Author Topic: Lending Club to stop screen scraping?  (Read 9848 times)

PennySaved

  • Full Member
  • ***
  • Posts: 139
    • View Profile
Lending Club to stop screen scraping?
« on: March 16, 2014, 04:24:52 PM »
This article says Lending Club third-party auto-investing with Lending Club, as it stands today, is going to be cut off. ??

http://www.lendingmemo.com/lending-club-auto-investing-screen-scraping/

I started using Interest Radar (IR) a few months back to Auto-Invest and I really like it.  I don't know much about programming and the behind the scenes of how it works, but does IR use screen scraping and will IR at some point, not be workable?  I was thinking about transferring some money to my LC Roth IRA from another Roth account, but without having a easy way to Auto Invest, I am a bit worried.  What do others think?

core

  • Hero Member
  • *****
  • Posts: 1784
  • Your loss is my gain
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #1 on: March 16, 2014, 04:47:33 PM »
Lengthy blog post which really boils down to just two paragraphs directly relating to this "news":

Quote from: Simon Cunningham
As a result, it seems in Lending Club’s best interest to end the practice of screen-scraping and force investors to use their platform in a way that is both safe and consistent. I spoke with Lending Club’s COO Scott Sanborn recently, and he seemed to affirm that screen scraping will eventually be cut off, focusing on the issue of server-load:

He then quotes Mr. Sanborn:

“Screen scraping is not allowed under our terms of use, but it is important to understand that we put this policy in place specifically to address activity that slows the platform and degrades the user experience for other investors, and currently that is our emphasis in enforcing it (i.e. we monitor the platform, and if we see something happening that has an impact on the platform’s performance, we take action).”


Maybe I'm misinterpreting this, but that quote from Lending Club does not in any way imply that all screen scraping is going to be cut off tomorrow or any time in the immediate future.  I read that to mean that they have always monitored things and took action when problems arose.  And that they will continue to do so.  I don't see anything that says otherwise.

Now whether they might do so in the distant future, or that it would be wise, that's a different matter.  But I'd like to see something a bit more concrete from Simon.  I'm not saying he's wrong, I'd just like to know what the basis was for this recent news?  That quote certainly doesn't cut it.

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Re: Lending Club to stop screen scraping?
« Reply #2 on: March 16, 2014, 04:55:45 PM »
You'll know that screen scraping is being killed when LC API starts to provide the missing functionality necessary to conduct business such as checking cash balance and the number of loans on the platform. The list has gotten smaller over the last year with the ability to query a subset of browseNotes and tokens.

Mr. Dumpling

  • Newbie
  • *
  • Posts: 9
    • View Profile
    • Email
Re: Lending Club to stop screen scraping?
« Reply #3 on: March 18, 2014, 01:22:05 PM »
I was going to start a new thread, however my question seems to be related directly to this post.

Over the past few days I have noticed a 300+ increase in notes (Some D,E & F) on the lending club platform. I am not logging in at feeding times and I have logged in a couple of times throughout each day. There has been a consistent 1000+ notes avalible.

Have they cut the screen scraping already?

I personally, love it becuase I have never used any thrid-party sites and did'nt plan to. Every time I log in I pick up D & E notes with my filters!

loan_trader

  • Newbie
  • *
  • Posts: 19
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #4 on: March 18, 2014, 01:51:00 PM »
I also noticed that there are about 300 more loans on here then there use to be. I don't think that has anything to do with screen scraping issue.

I wonder why it could be...

yojoakak

  • Hero Member
  • *****
  • Posts: 766
    • View Profile
    • Check out my Greasemonkey/Tampermonkey script for LendingClub here
    • Email
Re: Lending Club to stop screen scraping?
« Reply #5 on: March 18, 2014, 04:29:08 PM »
LendingClub should sell co-location services to the Mega-Traders and then use that money to upgrade their servers.

99greenballoons

  • Newbie
  • *
  • Posts: 8
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #6 on: March 18, 2014, 05:39:40 PM »
I just hope that if Lending Club does "end screen-scraping", they'll go by the definition presented in the article.

If their servers are buckling from individual accounts each executing thousands of hits per second, then throttle them.  Actually, I thought they already were  ???

Bohb Daishi

  • Sr. Member
  • ****
  • Posts: 481
  • I eat free lunches
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #7 on: March 19, 2014, 03:32:34 AM »
I think that there would be an uproar if LC got rid of "screen scraping". They would run into many problems:
  • How could they distinguish between someone like me (who takes only a minute or two to manually click on 100+ notes) and someone who is using a scraper?
  • How would investors monitor the collections log, FICO trend, and payment status on a large portfolio without some form of automation?
  • How do they handle the public relations nightmare that would come from disabling 3rd party tools/websites?
  • Most importantly, let's assume they throttled it or did something like "you can only click on 20 notes every minute". How can they stop a scraper from just hitting 19 notes every minute?

If they really want to get rid of screen scraping, they could simply increase the functionality of their platform/API in such a way that would make scraping unnecessary.
There are three ways to make a living in this business: be first, be smarter, or cheat.

core

  • Hero Member
  • *****
  • Posts: 1784
  • Your loss is my gain
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #8 on: March 19, 2014, 06:09:34 AM »
They would run into many problems

This was discussed at length in the past, I think in particular the "Fairer solutions..." thread, so I won't rehash it all here.  In short, they cannot stop screen scraping.  Not without it turning into a witch hunt, where you have to explain to them how you went 5 hours without a restroom break, why you were trading all day on Sunday instead of going to church, etc.  Anything they might do is easily defeated, except for the judgement of a human and the grand LC inquisition.  That is why it must necessarily come to that.  Nothing else will work.

I am still waiting to hear what makes Simon think that Lending Club "will end" screen scraping and 3rd party tools, since Sanborn didn't say anything had changed.  This story could have been posted in 2009 and it would have been just as valid/invalid.  It has always been in the terms as far as I know.   NOTHING is new.   Nobody (publicly) said they are to stop screen scraping except Simon Cunningham.

This all seems like picking out the following in Lending Club's prospectus:
Quote from: Lending Club prospectus, p. 25
We have incurred net losses in the past and have only recently become operating cash flow positive. If we are unable to sustain our positive growth and become insolvent or bankrupt, you may lose your investment.
... and coming out with this headline for one's blog:
Lending Club says it will go bankrupt
« Last Edit: March 19, 2014, 06:29:33 AM by core »

NorseBoric

  • Newbie
  • *
  • Posts: 8
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #9 on: March 19, 2014, 01:54:41 PM »
I've been screen-scraping for thirteen years. It's nigh impossible to prevent if the scraper knows what they're doing. There are only two ways to effectively deter it:

1. Provide the desired functionality via an API in the first place.
2. Continuously change your site markup.

Even assuming they do improve the API functions, there is also the issue of data latency. According to the API documentation, the API data is cached and not refreshed as quickly as one would see by simply hitting "refresh" on the website. This is a wee bit problematic, so they would also need to bring that API cache up to speed to encourage wider adoption. And of course there's their ridiculous process of obtaining API tokens as opposed to simply clicking "get/show token" like every other vendor I've ever worked with in my entire professional life.

BlueVestment

  • Full Member
  • ***
  • Posts: 100
    • View Profile
    • Email
Re: Lending Club to stop screen scraping?
« Reply #10 on: March 19, 2014, 05:41:51 PM »
I've been screen-scraping for thirteen years. It's nigh impossible to prevent if the scraper knows what they're doing. There are only two ways to effectively deter it:

1. Provide the desired functionality via an API in the first place.
2. Continuously change your site markup.

Even assuming they do improve the API functions, there is also the issue of data latency. According to the API documentation, the API data is cached and not refreshed as quickly as one would see by simply hitting "refresh" on the website. This is a wee bit problematic, so they would also need to bring that API cache up to speed to encourage wider adoption. And of course there's their ridiculous process of obtaining API tokens as opposed to simply clicking "get/show token" like every other vendor I've ever worked with in my entire professional life.

Don't forget adding CAPTCHA. Though even those can be gotten around.

NorseBoric

  • Newbie
  • *
  • Posts: 8
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #11 on: March 20, 2014, 03:59:03 PM »
Don't forget adding CAPTCHA. Though even those can be gotten around.
OCR has been a "solved problem" for about ten years. Granted it still isn't perfect, but at the end of the day, if you can articulate how a human does something, you can automate it.

Bohb Daishi

  • Sr. Member
  • ****
  • Posts: 481
  • I eat free lunches
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #12 on: March 21, 2014, 02:56:09 AM »
This alone would drop LC's computing footprint from a dozen racks in a data center to a laptop in the janitor's closet.

That would have to be a pretty beefy laptop.
There are three ways to make a living in this business: be first, be smarter, or cheat.

core

  • Hero Member
  • *****
  • Posts: 1784
  • Your loss is my gain
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #13 on: March 21, 2014, 03:04:12 AM »
The hardware is already there.  The janitor does have a laptop in his closet.  How else is he supposed to write these site updates that turn out oh-so-well?

Interesting idea on the server pushing out the new loans though.  That would eliminate a lot of hammering.  The devil is in the details, but with some randomization on the order of the listeners I think it could be done fairly.

core

  • Hero Member
  • *****
  • Posts: 1784
  • Your loss is my gain
    • View Profile
Re: Lending Club to stop screen scraping?
« Reply #14 on: March 21, 2014, 03:29:17 AM »
Multicast is the key word there. That fully addresses the issue of fairness.

The last time I even looked at multicast was a good 15 years ago and nobody was using it.  I just did a quick search to see what's changed, and it seems even today there's no guarantee you'll be able to receive the packets depending on where you are.  Surely you can't be suggesting LC implements something that will only work for certain ISPs?  It's gotta work for everybody or nobody.

If my information is old then I apologize in advance.  I obviously do not keep up with the latest developments so I could be wrong.