Author Topic: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval  (Read 6304 times)

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
P2P-Picks may be down for a number of loan loads beginning 11 Jun 2014 6 a.m. LC changed the way that users may retrieve the loan application data. This number may be 0 as we've attempted to adjust our scripts to account for the new method of downloading. However, it may take more than 0 attempts to debug our approach. Please bear with us.

lascott

  • Hero Member
  • *****
  • Posts: 1433
    • View Profile
    • Appreciate my post and want to try LendingRobot? URL below
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #1 on: June 11, 2014, 08:43:23 AM »
Thanks for letting everyone know.
Tools I use: (main) BlueVestment: https://www.bluevestment.com/app/pricing + https://www.interestradar.com/ , (others) Lending Robot referral link: https://www.lendingrobot.com/ref/scott473/  & Peercube referral code: DFVA9Y

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #2 on: June 11, 2014, 09:15:44 AM »
It seems I got it working on try #1. The speed may need to be worked on but picks got made.

rawraw

  • Hero Member
  • *****
  • Posts: 2792
    • View Profile
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #3 on: June 11, 2014, 09:39:14 AM »
It seems I got it working on try #1. The speed may need to be worked on but picks got made.
Are you human?

lascott

  • Hero Member
  • *****
  • Posts: 1433
    • View Profile
    • Appreciate my post and want to try LendingRobot? URL below
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #4 on: June 11, 2014, 09:39:55 AM »
It seems I got it working on try #1. The speed may need to be worked on but picks got made.
Nice job.
I used it to do the manual picks ... when I did not see BlueVestment buy any notes.  May want to double check that that part is working.
Tools I use: (main) BlueVestment: https://www.bluevestment.com/app/pricing + https://www.interestradar.com/ , (others) Lending Robot referral link: https://www.lendingrobot.com/ref/scott473/  & Peercube referral code: DFVA9Y

PeerSocialLending

  • Jr. Member
  • **
  • Posts: 79
    • View Profile
    • Email
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #5 on: June 11, 2014, 10:11:10 AM »
BV got 1 note for me and I manually picked up a few. Thanks Bryce!
-Ryan
p2p lending blogger @ www.peersociallending.com

mchu168

  • Full Member
  • ***
  • Posts: 147
  • “Integrity is the ability to stand by an idea.”
    • View Profile
    • Email
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #6 on: June 14, 2014, 09:36:37 AM »
I use is P2P picks and Bluevestment to auto invest, but I've noticed very few notes have been automatically purchased over the past few days.  At feeding times I see that at least a couple of notes are meeting my criteria (Profit Maximizer, =<10%, D-G) but are not purchased with Bluevestment.  Is it that these notes are so popular that even Bluevestment's API fails "catch" them?   I guess the combination of 36mo term and F grade is the absolute sweet spot for active note pickers.

I wonder if others are seeing this too?

lascott

  • Hero Member
  • *****
  • Posts: 1433
    • View Profile
    • Appreciate my post and want to try LendingRobot? URL below
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #7 on: June 14, 2014, 10:35:14 AM »
I use is P2P picks and Bluevestment to auto invest, but I've noticed very few notes have been automatically purchased over the past few days.  At feeding times I see that at least a couple of notes are meeting my criteria (Profit Maximizer, =<10%, D-G) but are not purchased with Bluevestment.  Is it that these notes are so popular that even Bluevestment's API fails "catch" them?   I guess the combination of 36mo term and F grade is the absolute sweet spot for active note pickers. I wonder if others are seeing this too?
It is working for me. This may be obvious (and something I've messed up myself) but you may want to check your "Maintain minimum cash reserve" value within the "Manage BlueVestment Account Settings" dialogue.
Tools I use: (main) BlueVestment: https://www.bluevestment.com/app/pricing + https://www.interestradar.com/ , (others) Lending Robot referral link: https://www.lendingrobot.com/ref/scott473/  & Peercube referral code: DFVA9Y

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #8 on: June 14, 2014, 11:48:51 AM »
The issue is that P2P-Picks relies on LC providing its loan application data via the CSV or API, and that these are no longer the most timely way to get loan application data. It is affecting me personally and all of my clients, too, and there isn't much I can do about it except complain to LC, which I have done.

It used to be that when you clicked the link to get the CSV, LC would instruct its systems to generate a fresh CSV based on the loan applications that were available at that very moment. This, I imagine, was expensive for them when thousands of people were clicking at the magic hours. Hence why we saw rate limiters in place last November. In the last few days, they moved to a model of linking to a static CSV available for download that is updated for freshness just once every minute. While this relieves the stress off their servers, it has had some unintended (or perhaps intended) consequences.

Here is the rub. LC releases the loans onto their platform around 30-40 second after the hour. The loans are available to website users a full 20-30 seconds before they ever even appear in the CSV. The API has always been out of date a number of seconds, just like the CSV is now. Now, the absolute fastest way to get loan application data is to use the website interactively via their filter processes (or to screen scrape for automation). This puts all of us who want to do active loan selection in a position of either sitting at our terminals every release twitching like some kid playing a video game or breaking the rules (even more than we already have to do) and resorting to screen scraping loan data off the website in order to automate things. Of course, the web pages don't even have a majority of the credit variables anyway, so this may be of limited value.

I have witnessed this occur on both the fractional and whole loan side. Reliably, at 30 after the hour, new loans will appear. Within a few seconds, the "good stuff" (to the extent you wish to believe in such a concept) is either purchased outright (in the whole loan market) or is mostly already full (on the fractional side) by the time the other two loan data retrieval methods are updated. Here is an example from 6pm a couple nights ago to illustrate:


5:55:00 - Initial count: 31 whole loans on platform
6:00:05 - Unused whole loans expire (12 hour rule): 19 whole loans on the platform
6:00:30 - 170 new whole loans added to platform (as viewed from the webpage) - total inventory = 189
6:00:45 - 24 whole loans disappear off the platform - total inventory = 165
6:01:00 - browseNotes.csv and LoansBrowseLoans update with just 165 whole loans in them

On the fractional side, some of my picks will already be completely full and most will be at least partially full by the time I get the loan apps to analyze.

Why would LC do this? I don't want to go into pet theories, even though I have some, before allowing LC a chance to respond. And to be clear LC is absolutely still providing an even playing field. Same rules for everyone. It's just that this change has benefitted a certain type of user (the twitching kid or the scraper).

My recommendation is to synch the timing of all forms of loan application data, website, CSV, and API.





Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #9 on: June 14, 2014, 12:12:29 PM »
Here is an example from 6pm a couple nights ago to illustrate:

5:55:00 - Initial count: 31 whole loans on platform
6:00:05 - Unused whole loans expire (12 hour rule): 19 whole loans on the platform
6:00:30 - 170 new whole loans added to platform (as viewed from the webpage) - total inventory = 189
6:00:45 - 24 whole loans disappear off the platform - total inventory = 165
6:01:00 - browseNotes.csv and LoansBrowseLoans update with just 165 whole loans in them

Thanks Bryce for this analysis.

From credit risk perspective, I cannot afford to make underwriting decisions without the information from the CSV file.  Therefore, I'd rather work with the 165 from CSV file rather than the 189 from the webpage.

I do not gamble with my investments.

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #10 on: June 14, 2014, 12:18:55 PM »
I agree Fred. It may be beneficial for you to repost in the main forum. I'd like a full discussion there as I find this so important.

rawraw

  • Hero Member
  • *****
  • Posts: 2792
    • View Profile
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #11 on: June 14, 2014, 04:34:13 PM »
Anil's doing work on this subject of how much speed matters.  It'll be interesting when we get the results soon

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #12 on: June 14, 2014, 05:16:45 PM »
To do this work, one would need to collect the fill time and then wait a long time to correlate to outcomes. Even then, stupid models may also act quickly, attenuating the results.

rawraw

  • Hero Member
  • *****
  • Posts: 2792
    • View Profile
Re: P2P-Picks Down While Adjusting for New LC browseNotes Retrieval
« Reply #13 on: June 14, 2014, 09:15:30 PM »
To do this work, one would need to collect the fill time and then wait a long time to correlate to outcomes. Even then, stupid models may also act quickly, attenuating the results.
Yea, he's been capturing the data for the past 10 months or so and captured the fill time.  He posted the first part of the multi-part analysis the other day