Author Topic: UPDATE: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API  (Read 10174 times)

viking

  • Sr. Member
  • ****
  • Posts: 370
    • View Profile
    • Email
One of the most popular sites that is using an API to capture Lending Club's loans is Interest Radar (IR). However, it appears that many loans are already gone by the time IR is able to display them on their website.

Here is an example of the number of available E-G loans on LC's (blue) and IR's (red) websites as they were released on 8/26/13 at 2 PM (each data point is spaced 4-5 sec apart):


Only about half the the number of LC's loans were available on IR when they were first posted.
The graph also shows that there were several "new" loans released throughout the 14 min period after the initial spike (possibly from loans released from holds in the shopping carts). Many of these loans were not detected by IR because the available loans are only captured at intervals.

I am guessing (but I don't know), that the Auto-buy on IR would use loans from the same pool as shown on IR's website and many LC loans may therefore not be purchased. Furthermore, I believe that the Auto-buy is only executed once, so the "new" loans may not be purchased? Rev??

It would be interesting to know if the other Auto-buy sites are also lagging LC's website (I would presume that they do!)?  I suspect that LC is releasing the loans on their website before the data is even available to the API? Is that possible...?


UPDATED 9/18/13. Because of a post by Rev today (http://www.lendacademy.com/forum/index.php?topic=1583.msg12564#msg12564), I compared the number of available loans on LC and IR again (9/18/13 at 2 PM); IR's posting time is now much improved! Thanks, Rev!



[attachment deleted by admin]
« Last Edit: September 18, 2013, 06:27:09 PM by viking »

jpildis

  • Full Member
  • ***
  • Posts: 158
    • View Profile
    • Email
Great analysis... I've seen the same thing and it's very troubling.  I would love to see a comparison between the available auto-invest tools that are currently available.  Anyone else up for a bake-off?

core

  • Hero Member
  • *****
  • Posts: 1784
  • Your loss is my gain
    • View Profile
Anyone else up for a bake-off?

Heck yeah!  Excellent idea which could prove fruitful in the long run.  I think that has the potential to turn into a food fight and I'm hungry for some entertainment around here.  P2P Throwdown!!!

Almost makes me want to whip up a batch of cupcakes.  Unfortunately there isn't the time.

LC Adv

  • Newbie
  • *
  • Posts: 18
    • View Profile
    • Email
If you want more speed locate your server in Santa Monica. You can probably cut your ping time in 1/2 although a microsecond is all you really need

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Experimenting with various methods of auto buying for my own site has proven quite the learning experience. I totally get this chart.

core

  • Hero Member
  • *****
  • Posts: 1784
  • Your loss is my gain
    • View Profile
If you want more speed locate your server in Santa Monica. You can probably cut your ping time in 1/2 although a microsecond is all you really need

Cutting your ping time in half doesn't help you if your approach/algorithm is causing you extra round trips compared to the other guy and/or you're not taking advantage of all that is available to you.   Also, an extra 100ms ping is hardly significant if you've got a 5000ms sleep between hits.

Speed for something like this is 99% what you do, and 1% speeding up how you do it. 

Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
I suspect that LC is releasing the loans on their website before the data is even available to the API? Is that possible...?

If this is the case, then the API approach is inferior to that of page-scrapping approach.

TonySaunders

  • Full Member
  • ***
  • Posts: 194
    • View Profile
Interest Radar has to somehow attempt to buy notes for ALL of their many clients during the 2-3 seconds that the best notes are available. I don't see how they can possibly succeed, and my experience with IR tells me that they are not succeeding at securing the best notes.

It's worth noting that the more capable the API, the worse this problem is. There's not enough of the best notes for everyone to have as much as they want and faster automation just means you get left out faster.

viking

  • Sr. Member
  • ****
  • Posts: 370
    • View Profile
    • Email
Interest Radar has to somehow attempt to buy notes for ALL of their many clients during the 2-3 seconds that the best notes are available. I don't see how they can possibly succeed, and my experience with IR tells me that they are not succeeding at securing the best notes.

It's worth noting that the more capable the API, the worse this problem is. There's not enough of the best notes for everyone to have as much as they want and faster automation just means you get left out faster.
As I wrote above, I suspect that Auto-buy buys from the same pool of loans that is available on IR's website. If so, then Auto-buy is not even capable of buying several of the LC available notes during the 1st minute after notes are posted on LC's website.

core

  • Hero Member
  • *****
  • Posts: 1784
  • Your loss is my gain
    • View Profile
faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!

viking

  • Sr. Member
  • ****
  • Posts: 370
    • View Profile
    • Email
faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!

I just want faster automation for ME..... ;D
« Last Edit: August 30, 2013, 05:17:11 PM by viking »

Rob L

  • Hero Member
  • *****
  • Posts: 2058
    • View Profile
Mini nuclear arms race for LC notes

TonySaunders

  • Full Member
  • ***
  • Posts: 194
    • View Profile
faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!

:)

core

  • Hero Member
  • *****
  • Posts: 1784
  • Your loss is my gain
    • View Profile
Mini nuclear arms race for LC notes

Yeah you aren't kidding.  Let's dig up Reagan and ask Him what to do.  "Lending Club, tear down this wall!"


AmCap

  • Full Member
  • ***
  • Posts: 248
    • View Profile
    • Email
I don't want to weigh in too heavily here, but if I can just make the larger point that folks forget that all of the LC universe seems focused on the very smallest fraction of LC notes that are available.  I see that in 2013, there have been 66,027 loans that LC has issued: assuming by "good notes" people mean E-G 400+, the number issued is 1,302.  Less than 2%.  Filters more restrictive than IR01 alone will yield even less. 

I really think we all should take Peter's post to heart and re-evaluate our expectations.  Unless users of other auto-invest services would like to weigh in, I think it's pretty unreasonable to target the same 2% of notes everybody else wants and be surprised to be shut out of them.  We may need to open some filters up...look at higher rated notes.  Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).  Expanding the "good note" universe to C and D notes expands the number of issued notes by a factor of 10.