UPDATE: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API

[viking]

One of the most popular sites that is using an API to capture Lending Club's loans is Interest Radar (IR). However, it appears that many loans are already gone by the time IR is able to display them on their website.

Here is an example of the number of available E-G loans on LC's (blue) and IR's (red) websites as they were released on 8/26/13 at 2 PM (each data point is spaced 4-5 sec apart):


Only about half the the number of LC's loans were available on IR when they were first posted.
The graph also shows that there were several "new" loans released throughout the 14 min period after the initial spike (possibly from loans released from holds in the shopping carts). Many of these loans were not detected by IR because the available loans are only captured at intervals.

I am guessing (but I don't know), that the Auto-buy on IR would use loans from the same pool as shown on IR's website and many LC loans may therefore not be purchased. Furthermore, I believe that the Auto-buy is only executed once, so the "new" loans may not be purchased? Rev??

It would be interesting to know if the other Auto-buy sites are also lagging LC's website (I would presume that they do!)?  I suspect that LC is releasing the loans on their website before the data is even available to the API? Is that possible...?


UPDATED 9/18/13. Because of a post by Rev today (http://www.lendacademy.com/forum/index.php?topic=1583.msg12564#msg12564), I compared the number of available loans on LC and IR again (9/18/13 at 2 PM); IR's posting time is now much improved! Thanks, Rev!



[attachment deleted by admin]

[jpildis]

Great analysis... I've seen the same thing and it's very troubling.  I would love to see a comparison between the available auto-invest tools that are currently available.  Anyone else up for a bake-off?

[core]

Anyone else up for a bake-off?

Heck yeah!  Excellent idea which could prove fruitful in the long run.  I think that has the potential to turn into a food fight and I'm hungry for some entertainment around here.  P2P Throwdown!!!

Almost makes me want to whip up a batch of cupcakes.  Unfortunately there isn't the time.

[LC Adv]

If you want more speed locate your server in Santa Monica. You can probably cut your ping time in 1/2 although a microsecond is all you really need

[brycemason]

Experimenting with various methods of auto buying for my own site has proven quite the learning experience. I totally get this chart.

[core]

If you want more speed locate your server in Santa Monica. You can probably cut your ping time in 1/2 although a microsecond is all you really need

Cutting your ping time in half doesn't help you if your approach/algorithm is causing you extra round trips compared to the other guy and/or you're not taking advantage of all that is available to you.   Also, an extra 100ms ping is hardly significant if you've got a 5000ms sleep between hits.

Speed for something like this is 99% what you do, and 1% speeding up how you do it. 

[Fred]

I suspect that LC is releasing the loans on their website before the data is even available to the API? Is that possible...?

If this is the case, then the API approach is inferior to that of page-scrapping approach.

[TonySaunders]

Interest Radar has to somehow attempt to buy notes for ALL of their many clients during the 2-3 seconds that the best notes are available. I don't see how they can possibly succeed, and my experience with IR tells me that they are not succeeding at securing the best notes.

It's worth noting that the more capable the API, the worse this problem is. There's not enough of the best notes for everyone to have as much as they want and faster automation just means you get left out faster.

[viking]

Interest Radar has to somehow attempt to buy notes for ALL of their many clients during the 2-3 seconds that the best notes are available. I don't see how they can possibly succeed, and my experience with IR tells me that they are not succeeding at securing the best notes.

It's worth noting that the more capable the API, the worse this problem is. There's not enough of the best notes for everyone to have as much as they want and faster automation just means you get left out faster.

As I wrote above, I suspect that Auto-buy buys from the same pool of loans that is available on IR's website. If so, then Auto-buy is not even capable of buying several of the LC available notes during the 1st minute after notes are posted on LC's website.

[core]

faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!

[viking]

faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!

I just want faster automation for ME.....

[Rob L]

Mini nuclear arms race for LC notes

[TonySaunders]

faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!

[core]

Mini nuclear arms race for LC notes

Yeah you aren't kidding.  Let's dig up Reagan and ask Him what to do.  "Lending Club, tear down this wall!"

[AmCap]

I don't want to weigh in too heavily here, but if I can just make the larger point that folks forget that all of the LC universe seems focused on the very smallest fraction of LC notes that are available.  I see that in 2013, there have been 66,027 loans that LC has issued: assuming by "good notes" people mean E-G 400+, the number issued is 1,302.  Less than 2%.  Filters more restrictive than IR01 alone will yield even less. 

I really think we all should take Peter's post to heart and re-evaluate our expectations.  Unless users of other auto-invest services would like to weigh in, I think it's pretty unreasonable to target the same 2% of notes everybody else wants and be surprised to be shut out of them.  We may need to open some filters up...look at higher rated notes.  Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).  Expanding the "good note" universe to C and D notes expands the number of issued notes by a factor of 10.