Lend Academy Network Forum

Lending Club Discussion => Investors - LC => Topic started by: viking on August 30, 2013, 06:20:42 AM

Title: UPDATE: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API
Post by: viking on August 30, 2013, 06:20:42 AM
One of the most popular sites that is using an API to capture Lending Club's loans is Interest Radar (IR). However, it appears that many loans are already gone by the time IR is able to display them on their website.

Here is an example of the number of available E-G loans on LC's (blue) and IR's (red) websites as they were released on 8/26/13 at 2 PM (each data point is spaced 4-5 sec apart):
(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Fwww.lendacademy.com%2Fforum%2Findex.php%3Faction%3Ddlattach%3Btopic%3D1507.0%3Battach%3D343&hash=39033a32808c9427c356f59024a0e38d)

Only about half the the number of LC's loans were available on IR when they were first posted.
The graph also shows that there were several "new" loans released throughout the 14 min period after the initial spike (possibly from loans released from holds in the shopping carts). Many of these loans were not detected by IR because the available loans are only captured at intervals.

I am guessing (but I don't know), that the Auto-buy on IR would use loans from the same pool as shown on IR's website and many LC loans may therefore not be purchased. Furthermore, I believe that the Auto-buy is only executed once, so the "new" loans may not be purchased? Rev??

It would be interesting to know if the other Auto-buy sites are also lagging LC's website (I would presume that they do!)?  I suspect that LC is releasing the loans on their website before the data is even available to the API? Is that possible...?


UPDATED 9/18/13. Because of a post by Rev today (http://www.lendacademy.com/forum/index.php?topic=1583.msg12564#msg12564), I compared the number of available loans on LC and IR again (9/18/13 at 2 PM); IR's posting time is now much improved! Thanks, Rev!
(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Fwww.lendacademy.com%2Fforum%2Findex.php%3Faction%3Ddlattach%3Btopic%3D1507.0%3Battach%3D362&hash=010d3a122ed22478c7d7643fb03b2534)


[attachment deleted by admin]
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: jpildis on August 30, 2013, 08:46:30 AM
Great analysis... I've seen the same thing and it's very troubling.  I would love to see a comparison between the available auto-invest tools that are currently available.  Anyone else up for a bake-off?
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: core on August 30, 2013, 09:09:32 AM
Anyone else up for a bake-off?

Heck yeah!  Excellent idea which could prove fruitful in the long run.  I think that has the potential to turn into a food fight and I'm hungry for some entertainment around here.  P2P Throwdown!!!

Almost makes me want to whip up a batch of cupcakes.  Unfortunately there isn't the time.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: LC Adv on August 30, 2013, 09:30:45 AM
If you want more speed locate your server in Santa Monica. You can probably cut your ping time in 1/2 although a microsecond is all you really need
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: brycemason on August 30, 2013, 09:34:04 AM
Experimenting with various methods of auto buying for my own site has proven quite the learning experience. I totally get this chart.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: core on August 30, 2013, 09:39:41 AM
If you want more speed locate your server in Santa Monica. You can probably cut your ping time in 1/2 although a microsecond is all you really need

Cutting your ping time in half doesn't help you if your approach/algorithm is causing you extra round trips compared to the other guy and/or you're not taking advantage of all that is available to you.   Also, an extra 100ms ping is hardly significant if you've got a 5000ms sleep between hits.

Speed for something like this is 99% what you do, and 1% speeding up how you do it. 
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: Fred on August 30, 2013, 02:22:33 PM
I suspect that LC is releasing the loans on their website before the data is even available to the API? Is that possible...?

If this is the case, then the API approach is inferior to that of page-scrapping approach.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: TonySaunders on August 30, 2013, 04:43:51 PM
Interest Radar has to somehow attempt to buy notes for ALL of their many clients during the 2-3 seconds that the best notes are available. I don't see how they can possibly succeed, and my experience with IR tells me that they are not succeeding at securing the best notes.

It's worth noting that the more capable the API, the worse this problem is. There's not enough of the best notes for everyone to have as much as they want and faster automation just means you get left out faster.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: viking on August 30, 2013, 04:52:32 PM
Interest Radar has to somehow attempt to buy notes for ALL of their many clients during the 2-3 seconds that the best notes are available. I don't see how they can possibly succeed, and my experience with IR tells me that they are not succeeding at securing the best notes.

It's worth noting that the more capable the API, the worse this problem is. There's not enough of the best notes for everyone to have as much as they want and faster automation just means you get left out faster.
As I wrote above, I suspect that Auto-buy buys from the same pool of loans that is available on IR's website. If so, then Auto-buy is not even capable of buying several of the LC available notes during the 1st minute after notes are posted on LC's website.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: core on August 30, 2013, 04:55:38 PM
faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: viking on August 30, 2013, 05:01:46 PM
faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!

I just want faster automation for ME..... ;D
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: Rob L on August 30, 2013, 05:17:41 PM
Mini nuclear arms race for LC notes
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: TonySaunders on August 30, 2013, 05:18:21 PM
faster automation just means you get left out faster

Hehehe best line I've read this week.  That's a good one!

:)
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: core on August 30, 2013, 05:50:05 PM
Mini nuclear arms race for LC notes

Yeah you aren't kidding.  Let's dig up Reagan and ask Him what to do.  "Lending Club, tear down this wall!"

Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: AmCap on August 30, 2013, 06:57:06 PM
I don't want to weigh in too heavily here, but if I can just make the larger point that folks forget that all of the LC universe seems focused on the very smallest fraction of LC notes that are available.  I see that in 2013, there have been 66,027 loans that LC has issued: assuming by "good notes" people mean E-G 400+, the number issued is 1,302.  Less than 2%.  Filters more restrictive than IR01 alone will yield even less. 

I really think we all should take Peter's post to heart and re-evaluate our expectations.  Unless users of other auto-invest services would like to weigh in, I think it's pretty unreasonable to target the same 2% of notes everybody else wants and be surprised to be shut out of them.  We may need to open some filters up...look at higher rated notes.  Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).  Expanding the "good note" universe to C and D notes expands the number of issued notes by a factor of 10. 
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: viking on August 30, 2013, 07:12:28 PM
I don't want to weigh in too heavily here, but if I can just make the larger point that folks forget that all of the LC universe seems focused on the very smallest fraction of LC notes that are available.  I see that in 2013, there have been 66,027 loans that LC has issued: assuming by "good notes" people mean E-G 400+, the number issued is 1,302.  Less than 2%.  Filters more restrictive than IR01 alone will yield even less. 

I really think we all should take Peter's post to heart and re-evaluate our expectations.  Unless users of other auto-invest services would like to weigh in, I think it's pretty unreasonable to target the same 2% of notes everybody else wants and be surprised to be shut out of them.  We may need to open some filters up...look at higher rated notes.  Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).  Expanding the "good note" universe to C and D notes expands the number of issued notes by a factor of 10.
Well, more than 50% of all E-G notes, in this example, were picked up in less than one minute. I expect that only a small fraction of those were 400+ notes. Including A-D notes may, of course, increase the absolute number of available notes.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: core on August 30, 2013, 07:16:02 PM
We may need to open some filters up...look at higher rated notes.  Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).

You may be right about that, opening up some filters.  But instead of buying a B note, wouldn't it be better to buy that E note that you really wanted... on Folio?   I am seriously not understanding why people are willing to take a 5-10% decrease in interest rate before buying Folio notes with an on-time payment or two already on them, at the rate the investor wanted, and possibly even at a small discount.  Is it just because the 3rd party sites don't have enough support for Folio at this time? 

I am honestly not understanding.   If there were no notes on Folio that precisely matched what everyone seems to want then I'd understand.  But there are seriously tons of them.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: thezfunk on August 30, 2013, 07:34:40 PM
We may need to open some filters up...look at higher rated notes.  Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).

You may be right about that, opening up some filters.  But instead of buying a B note, wouldn't it be better to buy that E note that you really wanted... on Folio?   I am seriously not understanding why people are willing to take a 5-10% decrease in interest rate before buying Folio notes with an on-time payment or two already on them, at the rate the investor wanted, and possibly even at a small discount.  Is it just because the 3rd party sites don't have enough support for Folio at this time? 

I am honestly not understanding.   If there were no notes on Folio that precisely matched what everyone seems to want then I'd understand.  But there are seriously tons of them.

I have thought about this too.  I thought about getting these favorable notes and letting them 'season' for a payment or two and then dumping them immediately on the secondary market.  Just rotate them out as fast as i can.  I hold the notes as little as possible and maybe not make my 22% but I'd take 18% :-)

I also have thought about buying notes on folio but it's still largely an unknown for me and I haven't taken the time to dig into it deeper.  I probably should.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: core on August 30, 2013, 07:49:35 PM
I also have thought about buying notes on folio but it's still largely an unknown for me and I haven't taken the time to dig into it deeper.  I probably should.

This seems to be the answer I hear every time someone mentions it.  Curiouser and curiouser.  It takes time to come here and complain about not finding any notes.  (And that activity is fun, I understand, don't get me wrong.)  But in the same amount of time one could have gotten fully invested in what one wanted in the first place.  And I'm not picking on you, thezfunk, everyone else says the same thing.

Here is my suggestion for 3rd party site developers:
When the user sets up their auto-buy filters, eagerly awaiting the next drop in 4 hours, show them this message:

Filters saved.

There are 189 notes matching your exact criteria right now at a discount.  Click here to buy them, or you may leave your filters in place and hope to catch a couple the next time LC releases some IF you're lucky.  You pick, Einstein.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: JDowding on August 30, 2013, 08:09:52 PM
We may need to open some filters up...look at higher rated notes.  Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).

You may be right about that, opening up some filters.  But instead of buying a B note, wouldn't it be better to buy that E note that you really wanted... on Folio?   I am seriously not understanding why people are willing to take a 5-10% decrease in interest rate before buying Folio notes

I can only speak for myself, but I tried Folio when I first got here (8/2012), thought the interface sucked, and haven't been back.     
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: core on August 30, 2013, 08:27:03 PM
I can only speak for myself, but I tried Folio when I first got here (8/2012), thought the interface sucked, and haven't been back.   

Certainly the interface does suck.  That's where the 3rd party developers can pick up the slack and make some coin at the same time.  Also make many people happy who wouldn't have been able to get invested otherwise.  But oops, it seems we are banning such 3rd parties from Folio these days??
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: Lovinglifestyle on August 30, 2013, 10:00:15 PM
My reason for not buying on Folio is that I haven't yet given up looking at the notes and total current debt balance at the time the note was issued--let alone present time--is not available on PeerCube for issued notes.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: AnilG on August 30, 2013, 11:43:51 PM
You will have detailed information for loans that are already issued on PeerCube very soon and ability to filter loans using your strategy for available loans. But PeerCube will not screen-scrape Foliofn site to show available notes. You will need to reconcile Foliofn notes with issued loan information yourself.

My reason for not buying on Folio is that I haven't yet given up looking at the notes and total current debt balance at the time the note was issued--let alone present time--is not available on PeerCube for issued notes.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: Rob L on August 31, 2013, 02:32:53 AM
Back to the chart for a moment. Doesn't that step function decline in available notes very early in the release imply a single buyer? With resolution of 2 sec it seems pretty unlikely multiple large buyers all happened to fall into that little window. Alternatively (and I've posted this before ) it may simply be a natural artifact of how the LC software works under heavy load. It's complicated stuff.


Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: viking on August 31, 2013, 04:26:27 AM
Back to the chart for a moment. Doesn't that step function decline in available notes very early in the release imply a single buyer? With resolution of 2 sec it seems pretty unlikely multiple large buyers all happened to fall into that little window. Alternatively (and I've posted this before ) it may simply be a natural artifact of how the LC software works under heavy load. It's complicated stuff.
Could you please explain "natural artifact of how the LC software works under heavy load"?
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: Fred on August 31, 2013, 10:08:57 AM
Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).  Expanding the "good note" universe to C and D notes expands the number of issued notes by a factor of 10.

+1

Not only does the universe expand by a factor of 10; the risks also decrease as significantly.  For loans originated in 2010, the charge-off rates of B loans (~6%) are about 1/3 of those of F+G loans (~17%).

These following charts are from LC Form 10-K:

(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Fwww.sec.gov%2FArchives%2Fedgar%2Fdata%2F1409970%2F000119312513329399%2Fg545900dsp043.jpg&hash=9e7cfdaa79a13fd86782d2299078b42f)

(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Fwww.sec.gov%2FArchives%2Fedgar%2Fdata%2F1409970%2F000119312513329399%2Fg545900dsp047.jpg&hash=df09e401060f40821bc98dfb3bf5e08c)
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: Rob L on August 31, 2013, 05:42:51 PM
Could you please explain "natural artifact of how the LC software works under heavy load"?
No, not at all; not that smart. I'm saying that complex things sometimes behave in unpredictable non-deterministic ways (even to those that built them). The LC software I think probably can be said to be complex and is being exposed to loads it has not seen before (at least not in actual operation). I do believe your graph is "ground truth" whatever the reason(s), and it ain't good.
Title: Re: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API.
Post by: rawraw on September 01, 2013, 12:30:30 PM
I don't want to weigh in too heavily here, but if I can just make the larger point that folks forget that all of the LC universe seems focused on the very smallest fraction of LC notes that are available.  I see that in 2013, there have been 66,027 loans that LC has issued: assuming by "good notes" people mean E-G 400+, the number issued is 1,302.  Less than 2%.  Filters more restrictive than IR01 alone will yield even less. 

I really think we all should take Peter's post to heart and re-evaluate our expectations.  Unless users of other auto-invest services would like to weigh in, I think it's pretty unreasonable to target the same 2% of notes everybody else wants and be surprised to be shut out of them.  We may need to open some filters up...look at higher rated notes.  Maybe a B note won't kill us (that is, you'll still get better yield than a lot of debt instruments available).  Expanding the "good note" universe to C and D notes expands the number of issued notes by a factor of 10.
This is what LC told me when I told them this forum's concern a month or two ago.
Title: Re: UPDATE: Is even the API too "Slow"? Many Loans are Gone Before Caught by the API
Post by: viking on September 18, 2013, 05:44:24 PM
UPDATED 9/18/13. Because of a post by Rev today (http://www.lendacademy.com/forum/index.php?topic=1583.msg12564#msg12564), I compared the number of available loans on LC and IR again; IR's posting time is now much improved! Thanks, Rev!
(see first post in this thread with an additional graph)