Lend Academy Network Forum

Lending Club Discussion => Foliofn - LC => Topic started by: Fred93 on January 12, 2018, 01:01:20 AM

Title: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 12, 2018, 01:01:20 AM
I buy notes on Folio, using the API.

When I place a buy order on Folio, the responses I get are a mix of NOTE_NOT_AVAILABLE and SUCCESS_PENDING_SETTLEMENT.

I'm going to be talking about just those orders that resulted in SUCCESS_PENDING_SETTLEMENT.  Those are the ones I think I bought.

A significant fraction of those never end up in my account.  I've been wondering why.

A few months ago I took a day's purchases, from the log written by my software, and checked them against my notes file to see how many actually arrived and how many did not.  I got a 50% failure rate!  Yipes.

I did this check again last month, and got a 38% failure rate.  38% of the time, the note does not show up.  (For the purposes of this little experiment, 38% and 50% are approximately the same number.)

I've written to LC a couple of times about this.  Most recently their response has been that the specific examples I sent them were all cancelled by payment.  I have quizzed them about the payment timeline, and how they decide which orders to cancel.  They say that the window of opportunity for payment after I place an order is only one day.  That would be 1 day out of about 30 days (because payments come about once every 30 days), so that would be 3.33% expected to fail.  But of course they only process payments on weekedays, so one of the 5 weekdays absorbs the hazard of the two weekend days, so maybe that would legitimately get them to 4.66%

Long way from 50%.

So I'd like to hear the experience of other LC folio buyers.  I'd like to know if I'm alone in this.

What fraction of successful orders fail to arrive in your account?

Do many of you see a large fraction of buy orders cancelled, ie say SUCCESS when you place the order, but it never makes it to your account?

LC is asking me questions aimed at a theory that my selection process accurately predicts who will pay on the day I order.  I've looked over what I'm doing, and I just don't think that's possible.

PS: Jheizer made a brief comment in another thread that he had seen "like 50%" failure.  I can send that to LC, but I'd prefer to send comments from several different people, showing that many people have similar experiences.  That will allow us to get past their theories that this is caused by some strange thing I'm doing.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Dave101 on January 12, 2018, 01:59:59 AM
Since I started tracking I've bought 519 notes, 175 of which are not presently in my account. A few are in pending and a few are sold, but I doubt those add up to 175. Still not close to 50%.

I'll re-tool my process so I can get an exact number and check back in in a couple of days.

Incidentally... any idea why it takes notes 4-6 minutes to be visible in the API listing, after which time several are not available? It seems to coincide with how often the downloadable CSV file is updated. Screen scraping doesn't seem to be much better. I know notes sold are available for purchase instantly, since I've tested it across accounts.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 12, 2018, 03:26:01 AM
I found a simpler way to explain what the cancellation rate should be...

Payments are processed 5 days every week, and there are 52 weeks in a year.  5x52 = 260 payment processing days.  A loan generates a payment 12 times a year.   That's 12 payments in 260 days, or a note should see a payment about 4.6% of days. 

Therefore, note purchasers should see only about 4.6% of their note purchases cancelled for payment.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 12, 2018, 04:40:27 AM
Getting a bit more serious, I decided not to rely on the stats I previously got by manually counting one day here, a half a day there, etc.  I wrote a program to process my log files for the past 6 months, and produce complete stats...

(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Ffred93.com%2Ffbi%2FCapture-folio-purchase-stats.PNG&hash=ab6424ff4d915cc1e514912d4476a98a)

If the hazard is truly one day, as LC says, then the failure rate should be 12/(5x52) = 4.62%

The last couple of months look to be about 5x this number.  Months before Nov 17 have lower failure rates.  Don't know why.  I've asked LC whether they made a change in this area in Oct '17.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: AnilG on January 12, 2018, 05:24:28 AM
Your analysis assumes loan payments and issuance are processed equally every day and not batched. I seriously doubt that is the case. Also, typically loans don't stay in payment processing for only one day. From payment due date to when payment is processed, completed and the payment processing flag is removed could be as much as 5 days.

Getting a bit more serious, I decided not to rely on the stats I previously got by manually counting one day here, a half a day there, etc.  I wrote a program to process my log files for the past 6 months, and produce complete stats...

(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Ffred93.com%2Ffbi%2FCapture-folio-purchase-stats.PNG&hash=ab6424ff4d915cc1e514912d4476a98a)

If the hazard is truly one day, as LC says, then the failure rate should be 12/(5x52) = 4.62%

The last couple of months look to be about 5x this number.  Months before Nov 17 have lower failure rates.  Don't know why.  I've asked LC whether they made a change in this area in Oct '17.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 12, 2018, 06:12:43 AM
Your analysis assumes loan payments and issuance are processed equally every day and not batched. I seriously doubt that is the case.

I asked, and they told me it was run every day.   (Doesn't make it so, of course.)

Here's what they wrote just yesterday...
Quote
In regards to timing dynamics on Folio, the key is that settlement on Folio happens once a day, at 3 pm. That settlement job checks whether the loan has received a payment since the last time the job was run. If it has, then it cancels the trade, and the trade does not SETTLE.

When I sell notes, I see the sale order sit in some sort of limbo for a day or two, so I figured there was a settlement process that took several days.  (like stock market trades)  I don't have a clue WHY it would take several days, but based on the timing of the email and the sell order screen, I just guessed that it did.  That's why I asked the question. 

When I purchase a note, there's no UI that shows me the darn purchases in process at all, so I don't see them sitting in limbo, and I get no email, so I have no idea what the settlement timing is.  I just know some of them mysteriously never arrive in my account.


Also, typically loans don't stay in payment processing for only one day. From payment due date to when payment is processed, completed and the payment processing flag is removed could be as much as 5 days.

I have repeatedly asked them to lay out the payment processing timeline vs folio.  They've answered with little pieces, and there is much that is still unclear to me.

Keep in mind that I'm talking about notes that I PURCHASED.  For me to purchase them, they have to be offered for sale.  I see sale offers routinely get cancelled for payment processing.  (I don't know what the timeline is exactly for that either.)  If the process that cancels sell orders due to payments runs once per day, then there should only be a 24 hour window during which I could buy a note for which a payment has been made.  The window is the time between these runs.

Seems to me that you or I could have designed this so that there was no need to tell the buyer SUCCESS and then cancel the thing silently mysteriously a day or so later.

LC needs to lay out the timeline. 

My thinking at this time is that the person who designed the process is probably gone, and the people maintaining the software don't really understand it, so they don't explain the timeline because they can't, and they can't explain my purchase failure rate because they have no idea what to look at.  I conclude this based on the difficulty I have had getting them to look into this, and the funky theories they've given me. 

For example one theory was that my purchase criteria maybe magically predicts who's gonna pay soon, making me a special case.  To counter this general category of excuse that supposes that I'm a special case, so should be ignored, I am attempting to gather info from other buyers.

Another point: LC folks claim surprise at the high fail numbers I've reported.  If my results were easily explained by the payment timeline, you'd think they would not be surprised.  And if they are surprised, and are engineers with a proper sense of curiosity, you'd think they'd want to investigate.


Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: apc3161 on January 12, 2018, 09:29:46 AM
I too had a similar problem with canceled payments, severe problem. I also talked to LC and we decided it was probably secondary to payments being too close to the sale date, such that payments automatically cancelled the sale. What I did to avoid this is stop purchasing notes (on-time notes) with payment date > 25 days (i.e. they have an upcoming payment). I did this recently. I'll see if this helps improve the situation.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 12, 2018, 04:57:26 PM
I'm thinkin I should add the "days since last payment" field to my log file.  That way I will be able to see whether this correlates to the notes that "disappear" during settlement.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Rob L on January 12, 2018, 05:21:37 PM
I want to add another voice and say I feel your pain so to speak. The comedy of Laurel and Hardy, Abbot and Costello, etc. are sadly in play.
The best of luck working this out. If the API worked I'd consider dabbling my toe in the water and maybe others would too.
When things are not a priority they can become an embarrassment; especially for the "tech" in "fintech".
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: hdsouza on January 12, 2018, 06:45:04 PM
I used to purchase notes with nsr investing earlier and they used to get about 1-2 notes/day. At that time I never really looks at the failures, although I did not see a few failures.
 
After recently automating  the process I purchased a bunch of Notes over the last few days and I noticed Like Fred that about HALF the notes were cancelled. This is after I received the SUCCESS_PENDING_SETTLEMENT.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 14, 2018, 08:21:29 AM
I think I have figured out the order processing timeline, and why so many buy orders are cancelled. 

In the past I'd been looking at historical data for months gone by, and LC gives us no way at all to track down what happened for notes we bought in prior months but did not settle, so I couldn't look at note by note details.  Generally, you can't look up the payment history on the note unless you own it, or it is offered for sale on Folio.  One exception is notes that you've tried to buy during the CURRENT month.  The "my account" page on Folio has links to the note detail for all these notes.  However, if you let time slip into the next month, all these links disappear.  This page provides an opportunity to look at the payment details of notes for which a buy order has been cancelled!  You've just got to do it before the links disappear!  That's what I did tonite.

Tonite I looked at notes I bought this month, looking up the status of the purchase transaction, and the note payment dates on the note detail pages for a considerable number of notes.  Lots of notes.  My fingers are very tired.

I now believe I know how the purchase timeline works.  Its pretty wonky.  We all know that lots of sell orders are cancelled for note payment, and I always figured this was done to protect the buyer.  Not so.   The sell orders are cancelled way too late to help buyers.  Sale offers are NOT cancelled while payments are processing.  As a result of this, the buyer has quite a hazard.   

Loan payment processing timeline...

Lets reference dates to the note "due date".  I'll call this 'D'.  This is when the payment processing window begins.  No payment has been made yet.  This is simply the day when a process starts.  On that day, or the closest business day, LC sends a request to the bank, via the ACH system, to grab some money from borrower's account.  A few days later, if the money has arrived, LC considers the payment "settled".  This is the "settlement date".  Lets call it 'S'.  (Due date and settlement date are shown on the note detail page.) 

Settlement date S can be = D+3, D+4, D+5 or D+6.  This occurs because LC only works on weekdays.  So if no weekend or holiday intervenes, settlement is D+3.  If a holiday is in there, then it's D+4.  If a weekend intervenes, it's D+5.  If you have both a weekend and a holiday, then settlement is D+6. 

So payment processing takes 3,4,5, or 6 days.  ...but there's more!

Now, the LC/folio note order processing timeline...

When I place a successful buy order on day 'B', LC does not process that the same day.  They process it the next day.  You can see this on your "my account" page.  The "order date" is always 1 day later than the day you placed the order.  Recent email from LC claimed the settlement process runs every day at 3PM, but this is misleading.  That may be the time that the software runs, but the time cutoff it uses is midnight.  Buy orders up to but not including midnight on 1/1/18 are logged by LC as ordered on 1/2/18.  In other words "order date" shown by LC is always B+1.

Now, when is a buy order cancelled?

A buy order is cancelled if B is in the range [D-1 thru S] inclusive.  In other words, the hazard window is one day more than settlement.

As an example, suppose a note had a due date on the 1st of a month.  Suppose there's a weekend and a holiday in there, so settlement is the 7th of the month.  Now you come along on the 7th and see this note listed for sale, and you buy it, and get a SUCCESS_PENDING_SETTLEMENT response.  Cool.  You think you've bought a note.  You haven't.  That order is gonna get cancelled for sure.

This is astonishing.  Its goofy that they know that they're gonna cancel the order if you buy, and they have known this for SEVERAL DAYS before you buy, but they let you buy, and then cancel.  Extra processing for them.  Extra processing for you.  Things are harder to track.  The right thing to do would have been to remove the sell order from folio once the window began, so that buyers didn't waste their time.  That's not what LC does. 

I've had orders where I bought on D-1 and the order was cancelled after I bought.  (For the record note# 126688481.  I bought 1/1/18.  LC order date 1/2/18.  Due 1/2/18.  Settle 1/5/18.)

I've had orders where I bought on D+6 and the order was cancelled after I bought.  (For the record note# 119577848.  I bought on 1/2/18.  LC order date 1/3/18.  Due 12/27/17.  Settle 1/2/18.)

So the bottom line is that for each payment, ie every month, each note has a window of 4,5,6, or 7 days in which if you order in that window, the order will be cancelled.  Most of the time its 4 or 6 days.  The other two possibilities require holidays, and there aren't very many of those.  There are 4 days of each non-holiday week which have 4 day windows, and three days that have 6 day windows.  From this we can calculate that the average window is 4.857 days.  (Not considering holidays, which would make it a little bit bigger.)

Payments come 12 times per year.  Each payment has a 4.857 day hazard window.  That's 12x4.857 hazard days per year.  (The occasional extra payment will bump this up a little, as will holidays.  Missed payments will bump it down a little.)

D can occur on any day of the year.  Yea, I know LC only processes on weekdays, but we've already handled that with all that stuff about settlement sometimes being D+3 and sometimes D+5.  So there are 365 days to consider, and ...

Probability that a note order is cancelled = 12x4.857 / 365 = 18.46%

Now you may recall that I've had some periods when I calculated much higher failure rates for my own account.  Keep in mind that this calculation is an average.   Different months have different numbers of holidays and different alignment to weekends.  Also the luck of the draw will cause some variation.  Finally some of the much higher numbers I calculated a few days ago were from manual calculations which because of the difficulty of calculating by hand were for single days, which of course will vary much more than months.  There is probably also some day-of-week bias when loans originate.  This number above is just an average calculation all other things being equal.

Here (again) are the calcs I did for the last 6 months on all buy orders in my account.
(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Ffred93.com%2Ffbi%2FCapture-folio-purchase-stats.PNG&hash=ab6424ff4d915cc1e514912d4476a98a)
November and December have a couple of holidays each, so they're expected to be a little higher.  These numbers look like they sorta agree with the theory.

The marketplace is simply designed so that about 1 out of 5 buy transactions will fail.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Dave101 on January 14, 2018, 10:53:29 AM
The marketplace is simply designed so that about 1 out of 5 buy transactions will fail.

This is great work, thank you.

Since we can't get the due date for a note until it's listed for sale and we've already bought it... there's no way to programatically avoid this window, is there?
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 14, 2018, 08:49:24 PM
Since we can't get the due date for a note until it's listed for sale and we've already bought it... there's no way to programatically avoid this window, is there?

Well yes, there are many imperfect ways...

The LC/folio API gives you a "days since last payment".  I use it to guess next payment date when calculating YTM, but for nothing else.  We certainly could use a next payment date guessed from last payment date to avoid notes whose orders might be cancelled due to payment.  It is imperfect in several ways.  The biggest one is that you want to know the date of the "next payment".  You could presume that the last payment was a regular payment, in which case the next payment will be on same day in the next month.  However, people make late payments and extra payments, so you don't know. 

Once upon a time, loans had a field called the "origination date" from which you could calculate the regular payment dates.  But then LC did two things to make that impractical.  First, the censored the origination dates by setting the day-of-month always to 1 to discourage competitors (possibly imagined competitors) reverse engineering some useful information out of their database.  Then they started allowing borrowers to pick the day-of-month that they wanted payments to fall on.  This was probably a good idea, but it adds a +- 15 day obscurity on top of the 30 day obscurity produced by censoring the origination date.

Finally, you can get the real "next payment date" from the note detail page.  Once a note is listed for sale, there is a link on the folio web page to the note detail page for each note offered.  However, you have to "screen scrape" to get the data.  LC discourages screen scraping.  (They've allowed me to do some minimal screen scraping in the past, when the API was so deficient that there were really basic things you couldn't do any other way.)  If you follow LC's rules you're supposed to do one web operation every 1 second.  Suppose you read the inventory and see 1000 notes newly offered for sale, and I wanted to examine the note detail pages for each.   Following the 1 second rule, it would take >20 minutes.  That's impractical in the extreme.  Even if they waived the 1 second rule, you couldn't access all those pages fast enough to get your bid in quickly enough to be practical.

I've had other ideas, but they are all extreme, and more work (which I avoid whenever possible).  Unless I see a big pot of gold at the end of this rainbow, I'm unlikely to do the work to try 'em.

Long ago I asked LC to add "next payment date" to the folio API, but as with most suggestions, there really isn't anyone at the receiving end to consider the suggestion or implement it.

Starting a day or so ago, after LC folks reacted with surprise and told me I must be accidentally choosing notes that are about to make a payment, I decided to add a field to my order log, showing "days since last payment".  After a few days of logging I should be able to see if indeed the ones that are getting cancelled are the ones where this number is large.  So far I can see that I the notes I pick seem to be scattered across the whole range of possible values of "days since last payment", which at least invalidates the adverse picking theory.  If these numbers line up with the cancellations nicely, then perhaps filtering on days since last payment is a pretty good method. 

Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Rob L on January 15, 2018, 09:40:02 AM
Since we can't get the due date for a note until it's listed for sale and we've already bought it... there's no way to programatically avoid this window, is there?

Well yes, there are many imperfect ways...

Finally, you can get the real "next payment date" from the note detail page.  Once a note is listed for sale, there is a link on the folio web page to the note detail page for each note offered.  However, you have to "screen scrape" to get the data.  LC discourages screen scraping.  (They've allowed me to do some minimal screen scraping in the past, when the API was so deficient that there were really basic things you couldn't do any other way.)  If you follow LC's rules you're supposed to do one web operation every 1 second.  Suppose you read the inventory and see 1000 notes newly offered for sale, and I wanted to examine the note detail pages for each.   Following the 1 second rule, it would take >20 minutes.  That's impractical in the extreme.  Even if they waived the 1 second rule, you couldn't access all those pages fast enough to get your bid in quickly enough to be practical.

Well, you only have to screen scrape notes you might intend to buy, right? So of the 1000 new ones listed how many are of any interest just using the data readily available from the API to exclude notes clearly uninteresting? Probably a small fraction, making screen scraping not so imposible, no? As an added bonus of screen scraping not only do you get the next payment date you also get the collections log. I'm talking in blissful ignorance as I'm not doing Folio buying myself, just curious about the topic. I always assumed screen scraping was required since the Folio API is so limited. However, I don't have the new API document and don't really know what data are available there.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: jheizer on January 15, 2018, 03:51:40 PM
I just tried to do a really rough guess on mine to see how it actually compared to my "I was annoyed too 50% guess".  I assumed for every time I logged out that a purchase ran that I only bought a single note.  I do see some had 2-3, but also some errors.  So if they do cancel each other out to average non per file, then about 20% of mine failed.  That's for the time period started about 2 weeks after the May mess in 2016 through October of that year. 
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: hdsouza on January 15, 2018, 05:05:52 PM
The concern about cancelled orders is further exacerbated by the fact that we hardly get good notes nowadays.
I remember about 6 months ago I would get about 3-4 good notes a day with NSR Investing. Now with my automation and  NSR Investing in parallel, I am lucky If I get one note in 2 days.. if it all.
.. And then after a couple days that note is suddenly cancelled!!!

Your analysis assumes loan payments and issuance are processed equally every day and not batched. I seriously doubt that is the case. Also, typically loans don't stay in payment processing for only one day. From payment due date to when payment is processed, completed and the payment processing flag is removed could be as much as 5 days.


A couple month ago I sent in a few hundred dollars to my LC account, and I believe (I am not positive) it took about 5 days to show up as available cash in my Lendingclub account. So probably the loan payments from borrowers are also delayed by 5 days.  Maybe that what Anil is referring to as Payment processing. I believe this 5 day time frame is atrocious. I have sent several transactions between financial institutions and they usually become available in 24 hours. I recently took a loan from Vanguard for several thousand dollars and it took 48 hours including the FULL APPROVAL process to show up in my bank account. 
Also delaying the payment from borrowers by 5 days is actually a huge loss to big investors, in terms of lost monies due to reinvestment. I guess LC is not affected as its no loss to them directly.

I wonder how long other Loan/mortgage companies take to process payments or is there some US regulation that states it should be 5 days?



Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 15, 2018, 05:27:52 PM
Well, you only have to screen scrape notes you might intend to buy, right? So of the 1000 new ones listed how many are of any interest just using the data readily available from the API to exclude notes clearly uninteresting?

You are correct.  Many optimizations possible.  I didn't explain every scheme.  I decided to stop typing at some point.

Quote
making screen scraping not so imposible, no?

Even doing no scraping,  I lose most of the notes I bid on.  Somebody gets to 'em before I do.  Perhaps I simply have many competitors and its all fair.  I don't know.  However, if I delay even a few seconds, I believe my results will be much worse.  Not impossible, but I think still impractical.

Each page of scrape takes >= 1 second, per the 1 second rule, so it seems to me this is stuff I want to avoid doing while evaluating newly posted offers.

Quote
As an added bonus of screen scraping not only do you get the next payment date you also get the collections log.

And the FICO history.

Quote
I always assumed screen scraping was required since the Folio API is so limited.

The LC folio API is indeed very limited.  I fix some of this by doing a join with a table I keep of loan characteristics, produced by processing the LC loan data files.  This gets me the loan credit variables without screen scraping.  Unfortunately, LC does not release info on new loans until the end of the quarter plus a few weeks.  So I don't have this data for very recent loans.  Right now, for example, I don't yet have data for 2017Q4 loans.

With increasing complexity (meaning I get to do more work), there are ways to do more, but each scheme comes with its own limitations.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 15, 2018, 05:36:14 PM
The concern about cancelled orders is further exacerbated by the fact that we hardly get good notes nowadays.
I remember about 6 months ago I would get about 3-4 good notes a day with NSR Investing. Now with my automation and  NSR Investing in parallel, I am lucky If I get one note in 2 days.. if it all.

Hmm.  I'm getting more now than I was a few months ago.

Maybe I'm now getting the notes you used to get?

Lack of volume in the secondary market is a big negative.  I can't invest nearly the amount I'd like to there.  Investing has been so slow that the time I invested in writing software and figuring out workarounds hasn't been worth the income I'm gonna make. 

I've been assuming that over time the secondary market would grow, but it doesn't seem like it has grown much if any. 

Aside: Hard to know if its growing, because LC doesn't publish any TRANSACTION VOLUME data.  I wrote to them last year explaining that they should do this, but judging from the response I got I don't think they understood.

I don't claim to know all the reasons the secondary market is so small, but LC's lack of attention contributes to the problem.  I believe their lack of attention is logical, because they're not makin' any money there.  Logical as that may be, it is a bit circular. 
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: jheizer on January 15, 2018, 05:40:35 PM
Investing has been so slow that the time I invested in writing software and figuring out workarounds hasn't been worth the income I'm gonna make. 

Exactly why I gave up and really have been trying to distance myself from all of LC stuff.  But more from the I'm not putting any more money into this so is it really worth this much time for what I do have here?  I decided no.  It stopped getting hobby play time so in turn it all just got shut off. 

Well, that and I wanted less notes of the same vintages that I already owned having started in 2015.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Dave101 on January 15, 2018, 11:52:10 PM
Even doing no scraping,  I lose most of the notes I bid on.  Somebody gets to 'em before I do.  Perhaps I simply have many competitors and its all fair.  I don't know.  However, if I delay even a few seconds, I believe my results will be much worse.  Not impossible, but I think still impractical.

I suspected that LC might offer institutional access earlier than we get to see notes on the market. Today it occurred to me that there's an easy way to test this. I put up a note for sale, a great seasoned note with a 5% discount that any reasonable algorithm would snatch up, and I timed it to see how long it would take to sell.

A little over 3 1/2 minutes later it appeared in the API, and was snatched up-- by my own process on another account. I now think that there are just several of us with automated processes doing purchasing, and sometimes someone else gets to the note first. But as you've said, with the lack of liquidity in this market, is it worth it to try and further optimize the process?
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on January 16, 2018, 07:13:43 AM
I suspected that LC might offer institutional access earlier than we get to see notes on the market.

Ah. Conspiracy theories.  Fun, but doesn't seem likely to me.  The volume on the secondary market is too low.  No reasonable sized "institution" would be able to use this tiny market.  Small hedge funds need to be >$100M in size to be reasonably profitable.  By the time you pay the lawyers, securities custodians, a software guy, a receptionist, the customer guy who scares up investors, office rent, share fees with finders, etc...  I believe the secondary market is dominated by retail investors.  The volume just isn't here.  Took me a year to get $100k invested in the secondary market.

Of course part of any good conspiracy theory would be that the volume of this hidden trading is ... hidden from us.  I just don't buy it.  (sorry for the pun)

Quote
Today it occurred to me that there's an easy way to test this. I put up a note for sale, a great seasoned note with a 5% discount that any reasonable algorithm would snatch up, and I timed it to see how long it would take to sell.

A little over 3 1/2 minutes later it appeared in the API, and was snatched up-- by my own process on another account.

Cool experiment.

Quote
I now think that there are just several of us with automated processes doing purchasing, and sometimes someone else gets to the note first. But as you've said, with the lack of liquidity in this market, is it worth it to try and further optimize the process?

My working presumption is that there are simply quite a few of us, in a fair competition.  I'm not sure of that tho.  I figure if its a fair competition, then I can win. 

In the primary market, I've been running my own software since January 2015, and in the first two years it almost always won.  In other words,  I got the loans I bid on a large fraction of the time.  I think this means I implemented well and figured out the tricks before other people did.  During 2017, my success rate decreased substantially.  There are now some days where I don't win any.  I am guessing this means other folks have caught up.  Luckily the volume in the primary market is large enough that I can still operate.  I only mention this to say I had a good experience automating the primary market, so figured the same was possible in the secondary market.

The investment of figuring out the secondary market note evaluation process, writing the software, is behind me, ie sunk cost.  Mostly I just let it run.  However, from time to time I do study where I'm failing and try to understand.  For example, the recent look into why so may successful transactions are cancelled.  I think I understand that pretty well now.  In the end there was no useful revelation there. 

It is a bit of a puzzle still why so many of my secondary market buy orders fail with NOTE_NOT_AVAILABLE status.  Could it be that there are enough other automated players, who have their software working well enough that they just beat me to the notes I want a good fraction of the time?  Are there tricks that they've figured out that I haven't?  Is the market broken in some way that just makes it look like somebody gets there first?  I don't know the answers.  The TOTAL lack of transparency in the secondary market makes it quite challenging to answer such questions.  One has to learn by experiment, as you have.

The protocol for the primary and secondary market, and the internal timing of the implementations, are so different that success in one doesn't tell you how to proceed in the other.  You've already discovered a lot of the minutia re how the secondary market works, the timing oddities etc. 

No conclusion.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Rob L on January 16, 2018, 10:10:29 AM
Even doing no scraping,  I lose most of the notes I bid on.  Somebody gets to 'em before I do.  Perhaps I simply have many competitors and its all fair.  I don't know.  However, if I delay even a few seconds, I believe my results will be much worse.  Not impossible, but I think still impractical.

I suspected that LC might offer institutional access earlier than we get to see notes on the market. Today it occurred to me that there's an easy way to test this. I put up a note for sale, a great seasoned note with a 5% discount that any reasonable algorithm would snatch up, and I timed it to see how long it would take to sell.

A little over 3 1/2 minutes later it appeared in the API, and was snatched up-- by my own process on another account. I now think that there are just several of us with automated processes doing purchasing, and sometimes someone else gets to the note first. But as you've said, with the lack of liquidity in this market, is it worth it to try and further optimize the process?

This is an interesting thread (from the bleachers). Just listening to the discussion there doesn't seem to be much mystery about why the best notes go so quickly; there are not very many of them and all the buyers more or less are attempting to buy the same "sure thing" notes. There's probably wide agreement among buyers regarding the definition of "sure thing" (it's not subtle; no secret sauce). Given the poor performance of recent vintages only a "sure thing" will do. Probably doesn't take a very large number of automated buying programs to produce the results observed. Clearly in this situation speed is king.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: hdsouza on January 19, 2018, 08:24:07 AM
I have seen several posts on screen scraping.  So i though I would mention that LC is against screen scraping.
I know it does not make sense, as there is no api to extract the original note data.. but that's their TOS (Maybe they give institutional investors access to some form of screen scraping and leave us, small fish out :) )

From one my emails with support:
Quote
I think what you are describing is referred to as "screen scraping" and per our Terms of Use, you may not “engage in the practices of "screen scraping," "database scraping," or any other activity with the purpose of obtaining content or other information

I did not engage the rep on the specific detail, as i had more pressing issues and live to fight another day.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Lovinglifestyle on January 19, 2018, 02:04:54 PM
I have seen several posts on screen scraping.  So i though I would mention that LC is against screen scraping.
I know it does not make sense, as there is no api to extract the original note data.. but that's their TOS (Maybe they give institutional investors access to some form of screen scraping and leave us, small fish out :) )

From one my emails with support:
Quote
I think what you are describing is referred to as "screen scraping" and per our Terms of Use, you may not “engage in the practices of "screen scraping," "database scraping," or any other activity with the purpose of obtaining content or other information

I did not engage the rep on the specific detail, as i had more pressing issues and live to fight another day.

Of course, I'm a dinosaur so I remember the "other information" such as actual words from the borrower that used to be provided.  I know most statistical folks don't care about that, and if I could count on the return "OF" my money I wouldn't either.  However, the current LC is not what I signed up for.  Meanwhile, 5% of my remaining 350 (nothing-but-green-arrows-up notes) lose credibility every month when the new FICO comes out.  Credibility matters for Folio sale prices.

Per the thread topic, I've noticed a disconnect regarding notes not available for me to sell on Folio because they are in pending process--except they aren't because they settled "yesterday" according to the log.  The next day I can usually list them.  Conversely, some pending sales are cancelled a day before the payment due date.  Does not account for 50% cancellations. 

No Foliofn at all would obviously be worse, and I like the sorting function for sellers that Folio offers. 
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Rob L on January 19, 2018, 05:25:08 PM
 
No Foliofn at all would obviously be worse, and I like the sorting function for sellers that Folio offers.

I'd like to second that. Foliofn could be better, but no Foliofn would be much worse. I left Prosper when they dropped Foliofn.
There are probably any number of us senior citizens who wonder what our estate executors and heirs could do with a portfolio of LC notes?
Likely they've never heard of LC. Like, whatszat?? Foliofn liquidation (like an estate sale) solves the problem. Heirs get cash and buyers get discounted notes.
One of the third parties should become the "estate sale" go-to for LC accounts. I've actually entertained the idea of starting an LLC to do just that.
Somebody doesn't do it, maybe I will. Being retired my cost of labor is zero and it isn't exactly rocket science.

Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: jheizer on January 19, 2018, 05:29:54 PM
Haha since all my buying is/was happening via my own code I gave my wife the tutorial one day on LC.  Login and kill this API key and just withdraw money as it becomes available.  Kind of nice having a master kill switch at least.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Half Right on January 22, 2018, 11:28:22 AM
I decided to stop reinvesting in LC when the news on LaPlanche first hit. Based on too many experiences with people having Madoff with my money etc. if the reputation gets smudged or tarnished at all , run ,dont walk away.
My portfolio has been running down since then and by luck the Private funds got sold off and closed out.
About 6 months ago I decided to look at Folio and buying "proven" notes... no more than 20 payments left, no late payments, etc etc etc. just for a test run.
I can now say in my opinion its absolutely not worth the effort.
I went back to my IT Maven who wrote the program for buying new notes directly from the platform and gave him my parameters. He comes back and said to me its a lot more work and based on his quick calculations it would not suffice for the amount of money I need to invest.
As for the notes I bought on Folio, I am not impressed with the return on those either.
There are currently much easier ways of investing large sums of money with a lot less work, while achieving higher returns with more security.
This business has become a boom to Goldman Sachs which lends out depositor money ( on which it pays 1.5%) at an average of 7% net( just my guess) and walks away with 5.5% with a ZERO investment. Now thats a nice return.

Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: wcwei5800 on July 08, 2018, 05:07:31 PM
Not sure if anyone have been experience this problem lately, but my cancellation rate has gone up in last few month to be around 50% consistently now. This really hurts since there are that many notes to invest on to begin with.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on July 08, 2018, 07:49:09 PM
Not sure if anyone have been experience this problem lately, but my cancellation rate has gone up in last few month to be around 50% consistently now. This really hurts since there are that many notes to invest on to begin with.

Sorry the first answer I posted an hour or so ago was wrong.  I researched my past writings and...

You should expect around 20% failure, based on how the secondary market works.

Here's the text I posted on Jan 14, 2018, somewhere on this forum, tho I can't find it here now, and also emailed to LC on same day...


I think I have figured out the order processing timeline, and why so many buy orders are cancelled.

In the past I'd been looking at historical data for months gone by, and LC gives us no way at all to track down what happened for notes we bought in prior months but did not settle, so I couldn't look at note by note details.  Generally, you can't look up the payment history on the note unless you own it, or it is offered for sale on Folio.  One exception is notes that you've tried to buy during the CURRENT month.  The "my account" page on Folio has links to the note detail for all these notes.  However, if you let time slip into the next month, all these links disappear.  This page provides an opportunity to look at the payment details of notes for which a buy order has been cancelled!  You've just got to do it before the links disappear!  That's what I did tonite.

Tonite I looked at notes I bought this month, looking up the status of the purchase transaction, and the note payment dates on the note detail pages for a considerable number of notes.  Lots of notes.  My fingers are very tired.

I now believe I know how the purchase timeline works.  Its pretty wonky.  We all know that lots of sell orders are cancelled for note payment, and I always figured this was done to protect the buyer.  Not so.   The sell orders are cancelled way too late to help buyers.  Sale offers are NOT cancelled while payments are processing.  As a result of this, the buyer has quite a hazard.   

Loan payment processing timeline...

Lets reference dates to the note "due date".  I'll call this 'D'.  This is when the payment processing window begins.  No payment has been made yet.  This is simply the day when a process starts.  On that day, or the closest business day, LC sends a request to the bank, via the ACH system, to grab some money from borrower's account.  A few days later, if the money has arrived, LC considers the payment "settled".  This is the "settlement date".  Lets call it 'S'.  (Due date and settlement date are shown on the note detail page.)

Settlement date S can be = D+3, D+4, D+5 or D+6.  This occurs because LC only works on weekdays.  So if no weekend or holiday intervenes, settlement is D+3.  If a holiday is in there, then it's D+4.  If a weekend intervenes, it's D+5.  If you have both a weekend and a holiday, then settlement is D+6.

So payment processing takes 3,4,5, or 6 days.  ...but there's more!

Now, the LC/folio note order processing timeline...

When I place a successful buy order on day 'B', LC does not process that the same day.  They process it the next day.  You can see this on your "my account" page.  The "order date" is always 1 day later than the day you placed the order.  Recent email from LC claimed the settlement process runs every day at 3PM, but this is misleading.  That may be the time that the software runs, but the time cutoff it uses is midnight.  Buy orders up to but not including midnight on 1/1/18 are logged by LC as ordered on 1/2/18.  In other words "order date" shown by LC is always B+1.

Now, when is a buy order cancelled?

A buy order is cancelled if B is in the range [D-1 thru S] inclusive.  In other words, the hazard window is one day more than settlement.

As an example, suppose a note had a due date on the 1st of a month.  Suppose there's a weekend and a holiday in there, so settlement is the 7th of the month.  Now you come along on the 7th and see this note listed for sale, and you buy it, and get a SUCCESS_PENDING_SETTLEMENT response.  Cool.  You think you've bought a note.  You haven't.  That order is gonna get cancelled for sure.

This is astonishing.  Its goofy that they know that they're gonna cancel the order if you buy, and they have known this for SEVERAL DAYS before you buy, but they let you buy, and then cancel.  Extra processing for them.  Extra processing for you.  Things are harder to track.  The right thing to do would have been to remove the sell order from folio once the window began, so that buyers didn't waste their time.  That's not what LC does.

I've had orders where I bought on D-1 and the order was cancelled after I bought.  (For the record note# 126688481.  I bought 1/1/18.  LC order date 1/2/18.  Due 1/2/18.  Settle 1/5/18.)

I've had orders where I bought on D+6 and the order was cancelled after I bought.  (For the record note# 119577848.  I bought on 1/2/18.  LC order date 1/3/18.  Due 12/27/17.  Settle 1/2/18.)

So the bottom line is that for each payment, ie every month, each note has a window of 4,5,6, or 7 days in which if you order in that window, the order will be cancelled.  Most of the time its 4 or 6 days.  The other two possibilities require holidays, and there aren't very many of those.  There are 4 days of each non-holiday week which have 4 day windows, and three days that have 6 day windows.  From this we can calculate that the average window is 4.857 days.  (Not considering holidays, which would make it a little bit bigger.)

Payments come 12 times per year.  Each payment has a 4.857 day hazard window.  That's 12x4.857 hazard days per year.  (The occasional extra payment will bump this up a little, as will holidays.  Missed payments will bump it down a little.)

D can occur on any day of the year.  Yea, I know LC only processes on weekdays, but we've already handled that with all that stuff about settlement sometimes being D+3 and sometimes D+5.  So there are 365 days to consider, and ...

Probability that a note order is cancelled = 12x4.857 / 365 = 18.46%

Now you may recall that I've had some periods when I calculated much higher failure rates for my own account.  Keep in mind that this calculation is an average.   Different months have different numbers of holidays and different alignment to weekends.  Also the luck of the draw will cause some variation.  Finally some of the much higher numbers I calculated a few days ago were from manual calculations which because of the difficulty of calculating by hand were for single days, which of course will vary much more than months.  There is probably also some day-of-week bias when loans originate.  This number above is just an average calculation all other things being equal.

Here (again) are the calcs I did for the last 6 months on all buy orders in my account.
(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Ffred93.com%2Ffbi%2FCapture-folio-purchase-stats.PNG&hash=ab6424ff4d915cc1e514912d4476a98a)

November and December have a couple of holidays each, so they're expected to be a little higher.  These numbers look like they sorta agree with the theory.

The marketplace is simply designed so that about 1 out of 5 buy transactions will fail.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: AnilG on July 10, 2018, 02:43:11 AM
Just FYI, new Folio API has order history call that you can use to capture order history. We implemented this on PeerCube so users are able to view order history and settled orders.

In my account, system placed 45 buy orders out of which 41 settled successfully. I don’t believe many buy orders placed successfully get cancelled.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: mark78 on July 10, 2018, 04:00:33 AM
Currently my https://www.lendingclub.com/foliofn/tradingAccount.action shows about 50 "Purchased this month" and 25 "Canceled Buy Orders", for a success rate of 67%.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: wcwei5800 on July 14, 2018, 03:33:13 AM
Not sure if anyone have been experience this problem lately, but my cancellation rate has gone up in last few month to be around 50% consistently now. This really hurts since there are that many notes to invest on to begin with.

Sorry the first answer I posted an hour or so ago was wrong.  I researched my past writings and...

You should expect around 20% failure, based on how the secondary market works.

Here's the text I posted on Jan 14, 2018, somewhere on this forum, tho I can't find it here now, and also emailed to LC on same day...


I think I have figured out the order processing timeline, and why so many buy orders are cancelled.

In the past I'd been looking at historical data for months gone by, and LC gives us no way at all to track down what happened for notes we bought in prior months but did not settle, so I couldn't look at note by note details.  Generally, you can't look up the payment history on the note unless you own it, or it is offered for sale on Folio.  One exception is notes that you've tried to buy during the CURRENT month.  The "my account" page on Folio has links to the note detail for all these notes.  However, if you let time slip into the next month, all these links disappear.  This page provides an opportunity to look at the payment details of notes for which a buy order has been cancelled!  You've just got to do it before the links disappear!  That's what I did tonite.

Tonite I looked at notes I bought this month, looking up the status of the purchase transaction, and the note payment dates on the note detail pages for a considerable number of notes.  Lots of notes.  My fingers are very tired.

I now believe I know how the purchase timeline works.  Its pretty wonky.  We all know that lots of sell orders are cancelled for note payment, and I always figured this was done to protect the buyer.  Not so.   The sell orders are cancelled way too late to help buyers.  Sale offers are NOT cancelled while payments are processing.  As a result of this, the buyer has quite a hazard.   

Loan payment processing timeline...

Lets reference dates to the note "due date".  I'll call this 'D'.  This is when the payment processing window begins.  No payment has been made yet.  This is simply the day when a process starts.  On that day, or the closest business day, LC sends a request to the bank, via the ACH system, to grab some money from borrower's account.  A few days later, if the money has arrived, LC considers the payment "settled".  This is the "settlement date".  Lets call it 'S'.  (Due date and settlement date are shown on the note detail page.)

Settlement date S can be = D+3, D+4, D+5 or D+6.  This occurs because LC only works on weekdays.  So if no weekend or holiday intervenes, settlement is D+3.  If a holiday is in there, then it's D+4.  If a weekend intervenes, it's D+5.  If you have both a weekend and a holiday, then settlement is D+6.

So payment processing takes 3,4,5, or 6 days.  ...but there's more!

Now, the LC/folio note order processing timeline...

When I place a successful buy order on day 'B', LC does not process that the same day.  They process it the next day.  You can see this on your "my account" page.  The "order date" is always 1 day later than the day you placed the order.  Recent email from LC claimed the settlement process runs every day at 3PM, but this is misleading.  That may be the time that the software runs, but the time cutoff it uses is midnight.  Buy orders up to but not including midnight on 1/1/18 are logged by LC as ordered on 1/2/18.  In other words "order date" shown by LC is always B+1.

Now, when is a buy order cancelled?

A buy order is cancelled if B is in the range [D-1 thru S] inclusive.  In other words, the hazard window is one day more than settlement.

As an example, suppose a note had a due date on the 1st of a month.  Suppose there's a weekend and a holiday in there, so settlement is the 7th of the month.  Now you come along on the 7th and see this note listed for sale, and you buy it, and get a SUCCESS_PENDING_SETTLEMENT response.  Cool.  You think you've bought a note.  You haven't.  That order is gonna get cancelled for sure.

This is astonishing.  Its goofy that they know that they're gonna cancel the order if you buy, and they have known this for SEVERAL DAYS before you buy, but they let you buy, and then cancel.  Extra processing for them.  Extra processing for you.  Things are harder to track.  The right thing to do would have been to remove the sell order from folio once the window began, so that buyers didn't waste their time.  That's not what LC does.

I've had orders where I bought on D-1 and the order was cancelled after I bought.  (For the record note# 126688481.  I bought 1/1/18.  LC order date 1/2/18.  Due 1/2/18.  Settle 1/5/18.)

I've had orders where I bought on D+6 and the order was cancelled after I bought.  (For the record note# 119577848.  I bought on 1/2/18.  LC order date 1/3/18.  Due 12/27/17.  Settle 1/2/18.)

So the bottom line is that for each payment, ie every month, each note has a window of 4,5,6, or 7 days in which if you order in that window, the order will be cancelled.  Most of the time its 4 or 6 days.  The other two possibilities require holidays, and there aren't very many of those.  There are 4 days of each non-holiday week which have 4 day windows, and three days that have 6 day windows.  From this we can calculate that the average window is 4.857 days.  (Not considering holidays, which would make it a little bit bigger.)

Payments come 12 times per year.  Each payment has a 4.857 day hazard window.  That's 12x4.857 hazard days per year.  (The occasional extra payment will bump this up a little, as will holidays.  Missed payments will bump it down a little.)

D can occur on any day of the year.  Yea, I know LC only processes on weekdays, but we've already handled that with all that stuff about settlement sometimes being D+3 and sometimes D+5.  So there are 365 days to consider, and ...

Probability that a note order is cancelled = 12x4.857 / 365 = 18.46%

Now you may recall that I've had some periods when I calculated much higher failure rates for my own account.  Keep in mind that this calculation is an average.   Different months have different numbers of holidays and different alignment to weekends.  Also the luck of the draw will cause some variation.  Finally some of the much higher numbers I calculated a few days ago were from manual calculations which because of the difficulty of calculating by hand were for single days, which of course will vary much more than months.  There is probably also some day-of-week bias when loans originate.  This number above is just an average calculation all other things being equal.

Here (again) are the calcs I did for the last 6 months on all buy orders in my account.
(https://forum.lendacademy.com/proxy.php?request=http%3A%2F%2Ffred93.com%2Ffbi%2FCapture-folio-purchase-stats.PNG&hash=ab6424ff4d915cc1e514912d4476a98a)

November and December have a couple of holidays each, so they're expected to be a little higher.  These numbers look like they sorta agree with the theory.

The marketplace is simply designed so that about 1 out of 5 buy transactions will fail.

your answer make sense. However, I been seem consistent high cancellation rates with my order which is really odd. My cancelled percentages are 45%, 37%, 40%, and 32% from March to June respectively. For July, this number is currently at 54%... The "day since last payment" is always around 14 days, so I don't this is the reason. The sample size is fairly large as well, so I don't believe they are due to random chances.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on July 14, 2018, 04:02:06 AM
your answer make sense. However, I been seem consistent high cancellation rates with my order which is really odd. My cancelled percentages are 45%, 37%, 40%, and 32% from March to June respectively. For July, this number is currently at 54%... The "day since last payment" is always around 14 days, so I don't this is the reason. The sample size is fairly large as well, so I don't believe they are due to random chances.

Strange. 

Just to be clear... these are orders where the order ended with "SUCCESS_PENDING..." and then you did not actually get.  Right?

Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: wcwei5800 on July 14, 2018, 04:28:10 AM
Yep, only count notes with SUCCESS status. I have emailed LC, but don't expect to get any fruitful answer from them.
Title: Re: Folio notes bought but never got -- ie buy orders cancelled by LC
Post by: Fred93 on July 14, 2018, 04:38:27 AM
Yep, only count notes with SUCCESS status. I have emailed LC, but don't expect to get any fruitful answer from them.

It is very difficult to get past the front line people to someone who actually understands what you're talking about.  Even the people the front line folks refer one to tend to not understand something like this. 

When I started complaining about all the cancelled orders last year sometime they gave me funky theories about how I must be choosing notes that had a payment coming up.  Later I showed them that wasn't true.  Finally I reverse engineered the cancellations, working case by case, and worked out the math. 

I suspect that the software developers who did the initial hookup between LC and folio are long gone, and the maintenance people don't really know how it works.  When you look into this monster, it is quite a hack job.