Lend Academy Network Forum

Lending Club Discussion => Foliofn - LC => Topic started by: Fred93 on October 22, 2016, 02:06:20 AM

Title: how often is folio inventory CSV file updated?
Post by: Fred93 on October 22, 2016, 02:06:20 AM
Suppose I write software that downloads this file and searches for notes to buy.  How often do I need to download this sucker?

Do they update it on a regular schedule, or whenever somebody posts a note for sale, or what?
Title: Re: how often is folio inventory CSV file updated?
Post by: jpildis on October 22, 2016, 10:07:20 AM
My understanding is that it's every 5 minutes.
Title: Re: how often is folio inventory CSV file updated?
Post by: fliphusker on October 22, 2016, 02:01:02 PM
Sometimes FOLIO will have notes listed as never late but notes slip through the cracks.  The odd thing of the notes that slip through the cracks is that it is not an IGP from a year ago it is generally a recent one. 
Would you be able to write the software that would account for notes that are taking a recent FICO drop in the past couple of months but still remains up over its lifetime?  Finding a note with a juicy up FICO just to find out it that the FICO is trending down sucks. 
Could it pick up multiple payments?  When I first started on FOLIO I bought a note and did not notice they were making triple payments monthly.  While I am not losing money on the note I am definitely not making the 17% YTM I expected on it. 
Have you tried out NSRs search on FICO?
Suppose I write software that downloads this file and searches for notes to buy.  How often do I need to download this sucker?

Do they update it on a regular schedule, or whenever somebody posts a note for sale, or what?
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on October 22, 2016, 02:27:34 PM
Would you be able to write the software that would account for notes that are taking a recent FICO drop in the past couple of months but still remains up over its lifetime?  Finding a note with a juicy up FICO just to find out it that the FICO is trending down sucks. 
Could it pick up multiple payments?

The information that is available to straightforward software (ie what is in the CSV file) does not give me access to FICO vs time.  It just lists a recent FICO and the up/down/flat flag.  Of course I can get the FICO at loan origination time from the LC loan history files.  I can therefore tell how much FICO has changed, but I cannot get the intermediate FICO values, unless I screen-scrape the folio web pages.  1 web page per loan and there are 621,416 loans listed, although one could obviously filter on other stuff first, and then scrape the FICO vs time for only a few interesting loans.  Ditto payments.  In years past, LC officially disallowed screen scraping, but since it was necessary for a few things (in years past), they tolerated some.  Whether or not it is allowed, it is a pain in the ass to program and maintain.  Therefore, the correct answer is probably no.

LC/folio has not put much effort into the API.  Its minimalist.

So the  case you want to avoid is this?  FICO jumps up sometime in the past, but so long ago that its not interesting, maybe 1 year ago, and has been trending down ever since.  folio reports that as "UP" but you think of it as "trending down".  Did I get that right?  Do you see this a lot?

Title: Re: how often is folio inventory CSV file updated?
Post by: fliphusker on October 22, 2016, 03:40:11 PM
You can see the FICO change at NSR as well after a filtered search and will tell you the change.  Let me toss out a scenario.  Borrower takes out 35k to consolidate debt as his CC are almost maxed out.  When you hit certain levels of CC % it impacts your FICO greatly.  10%, 30%, 60% and I think 90%.  So this guy is let's say 690 FICO.  He pays off those CC.  It is his CC that was killing his FICO and adding an installment payment has minimal impact on the FICO.  So he pulls that 690 up to 750 (which I have done actually.)  Now after 24 months of that 5-year note, he misses a car payment and is late by 2 months.  His FICO begins to plummet but still remains up over the life over the note and the FICO shows up 20.  Now the borrower has to decide to lose his car or start to miss LC payments. 
It is not so much the trending down slowly that is bothersome to me.  It is the spiking down. 
This note is a bit of a bad example as there is no real spike.  But it is trending lower.  I am going to make an assumption here.  I do not think they have missed any payments on other bills but running higher CC debt. 
https://www.lendingclub.com/foliofn
/browseNotesLoanPerf.action?showfoliofn=true&loan_id=46663002&order_id=65190991&note_id=79506210
Here is a perfect example actually.  30 point drop in past month and a 10 point drop shortly before.  From not seeing the FICO drop, the note is flat and has not missed a payment.  Sure they changed their payment date, but that was 10 months ago.  (Should you care about that is another caveat to worry about with FOLIO.)  From the outside, this is a desirable note.  Good YTM at a discount with a solid payment history.  Would automatic investing pick this up as a good note?  My guess is yes.  But what it does not see is that FICO going south. 
https://www.lendingclub.com/foliofn/browseNotesLoanPerf.action?showfoliofn=true&loan_id=48704474&order_id=99925929&note_id=80841285
FOLIO is all about risk to reward to what your willing to pay.  For me the 2 notes above I would not touch, no doubt others will, though. 
Did my rambling book make sense?  :)
Would you be able to write the software that would account for notes that are taking a recent FICO drop in the past couple of months but still remains up over its lifetime?  Finding a note with a juicy up FICO just to find out it that the FICO is trending down sucks. 
Could it pick up multiple payments?

The information that is available to straightforward software (ie what is in the CSV file) does not give me access to FICO vs time.  It just lists a recent FICO and the up/down/flat flag.  Of course I can get the FICO at loan origination time from the LC loan history files.  I can therefore tell how much FICO has changed, but I cannot get the intermediate FICO values, unless I screen-scrape the folio web pages.  1 web page per loan and there are 621,416 loans listed, although one could obviously filter on other stuff first, and then scrape the FICO vs time for only a few interesting loans.  Ditto payments.  In years past, LC officially disallowed screen scraping, but since it was necessary for a few things (in years past), they tolerated some.  Whether or not it is allowed, it is a pain in the ass to program and maintain.  Therefore, the correct answer is probably no.

LC/folio has not put much effort into the API.  Its minimalist.

So the  case you want to avoid is this?  FICO jumps up sometime in the past, but so long ago that its not interesting, maybe 1 year ago, and has been trending down ever since.  folio reports that as "UP" but you think of it as "trending down".  Did I get that right?  Do you see this a lot?
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on October 22, 2016, 05:13:55 PM
Let me toss out a scenario.  Borrower takes out 35k to consolidate debt as his CC are almost maxed out.  When you hit certain levels of CC % it impacts your FICO greatly.  10%, 30%, 60% and I think 90%.  So this guy is let's say 690 FICO.  He pays off those CC.  It is his CC that was killing his FICO and adding an installment payment has minimal impact on the FICO.  So he pulls that 690 up to 750 (which I have done actually.)  Now after 24 months of that 5-year note, he misses a car payment and is late by 2 months.  His FICO begins to plummet but still remains up over the life over the note and the FICO shows up 20.  Now the borrower has to decide to lose his car or start to miss LC payments. 

Wow.  That's some pretty sophisticated FICO curve interpretation. 

Hopefully someday they'll fix up the folio API so that it has all the capabilities that the web site has.
Title: Re: how often is folio inventory CSV file updated?
Post by: Rob L on October 22, 2016, 05:25:28 PM
My experience says that an update of the smallnotes.csv file once every 5 minutes sounds about right.
I do not know if there is a punctual schedule (like primary loan feeding times) but wouldn't be surprised. Not a hard thing to discover it one is interested.

Automating Folio selling is hard enough, and buying appears much tougher (I haven't attempted the buying part).
Sellers have no way to know if their notes are NeverLate without listing them. Kind of an important piece of information. My solution was to list all my notes at a ridiculous mark up, wait that 5 minutes or so  mentioned above, download smallnotes.csv (with my notes now listed in it), and capture their NeverLate status. Really!! Yeah, really. In hindsight I'd been better off if I'd simply assumed NeverLate as I didn't sell many not NeverLates anyway.

Fred93 is "almost" spot on saying that the "only" way to obtain the intermediate FICO scores between when listed and current is to screen scrape. There may be a few intrepid souls who have diligently captured over the long term the LoanStats.csv files (updated monthly) that could theoretically go back through them and piece together historical month by month FICO scores. Loans in the most recent quarter are not included.  From postings here on the forum only two handles come to mind that might possibly have such an extensive historic data base; Anil and Fred (not93). It may be an ugly solution but to me it seems reasonable one could whittle down the prospects with data available in the .csv files and that screen scraping as a final go/no go pass would be a reasonable solution, not overly objectionable (nor detectable) by LC.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on October 22, 2016, 06:21:29 PM
My experience says that an update of the smallnotes.csv file once every 5 minutes sounds about right.  I do not know if there is a punctual schedule

What is smallnotes.csv?  I'm downloading SecondaryMarketAllNotes.csv .

SMAN is a 100+MByte file.  Compresses to 30+MByte.  Every 5 minutes is 288 downloads/day, or 8.64 GB/day or 259 GB/month.  Seems like an awful lot of downloading! 

This has got to be the most inefficient API ever.
Title: Re: how often is folio inventory CSV file updated?
Post by: storm on October 22, 2016, 06:51:39 PM
I think FICO trend is pretty simplistic.  If the borrower's score is higher than when they first applied for the loan, then it shows an up arrow.  If the score is currently lower than when they originated the loan, then it will show down.  I've seen a quite a few notes where the chart only shows the initial score, and doesn't seem to get updated monthly.
Title: Re: how often is folio inventory CSV file updated?
Post by: fliphusker on October 22, 2016, 08:05:28 PM
There really is not a lot that goes into a FICO score nor things that can move a FICO score very much.  Some of this is speculation some of it has been monitoring my FICO with Credit Karma over the years.  (Ya I am a bit anal about my FICO.  OK, maybe a lot.)  Most people are going to have peaks and valleys with their FICO.  IMO it usually takes something fairly big to move a FICO 50 points, and something I do not want with a note that has that recent drop. 
Type of credit will not move your FICO score hardly at all even if you open up something new let's say a mortgage.
An inquiry should only move your FICO score less than 5 points.  It also comes off after 2 years.  The more accounts you have the better.  So someone opening up a new line their credit score may actually go up.
Length of credit history again will not move the FICO much.  And unless you are opening up a lot of lines at once, should not see much impact. 
So it is going to be DTI and late payments that will move that needle.  Even if that move is slight.  When I open up a note that I see a steep drop I am never surprised to see a corresponding 15-120 late payments around that time. 
    Payment history: (35 percent) -- Your account payment information, including any delinquencies and public records.
    Amounts owed: (30 percent) -- How much you owe on your accounts. The amount of available credit you're using on revolving accounts is heavily weighted.
    Length of credit history: (15 percent) -- How long ago you opened accounts and time since account activity.
    Types of credit used: (10 percent) -- The mix of accounts you have, such as revolving and installment.
New credit: (10 percent) -- Your pursuit of new credit, including credit inquiries and number of recently opened accounts.

Rob what I do with my notes that hit IGP is I spin them off into separate portfolios.  I have 2 different portfolios one for FOLIO and another one from LC primary notes.  I do list my notes through the FOLIO site.  I know this is not a real option for you with the size of your account or with most people either. 
A semi-regular poster here and just do not remember who mentioned about auto investing in FOLIO.  I just can not remember who it was.
Storm, you might want to look at the notes I posted above.  Those types of notes are really a lot of the reason why most do not want to get entwined with FOLIO because of the work involved with it. 
When Fred makes a program that will scrape FICO scores, he will be a rich man.  :)

Let me toss out a scenario.  Borrower takes out 35k to consolidate debt as his CC are almost maxed out.  When you hit certain levels of CC % it impacts your FICO greatly.  10%, 30%, 60% and I think 90%.  So this guy is let's say 690 FICO.  He pays off those CC.  It is his CC that was killing his FICO and adding an installment payment has minimal impact on the FICO.  So he pulls that 690 up to 750 (which I have done actually.)  Now after 24 months of that 5-year note, he misses a car payment and is late by 2 months.  His FICO begins to plummet but still remains up over the life over the note and the FICO shows up 20.  Now the borrower has to decide to lose his car or start to miss LC payments. 

Wow.  That's some pretty sophisticated FICO curve interpretation. 

Hopefully someday they'll fix up the folio API so that it has all the capabilities that the web site has.
Title: Re: how often is folio inventory CSV file updated?
Post by: Rob L on October 23, 2016, 10:53:06 AM
What is smallnotes.csv?  I'm downloading SecondaryMarketAllNotes.csv .

This has got to be the most inefficient API ever.

smallnotes.csv == SMAN

Agree; the Folio API is "spartan" to say the least but I'm glad it exists; much better than nothing.
Think awful then think about what Prosper just did.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred on October 25, 2016, 03:59:05 AM
Suppose I write software that downloads this file and searches for notes to buy.  How often do I need to download this sucker?

Do they update it on a regular schedule, or whenever somebody posts a note for sale, or what?

Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?

This link is more real-time; all Folio notes matching your search criteria will be included.  No noticeable delays.

My Folio program runs every 2-hr.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on October 25, 2016, 09:25:57 AM
Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?

No, I didn't.  I was blindly following the API instructions.  Silly me.  I presumed that the only way to make use of the filtering from software was to screen scrape, but I see that is not true.  I was wishing for a little filtering.  They implemented this feature but forgot to talk about it in the API docs.  It is slightly different than downloading the full file, because you have to be logged in to use the filter. 

The best filter would be "listed in last X hours."

My hangup now is that there doesn't seem to be any way to calculate how late a note (that I do not own) is.  I presume "days since last payment" is counting days since ANY payment, which could be early, late, partial, extra, etc, and therefore is not the same thing as "days in arrears".  Also, even for loans only making payments at scheduled dates, the "days since last payment" field seems to be a few days off.  I haven't researched this enough to understand what they are up to.

If you are a human using the web pages, you can clearly see the dates of payments, etc, but software, using only the download file has no access to payment dates.
Title: Re: how often is folio inventory CSV file updated?
Post by: fliphusker on October 25, 2016, 03:31:41 PM
Would you mind sharing what your program does?  Does it auto buy? 
Suppose I write software that downloads this file and searches for notes to buy.  How often do I need to download this sucker?

Do they update it on a regular schedule, or whenever somebody posts a note for sale, or what?

Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?

This link is more real-time; all Folio notes matching your search criteria will be included.  No noticeable delays.

My Folio program runs every 2-hr.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred on October 25, 2016, 10:20:18 PM
Would you mind sharing what your program does?  Does it auto buy? 

Yes, my program does auto buy.

It downloads the latest 500,000+ notes from Folio. runs risk-reward analysis on each, chooses some candidate notes to buy, and checks collection logs.  If nothing is problematic, submits purchase orders, then logouts.

It also does auto sell -- finding which of my notes are good to sell based on today's Folio prices.

This is all I do now in LC;  I do not buy notes from the primary market anymore.
Title: Re: how often is folio inventory CSV file updated?
Post by: fliphusker on October 26, 2016, 08:57:51 AM
So it scrapes through the FICO scores for downturns in recent months?  Or do you purchase so much in bulk that this is not a concern? 
Sounds pretty awesome and a lot of work was involved in building it, KUDOS!!
Would you mind sharing what your program does?  Does it auto buy? 

Yes, my program does auto buy.

It downloads the latest 500,000+ notes from Folio. runs risk-reward analysis on each, chooses some candidate notes to buy, and checks collection logs.  If nothing is problematic, submits purchase orders, then logouts.

It also does auto sell -- finding which of my notes are good to sell based on today's Folio prices.

This is all I do now in LC;  I do not buy notes from the primary market anymore.
Title: Re: how often is folio inventory CSV file updated?
Post by: RT45 on October 26, 2016, 03:44:21 PM
Can't you just join the Payment History file with LoanStats to get the time series for FICO scores?

Doesn't necessarily catch most recent FICO change except at the monthly posting, but should help build the FICO trend.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred on October 26, 2016, 11:16:25 PM
Can't you just join the Payment History file with LoanStats to get the time series for FICO scores?

Doesn't necessarily catch most recent FICO change except at the monthly posting, but should help build the FICO trend.

LoanStats only has 1 record per loan, no history.

Payment History contains LastFICOBand by month, but the file is not updated daily,
Title: Re: how often is folio inventory CSV file updated?
Post by: Rob L on October 27, 2016, 12:42:31 PM
It downloads the latest 500,000+ notes from Folio. runs risk-reward analysis on each, chooses some candidate notes to buy, and checks collection logs.
If nothing is problematic, submits purchase orders, then logouts.

Interesting how your approach has evolved over the years to add "memory" (collection log) to your initial "memoryless" Markov chain process. Must have been an interesting challenge to develop a statistically predictive model given a collection log as input. Just putting together a historic data base of enough loans with their collection logs for a representative sample to test must have taken quite a long while.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on October 27, 2016, 03:55:23 PM
Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?
This link is more real-time; all Folio notes matching your search criteria will be included.  No noticeable delays.
My Folio program runs every 2-hr.

When you download this, do you sometimes get a corrupted file?

I'm getting a bad download about 1 out of 10 tries.  A chunk of characters missing, resulting in malformed CSV, or a bunch of nulls in the file in place of some characters.  Unusual.

Don't know whether the problem is at my end or LC.  I download plenty of other CSV files from LC with no problem.  Something is different in this case, but I don't yet know what.
Title: Re: how often is folio inventory CSV file updated?
Post by: fliphusker on October 27, 2016, 04:07:23 PM
Is there something that I am missing that can not be done at NSR?  The only thing it will not do is filter out FICO drops that are recent with the overall FICO still showing that it is up. 
I do not download .csv files as I do not have Excel and I am not very good at dealing with spreadsheets when I get them into Open Office. 
Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?
This link is more real-time; all Folio notes matching your search criteria will be included.  No noticeable delays.
My Folio program runs every 2-hr.

When you download this, do you sometimes get a corrupted file?

I'm getting a bad download about 1 out of 10 tries.  A chunk of characters missing, resulting in malformed CSV, or a bunch of nulls in the file in place of some characters.  Unusual.

Don't know whether the problem is at my end or LC.  I download plenty of other CSV files from LC with no problem.  Something is different in this case, but I don't yet know what.
Title: Re: how often is folio inventory CSV file updated?
Post by: dmcnic on November 01, 2016, 09:45:47 PM
Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?
This link is more real-time; all Folio notes matching your search criteria will be included.  No noticeable delays.
My Folio program runs every 2-hr.

When you download this, do you sometimes get a corrupted file?

I'm getting a bad download about 1 out of 10 tries.  A chunk of characters missing, resulting in malformed CSV, or a bunch of nulls in the file in place of some characters.  Unusual.


That's about the same rate I am getting an error. I typically try again in ten minutes and every day except once, it worked the second time.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on November 01, 2016, 11:05:26 PM
When you download this, do you sometimes get a corrupted file?

I'm getting a bad download about 1 out of 10 tries.  A chunk of characters missing, resulting in malformed CSV, or a bunch of nulls in the file in place of some characters.  Unusual.


That's about the same rate I am getting an error. I typically try again in ten minutes and every day except once, it worked the second time.

I reported this problem to LC tonite.  I reproduced it with both Firefox and IE, and sent them images captured with a binary editor showing them blocks of null characters in the corrupted CSV files.

Could you please also report it?  Sometimes it is difficult to get priority on a bug unless it is reported by multiple people.  The more the merrier.
Title: Re: how often is folio inventory CSV file updated?
Post by: dmcnic on November 02, 2016, 08:28:37 PM

I reported this problem to LC tonite.  I reproduced it with both Firefox and IE, and sent them images captured with a binary editor showing them blocks of null characters in the corrupted CSV files.

Could you please also report it?  Sometimes it is difficult to get priority on a bug unless it is reported by multiple people.  The more the merrier.

I do have a person's name who contacted me just after I joined. I'll call her tomorrow and see if I can help get this bumped up the priority list.
Title: Re: how often is folio inventory CSV file updated?
Post by: dmcnic on November 05, 2016, 10:12:15 AM
I left a message Thursday afternoon. No response after the end of the day on Friday. I'll try again next week.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on November 08, 2016, 07:27:08 PM
A related question: There are 2 or 3 third-party web sites that do some automated LC folio trading.  How often do they run their algorithms?  In other words, if I post a note for sale, how long must I wait until I'm sure they've done a cycle and have decided not to buy my note at the present price?
Title: Re: how often is folio inventory CSV file updated?
Post by: Zach on November 12, 2016, 05:19:11 PM
A related question: There are 2 or 3 third-party web sites that do some automated LC folio trading.  How often do they run their algorithms?  In other words, if I post a note for sale, how long must I wait until I'm sure they've done a cycle and have decided not to buy my note at the present price?

NSR runs Lending Club Folio buying every minute.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on November 12, 2016, 06:45:44 PM
A related question: There are 2 or 3 third-party web sites that do some automated LC folio trading.  How often do they run their algorithms?  In other words, if I post a note for sale, how long must I wait until I'm sure they've done a cycle and have decided not to buy my note at the present price?

NSR runs Lending Club Folio buying every minute.

Wow.  That's pretty aggressive.  So if I have a note listed for a few hours, I can be pretty much 100% sure that your clients' automated programs didn't want it.

I presume that most of NSRs buying volume is in the small notes, <$25, <$50, etc.  Can you share a bit about that?  Do you buy (for your clients, from folio) any significant number of notes in higher sizes?  $500?  $1000?
Title: Re: how often is folio inventory CSV file updated?
Post by: Rob L on November 13, 2016, 09:19:38 AM
A related question: There are 2 or 3 third-party web sites that do some automated LC folio trading.  How often do they run their algorithms?  In other words, if I post a note for sale, how long must I wait until I'm sure they've done a cycle and have decided not to buy my note at the present price?

NSR runs Lending Club Folio buying every minute.

This says to me that from the point of view of the original question the folio inventory file is updated possibly more often than once a minute, and there is no "fixed update schedule". Otherwise NSR would have adapted to that reality. Instead the update algorithm might be something like "update as soon as N changes have occurred since the last update and update after any changes have been made and M seconds have passed since the last update.
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on November 13, 2016, 04:50:02 PM
NSR runs Lending Club Folio buying every minute.

This says to me that from the point of view of the original question the folio inventory file is updated possibly more often than once a minute, and there is no "fixed update schedule". Otherwise NSR would have adapted

LC has told me that the allnotes file is updated every 5 minutes.  However, other folks on this forum have told me that they believe the web site is more up to date than the file, and I think I have seen evidence to that effect.  My guess is that many are reading the web site rather than the file, in order to get faster than 5 minute access.
Title: Re: how often is folio inventory CSV file updated?
Post by: mark78 on January 28, 2018, 03:11:18 AM
Old topic, I know, but I did some tests and found it varied from 2 minutes 40 seconds to 3 minutes. That was over the course of a week (I wasn't really *that interested, but happened to be logging it).
Title: Re: how often is folio inventory CSV file updated?
Post by: Fred93 on January 28, 2018, 10:06:48 AM
Agreed.  It has changed.  Its twice as often as it used to be.  And as usual, LC forgot to notify us of this change.

Darn it tho, its not on a fixed schedule.  My guess is that instead of updating on a fixed schedule, at the end of the processing work, they set a timer which says "Wait 2.5 minutes, then do it again."  The result is that the gap between is always a little more than 2.5 minutes, but not predictable.  Not predictable means you have to poll.