Author Topic: how often is folio inventory CSV file updated?  (Read 11416 times)

Fred93

  • Hero Member
  • *****
  • Posts: 2234
    • View Profile
how often is folio inventory CSV file updated?
« on: October 22, 2016, 02:06:20 AM »
Suppose I write software that downloads this file and searches for notes to buy.  How often do I need to download this sucker?

Do they update it on a regular schedule, or whenever somebody posts a note for sale, or what?

jpildis

  • Full Member
  • ***
  • Posts: 158
    • View Profile
    • Email
Re: how often is folio inventory CSV file updated?
« Reply #1 on: October 22, 2016, 10:07:20 AM »
My understanding is that it's every 5 minutes.

fliphusker

  • Sr. Member
  • ****
  • Posts: 463
    • View Profile
    • Email
Re: how often is folio inventory CSV file updated?
« Reply #2 on: October 22, 2016, 02:01:02 PM »
Sometimes FOLIO will have notes listed as never late but notes slip through the cracks.  The odd thing of the notes that slip through the cracks is that it is not an IGP from a year ago it is generally a recent one. 
Would you be able to write the software that would account for notes that are taking a recent FICO drop in the past couple of months but still remains up over its lifetime?  Finding a note with a juicy up FICO just to find out it that the FICO is trending down sucks. 
Could it pick up multiple payments?  When I first started on FOLIO I bought a note and did not notice they were making triple payments monthly.  While I am not losing money on the note I am definitely not making the 17% YTM I expected on it. 
Have you tried out NSRs search on FICO?
Suppose I write software that downloads this file and searches for notes to buy.  How often do I need to download this sucker?

Do they update it on a regular schedule, or whenever somebody posts a note for sale, or what?
« Last Edit: October 22, 2016, 02:06:42 PM by fliphusker »

Fred93

  • Hero Member
  • *****
  • Posts: 2234
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #3 on: October 22, 2016, 02:27:34 PM »
Would you be able to write the software that would account for notes that are taking a recent FICO drop in the past couple of months but still remains up over its lifetime?  Finding a note with a juicy up FICO just to find out it that the FICO is trending down sucks. 
Could it pick up multiple payments?

The information that is available to straightforward software (ie what is in the CSV file) does not give me access to FICO vs time.  It just lists a recent FICO and the up/down/flat flag.  Of course I can get the FICO at loan origination time from the LC loan history files.  I can therefore tell how much FICO has changed, but I cannot get the intermediate FICO values, unless I screen-scrape the folio web pages.  1 web page per loan and there are 621,416 loans listed, although one could obviously filter on other stuff first, and then scrape the FICO vs time for only a few interesting loans.  Ditto payments.  In years past, LC officially disallowed screen scraping, but since it was necessary for a few things (in years past), they tolerated some.  Whether or not it is allowed, it is a pain in the ass to program and maintain.  Therefore, the correct answer is probably no.

LC/folio has not put much effort into the API.  Its minimalist.

So the  case you want to avoid is this?  FICO jumps up sometime in the past, but so long ago that its not interesting, maybe 1 year ago, and has been trending down ever since.  folio reports that as "UP" but you think of it as "trending down".  Did I get that right?  Do you see this a lot?


fliphusker

  • Sr. Member
  • ****
  • Posts: 463
    • View Profile
    • Email
Re: how often is folio inventory CSV file updated?
« Reply #4 on: October 22, 2016, 03:40:11 PM »
You can see the FICO change at NSR as well after a filtered search and will tell you the change.  Let me toss out a scenario.  Borrower takes out 35k to consolidate debt as his CC are almost maxed out.  When you hit certain levels of CC % it impacts your FICO greatly.  10%, 30%, 60% and I think 90%.  So this guy is let's say 690 FICO.  He pays off those CC.  It is his CC that was killing his FICO and adding an installment payment has minimal impact on the FICO.  So he pulls that 690 up to 750 (which I have done actually.)  Now after 24 months of that 5-year note, he misses a car payment and is late by 2 months.  His FICO begins to plummet but still remains up over the life over the note and the FICO shows up 20.  Now the borrower has to decide to lose his car or start to miss LC payments. 
It is not so much the trending down slowly that is bothersome to me.  It is the spiking down. 
This note is a bit of a bad example as there is no real spike.  But it is trending lower.  I am going to make an assumption here.  I do not think they have missed any payments on other bills but running higher CC debt. 
https://www.lendingclub.com/foliofn
/browseNotesLoanPerf.action?showfoliofn=true&loan_id=46663002&order_id=65190991&note_id=79506210
Here is a perfect example actually.  30 point drop in past month and a 10 point drop shortly before.  From not seeing the FICO drop, the note is flat and has not missed a payment.  Sure they changed their payment date, but that was 10 months ago.  (Should you care about that is another caveat to worry about with FOLIO.)  From the outside, this is a desirable note.  Good YTM at a discount with a solid payment history.  Would automatic investing pick this up as a good note?  My guess is yes.  But what it does not see is that FICO going south. 
https://www.lendingclub.com/foliofn/browseNotesLoanPerf.action?showfoliofn=true&loan_id=48704474&order_id=99925929&note_id=80841285
FOLIO is all about risk to reward to what your willing to pay.  For me the 2 notes above I would not touch, no doubt others will, though. 
Did my rambling book make sense?  :)
Would you be able to write the software that would account for notes that are taking a recent FICO drop in the past couple of months but still remains up over its lifetime?  Finding a note with a juicy up FICO just to find out it that the FICO is trending down sucks. 
Could it pick up multiple payments?

The information that is available to straightforward software (ie what is in the CSV file) does not give me access to FICO vs time.  It just lists a recent FICO and the up/down/flat flag.  Of course I can get the FICO at loan origination time from the LC loan history files.  I can therefore tell how much FICO has changed, but I cannot get the intermediate FICO values, unless I screen-scrape the folio web pages.  1 web page per loan and there are 621,416 loans listed, although one could obviously filter on other stuff first, and then scrape the FICO vs time for only a few interesting loans.  Ditto payments.  In years past, LC officially disallowed screen scraping, but since it was necessary for a few things (in years past), they tolerated some.  Whether or not it is allowed, it is a pain in the ass to program and maintain.  Therefore, the correct answer is probably no.

LC/folio has not put much effort into the API.  Its minimalist.

So the  case you want to avoid is this?  FICO jumps up sometime in the past, but so long ago that its not interesting, maybe 1 year ago, and has been trending down ever since.  folio reports that as "UP" but you think of it as "trending down".  Did I get that right?  Do you see this a lot?

Fred93

  • Hero Member
  • *****
  • Posts: 2234
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #5 on: October 22, 2016, 05:13:55 PM »
Let me toss out a scenario.  Borrower takes out 35k to consolidate debt as his CC are almost maxed out.  When you hit certain levels of CC % it impacts your FICO greatly.  10%, 30%, 60% and I think 90%.  So this guy is let's say 690 FICO.  He pays off those CC.  It is his CC that was killing his FICO and adding an installment payment has minimal impact on the FICO.  So he pulls that 690 up to 750 (which I have done actually.)  Now after 24 months of that 5-year note, he misses a car payment and is late by 2 months.  His FICO begins to plummet but still remains up over the life over the note and the FICO shows up 20.  Now the borrower has to decide to lose his car or start to miss LC payments. 

Wow.  That's some pretty sophisticated FICO curve interpretation. 

Hopefully someday they'll fix up the folio API so that it has all the capabilities that the web site has.

Rob L

  • Hero Member
  • *****
  • Posts: 2124
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #6 on: October 22, 2016, 05:25:28 PM »
My experience says that an update of the smallnotes.csv file once every 5 minutes sounds about right.
I do not know if there is a punctual schedule (like primary loan feeding times) but wouldn't be surprised. Not a hard thing to discover it one is interested.

Automating Folio selling is hard enough, and buying appears much tougher (I haven't attempted the buying part).
Sellers have no way to know if their notes are NeverLate without listing them. Kind of an important piece of information. My solution was to list all my notes at a ridiculous mark up, wait that 5 minutes or so  mentioned above, download smallnotes.csv (with my notes now listed in it), and capture their NeverLate status. Really!! Yeah, really. In hindsight I'd been better off if I'd simply assumed NeverLate as I didn't sell many not NeverLates anyway.

Fred93 is "almost" spot on saying that the "only" way to obtain the intermediate FICO scores between when listed and current is to screen scrape. There may be a few intrepid souls who have diligently captured over the long term the LoanStats.csv files (updated monthly) that could theoretically go back through them and piece together historical month by month FICO scores. Loans in the most recent quarter are not included.  From postings here on the forum only two handles come to mind that might possibly have such an extensive historic data base; Anil and Fred (not93). It may be an ugly solution but to me it seems reasonable one could whittle down the prospects with data available in the .csv files and that screen scraping as a final go/no go pass would be a reasonable solution, not overly objectionable (nor detectable) by LC.
« Last Edit: October 22, 2016, 05:40:09 PM by Rob L »

Fred93

  • Hero Member
  • *****
  • Posts: 2234
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #7 on: October 22, 2016, 06:21:29 PM »
My experience says that an update of the smallnotes.csv file once every 5 minutes sounds about right.  I do not know if there is a punctual schedule

What is smallnotes.csv?  I'm downloading SecondaryMarketAllNotes.csv .

SMAN is a 100+MByte file.  Compresses to 30+MByte.  Every 5 minutes is 288 downloads/day, or 8.64 GB/day or 259 GB/month.  Seems like an awful lot of downloading! 

This has got to be the most inefficient API ever.

storm

  • Full Member
  • ***
  • Posts: 139
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #8 on: October 22, 2016, 06:51:39 PM »
I think FICO trend is pretty simplistic.  If the borrower's score is higher than when they first applied for the loan, then it shows an up arrow.  If the score is currently lower than when they originated the loan, then it will show down.  I've seen a quite a few notes where the chart only shows the initial score, and doesn't seem to get updated monthly.

fliphusker

  • Sr. Member
  • ****
  • Posts: 463
    • View Profile
    • Email
Re: how often is folio inventory CSV file updated?
« Reply #9 on: October 22, 2016, 08:05:28 PM »
There really is not a lot that goes into a FICO score nor things that can move a FICO score very much.  Some of this is speculation some of it has been monitoring my FICO with Credit Karma over the years.  (Ya I am a bit anal about my FICO.  OK, maybe a lot.)  Most people are going to have peaks and valleys with their FICO.  IMO it usually takes something fairly big to move a FICO 50 points, and something I do not want with a note that has that recent drop. 
Type of credit will not move your FICO score hardly at all even if you open up something new let's say a mortgage.
An inquiry should only move your FICO score less than 5 points.  It also comes off after 2 years.  The more accounts you have the better.  So someone opening up a new line their credit score may actually go up.
Length of credit history again will not move the FICO much.  And unless you are opening up a lot of lines at once, should not see much impact. 
So it is going to be DTI and late payments that will move that needle.  Even if that move is slight.  When I open up a note that I see a steep drop I am never surprised to see a corresponding 15-120 late payments around that time. 
    Payment history: (35 percent) -- Your account payment information, including any delinquencies and public records.
    Amounts owed: (30 percent) -- How much you owe on your accounts. The amount of available credit you're using on revolving accounts is heavily weighted.
    Length of credit history: (15 percent) -- How long ago you opened accounts and time since account activity.
    Types of credit used: (10 percent) -- The mix of accounts you have, such as revolving and installment.
New credit: (10 percent) -- Your pursuit of new credit, including credit inquiries and number of recently opened accounts.

Rob what I do with my notes that hit IGP is I spin them off into separate portfolios.  I have 2 different portfolios one for FOLIO and another one from LC primary notes.  I do list my notes through the FOLIO site.  I know this is not a real option for you with the size of your account or with most people either. 
A semi-regular poster here and just do not remember who mentioned about auto investing in FOLIO.  I just can not remember who it was.
Storm, you might want to look at the notes I posted above.  Those types of notes are really a lot of the reason why most do not want to get entwined with FOLIO because of the work involved with it. 
When Fred makes a program that will scrape FICO scores, he will be a rich man.  :)

Let me toss out a scenario.  Borrower takes out 35k to consolidate debt as his CC are almost maxed out.  When you hit certain levels of CC % it impacts your FICO greatly.  10%, 30%, 60% and I think 90%.  So this guy is let's say 690 FICO.  He pays off those CC.  It is his CC that was killing his FICO and adding an installment payment has minimal impact on the FICO.  So he pulls that 690 up to 750 (which I have done actually.)  Now after 24 months of that 5-year note, he misses a car payment and is late by 2 months.  His FICO begins to plummet but still remains up over the life over the note and the FICO shows up 20.  Now the borrower has to decide to lose his car or start to miss LC payments. 

Wow.  That's some pretty sophisticated FICO curve interpretation. 

Hopefully someday they'll fix up the folio API so that it has all the capabilities that the web site has.

Rob L

  • Hero Member
  • *****
  • Posts: 2124
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #10 on: October 23, 2016, 10:53:06 AM »
What is smallnotes.csv?  I'm downloading SecondaryMarketAllNotes.csv .

This has got to be the most inefficient API ever.

smallnotes.csv == SMAN

Agree; the Folio API is "spartan" to say the least but I'm glad it exists; much better than nothing.
Think awful then think about what Prosper just did.

Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #11 on: October 25, 2016, 03:59:05 AM »
Suppose I write software that downloads this file and searches for notes to buy.  How often do I need to download this sucker?

Do they update it on a regular schedule, or whenever somebody posts a note for sale, or what?

Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?

This link is more real-time; all Folio notes matching your search criteria will be included.  No noticeable delays.

My Folio program runs every 2-hr.

Fred93

  • Hero Member
  • *****
  • Posts: 2234
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #12 on: October 25, 2016, 09:25:57 AM »
Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?

No, I didn't.  I was blindly following the API instructions.  Silly me.  I presumed that the only way to make use of the filtering from software was to screen scrape, but I see that is not true.  I was wishing for a little filtering.  They implemented this feature but forgot to talk about it in the API docs.  It is slightly different than downloading the full file, because you have to be logged in to use the filter. 

The best filter would be "listed in last X hours."

My hangup now is that there doesn't seem to be any way to calculate how late a note (that I do not own) is.  I presume "days since last payment" is counting days since ANY payment, which could be early, late, partial, extra, etc, and therefore is not the same thing as "days in arrears".  Also, even for loans only making payments at scheduled dates, the "days since last payment" field seems to be a few days off.  I haven't researched this enough to understand what they are up to.

If you are a human using the web pages, you can clearly see the dates of payments, etc, but software, using only the download file has no access to payment dates.

fliphusker

  • Sr. Member
  • ****
  • Posts: 463
    • View Profile
    • Email
Re: how often is folio inventory CSV file updated?
« Reply #13 on: October 25, 2016, 03:31:41 PM »
Would you mind sharing what your program does?  Does it auto buy? 
Suppose I write software that downloads this file and searches for notes to buy.  How often do I need to download this sucker?

Do they update it on a regular schedule, or whenever somebody posts a note for sale, or what?

Have you considered using Folio's "Download Search Results" link https://www.lendingclub.com/foliofn/notesRawData.action?

This link is more real-time; all Folio notes matching your search criteria will be included.  No noticeable delays.

My Folio program runs every 2-hr.

Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
Re: how often is folio inventory CSV file updated?
« Reply #14 on: October 25, 2016, 10:20:18 PM »
Would you mind sharing what your program does?  Does it auto buy? 

Yes, my program does auto buy.

It downloads the latest 500,000+ notes from Folio. runs risk-reward analysis on each, chooses some candidate notes to buy, and checks collection logs.  If nothing is problematic, submits purchase orders, then logouts.

It also does auto sell -- finding which of my notes are good to sell based on today's Folio prices.

This is all I do now in LC;  I do not buy notes from the primary market anymore.
« Last Edit: October 25, 2016, 11:59:44 PM by Fred »