Author Topic: stale incorrect data in notes-ext.csv file  (Read 2836 times)

Fred93

  • Hero Member
  • *****
  • Posts: 2231
    • View Profile
stale incorrect data in notes-ext.csv file
« on: November 25, 2016, 04:47:06 AM »
I sent a message to LC tonite, complaining about stale data in the notes-ext.csv file.  Some on this forum have noted that the last credit pull field shows data to be a month old.  The gist of that discussion has usually been that LC hasn't gotten around to pulling new data.  Turns out they have new data, and show it on the web page BUT JUST DON'T PUT THE NEW DATA INTO THE NOTES_EXT.CSV FILE!  On the same day, you can pull a notes_ext.csv file containing FICO scores more than a month old while the web page has the new data.  So, for example, if you have software that relies on the notes_ext.csv file for current FICO numbers, it will be using old stale data, while the people buying from you see the web page, and its new data.

My message to LC...

Today I sold a note, thinking that the borrower's FICO score had dropped significantly.  Turns out, that wasn't quite right.  I relied on the last-FICO fields in the notes-ext.csv file.  Rather, my software used these fields to decide which notes to sell.

However, I have learned that data is stale.  I noticed this first for note 46821141, (from loan 16041120) but I see that the same situation exists for many notes.

The notes-ext.csv file says that the last fico is 650-654, and the last credit update date is 10/20/2016.  That's just wrong.  Most of the current notes in the notes-ext.csv file contain a last credit pull date of 10/20/2016, but in all the cases I checked, the web page has november data.



If you look on the web page for this note, which my software cannot do of course, you see that the FICO score has been updated in November, and the new score is 700-704 !!!  A move of 50 points up is a substantial increase.  This data point is shown as dated 11/16/2016 on the graph.  That's 9 days ago.  However, the notes-ext.csv file pulled this evening, which sellers rely on, has not been updated.





I believe you should cancel this note sale, because you provided stale incorrect data.  I have no idea how many other note sales suffer this same condition.

You should be able to provide consistent data between the web pages and the notes-ext.csv file.  Both buyers and sellers should be given the same data.

Rob L

  • Hero Member
  • *****
  • Posts: 2117
    • View Profile
Re: stale incorrect data in notes-ext.csv file
« Reply #1 on: November 25, 2016, 10:55:53 AM »
FWIW I just downloaded the file containing all notes currently for sale:

https://resources.lendingclub.com/SecondaryMarketAllNotes.csv

There are two notes for sale now for your loan; one was listed on 11/18 and the other listed on 11/19.
The column titled FICO_End_Range has the value 700-704 and DaysSinceLastPayment column has the value 8.
So, automated buyers using this file also have the latest updated FICO (in addition to manual buyers using the web site).

It's interesting that notes-ext.csv does not have a NeverLate column but  AllNotes.csv does.
If LC were to fix the staleness problem they should also add NeverLate to notes-ext.csv as well.

The absence of NeverLate in notes-ext.csv forced me to "pre-list" my notes at a ridiculously high markup just to obtain NeverLate, then list them for real. As a work around you could do the same and get the latest FICO from your "pre-listed" notes in AllNotes.csv rather than from notes-ext.csv. Yeah, I know it shouldn't have to to work like that but how long will it be before (if ever) LC fixes the problem. As a bonus you will get the NeverLate flag too.

Fred93

  • Hero Member
  • *****
  • Posts: 2231
    • View Profile
Re: stale incorrect data in notes-ext.csv file
« Reply #2 on: November 25, 2016, 03:59:52 PM »
FWIW I just downloaded the file containing all notes currently for sale:

https://resources.lendingclub.com/SecondaryMarketAllNotes.csv

There are two notes for sale now for your loan; one was listed on 11/18 and the other listed on 11/19.
The column titled FICO_End_Range has the value 700-704

As I said, they're giving up-to-date info to note buyers, but stale info to note sellers.  No good.

Rob L

  • Hero Member
  • *****
  • Posts: 2117
    • View Profile
Re: stale incorrect data in notes-ext.csv file
« Reply #3 on: November 25, 2016, 05:07:23 PM »
FWIW I just downloaded the file containing all notes currently for sale:

https://resources.lendingclub.com/SecondaryMarketAllNotes.csv

There are two notes for sale now for your loan; one was listed on 11/18 and the other listed on 11/19.
The column titled FICO_End_Range has the value 700-704

As I said, they're giving up-to-date info to note buyers, but stale info to note sellers.  No good.

Yep, I was adding support to your complaint by pointing out that the flaw exists regarding automated buyers vs sellers as well.
Also I (we) don't know if the web site is updated simultaneously with the SecondaryMarketAllNotes file or not. Who knows? Three different times?
You might want to pass this along to LC as an update to your original complaint.

Meanwhile, there is a work around without resorting to https access.

Fred93

  • Hero Member
  • *****
  • Posts: 2231
    • View Profile
Re: stale incorrect data in notes-ext.csv file
« Reply #4 on: November 25, 2016, 05:30:58 PM »
Yep, I was adding support to your complaint by pointing out that the flaw exists regarding automated buyers vs sellers as well.

Thank you.

Quote
Also I (we) don't know if the web site is updated simultaneously with the SecondaryMarketAllNotes file or not. Who knows? Three different times?

Several folks have noted that the SecondaryMarketAllNotes file lags the web site.  I have observed this as well.  A person at LC told me that the SMAN file is updated once every 5 minutes. 

Quote
Meanwhile, there is a work around without resorting to https access.

...and the workaround is ..... What?

Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
Re: stale incorrect data in notes-ext.csv file
« Reply #5 on: November 30, 2016, 02:48:28 AM »
If you look on the web page for this note, which my software cannot do of course, ....

Have you tried calling https://www.lendingclub.com/fusionCharts/chart?  This should give you the FICO points in the chart:

  <trendLines>
    <line alpha="30" color="666666" displayvalue="499-" startValue="57.0"/>
    <line alpha="30" color="666666" displayvalue="520-524" startValue="152.0"/>
    ...
    <line alpha="30" color="666666" displayvalue="820-824" startValue="1292.0"/>
    <line alpha="30" color="666666" displayvalue="845-850" startValue="1387.0"/>
  </trendLines>
  <categories>
    <category label="04/13"/>
    <category label="05/13"/>
    ...
    <category label="10/16"/>
    <category label="11/16"/>
  </categories>

Fred93

  • Hero Member
  • *****
  • Posts: 2231
    • View Profile
Re: stale incorrect data in notes-ext.csv file
« Reply #6 on: November 30, 2016, 05:24:02 AM »
Have you tried calling https://www.lendingclub.com/fusionCharts/chart?  This should give you the FICO points in the chart:

I looked into it, but have not implemented.  For one thing, LC officially forbits "screen scraping", and I try to play it straight.  Also, you have to do that for each loan, and I have LOTS of loans.  Following their rule about hitting the server only 1 per second, my 5000 loans would take...  a long time.  Of course I could screen using the data I have and then scrape only a limited set of candidates, etc. 

I decided to confront them with the issue rather than writing a bunch of code and trying to hack around the problem.

Their first answer was as expected ... They told me they update on a different day every month so I should just get used to it.  I had to explain that I wasn't complaining about when they update, I was complaining about them giving different data to different users.  I told them that this doesn't seem important to them now because they've never taken folio seriously.  However, if they were stock brokers, and they were caught giving different data to different customers the implications would be substantial.  Time for them to grow up, take folio seriously, and do some damn software maintenance, and get rid of these manual update steps.  Using computers there is no reason in hell why one user community should get updated DAYS after another.

gaoc

  • Newbie
  • *
  • Posts: 10
    • View Profile
Re: stale incorrect data in notes-ext.csv file
« Reply #7 on: December 07, 2016, 11:10:46 AM »
Even the web page is not up to date. When a payment is just made, the payment history is updated but remaining payments is not updated. It takes a few days for remaining payments to catch up. Screen scraping is not a bad idea if you don't sell it too frequently.