New URL for Browsenotes Download

[brycemason]

I have no clue what is going on with the new browseNotes download. I sat at my terminal today at the 2pm load refreshing the "Browse Notes" page. I watched the count go from 720 or so at the top of the hour to well over 1000 somewhere around 2:00:30. Within a couple seconds it dropped to 900+. At the same time I was doing this, I was also downloading the loans via the new link to get a comparison of the application counts. It jumped from 720 to 923, but only after 2:01:00. Incidentally this is also when another server I have running saw the application count jump from 720 to 923. So, either I'm missing over 100 loans which got eaten up within seconds or they never really existed in the first place. It's at times like these when I hate this endeavor.

[brycemason]

Clearly downloading the CSV is a bunk way to do business. I'll have to explore the API.

Edit: I retract this statement. The jury's still out.

[sociallender]

Clearly downloading the CSV is a bunk way to do business. I'll have to explore the API.

Did you find that the CSV file is not updated in a timely manner relative to loan list time?  I am trying to determine if the new CSV file is the right way to download the notes.  The API does give a loan list (albeit in another format).  I am wondering if the delay also plagues the API?  Not sure where to invest time UGH!

[Fred]

Did you find that the CSV file is not updated in a timely manner relative to loan list time?

Yes.

I am trying to determine if the new CSV file is the right way to download the notes.  The API does give a loan list (albeit in another format).  I am wondering if the delay also plagues the API?  Not sure where to invest time UGH!

The CSV file contains much richer credit data.  I think the challenge now is to determine how frequent the CSV file is updated.

[RaymondG]

Did you find that the CSV file is not updated in a timely manner relative to loan list time?

Yes.

I am trying to determine if the new CSV file is the right way to download the notes.  The API does give a loan list (albeit in another format).  I am wondering if the delay also plagues the API?  Not sure where to invest time UGH!

The CSV file contains much richer credit data.  I think the challenge now is to determine how frequent the CSV file is updated.

I am using two auto-sliding time windows to assure covering the list release time and CSV update time while use minimal times to pinging the server or downloading the data file. The time windows will be moved according to the start time of the run and time of the events in last run. It works very well to handle uncertainty of the exact loan list release time and CSV updating following an loan release.

[GS]

It seems like any delay between the new loans posting and the Download file being updated is just going to force the API users who are obsessed with getting a speed advantage to go back to screen scraping ....or using the shopping cart lock-out exploit more to hold loans hostage while the CSV updates ....

Time will tell, but I don't this change was for the better..

[brycemason]

You've got it, GS. This delay between loans on the website and the API/CSV update completely killed me.

[faeriering]

So the "Retail" part of the link irked me.  for whatever reason it makes me feel a little like a second class citizen so I tried this address instead:
https://resources.lendingclub.com/secure/primaryMarketNotes/browseNotes_1-INSTITUTIONAL.csv

and it worked, but I think they both have the same data in them.  ah well.

[RaymondG]

It seems like any delay between the new loans posting and the Download file being updated is just going to force the API users who are obsessed with getting a speed advantage to go back to screen scraping ....or using the shopping cart lock-out exploit more to hold loans hostage while the CSV updates ....

Time will tell, but I don't this change was for the better..

So far, the # of loans showed on screen at the time of loan release is almost always equal to # of loans in the download file, about 1/3 chance to have one loan less. So, the impact is not significant at this moment. If the difference becomes bigger, shopping cart lock-out would be the easiest and fastest way to go.

The problem is, there were significant less loans passed my filter in past 3 days. Usually, there were average 5 to 6 passed my initial filter and close to half of them will be ordered everyday. However, there were total 3 loans were purchased in past 3 days. And half of times there was not any loan that even passed the initial filter. I am wondering if LC made change on rating the loans. I will see what will happen when my cash balance builds up and my worksheet expands its look-up range to include better rated loans.

[brycemason]

I don't know what counts you've been looking at, but I see usually a decline of 80-150 between loan drop and the CSV.

[RaymondG]

I don't know what counts you've been looking at, but I see usually a decline of 80-150 between loan drop and the CSV.

I am using the *retail* version. I only checked the # of loans (using json link) without comparing the list of loans on screen vs those in download file. Here is part of the log for the latest two loan releases, the timing is within 4 seconds after events.

@06/18 17:00:26      Loan listing was updated. Attempt # 6   # of loans: 1075
@06/18 17:00:44      Check N Loans in download file. Attempts: # 6:3   # of loans: 1075

@06/18 21:00:21      Loan listing was updated. Attempt # 5   # of loans: 1088
@06/18 21:00:46      Check N Loans in download file. Attempts: # 5:3   # of loans: 1088

The problem is that it's highly possible what we see in the download file may have already closed on LC BrowseNotes page. It has not occurred to the loans passed my filter, so far so good. But, I will add code to log the # of loans on screen at the time of downloading file to monitor it.

[Fred]

The problem is that it's highly possible what we see in the download file may have already closed on LC BrowseNotes page.   It has not occurred to the loans passed my filter, so far so good.

I can confirm this has happened to me a few times already.

[RaymondG]

That's bad news. So, I have to think about implementing shopping cart lockup in my worksheet so that it can use it when cash is over a limit.

[brycemason]

RaymondG,

I don't think my point is coming through.

Poll the website for the loan count continuously between 17:00:00 and 17:00:50. You will see it jump to X.
Download the loan file continuously between 17:00:00 and 17:01:15. You will see it the number of rows jump to Y around 17:01:00 (but no sooner).

X will be greater than Y. Therefore, some loans never even make it into the download file.

[RaymondG]

I saw your point but I had not done any test like you did. I have doubt because the two numbers are always either the same or having a difference of 1 according to my log. I have just added code to log the # of loans on screen, from json link, and in download files  in x seconds interval until the # rows in download file have big jump. I will see what will happen tomorrow.

As we know, another problem is the loans in download file might have already closed before we see it in file. So, either way, a solution is necessary.

Actually this change is not a big deal to me since it had always taken me 20 to 30 seconds to download the file before this change. The big concern now is that the loan rating algorithm in LC must have changed dramatically. Both the # of loans that passed my filter and the # of loans that were selected to order have dropped significantly in past 4 days. I am wondering if I should reexamine my filters, such as allow better rated loans.