Lend Academy Network Forum

Lending Club Discussion => Investors - LC => Topic started by: Rob L on June 07, 2014, 09:16:07 AM

Title: New URL for Browsenotes Download
Post by: Rob L on June 07, 2014, 09:16:07 AM
It appears that a few days ago LC changed the URL address for downloading browsenotes to:
https://resources.lendingclub.com/secure/primaryMarketNotes/browseNotes_1-RETAIL.csv (https://resources.lendingclub.com/secure/primaryMarketNotes/browseNotes_1-RETAIL.csv)
from:
https://www.lendingclub.com/browse/browseNotesRawDataV7.action (https://www.lendingclub.com/browse/browseNotesRawDataV7.action)
Did I miss the memo?
Title: Re: New URL for Browsenotes Download
Post by: brycemason on June 07, 2014, 10:36:51 AM
Are there any differences between the two links?
Title: Re: New URL for Browsenotes Download
Post by: lascott on June 07, 2014, 11:40:52 AM
Are there any differences between the two links?
I used Excel and saved them to a spreadsheet after shorting on the ID columns.

I use the compare tool: http://office.microsoft.com/en-us/support/basic-tasks-in-spreadsheet-compare-HA102834117.aspx

I didn't see any difference except in the very short time I downloaded both the funding amounts changed by $25 on some of the loans.

500+ rows in my example.
Title: Re: New URL for Browsenotes Download
Post by: Fred on June 07, 2014, 02:18:00 PM
The old link still works for me; seems to have the same number of rows and columns.
Title: Re: New URL for Browsenotes Download
Post by: AnilG on June 07, 2014, 02:21:56 PM
I smell some changes coming.  >:(

The old link still works for me; seems to have the same number of rows and columns.
Title: Re: New URL for Browsenotes Download
Post by: Rob L on June 07, 2014, 04:41:34 PM
This is another LC throttling maneuver.
A few months ago LC began returning an error rather than the browsenotes file for requests more closely spaced than 30 seconds.
Starting a few days ago a different error message (making reference to the new URL) is returned for requests more closely spaced than 2 minutes.
Note that if a download attempt is made and error returned, the timer resets to the full 2 minutes.

Yes Anil, "The times they are a changin"

Title: Re: New URL for Browsenotes Download
Post by: brycemason on June 10, 2014, 09:55:09 PM
RIP old link 14:01 10 June 2014.
Title: Re: New URL for Browsenotes Download
Post by: Fred on June 10, 2014, 10:42:12 PM
Yep, and the new link is hosted in what seems to be self-signed SSL certificate.  Had to write a custom X509TrustManager for that!
Title: Re: New URL for Browsenotes Download
Post by: Rob L on June 10, 2014, 11:08:53 PM
Gone but not forgotten!
@Fred; (I was lazy; just used the -k option)
Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 11, 2014, 12:33:03 AM
Be aware that there is a delay (probably more than 10 seconds) for new loans to be included in the downloaded file from the new download link.  I have to change my code to check the number of records in the downloaded CSV file to be sure that new loans are included.
Title: Re: New URL for Browsenotes Download
Post by: brycemason on June 11, 2014, 01:05:13 AM
What a headache. This may take many days to adjust.
Title: Re: New URL for Browsenotes Download
Post by: Fred on June 11, 2014, 01:07:47 AM
Be aware that there is a delay (probably more than 10 seconds) for new loans to be included in the downloaded file from the new download link.  I have to change my code to check the number of records in the downloaded CSV file to be sure that new loans are included.

Thanks.

The fact that the new link is a .csv seems suspicious to me.  I don't think the contents are generated on-demand.

The old link was an .action, which usually means some kind on-demand content generation.
Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 11, 2014, 01:25:09 AM
Be aware that there is a delay (probably more than 10 seconds) for new loans to be included in the downloaded file from the new download link.  I have to change my code to check the number of records in the downloaded CSV file to be sure that new loans are included.

Thanks.

The fact that the new link is a .csv seems suspicious to me.  I don't think the contents are generated on-demand.

The old link was an .action, which usually means some kind on-demand content generation.

This is exactly what I thought. They probably use this approach to reduce server load. The good news, at least I experienced, it seems there is no restriction on how frequently you can download that file. So, I am thinking just go directly download the file and check the number of loans in it, instead of use json to ping before download.
Title: Re: New URL for Browsenotes Download
Post by: GS on June 11, 2014, 02:49:49 PM
Peer Lending Server appears to not be working today.  LC's API is a disaster.  I feel for people who are trying to program around it.
Title: Re: New URL for Browsenotes Download
Post by: Fred93 on June 11, 2014, 03:44:44 PM
Peer Lending Server appears to not be working today.  LC's API is a disaster.  I feel for people who are trying to program around it.

Indeed.  I've done some of the work required to interface to LC's API, and this has made me aware of many missing pieces, difficulties, etc.  I thought I had worked around them all, but the constant changes which break implementations are frankly ridiculous.  If they don't want people to screen scrape, they should add the appropriate functions into the API so people won't be forced to screen scrape.  If they don't want people to poll, they should provide mechanisms allowing people to operate without polling.  If ...
Title: Re: New URL for Browsenotes Download
Post by: brycemason on June 11, 2014, 05:12:26 PM
I have no clue what is going on with the new browseNotes download. I sat at my terminal today at the 2pm load refreshing the "Browse Notes" page. I watched the count go from 720 or so at the top of the hour to well over 1000 somewhere around 2:00:30. Within a couple seconds it dropped to 900+. At the same time I was doing this, I was also downloading the loans via the new link to get a comparison of the application counts. It jumped from 720 to 923, but only after 2:01:00. Incidentally this is also when another server I have running saw the application count jump from 720 to 923. So, either I'm missing over 100 loans which got eaten up within seconds or they never really existed in the first place. It's at times like these when I hate this endeavor.
Title: Re: New URL for Browsenotes Download
Post by: brycemason on June 11, 2014, 05:14:33 PM
Clearly downloading the CSV is a bunk way to do business. I'll have to explore the API.

Edit: I retract this statement. The jury's still out.
Title: Re: New URL for Browsenotes Download
Post by: sociallender on June 11, 2014, 07:42:34 PM
Quote
Clearly downloading the CSV is a bunk way to do business. I'll have to explore the API.
Did you find that the CSV file is not updated in a timely manner relative to loan list time?  I am trying to determine if the new CSV file is the right way to download the notes.  The API does give a loan list (albeit in another format).  I am wondering if the delay also plagues the API?  Not sure where to invest time UGH!
Title: Re: New URL for Browsenotes Download
Post by: Fred on June 12, 2014, 10:35:54 AM
Did you find that the CSV file is not updated in a timely manner relative to loan list time?

Yes.

I am trying to determine if the new CSV file is the right way to download the notes.  The API does give a loan list (albeit in another format).  I am wondering if the delay also plagues the API?  Not sure where to invest time UGH!

The CSV file contains much richer credit data.  I think the challenge now is to determine how frequent the CSV file is updated.
Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 12, 2014, 11:04:44 PM
Did you find that the CSV file is not updated in a timely manner relative to loan list time?

Yes.

I am trying to determine if the new CSV file is the right way to download the notes.  The API does give a loan list (albeit in another format).  I am wondering if the delay also plagues the API?  Not sure where to invest time UGH!

The CSV file contains much richer credit data.  I think the challenge now is to determine how frequent the CSV file is updated.

I am using two auto-sliding time windows to assure covering the list release time and CSV update time while use minimal times to pinging the server or downloading the data file. The time windows will be moved according to the start time of the run and time of the events in last run. It works very well to handle uncertainty of the exact loan list release time and CSV updating following an loan release.
Title: Re: New URL for Browsenotes Download
Post by: GS on June 14, 2014, 12:12:21 AM
It seems like any delay between the new loans posting and the Download file being updated is just going to force the API users who are obsessed with getting a speed advantage to go back to screen scraping ....or using the shopping cart lock-out exploit more to hold loans hostage while the CSV updates ....

Time will tell, but I don't this change was for the better..
Title: Re: New URL for Browsenotes Download
Post by: brycemason on June 14, 2014, 09:10:19 AM
You've got it, GS. This delay between loans on the website and the API/CSV update completely killed me.
Title: Re: New URL for Browsenotes Download
Post by: faeriering on June 14, 2014, 03:32:10 PM
So the "Retail" part of the link irked me.  for whatever reason it makes me feel a little like a second class citizen so I tried this address instead:
https://resources.lendingclub.com/secure/primaryMarketNotes/browseNotes_1-INSTITUTIONAL.csv

and it worked, but I think they both have the same data in them.  ah well.
Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 18, 2014, 11:27:29 PM
It seems like any delay between the new loans posting and the Download file being updated is just going to force the API users who are obsessed with getting a speed advantage to go back to screen scraping ....or using the shopping cart lock-out exploit more to hold loans hostage while the CSV updates ....

Time will tell, but I don't this change was for the better..

So far, the # of loans showed on screen at the time of loan release is almost always equal to # of loans in the download file, about 1/3 chance to have one loan less. So, the impact is not significant at this moment. If the difference becomes bigger, shopping cart lock-out would be the easiest and fastest way to go.

The problem is, there were significant less loans passed my filter in past 3 days. Usually, there were average 5 to 6 passed my initial filter and close to half of them will be ordered everyday. However, there were total 3 loans were purchased in past 3 days. And half of times there was not any loan that even passed the initial filter. I am wondering if LC made change on rating the loans. I will see what will happen when my cash balance builds up and my worksheet expands its look-up range to include better rated loans.
Title: Re: New URL for Browsenotes Download
Post by: brycemason on June 18, 2014, 11:33:31 PM
I don't know what counts you've been looking at, but I see usually a decline of 80-150 between loan drop and the CSV.
Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 18, 2014, 11:42:07 PM
I don't know what counts you've been looking at, but I see usually a decline of 80-150 between loan drop and the CSV.

I am using the *retail* version. I only checked the # of loans (using json link) without comparing the list of loans on screen vs those in download file. Here is part of the log for the latest two loan releases, the timing is within 4 seconds after events.

@06/18 17:00:26      Loan listing was updated. Attempt # 6   # of loans: 1075
@06/18 17:00:44      Check N Loans in download file. Attempts: # 6:3   # of loans: 1075

@06/18 21:00:21      Loan listing was updated. Attempt # 5   # of loans: 1088
@06/18 21:00:46      Check N Loans in download file. Attempts: # 5:3   # of loans: 1088

The problem is that it's highly possible what we see in the download file may have already closed on LC BrowseNotes page. It has not occurred to the loans passed my filter, so far so good. But, I will add code to log the # of loans on screen at the time of downloading file to monitor it.
Title: Re: New URL for Browsenotes Download
Post by: Fred on June 19, 2014, 12:18:35 AM
The problem is that it's highly possible what we see in the download file may have already closed on LC BrowseNotes page.   It has not occurred to the loans passed my filter, so far so good.

I can confirm this has happened to me a few times already.
Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 19, 2014, 12:33:53 AM
That's bad news. So, I have to think about implementing shopping cart lockup in my worksheet so that it can use it when cash is over a limit.
Title: Re: New URL for Browsenotes Download
Post by: brycemason on June 19, 2014, 04:41:01 PM
RaymondG,

I don't think my point is coming through.

Poll the website for the loan count continuously between 17:00:00 and 17:00:50. You will see it jump to X.
Download the loan file continuously between 17:00:00 and 17:01:15. You will see it the number of rows jump to Y around 17:01:00 (but no sooner).

X will be greater than Y. Therefore, some loans never even make it into the download file.




Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 19, 2014, 09:06:33 PM
I saw your point but I had not done any test like you did. I have doubt because the two numbers are always either the same or having a difference of 1 according to my log. I have just added code to log the # of loans on screen, from json link, and in download files  in x seconds interval until the # rows in download file have big jump. I will see what will happen tomorrow.

As we know, another problem is the loans in download file might have already closed before we see it in file. So, either way, a solution is necessary.

Actually this change is not a big deal to me since it had always taken me 20 to 30 seconds to download the file before this change. The big concern now is that the loan rating algorithm in LC must have changed dramatically. Both the # of loans that passed my filter and the # of loans that were selected to order have dropped significantly in past 4 days. I am wondering if I should reexamine my filters, such as allow better rated loans.
Title: Re: New URL for Browsenotes Download
Post by: Fred on June 20, 2014, 02:38:56 AM
The big concern now is that the loan rating algorithm in LC must have changed dramatically.  Both the # of loans that passed my filter and the # of loans that were selected to order have dropped significantly in past 4 days.

Something similar occurred to me this past weekend; however, the number of loans meeting my criteria is back to normal today (Thursday). 

I think this is more about quality of weekend/early-week loans in the system, rather LC modifying their algo.
Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 20, 2014, 10:02:47 PM
The big concern now is that the loan rating algorithm in LC must have changed dramatically.  Both the # of loans that passed my filter and the # of loans that were selected to order have dropped significantly in past 4 days.

Something similar occurred to me this past weekend; however, the number of loans meeting my criteria is back to normal today (Thursday). 

I think this is more about quality of weekend/early-week loans in the system, rather LC modifying their algo.

It seemed to be back to normal today. Hope it's like what you thought.
Title: Re: New URL for Browsenotes Download
Post by: RaymondG on June 20, 2014, 10:11:00 PM
I saw your point but I had not done any test like you did. I have doubt because the two numbers are always either the same or having a difference of 1 according to my log. I have just added code to log the # of loans on screen, from json link, and in download files  in x seconds interval until the # rows in download file have big jump. I will see what will happen tomorrow.

The log of today showed that the # of loans on screen remained the same or dropped by 1, starting after the time when the jump of # of loans on screen was detected (at about 00:10) to the time when jump of # loans in download file was seen (after about 00:40). The interval is about 6 seconds. I am using the *retail* download link.