Author Topic: New URL for Browsenotes Download  (Read 9657 times)

Rob L

  • Hero Member
  • *****
  • Posts: 2028
    • View Profile
New URL for Browsenotes Download
« on: June 07, 2014, 09:16:07 AM »
It appears that a few days ago LC changed the URL address for downloading browsenotes to:
https://resources.lendingclub.com/secure/primaryMarketNotes/browseNotes_1-RETAIL.csv
from:
https://www.lendingclub.com/browse/browseNotesRawDataV7.action
Did I miss the memo?

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Re: New URL for Browsenotes Download
« Reply #1 on: June 07, 2014, 10:36:51 AM »
Are there any differences between the two links?

lascott

  • Hero Member
  • *****
  • Posts: 1410
    • View Profile
    • Appreciate my post and want to try LendingRobot? URL below
Re: New URL for Browsenotes Download
« Reply #2 on: June 07, 2014, 11:40:52 AM »
Are there any differences between the two links?
I used Excel and saved them to a spreadsheet after shorting on the ID columns.

I use the compare tool: http://office.microsoft.com/en-us/support/basic-tasks-in-spreadsheet-compare-HA102834117.aspx

I didn't see any difference except in the very short time I downloaded both the funding amounts changed by $25 on some of the loans.

500+ rows in my example.
Tools I use: (main) BlueVestment: https://www.bluevestment.com/app/pricing + https://www.interestradar.com/ , (others) Lending Robot referral link: https://www.lendingrobot.com/ref/scott473/  & Peercube referral code: DFVA9Y

Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
Re: New URL for Browsenotes Download
« Reply #3 on: June 07, 2014, 02:18:00 PM »
The old link still works for me; seems to have the same number of rows and columns.

AnilG

  • Hero Member
  • *****
  • Posts: 1088
    • View Profile
    • PeerCube
Re: New URL for Browsenotes Download
« Reply #4 on: June 07, 2014, 02:21:56 PM »
I smell some changes coming.  >:(

The old link still works for me; seems to have the same number of rows and columns.
---
Anil Gupta
PeerCube Thoughts blog https://www.peercube.com/blog
PeerCube https://www.peercube.com

Rob L

  • Hero Member
  • *****
  • Posts: 2028
    • View Profile
Re: New URL for Browsenotes Download
« Reply #5 on: June 07, 2014, 04:41:34 PM »
This is another LC throttling maneuver.
A few months ago LC began returning an error rather than the browsenotes file for requests more closely spaced than 30 seconds.
Starting a few days ago a different error message (making reference to the new URL) is returned for requests more closely spaced than 2 minutes.
Note that if a download attempt is made and error returned, the timer resets to the full 2 minutes.

Yes Anil, "The times they are a changin"


brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Re: New URL for Browsenotes Download
« Reply #6 on: June 10, 2014, 09:55:09 PM »
RIP old link 14:01 10 June 2014.

Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
Re: New URL for Browsenotes Download
« Reply #7 on: June 10, 2014, 10:42:12 PM »
Yep, and the new link is hosted in what seems to be self-signed SSL certificate.  Had to write a custom X509TrustManager for that!

Rob L

  • Hero Member
  • *****
  • Posts: 2028
    • View Profile
Re: New URL for Browsenotes Download
« Reply #8 on: June 10, 2014, 11:08:53 PM »
Gone but not forgotten!
@Fred; (I was lazy; just used the -k option)

RaymondG

  • Full Member
  • ***
  • Posts: 247
    • View Profile
    • Email
Re: New URL for Browsenotes Download
« Reply #9 on: June 11, 2014, 12:33:03 AM »
Be aware that there is a delay (probably more than 10 seconds) for new loans to be included in the downloaded file from the new download link.  I have to change my code to check the number of records in the downloaded CSV file to be sure that new loans are included.

brycemason

  • Hero Member
  • *****
  • Posts: 801
    • View Profile
    • P2P-Picks.com
    • Email
Re: New URL for Browsenotes Download
« Reply #10 on: June 11, 2014, 01:05:13 AM »
What a headache. This may take many days to adjust.

Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
Re: New URL for Browsenotes Download
« Reply #11 on: June 11, 2014, 01:07:47 AM »
Be aware that there is a delay (probably more than 10 seconds) for new loans to be included in the downloaded file from the new download link.  I have to change my code to check the number of records in the downloaded CSV file to be sure that new loans are included.

Thanks.

The fact that the new link is a .csv seems suspicious to me.  I don't think the contents are generated on-demand.

The old link was an .action, which usually means some kind on-demand content generation.

RaymondG

  • Full Member
  • ***
  • Posts: 247
    • View Profile
    • Email
Re: New URL for Browsenotes Download
« Reply #12 on: June 11, 2014, 01:25:09 AM »
Be aware that there is a delay (probably more than 10 seconds) for new loans to be included in the downloaded file from the new download link.  I have to change my code to check the number of records in the downloaded CSV file to be sure that new loans are included.

Thanks.

The fact that the new link is a .csv seems suspicious to me.  I don't think the contents are generated on-demand.

The old link was an .action, which usually means some kind on-demand content generation.

This is exactly what I thought. They probably use this approach to reduce server load. The good news, at least I experienced, it seems there is no restriction on how frequently you can download that file. So, I am thinking just go directly download the file and check the number of loans in it, instead of use json to ping before download.

GS

  • Sr. Member
  • ****
  • Posts: 413
    • View Profile
Re: New URL for Browsenotes Download
« Reply #13 on: June 11, 2014, 02:49:49 PM »
Peer Lending Server appears to not be working today.  LC's API is a disaster.  I feel for people who are trying to program around it.

Fred93

  • Hero Member
  • *****
  • Posts: 2128
    • View Profile
Re: New URL for Browsenotes Download
« Reply #14 on: June 11, 2014, 03:44:44 PM »
Peer Lending Server appears to not be working today.  LC's API is a disaster.  I feel for people who are trying to program around it.

Indeed.  I've done some of the work required to interface to LC's API, and this has made me aware of many missing pieces, difficulties, etc.  I thought I had worked around them all, but the constant changes which break implementations are frankly ridiculous.  If they don't want people to screen scrape, they should add the appropriate functions into the API so people won't be forced to screen scrape.  If they don't want people to poll, they should provide mechanisms allowing people to operate without polling.  If ...