Trouble with /foliofn/notesRawData.action

[mark78]

Hi, first time post.. From the Folio Browse Notes page, I can set my filters and "Download Search Results". That works fine. Now I want to do the same thing programatically using https://www.lendingclub.com/foliofn/notesRawData.action, but I get a "page does not exist" web page instead of the data I expect.

As Fred93 said (https://forum.lendacademy.com/index.php/topic,2231.15.html), you have to be logged in. Well I'm logged in with Chrome, Firefox, and IE, but it still doesn't work.

Has anyone else had this problem? Thanks!

[Fred93]

Hi, first time post.. From the Folio Browse Notes page, I can set my filters and "Download Search Results". That works fine. Now I want to do the same thing programatically using https://www.lendingclub.com/foliofn/notesRawData.action, but I get a "page does not exist" web page instead of the data I expect.

As Fred93 said (https://forum.lendacademy.com/index.php/topic,2231.15.html), you have to be logged in. Well I'm logged in with Chrome, Firefox, and IE, but it still doesn't work.

We have to be clear on what "you have to be logged in" means.  It means that when you issue the HTTPS operation, you have to provide cookies which identify you as logged in.  The program you wrote probably does not access the cookie store used by any of these browsers.  This means that you will need to perform the actions of logging in within your code, save the resulting cookies, and then provide those cookies when you ask for the data in question.  Then LC will see your request as coming from someone who is "logged in".  In addition, you should add logic which handles the situation if those cookies go bad, either because your login timed out, or they rebooted their server, etc.  In those cases your program would need to login again... just like a human.

Alternately, you can ask LC for the updated LC/folio API document from about a year ago, which contains an endpoint that gets you inventory information.  That will let you avoid screen scraping with its attendant cookie issues, etc.  Unfortunately the new API endpoint does not contain filters like the web page does, and of course it has foibles of its own.  They have promised me several times that they would get around to updating the LC/folio web pages to link to this newer document.  Its only been a year, so have patience.

[mark78]

Thanks for the overview, Fred93. It's working now with screen scraping, and I'll ask LC for the updated API docs as an alternate approach. The fact that the new API endpoint does not have filters doesn't bother me too much, as I've been working with SecondaryMarketAllNotes.csv so far, and can process that pretty fast.

[kuhnrl30]

Fred93- 

When you say endpoint, do you mean the link for downloading the csv file or is there now a true endpoint?

Sent from my SM-G920V using Tapatalk

[kuhnrl30]

Its a new endpoint.

[hdsouza]

I have been downloading the full CSV download of secondary notes, but of late I have been seeing that LC has been throttling that full CSV download. For example yesterday from 5 PM EST - past midnight, it was taking me +1 hour to down the CSV file. This would normally take about 10 seconds.

So this morning I attempted using the API to download the secondary notes and the data does not have the ORDER id. Without the ORDER ID you cannot buy a note. you need the note , loan and order IDs.
Anyone else seeing the same thing?

[mark78]

I saw the same thing with the API- no Order ID, so it's worthless. I emailed them a week ago and they said they would forward my concerns.

Also I don't think you could pre-populate a database to look up the missing Order ID, as the Order ID was just assigned at the time the note was offered for sale (like one second or one hour ago). I'm not sure about that though.

[Fred93]

They updated the folio API a year ago, but have never posted the update to the web site.

Ask them to post the updated API doc from a year ago.

[mark78]

Fred93, maybe we're missing something. I have the Word document 'Lending Club Secondary Market APIs Documentation.docx'. The Word properties say it was last modified 1/12/2018. Section 1 is the 'Listings' API, partially copied below. But it doesn't return the Order ID, so we don't see the value. (Sounds like you've gotten it to work though  )

---

1. Listings

This subresource allows an investor to see the Notes currently listed on the secondary market.

Operation: GET

URL: https://api.lendingclub.com/api/investor/v1/secondarymarket/listings

Query Parameters

Name   Type   Optional   Description
updatedSince   Integer   Yes   Filter listings to include only ones that were modified/created within the last X minutes.

---

[Fred93]

Fred93, maybe we're missing something. I have the Word document 'Lending Club Secondary Market APIs Documentation.docx'. The Word properties say it was last modified 1/12/2018. Section 1 is the 'Listings' API, partially copied below. But it doesn't return the Order ID, so we don't see the value. (Sounds like you've gotten it to work though  )

Wow.  Are you sure it's 2018?  I have a doc with last mod 1/12/2017 by Dhanya Moni, and one with 5/4/2017.  I think the later document just corrects some typos.  Same title as you wrote above.  Both of these documents contain the new buy & sell API endpoints which don't require the order ID.

Just to be super clear here... Are you saying that the doc dated 1/12/something that they sent you does not contain the new buy & sell API endpoints?

PS: When this was new, I tried and tried over and over to get them to return the "loan ID" and "order ID" in the listings endpoint, but was unsuccessful.  Wasn't even any conversation.  It was as if somebody snuck in a few minutes to wedge in a couple of new endpoints and was immediately assigned to other things and couldn't take time to consider anything.  My logic was that one needs to be able to get information about the LOAN, and the only way to do that is to have the loan ID.  If you want to access the note web page, for example, you need the triple: note ID, loan ID, order ID.  I think this was done way back when as some sort of wacko pseudo-security feature.  It's mostly a pain in the ass.

[mark78]

Fred93, thank you for being super-clear. By 'the new endpoint', I thought we were only talking about the 'Listings' endpoint shown above. I didn't notice that the new API documentation (whether from 2017 or 2018) also includes a new corresponding 'Buy' endpoint which does not require the Order ID.

(So when I complained to them a week ago, they should have told me 'RTFM, you don't need the Order ID anymore'. But no...)

[Fred93]

It is typical that when you speak with LC the front line folks don't know anything about your question.

Good to see you made it thru the maze.  Sorry to see another competitor tho.  The secondary market is damned competitive. 

[hdsouza]

Yeah. they have the new documentation on the site now.
Its at >  LC > Foliofn > My Account > look for "Folio API Documentation Pages" (at the bottom of the page)
Sadly to say that the new api does not work.... at least for me. It gives me the dreaded "Internal server error"   

The earlier APIs work fine, for now (with the order ID) so I hope they do not force us in time, to use the new APIs.

[Fred93]

Yeah. they have the new documentation on the site now.

Wow.  They really prettied it up since the version they gave me a year ago.  Looks like only very minor text changes, not counting the new and exciting legalistic footnotes.  Mostly formatting.

Sadly to say that the new api does not work.... at least for me. It gives me the dreaded "Internal server error"   

Unfortunately it is a common characteristic of these web APIs that they won't talk with you at all until you get about 20 things exactly right.
http://memory-alpha.wikia.com/wiki/Jarada

I can assure you that once you find the detail you have wrong, you will get past "internal server error".