Desktop-based FolioFn client?

[CapitalSigma]

Hey all,

I'm a software developer who's interested in P2P lending. I've been poking around the LC site and it appears that they've (perhaps accidentally) exposed a RESTful API for FolioFn. I don't think that it would be very difficult to wrap it up in a GUI and distribute it. My goal would be to make something lightweight and easy to use -- a replacement for the existing, painful UI that Folio provides.

Would the LC community be interested in a desktop-based client for FolioFn, likely for a one-time fee? If it's something that people would use, I'm open to feature requests.

[New Jersey Guy]

Rev has an outstanding dedicated platform on Interest Radar for Folio.
Unfortunately, Lending Club appears to have changed their attitude.  From what I understand, they keep throwing his ISP's out and his service has been up/down over the past 2-3 weeks.

Currently, it's down.  It's been down since the 10th.  He seems skeptical on whether or not he'll be able to continue his Folio platform at all.

This is a serious problem for us who are Folio-only.  Without the filters, it's very difficult to find fresh loans, and nearly impossible to find past-issued notes.

Also, I buy/trade distressed notes.  Without the filters, I can't operate that part of my investing (Unless I want to look at the same 5,400 notes everyday)

In other words, I've been unable to buy a single note in 2-days.  And it looks like the drought is going to continue.

Somebody needs to come up with a solution.  There are more of out there than you know.

[CapitalSigma]

Interesting. That sort of thing wouldn't be an issue with a desktop client.

What features would you consider to be a "must have" for a FolioFn platform?

[core]

Interesting. That sort of thing wouldn't be an issue with a desktop client.

Actually, it could be an issue, and with far worse consequences for the poor end user who happened to trigger the Wrath of Lending Club.  Particularly if the user has a semi static IP and doesn't have the skills/desire to monkey with changing it.  It would depend on how the app was designed.  Rev (the author of IR) has declined to share what exactly went on when he called up to find out why his IPs were blocked so I can only guess as to how much trouble it was for him to get things going again the first time it happened. 

I'm a software developer who's interested in P2P lending. I've been poking around the LC site and it appears that they've (perhaps accidentally) exposed a RESTful API for FolioFn.

That sounds like a pretty serious charge... LendingClub "accidentally" exposing something to the world which was never intended to be used.  And you discovered this just from "poking around" the LC site itself?  I'm no expert on the various labels folks put on stuff these days, but it's my understanding the web itself is REST architecture.  A web browser arguably uses REST all the time, nothing unique.  Based on the URLs I see on the site and how the site works, you could argue that this is just how the site works.  These javascript-triggered requests have to talk some language, take your pick, and I wouldn't consider normal site operation an "API" so to speak. 

I for one would like more information on what you have discovered and why you think it was accidental.  How long have you been "poking around" on LC's site, and to what extent?  Have you actually using this API or is this something you just discovered quite recently and haven't even had a chance to play with it?

[CapitalSigma]

Actually, it could be an issue, and with far worse consequences for the poor end user who happened to trigger the Wrath of Lending Club.  Particularly if the user has a semi static IP and doesn't have the skills/desire to monkey with changing it.  It would depend on how the app was designed.  Rev (the author of IR) has declined to share what exactly went on when he called up to find out why his IPs were blocked so I can only guess as to how much trouble it was for him to get things going again the first time it happened.

Well, maybe. When I talked to LC about getting access to their API, their main concern was that they wanted to keep the volume of automated trades relatively small. If Rev angered the LC gods, it seems likely that it was because the volume of his trades got too high. Also, as I'll explain in a minute, what I'm referring to as "the Folio REST API" is a hook into their database that the Folio page uses to pull up the list of loans. I can't think of any reason why someone using a desktop-based client would generate so much more traffic on that page than a normal browser that it would raise any eyebrows at LC.

That sounds like a pretty serious charge... LendingClub "accidentally" exposing something to the world which was never intended to be used.  And you discovered this just from "poking around" the LC site itself?

I looked through the source of the FolioFn page and I couldn't figure out how it was populating the list of loans, so I looked through all of the Javascript that it was pulling in from other parts of the site, and I found this page: https://www.lendingclub.com/mainapp/scripts/51731/combo/e16630ef2a2773a9bec9d81415634fabcf2d0b5539cfe377087b1a414b4a0.js. Inside of that, I found this fragment:

Code: [Select]

var buildQueryString = function (oState, oSelf) {
            oState = oState || {
                pagination: null,
                sortedBy: null
            };
            var sort = (oState.sortedBy) ? oState.sortedBy.key : "opa";
            var dir = (oState.sortedBy && oState.sortedBy.dir === YAHOO.widget.DataTable.CLASS_DESC) ? "desc" : "asc";
            var startIndex = (oState.pagination) ? oState.pagination.recordOffset : 0;
            var pSize = (oState.pagination) ? oState.pagination.rowsPerPage : 15;
            var myQuery = "&sortBy=" + sort + "&dir=" + dir + "&startindex=" + startIndex + "&newrdnnum=" + Math.floor(Math.random() * 100000000) + "&pagesize=" + pSize;
            return myQuery
        };
which lets you make queries against the Folio database.

I'm no expert on the various labels folks put on stuff these days, but it's my understanding the web itself is REST architecture.  A web browser arguably uses REST all the time, nothing unique.  Based on the URLs I see on the site and how the site works, you could argue that this is just how the site works.  These javascript-triggered requests have to talk some language, take your pick, and I wouldn't consider normal site operation an "API" so to speak. 

Yes, but you should care because the results of queries against that page look like this:

Code: [Select]

{
    "result": "success",
    "searchresult": {
        "loans": [{
            "days_since_payment": 29,
            "selfNote": 0,
            "accrued_interest": 0.04,
            "loan_status": "Current",
            "loanGrade": "B",
            "asking_price": 8.25,
            "isOnPayPlan": false,
            "noteId": 3642501,
            "ytm": "-149.76",
            "outstanding_principal": "4.87",
            "loanGUID": 662813,
            "title": "Debt Consolidation Loan",
            "markup_discount": "68.08",
            "credit_score_trend": 0,
            "loanClass": 36,
            "checkboxes": false,
            "loanRate": "10.00",
            "remaining_pay": 6,
            "orderId": 6551512
        },
                 
        .......
        {
            "days_since_payment": 30,
            "selfNote": 0,
            "accrued_interest": 0.03,
            "loan_status": "Current",
            "loanGrade": "A",
            "asking_price": 9.9,
            "isOnPayPlan": false,
            "noteId": 4227049,
            "ytm": "-110.23",
            "outstanding_principal": "6.14",
            "loanGUID": 712319,
            "title": "SPA LOAN",
            "markup_discount": "60.33",
            "credit_score_trend": 0,
            "loanClass": 36,
            "checkboxes": false,
            "loanRate": "5.79",
            "remaining_pay": 8,
            "orderId": 6331473
        }]
    },
    "totalRecords": 53376
}
That's JSON. I think it's significant because on this site I've seen it asserted that the only way to automate FolioFn is screen scraping. Screen scraping 53,376 pages worth of loans is prohibitively slow. Depending on how fast your internet connection is, it might take hours. If it was true that screen scraping was the only way to automate Folio, then the only kind of trading platform that could exist is what IR has now -- a huge, server-side, subscription-based service.

On the other hand, parsing JSON is fast -- you can do 53k loans in well under a minute -- and the fact that the data I've posted above exists means that it's possible to build something small and lightweight that gives control directly to the investor rather than forcing him to go through a 3rd party service like IR. It would also give people something to fall back on in case IR runs in to any more difficulties in the future.

I for one would like more information on what you have discovered and why you think it was accidental.  How long have you been "poking around" on LC's site, and to what extent?  Have you actually using this API or is this something you just discovered quite recently and haven't even had a chance to play with it?

See above. "Accidental" is probably too strong of a word, but the fact that it's completely undocumented and hidden inside pages of obfuscated Javascript doesn't suggest that it's something they expect anyone to use. I haven't seen anyone on here or Lending Club Talk mention it before. It provides a way to entirely avoid the terrible user interface that LC has provided for Folio users.

I haven't used it yet -- I found this yesterday during work while I was waiting for some tests to finish running. I started writing scripts to interact with it last night. I figured it was best to stick a thread up on here about it sooner rather than later because I didn't want to sink 40+ hours in to building something nice-looking that no one other than myself would ever use.

[core]

When I talked to LC about getting access to their API, their main concern was that they wanted to keep the volume of automated trades relatively small.

When you say "trades" I'm assuming you mean new note purchases, because that's all the normal API allows you to do.  You're serious about them saying they want to keep that volume low???  There are institutional investors making a LOT of purchases.  And I'm almost certain that some of them are hitting the site very very hard around feeding times trying to be the first to get the drop on the next guy.  Did they give you access?  Did they tell you what kind of traffic volume would be kosher?

I can't think of any reason why someone using a desktop-based client would generate so much more traffic on that page than a normal browser that it would raise any eyebrows at LC.

Well I sure can.  A person can't go through 77k notes in a minute like you were talking about.  I have a feeling that right there would raise some eyebrows eventually.

As for the rest of your post, thanks for all the techincal details.  I can tell you spent a while poking around.  I do have some bad news for you though:

That's just the standard way the site works, with a browser I mean.  If you want to call that a REST API, I suppose I can't argue with that.  Any time you see "Loading results" on a page you know this type of thing is going on.  Sure, the results are returned in JSON.  Yes, parsing JSON can be a tiny bit faster than other things. 

In this case though, that's the only way to get the results to my knowledge.  That's how your browser gets them.  Any "screen scraper" is already doing exactly what you propose.  I can understand how you might not want to call it a screen scraper just because some of it is parsing JSON.  But there are no longer any pages on the site that return the results embedded in the page html -- it's all like this now.

By the way, a sniffer is _much_ faster than digging through countless pages of javascript.  Finding the above request & response is as simple as starting the sniffer, doing one search, and then examining the generated traffic.  Takes maybe 20 seconds.  Of course that doesn't tell you if there are additional undocumented query string params which are not in use.  I didn't see any surprising ones in your javascript snippet though.

Anyway, regardless of what you want to call the method, that doesn't change the fact that there are no doubt several people here who would be interested in an application.  I'm sure they will chime in with what they'd want to see from one.  I'm kinda wondering if it would be economically feasible for someone to write something like this for a one-time fee.  How much were you thinking about charging?

[Randawl]

Although IR is nice for Folio, I and many others I know would be interested in something of this nature.  I think it would be worth your effort.

[Rob L]

Why is it presumed that IR is not already using this method of interacting with the FolioFn database?

[CapitalSigma]

When you say "trades" I'm assuming you mean new note purchases, because that's all the normal API allows you to do.  You're serious about them saying they want to keep that volume low???  There are institutional investors making a LOT of purchases.  And I'm almost certain that some of them are hitting the site very very hard around feeding times trying to be the first to get the drop on the next guy.  Did they give you access?  Did they tell you what kind of traffic volume would be kosher?

They gave me access -- they mostly wanted details about the amount that I was investing, when I told them I was only going to invest a few thousand dollars, they said it was fine. My impression was that they were concerned with institutional investors putting in huge amounts of cash through the API and locking out the normal folks entirely.

Well I sure can.  A person can't go through 77k notes in a minute like you were talking about.  I have a feeling that right there would raise some eyebrows eventually.

They get all of the notes in a single request from the server, there's no indication on the LC side that they're doing anything unusual.

That's just the standard way the site works, with a browser I mean.  If you want to call that a REST API, I suppose I can't argue with that.  Any time you see "Loading results" on a page you know this type of thing is going on.  Sure, the results are returned in JSON.  Yes, parsing JSON can be a tiny bit faster than other things. 

Parsing JSON is exponentially faster than reading through 500 pages of the site and parsing the HTML of each page to get what you want. It's a question of "Start up program, wait two hours for results to load, try to buy some things, discover that the loans you loaded two hours ago have already expired, get frustrated, wait two hours for the results to load again..." vs "Open program. Load loans in 30 seconds. Buy. Be done."

But there are no longer any pages on the site that return the results embedded in the page html -- it's all like this now.

I don't follow what you mean by this, was the Folio page previously formatted differently? It seems like the results are loaded into the page with AJAX with a call to the page that I listed above and rendered with HTML. It would not be easy to click through all of the page to get the AJAX calls to execute and parse the HTML that results from that.

By the way, a sniffer is _much_ faster than digging through countless pages of javascript.  Finding the above request & response is as simple as starting the sniffer, doing one search, and then examining the generated traffic.  Takes maybe 20 seconds.  Of course that doesn't tell you if there are additional undocumented query string params which are not in use.  I didn't see any surprising ones in your javascript snippet though.

I used a sniffer (Chrome's builtin dev tools) to find the URL of the page that the AJAX request was getting sent to and then searched through the Javascript loaded by the page for it to find out how the parameters of the request needed to be formatted. As far as I know there is no faster way to do this aside from looking at the request and hoping to guess the parameters correctly. I could be wrong.

Anyway, regardless of what you want to call the method, that doesn't change the fact that there are no doubt several people here who would be interested in an application.  I'm sure they will chime in with what they'd want to see from one.  I'm kinda wondering if it would be economically feasible for someone to write something like this for a one-time fee.  How much were you thinking about charging?

I'm not sure, it depends on how difficult it ends up being. I plan to release something small and limited (I'm thinking the basic LC filters plus whatever is easy to add based on the data) for fairly cheap at first and then release some more complete, comprehensive versions for a higher price down the line based on feedback. I don't want to commit to anything at the moment, but it should cost substantially less than a year's membership at IR. I don't have to pay for hosting or deal with any of the overhead that Rev does, so there's no reason for me to charge that much.

[core]

My impression was that they were concerned with institutional investors putting in huge amounts of cash through the API and locking out the normal folks entirely.

↑↑ Wow, are all the rest of you guys reading this?  Well, CapitalSigma I know you're not a LC shill based on the other stuff you wrote, so I'm taking this at face value.  Interesting indeed.

Parsing JSON is exponentially faster than reading through 500 pages of the site and parsing the HTML of each page to get what you want.

Where are you getting 500 pages?  There's no HTML to parse because the search results aren't in the html, like I previously said (and so did you).  It's a single request, provided you set your pagesize sufficiently high.  A scraper bot would need not even fetch the surrounding .html page first because it has no effect.  Aside from realism.

Let me ask you this:  If you could add ?pagesize=999999999 to a URL, and the results came back in an html <table> suitable for dynamic html use instead of JSON, I'm guessing that would change something?  Even though nothing is different besides besides there are <td>'s instead of double quotes around things? 

I don't follow what you mean by this, was the Folio page previously formatted differently? It seems like the results are loaded into the page with AJAX with a call to the page that I listed above and rendered with HTML. It would not be easy to click through all of the page to get the AJAX calls to execute and parse the HTML that results from that.

Yes as I recall year(s) back you could get results out of the html because they were present there.  And yes, the results are loaded into the page with AJAX or similar.  But if someone were to write a scraper they certainly would not monkey with "AJAX calls" or simulated clicks and parse rendered html... that would be absolute insanity.  Ohhhhh wait a minute, maybe by "screen scraper" you are thinking of some type of browser control or something?  Nah I'm strictly talking about raw HTTP requests.  Screen/page scraper to me means imitating a browser's HTTP requests, not actually using a browser, javascript engine, etc.

However keep one thing in mind with all this:  The more results you ask from them the longer it takes them to return em.  This may or may not be significant to someone's goals.  Also I believe 1000 is the max pagesize listed in the dropdown so any more than that and they have a real easy way to flag you as a bot.

[Deming]

I'm a software developer who's interested in P2P lending. I've been poking around the LC site and it appears that they've (perhaps accidentally) exposed a RESTful API for FolioFn. I don't think that it would be very difficult to wrap it up in a GUI and distribute it. My goal would be to make something lightweight and easy to use -- a replacement for the existing, painful UI that Folio provides.

Would the LC community be interested in a desktop-based client for FolioFn, likely for a one-time fee? If it's something that people would use, I'm open to feature requests.

Yes, absolutely!  Is it completed?