Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - sociallender

Pages: 1 2 [3] 4 5 ... 19
31
Investors - P / Re: Prosper API calls using cURL
« on: May 07, 2015, 08:16:16 PM »
Yep, if you try to get all ListingHistorical since 2007 in one API request, you will get timeout. It is 2+GB of data. Any API server call will time out with that much data. You need to use Odata to filter so that API response is smaller size. Also, don't make too many API requests in short time frame either as such requests start to look like DDoS attack pattern. The server security service will lock you out.

This is what I do at PeerCube:

1. Request about 500 to 1,000 records with one API request
2. Make no more than 60 API requests in an hour.


Just for the record, there is no limit to an HTTP request size that I am aware of (RESTful API).  If any are present, it is usually self imposed.  Also, I am not sure that the size of the data set would be 2+GB given proper compression (but dont' know since I have never seen the full data set).  The procedure you use above to download the data may work, but to anyone just trying to download the data, it seems a bit arduous. 

To be fair to Prosper, I can understand why they would want to break up the data into chunks.  Perhaps in the API, they could make different endpoints similar to the way LC gives separate links to their historicals based on date range. 


32
Investors - P / Re: Prosper API calls using cURL
« on: May 07, 2015, 07:55:25 PM »
What good is a richer dataset if you can't access it?  Or at least make it available with documentation on how to retrieve it. ... Using the API, connections are reset likely due to the amount of data that needs to be pushed.

I agree you should be able to do what the doc says, and it should work, and that they should be better at answering email problem reports.

However, to be fair to prosper, the timeout happens when doing big gets from ListingsHistorical, such as all listings, or all listings in 2007.

I can read all the Loan objects without error.  I would imagine, although I have not yet tried it, that one can then look at the listing number in each of these loan objects, and use ListingsHistorical to fetch the associated credit data for each of the 233,002 loans one at a time.   Of course some of the listings will be not yet available, right?

Of course tomorrow this scheme may also produce 404 errors or timeouts, and they may not answer email about these new problems, just as is happening now with the flat files and timeouts on ListingsHistorical.

I agree, I think the issue is the size of the data set.  They may need to do some tuning to get things working properly.  Perhaps go back to a flat file until they either get it working, or recommend a way to do it so that the developer doesn't have to guess what the correct filtering scheme should be.  Just a thought..

33
Investors - P / Re: Prosper API calls using cURL
« on: May 07, 2015, 07:50:54 PM »
Quote
Prosper always provided XML flat files for historical data till end of last year. Prosper still provides the download of data. Have you tried the links on API download page?

https://api.prosper.com/Downloads

Yes, and unfortunately, they do not contain the "richer" data set I was looking for.  It just contains a handful of variables.  Not of value to me.

Quote
Have you read Propser API documentation at https://api.prosper.com? It provides all the information you need to access API. It also explains how to use Odata. Also Odata is widely used so there is lot of information on the web to fine tune your queries.

Of course, that is the point.  I am fine with using the API, and actually prefer it over flat files.  As I mentioned before, you cannot download the historical listings without a connection reset from their servers unless you use Odata.  So, you are left with downloading the data set in chunks (such as by year or month).  Who knows if you are actually getting the full data set?  There is no mention of this limitation from Prosper.  Which leaves developers wasting hours of their time trying to determine if the connection resets were caused by the client.  I could point out more deficiencies but I am hoping you understand the point I am trying to make.   If you know of a way to gather the complete data set without a hack, please let me know.

Quote
See my point in last post, between Prosper and LC, whose data is used in published research on peer to peer lending. Lending Club has always provided very limited data compared to Prosper. Lending Club continue to reduce the amount of data it provides.

PeerCube supports both Lending Club and Prosper. So, I have deep insight into which platform is more in demand and how each platform is being used and what users are doing with each platform.

I agree with you the data is more robust.  I believe that more data is better.  There is no argument about that.  My sole issue is the ease in which a developer would obtain the historical data (should be an easy task).  Again, if you are going to remove the flat files in preference for the API, please make sure it works as advertised. 



34
Investors - P / Re: Prosper API calls using cURL
« on: May 07, 2015, 07:34:25 PM »
If you click on their export page you get 404 errors on the files (very unprofessional).

It is more professional now.  The 404 errors are gone.

https://www.prosper.com/invest/download.aspx
Quote
Performance > Download
Download Prosper Data
Data Export
We are sorry, but there are no files to download now.

Actually, I was referring to https://www.prosper.com/tools/DataExport.aspx
Click the first link "ProsperDataExport.xml.zip" to get 404 error.  Some of the other links show that Prosper site is under maintenance.  I am not even sure how to get to this tools page except through Google.  Anyways, they should remove any reference to flat file downloads from their site.



35
Investors - P / Re: Prosper API calls using cURL
« on: May 07, 2015, 06:29:09 PM »
Prosper only stopped providing flat files earlier this year. Prosper always has provided much richer dataset for their loans. Even now, richness of Prosper loan dataset is miles ahead than that of LC. There is lot more published research using Prosper dataset compared to LC dataset.

The lack of 3rd party ecosystem has nothing to do with dataset. It has to do with the lack of demand from retail individual investors.

Ok, this is absolutely ridiculous!  No flat files and an API that won't allow you to download the information unless you filter using Odata.  I always wondered why there was no real 3rd party developer ecosystem like LC...
What good is a richer dataset if you can't access it?  Or at least make it available with documentation on how to retrieve it.  If you click on their export page you get 404 errors on the files (very unprofessional). Using the API, connections are reset likely due to the amount of data that needs to be pushed.  Historical data is invaluable to data scientists for modeling purposes.  With the prevalence of automation these days one would expect more support in this area from prosper.  For the record, 2 emails have been sent to prosper for support without a response.

I completely disagree with you regarding the lack of 3rd party DEVELOPER ecosystem. 

36
Investors - P / Re: Prosper API calls using cURL
« on: May 07, 2015, 04:13:27 PM »
Ok, this is absolutely ridiculous!  No flat files and an API that won't allow you to download the information unless you filter using Odata.  I always wondered why there was no real 3rd party developer ecosystem like LC...

37
Investors - P / Re: Prosper API calls using cURL
« on: May 06, 2015, 09:59:05 AM »
I am having the same issue.  Connection reset when requesting Prosper historical loans.  Tried multiple REST clients.  Have you had any luck with this since?

38
Peer Lending Server / Re: Service Started but runs once then thats it!
« on: April 21, 2015, 02:57:34 PM »
Hmmm...  I am not sure what the issue is but will try and help.

First, when disabling the service from the scheduler, the status message on the main window has a 15 second maximum delay of notification.   Basically, its just a 15 second interval to see if there is a schedule to run the service.  Can you disable the service, wait at least 15 seconds, and see if it gets updated?

I am not sure what is causing the service not to execute on the second run.  Can you provide me with the messages on the log window?  You should be able to copy and paste the text and remove any info you don't want public.  On the second run, does it show "Starting PLS", then initial cash, list detection, etc?

SL

39
Peer Lending Server / Re: Multiple custom filters in version 7.7?
« on: April 10, 2015, 10:44:54 AM »
The custom filter box in the filter configuration window allows you to create a complex filter using joins and grouping.  You should be able to accomplish what you are asking by including the correct filter in the custom filter input box.  Note that custom filters use a slightly different syntax than standard SQL.  The bottom right portion of the filter screen shows how a filter is created (syntax) in real time when building from the GUI.  If the filter is incorrect, you should see errors on the right of the screen or no notes listed.  & and | are used for and and or.  Parenthesis are used for grouping.

SL

40
Peer Lending Server / Re: Service stalls after fetching note count
« on: March 02, 2015, 07:02:21 AM »
No problem.  Best, SL

41
Peer Lending Server / Re: Service stalls after fetching note count
« on: March 01, 2015, 09:42:25 PM »
Hmmm.. Not sure what the problem is.  Can you include the entries in the log up to where it hangs?  I haven't heard of anyone else having this problem yet and I am not seeing it on my end.  I am not sure at this point what it could be.  Just out of curiosity as well, can you restart PLS gracefully if you have not already tried that? 

SL


42
Investors - LC / Re: Payment History Incomplete?
« on: February 05, 2015, 04:29:57 PM »
Sorry, looks like the CSV download didn't complete.  Only got half of the file.  Re-downloading now...

43
Investors - LC / Payment History Incomplete?
« on: February 05, 2015, 04:27:26 PM »
I downloaded the payment history file from here (All Payments):

http://additionalstatistics.lendingclub.com/

I see approx 2.5MM rows of payment history with 103K unique loan ids present.  I don't understand why there are approx 400K unique notes in LC's historical statistics files but yet only 103K notes in the payment history?  Does anyone know why this is the case?  Am I missing something here?

SL
 

44
Peer Lending Server / Re: PLS 7 is a great improvement
« on: January 30, 2015, 06:43:34 PM »
Thanks, I really do appreciate the positive feedback.  I spent a good deal of time writing the code to be as efficient as possible using in-memory technology versus database ETL as in v6. I also worked on the intelligence model to also be very fast.  From list detection time including filter and modeling to sending the API order is usually less than a 1/10 of a second.  Probably faster for others using faster machines.  I haven't missed any notes that match my criteria in a while which is a big improvement over the previous version.  I was clocking around 15-20 seconds in previous versions with database schema validation, downloading the entire CSV, etc.. :)

The other nice benefit of in-memory tables is they are not susceptible to field removal (provided its not in your filter) and field re-ordering as LC does from time to time.  Should be more resilient than previous versions...

SL

45
Peer Lending Server / Re: What happens during a scheduled run?
« on: January 28, 2015, 08:35:38 AM »
You need to start PLS prior to loan list as the system queries LC for new notes.  If you start PLS after loan list, the full 300 queries will be sent before selecting your notes.  If you start 1 minute prior, which is the new default, you should get approximately 40 or so queries before new notes are listed and downloaded.

Pages: 1 2 [3] 4 5 ... 19