Topics

1

Investors - LC / Help on python script to download browseNotes

« on: November 09, 2018, 09:31:50 AM »

I am trying to automate my investment. The only piece of the puzzle left for me is to download the market notes CSV. I'm aware that you can download the data through the lending club API. However, the data is not the same; there are fewer columns and often fewer notes. So I want to write a script to download the CSV. To do this you have to navigate to the sign in page, sign in, navigate to the manual invest page, and find the download link to the CSV. The reason you can't go directly to the CSV download link is that there is dynamic information in the link. For example:

https://resources.lendingclub.com/secure/primaryMarketNotes/browseNotes_1-RETAIL.csv?signature=iMR1LOLGT7LO1Rjx5ktJTn8uA%3D&issued=15417747660

I can sign in, navigate to the manual investment page, and find the dynamic link. But I can't seem to download the file. Here is my python code so far.

Code: [Select]

import requests
import re

username = 'email_address'
password = 'password'


# Start session
s = requests.Session()

# Navigate to the login page with login information in the header
url_login = "https://www.lendingclub.com/account/login.action"
header = {'login_email': username,'login_password':password}
r1 = s.post(url_login, data=header)

# Navigate to the browse notes to find the download link (it changes)
url_manual_invest = 'https://www.lendingclub.com/browse/browse.action'
r2 = s.get(url_manual_invest)

# Get the html code on the manual investment page
browse_all_html = r2.text

# Find the and extract the download link in the html code
pattern = '"https://resources.lendingclub.com/secure/primaryMarketNotes.*"'
text_result = re.findall(pattern, browse_all_html)

# The address is the first and only element in the list
address_with_quotes = text_result[0]

# Remove the quotes
url_notes_download = address_with_quotes[1:-1]

# So far everything is successful I have the download link for the CSV file
# However I can't seem to get the CSV data. Perhaps because it's a download link
# Here's what I have tried

r3 = s.get(url_notes_download)
data = r3.text
The page that I'm directed to is:

https://static.lendingclub.com/www/terms-of-use-violation.html?TS-request-id=1652131192857168

Does anyone know how to get this CSV?