Author Topic: Model score  (Read 2109 times)

p2pinvestor2k

  • Newbie
  • *
  • Posts: 2
    • View Profile
Model score
« on: November 03, 2015, 10:18:20 PM »
I am looking at the filter criteria that indicates Model>85.

So what exactly is the model built on?

Is this a neural network?

Is the "brains" ever tweaked?

What can I look at to understand what the model score is based on?

sociallender

  • Sr. Member
  • ****
  • Posts: 273
    • View Profile
    • Email
Re: Model score
« Reply #1 on: November 09, 2015, 09:09:44 AM »
The model is based on a gradient boosted machine algo which is an ensemble of decision trees.  I haven't done a good job of explaining the model but will try to document it when I can find the time soon.  The model is probably 6 months old and due for an update. Probably get around to pushing a new model before beginning of new year.  Let me know if you have any specific questions...

SL



gstrubinsky

  • Newbie
  • *
  • Posts: 6
    • View Profile
    • Email
Re: Model score
« Reply #2 on: January 15, 2016, 09:05:33 AM »
Looking at the parameters in the VM I see only 2 options on the 'preset filters': 'Socially Responsible' and 'Balanced ROI' and above those two option I read 'under construction'.

Where does the algorithms kick in and how can I see what parameters are after the feedback loop adapts. (Assuming that's what it does). The system buys only 0 or 1 notes. Never more. In 2 weeks I got around 25 notes @ 25 USD bought. With this speed it will take a long time to invest the 6.5k I deposited. If the algorithm selecrt well, I would not mind. I just wished there was some transparency about if I set a parameter wrong, or there are no more good options.

If there was a description about how the program decides, I would love to understand it (I have some statistical knowledge). But if it is 'the secret sauce' which should stay that way, I would understand.

Re understanding: 'Maximum notes per order' I understand that an order is the # of notes for the chosen lending application is, not per session / run. Correct? Then why do I set the investment amount per note.
What does 'filter max percent' mean?

Any insight would be appreciates

gŁnter

sociallender

  • Sr. Member
  • ****
  • Posts: 273
    • View Profile
    • Email
Re: Model score
« Reply #3 on: January 15, 2016, 10:51:36 AM »
The "Social Lender" and "Balanced ROI" are just templates that you can use to set the note selection parameters.  If you click one of these it resets your filter and assigns the predefined filters for that selection.  You can see your parameters at the bottom of the bottom of the summary page.  For example, here are the parameters set for "Social Lender":

model >= 85 & inqLast6Mths <= 2 & installmentIncomeRatio <= 15 & delinq2Yrs <= 0 & pubRec <= 0 & intRate >= 14 & grade %in% c("B", "C", "D", "E")

Now, if you go to these parameters on the left you will see that they are set and you can change them for your particular note selection criteria.  Again, this is just a quick way of setting the filter if you have absolutely no idea of what parameters you want.  I recommend adjusting to suit your needs.

Since this filter has "model >= 85" as part of the selection filter, it will select only notes that have a model score above or equal to 85.  PLS assigns a probability score to each note that predicts the likelihood of the note to be fully paid.  The higher the score, the more likely the model believes the note will be paid.  Of course you dont have to use the model in your note selection criteria.  It is there to help but not required.

When I have time I will be creating a document that details the model  and the process of how it was built.  Its a gradient boosted model which uses an ensemble of decision trees. 

If you want more notes to be purchased, you will need to loosen your filter selection.  I am not sure what your filter looks like but it seems to be selective and perhaps a bit too tight.  Play with your parameters and review the summary page to see historically what percentage of notes match your criteria. 

Maximum notes per order is the max number of notes that can be placed per order.  It is a safety measure to ensure that if your filter is not set correct or too loose you don't buy too many notes.

The max percent is another saftey measure (probably a bit too paranoid) that will cancel the order if it is above the max percent of total notes.  So, if you have it set to 50% and there are 100 total notes that can be ordered, and your filter wants to order 50 or more, the order will be cancelled.  I tried to code in safety measures especially for people who don't understand how to use the software.  I have heard of other tools completely buying all available notes because of a bad filter and wanted to avoid this potential problem.

SL














gstrubinsky

  • Newbie
  • *
  • Posts: 6
    • View Profile
    • Email
Re: Model score
« Reply #4 on: January 15, 2016, 11:41:44 PM »
Thank you. So you get to the probability/model value by walking through a decision tree. The tree is binary (y/n) or weighted (last years requests = requests/12*learnt weight)? I am sure it's really cool what you did and I don't intend to steal your idea, just try to understand the entrails of the black box. Will the model be updated and sent tothe client (the weights and tree)? Since the software on our end is not capable to dig through the historical data trying to value each datapoint's weight and value in realtionship to the other datapoints. I guess you collect them over time to auto-adjust occasionally. The loading is way to short to load historical data on the client/our VM.

I have one more problem understanding: What is defined as order?
( logging in-reading new data-evaluating-order by filter not more than 'filter max percent and not more than 'maximum notes per order' for this session)
OR
is it to be understood as ( logging in-reading new data-evaluating-order by filter not more than 'filter max percent notes and not more than 'maximum notes per order' hit)

Or in other words for each login/logout there will not be more notes than  'maximum notes per order' bought
or for each hit (fitting id/memberid) there will not be more than 'maximum notes per order' bought and if that is the case what does then 'investment amount per note' mean? Are they not the same? one is a $ value the other a multiple of $-junks per note and therefore also a $ value?

I am new here obviously and try to get the best out of the software that obviously has an enormous amount of work invested. 

I appreciate your patience
This is by far not as boring as I thought it would be
gŁnter

sociallender

  • Sr. Member
  • ****
  • Posts: 273
    • View Profile
    • Email
Re: Model score
« Reply #5 on: January 21, 2016, 06:12:49 PM »
The model is actually an ensemble/combination of many decision trees (google gradient boosted models).  I will be updating the model hopefully soon when I can find the time.  I will create a blog post that will provide more details on how the model is implemented.

The order:

Login
API gets loans
Filter loans using your filter
If # filtered notes > max percent, cancel order
If # filtered notes > max notes per order, only order notes within max notes per order sorted by field (default intRate)

The amount invested per note is what is set in user settings.

Hope this helps. 

Sorry for late response, been traveling...

gstrubinsky

  • Newbie
  • *
  • Posts: 6
    • View Profile
    • Email
Re: Model score
« Reply #6 on: January 29, 2016, 11:57:16 PM »
I still have some understanding problems:

Can I change the distribution of each category?

Like 5% 'a', 10% 'b' , ... since this is obviously important for risk minimization. I don't want the system to buy only 'e' and 'f' since they are momentarily the only ones fitting ,my filter. I don't see those settings. Clicking the pie chart does not get me there either.

Now to the filters:

Is there an editor (vim, emacs, etc) for me to activate somehow to edit the filters since the custom text field is narrow and I can't balance the braces through indentation. copy/paste works only in the VM and not the host -  as expected.
Now, are those text custom filters attached to the selected custom filters or would I have to copy the original filter definitions into the custom field and edit it there?
Lastly, I probably forgot a brace, but my changes don't seem, to be taken over. I want to loosen my filters if '(purpose="car")' because - unless you live in one of the relatively few cities with working public transportation- I don't seem to get the car buyers which I do when I take that part out( =  it's the exact opposite result). I tried (purpose %in% c("car")) hoping that a one element list might do: Nada! Call me a 'sourpuss', but I do not want to invest into people who take a 3 year or even 6 year loan for a vacation. So would that be (purpose != 'vacation') or (purpose <> 'vacation') or (purpose # 'vacation')? 

I appreciate your effort helping us to tame your tiger.

guenter