Author Topic: LC database sloppiness  (Read 11453 times)

Fred

  • Hero Member
  • *****
  • Posts: 1421
    • View Profile
Re: LC database sloppiness
« Reply #15 on: April 26, 2014, 06:26:10 PM »
Some of them are just nonsensical sloppiness, such as giving the same field, containing the same data different names in different files, or providing the very same information in different files, but with different formats or conventions.  Then there are some things that just some kind of wrong.

I am not defending LC in any ways on this; however, even Bloomberg often produces incomplete, inconsistent, or even incorrect data.  This is from a company that charges a ton of money for its data!

I think the reasons are the same -- multiple teams working independently from each other.  There are different backend databases and different teams that update them.  I wouldn't be surprised if there are nightly batch jobs that simply copy data from one database to the others.

For display/downloadable data, there are yet different teams (depending on the pages or links) that handle precisions, formats, labels, etc.

Thatguybil

  • Newbie
  • *
  • Posts: 39
    • View Profile
Re: LC database sloppiness
« Reply #16 on: April 27, 2014, 09:31:08 PM »
Some of them are just nonsensical sloppiness, such as giving the same field, containing the same data different names in different files, or providing the very same information in different files, but with different formats or conventions.  Then there are some things that just some kind of wrong.

I am not defending LC in any ways on this; however, even Bloomberg often produces incomplete, inconsistent, or even incorrect data.  This is from a company that charges a ton of money for its data!

I think the reasons are the same -- multiple teams working independently from each other.  There are different backend databases and different teams that update them.  I wouldn't be surprised if there are nightly batch jobs that simply copy data from one database to the others.

For display/downloadable data, there are yet different teams (depending on the pages or links) that handle precisions, formats, labels, etc.

Heck I am amazed at the data integrity errors that can creep into databases when going from a real time database to a historical index database.

The back end guys never like it when you ask why you get different results when you run the same query on databases that are intended to be synced up.

P2PFact

  • Jr. Member
  • **
  • Posts: 91
    • View Profile
Re: LC database sloppiness
« Reply #17 on: May 07, 2014, 10:41:27 PM »
I also see 20k records in LC data that are almost empty. You can see loan id, member id, loan amnt, address. But that's about it. No loan_status, no term, etc.

I downloaded data 4/18. Maybe just bad luck? Anyone has similar issue?

Fred93

  • Hero Member
  • *****
  • Posts: 2166
    • View Profile
Re: LC database sloppiness
« Reply #18 on: May 07, 2014, 10:46:06 PM »
I also see 20k records in LC data that are almost empty. You can see loan id, member id, loan amnt, address. But that's about it. No loan_status, no term, etc.

I downloaded data 4/18. Maybe just bad luck? Anyone has similar issue?

I suspect you're referring to the "policy code = 2" loans.  Check out the policy code field.  I believe this is intentional.  These are, I believe, a class of loans which they are not yet offering to the public.  If you google you can find some blogs which mention them.  Just another detail not explained in the official documentation.

P2PFact

  • Jr. Member
  • **
  • Posts: 91
    • View Profile
Re: LC database sloppiness
« Reply #19 on: May 07, 2014, 11:29:20 PM »
I suspect you're referring to the "policy code = 2" loans.  Check out the policy code field.  I believe this is intentional.  These are, I believe, a class of loans which they are not yet offering to the public.  If you google you can find some blogs which mention them.  Just another detail not explained in the official documentation.

Yeah you are exactly right. All are policy 2 loans. Here are the stats:
loan count    total loan amount     avg loan amnt    state
41    $442,475     $10,792    AK
280    $2,528,325     $9,030    AL
138    $1,124,675     $8,150    AR
512    $4,332,925     $8,463    AZ
3335    $29,471,150     $8,837    CA
405    $3,893,325     $9,613    CO
280    $2,385,575     $8,520    CT
45    $351,675     $7,815    DC
52    $414,300     $7,967    DE
1376    $11,434,600     $8,310    FL
619    $5,776,200     $9,332    GA
141    $1,252,200     $8,881    HI
1    $6,000     $6,000    ID
651    $5,644,900     $8,671    IL
352    $2,807,550     $7,976    IN
163    $1,391,575     $8,537    KS
172    $1,506,600     $8,759    KY
175    $1,585,225     $9,058    LA
335    $3,808,525     $11,369    MA
495    $4,355,875     $8,800    MD
565    $4,669,925     $8,265    MI
365    $2,999,700     $8,218    MN
273    $2,407,625     $8,819    MO
53    $452,275     $8,533    MT
500    $4,505,325     $9,011    NC
89    $890,050     $10,001    NH
711    $6,507,675     $9,153    NJ
116    $1,160,650     $10,006    NM
378    $3,297,850     $8,724    NV
1634    $14,618,200     $8,946    NY
629    $5,355,275     $8,514    OH
146    $1,318,100     $9,028    OK
252    $2,135,175     $8,473    OR
578    $4,898,425     $8,475    PA
106    $898,175     $8,473    RI
184    $1,638,575     $8,905    SC
39    $316,975     $8,128    SD
281    $2,389,775     $8,505    TN
1108    $10,533,925     $9,507    TX
136    $1,254,525     $9,224    UT
555    $5,057,925     $9,113    VA
27    $198,925     $7,368    VT
437    $3,854,400     $8,820    WA
223    $1,938,350     $8,692    WI
71    $583,000     $8,211    WV
37    $320,800     $8,670    WY

Fred93

  • Hero Member
  • *****
  • Posts: 2166
    • View Profile
Re: LC database sloppiness
« Reply #20 on: May 08, 2014, 02:09:24 AM »
Some background on policy_code=2 loans can be found in this writeup...

http://www.lendacademy.com/policy-code-2-loans-lending-club/