03 / 05Case study
Credit scoring system
Kuda Technologies · African neobank, 7M+ customers
A scoring model for millions of monthly lending decisions, built on a feature-stability method that became the team standard.
50%
reduction in first-payment defaults
Credit scoring · Kuda Technologies
−50%
first-payment defaults, −75% across the portfolio
01
The problem.
Needed to pre-qualify millions of customers monthly for lending products. High default rates were eroding portfolio performance.
“At Kuda, when enough features drifted into instability, we watched first payment default rates move from 21% to 27% within months.”
“The screened population size didn’t change appreciably, which made it worse: the model was still approving at roughly the same rate while the quality of those approvals was quietly degrading.”
A feature that passes screening
“Spend velocity”
From the write‑up: The Problem
02
What I built.
Designed and owned the credit scoring function: an optimised LightGBM model built from a 300K borrower sample, processing millions of monthly pre qualification decisions. Developed the feature stability methodology (bootstrap learning curves) that became the team's standard for all subsequent model builds.
The write-up demonstrates the method on public Lending Club loan data, 2007 to 2018.
02.1
Bootstrap learning curves
“It varies pool size from small subsets up to the full dataset, draws multiple bootstrap resamples at each size, and computes distributional metrics (primarily Wasserstein distance and KS statistic) and target-dependent metrics (Spearman correlation, IV) against the full-sample reference. Then it fits k / √n + floor to the resulting curve.”
“floor measures the irreducible instability that remains even at large n. That’s structural instability, and more data will not help.”
From the write‑up: How It Works
k / √n + floor fitted to every feature
02.2
Two Kinds of Stability. Inversely Correlated.
“Marginal stability and SHAP stability answer different questions and are designed to diverge.”
“The Spearman rank correlation between the two axes is ρ = -0.556. Not just uncorrelated. Inversely correlated.”
From the write‑up: How It Works · Two Kinds of Stability. Inversely Correlated.
−0.556 Spearman ρ, marginal vs SHAP stability
02.3
Two Features. Two Different Decisions.
“Direction consistency of 55.8%. The model flips the sign of this feature’s effect in nearly half of bootstrap samples.”
“One conventional pipeline would have flagged dti (a reliable workhorse) and trusted delinq_2yrs (a directionally unstable feature masked by censoring). The stability analysis reverses both decisions.”
55.8% direction consistency of delinq_2yrs
02.4
Early Evidence: Structural Floors Predict Temporal Degradation
“Features flagged as structurally unstable in the development period degraded 15x more on average than stable features in the holdout.”
“This is a directional signal, not a predictive guarantee.”
15× more holdout degradation, unstable vs stable features
03
Architecture.
Schematic
- 01 Borrower sample 300K borrowers
- 02 Feature stability screen bootstrap learning curves
- 03 LightGBM scoring model (model step) optimised · SHAP
- 04 Monthly pre-qualification millions of decisions
- 05 Lending portfolio production scoring engine
Stability audit
- Marginal stability
- SHAP stability
Feeds back into step 02
04
Results.
-
50%
reduction in first-payment defaults
-
75%
reduction in total portfolio defaults
-
300K
borrower sample behind the model
-
Millions
of pre-qualification decisions a month
- Model serves as the production scoring engine for the entire lending portfolio.
05
Lessons.
“The problem is that standard screening evaluates marginal properties, not structural resilience.”
“If you only check distributional stability in a feature set with meaningful collinearity, you will tend to trust features the model uses erratically and flag features it relies on.”
“Bootstrap stability analysis isolates structural fragility from a single snapshot, making it viable for new product launches and thin-history portfolios where temporal holdout data doesn’t exist.”
06
Stack and sources.
- LightGBM
- Python
- SciPy
- SHAP
- SQL
- AWS
- Bootstrap framework