HomeSelected engagements

03 / 05Case study

Credit scoring system

Kuda Technologies · African neobank, 7M+ customers

A scoring model for millions of monthly lending decisions, built on a feature-stability method that became the team standard.

50%

reduction in first-payment defaults

Credit scoring · Kuda Technologies

−50%

first-payment defaults, −75% across the portfolio

Feature stability, by bootstrap. Schematic. Each line is one resample’s learning curve; a feature whose spread settles onto a low floor is safe to ship. The method behind a LightGBM model trained on 300K borrowers.

01

The problem.

Needed to pre-qualify millions of customers monthly for lending products. High default rates were eroding portfolio performance.

“At Kuda, when enough features drifted into instability, we watched first payment default rates move from 21% to 27% within months.”

“The screened population size didn’t change appreciably, which made it worse: the model was still approving at roughly the same rate while the quality of those approvals was quietly degrading.”

A feature that passes screening

“Spend velocity”

“Spend velocity, measuring how quickly money left an account after being deposited, would pass IV screening cleanly but flip its relationship with default depending on the mix of salaried versus gig workers in the sample.”

From the write‑up: The Problem

02

What I built.

Designed and owned the credit scoring function: an optimised LightGBM model built from a 300K borrower sample, processing millions of monthly pre qualification decisions. Developed the feature stability methodology (bootstrap learning curves) that became the team's standard for all subsequent model builds.

The write-up demonstrates the method on public Lending Club loan data, 2007 to 2018.

02.1

Bootstrap learning curves

“It varies pool size from small subsets up to the full dataset, draws multiple bootstrap resamples at each size, and computes distributional metrics (primarily Wasserstein distance and KS statistic) and target-dependent metrics (Spearman correlation, IV) against the full-sample reference. Then it fits k / √n + floor to the resulting curve.”

“floor measures the irreducible instability that remains even at large n. That’s structural instability, and more data will not help.”

From the write‑up: How It Works

k / √n + floor fitted to every feature

02.2

Two Kinds of Stability. Inversely Correlated.

“Marginal stability and SHAP stability answer different questions and are designed to diverge.”

“The Spearman rank correlation between the two axes is ρ = -0.556. Not just uncorrelated. Inversely correlated.”

From the write‑up: How It Works · Two Kinds of Stability. Inversely Correlated.

−0.556 Spearman ρ, marginal vs SHAP stability

02.3

Two Features. Two Different Decisions.

“Direction consistency of 55.8%. The model flips the sign of this feature’s effect in nearly half of bootstrap samples.”

“One conventional pipeline would have flagged dti (a reliable workhorse) and trusted delinq_2yrs (a directionally unstable feature masked by censoring). The stability analysis reverses both decisions.”

From the write‑up

55.8% direction consistency of delinq_2yrs

02.4

Early Evidence: Structural Floors Predict Temporal Degradation

“Features flagged as structurally unstable in the development period degraded 15x more on average than stable features in the holdout.”

“This is a directional signal, not a predictive guarantee.”

From the write‑up

15× more holdout degradation, unstable vs stable features

03

Architecture.

Schematic

  1. 01 Borrower sample 300K borrowers
  2. 02 Feature stability screen bootstrap learning curves
  3. 03 LightGBM scoring model (model step) optimised · SHAP
  4. 04 Monthly pre-qualification millions of decisions
  5. 05 Lending portfolio production scoring engine

Stability audit

  • Marginal stability
  • SHAP stability

Feeds back into step 02

Schematic. Component names from the engagement summary and the feature-stability write-up.

04

Results.

  • 50%

    reduction in first-payment defaults

  • 75%

    reduction in total portfolio defaults

  • 300K

    borrower sample behind the model

  • Millions

    of pre-qualification decisions a month

  • Model serves as the production scoring engine for the entire lending portfolio.

05

Lessons.

“The problem is that standard screening evaluates marginal properties, not structural resilience.”

From the write‑up: The Problem

“If you only check distributional stability in a feature set with meaningful collinearity, you will tend to trust features the model uses erratically and flag features it relies on.”

From the write‑up: Two Kinds of Stability. Inversely Correlated.

“Bootstrap stability analysis isolates structural fragility from a single snapshot, making it viable for new product launches and thin-history portfolios where temporal holdout data doesn’t exist.”

From the write‑up: How It Works