HomeSelected engagements

01 / 05Case study

LLM fine‑tuning for revenue estimation

CB Insights · Market intelligence platform

We finetuned Qwen 3 32B to replace an LLM API. The evaluation was harder than the training.

80%

cost reduction vs third-party LLM APIs

LLM fine-tuning · CB Insights

−80%

API cost vs third-party LLMs

Low-rank adaptation on a frozen 32B model. Schematic. Only the thin matrices B and A are trained; W stays frozen, which is what makes owning the model cheap. Measured on ~15,000 curated examples: precision in the upper 90s, recall in the lower 90s.

01

The problem.

Needed to estimate private company revenue from public signals at scale. The initial solution relied on expensive third-party LLM API calls, making costs prohibitive at platform scale.

“At CB Insights, we needed to extract financial metrics from public signals about private companies.”

One sentence, several fields

“the company expects to cross $100M ARR by Q3”

“…contains a projected metric, a value range, and a temporal context that the pipeline needs to decompose into separate structured fields.”

“Cost scaled linearly with volume, and at the throughput we were processing, API spend had become the primary cost driver for the extraction pipeline.”

“Standard LLM evaluation treats output as text a human reads. For structured extraction, that entire approach is wrong.”

From the write‑up: The Task That Doesn’t Fit in a Benchmark · Why Off the Shelf Evaluation Fails Here

02

What I built.

End to end product build: designed the data pipeline, normalisation layer, fine tuning approach (Qwen 3 32B, LoRA), and production FastAPI inference API. Defined the evaluation methodology (format validity, near match accuracy, and ground truth proximity), establishing how the team measures model quality for all subsequent LLM work.

02.1

Format Adherence: Why We Went 32B

“The 14B model could extract the right information but couldn’t reliably produce it in the complex JSON structure the pipeline required: missing fields, incorrect nesting, type errors.”

“Moving to Qwen 3 32B solved this. Format adherence reached 99.9 to 100%.”

“But healing masks problems. Format adherence at the model level prevents them, and it’s cheaper.”

From the write‑up

99.9–100% format adherence, Qwen 3 32B

02.2

The Matching Problem

“We treated this as a bipartite matching problem.”

“We then ran Hungarian algorithm assignment to find the optimal one to one matching that maximized total similarity, with a minimum similarity threshold below which a pair was rejected as unmatched.”

“Precision landed in the upper 90s. Recall in the lower 90s.”

From the write‑up

Upper 90s precision; recall in the lower 90s

02.3

Interextraction Differences: The Diagnostic That Changed the Story

“For every article, we compared the finetuned model’s extractions against the reference model’s extractions element by element.”

“Without interextraction difference tracking, we’d have seen comparable aggregate metrics and concluded the models were equivalent. The element level comparison showed the finetuned model was actually better, not just cheaper.”

From the write‑up

~15,000 hand-curated training examples

02.4

The Cost Decision

“The cost structure shifted from linear per extraction API pricing to fixed GPU infrastructure. A finetuned Qwen 3 32B served via vLLM on a RunPod instance costs the same whether it processes 10,000 or 100,000 articles.”

“When quality is comparable or better and cost drops by 80%, you don’t need a complex ROI model.”

From the write‑up

~80% lower amortised cost per extraction

03

Architecture.

Schematic

  1. 01 Public signals about private companies
  2. 02 Ingestion pipeline
  3. 03 Normalisation layer
  4. 04 Qwen 3 32B (model step) LoRA fine-tune · ~15,000 curated examples
  5. 05 FastAPI inference API served via vLLM
  6. 06 Revenue estimates company · metric · min/max · date

Evaluation methodology

  • Format validity
  • Near-match accuracy
  • Ground-truth proximity

Feeds back into step 04

Schematic. Component names from the engagement summary and the write-up.

04

Results.

  • 80%

    reduction in API costs vs third-party LLMs

  • 99.9–100%

    format adherence with Qwen 3 32B

  • Upper 90s

    precision, with recall in the lower 90s

  • ~15,000

    curated training examples

  • Production inference API serving revenue estimates across the platform.

05

Lessons.

“Define evaluation before training. We built the evaluation harness in parallel with finetuning, which meant we couldn’t use it to guide early training decisions.”

From the write‑up: What I’d Do Differently

“The model’s extraction quality degraded on longer articles in ways our evaluation didn’t catch, because our evaluation set didn’t contain articles of that length.”

From the write‑up: What Caught Us Off Guard: Article Length in Production

“Start with 32B for complex schemas. The 14B experiment confirmed what the schema complexity should have predicted.”

From the write‑up: What I’d Do Differently