01 / 05Case study
LLM fine‑tuning for revenue estimation
CB Insights · Market intelligence platform
We finetuned Qwen 3 32B to replace an LLM API. The evaluation was harder than the training.
80%
cost reduction vs third-party LLM APIs
LLM fine-tuning · CB Insights
−80%
API cost vs third-party LLMs
01
The problem.
Needed to estimate private company revenue from public signals at scale. The initial solution relied on expensive third-party LLM API calls, making costs prohibitive at platform scale.
“At CB Insights, we needed to extract financial metrics from public signals about private companies.”
One sentence, several fields
“the company expects to cross $100M ARR by Q3”
“Cost scaled linearly with volume, and at the throughput we were processing, API spend had become the primary cost driver for the extraction pipeline.”
“Standard LLM evaluation treats output as text a human reads. For structured extraction, that entire approach is wrong.”
From the write‑up: The Task That Doesn’t Fit in a Benchmark · Why Off the Shelf Evaluation Fails Here
02
What I built.
End to end product build: designed the data pipeline, normalisation layer, fine tuning approach (Qwen 3 32B, LoRA), and production FastAPI inference API. Defined the evaluation methodology (format validity, near match accuracy, and ground truth proximity), establishing how the team measures model quality for all subsequent LLM work.
02.1
Format Adherence: Why We Went 32B
“The 14B model could extract the right information but couldn’t reliably produce it in the complex JSON structure the pipeline required: missing fields, incorrect nesting, type errors.”
“Moving to Qwen 3 32B solved this. Format adherence reached 99.9 to 100%.”
“But healing masks problems. Format adherence at the model level prevents them, and it’s cheaper.”
99.9–100% format adherence, Qwen 3 32B
02.2
The Matching Problem
“We treated this as a bipartite matching problem.”
“We then ran Hungarian algorithm assignment to find the optimal one to one matching that maximized total similarity, with a minimum similarity threshold below which a pair was rejected as unmatched.”
“Precision landed in the upper 90s. Recall in the lower 90s.”
Upper 90s precision; recall in the lower 90s
02.3
Interextraction Differences: The Diagnostic That Changed the Story
“For every article, we compared the finetuned model’s extractions against the reference model’s extractions element by element.”
“Without interextraction difference tracking, we’d have seen comparable aggregate metrics and concluded the models were equivalent. The element level comparison showed the finetuned model was actually better, not just cheaper.”
~15,000 hand-curated training examples
02.4
The Cost Decision
“The cost structure shifted from linear per extraction API pricing to fixed GPU infrastructure. A finetuned Qwen 3 32B served via vLLM on a RunPod instance costs the same whether it processes 10,000 or 100,000 articles.”
“When quality is comparable or better and cost drops by 80%, you don’t need a complex ROI model.”
~80% lower amortised cost per extraction
03
Architecture.
Schematic
- 01 Public signals about private companies
- 02 Ingestion pipeline
- 03 Normalisation layer
- 04 Qwen 3 32B (model step) LoRA fine-tune · ~15,000 curated examples
- 05 FastAPI inference API served via vLLM
- 06 Revenue estimates company · metric · min/max · date
Evaluation methodology
- Format validity
- Near-match accuracy
- Ground-truth proximity
Feeds back into step 04
04
Results.
-
80%
reduction in API costs vs third-party LLMs
-
99.9–100%
format adherence with Qwen 3 32B
-
Upper 90s
precision, with recall in the lower 90s
-
~15,000
curated training examples
- Production inference API serving revenue estimates across the platform.
05
Lessons.
“Define evaluation before training. We built the evaluation harness in parallel with finetuning, which meant we couldn’t use it to guide early training decisions.”
“The model’s extraction quality degraded on longer articles in ways our evaluation didn’t catch, because our evaluation set didn’t contain articles of that length.”
“Start with 32B for complex schemas. The 14B experiment confirmed what the schema complexity should have predicted.”
06
Stack and sources.
- Qwen 3 32B
- LoRA
- FastAPI
- Python
- HuggingFace Transformers