ArticlesApplied ML

Your Text to SQL Model Didn’t Fail. Your Evaluation Metric Did.

End-to-end accuracy conflates two specification failures with different fixes. Most Text to SQL evaluation harnesses score correct queries as wrong because nobody wrote down what correct means.

Jordan Browne-Moore October 2026 16 min read

Contents · 9 sections
  1. The six decisions inside “most recent quarter profits”
  2. Where the system is wrong, and where the instrument is
  3. The report with two axes
  4. The comparator is a specification nobody wrote
  5. The policy page, and showing the reading
  6. The audit
  7. What a validator sends back
  8. If you are training
  9. Where this stops

“Show me the most recent quarter profits.”

The model reads that as net income for the last completed fiscal quarter, one row per obligor, in each obligor’s own reporting currency, with no comparative. It writes correct SQL for that reading and returns 48 rows, with the figure aliased net_income. The gold query reads the same sentence as profit attributable to the parent for the last completed calendar quarter, aggregated across the obligor group, converted to a common currency, with a prior year comparative. It returns 12 rows.

Strict result match scores this zero, and the SQL has no error in it. The model and the annotator read the sentence differently, and neither reading was ever written down.

You finetuned a Qwen model on your own financial data to replace a frontier API for Text2SQL. It scores four points below the API and nobody can say why. If the API’s advantage sits in interpretation, you are paying a per token premium for a prompt you could write down once as policy. If it sits in execution, you have a capability gap, and that is the case where better training data is the right answer. One accuracy figure cannot separate those two, because it adds together failures with different fixes, different costs and different owners. The harness would print that same zero if both sides had read the sentence identically and the comparator penalised the alias, or if the model had found a profit column in the wrong table.

The evidence below comes from public Text2SQL benchmarks and not from credit analytics, so read the magnitudes as reasons to test your own system. The audit near the end is a protocol I have not run.

The six decisions inside “most recent quarter profits”

A reasonable analyst, handed that sentence and a database, makes six decisions before writing any SQL, and the sentence contains no information about any of them.

Measure definition. Net income, operating profit, EBITDA, or profit attributable to the parent. In a credit file those sit at different distances from cash, and a covenant written against one is not satisfied by another.

Period convention. The last completed fiscal quarter, the last completed calendar quarter, the trailing three months to the latest reporting date, or the most recent quarter this obligor has filed. Across a portfolio on mixed fiscal calendars those four produce four different row sets.

Entity scope. A single legal entity, a consolidated group, the borrowing entity named in the facility agreement, or every entity in the obligor group including the ones that never report separately.

Comparatives. One number, or the number against the same quarter last year, the prior quarter, or plan. The analyst almost certainly wants a comparison and did not say so.

Output form. One row per obligor, one row per obligor and period, or a single aggregate. The shape decides whether the next question can be answered without a second query.

Currency and units. Reporting currency or a common currency, which rate on which date, thousands or millions, and whether a figure filed in thousands was converted anywhere upstream.

None of the six is in the question, and each gets resolved somewhere by something. Inside a finetuned model the resolution happens in weights, with no record of the choice. Inside a frontier API driven by a long prompt it happens in whatever the prompt says, which a person can at least read. That is the whole of the asymmetry, and it is not really about which model you bought, because a finetune can be handed the same long prompt. The variable that matters is whether the policy exists somewhere readable.

What makes the six difficult is not technical. Each is a policy decision with no technically correct answer, and credit and finance settle them differently without anyone noticing until two reports sit side by side. A gold query is one annotator’s resolution of all six. MCS-SQL’s manual failure analysis finds that matching the gold query’s tables and columns defeats human experts too, because several columns can share a meaning and the question often does not say which belong in the SELECT clause (MCS-SQL). A specification gap, recorded as a model failure.

Where the system is wrong, and where the instrument is

Your system has three places to fail. It resolves the question into an intent, which can disagree with the annotator’s. It translates that intent into tables and columns, which is schema linking. It executes. Two more failure sites sit outside the system, in the instrument you measure it with, since the gold set holds one person’s readings against a schema version that has probably moved, and the comparator is a function somebody wrote that decides when two result sets count as the same thing.

The layers leak into each other at linking. Column naming drives it, which makes it a specification problem, and retrieval over a wide schema is a learned capability, which makes it a modelling problem. Once metric artefacts are excluded, linking is the most common genuine error in MCS-SQL’s analysis, about a fifth of failures on both BIRD and Spider. The same manual analysis classed most of the recorded failures as correct answers scored wrong by the metric: 62 percent on BIRD and 73 percent on Spider. So the report below carries two axes and not three, with linking counted inside execution and named there. A policy document, a data dictionary and a training run have separate budgets, and the dashboard number cannot tell you which one to fund.

The report with two axes

The reporting fix is cheap and almost nobody does it. Report two numbers, and run the baseline in the same column layout.

illustrative mock. invented values, the shapes and labels are the point. run eval-2026-10-18 questions 200 annotators 2, independent policy interp-policy-v3 empty reference results 31, excluded below finetune frontier API INTERPRETATION, resolved reading graded against the written policy every dimension in play agrees 0.58 0.67 (of 200) any reading the policy permits 0.66 0.74 measure definition 0.88 0.90 (182 in play) period convention 0.63 0.81 (171) entity scope 0.74 0.76 (150) comparatives 0.69 0.72 (134) output form 0.91 0.92 (200) currency and units 0.95 0.95 (96) EXECUTION, SQL correct given the reading the model resolved strict match, content and column order 0.84 0.86 (of 169) containment, gold core columns present 0.91 0.93 schema linking share of strict failures 11/27 9/24 JOINT, policy reading and strict match 0.53 0.58 (of 169) AGGREGATE against the gold query 0.62 0.66 (of 200)

Each dimension is scored only on the questions where it is live, which is why the denominators differ and why the top line sits below every row beneath it. A reading has to agree on all six to count, so the all dimensions figure can only be lower than the worst of its parts. The last two lines are the ones to look at twice. Joint is graded against the policy and aggregate against the gold query, so they come out differently, and equal figures would mean the gold queries encode the policy perfectly, which is the opposite of the situation the policy exists to fix.

The period convention row is where the money is. The finetune resolves it to policy 63 percent of the time and the API 81 percent, against a two point gap on execution. The API has not learned anything about your fiscal calendars. Its system prompt says what they are. A figure that low also tells you what training will not fix, because a gold set encoding no policy gives the model nothing to converge on, and the seven points between strict and containment matching say you owe a written equivalence rule before any remaining error can be attributed to the weights.

The split also changes the procurement decision, since a frontier API and a model you host with the same aggregate score are not the same product. Section VII of the April 2026 model risk guidance asks a buyer to understand a purchased model’s conceptual soundness, design, development data and performance, and for a closed frontier model the first three go only as far as the vendor chooses to publish. Performance is the one that remains, and the only way to obtain it is to measure it on your own data.

The comparator is a specification nobody wrote

Result equivalence looks like the objective part of the problem. It is whatever your comparison function says it is, and nobody wrote those rules down. Your harness has already decided each of these.

  • Row order. Do the right rows in a different order match?
  • Column names. Does net_income match profit?
  • Column order and count. Do three correct columns plus two extra match a two column reference?
  • Float precision. Does 1842.9971 match 1842.9970, and at what tolerance?
  • Duplicates. Does a result with a row duplicated by a join that fans out match a deduplicated reference?
  • Empty against null. Does a zero row result match a one row result containing a null, and does either match a zero?
  • Units and rounding. Does a figure in thousands match the same figure in units?

The published size of the choice is substantial. On Spider, about a fifth of queries separate execution accuracy from set match, and the comparable share on BIRD is around a sixth (Fundamental Challenges in Evaluating Text2SQL). The benchmarks do not agree on the rule either. BIRD requires content and column order to match exactly, while Spider 2.0-Snow asks only that the gold answer’s core columns be present, which cuts false negatives and opens a surface where extra columns satisfy containment without matching intent (Spider 2.0). The comparator has the same disease as the model, which is an unwritten policy.

Rank the seven by what they cost, not by how often they fire, because frequency is what the harness surfaces. Units and rounding first, since a figure off by a factor of 1000 in a credit memo is a wrong decision presented with full confidence that passes every syntactic check in the pipeline. Duplicates second, because a join that fans out inflates an exposure total silently and the total still looks like a total. Empty against null third, treated below. Float precision fourth, because it makes noise in the harness and not wrong decisions in the file. Column names and row order last, because they are nearly always harness defects wearing a model defect’s clothes. That ordering is close to the reverse of how these failures appear in an error log.

Units has a twin on the other axis. Currency conflation is not a comparator decision at all, it is the sixth interpretation dimension, and it produces the identical failure at the identical cost.

The policy page, and showing the reading

The six dimensions get resolved either in a document a validator can read or in weights nobody can. Nothing that follows needs a model.

Write the defaults down, one page, per dimension. Profit means net income attributable to the parent unless the question says otherwise. Most recent quarter means the last completed fiscal quarter for that obligor, so obligors on different calendars appear with different period ends in the same table, and that is intended. Scope defaults to the borrowing entity named in the facility agreement. Every question carries a prior year comparative unless a single figure is asked for. Output is one row per obligor and period, in reporting currency, with a units column. Each default is arguable and each takes an afternoon to settle, and at the end of that afternoon a model risk function can challenge your system item by item.

Notice what the page does to the worked example at the top. Under that policy the model’s fiscal quarter reading is the right one and the gold query’s calendar reading is wrong, so the zero was the instrument’s error.

Show the resolved reading next to the answer. Before the query runs, in business language, with an edit control on it.

reading net income | fiscal Q2 2026 | borrowing entity | no comparative | GBP thousands [change]

A system that prints that line above its 48 rows has turned a silent failure into a correctable one, and it leaves a log of resolutions you can audit against the policy afterwards.

Grade intent against the policy, allowing a set of acceptable readings. That makes the grader a document instead of an annotator’s afternoon. Where the policy admits more than one reading, score any member of the acceptable set as correct and record which one the system picked.

I used to argue that a model has nothing to learn from a genuinely ambiguous question, since the input underdetermines the right reading. That assumes the target is a single reading. Recent work makes the target a set. Saparina and Lapata train with two rewards, recall on ambiguous inputs to cover the valid readings and precision on unambiguous ones to suppress invented alternatives, on supervision that is only several valid answers per question (Saparina and Lapata). A set of acceptable readings can be fully determined by the input even when no single reading is, and AMBROSIA already publishes that annotation, labelling multiple SQL queries per question across scope ambiguity, attachment ambiguity and vagueness (AMBROSIA).

What survives of my original claim is an argument about sequencing. You should not train on interpretation yet, because you have no gold set with multiple readings, no measured ambiguity rate on your own corpus, and no calibration metric that would catch the model growing more confident about an underdetermined guess. The audit below delivers the first two along the way. Where you can afford the extra turn, asking beats resolving.

The audit

This protocol is mine. It is specified tightly enough to run, and I have not run it.

Take 150 to 300 real questions from the engagement you are already in. Real means asked by an analyst in the course of work, in the words they used, including the vague ones. A synthetic question set measures your imagination.

Split the set in half and cross the conditions. Pair A annotates the first half against the schema and the existing instruction set, with no interpretation policy, then the second half after reading the one page policy. Pair B takes the halves in the opposite order. Four experts, and the policy effect no longer rides on the same people seeing the same questions twice, which the obvious design gets wrong, because a second pass by the same pair after reading a policy measures practice and recall as much as specification.

Classify every disagreement into the six dimensions, plus a seventh bucket for a plain mistake by one annotator. The taxonomy and the dimensions have to be the same list, or the audit cannot tell you which row of the policy page to argue about. Three measurements come out of it.

Expert disagreement without the policy. One minus that rate is roughly the best score to expect against a single annotator’s gold, since your own experts do no better against each other.

Expert disagreement with the policy. The drop between conditions is the measured value of the specification work, and the crossing pairs questions across conditions, so you can put an interval on the difference itself.

Model against policy, for the API and for the finetune, per dimension. The measurement the audit exists for. Grade each model’s resolved reading against the policy, dimension by dimension, and the two columns tell you whether the four points live in interpretation or in execution.

Write the thresholds down before anyone annotates. Mine would be a drop of at least 10 points in the disagreement rate with the 95 percent interval on the paired difference clear of zero, and a gap of at least 5 points between the two models on model against policy agreement, on the same test. For a rate near 30 percent that interval is about plus or minus 6.4 points at 200 questions and 12.7 at 50, which is why the range starts at 150. A 50 question pilot is worth running to find out whether the protocol survives your change control regime, and its rate is not quotable as a finding.

Report the taxonomy next to the rate. A 25 percent disagreement rate that is entirely period convention is a different project from a 25 percent rate spread evenly across all six dimensions. The first one is an afternoon. The second one is a quarter.

Two to three days of expert time each for a set in this range, plus a day of engineering for the comparison and the classification sheet. That is the cheap part and it is not the constraint. The constraint is that running expert written SQL against real data inside a bank is a change control event. You need a sandbox with no write access, a decision about which environment holds the data, and a data egress conversation with someone who will ask what leaves the building and in what form. Budget weeks of calendar time for that instead of days of effort, and start it before the expert time is scheduled.

What the gate decides

The likely result is mixed, which is what the per dimension columns are for. Expect the policy to close period convention almost completely, because a fiscal calendar is a fact somebody can look up, and to leave entity scope roughly where it was, because what counts as the obligor is a credit judgement two experienced people can hold differently after reading the same page.

Where disagreement collapses once the definitions are written down, specification was the fix, and the gap between your finetune and the API on that dimension was a prompt carrying an unwritten policy. Where it survives, interpretation is a real modelling problem there, and training targets in the Saparina and Lapata shape become a live agenda instead of a reading list. The deliverable depends on which way the dimensions fall, so say which in the engagement letter. Mostly the first kind and you ship the policy document and the gold set built against it. Mostly the second and you ship the harness and the evidence file.

What a validator sends back

Five things come back from a model risk function or a sponsor bank’s reviewer, and all five are about documents. Gold set provenance, the equivalence rule written as prose with both variants reported, a monitoring threshold with an action attached, a named owner for each of the four artefacts where the names cannot all be one person, and the count of gold questions whose reference result is empty. The companion piece sets all of it out as the evidence set a reviewer works through.

If you are training

Run reinforcement learning against execution results and the gold set stops being evaluation data. It becomes the objective, and it is attackable.

The cleanest documented case is a framework bug and its fix. SkyRL’s Text2SQL reward paid 1.0 when the predicted and gold result sets were equal. Where the gold query returns no rows, every well formed query that also returns no rows collects full reward, including SELECT 1 WHERE 0 and any filter restrictive enough to match nothing. On nearly half the tasks in SkyRL-SQL-653, 290 of 653, a query that did nothing earned full reward. An equality check cannot see this, because two empty sets are equal, and under group relative policy optimisation those samples collect positive advantage for the wrong reason (SkyRL). FINER-SQL names the same pattern as spurious correctness under an empty execution result (FINER-SQL).

Empty reference results were 4.7 percent of Spider dev, 7.1 percent of BIRD training and 44 percent of SkyRL-SQL-653. That spread across three corpora of one task family is the reason to count your own. Arctic-Text2SQL-R1 responds with a minimal reward, paying 1.0 for a result matching ground truth, 0.1 for executable SQL that does not match, and 0 otherwise, on the argument that composite rewards invite reward hacking while a reward based only on execution does not (Arctic-Text2SQL-R1).

Two results bear on whether to start at all. LIMIT moved Qwen3-8B from 57.4 to 65.4 on BIRD-Dev with supervised finetuning on 796 examples, to 66.9 by adding reinforcement learning, and to 69.1 with self-consistency at 8 on top of the trained model (LIMIT). Sampling bought its two points on top of the trained model and not in place of it, so nothing there licenses skipping the training leg. What it does support is that 796 curated examples bought eight points, a reason to look hard at curation before volume.

The second result is specific to your base model. Shao et al. report random rewards lifting Qwen2.5-Math-7B on MATH-500 by 21.4 points against 29.1 points from ground truth rewards, while the same spurious rewards do nothing on Llama3 and OLMo2 (Spurious Rewards). On a Qwen base, a run that improves has not thereby shown that your reward works. Budget for a control run on a base model from another family, or for an ablation showing the gain tracks reward quality.

The sequence that follows is a written specification, then sampling at inference with voting grouped by execution result, then distillation on execution verified traces, and only then a reinforcement learning leg, gated on a tested gold set and a budget for the control run. Most teams reach the training decision without having tried the three cheaper steps.

Where this stops

For a supervised bank, the supervisor fixes some of these defaults. SR 26-2 supersedes and replaces SR 11-7 and SR 21-8, and states its relevance as greatest for banking organisations with over $30 billion in total assets (SR 26-2), so which defaults are yours depends on your client’s regime. Worth knowing before you build the policy page, a Text2SQL layer is the weaker of the two scope cases under that guidance and under the UK equivalent.

Count the empty reference results in your gold set this week, write the equivalence rule down in a paragraph, and report two numbers at the next review. If they put the gap in interpretation, the deliverable was never a finetune.