Audit a Text-to-SQL harness before you retrain
- 150 to 300 real questions, asked by analysts in the course of work
- Two annotators and one written policy
- Treat the questions where the annotators disagree as the specification your harness is missing
- Build test pairs from those disagreements
Your Text to SQL Model Didn't Fail. Your Evaluation Metric Did. Field guide PDF, one page
