How to evaluate a text-to-SQL agent beyond accuracy

To evaluate a text-to-SQL agent, score more than execution accuracy. Also count how often its queries compare against values the column does not store, name tables or columns that do not exist, act on values that changed after the agent last looked, and answer questions it should refuse. All four can be scored mechanically. In our benchmark runs the largest effects appeared when the data changed, a case static benchmarks never test.

Part 4 of 4 in the series "Trustworthy agentic analytics", which starts with part 1. Previous: Making an analytics agent's work inspectable.

TL;DR

• Report five things per condition: execution accuracy, value hallucination, schema hallucination, stale answers after the data changes, and a reliability score that penalises wrong answers.
• On static BIRD questions, current models rarely invented filter values (1.0–2.2% of questions with plain introspection). A check of the final query against stored values took that to zero for all three hosted models, with accuracy within a 3-point margin.
• After stored values changed, agents reading a snapshot of the catalog gave stale answers on 28–55% of the changed questions. The live catalog cut that by 41–89%, most for the strongest model.
• Grounding costs context (1.25–2.6× the tokens per question), and optional lookups cost the frontier model 2.4 points of BIRD accuracy.
• The study was pre-registered and run once per model on the RushDB v3.0.1 release. The write-up is an unpublished draft.

Why isn't execution accuracy enough?

Execution accuracy scores whether the result matches, so it cannot tell you how the agent failed. Execution accuracy (EX) is the share of questions where the agent's query returns the same result set as the reference query. It is the standard headline number, and it hides three things.

First, an error, an empty result from filtering on 'Prague' instead of 'Hl.m. Praha', and a wrong join all score zero. Only the second is fixable by grounding.

Second, gold queries are imperfect: one CIDR 2026 study reports a 52.8% annotation-error rate on BIRD Mini-Dev (Jin et al.).

Third, benchmark databases do not change. An agent that read the data last week and filters on a value renamed since then fails in production, and no static benchmark contains that question. On hard benchmarks accuracy is also low: an o1-preview code agent solved 21.3% of Spider 2.0's 632 enterprise workflow problems (Lei et al.), so how the rest fails is most of what you learn.

What metrics should you use to evaluate a text-to-SQL agent?

Use the five in the TL;DR. All of them are scored by code, without an LLM judge.

The value and stale metrics do not depend on the gold answer being right, so the annotation errors above do not leak into them. Before any model ran, the pre-execution check used in one condition and the scorer judged all 4,683 distinct queries we compared identically.

RS(c) comes from TrustSQL (Lee et al.): +1 for a correct answer or a correct abstention, 0 for abstaining on a feasible question, −c for a wrong answer or for answering an infeasible one. We use c = 10. A system breaks even only when fewer than one answer in eleven is wrong. On our EHRSQL split, a system that refuses everything scores +0.53, above every condition of every model. Use RS(10) to rank conditions against each other; it is too harsh to serve as a deploy threshold.

How did we set up the runs?

We ran one tool-calling agent on four models over three datasets, with every condition, leg, metric, hypothesis and test pre-registered before the first row.

Agent and models. One agent loop, at most 12 turns, temperature 0, seed 0, reasoning off, one run per model:

• Qwen3.8-27B (primary) and Qwen3.5-9B, open-weight models from the Qwen family (Qwen3 report), served in bf16 through OpenRouter, pinned to one provider with fallback disabled;
• a frontier model at low effort, through its vendor's API;
• qwen3-14b (Q4_K_M, Ollama 0.40.7, 32K context) on a local GPU, which ran the drift and EHRSQL legs before the open models moved to hosted serving. Its BIRD leg stopped after 11 rows and is not analysed.

The panel changed once, after an interim look at qwen3-14b's results. The change was registered before any of the new rows, with the rule that no condition, metric or hypothesis changes and qwen3-14b's rows are reported in full. No claim compares models with each other.

Datasets.

• BIRD Mini-Dev: 500 questions on 11 SQLite databases (mini_dev), run without the evidence hints; 498 distinct questions because two repeat.
• TrustSQL EHRSQL: 200 test questions, stratified feasible and infeasible (seed 0), on the MIMIC-IV demo database shipped with TrustSQL. EHRSQL's questions come from 222 hospital staff and include questions the data cannot answer (Lee et al.).
• Drift set (built by us from BIRD Mini-Dev and EHRSQL): 281 questions on 11 databases. 221 receive one change after a snapshot: 131 renames (a compared value recoded to upper or snake case; the gold is rewritten and must return the original result) and 90 inserts (the rows holding a compared value are written after the snapshot). 60 controls compare values no change touches. Changed values are held by at most 2,000 rows. The live tenant receives the changes through RushDB's write path; the snapshot tenant is never written after load.

What changes between conditions. Every condition gets the same agent, prompt skeleton, foreign-key list, run_sql executor on SQLite, and terminal submit_sql and abstain tools. Only the grounding tools differ.

One episode, end to end:

RushDB v3.0.1, the public release binary, served the catalog in G2, G3V and G5A, with each database in its own tenant. Before any model ran, an audit compared every catalog read the tools use with SQL on the source: 18,889 reads, one known difference (a property name read back case-folded). Every changed drift value was checked on both tenants (724 checks, 0 differences).

Statistics. Every comparison is paired by question. Binary outcomes use exact McNemar tests; RS(10) uses a sign test and a bootstrap confidence interval; EX uses a paired confidence interval against a 3-point non-inferiority margin. Holm correction runs within each model and leg.

What happened on BIRD Mini-Dev?

With the checked submit, no hosted model's final query compared an unstored value on any of the 498 questions, and accuracy stayed within the 3-point margin of plain introspection. Setup: BIRD Mini-Dev, no hints. Each cell is EX / share of questions with an invented value.

Optional lookups (G3V) more than halved invented values on the two Qwen models and cut them by two thirds on the frontier model, but from 1.0–2.2% to 0.4–1.0% of questions: only 5 to 10 questions differ, too few for the pre-registered test to hold. Planner statistics changed little. On the frontier model, G3V scored 2.4 points below G1 (CI −5.1 to +0.3) and failed non-inferiority; G5A matched G1 exactly.

What happened on EHRSQL, where some questions have no answer?

The checked submit raised the reliability score over plain introspection by about two points on all three Qwen models (G5A − G1: +2.08 for qwen3-14b, +2.03 for Qwen3.5-9B, +1.87 for Qwen3.8-27B; Holm p ≤ 1.4×10⁻⁴), mostly by abstaining where introspection answered wrongly. Setup: 200 questions from TrustSQL's EHRSQL split.

Over G3V, the check added a significant gain only for qwen3-14b (+1.64), the model that invents most. The frontier model left nothing to remove: it abstained on about half the questions in every condition, invented no value in any, and scored above zero without grounding. Always abstaining (+0.53) still beats every condition of every model, so the gains reduce a penalty rather than earn a positive score.

What happened when the data changed?

After the data changed, the snapshot catalog (G2) was the worst condition on every model, and the live catalog removed most of the stale answers it caused. Share of the 221 changed questions with a stale answer:

Against the snapshot, G3V cut stale answers by 52%, 41%, 58% and 89% (in table order) and G5A by 68%, 37%, 53% and 97%; all eight contrasts hold (Holm p ≤ 6.2×10⁻⁵). The largest reduction is on the strongest model. Given the snapshot, the frontier model was as stale as the weakest model and used a renamed value's old spelling on 64% of the renamed questions; with the live catalog and the check, on none. It was stale under G2 and not under G5A on 119 questions, and the reverse on one.

Plain introspection also reads live data, so it is the harder comparison. Against introspection with planner statistics (G1P), the catalog was less stale on qwen3-14b and the frontier model (Holm p ≤ 0.011) and not on the two hosted Qwen models, whose live conditions already gave stale answers on fewer than 2.5% of the renamed questions. On the 60 control questions, we could not show that the live catalog costs nothing: no confidence interval fit within the 3-point margin, and on Qwen3.8-27B G3V scored 8.3 points below the snapshot.

Does the database engine matter, or the information?

The information removes the errors, and a database that maintains it on write makes it cheap and current. In our exploratory runs before this study, a control that answered the catalog tools with exact SQL on the source database at call time (COUNT(DISTINCT c), MIN/MAX, GROUP BY ... LIMIT k, LIKE) matched the catalog on the agent metrics. Any database can serve the same answers.

What a catalog maintained on write adds is exactness without recomputation and a read cost that grows slowly. A churn test ran three seeds of 20,000 randomized operations (creates, merges, type changes, nulls, deletes, links, committed and rolled-back transactions) and compared the catalog with a full recount at 60 checkpoints: 0 differences. We then replayed every catalog read Qwen3.8-27B made on three BIRD databases at their benchmark size and at 10× the rows, against exact SQL over the same SQLite file (warm pass, total seconds):

Every catalog output equalled the SQL reference (1,056 of 1,056 calls). The pre-registered "flat" (at most 1.5×) held only for formula_1. Value search over long text is the exception: the dictionary does not hold long strings, so the search scans them and grew 20× on european_football_2.

What happened when the query was checked before it ran?

The check left value hallucination at or near zero for every model on every leg: 0 of 498 BIRD questions and 0 of 200 EHRSQL questions for the three hosted models, 1 of 200 on EHRSQL for qwen3-14b, and 0.0–0.7% on drift, where the snapshot condition reached 31%. The pre-registered test against G3V held nowhere, because G3V was already low on static data and a residue cannot go below zero.

The comparison that shows what the check does is with the model left to itself. On drift, qwen3-14b compared an unstored value on 29.2% of questions with plain introspection, 9.6% with optional catalog lookups and 0.4% with the check. Optional lookups depend on the model deciding to look; the check does not.

G5A is a condition in our benchmark harness. It is not a RushDB feature today; RushDB is building checked submission into the product. Part 2 shows how to build the same check yourself.

When did grounding lower accuracy?

Grounding lowered accuracy in three places. On BIRD, optional lookups cost the frontier model 2.4 points. On EHRSQL, Qwen3.5-9B answered 2.5 points fewer questions correctly with the check, and the frontier model's RS(10) fell from +0.23 to −0.10 with optional lookups.

The exploratory runs showed a larger cost. With BIRD's evidence hints in the prompt, which spell out most literals, an earlier check on names and literals alone scored 27.1% against 33.5% for plain introspection (qwen3-14b, p = 0.007 after Holm). In the traces, the checked submit replaced running the query: plain introspection executed before submitting on 70% of hinted questions, the check on 18%. The final G5A therefore also returns queries that fail or return nothing. The hinted setting was not part of the final study.

What does grounding cost in tokens?

Grounding adds context, so it costs tokens. With the check, tokens per question were 1.25–2.6× those of plain introspection across models and legs: on BIRD, Qwen3.8-27B went from 10.1K to 12.6K and the frontier model from 3.0K to 5.2K; on EHRSQL, qwen3-14b went from 3.4K to 8.4K. Do not read these results as a token or cost saving.

What did we not test?

As of November 2026, the study has these limits:

• One engine, two benchmarks, twelve databases. Other engines could maintain the same information. Drift is generated (renames and inserts of compared values), not observed in production.
• One model family for the open models, plus one frontier model, each run once at temperature 0. Hosted inference is not guaranteed bit-stable.
• 60 control questions are too few to show that the live catalog costs nothing on unchanged data.
• Read cost on three databases at 10× only. Filtered catalog reads are not measured.
• No run gave the benchmark's hints in the final study.

How can you reproduce or adapt this evaluation?

The harness, both pre-registrations, the raw per-question rows with full traces, the analysis outputs and the drift builder are not public yet; until they are, these pins are what you need to rebuild the setup.

The pins are the RushDB v3.0.1 release binary (rushdb-x86_64-unknown-linux-gnu, sha256 821b3fe0…), qwen/qwen3.8-27b snapshot 20260814 and qwen/qwen3.5-9b in bf16 from one provider, the frontier model at low effort through its vendor's API, and qwen3:14b Q4_K_M on Ollama 0.40.7. The hosted runs cost $9.19 (Qwen3.8-27B), $12.53 (Qwen3.5-9B) and $44.70 (the frontier model) in API fees; the local model costs nothing to rerun. Run both conditions of a paired comparison back to back.

To adapt it to your own agent, pick questions that filter on string values plus some the data cannot answer, add a drift leg by renaming or inserting a few low-frequency values after the agent's last look (and rewriting gold to match), report RS(10) next to the always-abstain baseline, and run with and without the context your production prompt carries.

Where does RushDB fit?

RushDB was the catalog provider. It is a database for AI that maintains field support, stored types and value dictionaries on every commit (schema catalog docs). Agents reach it through getSchemaMarkdown and propertyValues in the MCP server. Its contribution is keeping those answers exact and current under writes without recomputing them per call.

FAQ

How do you measure value hallucination without an LLM judge?

Parse the predicted SQL, find each string literal compared with =, IN, != or <>, resolve its column, and check whether the column stores that value. Validate the scorer before the runs. Ours agreed with the pre-execution check on all 4,683 distinct queries compared, and the catalog it reads matched SQL on 18,889 reads.

Why include open-weight models?

The deployment we target is exploring confidential data with a model that can run next to it. Open-weight models also make the static-data errors visible: the frontier model rarely invented values on BIRD or EHRSQL, so on those legs there was little for grounding to remove.

How many questions do you need for a meaningful comparison?

Enough discordant pairs. A paired McNemar test only counts questions where the two conditions disagree. On BIRD, 498 questions left only 5 to 10 discordant ones for optional lookups, too few to show an effect, while 221 changed drift questions gave the frontier model 119 to 1.

Sources

• BIRD: Can LLM Already Serve as a Database Interface? (Li et al., NeurIPS 2023)
• BIRD Mini-Dev dataset
• TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring (Lee et al.)
• EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records (Lee et al., NeurIPS 2022)
• Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (Lei et al., ICLR 2025)
• Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors (Jin et al., CIDR 2026)
• Qwen3 Technical Report
• PostgreSQL documentation: pg_stats
• RushDB docs: schema catalog
• RushDB docs: MCP tools
• Benchmark numbers: our benchmark runs, November 2026 (draft, unpublished)