What they found
The authors built LuthorSQL, a 7B model that turns plain questions into database queries (SQL) in a single forward pass instead of writing text word by word. It scores slots in a fixed grammar of query plans, runs up to 32 candidate plans, and picks one with a trained validator. Alone, it reaches 68.03% execution accuracy on BIRD's hidden test set at 0.84 s per question on a single GPU. It keeps its accuracy in 9 GB with 8-bit quantized weights, and it transfers zero-shot to Snowflake, BigQuery, MySQL, PostgreSQL, and DuckDB within 3.1 points of its SQLite accuracy. Paired with a Claude Opus 5.5 agent in a pre-registered confidence cascade over 227 questions on databases published after Opus's training cutoff, it answers nearly half of the questions locally and cuts API cost by 33.3% at the same accuracy. The authors also report that Opus reproduces 48-92% of public benchmark gold queries verbatim without seeing the schema, yet its tool-assisted accuracy holds on renamed schemas and reaches 95.6-97.4% on post-cutoff databases.
Why it matters
Organizations that want to query their data with SI agents face slow, costly calls to hosted models, and their sensitive queries leave the building. This work shows a way to keep many of those queries on hardware they already own. The contamination results also matter for how we read benchmark scores, since a model may have seen the answers. The abstract reports the cost cut on only 227 questions, so how well it holds elsewhere is not stated.
How it changes the race
Fast small models that decide when to call a big agent could lower the cost and delay of deploying frontier SI on private data. This is a practical engineering gain in cost and privacy, not a jump in raw capability.
Safety and control
Keeping many queries local reduces exposure of sensitive organizational data to external APIs. The contamination probing is also a useful check that benchmark scores reflect real skill and not memorized answers.
The abstract, by the authors
Frontier LLM agents that query databases with tools are among the most accurate text-to-SQL systems, but per-query calls to a hosted model are slow, costly, and expose sensitive organizational queries to external APIs. For example, a Claude Opus 5.5 agent serving 10,000 queries a day costs over $70,000 a year at list price. To what extent can local hardware already owned by an organization offload this burden without compromising the agent's accuracy? We present LuthorSQL, a non-autoregressive 7B text-to-SQL model that serves as a rapid System 1 gate for a slower LLM agent (System 2), escalating queries only when uncertain. LuthorSQL makes a single forward pass to predict slot scores over a bounded grammar for query plans, compiles and runs up to 32 candidate plans, and chooses an answer using a trained validator. On its own, LuthorSQL reaches 68.03% execution accuracy on BIRD's hidden test set at 0.84 s per question on a single GPU, maintains its accuracy in 9 GB with 8-bit quantized weights, and transfers zero-shot to Snowflake, BigQuery, MySQL, PostgreSQL, and DuckDB within 3.1 points of its native SQLite accuracy. Combined with Claude Opus 5.5 in a pre-registered confidence cascade over 227 questions on databases published after Opus's training cutoff, LuthorSQL answers nearly half of all questions locally and cuts API cost by 33.3% at the same accuracy. Finally, contamination probing shows that Opus reproduces 48-92% of public benchmark gold queries verbatim without seeing the schema; however, its tool-assisted accuracy holds on renamed schemas and reaches 95.6-97.4% on post-cutoff databases. Code and data: https://github.com/IAMIbrahimmemon/luthorsql
Quoted from arXiv:2611.17780. The research and its words belong to its authors.
