An LLM will invent a table that doesn’t exist and hand you perfect SQL for it. The database is the wrong place to find that out.
Text-to-SQL benchmarks grade pass/fail: run it, compare rows. That hides the failure that actually hurts — confident, well-formed SQL against a table that was never in the schema. You find out at execution time, on a real warehouse, and the usual fix is to catch the error and retry the model.
Hallucination in SQL is a schema problem — and a schema is checkable before you execute anything.
So generated SQL is scored across seven weighted dimensions instead of one bit: correctness 35%, safety 20%, efficiency 15%, completeness and semantic accuracy 10% each, best practices and plan quality 5% each. Safety outranks speed on purpose.
Queries are parsed to an AST with sqlglot before execution. Every table, column and function reference is resolved against the live schema; anything that doesn’t resolve is reported with a severity score, not a stack trace.
Structural, not textual — so it survives transpilation across SQLite, DuckDB, PostgreSQL and BigQuery. Failures that do execute get classified by cause (schema, analysis, SQL) using the taxonomy from published work on SQL generation failures, which turns a score into a diagnosis.
It runs as a green (evaluator) agent over the AgentBeats A2A protocol, handing tasks to purple competitor agents and emitting one reproducible artifact per tournament. Docker compose, MIT licensed.