Text-to-SQL systems are often evaluated on academic benchmarks where schemas are fully specified, metadata is curated, and the task reduces to generating SQL from a natural-language question.
Real production environments look very different. Before a query can even be generated, systems must identify relevant tables, columns, and metric definitions across catalogs, dashboards, BI tools, tickets, and informal documentation—while operating in data ecosystems where schemas and business definitions constantly evolve.
In practice, the source of truth for metrics is fragmented: the same definition may exist in warehouses, dashboards, documentation, or team knowledge. This makes it difficult to constrain agents to trusted tables and consistent definitions, so even systems with strong benchmark performance can produce queries based on outdated metrics or the wrong data sources.
Building reliable self-service analytics therefore requires more than good SQL generation. Systems need guardrails that prevent unreliable queries and mechanisms that continuously update those guardrails as the data ecosystem evolves.
In this talk, we argue that Text-to-SQL systems should be viewed not just as SQL generators, but as systems that gradually build and refine a semantic layer through their own usage. Each query, correction, and failure becomes a signal that improves metric definitions, table mappings, and analytical abstractions.
Drawing on real production workflows, we explore the pitfalls organizations encounter when moving from benchmark-driven development to deploying self-service analytics for business users. The talk provides practical guidance for Python data teams building Text-to-SQL assistants in messy, evolving environments, showing how guardrails, feedback loops, and evolving semantic layers make analytical results trustworthy.