Data Governance and AI – 8 minute read
Today, large language models can turn a question asked in natural language into a working SQL query within a few seconds. The syntax of the queries is now rarely wrong. The real problem is whether the model is answering the right question.
In our context graph post this week, we explained that AI agents need context more than data. In this post, we focus on metadata, the most concrete part of context. We examine a study published by Atlan in December 2025 and the recent measurements that support it, and look at what they mean for organizations.
This is not just a technical detail. According to a prediction Gartner published in early 2025, by the end of 2026 organizations will abandon 60% of AI projects that are not supported by AI-ready data. Gartner also separately emphasizes that having sufficient metadata is essential to be able to show that data is AI-ready (Gartner).
In brief
Right SQL, wrong question
The report’s opening example sums up the problem very well. The model is asked, using Formula 1 data, “Which drivers were eliminated in the first round?” When no context is given, the model interprets the word “eliminated” as a field left empty and answers with a query that looks for null values. In this dataset, however, being eliminated means finishing at the bottom of the ranking. The query runs without errors and returns a result, but that result is completely wrong.
Although the example comes from the world of racing, the same situation arises in every organization. At an insurance company, the term “active policy” may mean a policy in force for one team and a policy whose premium has been paid for another. The “type” field in the customer table and the “type” field in the policy table can be confused with each other. The tables do not say which amounts a metric such as loss ratio is calculated with. Similarly, the report lists four types of errors: ambiguous business terms, column names that resemble each other, missing information about relationships between tables, and unwritten calculation rules.
What did the study measure and what did it find?
The Atlan team set up the experiment quite simply. GPT-5 wrote SQL for 174 questions on a dataset of 13 tables. Each question was run under two different conditions: Once with only the table and column names, and once with enriched metadata consisting of a business glossary, a data model and column descriptions added. The queries were run on Snowflake and then scored by Claude 4 Sonnet. Because the experiment was repeated three times, a total of 522 comparisons resulted.
In most of the results, the two methods received the same score. In the comparisons where there was a difference, however, metadata clearly came out ahead: Enriched metadata performed better in 22.2% of the comparisons, and schema information only in 16.1%.
The report presents this difference as a “38% improvement”. To read the figure correctly, it is worth keeping two points in mind. First, the average quality score rises from 0.53 to 0.56, and the report describes this as a small but consistent effect. Second, the result points in the same direction in all three of the three repetitions. In other words, metadata does not work miracles on every question, but it reliably makes a difference.
What is actually more interesting is which questions the difference appears in. Simple questions are often answered correctly with schema information alone. Very complex questions, on the other hand, require step-by-step examples, going beyond metadata. The effect of metadata is seen most in medium-difficulty questions that relate multiple tables and contain business terms. The improvement in this group is 2.15 times. According to the report, the questions most frequently encountered in organizations fall into exactly this group.
Other measurements point in the same direction
It would not be right to make a decision based on a single study. That is why we also looked at other recently published measurements and found that the most striking one came from the insurance sector.
In a study published in April 2026, dbt Labs used an insurance dataset called “ACME Insurance” consisting of 15 tables (dbt Labs). The team had 11 business questions answered with two methods, 20 times each. In the first method, the model wrote SQL directly against the tables. In the second method, a semantic layer was used in which metric definitions and table relationships had been modeled in advance.
The result is quite clear. Claude Sonnet 4.6’s accuracy was 90.0% when it wrote SQL directly and 98.2% with the semantic layer. For GPT-5.3-Codex, these rates were measured as 84.1% and 100%. These figures apply to the questions covered by the semantic layer. dbt Labs also notes that writing SQL directly still works for questions the layer does not cover, and recommends using the two approaches together.
There is one more detail in the same study. As models have improved, the overall success of direct SQL rose from 32.7% to 64.5% between 2023 and 2026. So the models really are getting better. However, for questions where the margin of error is unacceptable, such as money, risk or management reporting, the difference is still determined by how well the definitions and relationships are modeled.
Atlan and dbt Labs used different datasets, different models and different methods. Even so, both reach the same conclusion: The quality of the model matters, but what really determines the right answer is the context given to the model.
Not more, but the right one
In our view, the most instructive finding of the Atlan study comes from the first version of the experiment. In the first attempt, the team gave the model every possible piece of information and prepared a comprehensive metadata of 176 lines. They then curated this text and cut it down to 64 lines. The short version produced results 13.8% better than the long version and ran at 52% lower cost.
So it is not the amount of metadata that makes the difference, but its form. The report draws four principles from this experience.
When we translate these principles into corporate language, the result is as follows: A good business glossary does not just define a term. It also explains in which table and according to which rule that term is calculated, and which concept it should not be confused with.
A three-layer foundation
The report groups AI-ready metadata into three layers: the business glossary that links business terms to columns, the data model that shows the relationships between tables, and column-level metadata containing the description of each field. The natural home of these layers is a data catalog. However, the report gives an important warning: The raw information in the catalog is too long to be given to the model as is. The real value comes from bringing this information into a short, example-based form.
Nexa’s place in this picture
The three layers the studies point to, namely the business glossary, the data model and column-level metadata, are core topics of data governance. At Nexa, we build this foundation end to end with organizations. We design the governance framework, select the right platform without being tied to a specific product, and put metadata management, the business glossary, data lineage and personal data classification into practice. When needed, we also take over the operation of the platform after implementation.
With the spread of AI agents, the importance of this foundation grows even further. Our approach is to carry the experience accumulated in data governance over to AI agents as well: defining which data agents can access for which business purpose, and enforcing these rules at runtime as well.
Is your metadata AI-ready? Let’s assess your existing catalog and business glossary together with our experts, and determine together where you should start for your AI projects. [Schedule a meeting →]
Sources
