The Hidden Architecture of Enterprise Data Agents

This blog recaps the technical keynote, session 3A, revAlation Chicago, The Old Post Office.¹

I have been an engineer at Alation for around four years, having spent the last two focused on integrations. That means bringing Alation directly to where teams already communicate, like Slack, Teams, plugins, and MCP clients.² Along the way, I have helped many enterprise teams set up data agents across gaming, online betting, environmental services, and several other industries.

We almost always see the same pattern unfold. Getting that very first answer feels easy. Producing an accurate answer you can confidently defend in front of an auditor every single time, even as underlying data constantly changes, is an entirely different story. Gartner predicted in February 2025 that, through 2026, organizations would abandon 60% of AI projects not supported by AI-ready data.”³

Closing that exact gap is what brought me to revAlation Chicago 2026.

Getting the first answer feels like magic

To kick off the demo, I skipped the Alation UI entirely and opened Slack.

I typed a single question: Which 2024 race gave Max Verstappen the biggest undercut advantage? While the agent processed the request, I broke down the query for the audience. Seconds later, Slack pinged back with a spot-on answer and an automatically generated chart.

From the end user's perspective, the experience is effortless. I didn't need to know where the underlying data lived, how the tables were structured, or how data products are configured. I just asked a question in the tool I already use every day. 

Earlier in the keynote, my colleagues argued that the best governance is invisible: a seamless layer operating behind the scenes. I completely agree, but with one key caveat: invisible infrastructure isn't absent infrastructure. Everything the user didn't have to worry about in Slack had to be intentionally defined and governed beforehand.

For this demo, I intentionally used a public Formula 1 dataset from Kaggle rather than a pristine demo environment. Real enterprise data is chaotic, but F1 racing provides a shared domain where people can instantly spot if an answer feels off. If the underlying structure is wrong, you can't hide it behind a polished canned demo.

So what was the thing I didn't have to know about?

Since you are reading this writeup instead of chatting in Slack, let us take a quick peek behind the curtain:

The catalog holds every single piece of data. That includes governed objects, documents, business intelligence reports, key metrics, ontologies, custom fields, and all the metadata sitting underneath. It serves as our central source of truth, and we build it to be complete on purpose.

That is precisely why we do not throw the whole catalog at an AI agent. Being comprehensive is very different from being ready for a specific job. If you ask a model to hunt for answers across the entire data ecosystem, you make finding the right information unnecessarily difficult.

A data product gives you the exact setup you need for a specific task. Think of it as a carefully chosen package pulled straight from the catalog for one clear goal. You get the key tables, the documents containing business rules, the metrics for calculations, and the ontology showing how everything connects. We package it, focus it, and hand it to the agent.

The setup behind my demo kept things straightforward. It contained a set of Formula 1 tables, written context on how to query them, approved glossary terms, and a few sample questions so the first person opening it would not be staring at a blank screen.

Every single choice I highlighted gets recorded somewhere inside that package. Here is how that looks in practice.

Anatomy of a ten-word question

Which 2024 race gave Max Verstappen the biggest undercut advantage? That simple question brings five clear chances to go off track, and none of them come down to a flaw in the model itself:

"2024" is not a standard calendar year. The Formula 1 season usually runs from March to November, but schedule shifts can push a race into the next calendar year. It works much like a fiscal year. To help the agent set the right scope, someone had to lay out what the 2024 season includes, and a separate process had to confirm the table includes every race. That second part relies on a data quality check rather than a simple prompt.

"race" must map to an actual table. The table might be named races, or perhaps race_1, or it might use an entirely different name. The words in the question and the technical names in the catalog are not identical, so something has to bridge that gap. That is where the data product comes in by linking everyday vocabulary directly to the underlying assets.

"Max Verstappen" must point to one specific driver. Real data often contains typos or name variations. Two distinct records can look similar enough that a soft match seems reasonable. However, a loose match will not trigger an error; it will quietly bring back the wrong driver, which causes bigger issues down the line.

"biggest" needs clear context. It is a relative word that carries no real meaning until you specify what metric you are measuring.

"undercut advantage" is the core metric, and it is a specific technical term. An undercut happens when a driver makes a pit stop ahead of a rival to gain position on track, and it only applies under precise conditions. We certainly do not want a language model guessing or making up a definition for this. A person needs to define it, add it to the glossary, and mark it as official. When two different definitions pop up, that official approval tells the agent which one to trust.

Five key phrases and five decisions, all handled by people before anyone typed a question. The person asking received the answer smoothly without needing to touch any of that setup work. That is the true value of this approach.

So who handles all of that?

That is the main question I hear everywhere, and it makes complete sense. When you are working at enterprise scale with thousands of tables, nobody has time to sit down and type out descriptions by hand.

That is why most of the draft work happens automatically. You can set up a simple curation rule, like saying every table in a dataset needs a basic description alongside the questions it answers. Test it on one table, tweak your prompt, and run it across everything else. To get that right, we look far beyond simple table names. We inspect where the data came from, track its lineage, analyze how people query it, and pull context directly from attached documentation. A huge portion of company knowledge lives in random scattered documents, so pointing at those sources lets us extract the core meaning right away.

Data quality monitors follow the exact same path. Some of my setup rules were as basic as making sure every race lists a valid date, every result field contains data, and pit stop duration stays a positive number. Keeping things simple works great. The important part is that rules created for past data keep holding true when fresh records arrive next season.

There is still a firm line here that I emphasized on stage: an agent cannot take ultimate responsibility. Automated tools draft the context, but a real person has to own it. A specific human name stays attached to the definition of undercut advantage, so if that definition goes wrong, a person steps in to fix it instead of blaming the model.

What happens when you miss a definition

During the Q&A segment, an audience member asked the live agent something I had not prepared for ahead of time: who gained the biggest advantage from pit stops in 2024? The agent answered smoothly. Before the answer showed up, I mentioned out loud that everything hinged on how you define a pit stop advantage, and afterward I noted that we needed a clearer definition. That highlights the real risk in this setup. A missing definition will not throw an error message. Instead, it generates a polished, sensible response that no one can easily verify, which ends up far trickier to catch than a broken query.

That is why having a clear feedback route is so important. Users can click a quick rating button or type plain feedback explaining that a specific calculation is wrong because their fiscal year starts in February. The system recognizes that user feedback as a learning opportunity. We also monitor background signals by comparing active chat threads against catalog usage. A nightly automated check highlights queries that left users without clear answers, flagging those gaps directly on the data product.

Crucially, the system does not attempt to rewrite its own underlying logic automatically while live. That is by design. Every identified gap routes straight to the product owner, who decides how to update the setup.

Simple for users, detailed for owners

The core interaction maintains two distinct views on purpose:

Someone chatting in Slack sees a quick question and a clean chart. Meanwhile, the data product owner sees full details behind the scene: the exact prompt, how the model reasoned through it, which tools and credentials were accessed, processing time, token costs, and the exact model involved. You can easily filter usage by surface, allowing you to distinguish casual Slack questions from custom connection scripts or web browser queries.

Access permissions remain active across every level. That was the primary governance item I brought up: will users only view data they are authorized to see? The AI agent directly inherits access permissions from the main catalog⁴ and syncs seamlessly with your central identity system. Background tooling stays safe because it respects your configured security rules rather than bypassing them.

Maintaining quality through change

The final piece relies on evaluation tests, which function like unit tests for language models. You assemble a collection of core business questions with known answers, run them through, and have an evaluation model grade the output.⁵ We avoid forcing strict code matches since different AI models reach accurate answers through varied SQL logic. Instead, we evaluate the final output alongside the underlying reasoning steps.

The real power comes from snapshot tracking. Every evaluation run anchors to the precise configuration of the agent and data product at that moment, including the model choice, system prompts, connected tools, and defined terms. If you adjust a single table description and notice accuracy shift, frozen version histories make it easy to pinpoint what changed.

I showed how this works during the live presentation. My accuracy score dropped from around ninety percent down to seventy percent, and someone asked why. 

The drop happened because I had temporarily switched to a faster, lighter model to keep the presentation moving quickly, and I had refreshed some test questions. Built-in tracking made that trade-off clear right away. Tracking down known changes keeps things manageable, whereas discovering drops through user complaints creates real problems.

How to take the first step

Avoid trying to tackle the whole platform at once. Pick one real question that your team asks frequently, the kind of query people drop into chat and then double-check manually. Break that question down phrase by phrase, mapping out every core assumption needed for the answer to be completely right. You will quickly find your core decisions right there.

Focus on making those core assumptions reliable, and let everyone else enjoy the smooth experience without needing to worry about the heavy lifting behind it.

Curious to see Alation in action? Book a demo today.

Sources & notes

Every external claim on this page is independently verifiable. The public sources are listed here.

1. revAlation Chicago was held at The Old Post Office, September 16–17. Alation ↗ https://www.alation.com/revalation/ (accessed 12 Oct 2026)

2. Agent Studio agents can be deployed into apps and workflows via MCP or REST APIs. Alation ↗ https://www.alation.com/product/agent-studio/ (accessed 12 Oct 2026)

3. Gartner predicts that, through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data. Gartner ↗ https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk

4. Agents in Agent Studio inherit the access controls, policies and permissions the organization already uses. Alation ↗ https://www.alation.com/product/agent-studio/ (accessed 12 Oct 2026)

5. Agent Studio supports built-in evaluations, Q&A sets and custom judges to validate agents before production. Alation ↗ https://www.alation.com/product/agent-studio/ (accessed 12 Oct 2026)

ANALYST ATTRIBUTIONS & DISCLAIMERS

Gartner, Lack of AI-Ready Data Puts AI Projects at Risk (Q&A with Roxane Edjlali), 26 February 2025.

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. GARTNER and Magic Quadrant are registered trademarks and service marks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.