Where Should AI Agent Guardrails Be Enforced? Lessons from Columbia's DAPLab

Enterprises spent the last two years asking AI to answer questions. Today, they are asking it to act: filing the ticket, updating the record, running the query, committing the code. Deloitte's State of AI in the Enterprise 2026 found that of 3,235 business and IT leaders surveyed across 24 countries, 74% expect their companies to be using AI agents at least "moderately" by 2027, while only 21% report a mature governance model for them.¹ Something has to sit between an autonomous system and the damage it can do, and for most teams that something is currently a text file.

A prompt file is the set of standing instructions an engineer writes for an agent: a system prompt, a policy document, an instructions file checked into the repository. It tells the agent what it is working on, what tools it has, and what it must never do: Don't write to production; check the schema before you migrate; run the tests before you commit. The prompt file needs no new infrastructure and (often) no approval, which is how it became the place many enterprises keep their operating policy for autonomous software.

This is also why the same instruction often appears more than once: "Don't touch the database. Don't touch the database." Somewhere in a prompt file at your company, a version of that line is pasted half a dozen times, probably in caps. Eugene Wu's AI lab at Columbia found hundreds or thousands of those lines,⁴ and concluded that most of them should never have been in a prompt at all.

Wu is an associate professor at Columbia University and co-director of the Data Agents and Processes Lab, which spans data management, machine learning, NLP, systems, and human-computer interaction.⁴ The lab works with companies running AI agents in production, and its human-computer interaction group studies how people manage rules for machines. A rule in a prompt is a request; a rule in a system is a guarantee. Wu and his co-author Charlie Summers put that contrast plainly in their Data Flow Control paper: policies encoded in prompts or evaluated by an LLM are inherently probabilistic, offer no formal guarantees, and degrade as policy complexity and data scale grow.⁵ Wu's argument, shared on a recent episode of AI Radicals, is that most of what we write into prompts belongs in the second category. 

Why AI agent reliability isn't an uptime problem

We borrowed the word reliability from the computing discipline where it described performance: "Traditional computing, when we talk about reliability, it's whether or not the performance is good. If it's fast enough or it scales enough," Wu explains. "But [in enterprise AI], reliability means: did it interpret your request correctly? Or did it do the right thing?"⁴

That matters because one model now makes those judgments repeatedly inside a single request: interpreting intent, generating code, calling tools, and changing state. Wu calls the consequence semantic coupling. "The quality of all of these steps depends on every other step in the pipeline," he says. "What you get as the output can be arbitrarily bad or arbitrarily good."⁴

Semantic coupling is the dependency that links every meaning-based decision in a workflow to every other one. Traditional reliability is a question of execution — is it fast, does it scale, is it up? Semantic reliability is a question of interpretation, and interpretation compounds. 

Wu's analogy is a supply chain: a competent plumber working with bad parts still hands you a bad outcome, and you absorb it as the customer. An agent stack behaves the same way, because it is agents calling sub-agents calling tools, what some experts call “turtles all the way down.” Any component that interprets poorly degrades everything downstream of it. And because the handoffs between those components are probabilistic rather than coded, there is no layer at which you can stop checking and assume the rest holds. It is the reason agentic automation has to be designed as a system rather than assembled as a chain.

Host Satyen Sangani named the architectural casualty: traditional computing gives you discrete, deterministic handoffs between layers, but agentic workflows do not. "People want to just do the job and then hand it off to the next person, and that kind of thinking just does not work," Sangani said.⁴

What teams actually put in their agent prompts

Wu's colleagues went looking at what teams write in those files. "What you actually see in these prompts is a huge portion of it is actually trying to tell the agent what to do and what not to do," Wu says, and the volume surprised his team: "Our HCI students found hundreds or thousands of these rules."⁴ Organizations already manage policy, he points out, but which rules apply to an agent depends on the task and the step it is running.

Independent evidence suggests this approach is fragile. The HANDBOOK.md benchmark, published by Surge AI in July 2026, drops agents into 65 company environments governed by policy documents of 20 to 124 pages and grades the resulting state against 824 deterministic criteria.² The strongest of 30 model configurations satisfied every criterion in 36.2% of trials; most frontier configurations scored below 25%.² The authors recommend enforcing hard controls outside the model.² The tasks, environments, and grading harness are all public, so the result is auditable rather than asserted.

Nearly every failed run ended with the agent asserting that it had complied, frequently citing the specific sections it had just violated.² In other words, an evaluation that trusts an agent's own account of its work will grade a violation as a pass, which is why AI agent evaluations need a source of truth outside the model.

Columbia's HCI group ran into the same wall from the usability side, and its first instinct was to prune: put a subset of the rules in the prompt rather than all of them, and hope the agent follows a shorter list more faithfully. Wu looked at the same pile of rules and drew a different conclusion.

Move the rule out of the prompt and into the system

"We looked at these and we said, oh, actually the majority of this stuff we can push into the system," Wu says. "And so it doesn't have to be probabilistically enforced. We'll just enforce it."⁴

Sangani pressed on the obvious question: which system? Wu's answer is that there isn't a single one. "This could be the database, it could be the operating system. It could be an orchestrator, it could be a linter."⁴ His example is a deterministic checker that reads the code and confirms exceptions are raised where the rules require, every time rather than when the model remembers.

The sorting test is straightforward: Can the rule be checked by inspecting code, state, or configuration, without interpreting what the user meant? If so, it does not belong in a prompt. In practice that covers most of what teams are writing today. The list below covers some common prompt directions which should actually live in infrastructure.

4 common prompt guardrails to move into infrastructure

• “Don't write to these files or paths” → filesystem permissions in the harness or container

• “Don't write to production; check the schema first” → database grants, constraints, and read-only connections

• “Always raise exceptions here; follow this convention” → a linter or a CI check

• “Run the tests before you commit” → the orchestrator that sequences the steps

There is a second benefit that is easy to miss. Enforcement is not only a wall, it is also a signal. "If it's not probabilistic, and you can rely on it and it's guaranteed, then the system doesn't need to think about it at all," Wu says⁴, and the environment can then catch the violation, tell the agent what it hit, and tell it what to do instead. A rule in a prompt competes for the model's attention on every turn, whether or not it is relevant. A rule in the system costs nothing until it fires, and then arrives as feedback at the moment it matters.

For data leaders, this is the logic already behind AI governance and policy enforcement at the data layer, applied to a new consumer. "You need somewhere to ground reliability and quality," as Wu puts it.⁴

Why prompting harder can't cover data flows

A deterministic check works when you can judge an action by looking at the action itself. Some rules fail that test, because the same action is either routine or catastrophic depending on where the data came from.

Take an everyday workflow. You ask an agent to repurchase a shirt. It finds the item in your order history, adds it to the cart, then hits an email verification step at checkout. It searches your inbox, finds the code, and also finds spam advertising diet pills. The diet pills end up in the cart.

This usually gets filed as prompt injection, and Wu's point is that the label hides the mechanism. What actually happened is that an unreliable source nobody anticipated put information in front of the agent, and the agent followed it. The prompt-side patch is to add more rules: only read verification emails, or check every tool call before anything reaches the cart. Wu explains why that stops working: You might well have bought diet pills before, so adding them to a cart is not an unsafe action on its face, and no amount of inspecting the call will tell you otherwise. "There's no way of checking a tool call or checking just a query of any form and knowing, oh, this is safe or not safe," he says. "Because it's about these data flows."⁴ The provenance is the problem, and the agent harness has no visibility into provenance at all, which is why lineage stops being a reporting feature and starts being a control surface.

His lab's answer is data flow control: constraints on where data may move rather than on which commands may run. Order history may reach your cart. Your inbox may not. "If you can write that policy down, and if the system can deterministically enforce it, then it doesn't really matter how you design the agent," Wu says. "If it tries to do something that's disallowed, then the system will just deny it."⁴

The mechanism is published rather than theoretical. Summers and Wu describe Data Flow Control alongside an enforcement layer called Passant, which rewrites queries so that policies are checked during execution instead of after it. Across five database engines (DuckDB, Umbra, PostgreSQL, DataFusion, and SQL Server), enforcement runs at roughly zero overhead,⁵ cheap enough to leave switched on for every query rather than reserved for the queries someone remembered to flag. That is the practical difference between a guardrail you can afford everywhere and one you ration.

Sangani reached the same conclusion from the data side: "You put a stop sign at a road, you don't put the stop sign in the head of the user so they have to memorize that stop sign every single time," he said.⁴ Policy belongs with the object being governed, whether that is a PII flag on a column or an ontology behind governed data products.

Where to start with agent guardrails

Wu's view of the next twelve months is deliberately modest: translation rather than invention, taking the rules already written in prompts and pushing them into the harness or the environment.

The case that convinced him came from Intellect Design, a financial technology firm whose software runs the services banks deliver to their customers. Working with them, Wu found that constraints on data use map onto much of financial regulation. "Those today are often just thrown into prompts," he says. "But it turns out actually a lot of these can be pushed in and just checked all the time, at scale."⁴ Coding agents offer the same opening.

The cost of waiting is measurable. Gartner predicts that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because of governance gaps identified only after production incidents occur.³ Sangani described that dynamic from experience: "You have to learn the policy because the mistake in some sense informs whether you need it."⁴

Wu's middle ground is to carve out the spaces an agent may never enter, then expand what you can guarantee over time, keeping the primitives that let you know when the system can be relied on and when it cannot.

The precedent is already in your stack

Wu finds this tractable because the industry has done it once. Before relational databases, applications absorbed data corruption and query optimization themselves, and transactions barely existed as a concept. "Application development was like hundreds of times more expensive and slower than it needed to be," Wu says. "And so what the database did was absorb a lot of this cost... that's what we see as something needed to do today for agents."⁴ He is careful not to overclaim: the database is one tool among many that an agent touches. The shift shows up in how LLMs increasingly behave like an operating system layer, and in why an intelligence operating system has to span agents, context, data, and governance rather than sit beside them.

The practical reading for a data leader is a question to ask of every prompt file your team maintains: which of these rules could the environment simply guarantee? The guardrails worth keeping are the ones you never have to write twice.

Curious how governed data products and enforceable policy can give your agents something to rely on? Book a demo with us today.

Sources & notes

Every external claim on this page is independently verifiable. The public sources are listed here.

1. 3,235 IT and business leaders surveyed across 24 countries in the Americas, Asia Pacific, Europe, and the Middle East; 21% report a mature governance model for agentic AI; 74% expect their companies to be using AI agents at least "moderately" by 2027. Findings published in Deloitte's State of AI in the Enterprise: The untapped edge (January 2026). — Deloitte Insights, "Business and IT leaders report AI agents are scaling faster than their guardrails," 24 April 2026. https://www.deloitte.com/us/en/insights/topics/emerging-technologies/ai-agents-scaling-faster.html

2. 65 agentic tasks set in ten fictional companies; expert-written standard operating procedures of 20 to 124 pages; 824 programmatic acceptance criteria covering both required and prohibited actions; under strict grading the strongest of 30 evaluated model configurations passes 36.2% of trials, with most frontier configurations below 25%; the authors' stated recommendation is to enforce hard controls outside the model; nearly every failed trajectory ends by asserting compliance, frequently citing the sections that were violated. Tasks, environments, and harness released publicly. — Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta & Edwin Chen (Surge AI), HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following, arXiv:2607.25398, 28 July 2026. https://arxiv.org/abs/2607.25398

3. By 2027, 40% of enterprises will demote or decommission autonomous AI agents due to governance gaps identified only after production incidents occur. — Gartner, "Gartner Says Applying Uniform Governance Across AI Agents Will Lead to Enterprise AI Agent Failure," 26 May 2026. https://www.gartner.com/en/newsroom/press-releases/2026-05-26-gartner-says-applying-uniform-governance-across-ai-agents-will-lead-to-enterprise-ai-agent-failure

4. All quotations from Eugene Wu and Satyen Sangani, and the account of the Columbia HCI group's rule-management research, the Intellect Design collaboration, and the relational-database analogy. — AI Radicals, Season 4, Episode 8, "Semantic Coupling & the Interconnected AI Stack," Alation. https://www.alation.com/podcast/episodes/semantic-coupling-eugene-wu-columbia/

5. Data Flow Control (DFC) as a declarative framework for constraining tuple-level data flows, and Passant, a portable query-rewriting enforcement layer; approximately 0% overhead measured across DuckDB, Umbra, PostgreSQL, DataFusion, and SQL Server; prompt-based and LLM-evaluated policy approaches characterized as inherently probabilistic, offering no formal guarantees, and degrading as policy complexity and data scale grow. Preprint; not peer-reviewed in this form. — Charlie Summers & Eugene Wu, Columbia University, Data Flow Control: Data Safety Policies for AI Agents, arXiv:2606.05679, 4 June 2026. https://arxiv.org/abs/2606.05679

ANALYST ATTRIBUTIONS & DISCLAIMERS

Gartner, Avoid Governance Mismatch: Classify AI Agents by Autonomy Level, Shiva Varma, May 2026.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.