Semantic Coupling & the Interconnected AI Stack

Satyen Sangani: Hey, Eugene. How are you doing? Welcome to AI Radicals.

Eugene Wu: Awesome. Thank you for the invitation.

What is Columbia's Data Agents and Processes Lab?

Eugene Wu, Associate Professor at Columbia University, explains why he and his colleagues built a lab that spans data management, machine learning, NLP, systems, and human-computer interaction rather than a single research area.

Sangani: So you are a professor at Columbia University and are doing some pretty unique and interesting work around agents, with an agent lab. Tell us about the lab and why you founded it and what the impetus was behind starting it.

Wu: At Columbia, we've been building over the past year a lab called the Data Agents and Processes Lab, or we call it the DAP Lab. What we're studying is: how do we build a computing stack and the technologies and the user interfaces for AI agent automation? And the interesting thing about AI agents, which are proliferating everywhere, is that the layers of the technology stack in the system are all semantically coupled.

That's a very new type of challenge in computing, because an LLM is being used everywhere in an application. So it might interpret a user's intent, but also generate code and formulate queries and analyze data and call tools and modify the environment. All of these things are happening as part of serving a user's automation request. And so the quality of all of these steps depends on every other step in the pipeline. What you get as the output can be arbitrarily bad or arbitrarily good. Optimization anywhere affects the correctness, and what the outcome is semantically. And that's something very, very difficult to deal with.

Why doesn't fixing one layer of the AI stack fix agent quality?

Wu's colleagues at Columbia come from different research traditions, and all of them reached the same conclusion about single-layer progress.

Wu: My colleagues and I at Columbia have been thinking about this problem. My background is in data management, but my colleagues might be in machine learning, or in NLP, natural language processing, or in user interfaces, or in core systems. And what we all found ourselves thinking about is: oh, if we make progress in just one piece of the technology stack, that isn't really good enough in order to address these quality issues and these correctness and safety issues that people are just seeing all over the place when they deploy and use AI agents.

And so we came together a little bit over a year ago to decide, hey, let's build a research lab where we're working across the stack. So we co-advise students across multiple different areas, and our real question is: what does this computing stack look like if you want to be able to deploy and use agents in a way where you don't have to think about the fact that it might corrupt your data, or it might throw away data? Or it might just get things wrong.

Which agent tasks can you trust enough to build on?

Columbia's lab treats the boundary of agent trustworthiness as an open research question, not a settled one.

Wu: And it's an open question right now, for what types of use cases and tasks can you completely trust it? Or trust it well enough that you can rely on it to serve as a platform for other users, or to rely on it to do mission-critical use cases.

And so for that reason, we put together this lab, and we think it's fairly unique because of the fact that it spans multiple different research areas that are traditionally siloed. So traditionally, a research lab might have a bunch of people working in the same area — like a database lab or an NLP lab or an AI lab. But this is fairly unique in both that we're working across these areas, and we're working with a lot of companies that also span the computing stack, in order to get these use cases that can motivate longer-term research that are systemic, that all of these companies seem to be facing.

What is semantic coupling in AI agent systems?

Sangani asks Wu to define the term at the center of the lab's work. Wu's answer explains why quality in agentic workflows compounds the way it does in a supply chain.

Sangani: You mentioned when you started this term semantic coupling. Tell us a little bit more about what that means, and perhaps even provide an example of what that means.

Wu: I think a good analogy is, when you hire a contractor in New York City, you don't quite know what you're getting. So maybe I hire a plumber. And they come and they fix my toilet or they fix my sink. But the thing is, if I don't know if they're any good, then I kind of have to do all this supervision. I have to learn about plumbing myself well enough to know that they're doing a good job, that what they're doing makes sense and that the outcome is good. And that's a huge amount of work, and I may as well just do it myself. So that's one case where I don't completely trust who's providing me the service, so I have to do a lot of work myself.

But even if the plumber might be pretty good, their supplier might not be very good, and is unreliable. And so the quality of their outcome — even if they're pretty good with the tools that they have, if they're using bad tools or they're using bad parts, then that ultimately affects me as the end consumer of the entire service.

And so quality typically is something where everything needs to be good for the end outcome to be good. And so this is why companies like McDonald's, for instance — what they really promote is reliable taste and quality. It might not be at the quality that you would like, but it is reliable.

Sangani: It might not be In-N-Out, but you're gonna get the same thing every time.

Wu: That's right. Exactly. And so those are semantic coupling in the sense of: it's not about whether or not you get the service.

How is agent reliability different from traditional system reliability?

Wu draws the distinction that reframes the whole problem — traditional reliability is about performance, agent reliability is about meaning.

Wu: Traditional computing, when we talk about reliability, it's whether or not the performance is good. If it's fast enough or it scales enough, or if the service doesn't go down. But here, reliability means: did it interpret your request correctly? Or did it do the right thing? Because we're asking LLMs and AI to basically make semantic decisions for us. And if those are part of the automation process, then those can be semantically coupled.

Sangani: You're taking a systems-based approach, and I think your observation is basically, look, this is different because where in traditional computing you can have these discrete, deterministic, highly trustworthy handoffs, here you cannot. And in this systems thinking approach, you sort of use supply chain as an example, where there's compounding quality and provenance depending on the quality of the underlying providers and suppliers of a given good or service. Is that informative to how you think about your work in the lab?

How does one think about quality assessment? How does one think about — if you're all the way at the end of this chain, what have you learned about what you have to do, and how do you think about assurance of quality over time?

How do you redesign a computing environment so agents can be trusted?

Because an agent can generate arbitrary code and call arbitrary tools, Wu argues the environment it runs in has to change, not just the agent.

Wu: That's a really good question. The traditional way is you have these layered computing stacks, and you improve them one at a time. But the thing is, agents can just generate arbitrary code. They can execute just arbitrary tool calls, and they're operating in this just general computing environment. When you run Claude Code on your laptop, it just wants access to everything. And so the surface area of what it can interact with and do is very, very large.

But at the end of the day, they consume and generate information. They need to operate under these controls and guarantees and policies and expectations and user preferences, and they also change and manipulate computing state. This would be in the database, in the file system, in your process memory, and so on.

And so a lot of the questions here are: if it's operating in this complex, real computing environment, how do you redesign these environments so that with new capabilities, that can allow you to trust the agent automation a little bit more? So, for instance, can a data system help the agent search and find the relevant evidence in order to answer a question, even if the data is stored in a sprawling 10-terabyte data lake? Search and acquiring information is just fundamentally very, very hard. And so what kind of capabilities can you provide there?

But similarly, if the computing environment can say, "Hey, you can make as many mistakes as you want, but we can always recover from it," then a lot of the complexity in the application — in the agent or in the harness — can disappear.

What can agent infrastructure learn from 1970s relational databases?

Wu's central historical analogy: the relational database absorbed costs that applications used to carry, and something similar is owed to agents. Alation has written about LLMs as a new OS layer for the same reason.

Wu: This is very similar to just very classic database technology. Before the 1970s, relational databases as we know them today didn't really exist either. And so what that then meant is the database couldn't be trusted to manage data models, to do schema transformations. It wasn't reliable. Performance was pretty bad. And basically applications needed to handcraft programs in order to access data.

And so what that then meant is the application developers needed to absorb all these costs that we take for granted today. Like data corruption, query optimization — transactions, as a concept, didn't really exist. And so application development was like hundreds of times more expensive and slower than it needed to be. And so what the database did was absorb a lot, a lot of this cost. And so that's what we see as something needed to do today for agents.

Is the database the right place to make AI agents reliable?

Sangani asks whether the database simply expands to swallow these problems the way it has before. Wu's answer is that the database is now one tool among many.

Sangani: So do you see the same outcome playing out, in the sense that — I mean, ultimately the database became a singular thing, and famously Oracle every time they saw a problem would just build a new feature into the database, and the database would become this bigger and bigger and bigger thing. And even now today it's like, "Oh, we're just gonna build AI into the database." So do you see the database expanding as an artifact to effectively cover many of these failures and scenarios? And also, does that mean that it's sort of the extension of what we think about as a database today, or is it something entirely different and new?

Wu: There exist databases today that are providing new capabilities, such as branching and restore and so on. I think one of the things that makes agents tricky is the database is just one tool or one component that the agent interacts with in order to do a task.

And one of the exciting parts of LLMs and agents, and also one of the parts that makes it difficult, is that its interface is just text, which means anything that you can imagine, you could conceivably ask for it. And so the promise in terms of the classes of tasks that it can in principle support is very wide-ranging, beyond simply database queries.

Now, of course, databases house very important data and provide guarantees around it, but a lot of the tasks today are not purely database tasks that people use agents for. They're like reconfiguring and monitoring your cluster. Or they might be doing general housekeeping on your laptop. And so the range of what types of data and state are being manipulated and need to be managed is much, much larger and goes outside the database. So the database is, of course, very, very important, but it's just one tool at this point.

Why does layered architecture thinking break down with AI agents?

Sangani names the intellectual challenge for anyone classically trained in software architecture, and asks Wu how architects respond to it.

Sangani: I think for me, the interesting thing about this moment is that there's a lot of, to your point, blurring between the lines. And so systems thinking is really critical, because effectively you have to look at every part of the system to guarantee quality of the output. And what's hard, I think, for many people that are either classically trained in programming, or that have sort of thought about architecture, is that you always think about these stacks.

You think about layers, and you think about discrete handoffs. And that construct almost doesn't even make sense, because in many cases you can't define the quality of one layer without understanding the outcome or the output of the layer above and what happens underneath.

And so these layers bleed, and that bleed is not something that people care — like, people want to just do the job and then hand it off to the next person, and that kind of thinking just does not work. Have you seen people deal with that fundamental intellectual challenge? And certainly in industry, how do classically trained architects think about and respond to some of your work?

Wu: I think it's a fairly new way of thinking about things, because like you said, a lot of us are trained in a particular area, and that's what we traditionally focus on. And I think it's very difficult because what LLMs are really good at is learning expertise in one area and then transferring it over to a completely different domain.

And so that's something that LLMs and AI is very good at, but it's not something that a person might be very good at. And so I feel like for a variety of different reasons — it could be organizational reasons, or it could be existing legacy technology reasons — typically what we've found is that people are reconfiguring existing systems, but trying to do more in the harness or in the agent workflow structure in order to deal with the lack of reliability in the underlying environment.

What happens to AI agents when metadata quality is poor?

Wu uses cataloging as his example of a low-level quality problem that surfaces as user burden. It's the same dynamic behind metadata quality and model accuracy.

Wu: So that's generally what we see. And it kind of makes sense, because one example: you guys, Alation, are doing managing like huge amounts of data and cataloging and so on. If the quality of the metadata is not very good, then you can't really trust the search results.

And if you can't trust the search results, then any application you build on top of it needs to expose this information to the user. And the user has to make a judgment call. And that places a huge cognitive load on the end user trying to use this tool. And then they have to then make a decision, like is it worth it or not?

And so that's one example where something very, very low in the stack, just indexing, can permeate all the way up to the end user, and their experience.

How do data products and ontologies give agents something to trust?

Sangani describes how Alation's own answer to this problem moved from cataloging to a data products marketplace with ontologies underneath.

Sangani: And there's lots of different ways to solve that problem. We've been known as a catalog, and what we realized is that — our view on this is that when people use these things like catalogs and the like, they're doing it either for the purpose of trust or they're doing it for the purpose of discovery.

And what we've come to believe is that basically, whether you're building trust by describing, or somehow doing something with the data that allows you to tag it or improve it or make it better described, or you're consuming the data, you're gonna do that, in our view, using agents. Because, like many problems, you just don't have enough resources in order to analyze this stuff or improve it over time.

And so what we came to realize was that the catalog is sort of a thing, and to your point, whatever is there is there. If it's wrong, then agents — or when people are gonna both take the information and conclude wrong things.

So we ended up building sort of this abstraction, which we're calling a data product, alongside these ontologies, which are basically what agents use then in order to be able to do their work. That kind of solves the problem.

Why you're never done improving an agent system

Sangani's point about the improvement loop is the practical consequence of semantic coupling: errors can originate in the agent definition, the abstraction, or the data itself.

Sangani: But to the original point that you've made, what we've also then found is that, okay, great, well, you have these agents, these agents work on these ontologies and data products to do stuff, and then they're reading the underlying data, which hopefully is correct and governed.

But systemically, you can define whatever evals you want with the agent. The problem that we find is that whatever you do end up defining, there's always this kind of error detection improvement, and the improvement has to happen sometimes based on the tools that the agent is using, which is in the agent definition.

Sometimes it's in the actual description of this abstraction — the ontology or the product might be wrong, or underlying data might be wrong. And so you're in this constantly iterating loop of improvement that's needed, because you can't assure fidelity day one in the same way that when you build a software program you can.

And so it's just like you're never done with the work on some level. And this to us, this systems thinking is really something that has been very different. And we've had a challenge because a lot of our customers have said, "Okay, well, I just want to build my data layer, and I want to build my agent layer, and I want to build my context layer, and I want to think about these things discretely."

But these things are just completely not — they're completely tied together. So where you want to draw big lines with boxes, the lines are actually highly permeable. It's a little bit more like biology than it is like physics on some level.

Wu: Yep. This is that semantic coupling. It's like everything affects — somewhere that you've never even heard of or thought about is gonna affect whether or not you get correct results or good results. And that's very, very difficult to even measure and evaluate.

How do agents find the right data in a massive data lake?

Wu turns to the problem his lab has worked on longest: search at massive scale. Alation has compared the options in knowledge graph versus data catalog terms.

Wu: One of the things that we, from the data side, have been looking at and working on for actually many years is: how do you even find the right data if you have terabytes of just whatever is out there in your data lake, in an enterprise or even online?

Does a curated knowledge graph scale to millions of datasets?

Wu grants that curation works at small scale, then explains where it stops working.

Wu: I think, to your point, if you have a small enough corpus of data and you can highly curate a really high quality knowledge graph, and you've vetted and checked all the data, then yeah, you can rely on it. Because all of the different pieces — the search piece, the data piece, and then hopefully the analysis piece — are all reliable enough that you can trust and know, okay, it's good.

One of the things that we've been thinking about is: what if you scale to terabytes and terabytes of data and millions of datasets? At that scale, it's very, very expensive and maybe just impractical to build and curate a high quality knowledge graph. It might require a lot of humans, or very, very expensive models, or some combination of both.

And so one of the things that we've traditionally worked on is how do you do search if you have a machine learning question, like a machine learning model that you want to improve, or you have a causal inference question.

Why do multi-source questions break AI agents?

Wu's worked example — a question about New York City schools — shows how many separate pieces of evidence a single natural-language question can require.

Wu: But over the past year, what we've been looking at is: what if you have just a natural language QA, question answering, setup? Then how do you find even the right evidence in order to answer a question?

So for instance, if you ask, which New York elementary school is near or in the Clinton Hill area, has an average class size below some threshold — because you care about class sizes in the past five years — and has few violent incidents in the neighborhood?

That's a question that needs to piece together lots of pieces of evidence. You need to figure out the school districts, you need to, in New York City, figure out average class sizes, you need to figure out reasonable thresholds over the past years, as well as neighborhood violence statistics and so on.

And so that's a lot of pieces of data that, even if they're all public and you have access to it, how do you actually find them and know that those are the right pieces of data?

Why do AI benchmarks fail when models have memorized the internet?

Before you can improve retrieval, Wu argues, you need an evaluation that can't be passed from memory.

Wu: And so that's a challenge both in how do you even design and measure this type of question, and evaluate to know that you're making progress. Because one of the big challenges in question answering is models have memorized the internet.

So a lot of these question answering benchmarks and setups — you don't even need to read the data in order to answer the question, because it's just memorized it all. And that was a huge problem over the past couple years as people tried to build these kinds of systems.

And it's also a question of: how do you even design an evaluation where you're forced to even look for data? Because so many people in the data community, and companies, are working on data search, but how do you evaluate the end-to-end quality? And how much search is even the critical bottleneck in this end-to-end question?

Because what does the agent need to do? It needs to take your question, it needs to figure out how to decompose it into a series of sub-questions, such as schools near Clinton Hill. Then it needs to figure out, okay, I need to find information about Clinton Hill, and I need to find locations of schools. And it needs to then search over 40 million documents and datasets that we've collected, and find the right ones. And at any given step, if it didn't find the right data, then the whole thing falls apart and you can't answer the question.

Why does a user's unstated context change the right answer?

Sangani raises the part of search that lives in the searcher's head, and Wu agrees that the stated question is often not the real one.

Sangani: In traditional search, you look for something, and when you look for something, there's some set of returned values, and the person who's searching has a secondary set of — in the AI parlance, there's more context inside of the searcher's head.

So their initial query is going to inform a step, but it won't necessarily give all of the context that's required in order to be able to come up with that final list that you're describing. Because maybe I'm looking for all of the schools, but maybe I'm only looking for all of the schools with a lunch program, but I didn't think to ask about the lunch program in my initial query.

And so your initial prompt and then knowledge of the user are really important in getting to the right result. And sometimes that's not declared.

Wu: That's right. And the right result — if you just take the question at face value, it might not even be the right question, at the end of the day, that the user cares about. Exactly like you said, because they have their own preferences.

And often, a lot of times, no one knows what's out there. If you have terabytes of data, no one knows what that data is, fully. And so you're using the system to just figure out what questions to even answer and what can be answered. And so it's a very HCI type problem, because you're iteratively figuring out is the system even capable of answering questions, and what types of questions do I even ask of the system?

How do you show users what an AI system can actually do?

Sangani describes the interface design problem this creates when everything collapses into a chat window.

Sangani: That's one of the things that we found that's been so interesting in interface design for us, as we've been thinking about how we reveal what we do. So we had all sorts of things ranging from data quality to cataloging to discovery and all this data product creation and agent creation.

So you think about these things, and people don't actually know what they can do inside of our system. And so then the question becomes, well, it's a chat window. You could theoretically do anything inside of the system. How do you actually reveal it? And you can prompt people and show some samples. But this discovery of what can I do versus what I ought to do is this really —

And then you see people asking questions literally just like searching, like what can — and they're sort of like, "Can I do this? Can I figure it out?" So it's a really hard problem, I think, because everything seems to boil down to chat, but chat is both deeply powerful but also very mysterious.

How do you build a benchmark that forces an agent to search?

Wu's lab designed its evaluation so that every search returns a database the agent has never seen. Alation covers the practitioner version of this in AI agent evaluations.

Wu: That's right. And what we found basically, when we started this project last year, is that just evaluating even the simple case where you have the question, it's the right question, and there's one single answer, is very, very hard. And so most of these evaluations of these benchmarks are on very, very small datasets, or even they just give the data to the LLM.

So it's really measuring reasoning. And so we really wanted to build a benchmark where it requires search at massive scale, that requires complex planning and reasoning. And the search system, every time it does a search, it returns basically a new database that the agent hasn't seen before. Because it's doing top K, it's returning just some documents and some tables, and the agent needs to then figure out which of these is the right ones to even look at and analyze to figure out the next step.

How well do frontier models do on massive-scale data search?

The headline result from the lab's ablation study, presented at VLDB, one of the main database conferences.

Wu: And so what we found is, even very powerful models — using Opus, on just a small subset of this corpus — it doesn't do a particularly good job. And a big reason is because it just never even retrieves the right data. And even if it retrieves the right data, somewhere in this database of results, it doesn't look at the right one. And it's expensive.

Sangani: We have five results and look at the third, and you're just like, "That's not the one."

Wu: That's right. Exactly. And so if this is Opus, then you try weaker models, and if you're using like GPT Mini because it's cost-effective, it gets like 2% of the questions right. And so it's a very, very complicated problem. But we wanted to design a benchmark where you can carefully control the evaluation, and be able to measure progress on just search, or just planning, or any combination, so that you really know what you're getting.

'Cause otherwise, you get this end-to-end result, and we couldn't even know. We got Opus answers like 45 of the tasks correctly. What contributes? Who's responsible? And what's the ceiling? And even that wasn't very clear. And so that's why we both built this benchmark, and then ran some careful ablation studies to really understand.

Because if your setup is this massive scale data lake, then which components contribute to the end-to-end quality is gonna be very different than if you have this very small data lake that's fairly well curated, so that search is pretty good. And the failure modes between different setups and different types of questions is gonna be quite different.

And so that's one thing that we're presenting actually next week at VLDB. So that's one of the main database conferences. So we have this workshop paper where we do this careful ablation study to really try to understand what the ceiling is and what the floor is on each step in this type of large scale question answering.

What does trust mean when AI can persuade as well as answer?

Sangani raises a different definition of trust — not correctness, but persuasion — and asks whether it belongs in the lab's scope.

Sangani: You mentioned one of the things you talked about is this notion of trust. And this gets back to the fundamental thesis that you have, which is that to get to an outcome, you need to have the person really believe and trust it, and that's a really hard thing to do.

And yesterday I watched this recent interview with Yuval Harari, and he and Elon Musk have this thesis that in 10 years AI's gonna be — we're gonna have super intelligence, AI's gonna be in control. And that we're gonna be sort of doing the biddings of our AI overlords. And one of the interesting things that they talked about was this idea that AI's gonna learn you so well that it'll just be able to manipulate you very, very arbitrarily.

And so normally when we think about trust, we think about this from the perspective of, okay, what is it that we need in order to be able to get a correct answer that we can verify? And it's very much about absolute truth, and we think about evals and we think about what the standard happens to be. And obviously that's always traveling, but at least there's something that we're trying to target.

And in their case, they were talking about a much more relative trust, which is that I can get you to trust me by manipulating you, not necessarily by actually having the right answer. You mentioned HCI style interactions. How does that flow into the lab? How do you think about — are you thinking about some of those problems? Because to me, those are in some sense problems that are outside of the system, but deeply insidious and very hard.

Wu: That's a really good point. I guess the way I think about it is, there's always a technology gap between what people want and expect, and what technology can really provide and guarantee. And there's really two options. One is to try to improve what we have in technology to reach what people expect and want and would like to have, and the other is to lower expectations.

And I would like to think that research and systems work, and this multidisciplinary work, can focus on improving technology and getting to what we want, rather than lowering people's expectations. And I think one of the reasons is because new capabilities in a system immediately have effects on the usability of a system.

How do teams manage the hundreds of rules they put in agent prompts?

Wu's HCI colleagues catalogued the rules people write into agent prompts, and found thousands of them.

Wu: So one example of this cross-disciplinary work is, some of the HCI folks were looking at rule following. So if coding agents, or any of these automated agents — what you actually see in these prompts is a huge portion of it is actually trying to tell the agent what to do and what not to do.

Like obey this policy, don't write to this file. Hey, before you commit, you should always do X, Y, and Z, and so on. Or like, "Don't touch the database. Don't touch the database." And maybe that's copy and pasted like five times. And so there's all these things that people put in.

And our HCI students found hundreds or thousands of these rules. And so what they were looking at is actually how do you just manage this rule management process? And if you think about any organization, you're managing all these policies, all these implicit rules, and so on.

And we have existing systems to do that. But here, you're doing this for an agent, and which rules are relevant is contingent on the particular context and particular task — which step in a coding pipeline the coding agent is performing.

And so what our colleagues in HCI were looking at is just how do you manage all these rules from a usability perspective? And so in terms of the usability side, they studied how do you design this, how does a user reason about all these policies and when they're followed or not, and so on.

Should agent rules live in the prompt or in the system?

This is the pivot in Wu's argument: most rules don't belong in a prompt at all. Alation's take on policy enforcement at the data layer starts from the same premise.

Wu: But the actual implementation was, let's put a subset of the rules into the prompt rather than all of them, and hopefully the agent will follow these more reliably. But the thing is, we looked at these and we said, "Oh, actually the majority of this stuff we can push into the system." And so it doesn't have to be probabilistically enforced. We'll just enforce it.

Sangani: When you say pushing into the system, that's not the agent problem. What is that? What is that system?

Wu: This could be the database, it could be the operating system. It could be an orchestrator, it could be a linter. So for instance, it could just be as simple as check the code. And you have a deterministic checker over the code to see whether or not you're raising exceptions when there is an exception. Because somewhere in the rules, it asks for that.

And so what we found is actually the majority of these rules, you could just enforce all the time. And the nice thing here is, if it's not probabilistic, and you can rely on it and it's guaranteed, then the system doesn't need to think about it at all. And in fact, the environment can then identify these triggers, and then tell the agent, and tell it what to do instead. So these are new capabilities because you have something to rely on. You need somewhere to ground reliability and quality.

Why is prompt injection really a data flow problem?

Wu's shopping example reframes prompt injection: the failure is that data from one source reached a place it shouldn't have.

Wu: So we've been looking at this in the context of safety as well, because one of the things we noticed is agents are actually just really bad at reasoning about data and data flows.

So for instance, if you ask the agent, oh, go and repurchase a shirt that I bought. Then the agent might go to your order history and find that shirt and add it to the cart and check out. But then it might be, oh, I need to go do my email verification in order to check out. So then what it'll do, it'll go to your email and search the recent unread emails, and it finds a verification, but it might find another spam that says, "Hey, you should buy diet pills."

And then it also adds that to the cart, and then you bought diet pills as well. And so that's a simple example of prompt injection, but what it ultimately boils down to is it's a bad data flow. Prompt injection is just saying some unreliable source that you didn't anticipate has injected information into your agent, and it follows it.

And so on one hand, you can add a bunch of rules to say, "Hey, only look at email verification." Like, don't look at other emails. Add all these things, or check the tool calls to see, oh, if you're adding diet pills, then you shouldn't add it to the cart, something like that.

But the thing is, you might have even bought these in the past, you just didn't want to buy it today because you have enough. And so there's no way of checking a tool call or checking just a query of any form and knowing, oh, this is safe or not safe. Because it's about these data flows.

And so that's an example of something where the agent harness just doesn't have visibility of. Because it's really about, oh, the data from my order history is allowed to flow to my cart, but data from my email is not allowed to flow to my cart. And that's something that the agent simply doesn't have visibility on anyways. And so there's no way it could possibly enforce this kind of safety.

What is data flow control for AI agents?

Wu defines the lab's Dataflow Control project by contrasting it with the constraints and access controls databases already have.

Sangani: And so how would the system, in that example, how could the system work? How would the system work? Because the prompt never contemplated this idea of the context coming in from the email. How does that get resolved?

Wu: So over the past year, we've also been working on this project called Dataflow Control. And the idea there is, let's actually track and allow constraints over data flows rather than just over static data.

So in traditional databases, you have constraints, so for instance, NOT NULL. And what that allows is basically, certain types of data are not allowed to exist in the database. Like NULL values, or ages above 100 and so on. You can enforce those in the database. Or you have access controls that say you can access data, or you can write to data, or you can read the data and so on, if you're a particular user.

But then what we're interested in is, well, can we track the data flows and have constraints over them? So you might say something like, "You're allowed to make decisions and have data flow from order history to your cart, but from nothing else." Because that's the only thing I care for.

And if you can write that policy down, and if the system can deterministically enforce it, then it doesn't really matter how you design the agent. If it tries to do something that's disallowed, then the system will just deny it. So it's not about prompting harder and so on. Those can increase the likelihood of doing the correct thing, but you don't have to worry about it doing the wrong thing.

So that would be, in some case, like a whitelist. But it's not a whitelist of a command or an action, but of a particular data flow. And I think this is really important because the same query, depending on the content of the database or the state of the environment, could be safe or unsafe. Or policy-abiding or not policy-abiding.

The trivial example is, I don't want sensitive data to ever leak out of the database or the system. But the thing is, if your database is empty, then trivially that holds true. But if your database is full of sensitive data, then that's an unsafe action.

Should policy live with the data or with the agent?

Sangani's stop sign analogy makes the architectural claim explicit, and describes where Alation landed on the same question.

Sangani: And so on some level, the policy lives with the object that it's trying to govern, not necessarily with the agent that's trying to leverage it. It's like, you put a stop sign at a road, you don't put the stop sign in the head of the user so they have to memorize that stop sign every single time they are getting to the intersection, because they wouldn't know how to disambiguate when to stop and when not to stop.

On some level that sounds a lot like — we've sort of arrived at similar solutions, although I would probably describe it differently, if only 'cause we don't have the same semantics. But I think the way in which we've described it is that a lot of the policy lives with the data, and you sort of push it down to the lowest grain.

To your point, if it's PII, you're like, "Don't use this." Or if it's a more abstract process, you're building it into an ontology of some sort in order to be able to allow the agent to traverse the rules and the policies and the standards of the organization. And sometimes those are strictly defined rules, and some of those are more implicit.

The problem though, of course, is that you get into this place where, well, you didn't really realize that you needed a policy. You have to learn the policy because the mistake in some sense informs whether you need it. And that's I think where it gets really hard, because now you're encountering these new cases as the agents do their work.

How much agent behavior should you restrict up front?

Wu's pragmatic middle ground: carve out the spaces agents must never enter, and expand guarantees over time.

Wu: That's definitely true. And I feel like that also overlaps with expectation. I think our expectation is both high and low for agents. On one hand, we're kind of like, "Oh, the agents fail in a bunch of cases, so let's restrict what they do." On the other hand, we say, "Oh, but the surface area is anything I can possibly think of, so I expect everything out of it."

And probably a pragmatic middle ground is, let's try to build systems where we can carve out spaces for which we never want the agent to go into. And over time, we can guarantee more and more of reliability, or of safety and so on.

Because at the beginning of any enterprise or any application, you also don't know what the user really wants. Who the customer is even, at the beginning of a company or a new application. And you're learning over time what the expectations are.

And so I think what we really want there is the primitives that allow us to evolve over time, but still know when we can rely and when we cannot. I think that distinction is also very important. Because the same way, if you have a very heavily curated data set that you know is correct and golden, then you just allow people to analyze it and use it.

And the same thing in the agent world with software. It's not like agents are writing new operating systems for themselves to run the agents on. Maybe eventually that'll happen, but right now we still rely on high quality pieces of software as the building blocks. And there's a reason for that, which is because we know that those, in the particular context in which we can use them and agents use them, are way more reliable than anything else that we have access to.

What are the biggest unlocks for AI agents in the next 12 months?

Sangani asks where the bang for the buck is right now. Wu's answer is unglamorous and specific: move what's in the prompt into the harness.

Sangani: But what in your mind — so I guess there's problems in the next year that are really critical to solve, and then there's problems in the longer horizon that are really critical to solve. So let's talk a little bit about time. What do you see in the next 12 months as being the biggest unlocks for people to advance their utilization of agents, whether in the enterprise or outside?

In terms of the folks that you're interacting with who are contending with real world applications, where is there the most bang for the buck in the world in order to be able to get to some relative advancement in agents? And what are the problems that we need to really focus on solving right now?

Wu: I think right now on the safety side, there's a lot of low-hanging fruit in translating things we're doing in the prompt and asking agents to do and reason about, and just pushing it into the harness or into the environment.

How does data flow control map to financial regulation?

Wu's work with Intellect Design, which builds the products banks use to provide services, turned up a use case the lab hadn't anticipated. Building agents against governed policy is the idea behind Agent Studio.

Wu: So one of the companies that we work with, Intellect Design, they're in financial products. So they build the products that banks use to provide services. And one of the things we're working on with them is, it turns out, unbeknownst to us, that this data flow control — which is really about constraints over data use and how it's actually being used — is not only useful for the toy applications we're thinking about, but a lot of financial regulation.

Because those today are often just thrown into prompts, where you say, "Hey, I did this work, and can you check it against these federal financial regulation policies?" But it turns out actually a lot of these can be pushed in and just checked all the time, at scale.

And I think that's low-hanging fruit that we see in a lot of other places. For coding agents, the same thing. Oftentimes we ask coding agents to not touch files, or to make sure it does X, Y, and Z before it does something else. And those are things that you can really deterministically push into the harness.

And so I feel like that's really just low-hanging fruit that we can do. And the idea is just, let's remove as much as we can out of these prompts. A, because then they do the task a little bit better. And because you can then have these guarantees. So I think that's very important.

Why do agents need to fork and roll back their environment?

Wu's longer-horizon argument: if every component is stochastic, the environment has to let agents explore.

Wu: I think in the longer term there's a question of, what is computing infrastructure if any system component or any application that uses an LLM now becomes stochastic and sometimes wrong? What that then means is the computing infrastructure — for instance, the operating system or any application — needs to allow agents to basically explore.

Because what we've seen in a lot of other areas like math, or playing games and so on, is if agents are able to explore and do this test time compute, which just means it can trial and error a bunch of times and then see what happened — if you can do that with computing environments freely, then that means we can gain a similar amount of quality improvements on its ability to do tasks well.

So in order to do that, then the computing environment — like the operating system, the databases, Python — they all need the ability to basically fork their entire state and then to be able to then restore. And with these two capabilities, then agents are able to then explore and try things.

And so that's one of the systems that we've been working on for a couple of years here at Columbia, where it's basically like a Docker-like container environment that allows for this. And what we've shown is that if you run these computer use benchmarks on top of such a system, it can trial and error and try a bunch of things, and it just improves both the speed — because you're not resetting the entire environment and doing things from scratch each time — so it improves the speed, but also the quality of the outcomes.

How can an agent try risky actions without breaking anything?

Running a legacy web application inside a branchable environment lets a web agent explore destructive paths and roll them back.

Wu: But the other thing we've looked at is, what if you just run legacy applications in this kind of computing environment? So if we took a Shopify clone and we run it, then the web service itself can also branch itself.

So then now you can have a web agent, and you can say, "Hey, go find a good combination of outfits and get me to the purchase stage, and then I'll decide what to do." And it can go and in parallel try a bunch of different searches and click on things, and not worry about having destructive actions, because you can always roll back.

And it can then give back to the user and say, "Here's a bunch of options for you." And the cool thing there is it also interacts with both agents and how you design them, but also the HCI, the usability aspect. Because if the agent has tried a bunch of things, what do you show the user as options?

And so our sense is that the computing infrastructure might have to change as well, if we assume that agents will do a lot of things on our behalf. What that then means is they need to be able to fail and do incorrect things a lot of times, and the computing environment needs to then support that.

Is running hundreds of agent rollouts too expensive?

Sangani raises the cost objection: simulating the world many times over to make one decision is not how computing has worked.

Sangani: At least in these sandboxes. It's one of the reasons why people are so paranoid, and rightfully so, about running systems like OpenClaw. Because effectively it's this massively unrestricted agent that can do anything it wants to in your computing environment, which, if it gets access to your passwords and usernames, could even be like, "Oh, let's decrement all of the money in my bank account to go buy this car that I didn't need."

And so you can do all these things, and obviously agents, like in what people are normally constructing, are restricted to very specific prompts and tool calls. And then you've got this space in between, and this idea of a sandbox is super interesting.

You could theoretically run a very large number of them, but it is a compute problem, because it starts getting super expensive if you simulate the entire world like 15 times to get to some set of different decisions. And so there are lots of interesting compute challenges that come up in terms of determining — you generally don't run 15 compute — like you don't think about something 15 times to do it once. So that seems very expensive.

How are — are you finding that there are ways in order to optimize compute as you do this work better? And what are the learnings that you're coming up with as you're doing some of the simulation work?

How do you run thousands of agent rollouts on one machine?

Wu's answer is that you only ever capture the difference in state, using mechanisms operating systems already have.

Wu: Yeah, we definitely looked into this. Our early version of this actually just ran one at a time, like one step at a time, and just was able to roll back. But the thing is, most of the state — in principle, the only state you should be capturing is the different state from what you already have, that the other agents are working on.

And so what we basically do is we just track the minimum amount of state in the operating system, meaning the memory pages, the file system, the terminal and process state and so on, for a given agent rollout. So agent trial.

And so what we've recently had are experiments where we can run hundreds or thousands of these in parallel at the same time, on one machine. And part of the reason is because you don't need an entire new copy of the entire operating system state in order to fork and try something again.

And we have these mechanisms already in operating systems. For instance, if you fork a process, we don't just copy all of the memory state. We have shadow pages, and we do copy on write and so on. And so these types of mechanisms can make it a lot cheaper.

Why branching breaks when a database and an operating system don't coordinate

Wu walks through a concrete failure: a branchable database and a branchable process that don't know about each other.

Wu: And not only can you do it at the operating system level — increasingly, different software components, like database providers as well as Python notebooks and so on, are specializing this kind of mechanism inside their systems.

So the way we think about it is, the operating system's job traditionally is to manage resources on behalf of the application. And this branching is a type of resource, and it requires some kind of mediator in order to actually mix and match and do this. Because if you are running a database inside of this branchable operating system, the database should be managing branching for itself and not the operating system. Otherwise, you don't know what's going on inside the database as an operating system, and you're gonna incur a lot of costs.

So you want something where generally you can just drop in some kind of application and it just works, but if you have specialized components, then the system should orchestrate those.

And the orchestration turns out to be a little bit tricky. So for instance, we've tried just taking Dolt, which is a branchable database service, and just connecting it to a very, very simple web service that just makes a connection. So Dolt supports branching, and Python supports branching via the operating system.

But actually, if they don't coordinate between each other, then what happens is you checkpoint and you restore the web service, and then the database connection disappears. Because the database is like, "Oh, it's been a while. I haven't heard from the web service, so I'm gonna close the connection."

And so it's not as trivial as just throwing things together. There needs to be protocols as well in order to manage all of this. And so potentially — and this is why it's research — potentially EC2 or these kinds of compute infrastructure would provide these capabilities and allow orchestration between all of these in the future. And it's a big compute problem, and so some of it is a software problem, and some of it might end up being even like a hardware level or rack level problem in the data center.

What hardware changes make agent branching cheap?

Sangani asks about the hardware frontier, and Wu points to work by his Columbia colleagues Kostas Kafes and Assaf Sedon on rack-level memory transfer.

Sangani: And I guess every system has to provide some of their own orchestration capabilities, because on some level only it knows its primitives and what things are needing to retain state versus not. So there's a lot of work to do there for sure.

Talk a little bit about the hardware side. You obviously have done a lot of work on the software side and search and HCI, and you're alluding to, this is gonna be stuff that happens at the rack level or at the system level. What is the work that is most exciting to you for just the underlying compute, and where do you think things are going there? And educate folks about what is the most interesting cutting-edge stuff that either they may have heard of or may not be so familiar with.

Wu: The broad trend in my mind is there's a huge amount of work on AI infrastructure, making tokens generate faster. And in general, I feel like that's — of course, it's a very technically difficult problem, but it's a well-defined problem. Because you know what the ground truth tokens you should generate are, and you just need to generate faster.

So from that perspective, AI and token generation will drop in cost, but then it'll shift towards, in the automation world, to the environment and to general compute. Because if tokens are free to generate, and deciding what the next action is is free, then you're bounded by the cost of making tool calls, which means interacting with the general computing environment.

And so there, some of the work that my colleague Kostas Kafes, as well as Assaf Sedon, have been doing is looking at, at the rack level, there's these new technologies like CXL that allow very, very fast memory transfer between pools of compute.

And so what they've been looking at is super fast ways of shifting and forking virtual machines between compute instances, and being able to replicate and migrate. And if you have something like that, then that of course makes branching a lot, lot cheaper. If branching becomes so ubiquitously cheap, then that changes the user expectations of what it means to ask an agent to do a task.

What will the AI agent conversation be about in 12 to 18 months?

Wu closes on the problem he expects to outlast the current cycle: interaction design between people and probabilistic systems. Alation's own answer to this is the Alation Intelligence Operating System.

Sangani: It starts becoming very — I think we talked a little about biology before, but it starts becoming very organic. And then you have models that are spinning up other models, and you've got very small models doing discrete tasks and writing back. And so it just starts becoming a lot more like biology than anything else, which is super fascinating.

So maybe Eugene, take us out a little bit. You deal with a lot of enterprises, lots of people are working on agents in the enterprise. What are we gonna be talking about in 12 to 18 months? What is gonna change, and where will the conversation shift?

Wu: I think we already see this today where we're increasingly talking about verifiable tasks, and verifiable rewards and these sorts of things. But compute is generally easy to scale, and the difficult thing is understanding what the user actually wants, and the semantics.

This is something that I think database people have known for decades. Because in addition to transactional databases and keeping things from being corrupted and so on, data integration and data cleaning and data quality has been a perennial issue since the dawn of time.

And what we've all known is that the quality, and what it means to have clean data, is completely contingent on a lot of factors, and one of the main factors is what the user really wants to do. Oftentimes it's all in the service of user desire, or what they're trying to push a business or an application to be able to do. Or just their own imagination.

And I think understanding the interaction design between the human and this biological, probabilistic system is going to be a major, major question — not just over the next twelve to eighteen months, but for probably the next decade. I mean, computing often changes based on what the user wants and where applications end up going, and that's something that hopefully becomes more diverse and more creative and more exciting, rather than reducing to what AI is capable of today.

Sangani: Eugene, it's been awesome having you on. Every time I speak with you, I learn something new, and this episode did not disappoint. Would love to have you back on at some point soon, and thank you for taking the time.

Wu: Awesome. Thank you.

Want to give your agents data they can actually trust? Learn more about Alation.