Centralized or Federated? Data Architecture for Agentic AI

Enterprises spent thirty years making sure IBM is always spelled IBM in the database. Never International Business Machines, never Intl Business Mach, because a canonical value is what makes a record countable. Anant Jhingran, CTO of IBM Software, thinks a large language model can now resolve those variants on its own, and that this is roughly the least interesting consequence of the AI shift underway. The more interesting one is what happens to every architectural decision enterprises make in service of that precision.

Jhingran has spent his career on the layer nobody sees. He was CTO of IBM's Information Management Division,¹ scaled Apigee through its IPO and acquisition by Google,² and co-founded the GraphQL company StepZen before returning to IBM. He also co-hosts the podcast Context Window: Enterprise AI Reality Check,³ and joined Alation CEO Satyen Sangani for a recent conversation⁴ on Alation's podcast, AI Radicals.

Most of the debate about AI agents in the enterprise happens at the application layer. Jhingran spends his time one floor down, where the databases, integrations, and systems of record live, and where he argues the real constraint sits.

The foundation stays, the interaction changes

Sangani opened by asking whether this AI moment resembles the platform shifts Jhingran has already lived through. The answer came with a distinction worth holding onto.

"It's really, really different," Jhingran said, before adding the qualifier that shapes the rest of the hour: "the foundation is still the same. The interaction changes."⁴ Enterprises built their data estates for two kinds of consumer, he explained: pre-wired automated processes, and human operators doing the swivel-chair work between systems. Agentic workflows introduce a third, and the question is whether that is a new modality on top of an unchanged base or something that reaches further down.

His position is that the same problems persist. "The data systems are the same, the systems of record are the same, the problems of integration are still the same. The problem of which is the authenticated piece of data here is still the same. So problems remain the same, but the interaction paradigms change completely, and because of that, perhaps some of our old layering needs to be rethought."⁴

He reached back to the API economy he helped commercialize at Apigee. Physical retail became e-commerce, then mobile arrived; rather than build a third silo, companies built APIs that cut across all of them. "Every time there's a generational shift, something changes in the infrastructure also." What did not change was the value of good metadata and a defensible single source of truth. If anything, the stakes rose. Bad data exposed to a fast-reasoning agent, Jhingran noted, simply "amplifies the fast agentic reasoning on top of it, presumably in some incorrect directions."⁴

He is similarly unsentimental about the current interface layer. On the Model Context Protocol: "MCP is the wrapper around APIs. And if you poke into it, that's what it is in most of the cases. But is that the best thing? Who knows whether that's the best thing."⁴ Skills, CLIs, and whatever comes next will contend for the same job. The metadata-hygiene requirement underneath them will not move.

Two problems hiding inside one

Sangani pressed on the oldest unsolved problem in data management: Systems of record depend on humans to fill fields they have no incentive to fill accurately. "No salesperson is going to enter the full fidelity of the conversations that they've had," he said, which means the systems running the business frequently describe something other than reality. Fixing it has always been side of desk, expensive, and low on the CIO's list. Sangani probed: Is AI making this work more valuable?

Jhingran reframed, splitting the most critical work today in two:

• There is data for AI, the work of making enterprise data usable by models.

• And there is AI for data, which means pointing models at the cleansing, standardization, and integration problems that resisted automation for decades.

Asked how far enterprises have gotten with the second kind, his answer was blunt: "I think we are very early."⁴ Most have a bit of a handle on their data estate, their integration flows, and a governed data catalog, and they are changing the application paradigm on top of that finished work rather than using the new paradigm to change the infrastructure itself.

The second category is where he sees the larger prize. Resolving structured against unstructured data is a perpetual problem, and a shared embedding space may unify records older techniques could not. He pointed to Vanja Josifovski, Kumo's co-founder and CEO, who moved to NVIDIA when it acquired the company,⁵ and who has been applying transformer architectures to structured data problems.⁶ The opportunity is to use AI-powered data quality to get past "those very, very laborious and error-prone tasks, so that you can expose to the AI use cases or human use cases something that's much better and much cleaner, and then depend on them to actually work around, not perfection."⁴

Where agents tolerate mess, and where they don't

One of the episode's key threads: AI-agent consumers are more forgiving than the integrations that preceded them because they can reason. A malformed API path used to return an error and stop. "But today, AI can say, 'Okay, I'll type /member/ID,' and it'll come back with that code, and it'll reason among itself and say, 'Oh, I got it wrong.'"⁴ Enterprises pay the AI gods in tokens for that self-correction, as Jhingran put it, but the practical consequence is real. "If you believe that the users of all of this stuff are shifting towards AI, you have to be less prescriptive with it being exactly correct."⁴

Sangani marked the limit. Cleanup techniques get you to 85 or 90 percent quickly, and every order of magnitude after that costs more than an order of magnitude. "Now the work shifts to the last set of quality guarantees," he said, and agentic automation demands a higher trust threshold than the offline process it replaces. That makes agent evaluations the deciding discipline, and one most teams have not built. "People are struggling with this problem in its essence, because they just don't know how to think about what they can trust and what they can't trust."⁴

The cost of skipping that discipline shows up in the forecasts. Gartner predicted in June 2025 that more than 40% of agentic AI projects would be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls.⁷ None of those three are model problems.

Jhingran closed by emphasizing the value of foundational data. "If the data doesn't have high fidelity, any amount of lack of hallucination is not going to solve the problem."⁴ His proposed workaround for variant matching is characteristically direct: rather than indexing every spelling, "just embed every row," which is close to what semantic model mastering sets out to do.

The case for federation's return

The most consequential idea in the episode concerns why enterprises centralized data in the first place. Jhingran identified two reasons, cleansing and query efficiency, and noted the industry has always bundled them. Warehouses and lakes let you standardize data and run enormous queries against it in one move.

But agentic workloads only need one. "In the agentic interactions, really it's short bursts of queries. First let me find this, and let me find this. Oh, this didn't get me there. Let me now find something else."⁴ Rather than a fixed report or a day-long query somebody optimized, agents hill climb toward an answer. If the efficiency argument weakens, the case for moving everything into one place weakens with it, and what matters instead is whether enough active metadata exists for an agent to know what it can ask. "Without that metadata, the agents are basically groping around some space trying to figure this out." His conclusion: "It may be the re-emergence of federation with good, strong metadata space on top."⁴

Sangani named the conflation this has always turned on, between centralizing data and centralizing knowledge, and noted the boundary is getting muddier as context starts to include operational data. That question is closer to metadata versus context than to storage, and it sits underneath most current data mesh and data fabric debates.

Three pillars, and a harder question about building software

Asked what IBM's data strategy actually consists of, Jhingran named three pillars. The first is modernizing the systems customers bet their businesses on, where the core systems of record do not change but the work around them does; DB2 Genius, which lets AI observe and tune the database rather than leaving that to an administrator, is his example. The second is building new infrastructure for delivering context to AI applications, from vector stores to finding second lives for GPUs three years past their AI prime. The third is data in motion: IBM completed its $11 billion acquisition of Confluent in March 2026, folding Apache Kafka–based streaming in alongside watsonx.data, IBM MQ, and IBM Z.⁸ Adoption is fastest in the first pillar, Jhingran said, and combining streaming with structured data will take longest.

Late in the conversation, he raises a problem he says he would not have been thinking about six months ago, and it has nothing to do with databases. It is about how IBM should build products now that AI has changed what building costs.

"If you just say that AI is going to help us build products faster, then it doesn't actually create a competitive differentiation, because everybody else is creating products faster with AI. So you have to both do things differently and perhaps do different things."⁴

Doing different things might mean giving up the assumption that a product is one golden code path. Before AI, releases were slow by design because customers bet their operations on reliability, and support worked because everyone was trained on a common code base. If a model can debug an arbitrary branch, that constraint loosens. "Therefore, the reasons to have one code base may not exactly exist right now."⁴ What stays shared is the differentiated core, an entity graph or an optimization engine, staffed with your best engineers and wrapped in clean APIs. Inside IBM he called the practice very nascent, with two or three products testing it.

On whether any of this breaks the database, he is unconvinced for now. "I haven't seen inklings of a revolution yet."⁴ Databases and operating systems have a long record of absorbing each new phase, and he suspects pure-play vector stores will be absorbed the same way while conceding that nobody knows yet. He leaves open the possibility that many small changes accumulate into something else entirely, and that delivering context stops being sufficient.

That is the through-line of the episode. Nothing about provenance, integration, or trust matters less once agents arrive. What changes is the shape of the infrastructure built to serve it, and most of those design questions remain unsettled. Enterprises that treat metadata as an operational layer rather than a documentation layer will be positioned to answer them.

Curious how an AI operating system built on governed context could support your agentic strategy? Start a conversation with us today.

Sources & notes

Every external claim on this page is independently verifiable. The public sources are listed here.

1. Jhingran served as CTO of IBM's Information Management Division; he is now IBM Fellow and CTO, IBM Software, a role he took when IBM acquired StepZen in February 2023. — IBM Think ↗ https://www.ibm.com/think/author/anant-jhingran

2. Google completed its acquisition of Apigee (NASDAQ: APIC) on 10 November 2016, at $17.40 per share in cash, for a total value of approximately $625 million. — Apigee Corporation ↗ https://www.globenewswire.com/news-release/2016/11/10/888859/35581/en/Google-Completes-Acquisition-of-Apigee-Corporation.html

3. Jhingran co-hosts Context Window: Enterprise AI Reality Check with Ed Anuff; the show is produced by DataStax. — Apple Podcasts ↗ https://podcasts.apple.com/sg/podcast/context-window-enterprise-ai-reality-check/id1810242930

4. All quotations in this article are from AI Radicals, Season 4, Episode 6, "The Case for Federation in the Age of Agents," with Anant Jhingran and Satyen Sangani. Full transcript available at ↗ https://www.alation.com/podcast/episodes/ai-agent-federation-anant-jhingran-ibm/

5. Kumo AI's three co-founders — Vanja Josifovski, Hema Raghavan, and Jure Leskovec — transitioned to NVIDIA in May 2026 following NVIDIA's acquisition of the company. — Fortune, 3 June 2026 ↗ https://fortune.com/2026/06/03/nvidia-snaps-up-kumo-ai-in-latest-acquisition/

6. KumoRFM applies a Relational Graph Transformer to relational databases, converting each database into a temporal heterogeneous graph in which rows become nodes and foreign key relationships become edges. — Kumo.ai, 20 May 2025 ↗ https://kumo.ai/company/news/kumo-relational-foundation-model/

7. Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value, or inadequate risk controls. — Gartner, 25 June 2025 ↗ https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027

8. IBM completed its acquisition of Confluent on 17 March 2026, at $31 per share in cash, representing an enterprise value of approximately $11 billion, with day-one integrations across IBM watsonx.data, IBM MQ, IBM webMethods Hybrid Integration, and IBM Z. — IBM, 17 March 2026 ↗ https://www.prnewswire.com/news-releases/ibm-completes-acquisition-of-confluent-making-real-time-data-the-engine-of-enterprise-ai-and-agents-302716028.html

Analyst Attributions & Disclaimers

Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027," press release, 25 June 2025.

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. GARTNER and Magic Quadrant are registered trademarks and service marks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.