Open Source, Open Question: Why Model Supremacy Won't Last

Ashu Garg on the context graph and where AI value accrues

This episode preview, read by the AI Radicals producer, introduces guest Ashu Garg, a partner at Foundation Capital, and previews the discussion of the context graph, open source versus closed source AI models, and the SaaS apocalypse.

Producer: Welcome back to AI Radicals. Most venture investors chase patterns. Today's guest bets on breaking them. Ashu Garg is a partner at Foundation Capital, where he's backed one decacorn, Databricks, and six unicorns, including Cohesity, Eightfold, and Alation. Before venture, he ran Microsoft's online advertising business.

Satyen and Ashu dig into the context graph, the infrastructure layer Ashu believes will capture more value than the models themselves. They debate whether open source is really six months behind closed source, take a hard look at the SaaS apocalypse, and get specific. Bearish on Salesforce, bullish on ServiceNow.

If you want to understand where value accrues in the AI stack, this one's for you. This episode is brought to you by Alation. If your data teams are still spending weeks packaging data for AI and analytics, there's a better way. Alation's data products marketplace lets you build certified, governed, AI-ready data products, no code required. Get the guide at alation.com/data-products.

Meet Ashu Garg: Foundation Capital partner and early Alation investor

Host Satyen Sangani, CEO of Alation, introduces Ashu Garg, a partner at Foundation Capital and an investor in Alation, and the two open with Garg's childhood Rubik's Cube record.

Satyen Sangani (host, CEO of Alation): Hello, everyone, and welcome today to AI Radicals. As you know, on AI Radicals we try to pick some fairly interesting guests, and today we have somebody who I know will live up to the billing, that is Ashu Garg. Ashu is a dear friend and also an investor in Alation, and an investor at Foundation Capital.

I've known him for the better part of the last 14 years, and it has always been a gift to talk to him. He's somebody who has a take and a thought and does not hedge, and I enjoy that candidness and that willingness to both take a position and be right and wrong, and I think you'll see that in the podcast.

One thing that I actually learned as I went through the research in this episode is that Ashu is well known for solving the Rubik's Cube in 25 seconds at age 11, I think it is.

Ashu Garg (partner, Foundation Capital): Yeah, that was four decades ago.

Sangani: I see. And have you picked up one since?

Garg: I don't touch Rubik's Cubes anymore. It's so hard to live up to your childhood that it's better to stay away. And I have brought in a bunch of entrepreneurs who showed up and showed me how they beat my... People can do it now in four or five seconds. I once had an entrepreneur who was a world champion...

Sangani: Yeah, so you must feel like AI has come along, and you are like the DOS version, the pre-Windows version of the Rubik's Cube solver.

Garg: Apparently people can do it blindfolded, which I'm still trying to wrap my head around.

Sangani: Well, that's an entirely different episode. I think since then you've invested in one decacorn and apparently six unicorns, so it seems like you've done something between that and now.

Garg: I've been very fortunate.

How do investors tell lasting AI patterns from hype?

Ashu Garg of Foundation Capital describes how he forms investment views in AI: broad reading, pattern recognition, and the recognition that patterns often break.

Sangani: So let's get into it. One thing I say about you when people talk about who they should get to invest in their companies is that you're somebody who takes a very strong position, and I think a lot of that comes from pattern recognition. And you do that in a lot of different domains, but you've particularly done that in AI, and that's different from many investors. Many investors look for the pattern as the company to validate whatever it is that they see in the world. How do you do that? What's your method? Talk a little bit about how you do research, how you think about the world, how you approach markets.

Garg: In general, some of this is just personality, and a lot of it is I tend to do a lot of background research in the world. I'm an insomniac. I'm up till 2:00 or 3:00 in the morning most days. Whether it's The Economist, or Twitter or Reddit, I try to read a lot, and I use actually both conversations and writing as a way to synthesize my point of view, recognizing that I will be wrong as often as I will be right.

And so the trick in my mind is to take a combination of what's going on in the world, leverage the experiences, the pattern recognition, but also learn to recognize that patterns are often broken with time. I mean, the best entrepreneurs break past patterns. And so assuming that history will repeat is a very dangerous thing to do, and it's almost always wrong, but it tends to rhyme even when it doesn't repeat. So that's my philosophical framework.

And then in the context of companies that I work with or even conversations you and I have had, I try to find the balance between being very clear on my point of view, but not being prescriptive and saying, "Hey, this is an opinion. There's a real chance I'm right, but there's also a very meaningful chance I'm wrong. And so you as CEO or founder need to synthesize my opinion with what you're seeing in the market and what you believe in, and then make a decision. And I will be super supportive of whatever you decide to do, even if it's exactly the opposite of what I'm recommending."

How do you validate an AI market trend before betting on it?

Garg explains how he tests an AI market thesis through conversation and debate, and uses his early bet on open source models as an example of a non-consensus view that became consensus.

Sangani: Talk a little bit about your process. You mentioned that you read a lot. What do you do to validate or invalidate a particular market trend or theme that you think you're seeing?

Garg: It's mostly about conversations with people like you. You're triangulating based on, you read three different things, three different people are saying three different things. I go down all kinds of random rabbit holes on the data, but ultimately, we're all trying to predict the future. And you have to combine what you're hearing and seeing with experience. But most importantly, I think debate. I find huge value in conversations like this. That's why I love to do podcasts myself. The debate on those issues helps me refine the point of view.

And so very often I'll start with a point of view and six months later completely change it. Very, very often I'll have a point of view that was completely non-consensus, I'll give you an example in a second, and it becomes consensus over time. I, for example, was very bullish on open source models three and a half, four years ago. And I think it was completely non-consensus four years ago, and today I think it's generally consensus that open source will play a big, big role in the AI world. And so sometimes you just have to wait it out. You have to have the patience to see the market evolve to where you are. Now, as an investor, being four years ahead can be very painful. Right. So you also have to balance your opinion with the reality of where the market is today.

Should AI application companies build on open source or closed models?

Ashu Garg of Foundation Capital explains why he told portfolio companies to start on closed models such as OpenAI's while architecting their products so that open source models can be swapped in later.

Sangani: Yeah, because you have to not just pick the winning theme, but you also have to pick a moment, which matters in valuation but also matters in vintage.

Garg: Absolutely.

Sangani: Because you could be right, but you could be right too early. Was your view on open source completely stable through the last four years, or did you have moments of doubt?

Garg: We all have moments of doubt. I think what was stable for me was it was clear to me that even four years ago, the delta between open source models and closed source models was not dramatic. Yes, there was a 12-month delta, but 12 months in the fullness of time is nothing. It was also clear that while proprietary models, whether it's OpenAI, or at the time it was mostly OpenAI, not Anthropic... I mean, four years ago everyone thought OpenAI was going to rule the world. It was clear that while they were better in some things, for many other things it didn't matter that they were better. Open source was good enough even then for some things, but relatively few then, a lot more today.

So that gave me conviction that the gap would change over time in favor of open source. At the same time, there was a gap. And so as far as my portfolio was concerned, I was recommending to all of my portfolio companies that they should ignore open source in the short term. Just architect in a way that you can swap models with time. But in the near term, if an OpenAI model is working well for you, why bother? You've got to find product market fit with your application, and as long as you can swap things out two, three years from now, you're good.

Sangani: Yeah. And your alpha's not going to be coming from model optimization.

Garg: Exactly.

Sangani: It's going to be coming from building something that's a killer app on top.

Garg: Especially early on, when even if you had a model that was one-tenth the price, it didn't really matter because you didn't have any customers.

Sangani: Yeah, totally. I want to circle back to this conversation because I think this is actually at the essence of what matters right now.

What is a context graph?

Ashu Garg, who developed the context graph idea with his Foundation Capital partner, defines the term for host Satyen Sangani and describes how decision traces motivated it.

Sangani: But you've talked a lot about this construct of context graphs, and many of the listeners to this show and to this series are people who are responsible for modeling data, understanding data, understanding information. And so, this context graph conversation, I think it's still quite relevant, but it was very, very popular. So maybe first talk us through this: what is a context graph, and then talk us through the history of how you got to this notion of a context graph and why you were so excited about this construct.

Garg: In my mind, and my partner Jay and I worked very closely on this together, almost eighteen months ago, we started seeing what in hindsight is obvious, but at the time seemed like a new insight, which is that the companies that were winning in the marketplace were finding ways to capture customer or function or business-specific context in some sort of abstraction on top of the model layer.

And so what does that mean? That means that, hey, as you use an application, the users of those applications make a set of decisions. They interact with the application, they interact with the underlying data. As they make decisions, those decisions aren't necessarily captured in the model, but they should influence how the application does things.

So I'll take a very trivial example of sales. You have an application that's automating steps in a sales process, to the extent that you're making decisions around discounting, future decisions can be predicted based on the past decisions. Seems pretty obvious, but where do you capture them? They're not captured in a model. They're not even captured in Salesforce. In Salesforce, you may capture the end decision that you ended up giving XYZ customer a certain discount, but the process of that decision is captured in Slack, it's captured in emails, it's captured in a variety of other places, and it's part structured data and part unstructured data.

So the first insight we had was that there are decision traces that human beings generate as they interact with AI applications, but those decision traces are spread across a dozen or more different interfaces. Think of Slack or email as an interface, and you can call it a dozen or more applications.

Why can't a standard graph database capture decision traces?

Garg explains why the infrastructure for capturing decision traces, the context graph, is more than a traditional graph database, and why he expects the AI model layer to become interchangeable over time.

Garg: So our first insight 18 months ago was that the best AI applications will really need to capture decision traces over time, and then figure out how to use those decision traces to improve the decisions they make in the future. And so then as we started drilling into that, we realized very quickly that the infrastructure to capture decision traces didn't exist. It wasn't a traditional graph database, and so the graph part is somewhat of a misnomer because it's not as simple as just using a graph database. It's not a traditional structured database. So you needed to find a way to capture that information, and you'd have to build an infrastructure layer.

That infrastructure layer to capture the decision traces as human beings interact with AI applications is the context graph. And our belief is that the most valuable asset over time will be the context graph, because that's what's capturing both the reality as it stands today and as it evolves. Because even these decisions are not static, they're evolving. And so the graph has to keep evolving with time, and it's built on the thesis, which I've always had, that the model layer, while hugely valuable, will over time become interchangeable and, at the extreme, somewhat of a commodity.

How is a system of record different from decision traces?

Sangani, CEO of Alation, restates Garg's idea by contrasting a system of record such as Salesforce, which stores the final outcome, with the decision traces that happen before an outcome is recorded.

Sangani: Okay. So let me state back to you what I think I heard, and then I want to get into the specifics of semantics in this. You have this construct of a system of record, Salesforce. And that thing captures what is decided upon, what is the end result of some particular process, and that process may or may not be documented. But people do things, and then at the end of the day, sale, no sale, captured in Salesforce. But to get to that point, there's all these things that have happened. There's calls, there's conversations, there's documents that float back and forth, and all of that is what you're calling a decision trace.

And those decision traces, I assume there's infinite ones of them, and for any given decision there's some things that matter and there's some things that don't matter. But that all makes sense, because that's just a description of the world. You said infrastructure layer and graph. So why does it have to be a graph? Is layer singular or are there many of these things? And when it's infrastructure, it makes it sound like it's one thing. Tell us a little bit more about how you get to that place or what place you're getting to there.

What layers make up a context graph?

Garg describes how a context graph stitches time-stamped actions into decision traces and then abstracts them into business rules that an application can apply in milliseconds.

Garg: I think once you start digging in, you realize, I mean, there's almost too many decision traces. And the decision traces have a tree structure, they have a time component. So the first thing you have to do is find a way to stitch actions together to create a decision trace, and those actions have a time component. They have a person who did what. If a VP of sales approved an exception, that's different from having to go all the way to the CEO. And so the decision trace needs to capture a lot of granularity.

Then once you have these decision traces, you have to find some way to store them and then eventually abstract. In real time, you're not going to look at the actual decision traces, so the context graph is more than one layer. It's an enduring layer, but it has sub-layers to it. The first abstraction is you capture the individual decision traces. And then at some point, you have to find a way to extract business rules from these decision traces in real time. You know, Java [CHECK AUDIO] has been the topic of the day for the last couple of weeks.

Sangani: Yeah.

Garg: Conceptually, there's an analogy. You're not going to traverse the graph every time you have to make a decision. You're going to find some way to abstract the key decisions from that graph into a set of rules. Now, the trick is how do you make those rules semi-permanent versus permanent? How do you capture the right set of rules, and how do you make it really easy, in milliseconds and with relatively low cost, to be able to implement those decisions in the application layer that you have?

Should you build one enterprise context graph or one per application?

Garg, of Foundation Capital, says context graphs will exist at both the application level and the enterprise level. For how this compares with other knowledge structures, see this comparison of knowledge graphs versus data catalogs for AI agents.

Sangani: Yeah, makes sense. And when you're saying it's going to be a durable piece of infrastructure, I want to clarify. You don't mean it's a durable singular piece of infrastructure for a single enterprise, but that, like every application has its own database and database schema, this context graph would be singular to an application, or perhaps shared across a certain set of applications? Or are you saying there's going to be one for an entire enterprise?

Garg: I actually think over time it'll be all of the above.

Sangani: Okay.

Garg: And that's just how enterprise infrastructure evolves in reality. There's no one-size-fits-all answer. Every application, every AI application that is going to the enterprise, is going to build some sort of context graph in it. And at the same time, every enterprise is going to say, "There's a host of things we do that cut across the applications we use." There's decisions an enterprise makes that aren't sales specific, they aren't marketing specific, they cut across the company. And as you advance in your automation journey, it's a little bit like the levels of self-driving.

As companies go through that journey, they will want to capture many of these decisions in a context graph that's company wide. Now, the detail you'll have in that will be very different. If you try to capture every single decision in the company, think of a company the size of Goldman Sachs or Morgan Stanley, the graph will crash, and it'll actually just be more noise than value. And so that will happen over time. And will those be captured directly? Probably not for the large part. They'll actually be abstracted away from the existing context graphs that are application specific. We weren't trying to articulate an architecture for a product. Our goal is to articulate a direction of where we think value is going to be captured, what's going to be enduring, and what will make AI valuable in a large organization.

Why is building one universal context graph up front a mistake?

Sangani, whose company builds Alation's data catalog, argues that building a single enterprise-wide context graph before building applications is misguided, and Garg agrees.

Sangani: Yeah, it's a really interesting point, because a lot of people... we are obviously in the space of catalogs. Catalogs capture a lot of knowledge, and knowledge is a form of infrastructure, although it's not quite infrastructure. It never got to that point. But a lot of people hear what you wrote and think, "Oh, well then what I need to do is go build a singular context graph for the enterprise." Which I actually find to be deeply misguided thinking.

Garg: I agree. I totally agree.

Sangani: And I don't even think, to your point, I think you've just explained something that's actually not what you intended it ever. And I think what you've articulated is, look, context is very important because it's what makes the application run. And maybe over time we get multiple applications and we refactor them so that they all work off of a single context graph, because having context in five places is probably less efficient than having it in one place. By the way, that's no different from a database to a data mart to a data warehouse. Like a data lake.

Garg: Yeah.

Sangani: Right? The same metaphor applies, but it's not like I'm going to go run around and build to this theoretical thing before I've actually done something. I've actually got to go build these apps first and foremost.

Garg: Absolutely.

Sangani: And that's really how these things will ultimately evolve.

Who owns the context layer when you switch application vendors?

Garg tells enterprise buyers to ask their application vendors who owns the context layer and what happens to it if the customer changes applications.

Garg: Absolutely. And look, even the architecture of an application-specific context graph will evolve with time, just as your database schema evolves.

Sangani: Yep.

Garg: The most important thing is to recognize that it's a layer you need and to start thinking through who owns that layer, who has access to that layer. If you're working with an application vendor, have the conversation. Let's say you decide to change applications. What's going to happen to that layer? How will you extract... Is it going to get switched off or is it going to be persistent across time?

Sangani: Yeah. Where is my knowledge, where is my truth stored, and how does it evolve and who owns it?

Garg: Exactly.

Sangani: Yeah. I think that's right.

Why can a context graph only be built inside an application?

Garg uses a thought experiment about a company with 100,000 employees and no databases to argue that a context graph needs an application to anchor it.

Garg: And by the way, the context graph can only be constructed, in my opinion, in the context of an application. Because otherwise it's a theoretical concept. So, complete thought experiment. If you had a company with 100,000 employees and no databases, and you said, "Okay, I want to build this one universal data lake that captures all the information in the company," where would you start? What would the schema of that be? It's just not a realistic thing to do.

First of all, I don't know of any company that's truly gotten to this one data lake that just eliminated all databases. I mean, when that happens, Databricks will be 100 times bigger than it is today, and I'll hopefully make a little more money, but that's not reality. I think your data infrastructure evolves over time, and you've been a big part of building that out in the various things you've done, and I think the context graphs will evolve similarly.

How do you know if your AI agent has the right context?

Sangani, CEO of Alation, argues that context depends on the application above it, and that teams can only judge whether context is right through AI agent evaluations, known as evals.

Sangani: Yeah, I couldn't agree more. And it's funny, because while I'm in a space that occupies the conceptual predecessor of this context graph, I think a lot of people are talking about context layers. This is a new term, and people talk about it. "Oh, Satyen, what do you think about a context layer?" And I'm like, I actually believe in it less than... I think, one, context is fundamentally a function of the overlying application. And the only way you know if you have the right context is if the application lights up or if it doesn't light up.

In the case of agents, I think that is particularly true. Because the only way to assess whether an agent is successful or not is an eval, and the only way to know whether your eval passed or failed is whether or not you actually have the right underpinnings of that thing. And you can do 100% of it in the model, you can do 100% of it with model plus context, or 50% with model, 50% with context. But ultimately it comes down to, do I have the right answer? And you can't figure out your context unless you know what the application is on top.

I deeply believe that. And so I think these layers... But I think the trick now is going to be that these things are going to evolve very fast. It used to be that you'd have a database schema, and then I'd refactor the schema to a new schema, and that would be the entire profession of data modeling. And I think that in the new world, in the same way that biologically you've got to refactor your... I think you're going to have to refactor your context graphs constantly.

How is a context graph different from a database schema?

Garg identifies two differences between a context graph and existing infrastructure such as databases, schemas, and catalogs: the time dimension and stitching actions across applications. For more on the distinction, see metadata versus context for AI.

Garg: I agree with that, and I do think... So while there are many, many similarities with the existing infrastructure, whether it's databases, database schemas, catalogs and so on, the two things that are different are, one, the dimension of time becomes much more important in a context graph.

Sangani: Yes. Totally.

Garg: And the second thing that is different is the ability to stitch together actions that cut across traditional applications. Because that's the power of AI. The power of AI, especially in the current generation of AI, LLM-centric automation, is that you're bridging the gap. Traditionally we had ops functions for everything. What was sales ops doing? It was filling the gaps between what salespeople did in Salesforce and your sales forecasting app. Enterprises had 10 applications within a sales organization. And data was being captured in 10 different places, and some of it was being passed back and forth with APIs, but with the constraints of APIs and the constraints of structured databases.

And the glue functions were some form of ops: sales ops, marketing ops, production ops, IT ops. Pick your favorite ops. AI is automating away that glue, first and foremost, and is making flow of data across an entire business process seamless. And as you do that, your context graph does have to cut across those underlying applications. That's what makes that context graph hard to build, but also so enduring and so persistent and so valuable. Because ultimately you can swap the underlying applications back and forth in the fullness of time. No one's swapping out Salesforce today, but over time, if the context is being captured in a layer above Salesforce, then Salesforce is nothing but a database and an API.

Will AI make fixed data schemas obsolete?

Sangani and Garg discuss how large language models (LLMs) reduce the need to impose a fixed database schema on an unstructured world, and why business processes have always outpaced the schemas built to describe them.

Sangani: Yeah, look, I couldn't agree more, and I would actually argue even more that in the old world, you'd have a process, and the process would effectively manufacture a schema to capture the salient points within that process, and that's where structured data comes from. The world is fundamentally unstructured, but I'm going to impose structure on it by building an application, and I'm going to impose structure on it by modeling it in a database schema. But I think the entire world of LLMs says, "No, no, no," you can do analysis without having to structure anything. It's just about these effectively vectors underneath that describe these relationships arbitrarily. I think also what that means is that your process is dynamic, that on some level the time dimension is important, but it's also the rate of change that's important.

Garg: Yep.

Sangani: And on some level, every question should change your schema. Every action should change your schema, because your schema is this malleable thing. It's just, what relationships matter today? I don't know.

Garg: And look, that's been the challenge for enterprise apps for decades. You started with the business process, translated that into a schema, and then by the time you started implementing it in the organization, your business process had changed. You have a team in the UK, you have a team in the US, their business process was different, and you were constantly fighting to find a way to use human beings to bridge the gap between the reality of a business process and the somewhat theoretical business process that was used to define the schema.

I think AI changes all of that, as you very rightly said, and I think that's what's going to drive a massive amount of automation and value for customers over time. Yeah. And I do think a whole bunch of new companies... I don't think it's any one company. I don't think there is a context graph company. I think multiple people will build, and existing infrastructure providers, existing data infrastructure providers like yourself, will evolve to meet the needs of the context graph as those needs emerge.

Can a data catalog keep up with what is actually true in an organization?

Sangani explains the gap between what a data catalog documents and what is actually true in an organization, and why Alation data governance has to resolve conflicting versions of the truth.

Sangani: I think that's right. To me, this question of what the catalog, obviously what we do, has always been knowledge about data. That's kind of been the essence of it, and on some level no different than Wikipedia: people come to these things and they write things down and you get this knowledge, and what's documented is what theoretically is true. But of course, people wouldn't write fast enough, or even go to this place to do the writing fast enough, in order to be able to actually capture all the knowledge. So on some level, there was always this deficiency between what was documented and what is actually true.

And I think the trick is, okay, how do we actually start learning about the organization faster, so that what is true in the organization, not for an individual, but for the organization as a group, is captured much more rapidly and learned about much more rapidly and also governed much more rapidly. Because there might be four versions of the truth. Which one's actually right is the challenge of governance. That is the fundamental challenge of governance. And so that's how we are orienting, and it's certainly the world that we see.

When should you fix the model and when should you fix the context?

Sangani introduces Hippocratic AI, a company that builds AI nurses, and asks Garg when teams should fix the model and when they should fix the context. Garg explains that Hippocratic AI fine-tunes fully open source models based on human evaluations.

Sangani: You mentioned talking till yesterday, I guess, so you did a podcast yesterday where you were interviewing our friend Munjal. And it's funny, because I was talking to him. So Munjal runs this company called Hippocratic AI. Just for background, they basically are providing effectively the equivalent of AI nurses. And when I was talking to him, he said something very interesting. He's like, "Every time we see a bug or an issue, we basically just engineer it back into the model." So in some sense, they don't have a context graph. They're just basically putting everything back into the LLM. Speak a little bit to this pattern of, when do I fix the model? When do I fix the context?

Garg: So again, it's very Hippocratic specific, where their product doesn't take over a business process. They've architected their product to do very specific things, which is be a voice nurse, and it's designed so you have a nurse that's calling up to follow up on cardiac surgeries, because that's a slightly different nurse than the nurse who follows up on patients that have diabetes, which is chronic, and they need to be reminded to, you know, go for dialysis. It's just two examples.

And what they have done is they keep looking at the evals, and they do human evals. They do a very large number of human evals in that case. And based on the corner cases that keep coming up in these human evals, they fine-tune their models. And they use completely open source. They're a hundred percent open source, and so they have a lot of flexibility on what they can do with their models. Most companies are not a hundred percent open source, and so there's limitations on what you're going to do with the models. If you're working off Anthropic or OpenAI, you don't have as much flexibility there, and so you'll have to find some way to encode changes in some abstraction, that is, the context graph.

What is the difference between automating tasks and automating processes?

Garg, of Foundation Capital, distinguishes automating narrow, high-volume tasks, as Hippocratic AI does, from automating multi-role processes such as sales. Related reading: specialized enterprise AI agents.

Garg: But more importantly, most automation companies are automating processes as against tasks. And most is a strong word. I think there are two different ways to think about automation. If you're automating tasks, and nursing happens to be one of those things which are very specific tasks, done at very high volume. There's just literally nurses that sit on the phone, calling all day, every day, following up on people. And those kinds of tasks, the improvements can be built into the model because you're customizing. You have to have very specific AI nurses for different things.

If you have an AI process that's automating sales, it's just very different. There's not even one kind of salesperson. You have salespeople, SEs, sales managers, sales VPs. They're all interacting in that application, and you're automating away bits and pieces of what all of them do. But you're not eliminating any of them. They're all going to continue to exist in some shape or form. You're automating away things they do so they can focus on higher value work, and that's a very different approach to automation from the Hippocratic approach of going very deep on one task.

Will enterprises eventually build all their knowledge into the model?

Sangani asks whether companies will eventually build all their knowledge into models. Garg answers that the result will usually be a combination, and uses Hippocratic AI's nursing protocols as an example.

Sangani: I guess the question is, okay, I'm building a system with a box around it, and effectively at the limit this can be: when do I use the model out of the box, as is, just whatever it is. And then there's another version, which is search, which is, when do I just use the knowledge by itself. Raw search in its most retrieval-oriented form is effectively I'm not using any model. I'm just going to go out there and find something that's high reliability. And so there's a question of when do I improve the documents, or when do I improve the knowledge store, or when do I improve the underlying model and encapsulate the knowledge inside of the model?

Seems like a decision question. You could do either. Either approach in theory could work. But over time, do you see the world moving more towards engineering more into this open source model? Do you see a generation of companies where building models and engineering models become the layman's task, and now all of a sudden there will be a family of tools that will allow us to build everything into the model?

Garg: So I don't believe that we will get to a world where we will build everything into the model. I think the answer will almost always be a combination of both, depending on the use case that you have. In some use cases, again, Hippocratic is a good example. Let's say a nurse that's following up on a diabetic patient at Stanford versus a nurse who follows up on a diabetic patient at UCSF, by and large, they're following a very similar protocol. Because that protocol isn't decided... I mean, it's mostly, again, there's some gray here, but it's mostly decided by a protocol that's nationwide. There's a variety of organizations that specify protocols for these nurses, and all hospitals largely follow those protocols.

When does fine-tuning a model per customer stop making sense?

Garg explains why per-customer fine-tuning becomes expensive when decisions vary by customer, and why hospital-specific decisions belong in a context layer. Alation's approach to semantic context is described under semantic model mastering.

Garg: And when you have a situation like that, where you're aiming for uniformity of how a task is performed at a national level, it's much easier to embed most of those decisions in the model. Where you have a lot of variation customer to customer, the cost of embedding those decisions in the model will be exponential, because now you're fine-tuning a model for every single customer.

Sangani: Yeah.

Garg: And then re-fine-tuning every time there's a change in the decision. So I think it'll depend on the nature of the task. But even in a situation like Hippocratic, my own hypothesis is that over time it'll be much more efficient and effective to actually abstract away some decisions in a context layer. Because a nurse at Stanford and a nurse at UCSF may do ninety-five percent of the things exactly the same way, but my guess is... Again, I'm not an expert on how nurses do their jobs. Different hospitals do make some decisions that are hospital specific, and those decisions you would abstract into a context graph as against in the model.

Sangani: Yeah, I agree with that. And I think one way of extending the analogy or the metaphor is thinking through doctors, where it's highly unlikely that you're going to build all of the knowledge into an LLM because, you talked about Jev [CHECK AUDIO], right? Which is effectively not recommending language, but probabilistic decisions. I mean, you'd get a lot of fifty/fifty-five, like surgery. It's forty-five/fifty-five. Do it, don't do it. I don't know. What does the model recommend? I don't know. Like, maybe do a surgery, but I'm fifty-one percent likely that this is going to be a good idea. That would be a hard message to receive, I think.

Sponsor message from Alation

Producer: Quick break, brought to you by Alation. Here's a stat that should stop you cold. MIT research found only 5% of enterprise AI pilots delivered ROI in production. The problem isn't your models, it's knowing where to aim them. We've put together a free guide for enterprise leaders. It's a framework for identifying which processes are ripe for AI agents, what green flags and red flags to look for, and how to build a knowledge foundation that makes AI trustworthy at scale. It's not theory. It's drawn from how Global 2000 companies are actually doing this. Download it free at alation.com/ai-guide.

How far behind is open source AI compared with closed models?

Sangani asks Garg about the economics of foundation model providers. Garg, a partner at Foundation Capital, says closed source models are one to two model cycles, about six months, ahead of open source.

Sangani: So we've talked a lot about open source models and your ability to change them. Let's talk a little bit about economics moving forward. You obviously have to study the economics of both the application companies, the open source providers, and what that looks like. Also the foundation model providers. And in conversations that you and I have, you've said, "Look, I think the foundation models are great, but they may not be the only company that lives in the world," or, as many people hypothesize, that they could be. Tell us a little bit about your thinking about how these companies could evolve, and why maybe you're not as bullish as the most bullish people about these companies.

Garg: Look, I think what both OpenAI and Anthropic have done, and more recently Google with its Gemini family, is very impressive. I think they have set the pace for innovation. They are constantly raising the ceiling. I think it's now, in my opinion, consensus that closed source models are one to two model cycles ahead, which effectively means they're six months ahead of open source. They're not two years ahead. So I think that's why. I think you'll always have an open source model that is an alternative for almost everything in six months.

Do AI applications still need the best model for most tasks?

Garg argues that models across the board are now good enough for a large share of tasks, so the cost per token makes open source models part of the mix for any application operating at scale.

Garg: The second thing I think we're finding is the models across the board, open source and closed source, are getting so good that for, pick a number, seventy-five percent of the tasks, you don't need to get any better. The model is no longer the constraint, and that's why we're seeing Jev [CHECK AUDIO] and products like Jev, which are a hundred or a thousand times faster, one thousand the cost [CHECK AUDIO]. They're not better. They're just... They're recognizing that for so many decisions that we're looking to these models to make, you don't need to be better anymore.

So I think both things will be true. And if you believe in a world where you come to the conclusion that there's only a six-month gap, by and large, between open source and closed source, in that we could debate whether it's fifty percent or two-thirds or seventy-five percent of the tasks can actually be performed by existing models and the model is no longer the constraint, then the only question is what are the economics of open source versus closed source? And as of today... Now again, closed source model providers have choices they can make around the economics. But as of today, and in recent history, the cost per token between open source and closed source is so different that I think as an application developer that's doing anything at scale, you have to have open source be part of the mix.

That doesn't mean you have to be dogmatic about it. You don't have to be a hundred percent open source. But if it's not part of the mix, then I think you're doing something wrong. And if you now take that observation to its logical conclusion, I think OpenAI and Anthropic will be very successful and very valuable companies, and there will be hundreds of other very successful and valuable companies.

Sangani: Yeah.

Where does value accrue in the AI stack?

Garg compares AI model providers to database companies such as Oracle and Sybase and estimates how value has historically split between the application layer and the infrastructure layer.

Garg: Just like when you go back to the database era, I mean, yes, Oracle's a very successful database company. Sybase, not so successful, even though it was a better database, and maybe OpenAI will be Sybase or maybe Anthropic will be Sybase. Time will tell. Or maybe Gemini will be Sybase. Who knows? I do think we will see multiple such models, and I think open source databases have done very well. And I actually believe that in the fullness of time, the value will be captured by the applications. Like databases, the database companies, Oracle is a great example, is actually a very large application company.

Sangani: This is a little bit of my ignorance, but is that true? If you were to look at the market of technology right now, how is value distributed between the application layer... I mean, do you know the answer? You may not even know the answer to this question, but how is value distributed between the application layer and the infrastructure layer, exclusive of hardware? Forget about the hardware piece of it.

Garg: If I had to guess, and this is me guessing without the data in front of me.

Sangani: Yeah.

Garg: If I had to guess, I think roughly two-thirds of the value is in, historically.

Sangani: Yeah.

Garg: If you separate the recent AI era, two-thirds of the value has been captured by the application layer and one-third by the infrastructure layer. And the reason for that is twofold. Once application companies get large enough, they're always making a build versus buy decision on the infrastructure, and that's what keeps infrastructure prices under control for application companies. Where infrastructure companies have more pricing power is with large enterprises that are doing DIY on applications, because they're not going to build their own infrastructure.

Why are OpenAI and Anthropic building their own applications?

Sangani raises the SaaS apocalypse, and Garg argues that OpenAI and Anthropic are launching their own applications because they recognize that value accrues in the application layer.

Sangani: So infrastructure companies almost, in some sense, definitionally... But on some level, again, this is my data-uninformed story. These AI companies come along, they make application building significantly cheaper on some level. And as a result, everybody's like, "Oh, SaaS, SaaS apocalypse," that's going to affect the prices of all of the existing application companies. And so in some sense, what you're saying is, yeah, that's true, but over time, yes, there might be some big infrastructure companies that will continue to exist.

By the way, I'm putting words in your mouth, so you better tell me if I'm right or wrong. And yes, these might be very valuable companies and maybe more valuable companies than the application companies are, but there's going to be tens of thousands, lots and lots of application companies that will cumulatively be more valuable than the underlying infrastructure, because that's where all of the value is. Am I...

Garg: That's what I'm saying. And now the one caveat is I actually think the new generation of infrastructure companies, OpenAI and Anthropic, actually recognize that. If they didn't believe what I'm saying, they wouldn't have made the effort to launch their own applications.

Sangani: Yeah, which they're doing super rapidly, right? They're doing legal and finance and all these other things.

Garg: They are launching their own applications. I mean, ChatGPT is an application.

Who will own consumer and enterprise AI applications?

Garg predicts that model providers such as OpenAI, Anthropic, Google, and Meta will lead consumer AI applications, while large enterprises will either build their own applications or buy them from third parties.

Garg: And I do think, if I look at the future, I think you will see the consumer applications will, for the large part, not entirely, but for the large part, be owned by the infrastructure providers today. And I'm including in that Facebook, because they're doing their own models. Now, as we are seeing, even though Facebook's model family lags OpenAI, Anthropic, and Gemini, because their application is starting to take off, that's put them back in the game. You can be a laggard in infrastructure, but if you get the right consumer application, you will pull through your own infrastructure.

And so I do think OpenAI, Anthropic, Google, and Meta are, at this point in time, well-positioned to be leading application companies for consumer applications and consumer risk [CHECK AUDIO]. And at the other end of the spectrum, for large enterprises, with the folks that you at Alation serve, I think those companies will either build their own applications or buy applications from a variety of third-party application providers.

Which software categories will AI make obsolete?

Garg names lightweight categories such as project management software as the most exposed to AI, while arguing that enterprises still want vendors to take responsibility for maintenance, bugs, and security.

Garg: There is, at the same time, a lot of mush in the middle. And I do think in the mush in the middle, time will tell how it evolves. So if you were using something like Smartsheet, and I'm picking on Smartsheet just as an example, not to suggest that it will or will not go away, or you're picking some project management software in general, those things are such lightweight applications that it may not make sense to buy an application in the future. You could just use a model with some model artifacts, or you could cook up something on your own.

So I do think some applications that are relatively lightweight, just those categories will no longer be valuable. But for the large part, enterprises want an application provider that takes the responsibility for evolving the application, that takes the responsibility for maintenance, that takes the responsibility for bugs, for security issues. They don't want to deal with a lot of that.

Why can't enterprises replace SaaS by building their own apps?

Sangani and Garg discuss why enterprise applications earn their keep in the last 20 percent of functionality, and why large enterprises complain most about usability and business value.

Sangani: Yeah, it's interesting. I mean, I don't know him, but Chamath has this thing called 80/90 [CHECK AUDIO], and the premise behind it is, 80% of the value for one tenth of the cost, or a 90% discount on the cost. The thing about it is most applications earn their keep in that last 20%. Where you build all of the moats and all of the foundation and all of the differentiation is based on that last 20%, because it's that last 20% that actually allows you to be reliable and scalable, and all of those enterprise things don't come from the first 80%. If it did, then everybody would just be building their own apps.

Garg: Totally agree. And by the way, you talk to large enterprises, and of course they want cheaper applications. Who doesn't? But the real house on fire problem for large enterprises is not that I'm paying too much for my applications. It's they're damn hard to use, they're inflexible. I bought them to get a certain amount of value in terms of business value. I'm not seeing the business value. Those are all the things that people are really complaining about, not the, "Oh, I want to shave 10% off my application cost." I mean, in the grand scheme of things, that's a rounding error.

Sangani: Yeah. I couldn't agree more. I think in our world, exactly the same thing is true. Alation is something we've developed over 12 years, 12, 14 years. There's lots of capabilities, and most of our customers don't actually know how to use any of them. Or in some cases, they can do the same thing six different ways. And so the real problem isn't, oh, this is the thing. It's how do I actually get value out of the thing that my aspiration wants me to do but my people just don't know how to do? I think that's actually a much bigger opportunity, which makes me excited about the world.

Will Salesforce and ServiceNow survive the AI transition?

Garg gives his view on Salesforce and ServiceNow, and says uncertainty over which large application vendors will survive AI will continue for a couple of years.

Garg: Exactly. And I think what will happen is that when you have a disruption that is of the scale that AI is today, many existing application companies will fail to make that transition. And so in my mind, the SaaSpocalypse, what it represented well was the fact that there is a huge amount of uncertainty associated with every single large application provider today. Because it's hard to say which ones will survive and which ones will fail. And I'm not a public market investor, but I'm assuming the best public market investors will know who that is with time, and are cherry-picking when the entire sector crashed. Now of course, it's mostly back up. But I think that uncertainty will continue for a couple of years.

Sangani: And to put you on the spot, are there any that you're particularly bullish or bearish about?

Garg: I go back and forth on Salesforce. It's a hodgepodge of things they bought. It's unclear to me how they're going to make this transition from a technology and product standpoint. It seems almost too hard, and they don't have that culture and that ethos to do that. At the same time, betting against Marc Benioff's marketing prowess is a very dangerous thing. Now, when does this all fall apart? I don't know. I think ServiceNow I'm actually pretty bullish on. I think ServiceNow has actually had, you know, the applications themselves are very lightweight, but they've always had much more of an infrastructure ethos. A few years ago, they really made an effort to build out a consistent underlying infrastructure layer across all the applications that they have.

Now, how much of that is reality versus hype, it's hard for me to say outside in, but to the extent that they do have this consistent infrastructure across applications, I think that infrastructure will be very valuable for them. I think there's a whole bunch of things that do project management. They seem like spreadsheets on steroids. I think those will all go away. I think companies that do very niche things, like I can do sales compensation calculations, really? That never deserved to be an application. That was really a spreadsheet with some security and governance on it. I think those kinds of things will go away.

What separates an AI app that learns from one that just executes?

Sangani and Garg discuss why applications that learn from decisions over time, such as automating sales discount approvals, will outlast apps that perform narrow, well-defined tasks.

Sangani: So one, there's the aspect of coordination, and so does it build coordination across the enterprise or across the team that it's intended to serve in a way that reinforces communal learning? That's a really important... Because the individual productivity problem is really all done in sort of... Well, a lot of that's going to go to the clouds [CHECK AUDIO]. But then, can I do things? I think, again, the chat interface is going to allow me to do more and more things. I mean, Jabb [CHECK AUDIO] is showing you that already. So then it's a question of, is this thing allowing me to learn? Is this basically building more learning into my organization? And I think those applications are the ones that will win.

Garg: Are you building more learning? Are you, in an almost automatic way, evolving? Is there a continuous evolution in what the application can do?

Sangani: Totally.

Garg: Is the application that I use a month from now better than the application I use today? Are there decisions I was making manually today where it had to check... So I mean, we talked about discounting in the context of sales. If the application today has to check with the VP of sales every time a discount is being applied, but it learns over a period of time, and now it starts automatically doing it. And then it starts saying, "Hey, I made this decision, FYI." At some point, the FYI may not be necessary. So that evolution of decision-making is, I think, what will create persistent and enduring applications versus applications that perform narrow, well-defined tasks. Well, narrow, well-defined tasks can all be done in a chat interface in Claude Code, Work [CHECK AUDIO].

Sangani: Yeah.

How do you build software that customers never have to learn?

Sangani, CEO of Alation, describes the challenge of helping customers absorb new capabilities as they ship. For context on Alation's direction, see the Alation Intelligence Operating System (AIOS).

Sangani: As soon as you can define the task, on some level you must have to obviate it. I think this is one of the things that, as we've evolved the company, we've struggled with: customers say to us, "You guys are developing capabilities so fast, I don't know how to absorb them." But on some level, that absorption problem is the nature of it. You have to build the application so that these things are just metabolized as and when they're released, because customers don't even have to know how to think about using them. It just happens.

Garg: Exactly.

Sangani: And that, I think, is a pretty aspirational place to get to, and I would say that's probably one of our biggest problems, figuring out how to deliver that exact experience. But I think it's absolutely possible, and I do think there's a lot of differentiation around having that learning loop, both as a software company but certainly for the customers themselves.

Garg: And I think that's what will differentiate the winners from the losers.

How much automation is safe to hand to AI?

Garg explains that application builders must decide how far to push automation based on the cost of errors, and compares a $50 expense approval with a ledger entry at a public company. Related reading: AI governance.

Garg: And there's a balance. I mean, knowing how hard to push automation.

Sangani: Yeah.

Garg: And different people will make different choices on where, how close to the line are they willing to go with safety and trust issues? How close to the line are they willing to go with the risk that automation, when it fails, makes mistakes? Making a mistake on approving a $50 expense statement is very different from making a mistake on approving a ledger entry in a balance sheet for a public company that's going to get audited. And those are judgment calls that the application developers will have to make. And folks who are building an app like finance, I just use finance as the example. If you're building a finance application for mom-and-pop companies, the trade-offs that you will embed into your application will be very different from if you're building a finance automation application for public companies.

Closing the conversation with Ashu Garg

Sangani and Garg close the conversation and joke about continuing it over a glass of wine in a future episode.

Sangani: Yeah. Well, I think we're at time. I love talking to you, and it's because of exactly this conversation, and we do it in private, and this time we got to do it in public. So thank you for being so generous with your time and coming on the show. We'll have to get you back on.

Garg: Thank you so much. This was so much fun. I completely forgot we were doing a podcast. Next time we should just do it with two glasses of wine and...

Sangani: We will do that. So two glasses of wine, we'll do it here. All right. Take care. Ashu, thank you so much.

Garg: Thank you so much for having me on the show.

Sangani: Awesome. Till next time.

Episode recap: where AI value goes when models become interchangeable

The producer recaps the episode's argument that value shifts from the model layer to the context graph, and describes the SaaSpocalypse as real but narrow. A sponsor message from Alation follows.

Producer: Everyone wants to know which AI models are winning. Ashu says that's the wrong question. The models are getting good enough and fast enough that for most tasks, the model is no longer the bottleneck. Open source is only six months behind closed source, and the gap is closing. The question that matters is where the value goes when that happens. Ashu's answer, not to the model layer, but to the context graph, the infrastructure that captures how people actually make decisions and makes every app smarter over time.

That's what separates an app that learns from one that just executes. The SaaSpocalypse is real but narrow. Lightweight apps will disappear, but the real enterprise problem was never price. It was that software is hard to use, inflexible, and doesn't deliver enough value. The companies that fix that, the ones that build learning into their products, will be the ones worth backing. Thanks for tuning into AI Radicals. See you next time.

Sponsor message from Alation

Producer: A word from our sponsor, Alation. Here's a stat that should give every healthcare data leader pause. 80% of organizations have deployed AI in some form, but only 11% have achieved it at scale. The bottleneck isn't models or compute, it's the data underneath them. In healthcare, that's especially high stakes. Ungoverned data doesn't just slow down your AI program, it can introduce bias, expose patient information, and produce outputs clinicians can't trust or act on.

Alation's new white paper, produced with Healthcare Information and Management Systems Society, pulls from real implementations from Children's Hospital of Philadelphia. It covers what AI-ready healthcare data actually looks like, how to build data products that accelerate clinical AI, how to govern for HIPAA compliance without killing innovation, and a practical roadmap for going from pilot to production. If you're building AI strategy in a health system, this is one to read end to end. It's free at alation.com/ai-healthcare.