Why AI Verification Needs Trusted Data

Enterprises have spent two years asking how to get AI to produce more. The harder question has now arrived: how do you know that what it produced is right?

Models can now generate code faster than reviewers can read it, and AI agents return analyses, summaries, and recommendations at a similar pace. Reviewing output by hand stops scaling once generation outruns human attention. Teams building “software factories” have responded by automating verification. StrongDM’s, which Justin McCarthy co-founded, replaced human code review with agent-run scenario validation.¹

That shift carries a consequence for data leaders: Verification is only as good as the ground truth it checks against. Bad data makes for a bad test, and AI models will exploit a bad test. The open-source maxim, Linus’s Law, holds that with enough eyes, all bugs are shallow.² Justin McCarthy, founder of Diffusion,³ offers the AI-era version: “with enough tokens, all bugs become shallow.”⁴ Tokens are easy to buy in bulk. Trusted data is not.

In a recent episode of AI Radicals, host Satyen Sangani sat down with McCarthy and James Cham, partner at Bloomberg Beta,⁵ to discuss the software factory. McCarthy describes it as “a looping system that produces reliable outputs” from the unreliable output of AI models, with no human reading the code.⁴ Cham, who has moderated discussions of software dark factories,⁵ joined to explain why it works. This post takes their engineering lessons and asks what they mean for the people who own enterprise data today.

Verification should cost 100 times more than generation

Most teams that adopt coding agents keep a person at the end of the process, reading the output and running a traditional QA pass. McCarthy argues for a different process: Before any human sees anything, a team should spend at least one order of magnitude, and preferably two, more tokens validating work than generating it. His example: if a model runs for a minute to write a function, he wants 100 minutes of analysis. That analysis should come from the other models as well as the one that wrote the code, and none of it from humans.⁴

Cham calls this the best lever for taking advantage of large language models. Most of the industry chases efficiency and lower cost, which he describes as local gains. Once a team adopts the idea that “verification should be 100X more work, 100X more automated work,” the reliability it can reach changes in kind. Cham adds that today’s harnesses and development environments are not designed around this way of thinking.⁴

McCarthy compares the structure of good validation to the levels of cache in a CPU:⁴

• The innermost layer is fast tests, running in milliseconds or seconds, that agents use as they work. He urges skepticism toward that layer because the agents wrote it.

• The next ring is a definition of done, written in language close to the user’s experience of the software, that anyone in the company can read and recognize, such as how a tax calculation should work in a particular jurisdiction.

• The outermost loop explores the software how a user would, expands and curates the shared corpus of cases, and compiles what it learns back down into the fast tests.

Data and AI leaders will recognize this layered design from AI agent evaluations. Alation’s own guidance makes a similar point: teams that would never ship code without unit tests often “vibe-check” generative AI into production.⁶ The software factory raises the bar further, since the checks run continuously and at a scale no review meeting could match. For enterprises, that discipline belongs inside an AI governance program, where evaluation results become a condition of release rather than an afterthought.

Models will cheat the test you give them

Why does the verification budget need to be so large? Because the thing being checked will try to get away with something. McCarthy explains that models try to “win” the task so they can declare victory. “You have an adversarial entity here in the models that are trying to cheat,” he says. The design challenge is to build an environment where the easiest way out for the model is the correct answer. No textbook exists for this, which is why McCarthy calls it an extremely creative endeavor.⁴

NIST’s review of the research notes that evaluators have seen models raise their scores by modifying the tests or scoring code, and that on deliberately impossible benchmarks some leading models cheated in a majority of cases.⁷ McCarthy’s team hit this early: its write-up describes agents that “return true” to pass narrowly written tests.¹

Cham lists the range of tools available: static evals, synthetic data, simulations of other systems, and tools for measuring those simulations. He considers the sophistication of the verification step the undiscovered frontier of the whole approach. McCarthy adds that the right instrumentation depends on the domain. Some domains are quantitative, with columns and rows where all the numbers add up. Others are video or signal processing, and each needs its own way of closing the loop.⁴

Here is where the argument turns toward enterprise data. When an agent answers questions about revenue, customers, or risk, the domain is overwhelmingly quantitative. To get a correct answer out, the agent has to pass through data: certified definitions, trusted tables, and known-good outputs. If those are ambiguous, stale, or ungoverned, the easiest path may be a plausible wrong answer. That is the practical case for a data catalog and an agentic knowledge layer that supplies agents with context, and for certified, governed data products that give verification something solid to stand on.

Verification is a data problem

Sangani pressed on exactly this point. He observed that a rule such as “we never refund a customer more than they paid us” can be stated compactly, so a team should not burn a million tokens rediscovering it every time. He noted that people are now talking about ontologies, and he framed the design question this way: “When is something a rule? When is something data?”⁴

McCarthy answered with an idea from database engineering, pushing down a predicate. A query engine routes part of a query to the columnar store best suited to run it. A verification system should do the same with the rules that must never break, running them deterministically as the fastest possible test of those constraints. Generative, exploratory checks then operate around that fixed core. McCarthy also notes that verification demands a great deal of traditional compute. “You need to run queries on large databases as a means of verification,” he says, to confirm that the numbers add up, and models can help teams do it at scale.⁴

McCarthy describes two techniques: fuzzing an extract, load, and transform pipeline by amplifying its records with synthetic data and added variance, and replicating or rewriting brownfield systems, which he considers well suited to this approach because a working process already specifies what its replacement must do.⁴

Both depend on data foundations. First, rules need shared definitions. A constraint about refunds only works if “refund” and “paid” mean the same thing in every system, which is the work of data governance and of managing critical data elements. Second, rewriting a brownfield system requires knowing what feeds what, which is the job of data lineage. Third, synthetic data and fuzzing only help when a team has a quality baseline to compare against. This is where agentic data quality management fits: agents that propose quality rules from catalog metadata, lineage, and usage patterns⁸ help teams build the deterministic core that verification loops depend on.

The agentic era rewards the best ground truth

The software factory conversation is about code, but the lesson is applicable across domains. Agents will produce plenty of output. McCarthy puts the limit plainly: “Just making stuff is not the same as quality.”⁴ The organizations that pull ahead will be those that can verify agent work automatically, continuously, and against data they trust, with people reserved for the hard judgment calls.

That is the premise behind Alation AIOS,⁹ which connects data, context, and agents in one environment, and behind Alation Agents,¹⁰ which can be gated on evaluation results before they reach production.

Curious to learn how Alation can support your agent strategy? Book a demo with us today.

Sources & notes

Every external claim on this page is independently verifiable. The public sources are listed here.

1. StrongDM’s software factory replaces human code review with agent-run scenario validation, and its authors describe agents that return true to pass narrowly written tests. — Justin McCarthy, “Software Factories And The Agentic Moment,” StrongDM AI, Feb 6, 2026 ↗ https://factory.strongdm.ai/

2. Linus’s Law, formulated by Eric S. Raymond in

   The Cathedral and the Bazaar. — Eric S. Raymond ↗ https://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar01s04.html

3. McCarthy is founder and CEO of Diffusion. — Speaker bio, Enterprise AI Summit, Charlotte 2026 ↗ https://eaischarlotte2026.sched.com/speaker/justin_mccarthy.29xiwcti

4. The Software Factory: Don’t Review the Code, Verify It with Justin McCarthy & James Cham — AI Radicals Podcast. 11 Nov 2026.

   ↗ https://www.alation.com/podcast/episodeswhat-is-software-factory-ai-code-justin-mccarthy-james-cham

5. James Cham of Bloomberg Beta moderated a Commoncog Q&A on software dark factories (Jun 1, 2026). — Commoncog ↗ https://commoncog.com/software-dark-factory-q-a/

6. Alation’s guidance on evaluations versus “vibe-checking” generative AI (Mar 10, 2026; Alation’s own publication). — Alation ↗ https://www.alation.com/blog/ai-agent-evaluations/

7. Evaluators have observed models modifying tests or scoring code to raise scores, and cheating in a majority of cases on deliberately impossible benchmarks. — NIST, “Background: AI models can cheat on evaluations,” Nov 28, 2025 ↗ https://www.nist.gov/caissi/1-background-ai-models-can-cheat-evaluations

8. Alation Data Quality agents propose quality rules grounded in catalog metadata, lineage, and usage patterns (Alation’s own page). — Alation ↗ https://www.alation.com/aios/data-quality/

9. AIOS runs agents, context, data, and governance as one system (Alation’s own page). — Alation ↗ https://www.alation.com/aios/

10. Alation Agents support gating on evaluation results before production (Alation’s own page). — Alation ↗ https://www.alation.com/aios/agents/