Enterprise AI evaluation shifts to cohort analysis, cheaper models

Enterprise AI developers are overhauling how they test agent performance, moving away from grading individual conversations toward comparing broad user groups against baseline metrics. Executives from LangChain, Conviva and CoreWeave detailed the shift at VB Transform 2026, warning that traditional testing methods mask fundamental product flaws.

The core issue is that a single AI interaction can score perfectly in isolation while hiding systemic failures. Hui Zhang, CTO of Conviva, highlighted a retail scenario where a shoe recommendation interaction looked flawless individually. However, cohort data revealed a clarification ratio three times higher than the baseline, while purchase completion outside the conversation was five times higher. These metrics, invisible in single-trace testing, point directly to category-specific bugs that erode enterprise ROI.

This blind spot is creating operational bottlenecks. Harrison Chase, CEO of LangChain, noted that teams are falling into "eval paralysis," delaying product launches to build exhaustive test suites that still miss real-world failures. He framed evaluation criteria as a living product specification rather than a static checklist. "Evals are like the new PRD," Chase said. "They define what your agent should and shouldn't do."

To resolve this, CoreWeave's Emmanuel Turlay advocated for broad, always-on monitoring to catch real failures, followed by targeted offline testing. Once a failure pattern is identified, companies are significantly cutting costs by downsizing the models used for judging. Turlay recommended proving a task is solvable with a top-tier model before downsizing to cheaper, open-source alternatives for basic classification.

LangChain applied this directly by fine-tuning a Qwen model to detect "perceived error" when users believe an agent made a mistake. By combining hand labeling with distillation, the company achieved performance matching Anthropic's Claude Sonnet at a fraction of the expense. "Same as Sonnet, for, depending on how we served it, either 10 to 100x cost reduction," Chase said. He also noted that not all guardrails require AI, pointing to Anthropic's use of standard regexes in its Claude Code product.

Despite cheaper automated testing, human oversight remains a hard requirement for enterprise deployment due to legal and financial liability. Turlay, drawing on his experience in autonomous vehicles, emphasized that accountability still requires a human sign-off before models go live in high-stakes sectors like finance and healthcare. "Before we can remove a human to say, I endorse this and I take responsibility legally for it, it's going to be a while before agents can do that on their own," he said.