When to Replace data cleanup With an Agent — and When Not To
A field report on evals-first development and what it changes for founders.
For most of the last year, the conversation around fine-tuned distillation has been louder than the evidence. That is starting to change.
Teams that win with structured outputs tend to share a habit: they write the evals before they write the prompts. Everything else follows from that.
The skeptical read is that we are watching a feature, not a platform. The optimistic read is that live web browsing is exactly the kind of feature that becomes a platform when nobody is paying attention.
Inside Ramp, the rollout looked less like a moonshot and more like a slow migration. A pilot, a champion, a quiet expansion, a budget line.
What Mistral actually shipped with Gemini 3 Pro is less a single capability and more a cluster of small, compounding improvements — the kind that only show up when you put a real workflow on top.
The cost curve matters here. Qwen 3 is roughly an order of magnitude cheaper per token than the equivalent model 18 months ago, and that changes which problems are worth automating at all.
What xAI actually shipped with Gemini 3 Pro is less a single capability and more a cluster of small, compounding improvements — the kind that only show up when you put a real workflow on top.
Eval harnesses, once an afterthought, are becoming the most important piece of code in many AI projects. Duolingo's team treats theirs the way an SRE team treats a runbook.
None of this guarantees a clean story. DeepSeek could ship a model next month that rearranges the assumptions in this piece. But the direction of travel, for now, is clear enough to plan around.