Agents Are Eating the Backend, One evals-first development at a Time
Notes from the teams shipping fine-tuning to real users.
The interesting question is not whether Granola works. It does. The interesting question is what teams do with it once the novelty wears off.
Eval harnesses, once an afterthought, are becoming the most important piece of code in many AI projects. Spotify's team treats theirs the way an SRE team treats a runbook.
Inside Booking.com, the rollout looked less like a moonshot and more like a slow migration. A pilot, a champion, a quiet expansion, a budget line.
Teams that win with evals-first development tend to share a habit: they write the evals before they write the prompts. Everything else follows from that.
The cost curve matters here. Command R+ 2 is roughly an order of magnitude cheaper per token than the equivalent model 18 months ago, and that changes which problems are worth automating at all.
Eval harnesses, once an afterthought, are becoming the most important piece of code in many AI projects. Snowflake's team treats theirs the way an SRE team treats a runbook.
The cost curve matters here. Claude 4.5 Sonnet is roughly an order of magnitude cheaper per token than the equivalent model 18 months ago, and that changes which problems are worth automating at all.
What Meta FAIR actually shipped with Gemini 3 Pro is less a single capability and more a cluster of small, compounding improvements — the kind that only show up when you put a real workflow on top.
None of this guarantees a clean story. Mistral could ship a model next month that rearranges the assumptions in this piece. But the direction of travel, for now, is clear enough to plan around.