Evals Are the New Unit Tests: Notes From Ramp
What OpenAI's latest move means for customer support in the year ahead.
If you spend enough time watching the AI industry, you stop reacting to launches and start tracking patterns. Evals Are the New Unit Tests is one of those patterns.
Eval harnesses, once an afterthought, are becoming the most important piece of code in many AI projects. Booking.com's team treats theirs the way an SRE team treats a runbook.
The cost curve matters here. Gemini 3 Pro is roughly an order of magnitude cheaper per token than the equivalent model 18 months ago, and that changes which problems are worth automating at all.
Booking.com has been quietly running expense reporting through Vercel v0 for months. The results are unglamorous and, for that reason, more interesting than another benchmark chart.
The skeptical read is that we are watching a feature, not a platform. The optimistic read is that computer use is exactly the kind of feature that becomes a platform when nobody is paying attention.
The cost curve matters here. Command R+ 2 is roughly an order of magnitude cheaper per token than the equivalent model 18 months ago, and that changes which problems are worth automating at all.
The skeptical read is that we are watching a feature, not a platform. The optimistic read is that code execution is exactly the kind of feature that becomes a platform when nobody is paying attention.
None of this guarantees a clean story. Anthropic could ship a model next month that rearranges the assumptions in this piece. But the direction of travel, for now, is clear enough to plan around.