Coding Agents and How to Tame Them

Coding agents are remarkably capable one minute and burning 50,000 tokens in a broken loop the next. Over the past few months at Google DeepMind, I've worked with Kaggle to audit over 50 coding benchmarks trace by trace and build a platform to evaluate frontier agents at scale. In this short 18-minute talk, we'll cover the weird failure modes we caught in the wild, the lessons we learned building air-gapped sandboxes, and how to take those same lessons to design better harnesses, verifiers, and coding environments for your own agents.