Inference Engineering for Coding Agents

Coding agents are a dominant use case for LLMs, but they tend to use generalized inference setups even at large scale. Like any optimization problem, in inference you can get better results with better constraints. This talk shares specific techniques that you can apply when building an inference service for exclusive use in coding agents, including targeted quantization and pruning schemes, domain-specific post-training, KV cache management around compaction events, subagent session affinity in a world of autoscaling and load balancing, and code-specific speculative decoding techniques across both common model-based approaches (DFlash, DSpark, EAGLE) and deterministic speculation using n-grams.