Where the time goes in an agent turn
Serving an open 8B model with vLLM and measuring each agent turn with and without prefix caching, against a hosted API.
Builder · 2026–
What it is
An open 8B model served with vLLM, with every agent turn measured with and without prefix caching and compared against a hosted API.
How it's built
- Prefill versus decode. Each turn is split into prefill and decode, reporting time to first token and inter-token latency.
- KV-cache math. Working out how much memory the cache needs for a given context length and batch.
- Prefix caching. Agent turns share long prefixes (system prompt, tool definitions, earlier turns), so caching them is the main lever.
- Structured output. Constrained decoding for tool-call arguments.