Evals and tracing for agents
An eval suite and OpenTelemetry traces for the harness, used to measure which harness components help which model.
Builder · 2026–
What it is
An eval suite and tracing layer for the agent harness. It answers one question: which harness components actually help which model.
How it's built
- Metrics. Task success, tool-call accuracy, and cost per task.
- Tracing. OpenTelemetry traces that follow the GenAI conventions, covering every model call and every tool call.
- Ablations. Planning on and off, typed tools versus bash-only, context management at 32k and 128k windows, and a small model versus a large one.
- Turn labels. Each turn is labeled localize, fix, or verify, so failures can be placed in the loop.
- Tasks. A slice of SWE-Bench Verified or Terminal-Bench.
- Statistics. A paired McNemar test compares configurations on the same tasks.
Write-up
"How I debug a nondeterministic agent."