AP46 · No evals, only fixes
Family: Verification and evaluation
Moderate evidence
Original names: no evals, reactive loops, flying blind
Every complaint gets a fix, and nobody knows whether the agent is getting better or worse.
Definition
The agent evolves complaint by complaint, each fixed in turn, with no evaluation set. Prompt, tool or model changes ship to production with no before-and-after measurement.
Observable symptoms
- Every fix breaks something else.
- Nobody can tell whether a new model does better or worse than the old one.
- Decisions, such as switching models or granting more autonomy, are made without data.
Why it hurts
Without a baseline, a real regression cannot be told apart from noise. Each local fix moves the problem elsewhere. Patterns that rely on measurement, such as the evaluator or the model cascade, can no longer be applied.
A defensible choice when…
- A throwaway prototype, long enough to find the first failure modes that will supply the first evaluation tasks.
Figures
- According to Anthropic, teams without evals get bogged down in reactive loops: they fix one failure, create another, and can no longer tell real regressions from noise. Suggested starting point: 20 to 50 simple tasks drawn from real failures.
Remedies from the catalogue
- B5 · Generator-evaluator — An automatic evaluator reruns the task set on every prompt, tool or model change.
- F3 · Model cascade and routing — Prototype with the most capable model to set a baseline, then move down to cheaper models while measuring the gap.
Measures outside the catalogue
- A set of 20 to 50 tasks drawn from real failures, extended after every incident.
- Blocking regression evals in continuous integration.
- Error analysis on real traces before writing the evaluators.
Related anti-patterns
Sources
- Demystifying evals for AI agents — Anthropic, 2026-01-09
Content reviewed on 2026-10-05