AI Agent Evaluation: Why Zero Failures Can Be Misleading

The early Iboga report put recurring failures at 0 out of 22 checkpoints under a correction contract, compared with 9 out of 22 under reflection. That comparison was later retracted. The correction arm never produced a label the detector needed, so the detector could not register a recurrence there.

The published zero described a blind spot in the detector, even though it appeared in a comparison of agent behavior.

In AI agent evaluation, “no failures detected” is useful only if the evaluator could detect the failure in the first place. Iboga makes the problem concrete: the intervention changed the information available to the instrument used to judge it.

What Iboga’s zero meant

Iboga studies how guidance at a checkpoint affects a coding agent’s subsequent work. In the Phase 2A pilot, one condition used reflection and another supplied a machine-checkable correction contract.

The recurrence detector depended on a failure code derived from a confession. Reflection produced that artifact. The correction condition did not run the confession step, leaving the code null by design. A repeated failure in that condition could therefore pass through the detector uncounted.

The June 5, 2026 erratum withdrew the comparison. A retrospective check using a rule-based label available in both arms found recurrence at approximately 12 out of 22 checkpoints in each. The corrected findings pack records the change.

The recheck does not prove that the interventions are equivalent. This was exploratory work, and changing the measurement retrospectively does not turn it into a confirmatory experiment. What it does establish is that the original zero could not support the conclusion attached to it.

A number can hide a failed measurement

Two other issues in the close-out are worth keeping alongside the recurrence error.

Solve rate was null because the parser expected a different evaluation-file structure. It looked for a flat tests_passed field, while the SlopCodeBench v1 artifacts supplied pass and total counts alongside a map of test suites. The repair was checked against 113 historical evaluation files and used to backfill existing trajectories.

The erosion audit found 11 values of 0.0 that represented snapshots the tool could not parse. Those were missing measurements, not observations of no erosion.

These failures need different fixes. One detector lacked an input in a particular condition. Another parser expected the wrong schema. A third path encoded a parse failure as a plausible number. A chart could conceal all three unless the underlying records preserved what happened.

Keep the value separate from its status

For a future evaluation, I would want a record like this:

{
  "recurrence": null,
  "measurement_status": "unavailable",
  "reason": "required_failure_label_missing"
}

This is a proposed schema, not the original Iboga format. Its purpose is to keep an unavailable result from becoming a zero during aggregation.

Three statuses cover different situations:

• Observed: the metric applied, the required evidence was present, and scoring succeeded. Zero can be a valid value here.
• Unavailable: the case should have been measurable, but a missing input or broken measurement path prevented scoring.
• Not applicable: the metric’s definition excludes the case. For example, a recurrence measure that requires a prior checkpoint may not apply to the first checkpoint.

I would keep the reason as well as the status. A missing label, a parser exception, and a timeout may all prevent scoring, but they point to different parts of the system.

Audit coverage before comparing scores

Start with the event you want to count. What counts as a repeated failure? Is the unit a task, checkpoint, or complete agent run? Which earlier observations does the detector need? Write those choices down before calculating a rate.

Then trace the prerequisites through each condition. Check whether every arm produces the labels, files, and state transitions the evaluator expects. A field can be optional in the logging schema and essential to the analysis.

Report coverage beside the result. In a hypothetical arm with 22 eligible checkpoints, suppose only 14 have the required label. A detector finding no recurrence among those 14 does not justify reporting zero across all 22. State what happened in the measurable subset and account for the eight unavailable cases.

That still leaves a harder question: why are those eight missing? If missingness depends on the condition or the outcome, comparing only the measurable subsets can remain biased. A coverage percentage makes the gap visible; it does not repair it.

Finally, give the detector a few deliberately constructed cases before trusting an aggregate. I would include a repeated failure with its label, a non-repeated failure with its label, and the same repeated failure with the label removed. The last case should become unavailable. If it silently passes, the test has exposed a blind spot.

These are checks readers can adapt. No new detector test was run for this article.

Can a retrospective recheck rescue the experiment?

Sometimes it can recover a useful observation. In Iboga, a shared rule-based label allowed the recurrence comparison to be reconsidered. That was enough to withdraw the earlier interpretation. It did not settle what the intervention would do in a clean, matched rerun.

The same care applies to the outcome we choose. The pilot’s retrospective contract-compliance scores were 0.885 for correction across 16 scored checkpoints and 0.881 for reflection-derived contracts across 14. Those figures describe compliance in the scored records; they do not establish one method’s superiority.

An agent can follow a contract while repeating a mistake. It can avoid a repeated mistake while producing poor code. Compliance, recurrence, and code quality need their own evidence. The coding-agent reliability overview sets out the project’s broader scope.

Other researchers have documented reasons to inspect the apparatus around a score. Anthropic’s agent evaluation guide treats graders, traces, outcomes, and the harness as parts of the evaluation. Its analysis of infrastructure noise examines how runtime conditions affect coding benchmark results. Iboga’s detector error is a different failure mode, but it belongs in that wider account of what an agent score actually measures.

For our work at Walking Stick Labs, this matters whenever an agent acts through an environment we built. If a move fails, I want to be able to distinguish the agent’s choice, the interface’s handling of it, and the evaluator’s ability to observe the result.

Reading the Failure Logs describes the trace-inspection side of that work. The question I would put beside any impressive zero is more immediate: show me a failure this detector is capable of counting.