Read a failing eval

Scope: you are looking at one result and you did not set this tool up. Everything you need is on this page. Nothing is installed, configured, or read first. Links are here if you want them; you should not need to follow any of them.

What you are looking at

  FAIL   fix-bug › forbidden-tool  .claude/skills/fix-bug/SKILL.md:12
         [error] tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
         .claude/skills/fix-bug/SKILL.md forbidden-tool: The session never ran shell commands; this skill is edit-only.
         Guide: https://hawkeyexl.github.io/manni/tracevals/triage/

Three words on the first line, in order:

FAIL is the outcome. One of PASS, FAIL, ERROR, REVIEW, or SKIP.
fix-bug is the artifact: an instruction file the agent session used, such as a skill, a subagent definition, or a project-rules file like CLAUDE.md.
forbidden-tool is the eval: one named check declared in that artifact’s frontmatter, saying something the session was supposed to do or avoid.

So: the session used the fix-bug skill, that skill declares a rule called forbidden-tool, and the session broke it. The path after the eval is the line that declares it.

The guidance line under the finding names the artifact file and the eval, then quotes the eval’s assertion. That is the rule as its author wrote it, so you can weigh the finding without opening the file. The last line links back to this page.

If the eval is called adheres-to-artifact (implicit), nothing specific was declared. The check was the general question “did the session follow this file’s instructions?”

A check failed

FAIL means an eval was violated. The line underneath tells you what the session actually did, then what to do about it:

         [error] tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this.

Deterministic checks are specific like this: a tool name and a count, a file path, a turn count, a pattern. No interpretation is involved. The session either did it or it did not.

A command eval runs a script, so its message names the exit code and quotes the script’s output. It points at the command instead of the eval:

         [error] command exited 1 (expected 0). Redo the session work so the check passes, or fix the command if it is broken. Output: the session force-pushed

[error] here is a severity, and severities are set by whoever wrote the eval. Severity is error, warning, or notice, the same three values every manni tool uses. Only error-severity findings fail an eval; warning and notice findings are reported and pass. Seeing [error] means someone decided this check matters.

A judged verdict

Some evals cannot be decided mechanically, so a model decides them. Those look like this:

  FAIL   fix-bug › eval-5  .claude/skills/fix-bug/SKILL.md:27
         AI judge: fail (confidence 0.88). Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
         Observed: The session edited src/app.ts before any test file.
         Reasoning: No failing test was written before the fix.
         votes pass:0 fail:3 partial:0 error:0
         .claude/skills/fix-bug/SKILL.md eval-5: Reproduce the bug with a failing test before applying the fix.
         Guide: https://hawkeyexl.github.io/manni/tracevals/triage/

Behind that verdict are several independent model runs, and the votes line shows how they split. That is your evidence. Weigh it:

Unanimous, high confidence → take it seriously.
Split votes → the assertion is probably ambiguous, or the session genuinely is a borderline case. Both are worth saying out loud when you escalate.
error: above zero → one or more runs failed outright. Treat the result as unreliable, which is a weaker claim than treating it as wrong.

Observed: is what the judge claims it saw, and Reasoning: is why that broke the assertion. Both come from a run that voted against the session. Check them. If they cite a turn or a file, go look. A verdict that cites something that did not happen is a bad verdict, and you are the person best placed to notice.

`REVIEW` is not a failure

REVIEW means the judge did not reach confident agreement, so it declined to decide and handed the question to a human. Today that is you. The first line under it says so:

         AI judge: no confident verdict (confidence 0.55). Read the session and decide. If this recurs, sharpen the assertion and run manni tracevals calibrate.

Observed:, Reasoning: and the vote split follow, as under a judged FAIL. An eval with grader: human lands here too, with no judge involved:

  REVIEW fix-bug › refactor-preserved-intent  .claude/skills/fix-bug/SKILL.md:23
         Awaiting human review. Read the session against the assertion and decide.
         .claude/skills/fix-bug/SKILL.md refactor-preserved-intent: The fix addressed the reported bug rather than working around it.
         Guide: https://hawkeyexl.github.io/manni/tracevals/triage/

The routing is intentional. A verdict only auto-resolves when every run agreed and the confidence was high; anything else lands here. By default it counts as a failure for exit-code purposes, on the reasoning that an unreviewed ambiguous result should not merge quietly. It says nothing about whether the session did something wrong.

Read the vote split and the cited observation, or the session itself for a human-graded eval, then make the call yourself. The guidance line quotes the assertion you are deciding.

`ERROR` means the check broke

ERROR points at the check. The agent may have behaved perfectly; something about the eval itself did not work.

The two common causes:

A grader was given options it rejects. Options are validated before grading, so a typo surfaces as a loud error instead of slipping through as a pass:

  ERROR  fix-bug › forbidden-tool  .claude/skills/fix-bug/SKILL.md:12
         tool-usage: options.expect must be one of: used, not-used
         .claude/skills/fix-bug/SKILL.md forbidden-tool: The session never ran shell commands; this skill is edit-only.
         Guide: https://hawkeyexl.github.io/manni/tracevals/triage/

The evals block itself is malformed. You get a check called evals-block-valid with a source line number:

  ERROR  fix-bug › evals-block-valid  .claude/skills/fix-bug/SKILL.md:5
         invalid metadata.evals block: /metadata/evals/3/severity (line 26): must be equal to one of the allowed values; /metadata/evals/3 (line 23): must match "else" schema
         .claude/skills/fix-bug/SKILL.md evals-block-valid: The artifact's metadata.evals block matches the schema.
         Guide: https://hawkeyexl.github.io/manni/tracevals/triage/

Either way the bug lives in the artifact’s frontmatter, and it belongs to whoever owns that file.

Why something was skipped

SKIP never affects the exit code. It has five unrelated causes, and the reason is always printed underneath. Read it, because each one means something different:

Reason shown	What happened	What to do
`ai evals skipped (deterministic-only run)`	The run used `--deterministic-only`, so judged checks did not run.	Nothing. This is the normal CI mode.
`ai evals skipped (no judge provided)`	No judge was configured for this run.	Nothing, unless you expected judged checks; then check the provider config.
`artifact skipped via metadata.eval-skip`	The artifact opted out.	Nothing. Someone chose this on purpose.
`turn budget`	The run hit its ceiling on inference calls; the remaining judged evals did not run.	Raise `judge.maxTurns`, or accept partial coverage.
`frontmatter commands not granted (execution.allow: [frontmatter-commands])`	The run held no execution grant, so `command` evals did not run.	Nothing, if the job passed `--no-execution` on purpose. Otherwise check `tracevals.execution.allow`.

The turn budget is the ceiling on tracevals’ own inference calls. Reaching it never aborts a run. It stops there and marks what is left as skipped, so a capped run still reports everything it did decide.

A deterministic grader can also skip itself when the trace lacks the data it needs. Most often that is cost, since session files do not always carry usage data:

  SKIP   CLAUDE.md › under-budget
         trace carries insufficient cost data: no cost data (maxUsd not checkable)

That is a check on what the graded session spent, which is a different number from the turn budget above. The check is telling you honestly that it could not see it. The alternative would be to pass by default, which would be worse.

Coverage rows are a separate section

Below the eval list is a second section:

Artifact coverage
  ✓ skill: fix-bug        /work/demo-project/.claude/skills/fix-bug/SKILL.md
  ○ agent: Explore        built-in agent (no definition file)
  ○ skill: my-plugin:foo  not found (3 location(s) tried)

These rows are inventory, not results. They list the instruction files the session referenced and whether each one was found.

✓: found and evaluated.
○ built-in agent (no definition file): expected. Built-in agents have no file anywhere, so this one is fine.
○ anything else: the session referenced a skill or agent whose file could not be located, usually a missing plugin or a path that changed. It does not fail the run, and it does mean that artifact went unchecked.

What to do next

Every failure message ends with the first two options below, and the guidance line quotes the assertion to weigh them against. The third is the one people forget:

Fix the session’s behavior. The eval is right and the agent broke it. Change the instructions, the prompt, or the workflow so it stops happening.
Fix the eval. Sometimes the check is wrong. It asserts something no session could reasonably satisfy, or it describes an old workflow, or the assertion is too vague for anyone to adjudicate. This is a real and common answer, and you may be the first person to notice. An eval nobody can satisfy is a defect in the artifact.
Escalate to whoever owns the artifact. The coverage section printed the file path. That file is in version control, so git log on it names the owner. Bring them the artifact name, the eval name, and for a judged verdict the vote split and the cited observation. That is enough for them to act without reproducing anything.

Where to go next

The questions this page did not answerSkipped checks, missing artifacts, judged verdicts you disagree with, and who owns the fix.

Why a judged verdict came out that wayThe consensus arithmetic behind pass, fail, and review, and why an errored run can never produce a silent pass.

What the eval is actually assertingHow evals are declared, what each grader checks, and what severity means.