Exit codes & reports

Scope: the contract a pipeline depends on, meaning what the process returns and what it can leave behind. Installing the gate is Run it in CI; the full field-by-field report structure is in the reference.

The exit-code contract

Code	Meaning	What it tells you
0	every eval passed or was skipped	Nothing to do.
1	some eval failed, errored, or needs review	The agent session did something worth looking at. In a batch, also a corpus the run stopped measuring. See A batch that ran out of budget.
2	tracevals could not run	The tool could not do its job, through bad usage, an unreadable trace, an unknown format, or a nonexistent path.

The 1 versus 2 distinction is the most useful thing on this page. 1 means the check worked and the answer was bad; 2 means you never got an answer. A pipeline that treats them identically will eventually report a broken trace path as an adherence violation.

On 2, the message is written to stderr and prefixed with the command you typed:

manni: no trace given; pass a trace file or run `manni tracevals list`

fill uses the same contract: 0 normally, 1 when any artifact result has status error, 2 for operational failures, including a numeric flag out of range.

The third outcome

An eval has five possible outcomes, not two:

Outcome	Meaning	Counts as failure?
pass	the eval held	no
fail	the eval was violated	yes
error	the check itself broke	yes
needs-review	the judge did not reach confident agreement	yes, by default
skipped	the check did not run	no

needs-review is the one that surprises pipelines. It is a designed outcome. When an ensemble of judge runs does not land confidently in the pass or fail zone, tracevals declines to decide. It routes the eval to a human instead. Errored judge runs count against consensus, which means they can push an eval into review. They can never push one into a silent pass.

By default that counts as a failure, on the reasoning that an unreviewed ambiguous result should not merge quietly. If your pipeline has somewhere better to send those, change it:

manni.config.yaml

tracevals:
  failOnReview: false

Now needs-review no longer contributes to a non-zero exit. It still appears in the report and in the summary counts, so you can surface it without blocking on it.

To decide per invocation instead, override it on the command line. That lets a strict pre-merge job and a lenient nightly one share one config:

npx @hawkeyexl/manni tracevals run <trace> --no-fail-on-review

--fail-on-review forces the opposite. Passing neither leaves the config value in force.

Decide this before you meet it

Pick a policy when you install the gate. Meeting needs-review for the first time as a red build nobody can explain is the avoidable version of this.

skipped never fails a run, and it has five unrelated causes:

Cause	What it looks like
a --deterministic-only run	every judged eval, with the reason naming the flag
an exhausted turn budget	the reason names the budget, and the batch says so beside the headline
a cost grader on a trace with no usage data	no cost data (maxUsd not checkable)
an artifact that opted out	the eval’s own metadata.eval-skip
a command eval without its grant	frontmatter commands not granted (execution.allow: [frontmatter-commands])

Each is listed with its reason in the report. See Read a failing eval.

A batch that ran out of budget

--max-turns (and judge.maxTurns) is a ceiling on the whole invocation, shared across every trace in a batch. That is deliberate: one judge, one budget, so fifty traces cannot spend fifty times the configured cap. A turn is one ensemble run, and a cached ensemble costs none.

The consequence is that a budget can run out part way through the corpus. Every remaining judged eval is then skipped. Nothing failed and nothing errored, so scoring the batch on those two counts alone would exit 0 after judging four traces out of fifty. It does not. A batch cut short by its own budget exits 1, says so beside the headline, and carries a budget block:

manni tracevals — 1 trace(s)
1 passed · 0 failed · 0 unreadable
BUDGET EXHAUSTED  1 eval(s) across 1 trace(s) were never judged — judge turn budget exhausted (0)

In --format json:

"budget": {
  "skippedEvals": 1,
  "traces": 1,
  "reason": "judge turn budget exhausted (0)"
},
"exitCode": 1

Exit 1 rather than 2, because the report is real: the traces that were judged carry genuine verdicts and are worth keeping. The same reasoning makes a selector matching no trace exit 2, since there is nothing to keep in that case. Neither may quietly exit 0.

Raise the ceiling, or narrow the corpus

The warning names both remedies. A gate that keeps hitting this is budgeted for one trace and pointed at fifty. Or it is judging evals a --deterministic-only job could have answered for free.

Keep the report

run writes its report to stdout in one of six formats, and --output additionally writes it to a file.

npx @hawkeyexl/manni tracevals run <trace> --format markdown --output tracevals-report.md

Format	Use it for
pretty (default)	reading in a terminal, colored, with vote splits on non-passing judged evals
json	anything programmatic; the full report structure
markdown	a build summary or a PR comment
github	GitHub Actions annotations on the lines that declare each eval, then the markdown summary
sarif	code scanning, with one rule per grader as tracevals/<grader>
junit	a test-report viewer; needs-review lands as skipped, not failed
calibrate takes pretty, json and markdown. The CI recipes for github and sarif are in
Run it in CI.

The markdown format renders as tables:

# manni tracevals report

- **Trace**: `/path/to/session.jsonl`
- **Source**: claude-code (claude-opus-4-8)
- **Project**: `/work/demo-project`
- **Turns**: 2

## Evals

| Outcome | Artifact | Eval | Grader | Detail |
|---|---|---|---|---|
| pass | fix-bug | used-read `.claude/skills/fix-bug/SKILL.md:6` | tool-usage |  |
| fail | fix-bug | forbidden-tool `.claude/skills/fix-bug/SKILL.md:12` | tool-usage | tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this. |
| pass | fix-bug | no-force-push `.claude/skills/fix-bug/SKILL.md:18` | command |  |
| needs-review | fix-bug | refactor-preserved-intent `.claude/skills/fix-bug/SKILL.md:23` | human | awaiting human review |
| skipped | CLAUDE.md | adheres-to-artifact (implicit) `CLAUDE.md` | ai | ai evals skipped (deterministic-only run) |

## Findings

- **FAIL** fix-bug › forbidden-tool · `.claude/skills/fix-bug/SKILL.md:12`
  - tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
  - .claude/skills/fix-bug/SKILL.md forbidden-tool: The session never ran shell commands; this skill is edit-only.
  - Guide: https://hawkeyexl.github.io/manni/tracevals/triage/
- **REVIEW** fix-bug › refactor-preserved-intent · `.claude/skills/fix-bug/SKILL.md:23`
  - Awaiting human review. Read the session against the assertion and decide.
  - .claude/skills/fix-bug/SKILL.md refactor-preserved-intent: The fix addressed the reported bug rather than working around it.
  - Guide: https://hawkeyexl.github.io/manni/tracevals/triage/

## Artifact coverage

| Resolved | Kind | Ref | Availability | Where |
|---|---|---|---|---|
| yes | skill | fix-bug | used | `/work/demo-project/.claude/skills/fix-bug/SKILL.md` |
| no | agent | Explore | used | built-in agent (no definition file) |
| yes | project-rules | project rules | used |  |

## Availability

- 3 skill(s) offered, 1 used, 2 never used
- 1 agent(s) offered, 1 used, 0 never used

## Warnings

- 1 unparseable JSONL line(s) were skipped

**9 eval(s)** — 2 pass, 1 fail, 0 error, 1 needs-review, 5 skipped.

The last coverage row has an empty location on purpose. Project rules resolve as one aggregated entry covering several files, so it is resolved with no single path. Every location it checked is in the entry’s tried list. See Reports & exit codes.

Each eval names the file and line that declare it, after its name. That is the artifact, or the manifest that holds a relocated block. The implicit eval has no declaring entry, so it names its artifact alone. The github, sarif and junit formats point at the same file and line.

## Findings lists what to act on: each failed, errored or needs-review eval. Under the finding, a line quotes the eval’s assertion, and a last line links to reading a failing eval. The other formats carry the same lines under each finding. The section is left out when nothing needs acting on.

--format json emits the same run as a structured document. That document carries trace metadata, per-eval outcomes with findings, consensus, the declaring location and guidance, and the coverage list including every location that was tried. It also carries warnings, summary counts, turn telemetry, and duration. Every field is documented in Reports & exit codes.

The commands are re-run, so the samples stay true

Every command that produces output on this page runs on every push. It runs against this repository’s committed fixtures, through the Doc Detective steps in this file’s source. A sample output nothing re-runs is a sample that will eventually be wrong.

Where to go next

The full report structureEvery field in the JSON report, the history file format, and what each outcome carries.

Decide whether to trust the judgeWhy needs-review exists, how consensus is computed, and what an errored run does to a verdict.