Exit codes & reports

Scope: the contract a pipeline depends on: what the process returns and what it can leave behind. Installing the gate is Add moose-tracevals to CI; the full field-by-field report structure is in the reference.

The exit-code contract

Code	Meaning	What it tells you
0	every eval passed or was skipped	Nothing to do.
1	some eval failed, errored, or needs review	The agent session did something worth looking at. In a batch, also a corpus the run stopped measuring. See A batch that ran out of budget.
2	moose-tracevals could not run	The tool could not do its job, through bad usage, an unreadable trace, an unknown format, or a nonexistent path.

The 1 versus 2 distinction is the most useful thing on this page. 1 means the check worked and the answer was bad; 2 means you never got an answer. A pipeline that treats them identically will eventually report a broken trace path as an adherence violation.

On 2, the message is written to stderr and prefixed:

moose-tracevals: no trace given; pass a trace file or run `moose-tracevals list`

fill uses the same contract: 0 normally, 1 when any artifact result has status error, 2 for operational failures, including a numeric flag out of range.

The third outcome

An eval has five possible outcomes, not two:

Outcome	Meaning	Counts as failure?
pass	the eval held	no
fail	the eval was violated	yes
error	the check itself broke	yes
needs-review	the judge did not reach confident agreement	yes, by default
skipped	the check did not run	no

needs-review is the one that surprises pipelines. It is a designed outcome. When an ensemble of judge runs does not land confidently in the pass or fail zone, moose-tracevals declines to decide. It routes the eval to a human instead. Errored judge runs count against consensus, which means they can push an eval into review. They can never push one into a silent pass.

By default that counts as a failure, on the reasoning that an unreviewed ambiguous result should not merge quietly. If your pipeline has somewhere better to send those, change it:

moose.config.yaml

tracevals:
  failOnNeedsReview: false

Now needs-review no longer contributes to a non-zero exit. It still appears in the report and in the summary counts, so you can surface it without blocking on it.

To decide per invocation instead, override it on the command line. That lets a strict pre-merge job and a lenient nightly one share one config:

npx moose-tracevals run <trace> --no-fail-on-needs-review

--fail-on-needs-review forces the opposite. Passing neither leaves the config value in force.

Decide this before you meet it

Pick a policy when you install the gate. Meeting needs-review for the first time as a red build nobody can explain is the avoidable version of this.

skipped never fails a run, and it has several unrelated causes. Those are a deterministic-only run, an exhausted cost budget, a cost check on a trace with no usage data, or an artifact that opted out. Each is listed with its reason in the report. See Read a failing eval.

A batch that ran out of budget

--max-cost-usd (and judge.maxCostUsd) is a ceiling on the whole invocation, shared across every trace in a batch. That is deliberate: one judge, one budget, so fifty traces cannot bill fifty times the configured cap.

The consequence is that a budget can run out part way through the corpus. Every remaining judged eval is then skipped. Nothing failed and nothing errored, so scoring the batch on those two counts alone would exit 0 after judging four traces out of fifty. It does not. A batch cut short by its own budget exits 1, says so beside the headline, and carries a budget block:

moose-tracevals — 1 trace(s)
1 passed · 0 failed · 0 unreadable
BUDGET EXHAUSTED  1 eval(s) across 1 trace(s) were never judged — judge cost budget exhausted ($0)

In --format json:

"budget": {
  "skippedEvals": 1,
  "traces": 1,
  "reason": "judge cost budget exhausted ($0)"
},
"exitCode": 1

Exit 1 rather than 2, because the report is real: the traces that were judged carry genuine verdicts and are worth keeping. The same reasoning makes a selector matching no trace exit 2, since there is nothing to keep in that case. Neither may quietly exit 0.

Raise the ceiling, or narrow the corpus

The warning names both remedies. A gate that keeps hitting this is budgeted for one trace and pointed at fifty. Or it is judging evals a --deterministic-only job could have answered for free.

Keep the report

run writes its report to stdout in one of three formats, and --output additionally writes it to a file.

npx moose-tracevals run <trace> --format markdown --output moose-tracevals-report.md

Format	Use it for
human (default)	reading in a terminal, colored, with vote splits on non-passing judged evals
json	anything programmatic; the full report structure
markdown	a build summary or a PR comment

The markdown format renders as tables:

# moose-tracevals report

- **Trace**: `/path/to/session.jsonl`
- **Source**: claude-code (claude-opus-4-8)
- **Project**: `/work/demo-project`
- **Turns**: 2

## Evals

| Outcome | Artifact | Eval | Grader | Detail |
|---|---|---|---|---|
| pass | fix-bug | used-read | tool-usage |  |
| fail | fix-bug | forbidden-tool | tool-usage | tool Bash was used 1 time(s) but must not be |
| pass | fix-bug | no-force-push | command |  |
| needs-review | fix-bug | refactor-preserved-intent | human | awaiting human review |
| skipped | CLAUDE.md | adheres-to-artifact (implicit) | ai | ai evals skipped (deterministic-only run) |

## Artifact coverage

| Resolved | Kind | Ref | Where |
|---|---|---|---|
| yes | skill | fix-bug | `/work/demo-project/.claude/skills/fix-bug/SKILL.md` |
| no | agent | Explore | built-in agent (no definition file) |
| yes | project-rules | project rules |  |

## Warnings

- 1 unparseable JSONL line(s) were skipped

**9 eval(s)** — 2 pass, 1 fail, 0 error, 1 needs-review, 5 skipped.

The last row has an empty location on purpose. Project rules resolve as one aggregated entry covering several files, so it is resolved with no single path. Every location it checked is in the entry’s tried list. See Reports & exit codes.

--format json emits the same run as a structured document. That document carries trace metadata, per-eval outcomes with findings and consensus, and the coverage list including every location that was tried. It also carries warnings, summary counts, cost, and duration. Every field is documented in Reports & exit codes.

Capture output rather than composing it

Both examples above were produced by running the CLI against this repository’s committed fixtures. The commands that produce them are re-run on every push. A hand-written sample output is a sample that will eventually be wrong.

Where to go next

The full report structureEvery field in the JSON report, the history file format, and what each outcome carries.

Decide whether to trust the judgeWhy needs-review exists, how consensus is computed, and what an errored run does to a verdict.