Reports & exit codes

Scope: everything run, calibrate and check emit. The policy discussion, meaning what to do about needs-review in a pipeline, is in Add tracevals to CI. Source of truth: src/tracevals/types.ts, src/tracevals/calibrate/types.ts, src/tracevals/reporters/, and src/tracevals/history.ts. The verdict schema’s $id is manni:tracevals:verdict.

-f, --format on run is pretty | json | markdown | github | sarif | junit, default pretty. On calibrate it is pretty | json | markdown. On fill, capture, prepare, check, release, and list it is pretty | json. Every default is pretty.

Exit codes

Code	Meaning
0	every eval passed or was skipped
1	any fail or error, and needs-review unless failOnReview: false
2	tracevals itself could not run, from bad usage, an unreadable trace, an unknown format, or a nonexistent path

2 is written to stderr as manni: <message>; 0 and 1 write the report to stdout.

calibrate reads 0 and 1 differently, because it measures rather than gates. Disagreeing with your labels is exit 0. Only a threshold you asked for, or a trace that could not be evaluated, makes it 1.

Outcomes

Outcome	Meaning	Fails the run?
pass	the eval held	no
fail	the eval was violated by an error-severity finding	yes
error	the check itself broke, through invalid options, an invalid evals block, or an unknown grader kind	yes
needs-review	the judge did not reach confident agreement	yes, by default
skipped	the check did not run; the reason is always given	no

Finding severity is notice | warning | error, the same three values every manni tool reports. Only error turns an eval into a fail.

Skip reasons stay distinct, because they mean different things about your coverage:

skipReason	Cause
ai evals skipped (deterministic-only run)	--deterministic-only
turn budget	the judge budget was spent; see judge.maxTurns
no cost data (maxUsd not checkable)	the trace carried no usage record for a cost grader to read
artifact skipped via metadata.eval-skip	the artifact opted out
frontmatter commands not granted (execution.allow: [frontmatter-commands])	--no-execution, or an execution.allow without the grant

--format json

The complete RunReport:

{
  "trace": {
    "file": "/path/to/session.jsonl",
    "source": "claude-code",
    "sessionId": "11111111-1111-1111-1111-111111111111",
    "cwd": "/work/demo-project",
    "model": "claude-opus-4-8",
    "turnCount": 2
  },
  "warnings": ["1 unparseable JSONL line(s) were skipped"],
  "coverage": [ /* CoverageEntry[] */ ],
  "availability": {
    "recorded": true,
    "skills": { "offered": 5, "used": 2, "unused": 3 },
    "agents": { "offered": 3, "used": 2, "unused": 1 },
    "listed": false
  },
  "evalResults": [ /* EvalResult[] */ ],
  "summary": {
    "total": 12, "pass": 4, "fail": 1,
    "error": 0, "needsReview": 1, "skipped": 6
  },
  "exitCode": 1,
  "turns": 0,
  "durationMs": 118
}

turns is what this run spent, meaning uncached ensemble runs the judge dispatched. A fully cached or --deterministic-only run reports 0. It is the same unit judge.maxTurns bounds, so a report can be read against the budget that produced it. tracevals counts turns and reports no spend of its own.

trace

Field	Type	Notes
file	string	Absolute path to the trace.
source	string	Trace source. claude-code today.
sessionId	string?	Present when the trace records one.
cwd	string	The session’s recorded working directory.
model	string?	The model the session ran on.
turnCount	number	Non-sidechain user prompts. This describes the session, not what tracevals spent.

EvalResult

Field	Type	Notes
evalName	string	The eval’s name, or adheres-to-artifact / evals-block-valid.
artifact	string	Absolute path to the artifact file.
artifactName	string	Its display name, such as fix-bug or CLAUDE.md.
artifactType	string	skill, agent, project-rules, or slash-command.
grader	string	The grader kind, ai for judged.
implicit	boolean	true for the whole-artifact eval.
location	object	Where the eval is declared, { file, line? }. See below.
outcome	string	One of the five outcomes above.
findings	array?	Deterministic findings: { evalName, artifact, message, severity }, severity one of notice, warning, error.
consensus	object?	Judged evals only, see below.
error	string?	Why the check broke, when outcome is error.
skipReason	string?	Why it was skipped.
turns	number?	Judged evals only. Uncached ensemble runs spent on this eval; 0 when the ensemble came from cache.
guidance	object	The rule behind the result. Present on every result, passing ones included. See below.
durationMs	number	Time for this eval.

location

The item of metadata.evals that declares the eval. That is the artifact when its front matter holds the block. It is the manifest when relocation put the block there, so file can differ from artifact.

Field	Type	Notes
file	string	Relative to the working directory and forward-slashed inside it, absolute outside it, or a hosted manifest’s URL.
line	number?	The 1-based line of the entry. Absent when nothing records one.

The implicit adheres-to-artifact eval has no entry, so its location is the artifact alone, with no line. A schema failure (evals-block-valid) points at the block, and an eval-skip at its key.

"location": { "file": ".claude/skills/fix-bug/SKILL.md", "line": 12 }

guidance

What the result was judged against, and where to read about acting on it. Every report format prints it under a fail, error or needs-review result.

Field	Type	Notes
id	string	The artifact file, relative to the working directory, absolute outside it.
rule	string	The eval’s name, the same as evalName.
description	string | null	The eval’s assertion, or null when it declares none. The implicit eval reads The session follows every instruction in this artifact., and evals-block-valid reads The artifact's metadata.evals block matches the schema.
guide	string	https://hawkeyexl.github.io/manni/tracevals/triage/, the page on reading a failing eval.

"guidance": {
  "id": ".claude/skills/fix-bug/SKILL.md",
  "rule": "forbidden-tool",
  "description": "The session never ran shell commands; this skill is edit-only.",
  "guide": "https://hawkeyexl.github.io/manni/tracevals/triage/"
}

The text reporters print it as two lines, <id> <rule>: <description> and then Guide: <guide>. A null description drops the colon and what follows it.

consensus

Present on every judged eval.

Field	Type	Notes
runs	array	Each run: { verdict?, error?, provider, model, cached, usage?, durationMs }. verdict is absent when the run errored.
votes	object	{ pass, fail, partial, error } counts.
verdict	string	pass, fail, or partial. partial counts as fail for the binary outcome.
agreement	number	Fraction of non-errored runs agreeing with the majority. Reported, but not what routes the outcome.
meanConfidence	number	Mean confidence across non-errored runs.
zone	string	auto-pass, auto-fail, or human-review. This is what maps to the outcome.

Each verdict carries claim, observed, match, confidence, and reasoning, and validates against manni:tracevals:verdict. See How the judge decides.

CoverageEntry

One entry per artifact reference found in the trace, resolved or not.

Field	Type	Notes
ref	string	The reference as it appeared, e.g. fix-bug, plugin:skill.
kind	string	skill, agent, project-rules, or slash-command.
resolved	boolean	Whether a file was found.
path	string?	The resolved file. Absent for an aggregated project-rules entry covering several files.
tried	string[]	Every location searched, in order. Usually enough to diagnose a miss.
note	string?	Why it will never resolve, e.g. built-in agent (no definition file).
availability	string?	offered-and-used, offered-not-used, not-offered, or unknown. Absent for project-rules, which is always in force.
stale	boolean?	The file is newer than the session’s end time. Absent when the trace records no end time.
modifiedAt	string?	The file’s mtime, ISO-8601. Present whenever stale is.

An unresolved reference is not a failing eval. It is a coverage note plus a warning, so a missing plugin never crashes a run or turns a build red on its own.

stale is likewise never an eval outcome and never an exit code. It is a heuristic that says how much to trust the verdicts, not what they are. mtime is not content identity, and a checkout rewrites every mtime. See Artifact coverage.

The location shown for an entry

Both the pretty and markdown reporters derive the location cell the same way, in this order:

1. path, when the entry resolved to a single file.
2. note, when there is no single path.
3. Empty, otherwise.

The third case is real rather than theoretical: project rules resolve as one aggregated entry covering several files (CLAUDE.md and AGENTS.md), so the entry is resolved: true with no path. Its locations are all in tried. An unresolved entry falls back to not found (<n> location(s) tried).

availability

What the session was offered, next to what it used. recorded: false means the trace carried no availability roster at all. That is unknown, not nothing offered, and every CoverageEntry then reads unknown.

Field	Type	Notes
recorded	boolean	Whether the trace carried a roster.
skills, agents	object	{ offered, used, unused } counts of distinct names.
listed	boolean	Whether the offered-but-unused rows were included in coverage too. --report-unused-artifacts turns this on.

It is never a verdict. Availability contributes nothing to summary and nothing to exitCode; the report is identical both with and without --report-unused-artifacts. Reading the three states is covered in Artifact coverage.

--format pretty

The default. Structure:

manni tracevals — /path/to/session.jsonl
claude-code · claude-opus-4-8 · 2 turn(s) · /work/demo-project

  PASS   fix-bug › used-read  .claude/skills/fix-bug/SKILL.md:6
  FAIL   fix-bug › forbidden-tool  .claude/skills/fix-bug/SKILL.md:12
         [error] tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
         .claude/skills/fix-bug/SKILL.md forbidden-tool: The session never ran shell commands; this skill is edit-only.
         Guide: https://hawkeyexl.github.io/manni/tracevals/triage/
  PASS   fix-bug › no-force-push  .claude/skills/fix-bug/SKILL.md:18
  REVIEW fix-bug › refactor-preserved-intent  .claude/skills/fix-bug/SKILL.md:23
         Awaiting human review. Read the session against the assertion and decide.
         .claude/skills/fix-bug/SKILL.md refactor-preserved-intent: The fix addressed the reported bug rather than working around it.
         Guide: https://hawkeyexl.github.io/manni/tracevals/triage/
  PASS   reviewer › reviewer-is-read-only  .claude/agents/reviewer.md:6
  SKIP   doc-writer › doc-writer-wrote-docs  .claude/agents/doc-writer.md:6
         agent "doc-writer" was spawned but recorded no subagent turns (no inline
         sidechain records and no sidecar transcript), so it governed no turns
  SKIP   CLAUDE.md › adheres-to-artifact (implicit)  CLAUDE.md
         ai evals skipped (deterministic-only run)

Artifact coverage
  ✓ skill: fix-bug   /work/demo-project/.claude/skills/fix-bug/SKILL.md
  ○ agent: Explore   built-in agent (no definition file)
  ✓ agent: reviewer  /work/demo-project/.claude/agents/reviewer.md

Availability
  5 skill(s) offered, 2 used, 3 never used
  3 agent(s) offered, 2 used, 1 never used
  pass --report-unused-artifacts to list them

Warnings
  ! 1 unparseable JSONL line(s) were skipped

12 eval(s): 4 pass, 1 fail, 0 error, 1 needs-review, 6 skipped

forbidden-tool forbids Bash, and this session ran it twice. Only one of those came after the fix-bug skill was invoked, and only that one is the skill’s business. doc-writer was spawned and recorded no turns, so its evals report skipped rather than a verdict about work it had no part in. Both are eval windowing.

Outcome labels are PASS, FAIL, ERROR, REVIEW, SKIP. Each eval line ends with its location as <file>:<line>, or the file alone when no line is recorded. A FAIL, ERROR or REVIEW line ends with the result’s guidance, dimmed.

The message of every built-in deterministic failure ends with Redo the session work so it holds, or fix the eval if the artifact does not ask for this. A command failure reads command exited <code> (expected <codes>). Redo the session work so the check passes, or fix the command if it is broken. The command’s stderr follows as Output: <stderr>.

A non-passing judged eval gets a judged block above the vote split. It quotes the first run that voted against the session, or the first run with a verdict when none did.

  FAIL   fix-bug › eval-5  .claude/skills/fix-bug/SKILL.md:27
         AI judge: fail (confidence 0.88). Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
         Observed: The session edited src/app.ts before any test file.
         Reasoning: No failing test was written before the fix.
         votes pass:0 fail:3 partial:0 error:0
         .claude/skills/fix-bug/SKILL.md eval-5: Reproduce the bug with a failing test before applying the fix.
         Guide: https://hawkeyexl.github.io/manni/tracevals/triage/

Outcome	First line of the judged block
judged fail	AI judge: fail (confidence <n>). Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
judged needs-review	AI judge: no confident verdict (confidence <n>). Read the session and decide. If this recurs, sharpen the assertion and run manni tracevals calibrate.
grader: human	Awaiting human review. Read the session against the assertion and decide.

Observed: and Reasoning: follow the first line when the quoted run carries them. A human-graded eval has no judge, so its block is the one line.

Coverage marks: ✓ resolved, ○ referenced but not found, · offered and never used. The location follows the rule above. A · row is the exception. It reports offered, never used plus the roster’s own description, because nothing was ever looked for on disk. A stale entry appends ⚠ modified after the session ended (<iso>) after the location. In markdown that goes into the same Where cell, so the table keeps its column count.

--format markdown

The same run as tables: # manni tracevals report, bullet metadata, then ## Evals (| Outcome | Artifact | Eval | Grader | Detail |, with the eval’s <file>:<line> in code after its name), ## Findings, ## Artifact coverage (| Resolved | Kind | Ref | Availability | Where |), ## Availability, ## Warnings, and a bold summary sentence. An offered-but-unused row reads n/a under Resolved, never no. Pipe characters in messages are escaped. Suited to a build summary or a PR comment.

## Findings lists each fail, error and needs-review result, and is left out when there are none. Each item is labelled FAIL, ERROR or REVIEW. Under it come the findings, the error, or the judged block, then the guidance lines.

## Findings

- **FAIL** fix-bug › forbidden-tool · `.claude/skills/fix-bug/SKILL.md:12`
  - tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
  - .claude/skills/fix-bug/SKILL.md forbidden-tool: The session never ran shell commands; this skill is edit-only.
  - Guide: https://hawkeyexl.github.io/manni/tracevals/triage/
- **REVIEW** fix-bug › refactor-preserved-intent · `.claude/skills/fix-bug/SKILL.md:23`
  - Awaiting human review. Read the session against the assertion and decide.
  - .claude/skills/fix-bug/SKILL.md refactor-preserved-intent: The fix addressed the reported bug rather than working around it.
  - Guide: https://hawkeyexl.github.io/manni/tracevals/triage/

--format github

GitHub Actions workflow commands, then the markdown summary, so the step’s log and $GITHUB_STEP_SUMMARY read the same run. Each annotation lands on the eval’s location, meaning the file and line that declare it. That is the line someone edits to change the outcome, in the artifact or in its manifest. The annotation carries no line when the location has none. The trace it was graded against ends the message’s first line, and the guidance lines follow it, joined by %0A:

::error file=test/tracevals/fixtures/project/.claude/skills/fix-bug/SKILL.md,line=12,title=manni tracevals%3A forbidden-tool::tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this. (trace: test/tracevals/fixtures/traces/claude-session.jsonl)%0Atest/tracevals/fixtures/project/.claude/skills/fix-bug/SKILL.md forbidden-tool: The session never ran shell commands; this skill is edit-only.%0AGuide: https://hawkeyexl.github.io/manni/tracevals/triage/

A finding takes its own severity as the annotation level. A judged fail and an error outcome are annotated as errors. A needs-review outcome is one ::warning, since nobody has decided it yet. Its message is the judged block, or the human-review line:

::warning file=test/tracevals/fixtures/project/.claude/skills/fix-bug/SKILL.md,line=23,title=manni tracevals%3A refactor-preserved-intent::Awaiting human review. Read the session against the assertion and decide. (trace: test/tracevals/fixtures/traces/claude-session.jsonl)%0Atest/tracevals/fixtures/project/.claude/skills/fix-bug/SKILL.md refactor-preserved-intent: The fix addressed the reported bug rather than working around it.%0AGuide: https://hawkeyexl.github.io/manni/tracevals/triage/

--format sarif

One SARIF 2.1.0 run from the driver manni-tracevals. Each rule is a grader, with the id tracevals/<grader>, such as tracevals/tool-usage or tracevals/ai. A trace that could not be read is filed under tracevals/trace. Levels map error to error, warning to warning and notice to note. A judged fail and an error outcome are error results, and each needs-review eval is one warning result.

A built-in grader’s rule carries a fullDescription that says what the grader checks, and a helpUri of https://hawkeyexl.github.io/manni/tracevals/triage/. A plugin grader’s rule carries neither.

{
  "id": "tracevals/tool-usage",
  "name": "tool-usage",
  "shortDescription": { "text": "manni tracevals tool-usage grader" },
  "fullDescription": { "text": "Checks whether the session used a tool, and how often." },
  "helpUri": "https://hawkeyexl.github.io/manni/tracevals/triage/",
  "defaultConfiguration": { "level": "error" }
}

A result’s message.text is its lines joined by newlines: the finding, the error, or the judged block, then the guidance lines.

A location is the eval’s location, with its line as region.startLine. A location with no line has no region. Under the working directory the file is relative to SRCROOT. A user-level skill in ~/.claude gets an absolute file: URI, since no relative one reaches it, and a hosted manifest keeps its URL. Each result’s properties carry eval, artifactName, artifactType, trace, sessionId and guidance.

--format junit

One <testsuite> per trace and one <testcase> per artifact and eval. classname is the artifact file and name is the eval, so a JUnit viewer groups by the file someone edits. file and line are the eval’s location, and line is left out when the location has none. fail is a <failure> element and error an <error> element. Their body is the findings or the judged block, then the guidance lines, one per line. skipped and needs-review are both <skipped>, because JUnit has no third state and a review queue is not a broken build.

<failure message="forbidden-tool failed" type="tracevals/tool-usage">tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
test/tracevals/fixtures/project/.claude/skills/fix-bug/SKILL.md forbidden-tool: The session never ran shell commands; this skill is edit-only.
Guide: https://hawkeyexl.github.io/manni/tracevals/triage/</failure>

Batch runs

Naming more than one trace emits a BatchReport instead, as does selecting them with --all-projects, --newer-than, or --limit. It answers a different question: not did this hold? but how often does it hold?

Naming exactly one trace still emits the RunReport above, byte for byte. What decides the shape is how the traces were selected, never how many came back; see the CLI reference.

{
  "traces": [ /* BatchTraceEntry[] */ ],
  "artifacts": [ /* AggregateRow[] — one per artifact */ ],
  "evals": [ /* AggregateRow[] — one per eval */ ],
  "summary": {
    "total": 14, "pass": 4, "fail": 1, "error": 0, "needsReview": 1, "skipped": 8,
    "traces": 2, "tracesPassed": 1, "tracesFailed": 1, "tracesErrored": 0
  },
  "warnings": [],
  "exitCode": 1,
  "turns": 0,
  "durationMs": 75
}

summary is the single-run RunSummary counts summed over every trace, plus four per-trace tallies: traces selected, tracesPassed, tracesFailed (reported exit 1), and tracesErrored (could not be parsed at all).

AggregateRow

One artifact, or one eval within it, summed across the batch.

Field	Type	Notes
key	string	<type>:<name>, plus ::<evalName> on eval rows. The sort key.
artifactName	string	fix-bug, CLAUDE.md.
artifactType	string	skill, agent, project-rules, or slash-command.
artifacts	string[]	Distinct paths this row aggregated from. More than one means the row spans projects.
evalName	string?	Absent on artifact rows.
graders	string[]?	Grader kinds seen, sorted. Eval rows only.
traces	number	Traces that produced at least one result for this row.
total	number	Eval results contributing.
counts	object	{ pass, fail, error, needsReview, skipped }.
graded	number	total minus skipped, the pass-rate denominator.
passRate	number|null	pass / graded, or null when nothing was graded.
failingTraces	string[]	Traces where this row produced a fail or error, in batch order.
reviewTraces	string[]	Traces where it produced a needs-review.
skipReasons	string[]	Distinct skip reasons, sorted.

Two things about passRate are deliberate and worth knowing before you build on it:

• skipped is out of the denominator. A check that never armed is not evidence in either direction, so it is excluded rather than counted as a pass. An artifact that was resolved but never invoked reports null, rendered as —, not 100%.
• needs-review is in the denominator and not in the numerator. It fails the run by default, so counting it as a pass would contradict the exit-code contract. A row that is all review therefore reads 0%, with the traces named in reviewTraces.

Rows are keyed by artifact type and name, not by path, so the same skill evaluated across several projects is one rate. Ordering is a byte comparison on key, so two runs over the same corpus produce comparable reports on any platform.

BatchTraceEntry

Field	Type	Notes
file	string	The trace, as it was named or discovered.
sessionId	string?	When the trace records one.
error	string?	Set when the trace could not be evaluated at all. summary is then absent.
summary	object?	That trace’s own RunSummary.
warnings	string[]	That trace’s warnings.
exitCode	number	0 or 1. An errored trace is 1.
turns	number	Judge turns spent on that trace.
durationMs	number	Time for that trace.

The turn budget is shared, not per trace

judge.maxTurns applies to the whole invocation. Once it is exhausted the remaining evals report skipped with the reason turn budget. That applies on later traces as well as later evals, and the reason survives into the row’s skipReasons. It is never a pass, and never a silently absent row.

--format pretty and --format markdown

The pretty batch report leads with the rate table, names the outlier traces beneath each row, then lists artifact rates and one line per trace. Rows are in key order, so a batch is comparable to the one before it. Run against this repository’s own fixture corpus:

manni tracevals — 2 trace(s)
1 passed · 1 failed · 0 unreadable

Eval pass rates
       —  doc-writer › doc-writer-followed-its-brief (1 trace(s))
          0 pass, 0 fail, 0 error, 0 review, 1 skip
          skipped: agent "doc-writer" was spawned but recorded no subagent turns (no inline sidechain records and no sidecar transcript), so it governed no turns
       —  doc-writer › doc-writer-wrote-docs (1 trace(s))
          0 pass, 0 fail, 0 error, 0 review, 1 skip
          skipped: agent "doc-writer" was spawned but recorded no subagent turns (no inline sidechain records and no sidecar transcript), so it governed no turns
    100%  reviewer › reviewer-is-read-only (1 trace(s))
          1 pass, 0 fail, 0 error, 0 review, 0 skip
    100%  reviewer › reviewer-read-something (1 trace(s))
          1 pass, 0 fail, 0 error, 0 review, 0 skip
       —  AGENTS.md › eval-1 (2 trace(s))
          0 pass, 0 fail, 0 error, 0 review, 2 skip
          skipped: ai evals skipped (deterministic-only run)
       —  CLAUDE.md › adheres-to-artifact (2 trace(s))
          0 pass, 0 fail, 0 error, 0 review, 2 skip
          skipped: ai evals skipped (deterministic-only run)
       —  fix-bug › eval-5 (1 trace(s))
          0 pass, 0 fail, 0 error, 0 review, 1 skip
          skipped: ai evals skipped (deterministic-only run)
      0%  fix-bug › forbidden-tool (1 trace(s))
          0 pass, 1 fail, 0 error, 0 review, 0 skip
          failing: claude-session.jsonl
    100%  fix-bug › no-force-push (1 trace(s))
          1 pass, 0 fail, 0 error, 0 review, 0 skip
      0%  fix-bug › refactor-preserved-intent (1 trace(s))
          0 pass, 0 fail, 0 error, 1 review, 0 skip
          review: claude-session.jsonl
    100%  fix-bug › used-read (1 trace(s))
          1 pass, 0 fail, 0 error, 0 review, 0 skip
       —  writing-toolkit:identify-ai-tells › adheres-to-artifact (1 trace(s))
          0 pass, 0 fail, 0 error, 0 review, 1 skip
          skipped: artifact skipped via metadata.eval-skip

Artifact pass rates
       —  agent: doc-writer 0 pass, 0 fail, 0 error, 0 review, 2 skip
    100%  agent: reviewer 2 pass, 0 fail, 0 error, 0 review, 0 skip
       —  project-rules: AGENTS.md 0 pass, 0 fail, 0 error, 0 review, 2 skip
       —  project-rules: CLAUDE.md 0 pass, 0 fail, 0 error, 0 review, 2 skip
     50%  skill: fix-bug 2 pass, 1 fail, 0 error, 1 review, 1 skip
       —  skill: writing-toolkit:identify-ai-tells 0 pass, 0 fail, 0 error, 0 review, 1 skip

Traces
  FAIL   claude-session.jsonl 4 pass, 1 fail, 0 error, 1 review, 6 skip
  PASS   claude-session-sidecar.jsonl 0 pass, 0 fail, 0 error, 0 review, 2 skip

14 eval(s) across 2 trace(s): 4 pass, 1 fail, 0 error, 1 needs-review, 8 skipped

Three rows worth reading closely. forbidden-tool is 0% over the one trace that invoked the skill. The other trace never did, so it is not in the denominator. refactor-preserved-intent is 0% for a different reason: it is awaiting human review, named under review: rather than failing:. And doc-writer’s rows are —, because that agent governed no turns in either trace; a 100% there would be a rate computed from nothing.

The markdown form is the same content as three tables: ## Eval pass rates (| Rate | Artifact | Eval | Grader | Traces | Outcomes | Outliers |), ## Artifact pass rates, and ## Traces. It suits a build summary or a PR comment.

Calibration runs

calibrate emits a CalibrationReport, a third shape alongside RunReport and BatchReport. It answers was it right?, so it leads with the two kinds of mistake rather than with a rate.

{
  "labelsFile": "/repo/tracevals/labels.yaml",
  "corpus": [ /* absolute trace paths, in batch order */ ],
  "setting": { "ensembleRuns": 3, "zones": { "autoPass": 0.8, "autoFail": 0.8 } },
  "counts": {
    "labels": 10, "scored": 8, "agree": 5,
    "falsePass": 1, "falseFail": 1, "review": 0, "missedReview": 1,
    "errored": 0, "skipped": 2,
    "reviewVolume": 1, "insufficient": 0, "agreement": 0.625
  },
  "disagreements": [ /* Disagreement[] */ ],
  "unscored": [ /* Disagreement[] — labelled, but no evidence either way */ ],
  "sweep": [ /* SweepCell[] — only with --sweep */ ],
  "gates": [ /* CalibrationGate[] — only for thresholds you set */ ],
  "batch": { /* the BatchReport the counts were computed from */ },
  "warnings": [],
  "exitCode": 0,
  "turns": 0,
  "durationMs": 240
}

Field	Type	Notes
counts.labels	number	Labels that joined to a result. A label matching nothing is exit 2, never a missing row.
counts.scored	number	labels minus skipped, the agreement denominator.
counts.falsePass	number	Labelled fail, judged pass.
counts.falseFail	number	Labelled pass, judged fail.
counts.review	number	Labelled pass or fail, routed to a human instead.
counts.missedReview	number	Labelled needs-review, decided anyway.
counts.reviewVolume	number	needs-review across every eval in the corpus, labelled or not.
counts.insufficient	number	Rows whose cached ensemble was too small to score at this setting. Non-zero also raises a warning.
counts.agreement	number | null	agree / scored; null when nothing was scored.
disagreements[].kind	string	false-pass, false-fail, review, missed-review, or error.
disagreements[].consensus	object?	Vote tally, agreement, mean confidence, and run count at the setting scored, not at the depth the corpus was judged.
disagreements[].note	string?	Whatever you wrote in the labels file.
sweep[].axis	string	baseline, ensembleRuns, zones.autoPass, or zones.autoFail.
sweep[].value	number	The value of that axis on this row.
gates[].exceeded	boolean	Any true makes exitCode 1.

The pretty and markdown forms carry the same content. That is the headline counts, the disagreement list with its evidence and notes, and the unscored rows with their skip reasons. It also covers the sweep table and the thresholds. See Calibrate the judge.

check reports

manni tracevals check judges one turn rather than a whole session, so its report has a shape of its own. -f json prints it, and -o writes the same bytes to a file. The pretty form is in the check reference.

{
  "trace": "/abs/3b265d00.jsonl",
  "sessionId": "3b265d00-0000-4000-8000-000000000001",
  "agentId": null,
  "turn": { "from": 4, "to": 15 },
  "judge": { "provider": "mock", "model": "mock-model", "mode": "generative", "runs": 3 },
  "extraction": { "provider": "mock", "model": "mock-model" },
  "sources": [
    { "path": "CLAUDE.md", "format": "claude-md", "trigger": "always",
      "rules": 2, "origin": "declared" },
    { "path": ".cursor/rules/web.mdc", "format": "cursor-rule",
      "trigger": "globs matched src/web/App.tsx", "rules": 1, "origin": "extracted" },
    { "path": "prompt", "format": "prompt", "trigger": "typed prompts 1-2",
      "rules": 0, "origin": "extracted" }
  ],
  "findings": [
    { "source": "CLAUDE.md", "rule": "no-force-push",
      "text": "Never run `git push --force`.", "outcome": "fail", "severity": "error",
      "observed": "not-followed 100, followed 0, not-applicable 0. The turn shows `git push --force`.",
      "confidence": 1, "reasoning": "The turn shows `git push --force`." }
  ],
  "summary": { "sources": 6, "rules": 7, "notApplicable": 0, "fail": 2, "needsReview": 1,
    "reported": 0 },
  "session": {
    "turns": 6,
    "rules": [
      { "source": "CLAUDE.md", "rule": "no-force-push", "followed": 4, "broken": 1,
        "repaired": 1, "needsReview": 0, "notApplicable": 1, "notInScope": 0,
        "last": { "turn": 3, "outcome": "followed" } }
    ]
  },
  "skipped": null,
  "warnings": [],
  "exitCode": 1
}

Field	Type	Meaning
trace	string	The transcript that was judged, as an absolute path.
sessionId	string	The session’s id. The closing line shows its first eight characters.
agentId	string or null	The subagent a SubagentStop judged. null for the main session.
turn	{from, to}	The turn’s first and last event, as ordinals in the trace.
judge	object or null	The provider and model that judged, its mode, and runs. null when the run stopped before a judge was asked.
judge.mode	decision or generative	Detected from the provider. decision is a decision-only provider such as jev, which makes one call, so runs is 1. Every other provider is generative, and scores each rule in each run.
extraction	object or null	The out-of-loop provider and model. null when no source needed them.
sources[]	object	One entry per source in scope: path, format, trigger, rules, origin, and skill for a skill’s files. trigger says why the source was in scope. origin is declared for a file’s own ai evals, and extracted otherwise. Six formats say what the session was asked, in the table below.
findings[]	object	Only fail and needs-review outcomes, each with its source, rule, text, severity, observed and confidence. severity is error when the rule’s source blocks, and warning when it is only reported. Only Claude Code’s default system prompt gives warning. observed gives the rule’s three scores, averaged over the runs. confidence is the not-followed score as a fraction. reasoning is the judge’s sentence or two on what the transcript shows, and observed ends with it. A decision-only provider such as jev gives none. A rule left out was followed or did not apply.
summary	object	The counts of sources, rules, notApplicable, fail, needsReview and reported. fail counts the error breaks, and reported counts the warning breaks. needsReview counts the unsettled rules of both severities.
session	object or null	What the hook recorded for this session in the session ledger, summed up per rule. null when the session has no ledger.
session.turns	number	The turns the ledger records.
session.rules[]	object	One entry per rule the ledger holds, with its source and rule. followed, broken, needsReview and notApplicable count its entries. repaired counts the broken entries that a later followed entry came after. notInScope counts the recorded turns where the rule’s file was not in scope. last is its latest entry’s turn and outcome.
skipped	string or null	null, empty-turn, no-sources or not-applicable. The other gates apply only under a hook.
warnings	array	Strings, such as a rules file whose front matter does not parse.
exitCode	number	0, or 1 when summary.fail is above zero. A warning break alone exits 0.

The six formats that say what the session was asked never declare evals, so their origin is always extracted.

format	path	trigger
prompt	prompt	typed prompts 1-<n>, every prompt typed up to the turn’s end
plan	The plan file under your Claude config directory, or plan when the plan came only as the call’s input	approved at turn <n>, the ordinal of the last approved ExitPlanMode
speckit-spec	specs/<feature>/spec.md, plan.md or tasks.md	touched <path>, the first file the session touched under specs/<feature>/
kiro-spec	.kiro/specs/<name>/requirements.md, design.md or tasks.md	touched <path>, the first file touched under that directory
openspec-change	A .md file under openspec/changes/<id>/, never under changes/archive/	touched <path>, the first file touched under that change
plans	A file runtime.plans names	touched <path>, the file itself

Two more formats come from the transcript itself. They are read from the last record of their kind at or before the turn’s end, and never declare evals.

format	path	trigger
system-prompt	system-prompt	recorded at turn <n>, Claude Code <version>, the ordinal of the last prompt_snapshot. , Claude Code <version> is left out when the record has no version. , custom ends it when the boundary marker is missing.
output-style	The file that defines the style, relative to the project root or ~/… under home	style <name>, the style’s name as the transcript recorded it

summary.sources counts every row, so the closing line says “sources” rather than “files”.

Only a confident violation counts as fail: a not-followed score at judge.zones.autoFail, as a percentage, in every run. A rule the judge could not settle, or errored on, is needs-review, which never fails check. Only an error break exits 1 or blocks a Stop. A confident warning break counts in reported and does neither. A hook does not judge the default system prompt, so its report holds no warnings.

The session ledger

The hook records each turn it sees, so the judge knows what each rule found on earlier turns. That covers a turn that ended before the judge was asked, such as one with no rules in scope. A judged turn also records each judged rule’s result. The main session writes .manni/tracevals/sessions/<session hash>.ledger.json under the project. A subagent writes <session hash>.<agent id hash>.ledger.json, so parallel subagents never share a file. manni tracevals check reads the ledger of the session it judges and never writes it.

{
  "version": 1,
  "turns": [
    { "turn": 3, "inScope": true, "sources": ["CLAUDE.md"],
      "commands": ["npm ci", "git push --force origin main"], "wrote": ["src/web/App.tsx"],
      "read": ["CLAUDE.md"], "skills": [], "agents": [] }
  ],
  "rules": {
    "CLAUDE.md#no-force-push": {
      "text": "<sha256 of the rule text>",
      "entries": [
        { "turn": 3, "outcome": "broken", "note": "The turn shows `git push --force`." }
      ]
    }
  }
}

Field	Type	Meaning
turns[]	object	One record per turn, oldest first. The ledger keeps the last 30. Recording a turn again replaces its record.
turns[].turn	number	The turn’s first event, as an ordinal in the trace.
turns[].inScope	boolean	Whether any rules file governed the turn. An empty turn records false.
turns[].sources	array	The rules files in scope, by path.
turns[].commands	array	Up to 10 Bash commands, each on one line and cut to 120 characters.
turns[].wrote	array	Up to 10 files written or edited, relative to the project.
turns[].read	array	Up to 5 rules files the turn read.
turns[].skills	array	The skills the turn ran.
turns[].agents	array	The subagent types the turn spawned.
rules	object	One entry per rule, keyed by its file and id.
text	string	A sha256 of the rule’s text. An edited rule starts with no history.
entries[].turn	number	The turn’s first event, as an ordinal in the trace. Judging a turn again replaces its entry.
entries[].outcome	string	followed, broken, needs-review or not-applicable.
entries[].note	string	The judge’s reasoning, cut to 200 characters. Empty for a decision-only provider.

Every call the judge makes for a turn opens with what the session did before it. That block is read from the transcript, with no model, so it covers turns the hook never saw. It gives every earlier turn that did something one line, newest last. The ledger only marks a turn it recorded with no rules in scope.

# Earlier in this session

- (turns 4–31: 12 lines left out)
- turn 40: ran npm ci; wrote src/web/App.tsx; read CLAUDE.md; ran skill authoring-workflow
- turn 52: spawned technical-nonfiction-reviewer, which returned {"route_to":"writer","issues":[…
- turn 61: wrote specs/001-login/tasks.md; ticked T014 in specs/001-login/tasks.md
- turn 70: asked the user; had a plan approved
- turn 77: ran ls (no rules in scope)

A line joins its facts with ; , in this order.

Fact	Line
Bash commands	ran <command>, <command>, up to 10, each on one line and cut to 120 characters
Files written or edited	wrote <path>, <path>, up to 10, relative to the project
Rule sources read	read <path>, up to 5. Other files the turn read are left out.
A skill call	ran skill <name>, one per skill
An agent spawn	spawned <type>, which returned <first line>, up to 10 per turn. The result’s first line is cut to 120 characters. With no result, the line is spawned <type>.
A question to the user	asked the user
A task ticked	ticked <id> in <path>
A plan approved	had a plan approved, for an ExitPlanMode whose result is not an error

A task is ticked when an Edit or MultiEdit turns a - [ ] <text> line into - [x] <text> with the same text. Spec Kit, Kiro and OpenSpec all keep tasks this way. The line names the task’s leading id, such as T014, FR-001 or 1.2. A task with no id is named by its text, cut to 60 characters. Commands, paths and results pass through the judge’s redaction before they are cut.

That block shares the render budget with the turn, and takes at most a quarter of it. Over that, its oldest lines go first, and a marker line in their place says which turns went, as - (turns 4–31: 12 lines left out) or - (turn 4: 1 line left out). Each rule keeps its last 10 entries. Its first followed entry and every broken entry stay, so the others go first. A rule’s item shows the judge up to five results from earlier turns. Those are its first followed, its broken ones and its latest, leaving out the turns it did not apply to. A run of recorded turns where the rule’s file was not in scope shows as (turns 4–8: not in scope). A rule with only turns it did not apply to has no history to show. Both blocks are part of the verdict cache key. A missing or unreadable ledger is an empty history, and a ledger that cannot be written is a warning that never blocks.

--output

Writes the rendered report to a file in addition to stdout, in whichever format --format selected.

npx @hawkeyexl/manni tracevals run <trace> --format markdown --output tracevals-report.md

History

--history appends one line to history.file (default .manni/tracevals/history.jsonl) and compares this run against the most recent previous entry for the same session.

Each line is a HistoryEntry:

Field	Type	Notes
timestamp	string	ISO 8601.
traceFile	string	Path to the trace.
sessionId	string?	When known.
project	string	The trace’s recorded cwd.
summary	object	The same summary counts as the report.
exitCode	number	0 or 1.
turns	number	Judge turns spent on that run.
evals	array	{ artifact, evalName, outcome } per eval, outcomes only, not findings.

The comparison matches on sessionId when present, falling back to traceFile, and is appended to non-JSON output:

History vs 2026-07-01T09:14:22.104Z:
  regression: fix-bug › used-read
  added: fix-bug › reproduces-before-fixing

Line	Meaning
regression	previously pass, now fail or error
improvement	previously fail or error, now pass
added	an eval that did not exist in the previous run
removed	an eval that existed then and does not now
no changes	the outcome set is identical

History never aborts a run

A history write failure warns and continues. The evaluation result is the product; the log is a convenience.