Skip to content

CLI reference

Eight commands under one subcommand. Every flag below is verified against src/tracevals/cli.ts, and the exit codes are exercised by the inline tests at the foot of this page.

Terminal window
manni tracevals <command> [command options] [arguments]

There is no default subcommand. manni tracevals with no verb prints the command list and exits 2, and a trace path with no verb is an unknown command. Every invocation names run, calibrate, fill, capture, prepare, check, release, or list.

run and calibrate take positional trace files, and check takes exactly one. fill takes positional artifact paths. list, capture, prepare and release take none: they read the session store or a hook payload instead. tracevals declares no collection of its own, and a collection never chooses which traces a run covers. Declared collections are still read, so that evals relocated into an external manifest are graded rather than silently missing.

FlagCommandsMeaning
-c, --config <path>run, calibrate, fill, capture, prepare, checkPath to manni.config.yaml. Without it, manni tracevals looks for that file from the working directory up to the repository root. It reads the file’s tracevals: key, plus the top-level providers: and collections: keys.
--no-configsame as aboveIgnore any discovered config file and run on built-in defaults. -c and --no-config set the same option, so the one written later wins. With no config there are no collections, so every artifact’s evals are its own front matter’s.
--project <dir>run, calibrate, fill, capture, prepare, check, listWhere artifacts are looked up. run, calibrate and check override the cwd the trace recorded; fill and list default to the working directory; capture and prepare default to the hook payload’s cwd.
--provider <name>run, calibrate, fill, checkauto, anthropic, openai, claude-cli, or llama-cpp. check also accepts jev. Selection runs through the family’s shared provider layer: --local, then this flag, then the eval’s own provider:, then tracevals.provider, then providers.provider, then auto. An unknown name is exit 2.
--model <model>run, calibrate, fill, checkThe model within the provider. It needs a named provider, from this flag or from config, because a model name does not say which provider owns it. No model id is hardcoded anywhere in the tool.
--localrun, calibrate, fill, checkRun inference on this machine with llama-cpp, over every configured or eval-level choice, with one notice per choice it replaced. --local next to a hosted --provider is a contradiction: exit 2.
-f, --format <format>all eightOutput format. run accepts pretty, json, markdown, github, sarif, and junit. calibrate accepts pretty, json, and markdown. fill, capture, prepare, check, release, and list accept pretty and json. Anything else is a usage error, with the allowed set in the message.
--require <module>run, calibrate, fillLoad a grader plugin. Repeatable, one module per occurrence, never comma-separated. It adds to tracevals.plugins rather than replacing it, and a specifier that will not import is an error rather than a skip.

Flags override config; they do not bypass it. An unset flag falls through to the resolved config value, so a config file and the command line reach the same code paths.

Each command’s table below lists every flag it accepts, the shared ones included. A table is the whole surface of its command.

Global options are accepted before the subcommand.

OptionDescription
-V, --versionPrint the manni version and exit.
-h, --helpPrint help for the program or a subcommand.

The session adherence tool. Every command on this page lives under it. tracevals accepts its own -V, --version, which reports the manni version, and -h, --help.

OptionDescription
--no-colorDisable colored output. Color is on only when stdout is a TTY, and NO_COLOR turns it off too.
Terminal window
manni tracevals <command> [command options] [arguments]

Evaluates one or more traces against the skills, agents, slash commands, and project rules they used. Artifacts are resolved from the trace, evals are read from those artifacts, deterministic graders run first, then the judge.

Terminal window
manni tracevals run [traces...] [options]
Terminal window
$ manni tracevals run test/tracevals/fixtures/traces/claude-session.jsonl \
--project test/tracevals/fixtures/project --deterministic-only
manni tracevals — test/tracevals/fixtures/traces/claude-session.jsonl
claude-code · claude-opus-4-8 · 4 turn(s) · C:\work\demo-project
PASS fix-bug › used-read
FAIL fix-bug › forbidden-tool
[error] tool Bash was used 1 time(s) but must not be. Redo the session work so it holds, or fix the eval if the artifact does not ask for this.
test/tracevals/fixtures/project/.claude/skills/fix-bug/SKILL.md forbidden-tool: The session never ran shell commands; this skill is edit-only.
Guide: https://hawkeyexl.github.io/manni/tracevals/triage/
REVIEW fix-bug › refactor-preserved-intent
Awaiting human review. Read the session against the assertion and decide.
test/tracevals/fixtures/project/.claude/skills/fix-bug/SKILL.md refactor-preserved-intent: The fix addressed the reported bug rather than working around it.
Guide: https://hawkeyexl.github.io/manni/tracevals/triage/
SKIP fix-bug › eval-5
ai evals skipped (deterministic-only run)
16 eval(s): 7 pass, 1 fail, 0 error, 1 needs-review, 7 skipped
ArgumentDescription
[traces...]Trace files to evaluate, space-separated. Optional. With none given and a TTY on both ends, an interactive picker opens. Without a TTY it is exit 2, because a scripted caller must name what it graded.
OptionArgumentDefaultDescription
-c, --config<path>discoveredPath to manni.config.yaml. See Common conventions.
--no-confign/aoffIgnore any discovered config file.
--project<dir>the trace’s recorded cwdArtifact-lookup root. Use it when the session ran somewhere this checkout is not.
--provider<name>config provider, else autoJudge provider: auto, anthropic, openai, claude-cli, or llama-cpp.
--model<model>config model, else the provider’s defaultJudge model. Needs a named provider.
--localn/aoffRun inference on this machine with llama-cpp, over every configured or eval-level choice.
--runs<n>config judge.ensembleRuns, which is 3Ensemble runs per judged eval. A whole number of at least 1; anything else is exit 2.
--deterministic-onlyn/aoffSkip every LLM-judged eval, which then reports skipped with that reason. No provider is constructed, so this is the offline path.
--offlinen/aoffNever fetch a remote manifest. External metadata is read from local manifests only, so a hosted eval trail is skipped rather than requested.
--no-cachen/aoffBypass the judge cache under judge.cacheDir.
--max-turns<n>config judge.maxTurns, unboundedStop after this many uncached ensemble runs. A cached ensemble costs none. Evals left unjudged report skipped with reason: "turn budget".
-f, --format<pretty|json|markdown|github|sarif|junit>prettyOutput format. github prints workflow annotations on the lines that declare each eval, then the markdown summary. sarif files each result under the rule tracevals/<grader>. junit reports needs-review as skipped. An unknown value is a usage error (exit 2), with the allowed set named.
-o, --output<file>n/aAlso write the report to a file, in the chosen format. stdout still gets it.
--report-unused-artifactsn/aoffList every skill and agent the session was offered and never used. Without it the counts are still reported, and only the names are withheld.
--fail-on-reviewn/aconfig failOnReview, which is trueTreat needs-review as a failure, so an eval the judge could not decide exits 1.
--no-fail-on-reviewn/aoffDo not fail the run on needs-review. Declared as the pair to the flag above so that the config value stays reachable from the command line.
--allow-execution<kind>config execution.allow, which is every grantKeep only the named execution grants. Repeatable, one grant per occurrence. The one grant is frontmatter-commands, which runs command evals. It narrows the config and never widens it. Any other value is exit 2.
--no-executionn/aoffClear every execution grant for this run. Every command eval reports skipped, never pass.
--require<module>config plugins, else noneLoad a grader plugin. Repeatable; appends to the configured list.
--all-projectsn/aoffEvaluate every project’s traces in the session store rather than this project’s.
--newer-than<duration>n/aOnly traces whose file changed within this window, such as 30m, 24h, 7d, or 2w. Any other spelling is exit 2.
--limit<n>n/aMaximum traces to evaluate. A whole number of at least 1.
--historyn/aoffAppend this run to history.file and compare it with the previous run, so a regression is a check that passed before and does not now.
--manifest<file>looked for beside the trace and under the projectThe session manifest to compare artifacts against. One session’s evidence, so it cannot be combined with a corpus.

What decides the report shape is how traces were selected, not how many came back:

  • One named trace is a single-run report. The pretty form is the per-eval list above; --format json is the RunReport object.
  • Two or more named traces, or any discovery selector (--all-projects, --newer-than, --limit), is an aggregate report. It carries eval pass rates across the corpus, the traces that failed, and the skip reasons that survived aggregation.

A selector that matches exactly one trace still produces the aggregate shape, so a script reading --format json gets one answer rather than two. Field by field, both shapes are in Report and exit codes.

Naming traces and selecting them are alternatives, not a filter applied to a list. Doing both is a usage error:

Terminal window
$ manni tracevals run a.jsonl --newer-than 7d --deterministic-only
manni: name traces or select them with --all-projects/--newer-than/--limit, not both

The selectors compose with each other. --all-projects --newer-than 30d --limit 20 reads the whole session store, keeps the last thirty days, and evaluates the twenty most recent of those.

run looks for a session manifest beside the trace and under the project, and reports the one it used. Without a manifest, “was this artifact edited after the session?” is answered from mtime, which is a heuristic and says so in the warnings. With one, the answer is content identity. --manifest names a file outright, which only a single trace can mean:

Terminal window
$ manni tracevals run a.jsonl b.jsonl --manifest m.json --deterministic-only
manni: --manifest names one session's manifest, so it cannot be used with a corpus; drop it and each trace will find its own

capture is what writes one.

Measures judged and graded verdicts against a labels file: false passes, false fails, and how much review volume the current settings produce. It is a run plus the labels it is measured against. It stays a separate command because the report answers a different question and carries a different shape.

Terminal window
manni tracevals calibrate [traces...] [options]
ArgumentDescription
[traces...]Trace files to score, space-separated. Optional; the same discovery selectors apply.
OptionArgumentDefaultDescription
--labels<file>config calibrate.labels, which is tracevals/labels.yamlThe calibration labels sidecar, holding the human verdict each eval is scored against.
--sweepn/aoffRe-score the corpus across the configured grid of zones and ensemble sizes, from cached verdicts. It makes no further model calls.
--max-false-pass<n>n/aExit 1 above this many false passes. A whole number of at least 0.
--max-false-fail<n>n/aExit 1 above this many false fails.
--max-review<n>n/aExit 1 above this many needs-review outcomes.
-c, --config<path>discoveredPath to manni.config.yaml. See Common conventions.
--no-confign/aoffIgnore any discovered config file.
--project<dir>the trace’s recorded cwdArtifact-lookup root.
--provider<name>config provider, else autoJudge provider.
--model<model>config model, else the provider’s defaultJudge model. Needs a named provider.
--localn/aoffRun inference on this machine with llama-cpp.
--runs<n>config judge.ensembleRuns, which is 3Ensemble runs per judged eval.
--deterministic-onlyn/aoffSkip LLM-judged evals. A calibration that scored nothing is never green.
--offlinen/aoffNever fetch a remote manifest.
--no-cachen/aoffBypass the judge cache.
--max-turns<n>config judge.maxTurns, unboundedStop after this many uncached ensemble runs.
-f, --format<pretty|json|markdown>prettyOutput format.
-o, --output<file>n/aAlso write the report to a file.
--report-unused-artifactsn/aoffList every skill and agent the session was offered and never used.
--fail-on-reviewn/aconfig failOnReview, which is trueTreat needs-review as a failure.
--no-fail-on-reviewn/aoffDo not fail on needs-review.
--allow-execution<kind>config execution.allow, which is every grantKeep only the named execution grants. Repeatable.
--no-executionn/aoffClear every execution grant for this run.
--require<module>config plugins, else noneLoad a grader plugin. Repeatable.
--all-projectsn/aoffScore every project’s traces in the session store.
--newer-than<duration>n/aOnly traces whose file changed within 30m, 24h, 7d, 2w, and so on.
--limit<n>n/aMaximum traces to score.

calibrate takes no --history and no --manifest. Calibration measures a corpus rather than a point in one session’s timeline, and a manifest belongs to exactly one session.

Disagreement on its own exits 0, because a measurement is not a failure. The three thresholds are what turn the measurement into a gate. The journey is in Calibrate.

Proposes evals for skills, agent definitions, and project rules, and writes those above the confidence threshold. This is the one write path that grades nothing, and run never calls it.

Terminal window
manni tracevals fill [paths...] [options]
Terminal window
$ manni tracevals fill test/tracevals/fixtures/project --provider mock --dry-run --no-cache
proposed test/tracevals/fixtures/project/.claude/skills/fix-bug/SKILL.md +1 (reads-before-editing 0.90) — dry run, not written
below 0.7: explains-its-reasoning 0.30
needs sharpening: "Produce high-quality work." — "high-quality" has no measurable bar, so no session could fail it
proposed test/tracevals/fixtures/project/CLAUDE.md +1 (reads-before-editing 0.90) — project rules are never written; copy what you want
Threshold 0.7 · 6 evals proposed, none written (dry run)
ArgumentDescription
[paths...]Artifact files or directories to scan, space-separated. Optional; without them, --project, which defaults to the working directory.
OptionArgumentDefaultDescription
-c, --config<path>discoveredPath to manni.config.yaml. See Common conventions.
--no-confign/aoffIgnore any discovered config file. With no config there are no collections, so evals are read from and written to front matter.
--project<dir>the working directoryProject root to scan for skills, agents, slash commands, and project rules.
--exclude<glob>n/aGlob to exclude. Repeatable, one glob per occurrence, never comma-separated.
--dry-runn/aoffReport proposals without writing them. Run this first: you see every proposal, and what the pass cost, before anything touches the repository.
--confidence<n>config fill.confidenceThreshold, which is 0.7Minimum confidence to write, from 0 to 1. Outside that range is exit 2.
--max-evals<n>config fill.maxEvalsPerArtifact, which is 8Maximum evals per artifact, counting the ones already declared.
--max-turns<n>config fill.maxTurns, unboundedStop after this many inference calls. A cached artifact costs none.
--no-cachen/aoffBypass the proposal cache under fill.cacheDir. Proposals are cached before the confidence gate, so re-running at a different --confidence spends nothing.
--provider<name>config provider, else autoProvider that writes the proposals.
--model<model>config model, else the provider’s defaultModel. Needs a named provider.
--localn/aoffRun inference on this machine with llama-cpp.
--offlinen/aoffNever fetch a remote manifest.
--require<module>config plugins, else noneLoad a grader plugin, so its kinds are proposable. Repeatable.
-f, --format<pretty|json>prettyOutput format.

There is no --collection. A documentation collection is not an artifact set, and fill finds artifacts under --project instead. Declared collections are still read. An artifact whose evals were relocated into a manifest is filled back into that manifest, not duplicated into its front matter. Two collections owning one artifact’s metadata is exit 2.

Project rules are proposed and never written. What fill does write is recorded in the artifact’s metadata.meta-provenance, in the same edit. The journey is in Propose evals with fill.

Records a session manifest, which holds the sha256 of every instruction artifact plus the git SHA. It reads a Claude Code hook payload on stdin. It is the second write path, and run never calls it either.

Terminal window
manni tracevals capture [options]
OptionArgumentDefaultDescription
-c, --config<path>discoveredPath to manni.config.yaml. See Common conventions.
--no-confign/aoffIgnore any discovered config file.
--project<dir>the payload’s cwdProject root to scan for instruction artifacts.
--session-id<id>the payload’s session_idSession id the manifest is filed under. Required when no payload is piped in.
-o, --out<file>capture.dir, which is .manni/tracevals/sessions/<id>.jsonWrite here instead of the configured directory.
-f, --format<pretty|json>prettyOutput format.

The manni Claude Code plugin runs capture at the start of every session for a user who turned on its tracevals option. There is nothing to wire. Install the plugin and turn tracevals on, and the manifest is written for you. To run capture without the plugin, add a hook to .claude/settings.json. capture takes its payload on stdin, which is what a SessionStart hook supplies:

{
"hooks": {
"SessionStart": [
{
"hooks": [{ "type": "command", "command": "manni tracevals capture" }]
}
]
}
}

With a payload on stdin, nothing goes to stdout. A SessionStart hook’s stdout is added to the model’s context. A report printed there would be a side effect on the session being observed. In that mode the report goes to stderr instead, where an exit-0 hook’s output is a debug channel:

Terminal window
manni: Captured session 11111111-1111-1111-1111-111111111111
manifest .manni/tracevals/sessions/11111111-1111-1111-1111-111111111111.json
artifacts 6 hashed
commit fe7ea9b231ff683aa64eff48f28021f7764b4e45 (main)
device e1dafb2151ca3cbc (win32)

Run by hand, with --session-id and no payload, the same report goes to stdout. Either way the manifest is the only file written. Why it is worth wiring up is in Artifact coverage.

Fetches the local models judging needs and extracts the rules of every file that applies to every session. The manni plugin runs it in the background when a session starts, for a user who turned on its tracevals option. A person or a CI runner can run it too, so the first turn finds its rules cached and its model on disk. With no tracevals.runtime section it says so and exits 0.

Terminal window
manni tracevals prepare [options]
Terminal window
$ manni tracevals prepare --project . --no-cache
Extracted 2 rules from 1 file that applies to every session.
OptionArgumentDefaultDescription
-c, --config<path>discoveredPath to manni.config.yaml. See Common conventions.
--no-confign/aoffIgnore any discovered config file.
--project<dir>the payload’s cwd, else the working directoryProject root.
--no-cachen/aoffExtract again, even from cached sources.
--offlinen/aoffRefuse network providers, so only local models are fetched. A hosted extraction model is then skipped, and extraction in the JSON report is null.
-f, --format<pretty|json>prettyOutput format.

A hook payload on stdin is detected, as capture detects one. With a payload, nothing goes to stdout and the report goes to stderr, because a SessionStart hook’s stdout becomes model context. Under a SessionStart payload with a local hook model, prepare also takes the session’s lease on the model host. It loads the model too, so the first turn finds it warm.

MessageStreamExit
tracevals runtime evals are not set up; nothing to prepare.stdout0
manni: could not download <model>: <first line of the error>stderr2

Judges the last turn of one session against the rules that governed it. The turn starts at the last prompt the user typed. By hand it uses the out-of-loop model and judge.ensembleRuns. The journey is in Check each turn.

Terminal window
manni tracevals check [traces...] [options]
Terminal window
$ cd test/tracevals/fixtures/runtime/project
$ manni tracevals check ../traces/breaks.jsonl --project . --no-cache
CLAUDE.md
? run-npm-ci-first Run `npm ci` before `npm test` in a fresh worktree.
not-followed 0, followed 0, not-applicable 0. The turn does not show `npm ci`. (0.00)
✖ no-force-push Never run `git push --force`.
not-followed 100, followed 0, not-applicable 0. The turn shows `git push --force`. (1.00)
.cursor/rules/web.mdc
✖ never-use-innerhtml-in-components Never use `innerHTML` in components.
not-followed 100, followed 0, not-applicable 0. The turn shows `innerHTML`. (1.00)
Last turn of 3b265d00: 7 rules from 6 sources. 2 broken, 1 needs review.

Findings are grouped under the source each rule came from. A source that is not a file prints under its format’s name. The rules taken from your typed prompts print under prompt, and a plan given only as input prints under plan, and the system prompt prints under system-prompt. The closing line counts sources rather than files for the same reason.

ArgumentDescription
[traces...]One transcript file. A turn belongs to one session, so no trace, or two, is exit 2.
OptionArgumentDefaultDescription
-c, --config<path>discoveredPath to manni.config.yaml. See Common conventions.
--no-confign/aoffIgnore any discovered config file.
--project<dir>the trace’s recorded cwdProject root for resolving rule sources.
--provider<name>config provider, else autoJudge provider: auto, anthropic, openai, claude-cli, jev, or llama-cpp.
--model<model>config model, else the provider’s defaultJudge model. Needs a named provider.
--localn/aoffRun inference on this machine with llama-cpp, over every configured choice.
--runs<n>config judge.ensembleRuns, which is 3Judge calls per rule. A whole number of at least 1; anything else is exit 2. jev makes one call and ignores it.
--no-cachen/aoffBypass the rules cache and the verdict cache.
--offlinen/aoffRefuse network providers. Judge and extract with local models only.
-f, --format<pretty|json>prettyOutput format.
-o, --output<file>n/aAlso write the report to a file, in the chosen format.

The flags override the judge only. Extraction always uses tracevals.provider and tracevals.model, so prepare, the hook and check share one rules cache.

Exit 0 means no rule was broken with a confident verdict, needs-review included. Exit 1 means at least one was. A break of a rule from Claude Code’s default system prompt is a warning. It prints with ! in place of ✖, counts as reported, and never exits 1. Exit 2 is an operational or usage error. The closing line says which case applied:

Closing lineExit
Last turn of <session>: <n> rules from <n> sources. None broken.0
Last turn of <session>: <n> rules from <n> sources. None broken, <n> needs review, <n> reported.0
Last turn of <session>: <n> rules from <n> sources. <n> broken, <n> needs review, <n> reported.1
Last turn of <session>: <n> rules from <n> sources, none apply to this turn.0
Last turn of <session>: no rule sources governed it.0
Last turn of <session>: nothing happened after the last prompt.0

The needs-review and reported counts are left out when they are zero.

When the hook has kept a session ledger for the session, a Session so far section follows the findings. It never changes the exit code. It lists each rule the ledger ever recorded as broken or needing review, on one line, then counts the rest.

Session so far
CLAUDE.md#no-force-push broken 1, repaired 1, followed 4
6 rules held every turn they applied to.
Message on stderrExit
manni: no trace given; pass a trace file or run manni tracevals list2
manni: tracevals check takes one trace, got 22
manni: --runs must be a whole number of at least 1, got 02
manni: cannot read trace <path>: <reason>2
manni: <model> is not downloaded; run manni tracevals prepare2
manni: --offline runs no network provider, and <the judge or extraction> uses <provider>2
manni: jev needs an API key in TYPESAFE_API_KEY2
manni: jev answers decisions only, so it cannot extract rules or write verdicts. Use it as tracevals.runtime.hook.provider.2
manni: could not judge the last turn: <first line of the error>2

run and fill answer --provider jev with the same decisions-only message. The JSON report’s fields are in Report and exit codes.

Returns a session’s lease on the model host. The manni plugin runs it when a session ends, for a user who turned on its tracevals option. A person runs it with --all to free the memory now.

Terminal window
manni tracevals release [options]
Terminal window
$ manni tracevals release --all
No model host is running.
OptionArgumentDefaultDescription
--alln/aoffDrop every lease, unload every model and stop the host.
-f, --format<pretty|json>prettyOutput format.

release reads no config, so it takes no -c and no --no-config. Under a SessionEnd payload the session comes from session_id, and nothing goes to stdout. By hand, without --all, there is no session to release.

MessageStreamExit
No model host is running.stdout0
manni: release needs a session from a SessionEnd hook, or --allstderr2

Lists discoverable traces, newest first, and runs nothing. Use it to find the path a run should name.

Terminal window
manni tracevals list [options]
Terminal window
$ manni tracevals list --all-projects --limit 5
1. 2026-07-02 00:00 Write a README for this repo.
C:\work\other-project · 22222222-2222-2222-2222-222222222222 · 1KB
...\projects\C--work-other-project\22222222-2222-2222-2222-222222222222.jsonl
2. 2026-07-01 00:00 Fix the crash in src/app.ts.
C:\work\demo-project · 11111111-1111-1111-1111-111111111111 · 1KB
...\projects\C--work-demo-project\11111111-1111-1111-1111-111111111111.jsonl

The session store is detected, not configured. list reads Claude Code’s own CLAUDE_CONFIG_DIR and falls back to ~/.claude, so a CI job points at a fixture tree with the same variable the agent reads. Nothing found is exit 0 with a line saying which flag would widen the search.

OptionArgumentDefaultDescription
--project<dir>the working directoryProject directory to scope to.
--all-projectsn/aoffScan every project in the session store.
--newer-than<duration>n/aOnly traces whose file changed within this window, as on run. Use it to see what a run with the same window would evaluate.
--limit<n>n/aMaximum traces to list. A whole number of at least 1, because --limit -1 would quietly list everything but the oldest.
-f, --format<pretty|json>prettyOutput format.

list takes no -c and no --no-config. It enumerates the session store and reads no config at all, and an accepted flag that quietly does nothing is worse than an absent one. It takes the long spellings of --project and --all-projects only, matching run.

CodeMeaning
0Everything that ran passed.
1A failed or errored eval, or a calibration threshold that was exceeded. The author’s problem.
2Operational or usage error, such as a bad flag, an unreadable trace, invalid config, or a contradictory provider. Not the author’s problem.

Every operational message leaves through the family’s stderr writer, so it carries the manni: prefix and nothing spells that prefix by hand. Full detail in Report and exit codes.