Catch a broken rule before the agent hands back

Scope: judging the turn that just ended against the rules your repository declares for agents, and against what you asked for. It serves Maya, and it follows the post-hoc run, which grades a whole session after the fact. For the files a session used, see artifact coverage.

You wrote the rules. The agent read them. It still ran npm test in a fresh worktree before npm ci, and you found out in the diff. Fixing it costs a review cycle and a new prompt. The rule sat in the agent’s context the whole time, and nothing told it.

tracevals can check at the end of every turn instead. It reads the rules files that govern the agent, and what you asked it to do. It judges the last turn against the session so far, and names each broken rule with the source it came from. The agent fixes the work, or says why the rule does not apply, before the turn reaches you.

The same check catches a settings toggle you asked for and never got. It catches a task ticked with no work behind it, and a skill’s step run out of order.

Set it up

Install the manni plugin. It carries the hooks that run the check. To run the check by hand, skip this step.
Add one section to manni.config.yaml. An empty one is enough. It says turns can be judged in this repository, for everyone who opens it.
manni.config.yaml
```
tracevals:
  runtime: {}
```
Choose the models. Two roles do different work, so each gets a model of its own.
Turn on the plugin’s tracevals option for yourself. Open /plugin, then Installed, manni and Configure options, check Judge each turn with tracevals, and restart Claude Code. Turn on tracevals for your sessions covers the other ways to set it.
Run manni tracevals prepare once. The first turn then finds its rules cached and its local model on disk.

A committed runtime section alone runs nothing on a contributor’s machine. Each person whose turns should be judged turns the option on, and Claude Code keeps that answer in their own settings. A contributor who never turns it on gets no manifest, no model download and no judgement.

Setup	The tracevals row in the session briefing
No `runtime` section	`not checked judges recorded sessions with a model`
The section, and the option off	`not checked runtime evals are set up, and turned off here`
The section, and the option on	`judges each turn`, with both models

To turn the check off for yourself, turn the option off. To turn it off for everyone, delete the section.

Choose the models

Work inside the loop and outside it have different budgets.

Role	Does	Set by
Out of the loop	Extracts rules from files, runs `prepare`, and judges `check` by hand	`tracevals.provider` and `tracevals.model`
In the loop	Judges a turn inside a Stop or SubagentStop hook	`tracevals.runtime.hook`, then the keys above

The hook runs on every turn, so it wants a fast model. Extraction runs once per file version, so it can afford a stronger one. These recipes are examples. tracevals pins no model.

tracevals:
  provider: anthropic
  model: claude-sonnet-5-5
  runtime:
    hook: { provider: anthropic, model: claude-haiku-4-5 }

tracevals:
  provider: llama-cpp
  model: qwen3.5-4b
  runtime: {}

tracevals:
  provider: anthropic
  model: claude-sonnet-5-5
  runtime:
    hook: { provider: jev, model: jev-latest }

Jev only decides, so it can judge inside a hook and can never be tracevals.provider. Its key comes from TYPESAFE_API_KEY. Every key is in the configuration reference.

Which sources count

Resolution is deterministic, from the filesystem and the trace. No model decides which sources count. A model only reads the sources that do and returns their rules.

The list of formats is closed. Each one is scoped by its own trigger, as the tool that owns it scopes it.

Format	Files	In scope when
Claude memory	`CLAUDE.md`, `.claude/CLAUDE.md`, `CLAUDE.local.md`, and the one in your Claude config directory	Always for the working directory and its ancestors. A subdirectory’s file applies once the session touches a file under it.
Claude rules	`.claude/rules/*/.md`, and the same under your Claude config directory	Without `paths:`, always. With `paths:`, once the session touches a match.
`AGENTS.md`	`AGENTS.md` and `.claude/AGENTS.md`, nested too	As for Claude memory.
`GEMINI.md`	`GEMINI.md`, the one in `~/.gemini`, nested too	As for Claude memory.
Cursor rules	`.cursor/rules/*/.mdc`, and `.md` files there with front matter	`alwaysApply: true` means always. `globs` means once the session touches a match. Others apply once the agent reads them.
Kiro steering	`.kiro/steering/*.md`, and the same under `~/.kiro`	`inclusion: always`, or none, means always. `fileMatch` follows `fileMatchPattern`. Others apply once read.
Spec Kit constitution	`.specify/memory/constitution.md`	Always
OpenSpec project	`openspec/project.md`	Once the agent reads it
Imports	Files an `@path` line pulls into a memory file, up to four hops	With the importing file
Skills, commands, agents	An invoked skill’s `SKILL.md` and every file read under its directory, a slash command, and the running subagent’s definition	A skill or command from its first invocation to the end of the session. A subagent’s definition for its own run.
Designated	Files matching `runtime.include`	Always

To touch a file is to read, write or edit it. That is Claude Code’s own trigger for path-scoped rules. paths:, globs and fileMatchPattern each accept one glob or a list. A nearer file overrides a farther one, and a prompt you typed overrides any rule. Doing what you asked is never a violation.

Reading a file never makes it a rule

A README, a docs page, a design note or an API reference describes other systems or other readers. Judging the agent against one would fail it for sentences never meant for it. Two things keep those out.

The list of files. It is closed. include adds files you name, and it is empty by default. A repository whose CLAUDE.md tells agents to consult docs/content-strategy/ would name that tree there.
The sentences. Extraction keeps only directives addressed to the agent working in the repo. A description of a system, an instruction for end users, an example or a quoted rule yields nothing. A skill reference that documents an API yields no rules, though its skill counts.

exclude removes a file from every row, for a product docs tree or a stale .cursor/rules file. That covers plan, spec and output style files too.

Spec-driven toolkits

Three spec-driven toolkits keep agent guidance apart from specs, and tracevals reads both. Spec Kit’s constitution, Kiro’s steering files and OpenSpec’s project.md govern the agent, as the table above says. Their specs say what to build. Each counts once the session touches it, as What the session was asked says.

What a rule looks like

Each rule has an id, its text, and a when. The id is kebab-case, or a spec’s own id. The when uses the same grammar as conditional triggers, so a rule about commands arms only for the turns that ran one.

{ "id": "run-npm-ci-first",
  "text": "Run npm ci first when working in a worktree.",
  "when": { "command-matches": "\\bnpm (test|run)\\b" } }

A rule a transcript cannot show, such as “prefer boring code”, is dropped. A file that declares its own ai evals is not extracted. Its eval assertions are its rules, with their own when. Extraction is cached under .manni/tracevals/cache/rules. The key is the provider, the model, the extraction prompt and the sha256 of the whole source. So a source pays for extraction once per version. Rules files and the sources from what you asked are read by two different prompts.

What the session was asked

A rules file says how to work. What you asked for says what to build, and a turn can call that work done when it is not. Six more formats carry it.

Format	Source	In scope when
Typed prompts	Every prompt you typed in the session, read as one numbered sequence	Always
Approved plan	The plan of the last approved `ExitPlanMode`, from its input or the plan file written before it	From the approval on
Spec Kit spec	`specs/<feature>/spec.md`, `plan.md` and `tasks.md`	Once the session touches a file under `specs/<feature>/`
Kiro spec	`.kiro/specs/<name>/requirements.md`, `design.md` and `tasks.md`	Once the session touches a file under that directory
OpenSpec change	Every `.md` file under `openspec/changes/<id>/`	Once the session touches a file under that change. An archived change never counts.
Plans	A file that `runtime.plans` names	Once the session touches it

Prompts. Extraction returns the items that stand after your last prompt, so “forget the toggle” drops the toggle. A question with nothing to build yields no rules. Each new prompt costs one extraction.
A plan. A later approval replaces the plan. A rejected plan never counts.
Specs. Only a touched feature counts, so a tree of many specs brings in only the one in play. A rule keeps the spec’s own id, such as FR-001, T014 or 1.2.

Plans in a format of your own go in runtime.plans. A folder of plans holds plans for other work too, so a file counts only once the session touches it. include is for files that govern every turn, which is why plans have a key of their own.

tracevals:
  runtime:
    plans: [docs/plans/*.md]

exclude removes a plan or spec file here too. Your typed prompts are not a file, so a glob never matches them. exclude drops them by the name prompt, and a plan given only as input by plan.

A rule from what you asked is judged like any other. “Add a dark mode toggle” does not apply yet while the work goes on. It is followed once the toggle exists. It is broken on the turn that says the work is done without one.

Ticks

A ticked task is the agent saying that task is done. An Edit that turns - [ ] T014 … into - [x] T014 … is a tick. Spec Kit, Kiro and OpenSpec all keep tasks this way. The judge sees each earlier tick by its id, so a tick with nothing behind it reads as a claim to check. A Write that rewrites the whole tasks file is not read as a tick.

The system prompt the session ran under

Claude Code records its system prompt in the transcript, as a prompt_snapshot entry. It writes one at session start, and again after a compaction or a resume. tracevals reads the last one at or before the turn as one more source, system-prompt. A session that records none has no such source, and there is nothing to configure.

Who wrote the prompt decides what a break does.

Prompt	How tracevals tells	A break
Claude Code’s default	It holds Claude Code’s boundary marker.	Reported with `!` by `tracevals check`. It never blocks and never fails `check`. A hook skips it.
Replaced, with `--system-prompt` or `--system-prompt-file`	The boundary marker is missing, and the trigger ends `, custom`.	Blocks, like any rule.
A user output style	A file under `.claude/output-styles/` or `~/.claude/output-styles/` defines it.	Blocks, under the style file’s path.
Appended, with `--append-system-prompt` or `--append-system-prompt-file`	It cannot be told apart. It is a last block with no name, as the desktop app’s own block is.	Counts as default, so it is reported.

An output style. A user style is a source of its own, and its trigger is style <name>. Its content is the text the transcript recorded, which is what the agent saw. A built-in style has no user file, so its text joins the default prompt.
Redaction. The prompt names paths under your home directory. So judge.redact scrubs the prompt and a style’s text before extraction.
Cost. Extraction is cached by the redacted text, so one Claude Code version pays once per project. The default prompt yields many rules. A Stop or SubagentStop hook never extracts or judges them, because a warning could not block. Run tracevals check to see them.
The judge. A rule from the system prompt is a default. A rules file, or a prompt you typed, overrides it.

The fixture’s default prompt says never to claim the tests pass without showing them, and its last turn does.

$ cd ../project
$ manni tracevals check ../traces/system-prompt.jsonl --project . --no-cache
system-prompt
  ! never-say-tests-pass-without  Never say `Tests pass` without showing the test output.
      not-followed 100, followed 0, not-applicable 0. The turn shows `Tests pass`. (1.00)

Last turn of 3b265d00: 7 rules from 4 sources. None broken, 1 reported.

The ! row is a confident break of a default rule. The closing line counts it as reported, and the command exits 0. The same rule in a replaced prompt prints ✖, and the command exits 1.

To leave the system prompt out, name it in exclude. The names prompt, plan and system-prompt match as whole names, never as globs, so ** keeps them.

tracevals:
  runtime:
    exclude: [system-prompt]

Procedures are rules

A skill’s procedure is rules too. Extraction keeps its sentences about the order of steps, a gate one step must pass before another, and a route a step’s result chooses. It also keeps a condition that calls for a step, a required step, and the step that comes last. The judge reads each one against what the session did, which names every skill call and every agent spawn with its result.

{ "id": "follow-reviewer-route",
  "text": "Send the draft to the pass the Reviewer names in route_to." }

A skill’s rules stay in scope from its first invocation to the end of the session. A skill it calls partway through does not end that. A slash command’s rules work the same way. The post-hoc manni tracevals run still grades each skill against its own window.

Run it by hand

manni tracevals check judges the last turn of one session. A turn starts at the last prompt you typed, and tool results do not start one. Run it on yesterday’s session to see what the hook would have said. The examples below run from the committed runtime evals fixture, a small project with a CLAUDE.md that declares two rules and one AGENTS.md. Your typed prompts are a source on every turn, so the count includes them.

$ cd test/tracevals/fixtures/runtime/project
$ manni tracevals check ../traces/breaks.jsonl --project . --no-cache
CLAUDE.md
  ? run-npm-ci-first  Run `npm ci` before `npm test` in a fresh worktree.
      not-followed 0, followed 0, not-applicable 0. The turn does not show `npm ci`. (0.00)
  ✖ no-force-push  Never run `git push --force`.
      not-followed 100, followed 0, not-applicable 0. The turn shows `git push --force`. (1.00)
.cursor/rules/web.mdc
  ✖ never-use-innerhtml-in-components  Never use `innerHTML` in components.
      not-followed 100, followed 0, not-applicable 0. The turn shows `innerHTML`. (1.00)

Last turn of 3b265d00: 7 rules from 6 sources. 2 broken, 1 needs review.

The fixture’s config judges with a test provider that scores each rule 0 or 100. A real model scores in between, from what the turn shows. Each finding names the rule’s id, its text, its three scores and the judge’s reasoning, grouped under the source the rule came from. The number in parentheses is the not-followed score as a fraction. The ? row is a rule the judge could not settle. The ✖ rows are broken, so the command exits 1.

A rule from what you asked prints under its format’s name. In the fixture’s second project, a later prompt said not to edit src/legacy.ts, and the last turn edited it.

$ cd ../requests
$ manni tracevals check ../traces/requests.jsonl --project . --no-cache
prompt
  ✖ do-not-edit-src-legacy  Do not edit `src/legacy.ts`.
      not-followed 100, followed 0, not-applicable 0. The turn shows `src/legacy.ts`. (1.00)

Last turn of 3b265d00: 14 rules from 10 sources. 1 broken.

Its other sources are the approved plan, the touched Spec Kit and Kiro specs, an OpenSpec change and a file under docs/plans/. -f json lists each one with its trigger.

The other closing lines all exit 0.

Closing line	What it means
`Last turn of 3b265d00: 4 rules from 2 sources. None broken.`	Every rule that applied was followed.
`Last turn of 3b265d00: 7 rules from 4 sources. None broken, 1 reported.`	Each confident break came from Claude Code’s default prompt.
`Last turn of 3b265d00: 2 rules from 1 source, none apply to this turn.`	Each rule’s `when` failed over the turn. A rule that never armed is skipped, never passed.
`Last turn of 3b265d00: no rule sources governed it.`	No source was in scope. Your typed prompts are one, so this is rare.
`Last turn of 3b265d00: nothing happened after the last prompt.`	The turn is empty.

For a script, -f json prints the whole report and -o also writes it to a file. It lists every source that was in scope and why, so a rule you expected and did not get is easy to trace. Its fields are in Report and exit codes.

$ manni tracevals check ../traces/breaks.jsonl --project . --no-cache -f json

--provider, --model, --local and --runs change the judge only. Extraction keeps its own model, so prepare, the hook and check share one cache.

What the agent sees

At Stop, the hook judges the turn with the in-loop model. Only a confident violation blocks.

A broken rule. The hook blocks the stop once. The reason is the report above, with color off. The agent fixes the work, or says why the rule does not apply here, and finishes.
A broken default. A rule from Claude Code’s default prompt never blocks. When another rule blocks, the report shows the default’s break with !. The count in the reason leaves it out.
A rule the model could not settle. It shows as needs-review in check. It never blocks, and it never fails check.
A rule still broken after the repair pass. The agent stops, and you see one message. It names manni tracevals check and the transcript, so you can read what remains.

{"decision":"block","reason":"This turn broke 1 rule from the files and requests that governed it. Fix the work, or say why the rule does not apply here, then finish.\n\n<report>"}

The repair pass judges the whole turn once more. The turn has grown, so the verdict cache misses, and the pass costs one call. It is the only way to tell you a rule is still broken.

The judge asks about each rule on its own. It sees the last turn, what the session did before it, and that one rule. It first says in a sentence or two what the session shows about the rule. Then it returns three scores from 0 to 100: not applicable, followed, and not followed.

The scores read different spans. A rule applies once the session does the kind of work it covers, and followed reads the whole session. Not followed reads only the last turn. A rule is broken for what the last turn did, or for the last turn saying the work is done without it. So an old break blocks once. A later turn that neither repeats it nor claims the work is done does not block again. The three are independent, so a long record of keeping a rule never hides a break in the last turn.

Some rules cover a procedure that spans turns, such as running npm ci once before any test. So every call opens with what the session did before the last turn, read from the transcript. Each earlier turn gets a line with its commands, the files it wrote, the rule files it read and its skill calls. The line also names each agent it spawned, with the first line of the result. It names a question to you, each task it ticked and a plan you approved. So npm ci run three turns ago still counts.

# Earlier in this session

- turn 40: ran npm ci; ran skill authoring-workflow
- turn 52: spawned technical-nonfiction-reviewer, which returned {"route_to":"writer","issues":[…
- turn 61: wrote specs/001-login/tasks.md; ticked T014 in specs/001-login/tasks.md

That block takes at most a quarter of the judge’s budget. In a long session its oldest lines go first, and a marker line says which turns went.

The hook also keeps a ledger for each session. It records each turn’s facts and each judged rule’s results. Each rule shows the judge its earlier results, and keeps its first followed and every broken one in view. The ledger marks a stretch of turns that left the rule’s file out of scope. So the judge can tell a gap from a quiet history. A subagent keeps a ledger of its own.

manni tracevals check reads a session’s ledger and never writes it. Its Session so far section names each rule that was ever broken or needed review, and counts the rest.

Session so far
  CLAUDE.md#run-npm-ci-first  broken 1, repaired 1, followed 6
  31 rules held every turn they applied to.

The ledger’s shape is in Report and exit codes.

Only a confident violation blocks. A rule blocks when its not-followed score reaches judge.zones.autoFail as a percentage, in every run, and no run errored. It passes when its not-applicable or followed score reaches judge.zones.autoPass in every run. Anything else needs review, including a rule the judge errored on. runtime.hook.runs sets the number of runs in a hook, and it defaults to 1. Jev only returns probabilities, so it makes one call, and each option’s probability becomes its score.

When tracevals stays out of the way

A Stop runs a short list of cheap checks in order, and the first one that finds nothing to do ends the run. Most of them are silent.

Turned off for this user means tracevals never runs. The Stop runs the file checks alone and never reads the transcript. Only MANNI_TRACEVALS_HOOKS=1 turns it on, and the plugin sets that when its option is on.
No runtime section means tracevals never runs.
A judge’s own session never judges itself. tracevals sets MANNI_TRACEVALS_JUDGE on every process it spawns, so a claude-cli judge cannot set off the hook again.
An empty turn, or one with no rule source in scope, ends the run. Your typed prompts are a source, so a turn rarely has none.
A turn where every rule’s when fails is skipped, never passed.
A turn already judged, with the same rules, history and model, reuses its cached verdict.

Three local-model cases speak, once per session, and judge nothing. The model is still downloading, or it is not on disk, or it does not fit in free memory. Each message says which. A hosted model skips all three.

Prepare, release and the model host

manni tracevals prepare fetches a local model that is not on disk and extracts the rules of every file that applies to every session. For a user who turned tracevals on, the plugin runs it in the background at session start, so it never holds the session up. The first turn then finds both ready. Prompts, plans and specs are not known at session start, so the turn that first has them extracts them.

A local model loads once and stays loaded in a model host. That is one background process per machine and user. Every hook is its client, and requests queue in the order they arrive. Without the host, each Stop would pay a model load, and parallel subagents would each load their own copy.

A session takes a lease on the host when it starts, and manni tracevals release returns it at session end. The plugin runs neither for a user who left tracevals off. A model stays loaded while a lease is held and used within providers.llama-cpp.keepAlive, which defaults to 10m. A crashed session’s lease lapses when it goes idle. With no lease left, the model unloads, and the host exits when nothing is loaded.

By hand, check uses a host that is running and never starts one, so CI and one-off runs leave no process behind. To free the memory right now, run manni tracevals release --all.

CLI referenceEvery flag of check, prepare and release, with the messages and exit codes.

ConfigurationEvery runtime key, the two model roles and the provider precedence.

Keep an agent's docs greenThe plugin whose hooks run the check at Stop and SubagentStop.