Fill in evals across a project

Scope: bulk authoring with fill, the one command in moose-tracevals that writes. This page assumes the vocabulary from Declare what to check and is about volume and trust.

What fill does

fill scans a project for instruction artifacts. It asks a model to extract the evals each artifact’s own instructions already imply, then appends the ones that survive a gate. It needs no trace.

This is the only command that touches your files, so here is exactly what it refuses to do:

• It never modifies an existing eval. A name collision fails the run.
• It only rewrites the frontmatter block. Everything from the closing --- onward is byte-identical, including CRLF line endings and a byte-order mark if you had one.
• It never writes project rules. See below.
• It is never called by run. Evaluation is read-only. This is a separate authoring step you invoke on purpose.

Dry-run first

Always. This is the fastest way to see what the tool thinks of your artifacts with no consequence:

npx moose-tracevals fill --dry-run

proposed  .claude/agents/doc-writer.md  +1 (reads-before-editing 0.90) — dry run, not written
          below 0.7: explains-its-reasoning 0.30
          needs sharpening: "Produce high-quality work." — "high-quality" has no measurable bar, so no session could fail it
proposed  .claude/commands/ship-it.md  +1 (reads-before-editing 0.90) — dry run, not written
          below 0.7: explains-its-reasoning 0.30
          needs sharpening: "Produce high-quality work." — "high-quality" has no measurable bar, so no session could fail it
proposed  .claude/skills/fix-bug/SKILL.md  +1 (reads-before-editing 0.90) — dry run, not written
          below 0.7: explains-its-reasoning 0.30
          needs sharpening: "Produce high-quality work." — "high-quality" has no measurable bar, so no session could fail it
proposed  AGENTS.md  +1 (reads-before-editing 0.90) — project rules are never written; copy what you want
proposed  CLAUDE.md  +1 (reads-before-editing 0.90) — project rules are never written; copy what you want

Threshold 0.7 · 6 evals proposed, none written (dry run)

Read the proposals

Each line is one artifact. Reading left to right:

• proposed / filled / skipped / no-op / error: what happened to this artifact. proposed means nothing was written, either because you passed --dry-run or because it is a project-rules file.
• +1 (reads-before-editing 0.90): how many evals were accepted, with each name and the model’s self-reported confidence.
• below 0.7: …: proposals rejected for confidence, named so you can see what you are missing.
• needs sharpening: …: an instruction too vague to test. See below.

Why a proposal was rejected

Here is where intuition misleads. Confidence feels like the filter, so lowering the threshold feels like it should let more evals through. It does not. Confidence is the last of six gates, and no confidence score overrides any of the five before it.

Proposals are checked in this order:

1. duplicate-name: an eval by that name already exists on the artifact.

2. grader-not-allowed: the grader’s scope does not fit this artifact type. Each type has an allowlist:

   Artifact type	Graders it may use
   skill	ai, tool-usage, file-access, regex
   slash command	ai, tool-usage, file-access, regex
   agent	ai, tool-usage, file-access
   project rules	ai, skill-invoked, tool-usage

   A slash command carries the skill list unchanged, because it is the same shape of artifact: an injected procedure graded over the span it governed.

   Two exclusions are deliberate. cost and json-output are whole-session graders. Tokens and the session’s final message are not attributable to whichever artifact was governing. A budget declared inside one skill would therefore silently constrain everything, so they are never proposed. turn-count is windowed and no longer has that problem, but it stayed out of the allowlist: widening it is a separate decision. And skill-invoked is excluded from skills, slash commands, and agents, because an artifact asserting that it was itself invoked is permanently green.

3. invalid-options: the grader itself refused the options. Every grader validates its own options, and a grader that cannot do so is rejected outright.

4. ungrounded-target: the proposal names something that does not exist here. The target may be a tool absent from the project’s vocabulary, or a skill the scan never found. It may be an absolute file path where a repository-relative suffix belongs.

5. low-confidence: and only now, below --confidence (default 0.7).

6. Cap. Survivors are sorted by confidence and truncated to --max-evals (default 8). The cap counts existing evals too, so an artifact already carrying eight gets none. The rest are reported as over per-artifact cap, so nothing disappears quietly.

That ordering is the point. A high-confidence proposal naming a tool your project does not have is still rejected. Confidence can only lose you an eval, never win you one.

Re-gating is free

The cache stores the raw, pre-gating proposal. Re-running with a different --confidence or --max-evals re-gates the cached proposals without another model call.

Instructions that cannot be tested

The needs sharpening notes are the most valuable thing on this page and the easiest to skim past.

When fill meets an instruction too vague to decide, such as “Produce high-quality work.”, it says so instead of converting it into an assertion that can never fail. A soft eval inflates your pass rate and hides the fact that nothing is being checked.

Treat these as a to-do list for the artifact. The fix is upstream: rewrite the instruction so a session could observably fail it. "high-quality" has no measurable bar is telling you the instruction does no work.

Write and review

Drop --dry-run to write:

npx moose-tracevals fill

Everything lands in a reviewable diff. Read it before you commit it. The confidence score is the model’s opinion of its own proposal; you are the last gate.

Useful knobs:

npx moose-tracevals fill --confidence 0.85 --max-evals 4 --dry-run

Confidence and rationale are report-only: shown to you, never written into the artifact. Newly written evals are always type: regression, the conservative default; see capability vs regression.

Project rules are never written

CLAUDE.md, AGENTS.md, and GEMINI.md get proposals printed for you to copy by hand. They are never written automatically. That is a design decision, and here is the reasoning.

Project rules are read by the agent before it acts. An eval written into that file would tell the agent exactly what it is about to be measured on, which is teaching to the test. The measurement stops meaning anything the moment the subject can read it.

So the proposals are printed, and you decide which ones to carry over by hand. The reasoning is recorded in ADR 01005.

Slash commands are written, like skills and agents. What matters is not whether the agent reads the file. What matters is whether it reads that file’s frontmatter. Claude Code consumes a command’s description, argument-hint, and allowed-tools itself, and injects only the body as the prompt. An eval block there never reaches the session under test. Project rules are injected verbatim, frontmatter included, which is what makes them the exception.

Where to go next

Check your new evals fireRun against a session you know violated the rule, and confirm the failure names what actually happened.

Every fill flag--confidence, --max-evals, --max-cost-usd, --no-cache, --provider, and the JSON report shape.