Declare what to check
Scope: authoring one eval by hand, deliberately. Proposing many at once is Fill in evals; tuning how judged evals are decided is Trust the judge.
The implicit eval
Section titled “The implicit eval”Before you declare anything, every artifact a session used already gets one eval:
SKIP CLAUDE.md › adheres-to-artifact (implicit)The assertion is “the session adhered to the instructions in this artifact”, judged against the artifact’s full text. That is why a first run says something with zero configuration, and also why you will quickly want more.
From implicit to declared
Section titled “From implicit to declared”The implicit eval is coarse on purpose. It answers did the session follow this file? as a single verdict, which is hard to act on when the file contains fifteen instructions and only one of them matters to you today.
A declared eval is named, scoped to one instruction, and stable across runs. You can point at it when it fails, and watch it specifically over time.
Evals live in the artifact’s own frontmatter, next to the instruction they check. Not in a
separate suite or a config file, but in the same SKILL.md you edit when the instruction changes.
The check and the rule then move together, so they cannot drift apart.
Declare your first eval
Section titled “Declare your first eval”Add a metadata.evals block to a skill, an agent definition, a slash command in
.claude/commands/, or a project-rules file:
---name: fix-bugdescription: Fix a reported bug, reproducing it with a failing test first.metadata: evals: # String shorthand: an LLM-judged assertion at error severity. - Reproduce the bug with a failing test before applying the fix.
# Object form: a deterministic grader. - id: forbidden-tool assertion: The session never ran shell commands; this skill is edit-only. grader: tool-usage options: tool: Bash expect: not-used---
# Fix a bug
...the rest of the file is untouched.Two forms, and the difference matters more than it looks:
- A string is shorthand for a judged assertion at
errorseverity. One line, no ceremony. It gets an automatic name derived from its position (eval-3for the third entry). That suits anything you will not need to refer to by name. - An object gives you a stable
name, a grader, and options. Use it for anything you want to track, reference, or decide deterministically.
The full block shape, including every optional field, is in the evals schema reference.
Judged or deterministic?
Section titled “Judged or deterministic?”This is the decision the whole page exists for, and the one most likely to be made badly.
Ask first: can this be decided from the trace alone? If yes, use a deterministic grader.
Reaching for the judge when tool-usage would answer the same question is a downgrade. You trade a
certain answer for a probabilistic one, and you pay for it.
| If the instruction is about… | Use | Grader |
|---|---|---|
| whether a tool was used, avoided, or used within bounds | deterministic | tool-usage |
| whether a skill was invoked | deterministic | skill-invoked |
| whether a file was read, written, or edited | deterministic | file-access |
| how long the conversation ran | deterministic | turn-count |
| what the session cost | deterministic | cost |
| whether specific text appeared | deterministic | regex |
| whether the final output is valid JSON of a given shape | deterministic | json-output |
| reasoning, sequence, judgement, or intent | judged | ai |
Every deterministic grader, its options, and its exact failure messages are in the grader reference.
The third answer is not testable at all
Section titled “The third answer is not testable at all”Some instructions are neither. “Produce high-quality work” has no measurable bar, so no session could fail it. Turning it into a judged assertion produces a check that is permanently green and tells you nothing.
When that happens, sharpen the instruction instead of softening the eval. A rule no session
could violate is a defect in the artifact. fill names these for you explicitly; see
instructions that cannot be tested.
Severity
Section titled “Severity”Every eval has a severity, and the default is error.
| Severity | Effect |
|---|---|
error |
a finding fails the eval |
warning |
reported, but the eval passes |
info |
reported, but the eval passes |
This is the quietest trap in the system in both directions. Declare everything at warning and
nothing will ever fail; assume warning is the default and you will be surprised the first time a
soft preference goes red.
warning earns its place during rollout: introduce a new convention at warning, drive coverage
across your artifacts, and promote it to error once they are ready.
Confirm it fires
Section titled “Confirm it fires”An eval you have not seen fail is an eval you should not trust yet. Run against a session you know violated the rule:
npx moose-tracevals run <trace-file> --deterministic-only FAIL fix-bug › forbidden-tool [error] tool Bash was used 1 time(s) but must not beThe finding names the artifact, the eval, and what the session actually did.
If your new eval shows ERROR instead, the check itself is broken and the session is fine.
Most often that is a grader given options it rejects. Grader options are validated before
grading so that a typo surfaces loudly:
ERROR fix-bug › forbidden-tool options.expect must be one of: used, not-used