Evolve the eval standard

Scope: the evals block as a shared contract, and changing it safely. The block’s fields are in the evals schema reference; tuning how a judged eval is decided is Calibrate the judge.

The block is a shared contract, not ours alone

The shape of metadata.evals is not an implementation detail, and it is not this tool’s to change. It implements manni:artifact-evals:1.1.0, a built-in vocabulary published by manni meta that any session-grading tool can implement. manni tracevals implements behavior against it.

Two consequences for you:

• A shape change is a vocabulary conversation. If a field is missing, the place to argue for it is a proposal against the vocabulary, not this tool’s issue tracker. What lands there lands for every implementer at once, manni docevals and manni meta fill included.
• Your artifacts are portable. An artifact carrying evals is not locked to this runner.

The schema is published at https://hawkeyexl.github.io/manni/schemas/artifact-evals/1.1.0.json, and the package carries the same bytes. A finding about the block names manni:artifact-evals:1.1.0 as its schema. Copy that id into manni meta validate -s and it resolves.

Do I need to pin anything?

Not to author evals. manni tracevals validates against the vocabulary it was built with, and you write the block. Pinning matters when your own tooling validates these blocks and you need the shape to hold still. Name the exact id, such as manni:artifact-evals:1.1.0, in your config. Ids resolve by exact string, so nothing moves to a later version without a config edit.

What a version bump means

manni meta versions these vocabularies with three-segment semver, and each segment is a claim about validation behavior rather than a mood:

Segment	Means	Example
MAJOR	a block that used to validate now fails	a new required field, a removed field, a narrowed enum
MINOR	a block that used to fail may now pass; every old one still validates	a new optional field, a widened enum
PATCH	no validation-behavior change at all	a corrected description

So a bump tells you directly whether your corpus is at risk: only MAJOR can break an artifact that validates today.

What changed on the way to 1.1.0

Two renames landed while the vocabulary was still a draft. Both are the kind that fail loudly rather than quietly:

Was	Is	What to do
metadata.eval-provenance	metadata.meta-provenance	Rename the key. The old spelling is now a reserved-prefix error, not an ignored member.
severity: info	severity: notice	Rename the value. Severity across the family is error | warning | notice.

The whole top-level metadata key carries x-manni-location: external. The mark sits on the map, not on one field, so evals, eval-skip and meta-provenance move as one block. They cannot be split across an artifact and a manifest. manni meta relocate moves that block into the external-metadata manifest a collection declares, which leaves the instructions a person reads uncluttered. tracevals reads the block wherever it lives. Collections and a relocated block has the rules.

1.1.0 is a MINOR release over 1.0.0. It names the tool-order grader, so the strict overlay accepts it. Every block that validated at 1.0.0 still validates, and 1.0.0 keeps its bytes.

The provenance rename is worth a second look, because the guard changed under it. The reserved prefix inside metadata is ^eval-(?!skip$): every eval-* member is rejected except metadata.eval-skip. eval-provenance used to be the one other exception, so an artifact still carrying it now fails validation with a pointer to the key.

That is deliberate. A provenance trail silently ignored is worse than one that stops the build, because the criterion axis of the self-preference check reads it. An artifact whose trail went unread would report no bias while the model graded its own assertion.

The judge prompt moved with it

PROMPT_VERSION is 4. Prompt version is part of the cache key, so a cache written under the old vocabulary misses once and the next run judges fresh. Nothing to clear by hand.

Adding a grader is a strict-overlay change

The grader field is an open enum in the vocabulary. Any kebab-case name validates, and the grader registry is what rejects an unknown one at run time. So a misspelled or retired name (llm, the pre-1.0 spelling of ai) passes the open vocabulary and fails during the run instead. The strict overlay closes the enum to the graders it names, so a new grader needs a new overlay version. That is why tool-order arrived with 1.1.0.

Separate probes from guards

type answers a question that otherwise makes a pass rate meaningless:

Value	Meaning	A failure means
regression	Protects behavior that already works.	Something broke. Act on it.
capability	Probes the edge of what the agent can do. Expected to fail sometimes.	Information about where the boundary is.

- id: handles-nested-monorepo
  assertion: The session located the correct package in a nested workspace without being told.
  type: capability
  grader: ai

Neither changes enforcement. A failing capability eval still fails and still contributes to a non-zero exit code. That restraint is deliberate: an eval that could not fail the build would just be a comment. The field instead lets you read a report or a history log and know that a red probe is not a regression.

New evals default to regression, including everything fill writes. That is the conservative choice, since mislabelling a real guard as a probe is the more expensive mistake.

`capability` is reported, not excused

If you want probes to stop blocking a pipeline, that is a severity decision, not a type decision. Set severity: warning so the finding reports and the eval passes. Use type for what the eval is, and severity for what should happen when it fails.

Roll out a convention in stages

The transferable technique, whether you are adding a field, a grader convention, or a tightened assertion: never flip a whole corpus to error at once.

1. Introduce it at warning. The eval reports on every run and fails nothing:

   - id: cites-a-test-run
     assertion: The session ran the test suite before declaring the fix complete.
     severity: warning

   notice is quieter still, for a convention you want recorded but not discussed yet.

2. Measure coverage. Count how many artifacts carry it, and how often it actually fires:

   npx @hawkeyexl/manni tracevals run <trace> --format json \
     | jq '[.evalResults[] | select(.evalName == "cites-a-test-run")]
           | group_by(.outcome) | map({(.[0].outcome): length}) | add'

3. Fix the artifacts, not the eval. A rule failing everywhere usually means the instructions never said it clearly. This is the step that does the work.

4. Promote to error once the corpus is ready. Now it gates.

The same ratchet works in reverse for tightening an existing assertion. Add the sharper eval under a new id at warning, let both run, and remove the old one when the new one is clean.

Why not just tighten in place?

Editing an assertion changes the eval plan, which changes the cache key. Every session is then re-judged against a rule that half your artifacts have never been written for. You get a wall of red that tells you nothing about whether behavior changed. Adding a second eval keeps the old signal intact while the new one settles.

A malformed block fails loudly

Rolling a change across many repositories is survivable mostly because breakage is precise. An invalid block is never silently ignored: it produces an eval named evals-block-valid with a JSON Pointer to the offending field and a source line number.

ERROR  fix-bug › evals-block-valid
       invalid metadata.evals block: /metadata/evals/1/grader (line 12): must be string

Eval entries are closed, so a misspelled field inside one is an error rather than a key that quietly does nothing. metadata itself must stay open, because it is the host tool’s extension bag. The eval- prefix is reserved inside it instead, so eval-skpi is rejected the same way, and so is the retired eval-provenance. When you roll out a convention and something breaks, you get the file, the pointer, and the line.

Keep ids stable

Eval ids are the join key for history. A regression is defined as this artifact and this eval id went from pass to fail. Renaming an eval therefore reads as one eval removed and another added, and the trend line resets.

This is exactly why id is required on the object form. The retired 0.2 shape let a name be derived from list position, so reordering entries silently reset every trend line that depended on them.

Rename when the meaning genuinely changed. Otherwise leave the id alone and edit the assertion.

Where to go next

Every eval fieldThe full metadata.evals block, the manni meta vocabulary behind it, and what each field does.

The vocabulary itselfartifact-evals as manni meta publishes it, with every key, the location marks, and the strict overlay.

Track it over timeThe history file, what counts as a regression, and how to compare two runs.