Evolve the eval standard

Scope: the evals block as a shared contract, and changing it safely. The block’s fields are in the evals schema reference; tuning how a judged eval is decided is Calibrate the judge.

The block is a shared contract, not ours alone

The shape of metadata.evals is not an implementation detail, and it is not this tool’s to change. It implements docmeta:artifact-evals:1.0.0-proposal.1, a common vocabulary published by docmeta that any session-grading tool can implement. moose-tracevals ships a byte-identical copy of the schema and implements behavior against it.

Two consequences for you:

• A shape change is an upstream conversation. If a field is missing, the place to argue for it is docmeta’s proposal, not this repo. What lands there lands for every implementer at once.
• Your artifacts are portable. An artifact carrying evals is not locked to this runner.

import schema from "moose-tracevals/schemas/artifact-evals-1.0.0-proposal.1.json" with { type: "json" };

Do I need to pin anything?

Not to author evals. moose-tracevals validates against the copy it ships. Importing the schema matters when your own tooling reads or generates these blocks and you need the shape to hold still.

`-proposal.1` is a prerelease, and the hyphen is load-bearing

The vocabulary is under community review. 1.0.0-proposal.1 is a semver prerelease. It sorts below the 1.0.0 it will register as, and standard range matching excludes it, so nobody depends on a draft by accident. Spelled +proposal.1 it would be build metadata and compare equal to the release, which is the opposite of what a draft wants.

What a version bump will mean

docmeta versions these vocabularies with three-segment semver, and each segment is a claim about validation behavior rather than a mood:

Segment	Means	Example
MAJOR	a block that used to validate now fails	a new required field, a removed field, a narrowed enum
MINOR	a block that used to fail may now pass; every old one still validates	a new optional field, a widened enum
PATCH	no validation-behavior change at all	a corrected description

So a bump tells you directly whether your corpus is at risk: only MAJOR can break an artifact that validates today.

Adding a grader is not a schema change

The grader field is an open enum. Any kebab-case name validates, and the grader registry is what rejects an unknown one at run time. A new grader therefore never needs a new schema version. The cost is that a misspelled or retired name (llm, the pre-1.0 spelling of ai) passes validation and fails during the run instead.

Separate probes from guards

type answers a question that otherwise makes a pass rate meaningless:

Value	Meaning	A failure means
regression	Protects behavior that already works.	Something broke. Act on it.
capability	Probes the edge of what the agent can do. Expected to fail sometimes.	Information about where the boundary is.

- id: handles-nested-monorepo
  assertion: The session located the correct package in a nested workspace without being told.
  type: capability
  grader: ai

Neither changes enforcement. A failing capability eval still fails and still contributes to a non-zero exit code. That restraint is deliberate: an eval that could not fail the build would just be a comment. The field instead lets you read a report or a history log and know that a red probe is not a regression.

New evals default to regression, including everything fill writes. That is the conservative choice, since mislabelling a real guard as a probe is the more expensive mistake.

`capability` is reported, not excused

If you want probes to stop blocking a pipeline, that is a severity decision, not a type decision. Set severity: warning so the finding reports and the eval passes. Use type for what the eval is, and severity for what should happen when it fails.

Roll out a convention in stages

The transferable technique, whether you are adding a field, a grader convention, or a tightened assertion: never flip a whole corpus to error at once.

1. Introduce it at warning. The eval reports on every run and fails nothing:

   - id: cites-a-test-run
     assertion: The session ran the test suite before declaring the fix complete.
     severity: warning

2. Measure coverage. Count how many artifacts carry it, and how often it actually fires:

   npx moose-tracevals run <trace> --format json \
     | jq '[.evalResults[] | select(.evalName == "cites-a-test-run")]
           | group_by(.outcome) | map({(.[0].outcome): length}) | add'

3. Fix the artifacts, not the eval. A rule failing everywhere usually means the instructions never said it clearly. This is the step that does the work.

4. Promote to error once the corpus is ready. Now it gates.

The same ratchet works in reverse for tightening an existing assertion. Add the sharper eval under a new id at warning, let both run, and remove the old one when the new one is clean.

Why not just tighten in place?

Editing an assertion changes the eval plan, which changes the cache key. Every session is then re-judged against a rule that half your artifacts have never been written for. You get a wall of red that tells you nothing about whether behavior changed. Adding a second eval keeps the old signal intact while the new one settles.

A malformed block fails loudly

Rolling a change across many repositories is survivable mostly because breakage is precise. An invalid block is never silently ignored: it produces an eval named evals-block-valid with a JSON Pointer to the offending field and a source line number.

ERROR  fix-bug › evals-block-valid
       invalid metadata.evals block: /metadata/evals/1/grader (line 12): must be string

Eval entries are closed, so a misspelled field inside one is an error rather than a key that quietly does nothing. metadata itself must stay open, because it is the host tool’s extension bag. The eval prefix is reserved at run time instead, so eval-skpi is rejected the same way. When you roll out a convention and something breaks, you get the file, the pointer, and the line.

Keep ids stable

Eval ids are the join key for history. A regression is defined as this artifact and this eval id went from pass to fail. Renaming an eval therefore reads as one eval removed and another added, and the trend line resets.

This is exactly why id is required on the object form. The retired 0.2 shape let a name be derived from list position, so reordering entries silently reset every trend line that depended on them.

Rename when the meaning genuinely changed. Otherwise leave the id alone and edit the assertion.

Where to go next

Every eval fieldThe full metadata.evals block, the docmeta vocabulary behind it, and what each field does.

Track it over timeThe history file, what counts as a regression, and how to compare two runs.