Evals schema reference

Scope: the exact shape of the metadata.evals block. How to decide what to put in it is Declare what to check. Source of truth: schemas/artifact-evals-1.0.0-proposal.1.json.

The vocabulary is docmeta’s

The block implements docmeta:artifact-evals:1.0.0-proposal.1, a common metadata vocabulary published by docmeta rather than by this tool. moose-tracevals ships a byte-identical copy of the schema and implements the behavior against it. That behavior is the graders, the judge, and the reports. Any session-grading tool can implement the same vocabulary.

Still a proposal

The vocabulary is open for community review, which is what the -proposal.1 suffix records. It is a semver prerelease, so it sorts below the 1.0.0 it will register as. If the review changes the shape, the vendored copy is re-synced.

Where the block goes

In the YAML frontmatter of any instruction artifact: a SKILL.md, an agent definition, or a project-rules file. An artifact’s top-level frontmatter belongs to the tool that runs it, so everything here lives one level down under metadata, its sanctioned extension bag:

---
name: fix-bug
description: Fix a reported bug, reproducing it with a failing test first.
metadata:
  evals:
    - id: used-read
      assertion: The session read at least one source file before editing.
      grader: tool-usage
      options:
        tool: Read
        expect: used
    - Reproduce the bug with a failing test before applying the fix.
---

metadata stays open, so other tools’ members pass through untouched. Eval entries are closed, so a misspelled field inside one fails loudly instead of being ignored.

The three keys

Key	Type	Default	Description
metadata.evals	string, or non-empty list	none	One assertion, or the list of them.
metadata.eval-skip	boolean	false	Skip this artifact’s evals entirely. Reported as skipped, never silently dropped.
metadata.eval-provenance	list	none	Which model proposed which evals, at what confidence. Written by fill.

Because metadata must stay open, the schema alone cannot catch a typo like eval-skpi. So the eval prefix is reserved: any unrecognized metadata.eval* key is an error, not a key that quietly does nothing.

An empty list is not a declaration. An artifact with no evals omits the key.

One eval

Each entry in evals is a string or an object.

String form

- Reproduce the bug with a failing test before applying the fix.

Shorthand for an AI-judged assertion. It normalizes to:

Field	Value
id	eval-<n>, derived from position
assertion	the string
grader	ai
type	regression
severity	error

The whole block can also be one string, which is the same thing for an artifact with a single claim:

metadata:
  evals: Reproduce the bug with a failing test before applying the fix.

The string form is the legitimately id-less one, so its identity is positional and shifts if you reorder the list. Use the object form for anything you want to reference or track over time.

Object form

id and assertion are both required.

Field	Type	Default	Description
id	string	required	Stable identifier, unique per artifact. Must match ^[a-z0-9][a-z0-9-]*$, lowercase kebab-case.
assertion	string	required	What must have held in the session.
grader	string	ai	How the assertion is checked. See graders.
options	object	none	Grader-specific options, validated by the grader itself.
severity	error | warning | info	error	Only error findings fail the eval.
type	capability | regression	regression	Reported, never enforced. See below.
skip	boolean	false	Skip this one eval, reported as skipped.
evidence	string	none	A hint telling the judge where in the session to look. Judged evals only.
examples	object	none	{ pass, fail }, anchor examples for the judge. Each is one string or a list of them.
provider	string	config default	Judge this eval with a specific configured provider.
severity-map	object	none	Maps a tool:* grader’s own severities onto eval severities. Accepted for vocabulary parity; moose-tracevals implements no tool:* graders, so it is ignored here.

Command-graded evals take four more fields, covered under the command grader.

Ids are required on the object form because a position-derived name breaks the history join whenever entries move. A rename reads as one eval removed and another added, and the trend line resets. If an eval is worth an object, it is worth a stable name.

- id: reproduces-before-fixing
  assertion: The session ran a failing test that reproduced the bug before editing any source file.
  grader: ai
  severity: error
  type: regression
  evidence: Look for a test command in the tool calls preceding the first Edit.
  examples:
    pass:
      - "Ran `npm test -- crash.test.ts`, saw it fail, then edited src/app.ts."
    fail: "Edited src/app.ts at turn 2 with no preceding test run."

evidence and examples are prompt surface. They are given to the judge and measurably improve verdict quality on assertions that are easy to misread. They have no effect on a deterministic grader.

capability vs regression

type separates two different reasons for an eval to exist:

Value	Meaning
regression	Protects behavior that already works. A failure is news.
capability	Probes the boundary of what the agent can do. It is expected to fail sometimes.

Both are reported; neither changes enforcement. A failing capability eval still fails and still contributes to a non-zero exit code.

That restraint is deliberate. A capability eval that could not fail the build would just be a comment. The value of the field is that reports and history can tell the two apart. A failing probe is never read as a regression. New evals, including everything fill writes, default to regression, the conservative choice.

Provenance

fill records what it proposed, so a reviewer can tell an unreviewed suggestion from an eval a human signed off on:

metadata:
  eval-provenance:
    - generated-by: anthropic:claude-opus-5
      evals: [used-read, honored-tdd]
      confidence:
        used-read: 0.91
        honored-tdd: 0.86

Entries merge by generated-by, so repeated fills by one model extend one entry. You delete the entry once you have reviewed those evals. That is what makes the trail mean something.

The implicit eval

An artifact with no declared evals still gets one judged eval, named adheres-to-artifact:

The session adhered to the instructions in this <type> (“<name>”). Cite the specific instructions followed or violated.

The artifact’s full text is given to the judge, capped at 8,000 characters. This is why a first run produces meaningful output before you have authored anything.

Declaring any eval replaces the implicit one for that artifact. Once you start declaring, declare what matters: the general check no longer runs alongside.

Importing the schema

The schema ships with the package:

import schema from "moose-tracevals/schemas/artifact-evals-1.0.0-proposal.1.json" with { type: "json" };

You do not need to pin anything to author evals. moose-tracevals validates against the shipped copy. Importing it matters when your own tooling reads or generates these blocks.

When the block is invalid

A malformed block is never silently ignored. It produces an eval named evals-block-valid with an error outcome, a JSON Pointer to the offending field, and a source line number:

  ERROR  fix-bug › evals-block-valid
         invalid metadata.evals block: /metadata/evals/1/grader (line 12): must be string

Migrating from artifact-evals-0.2

The pre-1.0 spellings fail loudly rather than degrading, which is the point:

• metadata.evals.criteria: is gone. The list is now the value of evals.
• - name: on an object entry is now - id:, and required.
• metadata.evals.skip is now metadata.eval-skip, a sibling of evals rather than nested in it.
• grader: llm is now grader: ai. It passes the schema, because the grader enum is open. The grader registry rejects it at run time, with a message naming every kind it knows.

Only YAML frontmatter can be written

fill writes to YAML frontmatter only. TOML (+++) and JSON (;;;) frontmatter are refused: only YAML frontmatter can be edited (found toml frontmatter). Reading and evaluating an existing block is unaffected.