docmeta:artifact-evals:1.0.0-proposal.2

The question it answers: what must a session that used this artifact have done? Instruction artifacts (skills, agent definitions, project-rules files) carry evals about the sessions that run with them. This is a common vocabulary that any session-grading tool can implement.

Keys

The page-side keys, one level down. An artifact’s top-level frontmatter is the host tool’s contract, and metadata is the host’s sanctioned extension bag. Any other eval-* key inside metadata is rejected.

Key	Type	Notes
`metadata.evals`	string, or non-empty list of entries	One assertion, or the list
`metadata.eval-skip`	boolean (default false)	Skip the artifact’s evals. Tooling reports the skip
`metadata.eval-provenance`	list of entries	The machine-proposal trail

Entries share the page-side shape. id is required on the object form. assertion is required for ai, for human, and for entries with no grader; command needs an assertion or a command. The rest: type, severity (default error), evidence, examples with one or several anchors, provider, runs and model (the three legal only on ai graders), target, weight, the command family with the same guard rails, options, severity-map, and skip.

target selects what the grader receives. The options are transcript (the default, the whole session), last-message (the final assistant message), files (the files the session wrote), and artifact (the instruction artifact itself). It can also be a named file, with source: file plus a path relative to the session’s working directory.

Graders

The grader is a fully open list. Recommended values: ai (default), human, command, tool-usage, skill-invoked, file-access, turn-count, cost, regex, and json-output, plus the page side’s tool:* spelling so one grader name works in both vocabularies. Any kebab-case name is legal. The implementer’s grader registry validates it at run time, so a new grader never needs a schema version.

Two graders behave differently from the page side by nature. human is judged per session, because every trace is new, so verdicts are never cached. command receives {trace}, the session’s trace path, where the page side substitutes {file}.

Example

---
name: fix-bug
description: Fix a reported bug, reproducing it with a failing test first.
metadata:
  evals:
    - id: used-read
      assertion: The session read at least one source file before editing.
      grader: tool-usage
      options: { tool: Read, expect: used }
    - id: honored-tdd
      assertion: The session wrote a failing test before the fix.
      grader: ai
      type: capability
      weight: 2
      examples:
        pass:
          - A test edit lands before the src edit, and the first run fails.
        fail: The fix lands first and a test is added afterwards.
    - id: summary-names-the-cause
      assertion: The final message names the root cause in one sentence.
      target: last-message
    - Reproduce the bug with a failing test before applying the fix.
  eval-provenance:
    - generated-by: claude-fable-5
      evals: [used-read, honored-tdd]
      confidence: { used-read: 0.91, honored-tdd: 0.86 }
---

Design decisions

The envelope belongs to the host. This is the one place the family’s flat-keys rule cannot reach. A skill’s top-level frontmatter belongs to the tool that runs it, and metadata is its extension point. metadata stays open, so other tools’ members pass untouched, while entry objects are closed. The schema rejects any eval-* key inside metadata other than the ones above. That gives it the same protection the page side applies at its root, where a misspelled key fails instead of being ignored. In proposal.1 this was a request to implementers, and the implementer that followed the description faithfully ended up without the guard.
Ids are required on object entries. Names derived from position orphan cached verdicts whenever entries move. An eval that is worth an object is worth a stable name.
assertion follows the page side’s rule. It was unconditional in proposal.1. A tool-usage criterion says everything in options, just as a page-side tool:freshness one does, so the old requirement forced authors to write a sentence no grader reads. The page side’s conditional block is ported verbatim.
The grader list stays open. The schema’s conditionals branch only on ai, human and command, the three names both sides share. Any other kebab-case name matches none of those branches, passes the schema, and is validated by the registry at run time. The one cost, stated in the schema: a stale legacy grader name passes the schema and is rejected by the registry.
weight, target, runs and model mean what they mean on the page side. See the evals vocabulary for the reasoning. Only the target members differ, because the subject is a session rather than a page.

What reviewers should poke at

Two questions. Does one entry vocabulary across pages and artifacts (differing only in envelope, target members and grader family, each for a stated reason) carry its weight? And are the per-session human semantics documented clearly enough that implementers will not try to cache verdicts?