manni:artifact-evals:1.0.0-proposal.4

Proposal under review

Part of the metadata vocabularies proposal, open for community review. Draft schema: docs/proposals/0023/schemas/artifact-evals/1.0.0-proposal.4.json. Review round 6 took artifact-evals from proposal.1 to proposal.2: it added weight, target, runs and model, made assertion conditional, and moved the eval- prefix guard from prose into the schema. Proposal 0046 took it to proposal.3 by renaming metadata.eval-provenance to metadata.meta-provenance and info to notice, as the end of this page describes. Proposal 0047 took it to proposal.4 by marking the preferred location of its top-level key, and changed no key.

The question it answers: what must a session that used this artifact have done? Instruction artifacts (skills, agent definitions, project-rules files) carry evals about the sessions that run with them. This is a common vocabulary that any session-grading tool can implement.

Keys

The page-side keys, one level down. An artifact’s top-level frontmatter is the host tool’s contract, and metadata is the host’s sanctioned extension bag. Any other eval-* key inside metadata is rejected.

Key	Type	Location	Notes
metadata.evals	string, or non-empty list of entries	external	One assertion, or the list
metadata.eval-skip	boolean (default false)	external	Skip the artifact’s evals. Tooling reports the skip
metadata.meta-provenance	list of entries	external	The machine-proposal trail, in the shape of the page-level meta-provenance in ai-context

The Location column comes from the one mark in manni:artifact-evals:1.0.0-proposal.4, which sets "x-manni-location": "external" on the top-level metadata key. The keys inside it carry no mark of their own, so each row takes its parent’s. page means the value belongs in the document’s own metadata and is meant to reach delivered output. external means it belongs in the collection’s external-metadata manifest. A key is page when a third-party or user-owned agent fetching the page would act on it. Artifact evals are CI’s, so the block is external. See field location for how the marks are used.

Entries share the page-side shape. id is required on the object form. assertion is required for ai, for human, and for entries with no grader; command needs an assertion or a command. The rest: type, severity (default error), evidence, examples with one or several anchors, provider, runs and model (the three legal only on ai graders), target, weight, the command family with the same guard rails, options, severity-map, and skip.

target selects what the grader receives. The options are transcript (the default, the whole session), last-message (the final assistant message), files (the files the session wrote), and artifact (the instruction artifact itself). It can also be a named file, with source: file plus a path relative to the session’s working directory.

Graders

The grader is a fully open list. Recommended values: ai (default), human, command, tool-usage, skill-invoked, file-access, turn-count, cost, regex, and json-output, plus the page side’s tool:* spelling so one grader name works in both vocabularies. Any kebab-case name is legal. The implementer’s grader registry validates it at run time, so a new grader never needs a schema version.

Two graders behave differently from the page side by nature. human is judged per session, because every trace is new, so verdicts are never cached. command receives {trace}, the session’s trace path, where the page side substitutes {file}.

Example

---
name: fix-bug
description: Fix a reported bug, reproducing it with a failing test first.
metadata:
  evals:
    - id: used-read
      assertion: The session read at least one source file before editing.
      grader: tool-usage
      options: { tool: Read, expect: used }
    - id: honored-tdd
      assertion: The session wrote a failing test before the fix.
      grader: ai
      type: capability
      weight: 2
      examples:
        pass:
          - A test edit lands before the src edit, and the first run fails.
        fail: The fix lands first and a test is added afterwards.
    - id: summary-names-the-cause
      assertion: The final message names the root cause in one sentence.
      target: last-message
    - Reproduce the bug with a failing test before applying the fix.
  meta-provenance:
    - generated-by: claude-fable-5
      evals: [used-read, honored-tdd]
      confidence: { used-read: 0.91, honored-tdd: 0.86 }
---

Design decisions

• The envelope belongs to the host. This is the one place the family’s flat-keys rule cannot reach. A skill’s top-level frontmatter belongs to the tool that runs it, and metadata is its extension point. metadata stays open, so other tools’ members pass untouched, while entry objects are closed. The schema rejects any eval-* key inside metadata other than the ones above. That gives it the same protection the page side applies at its root, where a misspelled key fails instead of being ignored. In proposal.1 this was a request to implementers, and the implementer that followed the description faithfully ended up without the guard.
• Ids are required on object entries. Names derived from position orphan cached verdicts whenever entries move. An eval that is worth an object is worth a stable name.
• assertion follows the page side’s rule. It was unconditional in proposal.1. A tool-usage criterion says everything in options, just as a page-side tool:freshness one does, so the old requirement forced authors to write a sentence no grader reads. The page side’s conditional block is ported verbatim.
• The grader list stays open. The schema’s conditionals branch only on ai, human and command, the three names both sides share. Any other kebab-case name matches none of those branches, passes the schema, and is validated by the registry at run time. The one cost, stated in the schema: a stale legacy grader name passes the schema and is rejected by the registry.
• weight, target, runs and model mean what they mean on the page side. See the evals vocabulary for the reasoning. Only the target members differ, because the subject is a session rather than a page.

What reviewers should poke at

Two questions. Does one entry vocabulary across pages and artifacts (differing only in envelope, target members and grader family, each for a stated reason) carry its weight? And are the per-session human semantics documented clearly enough that implementers will not try to cache verdicts?

Proposal 0046 renames metadata.eval-provenance to metadata.meta-provenance, in draft 1.0.0-proposal.3. Its entries take the same shape as the page-level meta-provenance in ai-context, which manni meta fill already writes on a page. They sit one level down, because an artifact’s top level still belongs to its host tool. The eval- guard inside metadata no longer excepts provenance, so metadata.eval-provenance is rejected like any other unknown eval-* key. The same draft renames the info severity to notice, the level manni’s other tools already use.