manni:evals:1.0.0-proposal.4

Proposal under review

Part of the metadata vocabularies proposal, open for community review. Draft schema: docs/proposals/0023/schemas/evals/1.0.0-proposal.4.json. Review round 6 took evals from proposal.1 to proposal.2: it added weight, target, runs and model, and moved the eval- prefix guard from prose into the schema. Proposal 0046 took it to proposal.3 by removing eval-provenance and renaming info to notice, as the end of this page describes. Proposal 0047 took it to proposal.4 by marking each key’s preferred location, and changed no key.

The question it answers: what must be true of this page? A common vocabulary for per-page quality assertions. Any grader, judge, or CI tool can implement it, and other schemas can compose on top of it.

Keys

Three page-level keys. Two carry the reserved eval- prefix, and the third is evals itself. Any other eval-* key at the page root is rejected.

Key	Type	Location	Notes
evals	string, or non-empty list of entries	external	One assertion, or the list
eval-suite	string, non-empty	external	A named suite from the implementer’s config. On an id collision, the page’s entry wins
eval-skip	boolean (default false)	external	Skip the page’s evals. Tooling reports the skip

The Location column is the x-manni-location mark each key carries in manni:evals:1.0.0-proposal.4. page means the value belongs in the document’s own metadata and is meant to reach delivered output. external means it belongs in the collection’s external-metadata manifest. A key is page when a third-party or user-owned agent fetching the page would act on it. All three keys are external, because evals are CI’s and an agent reading the page does not act on them. The fields inside an entry carry no mark of their own. See field location for how the marks are used.

An entry takes one of three forms. It can be a string shorthand, meaning an AI-judged assertion at error severity. It can be a use: reference to an eval defined in config. Or it can be an inline definition. An inline definition can carry:

• id (required, kebab-case) and assertion
• type: capability or regression
• grader, and for AI graders provider, model (the judging model) and runs (ensemble runs, 1 to 50)
• target: what the grader receives. The options are body (the default, with frontmatter stripped), raw (the file verbatim), and frontmatter (the parsed frontmatter alone). It can also be an object naming a companion file, with source: file plus a path relative to the page’s directory.
• weight: a positive number, default 1, that sets how much this eval’s outcome moves an aggregate score
• evidence, and examples with one or several pass/fail anchors
• the command family: command, success-exit-codes, timeout-ms, generated-assertion-hash
• options, severity (error, warning, or notice; default error), severity-map, and skip

A use: reference may override skip, type, severity, options, and weight. It may not override provider, model, runs or target. Those say how the tool executes, and a named eval must read the same bytes on every page that uses it.

Graders

ai is the default: a model or an agent, chosen per eval with provider. command runs an executable, with {file} as the page path. human puts the entry in a review queue. tool:* names an integration, and that namespace is open. The first three stay closed because schema conditionals branch on them. ai and human require an assertion. command requires an assertion or a command. Omitting the command is the generation contract: tooling generates a check script from the assertion and writes command and generated-assertion-hash back. An entry carrying a command must say grader: command, and a hash is never legal without its command.

Examples

evals: The documented install command matches the current package name.

eval-suite: how-to
evals:
  - id: install-command-current
    assertion: The install command matches the current package name.
  - id: links-resolve
    grader: command
    command: ["npx", "linkinator", "{file}"]
  - id: install-verified
    assertion: The Helm install steps produce a Ready operator pod.
    grader: human
    severity: warning
  - id: description-says-what-for
    assertion: The description says what the reader can do after reading.
    target: frontmatter
    weight: 2
    runs: 3

Design decisions

• Flat, like the rest of the family. There is no settings object. The list is the value, and the settings are page-level eval-* keys. Because the page root is open, the schema rejects any eval-* key other than the two above. That gives the root the property a closed container would have: a misspelled key fails instead of being ignored. In proposal.1 this was a request to implementers; one implementer honoured it and one did not, so proposal.2 encodes it.
• Ids are required on object entries. The string shorthand is the only id-less form. Names derived from position orphan cached verdicts whenever entries move.
• Errors name the actual fault. Entries branch with if/then rather than oneOf, so a misspelled field inside an object is reported against that key instead of collapsing to “must be string”.
• weight scores; it does not judge. It changes how much an outcome moves a suite or run total, and never the eval’s own pass or fail. The binary outcome is what SARIF, JUnit and findings baselines consume downstream. Zero is excluded: a weightless eval is a silent disable, and skip already means that, visibly.
• target selects the bytes; evidence hints where to look. A deterministic grader has nothing to focus on, but can still be pointed at the frontmatter or a companion file. So the selector is structural, and every grader honours it. The name is target rather than focus for the same reason: it selects data, and it does not collide with evidence.
• runs and model belong to ai evals. A conditional makes provider, runs and model legal only with the ai grader. runs is capped at 50 because runs multiply cost directly. model also lets an eval name a judge other than the model that wrote the page. That turns the self-preference warning that provenance in ai-context enables into something a tool can act on.
• Machine-proposed evals go in the family’s one trail. Their ids sit under evals in a meta-provenance entry (see ai-context). Entries are deleted under human review, and a surviving entry means machine-proposed evals nobody has checked.

What reviewers should poke at

Three questions. Is the encoded eval- prefix rejection an acceptable constraint on the page root? Is the split between closed structural graders and the open tool:* namespace the right one? (Its artifact-side counterpart is a fully open list, for reasons stated there.) And should a page be able to set runs and model inline at all, given that a use: reference cannot?

Proposal 0046 removes eval-provenance, in draft 1.0.0-proposal.3. Machine-proposed eval ids move to the page-level meta-provenance in ai-context, under its evals list, so one trail covers fields and evals alike. The eval- prefix guard no longer excepts provenance, so an eval-provenance key at the page root is rejected like any other unknown eval-* key. manni meta derive already records the machine behind each range of a body, in ai-context’s provenance. The self-preference warning would read them there for a body target, meta-provenance for frontmatter, and both for raw, rather than one page-level generated-by. The same draft renames the info severity to notice, the level manni’s other tools already use.