Write evals

Every eval declaration lives under the evals key in page frontmatter. This page is the contract; Frontmatter reference is the exhaustive field list.

The minimum

The evals key is the list of checks for a page:

---
title: Concepts
evals:
  - use: no-future-promises
  - id: defines-core-terms
    assertion: The page defines every core concept it introduces.
---

A suite assignment and a page-level skip are flat keys beside it, not an enclosing object. Every setting this vocabulary claims carries the reserved eval- prefix, so a typo like eval-suit: is an error rather than a key nobody reads:

---
title: Installation
eval-suite: how-to
evals:
  - use: no-future-promises
---

---
title: Index
eval-skip: true
---

Three ways to declare an eval

A bare string is an assertion, judged by AI at error severity. It is the fastest way to guard something once:

evals:
  - The install command names the current package.

A string is never a reference

- no-todo-markers sends those words to the judge as an assertion. It does not run the eval of that name. manni docevals warns when a string matches a defined eval id. Write use: to reference one.

Reference a named eval from manni.config.yaml with use::

evals:
  - use: no-future-promises

Reference with overrides, to narrow it for this page only:

evals:
  - use: names-an-action
    severity: error
    options: { flags: i }

options merges over the base rather than replacing it, so you can change one setting without restating the rest. type, severity, and skip replace.

Define one inline, when it exists only on this page:

evals:
  - id: install-command-present
    assertion: The page contains a bash code block with `npm i -g doc-detective`.
    grader: command

Inline evals need an id, kebab-case and unique on the page. Everything else is optional: grader defaults to ai and type to regression. The id is required rather than derived from position, because a position-derived name orphans every cached verdict as soon as entries move.

Evals can live beside the pages instead

Everything above is a page carrying its own evals, and that is where they start. They are maintainer’s bookkeeping though, not something a reader of the page wants, and some platforms serve frontmatter unchanged to whoever fetches the markdown.

So a collection can keep them in one external-metadata manifest instead, keyed by page path:

site.metadata.yaml

docs/install.md:
  eval-suite: reference
  evals:
    - use: no-todo-markers

manni.config.yaml

collections:
  - name: site
    paths: ["docs/**/*.md"]
    externalMetadata:
      - file: ./site.metadata.yaml
        keys: [evals, eval-suite, eval-skip]

docs/install.md then carries no evals key at all, and every command reads the manifest’s entry as though it sat on the page:

$ npx @hawkeyexl/manni docevals list
docs/install.md (suite: reference)
  - no-todo-markers [tool:regex, regression, config]

1 pages, 1 evals resolved

Declaring the same key in both places is an error rather than a merge. Nothing then says which declaration is the one CI runs, so pick one home per key. Moving an annotated corpus is one command, and the frontmatter reference has the rules.

Prefer named evals

An assertion pasted into twelve pages drifts into twelve slightly different assertions, and nobody can then say what a how-to page is checked for. Move anything used more than once into the config and reference it. See Named evals and suites.

Inline is right for something genuinely specific to one page, like the exact install command above.

What wins

The plan is built suite-first, then page entries in document order. Page entries win on name collision, so a local override beats the global default.

That is the correct default and completely invisible until it surprises you, so check rather than guess:

npx @hawkeyexl/manni docevals list docs/actions/goTo.mdx

docs/actions/goTo.mdx (suite: reference)
  - no-future-promises [ai, regression, config]
  - names-an-action [tool:regex, regression, config]
  - no-todo-markers [tool:regex, regression, config]

1 pages, 3 evals resolved

Each line is name [grader, type, source]. source tells you whether the definition came from the config or the page. That is exactly what you need when a page behaves differently from its neighbours.

Problems are reported, not thrown

A malformed page does not stop the run. Unknown suite names, unknown eval references, and schema-invalid frontmatter are reported as page errors with a source line. Duplicate names and inline ai evals missing examples are warnings.

Validate the declarations themselves with manni meta validate. It checks them against the shipped manni:evals:1.0.0 vocabulary, so bad frontmatter fails in the metadata gate.

Next

• Write good assertions, the wording that decides whether an eval is worth having
• Deterministic checks, the graders to prefer
• Named evals and suites, turning this into a library