Skip to content
manni

Move an annotated corpus into a manifest

Evals are bookkeeping. They tell CI what to check, and they tell a reader of the page nothing. Several docs platforms serve frontmatter unchanged in the markdown variant coding agents and retrieval pipelines fetch. On that surface every eval is bytes an agent reads and cannot use.

The fix is to keep them in the collection’s external-metadata manifest instead. This page moves a corpus that is already annotated, and shows that the move changes nothing about what is checked.

Start from a run you can compare against

Section titled “Start from a run you can compare against”

Record what the corpus reports now. That is the thing the move has to preserve:

Terminal window
$ npx @hawkeyexl/manni docevals run --deterministic-only
docs/draft.md
  FAIL no-todo-markers
       error:9 [regex/found] Pattern /\b(TODO|TBD|FIXME)\b/ found in body, expected absent
docs/install.md
  pass no-todo-markers


Suites
  default: 0/1 passed — 0% vs target 100% below target
  reference: 1/1 passed — 100% vs target 100% ok

One page passes, one fails. Both numbers have to survive.

See what would move

Section titled “See what would move”

manni meta relocate moves a value to where its schema says it belongs. The evals vocabulary marks all three eval keys external. Ask it first, and write nothing:

Terminal window
$ npx @hawkeyexl/manni meta relocate --dry-run --fields evals,eval-suite -s manni:evals:1.0.0
Using manni.config.yaml (.)
Would create site.metadata.yaml; collections[site].externalMetadata[0] would own evals, eval-suite.
docs/draft.md
    evals       → site.metadata.yaml
docs/install.md
    eval-suite  → site.metadata.yaml
    evals       → site.metadata.yaml
2 files, 3 values would move to 1 manifest

Read the first line before the file list. It says the manifest does not exist yet and that relocate will declare it on the collection. Both are edits to files you own. A dry run is the only place to catch a manifest landing somewhere you did not want it.

--fields names the keys to move, comma-separated, and eval-skip joins the list on a corpus that uses it. Without the flag relocate moves every marked value it finds. That is the right call for a corpus adopting a whole standard and the wrong one for a first pass.

-s manni:evals:1.0.0 names the built-in vocabulary, so no copy of the schema is needed. Drop the flag once your own schema composes the vocabulary.

Move them

Section titled “Move them”
Terminal window
$ npx @hawkeyexl/manni meta relocate --fields evals,eval-suite -s manni:evals:1.0.0
Using manni.config.yaml (.)
Created site.metadata.yaml; collections[site].externalMetadata[0] owns evals, eval-suite.
docs/draft.md
    evals       → site.metadata.yaml:2
docs/install.md
    eval-suite  → site.metadata.yaml:5
    evals       → site.metadata.yaml:6
2 files, 3 values moved to 1 manifest

Three files changed. The pages lost their eval keys and kept everything else:

docs/install.md
---
title: Install
---

The manifest holds them, keyed by the page’s path:

site.metadata.yaml
docs/draft.md:
  evals:
    - use: no-todo-markers
docs/install.md:
  eval-suite: reference
  evals:
    - use: no-todo-markers

And the collection declares which keys that manifest owns, which is what every manni tool reads:

manni.config.yaml
collections:
  - name: site
    paths:
      - "docs/**/*.md"
    externalMetadata:
      - file: ./site.metadata.yaml
        keys: [evals, eval-suite]

Commit all three together. A manifest without the externalMetadata: entry declaring it is a file nothing reads, and pages stripped of evals with no manifest to supply them resolve nothing.

Check that nothing changed

Section titled “Check that nothing changed”
Terminal window
$ npx @hawkeyexl/manni docevals run --deterministic-only
docs/draft.md
  FAIL no-todo-markers
       error:7 [regex/found] Pattern /\b(TODO|TBD|FIXME)\b/ found in body, expected absent
docs/install.md
  pass no-todo-markers


Suites
  default: 0/1 passed — 0% vs target 100% below target
  reference: 1/1 passed — 100% vs target 100% ok

The same verdicts, down to the failing page. Only the finding’s line moved, because the page’s frontmatter is two lines shorter. Relocation changes where a value is stored, never what it means. manni docevals list is the same comparison one step earlier, and it costs no grading.

Findings still point at the page, because the page is what the grader read. A problem with a declaration now points at the manifest and its line there, since that is the file to edit.

What the writers do afterwards

Section titled “What the writers do afterwards”

fill, generate and promote --write follow the same rule from here on. Each writes into your page’s entry in the manifest, leaving every other entry, and every comment in the file, alone:

filled   docs/install.md  +1 evals (states-the-problem 0.90)
    evals → site.metadata.yaml

A page the manifest has no entry for is refused rather than written somewhere else. Where the manifest joins on a field rather than the path, a page missing that field gets the sentence naming what it lacks:

docs/no-id.md carries no doc-id, which site.metadata.yaml joins on, so its evals has no entry there.

Two things not to do

Section titled “Two things not to do”

Do not leave a copy behind. A page that keeps a key its manifest owns is an error, not a merge:

error docs/install.md:4 "evals" is owned by manifest site.metadata.yaml (collection site); remove it from the document

Two declarations of what to check have no tiebreak. Delete the page’s copy; relocate does that for you, and this error only appears where something else put one back.

Do not point file: at a URL. A shared manifest over HTTP is fine for keys manni only reads. docevals writes eval keys, and nothing fetched can be written, so the run is refused with exit 2 before a byte is fetched. The same goes for two collections both claiming one page’s evals. The frontmatter reference lists all three refusals.

Next

Section titled “Next”