manni test

Prove the procedures in your docs work, and that a reader can finish a task with them.

CLI reference Attempt a task Run it in CI

A page in the docs makes two claims. Its procedures work, and a reader can finish a job with it. manni test checks each claim with a verb of its own.

Command	Checks	Tool
`manni test procedures`	The procedure tests embedded in the pages pass.	Doc Detective
`manni test tasks`	A synthetic user, playing a persona, finishes a task from the pages.	Claude Code

The verb names the job, and config names the tool that answers it. Both verbs report the way every other manni gate does. CI calls each one it wants.

A procedure in the docs is a claim that following it works. Doc Detective checks that claim. It reads tests embedded in a page as comments, runs each step, and records whether it passed. manni test procedures starts Doc Detective, reads the results it writes, and reports them. The sections below cover procedures, and the last one covers tasks.

What it adds to Doc Detective

An exit code that is a verdict. Doc Detective exits 0 when a test fails. manni test procedures exits 1 when anything failed, and 2 when the run itself went wrong.
The page and line that broke. Each failing step is listed under its page with its line, or written as a GitHub annotation with -f github.
A run that tested nothing fails. Doc Detective skips an input path that does not exist, and passes a run that found no tests. manni refuses both with exit 2, unless you pass --allow-empty.

Doc Detective is installed on its own, beside manni:

npm install -g doc-detective

A 30-second proof

Two pages in test/test/fixtures/ hold one test each, written as comments. Each test has one runShell step that expects exit 0. In pass.md, the step runs echo hello. In fail.md, it runs exit 1, on line 4.

Run the failing page. The step’s line is in the report, and the run exits 1:

$ manni test procedures test/test/fixtures/fail.md
test/test/fixtures/fail.md
  4  FAIL     Returned exit code 1. Expected one of [0]. Fix the step, fix the procedure it documents, or report a product regression.
              Doc Detective runShell
              Guide: https://docs.doc-detective.com/docs/actions/runshell

1 test: 0 passed, 1 failed, 0 warnings, 0 skipped

Run the passing page. A clean run prints the summary alone, and exits 0:

$ manni test procedures test/test/fixtures/pass.md
1 test: 1 passed, 0 failed, 0 warnings, 0 skipped

Name a page that is not there. Doc Detective alone would skip it and pass. manni stops with exit 2:

$ manni test procedures test/test/fixtures/missing.md
manni: File not found: "test/test/fixtures/missing.md". Check the path. Named paths resolve from the working directory.

Which files it tests

A run tests the first of these that is present:

The paths you name. Files and directories go to Doc Detective as you typed them, and Doc Detective walks a directory itself. manni expands a glob.
--collection <name>. The files of a collection declared in manni.config.yaml, minus its exclude:. Repeat the flag for more than one. It cannot be combined with paths.
Doc Detective’s own input. From its config file, which it finds as .doc-detective.json, .yaml or .yml in the working directory.

A bare manni test procedures reads no collection, unlike manni meta validate and manni lint check. A repository that tests its docs already has a Doc Detective config, and that config already says what to test. Reading collections too would give one question two answers.

Every other setting stays in Doc Detective’s config, even when manni names the inputs. That covers fileTypes, beforeAny and loadVariables.

tools:
  doc-detective:
    config: .doc-detective.json

Run it in CI

Install Doc Detective, then run manni with the GitHub format:

- run: npm install -g doc-detective
- name: Run the docs' tests
  run: manni test procedures -f github --progress

-f github writes one annotation per failing step, on its page and line. --progress streams Doc Detective’s own log to stderr as it runs. That is the default on a terminal, and a CI job is not one. stdout stays clean either way.

The exit code says whose problem it is:

Exit	Means
`0`	No test failed. Warnings and skipped tests pass.
`1`	A test failed. The page needs fixing.
`2`	The run went wrong. Doc Detective is missing, a path is wrong, or no tests were found.

For a dashboard, -f json prints Doc Detective’s own results object, unchanged. A script written against Doc Detective’s report reads it as it is.

Attempt a task as a persona

A procedure test proves the steps work in the author’s order. It does not prove a reader finds them. manni test tasks hands a task in plain text to a synthetic user, which is Claude Code run headless in a temporary directory. It can play one of your persona records, constraints included:

manni test tasks docs/src/content/docs/meta/fix/index.mdx --persona theo   --workspace test/test/fixtures/tasks/red-pr   --task "The docs check on my pull request fails. Make it pass."

The synthetic user starts from a copy of the docs, never your checkout. A judge then reads its trace and its workspace. It decides whether the reader completed the task and followed the docs, and whether it kept each persona constraint. The report ends with the reader’s own account of where it got stuck. A run that fails exits 1.

A journey that declares evals: is a golden task. Its evals say what done means beyond the task’s own words. manni test tasks --journey t1 runs one, and a bare manni test tasks runs them all. Attempt a task as a persona walks through both.

Attempt a task as a personaAsk whether a persona can finish a task from a page, read its account, fix the page, and keep the task as a golden journey.

CLI referenceEvery argument and flag of test procedures and test tasks, the output formats, the exit codes, and every message.

tools: configurationThe tools.doc-detective and tools.claude-cli keys, beside the settings of every other outside tool manni runs.