Run it in CI

A gate that is not in CI is a linter someone runs sometimes. This page is the GitHub Actions recipe; other platforms follow the same shape.

The recipe

name: docs

on:
  pull_request:
    paths: ["docs/**", "manni.config.yaml"]
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  docevals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm
      - run: npm ci

      # Persist the judge cache between runs — without it every push
      # re-judges the whole corpus and the bill is the story.
      - uses: actions/cache@v4
        with:
          path: .manni/docevals/cache
          key: manni-docevals-${{ hashFiles('manni.config.yaml') }}-${{ github.sha }}
          restore-keys: manni-docevals-${{ hashFiles('manni.config.yaml') }}-

      - run: npx @hawkeyexl/manni docevals run --format github
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Three things are doing real work here.

--format github emits workflow commands that annotate the offending line in the diff, followed by a markdown summary. A failure a contributor can see on the line they wrote gets fixed in minutes; the same failure in a job log gets ignored. See Exit codes and annotations.

The cache step is not an optimisation. The whole approach assumes unchanged pages never re-judge; a cold cache every run re-judges everything. The key includes the config hash because changing an assertion legitimately invalidates verdicts. See Caching and turn budgets.

paths: keeps the job off pull requests that cannot affect the docs.

A local llama-cpp judge belongs in CI only on a runner with a GPU. See Local models and CI.

Post the summary

--format github output doubles as a job summary:

      - run: npx @hawkeyexl/manni docevals run --format github | tee "$GITHUB_STEP_SUMMARY"

Before you enable it on pull requests

Two things will bite, and both have their own page:

• Fork pull requests can execute their author’s code, and the obvious-looking config flag does not prevent it. Read Untrusted pull requests before this job runs on a public repo.
• Every uncached ai eval is live model calls, judge.ensembleRuns of them. Bound a run with judge.maxTurns or --max-turns: evals past the budget are skipped and reported, and evals served from cache never count against it. See Caching and turn budgets.

Start deterministic

You do not need a provider to get value on day one:

      - run: npx @hawkeyexl/manni docevals run --deterministic-only --format github

No secret, no cost, no model in the critical path. Every tool:regex eval and every command eval still runs. This is also the correct job for forks.

Frontmatter and structure are checked in their own steps, by manni meta validate and manni lint.

Exit codes route to different people

Code	Meaning	Whose problem
0	Pass	Nobody
1	Findings, or a suite below target	The author
2	Operational, such as bad config or a missing key	The pipeline owner

A recipe that treats any non-zero exit as “the docs are bad” blames authors for infrastructure problems. It also teaches the team the check is flaky. Do not collapse 1 and 2.

Verify the gate can fail

Before you trust it, make it fail once. Break a page deliberately, confirm the job goes red with an annotation on the right line, and revert. A gate that has only ever been seen passing has not been shown to work.