Fix a failing eval

You did not configure this check and you do not need to learn the tool. Start with the table.

manni docevals runs quality checks, called evals, against documentation pages. Each eval is one assertion about a page, decided either by a script or by an AI judge.

Which failure is this?

Find your CI output in the left column.

What you see	What it is	What to do
A message with a file and line, like `error:14 [regex/found] …`	A script-based check	Fix the line
A message with reasoning but no line, mentioning “AI judge”	An AI verdict	Read the assertion
`needs-review`	Waiting on a person	Not yours to fix
An assertion changed since its script was generated	A generated script is out of date	Not yours to fix
Exit code 2, or “could not run” / missing key / bad config	The tool itself failed	Not your fault

The distinction that matters most is this. Exit 1 means a docs problem, exit 2 means a pipeline problem. If you are looking at exit 2, stop and tell whoever owns CI.

A finding with a line

docs/actions/goTo.mdx
  FAIL no-todo-markers
       error:14 [regex/found] Pattern /\b(TODO|TBD|FIXME)\b/ found in body, expected absent

Read it as severity:line [rule] message. A script found something specific at line 14.

Go to that line and make the change the message describes, then re-run that one eval locally before you push.

The rule says what the pattern check wanted:

Rule	Meaning
`regex/found`	The page contains text it must not. The line is the first match.
`regex/not-found`	The page lacks text it must contain.
`regex/count`	The text appears a different number of times than required.
`regex/unreadable-target`	The check could not read the part of the page it targets. Ask whoever owns the eval.

A message starting Exit code has no rule. It comes from a command check, and the rest of the message is that command’s own output.

Only error fails the build. If your message says warning or notice, that check is not what turned the build red. Keep reading the log.

An AI verdict

::error file=docs/tutorial.mdx,title=manni docevals%3A no-future-promises::AI judge: fail (confidence 0.93). The "Coming in v3" section describes unreleased functionality.

No line number, because the judgement is about the page rather than a character position.

The rationale tells you what is wrong; the assertion tells you what was required. Look both up:

npx @hawkeyexl/manni docevals list docs/tutorial.mdx

That prints which evals apply. Find the named eval, here no-future-promises, in manni.config.yaml, or inline in the page’s own frontmatter. It looks like this:

  no-future-promises:
    assertion: The page makes no claims about unreleased or future functionality.
    examples:
      pass: Describes only shipped behavior.
      fail: Says "coming soon" or references an unreleased version.

The examples.fail line is the useful one. It shows what violating this looks like. Between the rationale and that example, the offending sentence is usually obvious.

If you genuinely disagree with the verdict, that is a conversation with whoever owns the eval, not a thing to work around. Say so in the pull request.

`needs-review`

The judge was not confident enough either way, so the eval is waiting on a person.

You cannot resolve this yourself. It needs someone with standing to record a verdict. Ask whoever owns the docs quality checks in your repo; the command they run is manni docevals review <file> <eval> pass|fail.

This is a legitimate outcome, not you having done something wrong.

Stale generated script

Some checks are scripts generated from a plain-language assertion. If someone edited the assertion without regenerating, the script no longer matches it.

Not yours to fix unless you were the one who edited the assertion. In that case, regenerate:

npx @hawkeyexl/manni docevals generate <the file>

and commit the updated script alongside your change.

Exit 2

Bad config, a missing provider key, a malformed flag, or a run that would have checked nothing. This is not a docs problem and not your fault. Tell whoever owns the pipeline. Re-running the job will not help.

No evals resolved is the one worth naming, because it looks like the tool is complaining about your pages. It is not. Nothing in the config attaches an eval to any page the run would check. The job stopped rather than reporting success over an empty report. That is a configuration fix, not a content fix.

Reproduce it locally

CI had a provider key and a warm cache; your laptop has neither. A naive local run either fails on a missing credential or costs money.

Run the deterministic checks on just your file:

npx @hawkeyexl/manni docevals run docs/your-page.mdx --deterministic-only

No API key needed, no cost. This covers everything except AI verdicts.

Re-run only the eval that failed

Your CI output names the eval. Hand that name to --eval and nothing else runs:

npx @hawkeyexl/manni docevals run docs/your-page.mdx --deterministic-only --eval no-todo-markers

Seconds instead of a whole pass, and the output is the one check you are trying to fix. Repeat after each edit until it goes green.

Two things about that output:

The Suites lines will say partial — filtered run, target not evaluated. That is not a failure. A filtered run only measured part of the suite, so it prints the numbers and withholds the verdict. Read the eval lines above it; those are yours.
A name that matches nothing is exit 2, not a clean run. matched no evals means you mistyped the name, or that eval does not apply to this page. Check with list, below.

Because it never reports a suite as meeting its target, --eval is a debugging aid and not a replacement for the real check. Push and let CI run everything.

To see what applies to a page without running anything:

npx @hawkeyexl/manni docevals list docs/your-page.mdx

Exit 0 means clean.

Still stuck?

FAQ, the recurring questions
Ask whoever configured the check. A check that blocks contributors who cannot self-serve is a problem with the check, and worth saying so.