How the judge decides

Scope: how a judged eval becomes a pass, fail, or needs-review, with the arithmetic. What an eval should assert is Declare what to check; every configuration key mentioned here is in the configuration reference.

If you are deciding whether to let a judged eval block a merge, this page is the argument. It is made with numbers, because numbers are the only form of this argument worth anything.

One sample is not a measurement

Ask a model once whether a session followed an instruction and you have a sample. moose-tracevals runs the question N independent times, three by default, each a fresh request with no shared context, at temperature 0.

Each run returns a structured verdict:

Field	Meaning
claim	the assertion under evaluation
observed	what actually happened in the session, citing specific messages, tool calls, or instructions
match	pass, fail, or partial
confidence	the model’s own 0–1 confidence
reasoning	why

A run that fails to produce a valid verdict is recorded as an errored run. That covers an API failure and malformed JSON after a retry. It does not disappear. That single fact is what the rest of this page is built on.

From votes to consensus

The runs are tallied into four buckets: pass, fail, partial, and error.

failVotes = fail + partial            // partial counts as fail for the binary outcome,
                                      // but stays visible in the vote counts
verdict   = pass > failVotes ? "pass" : "fail"
graded    = pass + fail + partial     // errored runs are not "graded"
agreement = max(pass, failVotes) / graded
meanConfidence = mean confidence across non-errored runs

Two choices here have consequences:

• partial counts as a failure for the binary outcome. A session that partly followed an instruction did not follow it. The partial votes stay visible in the report. You can still see the difference between a clean three-nil fail and a genuinely ambiguous one.
• A tie goes to fail. pass > failVotes is a strict comparison, so an evenly split ensemble does not resolve as a pass. In practice such a split routes to review anyway (see below).

Confidence zones

Consensus produces numbers. The zone turns them into an outcome, and the bar is high:

Zone	Requires	Outcome
auto-pass	every run voted pass (no fails, no partials, no errors) and mean confidence ≥ judge.zones.autoPass (default 0.8)	pass
auto-fail	no run voted pass, no errors, at least one fail or partial, and mean confidence ≥ judge.zones.autoFail (default 0.8)	fail
human-review	anything else	needs-review

Read that table for what it excludes. Only a unanimous, confident ensemble auto-resolves. One dissenting vote, or one low-confidence run dragging the mean below the threshold, sends the eval to a human.

An errored run can never produce a silent pass

Both auto-resolving zones require zero errored runs. So if a judge run fails, whether the API timed out or the model returned unparseable JSON, the eval cannot land in auto-pass. The worst it can do is push the result to needs-review.

This is the invariant to check if you are deciding whether to trust this system. A failure of the measuring instrument surfaces as “a human should look”, never as “everything is fine”.

One subtlety to absorb before you tune anything: agreement is reported, but it does not route the outcome. Zones are decided by strict unanimity plus meanConfidence. An ensemble two-thirds in agreement has an agreement of 0.67 and goes to review however confident those two runs were.

A worked example

Three runs on the assertion “the session reproduced the bug with a failing test before fixing it”:

Run	match	confidence
1	fail	0.90
2	fail	0.85
3	partial	0.80

votes          = { pass: 0, fail: 2, partial: 1, error: 0 }
failVotes      = 3
verdict        = "fail"          (0 > 3 is false)
graded         = 3
agreement      = 3 / 3 = 1.0
meanConfidence = (0.90 + 0.85 + 0.80) / 3 = 0.85

No run voted pass, there are no errors, and 0.85 ≥ 0.8. That is auto-fail, reported as FAIL.

Change run 3 to pass at 0.95 and nothing auto-resolves. The ensemble is split, so it routes to human-review and is reported as REVIEW, even though mean confidence went up.

What the judge cites

For any non-passing judged eval, the human report shows the vote split and the first observed field:

  REVIEW fix-bug › eval-5
         votes pass:1 fail:2 partial:0 error:0 · confidence 0.87
         The session edited src/app.ts at turn 2 without a preceding test run.

That observed line is the evidence, and it is what makes a judged verdict something you can check. A verdict citing a specific turn and file can be audited; one that says “the session did not comply” cannot.

The prompt asks for exactly that. It wants cited messages, tool calls, and instructions followed or violated. It is also why an assertion written so a stranger could adjudicate it produces better verdicts than one written as a slogan.

What leaves the machine

A judged eval sends one request per ensemble run to the configured provider. With --deterministic-only, no provider is constructed and nothing is sent at all. That is the only setting under which this section has no content.

Otherwise, each request carries exactly this:

Part	Redacted?
The eval’s assertion, evidence, and examples	No, this is text you authored
The source artifact’s body, capped at 8 000 characters	No, see below
Session header, with source, model, cwd, git branch, turn count, skill and agent names	Yes
User prompts and assistant text, within the eval’s window	Yes
Tool names and tool call inputs	Yes
Tool results, file contents, command stdout/stderr	Not sent at all

Two things are worth reading twice. Tool results are never rendered, so a file this session read does not travel with it. And an eval is judged against the window its artifact governed. A skill’s request usually carries a fraction of the session rather than all of it.

Redaction

Everything marked Yes above is filtered before it is sent. The built-in shapes are always on:

• vendor API-key prefixes such as sk-…, AKIA…, gh[pousr]_…, xox…, AIza…, and JWTs
• Authorization values behind Bearer, Basic, and Token
• -----BEGIN … PRIVATE KEY----- blocks
• secret-named assignments in shell (DATABASE_PASSWORD=…) and JSON ("apiKey": "…") form, plus every AWS_*

Add your own with judge.redact; they are applied on top of the built-ins, never instead of them.

Matches are replaced, not deleted, with a labelled placeholder. The labels are [redacted:api-key], [redacted:auth-token], and [redacted:secret-value]. The judge can still see that a credential was used, which is often the adherence question. Bearer [redacted:auth-token] keeps the scheme; AWS_SECRET_ACCESS_KEY=[redacted:secret-value] keeps the variable name.

Redaction runs before the truncation caps, so a maxBlockChars boundary can never land mid-secret and ship the surviving prefix.

Best-effort, not a guarantee

Pattern matching recognizes shapes. A secret that looks like ordinary prose will not be caught, and no pattern list will catch it. That covers an internal hostname, a customer name, and a password with no surrounding assignment. --deterministic-only is the only guarantee that nothing leaves the machine.

The artifact body is deliberately not redacted: it is your own instruction file, and its exact text is part of the cache key. A secret in a SKILL.md is a problem to fix in the SKILL.md (ADR 01020).

Redaction and the cache work together without a knob. The digest is hashed into the cache key, so a redacted run gets its own entry. It can never replay a verdict formed on unredacted text. A pattern that matches nothing leaves the digest byte-identical and still hits the existing entry, so adding a precautionary pattern costs nothing.

What it costs

• Nothing, if you never invoke it. --deterministic-only skips judged evals entirely, and --provider mock exercises the pipeline offline.
• Nothing extra on a repeat. Verdicts are cached, keyed on the provider, model, prompt version, run count, temperature, the rendered trace, and the eval plan. Change any of those and you get a fresh judgement instead of a stale replay.
• Bounded, if you set a ceiling. judge.maxCostUsd is checked before each eval. Once spent, remaining evals are reported as skipped with the budget named, so nothing is dropped silently.

A budget is only as good as the price table

Cost is computed from a built-in per-model price table. A model the table does not recognize is costed at zero, which silently disables maxCostUsd. If you run a model the table does not know, set pricing explicitly in your provider config. See configuration.

Where to go next

Configure runs, zones, and budgetsEvery judge key — ensembleRuns, temperature, zones, cacheDir, redact, maxCostUsd — with its type and default.

Read a verdict you disagree withWhat to do when an eval comes back REVIEW or FAIL and you think it is wrong.