Custom verdict schema

Wording the verdict in your own domain’s language, and guarding the one contract the library cannot check for you.

Running the ensemble is judge and consensus. The canonical schema is in the judge reference.

Why the seam exists

EnsembleOptions.schema replaces the built-in VERDICT_SCHEMA for an ensemble. The reason is not structural flexibility — it is this:

Field descriptions are prompt surface. They reach the model and measurably steer it.

A claim field described in terms of documentation pages produces different verdicts than one described in terms of agent traces. Every production consumer of this package overrides the schema, and every one of them changes only $id, title, and the descriptions. None change the structure.

That is the pattern to copy.

Do it

// Wording a verdict schema for your own domain, and the runtime hole to guard.
// Runs with no API key: MockProvider stands in for a real provider.

import { MockProvider, VERDICT_SCHEMA, judge, runEnsemble } from "@hawkeyexl/inference";

// Start from the canonical schema and change only $id, title, and descriptions.
// Descriptions are prompt surface: they reach the model and steer it.
const traceVerdictSchema = structuredClone(VERDICT_SCHEMA);
traceVerdictSchema["$id"] = "my-tool:trace-verdict:1";
traceVerdictSchema["title"] = "Agent trace adherence verdict";
traceVerdictSchema["properties"]["claim"]["description"] =
  "The instruction the agent session was supposed to follow.";
traceVerdictSchema["properties"]["observed"]["description"] =
  "What the agent actually did, quoted from the trace.";
traceVerdictSchema["properties"]["reasoning"]["description"] =
  "Why the trace does or does not satisfy the instruction.";

console.log("canonical $id:", VERDICT_SCHEMA["$id"]);
console.log("override $id: ", traceVerdictSchema["$id"]);
console.log("structure unchanged:",
  JSON.stringify(Object.keys(traceVerdictSchema["properties"])) ===
  JSON.stringify(Object.keys(VERDICT_SCHEMA["properties"])));

const provider = new MockProvider([
  {
    json: {
      claim: "The agent reads the skill file before acting.",
      observed: "The session opened SKILL.md at step 2.",
      match: "pass",
      confidence: 0.94,
      reasoning: "The read precedes every tool call that depends on it.",
    },
  },
]);

const consensus = await judge({
  provider,
  system: "You evaluate whether an agent session followed its instructions.",
  user: "# Instruction\nRead the skill file first.\n\n# Trace\n...",
  runs: 3,
  schema: traceVerdictSchema, // the override seam
});

console.log("verdict:", consensus.verdict, "zone:", consensus.zone);

// The provider was asked for YOUR schema, not the built-in one.
console.log("schema sent was the override:", provider.requests[0].schema["$id"] === "my-tool:trace-verdict:1");

// The hole: the library validates against whatever schema you give it. It does not
// check that your schema produces JudgeVerdict-shaped objects. Drop a field and you
// get runs that validate and then break the consensus math — at runtime, not compile time.
const missingConfidence = structuredClone(traceVerdictSchema);
delete missingConfidence["properties"]["confidence"];
missingConfidence["required"] = missingConfidence["required"].filter((f) => f !== "confidence");

const badProvider = new MockProvider([
  {
    json: {
      claim: "c",
      observed: "o",
      match: "pass",
      reasoning: "r", // no confidence
    },
  },
]);

const runs = await runEnsemble({
  provider: badProvider,
  system: "s",
  user: "u",
  runs: 1,
  schema: missingConfidence,
});

// It validated happily, because the schema said confidence was not required.
console.log("run validated:", runs[0].error === undefined);
console.log("but confidence is:", runs[0].verdict?.confidence);
console.log("guard this with a round-trip test in your own suite");

canonical $id: inference:verdict:0.1
override $id:  my-tool:trace-verdict:1
structure unchanged: true
verdict: pass zone: auto-pass
schema sent was the override: true
run validated: true
but confidence is: undefined
guard this with a round-trip test in your own suite

Clone VERDICT_SCHEMA. It is exported. Starting from it is what keeps the structure right.
Change $id and title. Give it your own namespace so a cached entry written against your schema is never confused with one written against the canonical shape.
Rewrite the descriptions in your domain’s vocabulary. This is the part that changes results.
Pass it as schema to judge or runEnsemble. The validator is compiled once for the whole ensemble.

The hole you have to guard

The last three lines of the sample output are the failure, reproduced deliberately: a schema with confidence removed yields runs that validate cleanly and then carry undefined where the consensus math expects a number. Nothing throws. meanConfidence quietly becomes meaningless, and zone routing with it.

TypeScript cannot catch this — schema is a Record<string, unknown>, so there is no type relationship between it and JudgeVerdict.

Guard it yourself, in three lines, in your own test suite:

import { MockProvider, runEnsemble } from "@hawkeyexl/inference";
import myVerdictSchema from "./my-verdict-schema.json" with { type: "json" };

it("produces JudgeVerdict-shaped verdicts", async () => {
  const provider = new MockProvider([{ json: aRepresentativeVerdict }]);
  const [run] = await runEnsemble({ provider, system: "s", user: "u", runs: 1, schema: myVerdictSchema });
  expect(typeof run.verdict?.confidence).toBe("number");
  expect(["pass", "fail", "partial"]).toContain(run.verdict?.match);
});

What must not change

Field	Why it is load-bearing
`match`	`computeConsensus` counts `pass`/`fail`/`partial`. An unknown value is not counted.
`confidence`	Bounded 0–1. `meanConfidence` and both zone thresholds read it.
`claim`, `observed`, `reasoning`	Not read by the math, but they are what a human reviewing a `human-review` result actually reads. Dropping them makes the review zone useless.

Keep additionalProperties: false. It is what stops a model from padding the object with fields your downstream code will not expect.

Under a local model

Grammar-constrained decoding changes four things, and a custom schema is where they bite:

required is ignored — every key in properties is emitted regardless.
additionalProperties defaults to false.
Numeric bounds are not enforced by the grammar; an out-of-range confidence comes back well-formed and is caught by Ajv and the retry.
Descriptions are invisible to the grammar, so the provider restates the schema in the system prompt. Your wording still works — it arrives by a different route.

None of these affect the canonical schema, which requires all its fields. They matter as soon as yours has optional ones. See choosing a model.

Caching — put a schema version in your cache key
Upgrading — what a schema change does to entries you already hold
ADR 01001 — the full reasoning