Add moose-tracevals to CI

Scope: getting a dependable gate into automation. Reading results downstream (JSON, history, regressions) is covered in Exit codes & reports. Deciding what the evals assert is Declare what to check.

moose-tracevals can run in a build with no network call and no API key. That is --deterministic-only, and it is a complete mode in its own right.

First, where does the trace come from?

This is the question that decides your whole setup, so it comes before the recipe.

Agent sessions are written on the machine the agent ran on. A CI runner has no session store of its own, so a trace has to get there. Three patterns work:

Pattern	When it fits
Evaluate in the same job that ran the agent	An agent runs in CI and you evaluate its session immediately afterwards. Simplest, and the trace never leaves the runner.
Upload the trace, evaluate in a later job	The agent runs in one job or workflow; the session file is uploaded as an artifact and downloaded by the evaluating job.
Evaluate a committed trace	A small corpus of representative sessions lives in the repo and is re-evaluated whenever artifacts change, giving you a regression suite for your instructions.

The third is the one people underrate. Commit a handful of traces, and this gate answers a useful question on every pull request. Does this edit to CLAUDE.md break a check that used to pass?

A corpus is worth more than a trace

Point run at the whole directory’s worth of traces and it reports rates rather than verdicts:

npx moose-tracevals run traces/*.jsonl --deterministic-only

That changes the question from did this session comply? to how often does this check hold? That second question is the one a skill’s author actually has. One session failing an eval is an anecdote. The same eval at 40% across twenty sessions is a finding. The report names the traces that pulled it down, so you can open them.

Everything about a batch is built to keep that number honest:

• The judge cost budget covers the whole invocation, not each trace. A fifty-trace batch with --max-cost-usd 5 spends at most five dollars. Once it is exhausted the remaining evals report skipped with the reason, never a pass.
• A trace that will not parse becomes a report entry and the batch continues. One corrupt file does not cost you the other forty-nine.
• Skipped evals stay out of the rate. An artifact the session never invoked reports —, not 100%.
• Rows are ordered deterministically, so two runs over the same corpus are comparable.

Exit 1 if any trace fails, exactly as for a single one. Naming exactly one trace still produces the familiar single-session report. See which report you get.

Or let the tool find the sessions

When the agent ran on the same machine, three selectors pull traces out of the session store instead of naming files. They are the same --all-projects and --limit that list uses, plus a time window:

npx moose-tracevals run --since 7d --limit 20 --deterministic-only

A selector always produces the aggregate report, even when it matches one trace. A script’s parsing therefore does not depend on how busy the week was. A selector that matches nothing exits 2 rather than 0. An empty corpus and a clean corpus look identical downstream, and a gate that stopped gating should say so.

Stay offline

Two flags cover every offline case:

• --deterministic-only makes no model call at all. Judged evals are reported as skipped instead of failed, so an artifact with only judged evals does not turn your build red for having been skipped.
• --provider mock exercises the whole pipeline, judge path included, without a network call. Use it to smoke-test wiring, not to check adherence: it returns a canned verdict.

If you do let the judge run in CI

Set a ceiling with judge.maxCostUsd and provision the API key as a secret. Also check that your model’s pricing is known. A model the pricing table does not recognize is costed at zero, which silently disables the budget you set. See configuration.

Control the environment

Trace discovery reads a session store under a home directory. On a shared runner, pin it explicitly instead of inheriting whatever the runner has:

MOOSE_TRACEVALS_HOME=./session-store npx moose-tracevals run ./traces/session.jsonl

MOOSE_TRACEVALS_HOME overrides the home directory for both trace discovery and user-level artifact lookup. A relative value resolves against the current working directory. This repository’s own CI uses it to make runs reproducible on every machine.

Use relative paths in workflow steps

On Windows runners, an absolute POSIX path like /tmp/x resolves differently in a shell than in node.exe. If a step is touched by both, keep the paths relative.

GitHub Actions

A gate that evaluates a committed trace corpus on every pull request:

.github/workflows/moose-tracevals.yml

name: Agent adherence

on:
  pull_request:
    paths:
      - ".claude/**"
      - "CLAUDE.md"
      - "AGENTS.md"
      - "traces/**"
  workflow_dispatch:

permissions:
  contents: read

jobs:
  adherence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7

      - uses: actions/setup-node@v6
        with:
          node-version: 24

      - run: npm install --no-fund --no-audit

      # No network call, no API key. Judged evals are skipped, not failed.
      - name: Evaluate agent sessions
        run: |
          npx moose-tracevals run traces/refactor-session.jsonl \
            --deterministic-only \
            --format markdown \
            --output moose-tracevals-report.md

      # Runs whether or not the gate passed — a failing report is the one you
      # most want to keep.
      - name: Publish the report
        if: always()
        uses: actions/upload-artifact@v5
        with:
          name: moose-tracevals-report
          path: moose-tracevals-report.md

The gate is the exit code. The step fails when run exits non-zero, which is exactly when an eval failed, errored, or needs review.

Evaluating a session the same job just produced

An agent that runs earlier in the workflow writes its session under the runner’s home directory. You can discover it instead of naming a file:

      - name: Evaluate the session that just ran
        run: |
          npx moose-tracevals list --all-projects --limit 1 --json > traces.json
          npx moose-tracevals run "$(node -p "require('./traces.json').traces[0].file")" \
            --deterministic-only

list --json emits { "traces": [...] } sorted newest-first, so the first entry is the session that just finished.

Carry the manifest with the trace

CI gets one thing badly wrong on its own, and it is worth fixing before you trust the report. A checkout rewrites every file’s mtime, so the staleness heuristic flags every artifact in your repository on every run. The warning is technically correct and completely useless.

A session manifest fixes it, because it compares content rather than timestamps. The mechanics are the same three patterns above:

Pattern	Where the manifest is
Evaluate in the same job	capture wrote it to .moose-tracevals/sessions/<id>.json in the workspace, and run finds it there. Nothing to do.
Upload the trace	Upload the manifest with it. Name it <trace>.manifest.json and run picks it up beside the trace with no flag.
Committed corpus	.moose-tracevals/ is gitignored, so copy the manifest next to the trace when you commit it, as traces/refactor-session.manifest.json.

      - name: Upload the session and its manifest
        uses: actions/upload-artifact@v4
        with:
          name: agent-session
          path: |
            session.jsonl
            session.manifest.json

The evaluating job downloads both into one directory, names the trace, and gets exact staleness for free.

Nothing breaks without one

An absent manifest is the default and always will be. Every hash check reports skipped with the reason, the mtime heuristic stands, and no verdict, summary count, or exit code moves either way. A manifest is an observation about the artifacts, never an input to a verdict.

Other CI platforms

There is no platform-specific behavior to account for. The contract is a command, an exit code, and optionally a file. Any system that can run Node 24 and branch on an exit code works:

GitLab CI

.gitlab-ci.yml

agent-adherence:
  image: node:24
  script:
    - npm install --no-fund --no-audit
    - npx moose-tracevals run traces/refactor-session.jsonl --deterministic-only
  artifacts:
    when: always
    paths:
      - moose-tracevals-report.md

pre-commit

.pre-commit-config.yaml

repos:
  - repo: local
    hooks:
      - id: moose-tracevals
        name: agent adherence
        entry: npx moose-tracevals run traces/refactor-session.jsonl --deterministic-only
        language: system
        pass_filenames: false

Where to go next

The exit-code contractWhat 0, 1, and 2 mean, why needs-review counts as a failure by default, and how to change that.

Every configuration keyCost ceilings, provider selection, cache locations, and the knobs that have no CLI flag.