moose-tracevals

Agent instructions accumulate. A SKILL.md here, a subagent definition there, a CLAUDE.md that has grown past the point anyone rereads it. Nothing checks whether any of it is still being followed.

moose-tracevals closes that loop. Give it a session that already happened, and it deterministically looks up every skill, agent definition, slash command, and project-rules file the session used. Then it evaluates whether the session adhered to the instructions in those artifacts.

What you want to do

Evaluate my first sessionInstall, find a session, and read a real result, offline and with no API key.

Check one specific instructionDeclare an eval in an artifact's frontmatter and make it fail when it should.

Gate agent work in CIOne workflow, no network calls, an exit code you can branch on.

Decide whether to trust the judgeEnsembles, consensus, and why an errored run can never produce a silent pass.

Understand why my build went redOne page: what failed, whether the verdict holds, and what to do about it.

Look something upEvery flag, config key, grader option, and report field.

The 30-second proof

Everything below runs with no API key and no network call.

npm install --save-dev moose-tracevals

npx moose-tracevals run ./session.jsonl --deterministic-only

  PASS   fix-bug › used-read
  FAIL   fix-bug › forbidden-tool
         [error] tool Bash was used 1 time(s) but must not be
  SKIP   CLAUDE.md › adheres-to-artifact (implicit)
         ai evals skipped (deterministic-only run)

9 eval(s): 2 pass, 1 fail, 0 error, 1 needs-review, 5 skipped

The skill said “this skill is edit-only”. The session ran a shell command. Nobody had to notice.

Nothing to install first

The package and the binary are both named moose-tracevals, so npx moose-tracevals reaches this tool whether or not it is already a dependency of your project.

How it works

trace (~/.claude/projects/<project>/*.jsonl, or a file you name)
  |
  v
parse ──> resolve artifacts used ──> extract evals ──> plan evals
              (skills, agents,          (metadata.evals
               CLAUDE.md/AGENTS.md)      frontmatter)
  |
  v
deterministic graders ──> ensemble LLM judge (N runs, consensus, zones)
  |
  v
report (human / json / markdown) + artifact coverage + history

Nothing to instrument

Evaluation runs against sessions that already happened. No wrapper, no re-running work, and run never modifies a trace or the artifacts it evaluates.

Deterministic lookup

Which skills, agents, and rules a session used is read from the trace and the filesystem, never guessed by a model. An artifact that cannot be found becomes a coverage note, never a crash.

Useful with zero configuration

An artifact that declares no evals still gets one judged eval: did the session adhere to this? Your first run says something before you have authored anything.

A judge you can audit

Independent runs at temperature 0, combined by consensus. An errored run can push an eval to human review. It can never push one to a silent pass.