Skip to content
moose-tracevals

Get started

Scope: installing moose-tracevals and getting a real, readable result from a session that already happened. It stops once you trust the output. Making the checks say something specific is Declare what to check; putting this in a pipeline is Run it in CI.

Everything on this page runs offline, with no API key.

  1. Install

    Section titled “Install”

    moose-tracevals needs Node.js 24 or newer.

    Terminal window
    npm install --save-dev moose-tracevals

    Then invoke it as npx moose-tracevals.

  2. Find a session

    Section titled “Find a session”

    moose-tracevals evaluates a trace: a record of a session that already happened, sitting on disk. You do not instrument anything and you do not re-run any work.

    Claude Code writes one JSONL file per session under ~/.claude/projects/<project-slug>/. To see what is evaluable:

    Terminal window
    npx moose-tracevals list --all-projects --limit 5
      1. 2026-07-02 00:00  Write a README for this repo.
         C:\work\other-project · 22222222-2222-2222-2222-222222222222 · 1KB
         C:\...\.claude\projects\C--work-other-project\22222222-....jsonl
      2. 2026-07-01 00:00  Fix the crash in src/app.ts.
         C:\work\demo-project · 11111111-1111-1111-1111-111111111111 · 1KB
         C:\...\.claude\projects\C--work-demo-project\11111111-....jsonl

    Each entry shows the timestamp and first prompt, then the project and session id, then the file path you can pass to run. Drop --all-projects to scope to the current directory’s project.

  3. Run your first evaluation

    Section titled “Run your first evaluation”

    Start with the mode that makes no model call at all:

    Terminal window
    npx moose-tracevals run <trace-file> --deterministic-only

    --deterministic-only runs every check that needs no inference call. That is the seven deterministic graders decidable from the trace alone, plus command evals and the human review queue. It skips AI-judged evals instead of failing them.

    To exercise the whole pipeline, including the judge, without a network call:

    Terminal window
    npx moose-tracevals run <trace-file> --provider mock

  4. Read the report

    Section titled “Read the report”

    A deterministic-only run against a project with one skill, one subagent, and two rules files looks like this:

    moose-tracevals — /path/to/session.jsonl
    claude-code · claude-opus-4-8 · 2 turn(s) · /work/demo-project
    

      SKIP   writing-toolkit:identify-ai-tells › adheres-to-artifact (implicit)
             artifact skipped via metadata.eval-skip
      PASS   fix-bug › used-read
      FAIL   fix-bug › forbidden-tool
             [error] tool Bash was used 1 time(s) but must not be
      PASS   fix-bug › no-force-push
      REVIEW fix-bug › refactor-preserved-intent
             awaiting human review
      SKIP   fix-bug › eval-5
             ai evals skipped (deterministic-only run)
      SKIP   doc-writer › adheres-to-artifact (implicit)
             ai evals skipped (deterministic-only run)
      SKIP   CLAUDE.md › adheres-to-artifact (implicit)
             ai evals skipped (deterministic-only run)
      SKIP   AGENTS.md › eval-1
             ai evals skipped (deterministic-only run)
    

    Artifact coverage
      ✓ skill: writing-toolkit:identify-ai-tells /path/to/plugins/.../SKILL.md
      ✓ skill: fix-bug                           /work/demo-project/.claude/skills/fix-bug/SKILL.md
      ○ agent: Explore                           built-in agent (no definition file)
      ✓ agent: doc-writer                        /work/demo-project/.claude/agents/doc-writer.md
      ✓ project-rules: project rules
    

    Warnings
      ! 1 unparseable JSONL line(s) were skipped
    

    9 eval(s): 2 pass, 1 fail, 0 error, 1 needs-review, 5 skipped

    Five things are worth naming, because the rest of the documentation assumes them:

    • An artifact is an instruction file the session used: a skill, an agent definition, or a project-rules file. Here: fix-bug, doc-writer, CLAUDE.md, AGENTS.md, and a plugin skill.
    • An eval is one named check declared on an artifact. used-read and forbidden-tool were declared in fix-bug’s frontmatter.
    • The grader decides how a check is answered. used-read reads the trace directly, and no-force-push runs a script over it. refactor-preserved-intent is REVIEW because it asks for a human, and the SKIPs are AI-judged evals, which --deterministic-only does not run.
    • adheres-to-artifact (implicit) is what you get for free. An artifact that declares no evals still gets one judged eval asking whether the session followed its instructions. A first run therefore says something before you have authored anything.
    • Artifact coverage is the honest answer to what did you not check? A ✓ resolved to a real file; a ○ did not. Explore is a built-in agent with no definition file anywhere, which is expected, and fine.

    The one failure is the point of the exercise. fix-bug declares that it is edit-only; the session ran a shell command; the finding names the tool and the count.

  5. Check the exit code

    Section titled “Check the exit code”
    Code Meaning
    0 every eval passed or was skipped
    1 something failed, errored, or needs review
    2 moose-tracevals itself could not run: bad usage, unreadable trace, unknown format

    The distinction between 1 and 2 is the one to hold on to: 1 means the agent did something, 2 means the tool did. Full detail in Exit codes & reports.

Where to go next

Section titled “Where to go next”

Two doors from here, depending on what you came for.

Make the checks specificThe implicit eval is coarse. Declare an eval on the one instruction you actually care about — then let fill propose the rest.
Put this in a pipelineA GitHub Actions recipe that runs offline, plus the exit-code contract and a policy for needs-review.