Eval and replay

This page is the DIY pattern against the substrate. For the published SKU references, see @pleach/eval and @pleach/replay.

The substrate ships the primitives @pleach/eval and @pleach/replay build on: deterministic fingerprints, the append-only audit ledger, per-channel checkpoints, and the three runtimeMode modes. @pleach/replay@0.1.0 and @pleach/eval@0.1.0 are shipping today — createReplayRuntime's four entry points (replayTurn, fromSnapshot, fork, aggregateMultiTenant) plus verifyChainIntegrity all have real bodies, and the event-granular ReplayHandle.step() / seek() / replayTurn() stepper is wired against runtime.events.iterate. This page documents the underlying workflow so you can DIY against the substrate before adopting the SKUs, or read what the SKUs do.

This page walks the eval / replay workflow end-to-end. Each section is implementable with only the substrate; sibling-SKU adoption layers typed contracts and scoring on top.

The three workflows

Replay and regression eval are correctness signals — yes/no. Behavioral eval is a quality signal — score-based.

runtimeMode is the eval seam

The runtime distinguishes three operating modes via runtimeMode in the fingerprint key. Picking the right mode is how you opt into the eval workflow.

Set via runtime config or HARNESS_RUNTIME_MODE. The mode is in the fingerprint, so a turn recorded in one mode never collides with the cache of another.

Recording a turn for replay

In production mode, every turn writes a full audit-ledger row per LLM call. The row carries the fingerprint, the metadata, the model id, the token usage, and the decision payload — enough to reconstruct the call.

const runtime = new SessionRuntime({
  storage:                 supabaseAdapter,
  userId:                  "user_123",
  // runtimeMode defaults to "production"
});

for await (const event of runtime.executeMessage(sessionId, prompt)) {
  // ... handle the stream
}

// After the turn, every call's row is in `harness_auditable_calls`.

No additional capture step. The ledger is the recording.

Replaying a turn

Build a fresh runtime around the same storage and ledger, open a ReplayHandle over the chat's event log via @pleach/replay, then advance one assistant turn with handle.replayTurn():

import { SessionRuntime } from "@pleach/core";
import { ReplayClient, type ReplayRuntimeFacet } from "@pleach/replay";

const runtime = new SessionRuntime({
  storage:                supabaseAdapter,
  userId:                 "user_123",
});

// ReplayClient reads only the runtime's `events` facet (iterate + fold).
const replay = new ReplayClient({
  runtime: runtime as unknown as ReplayRuntimeFacet,
});

const handle = await replay.fromEventLog(sessionId, {
  tenantId: "tenant_xyz",   // REQUIRED — replay inherits the runtime's RLS posture
});

const turn = await handle.replayTurn();   // advance to the next turn.completed boundary

console.log(turn.messageId);     // the assistant message that closed the turn
console.log(turn.stepResults);   // the per-event step results, in fold order
console.log(turn.nextState);     // the folded HydratedHarnessState after the turn
await handle.close();

replayTurn(messageId?) loops step() to the next turn.completed boundary, folding each event through @pleach/core's shared hydrateFromEvents reducer. Because incremental step() and a full seek() re-fold share one reducer, N steps produce a state byte-identical to seek(N) — the determinism invariant.

To replay against a changed seam (a different provider, an updated prompt contribution, a new safety policy), construct a fresh runtime around the same storage with that seam swapped in, then open a new handle on the new runtime. The runtime has no withProvider() / withSystemPrompt() mutators — every seam change is a fresh construction.

Verifying byte-identical replay

createStrictHandleReplay from @pleach/replay/strict is the callable form of the determinism invariant. It opens N independent handles over the same (chatId, tenantId, window), walks each to done, and byte-compares the folded state at every step — surfacing the step index where two replays first diverge.

import { ReplayClient, type ReplayRuntimeFacet } from "@pleach/replay";
import { createStrictHandleReplay } from "@pleach/replay/strict";

const replay = new ReplayClient({
  runtime: runtime as unknown as ReplayRuntimeFacet,
});
const strict = createStrictHandleReplay({ client: replay });

const verdict = await strict.replay({ chatId: sessionId, tenantId: "tenant_xyz" });
// → { chatId, deterministic: true, steps: 12 }
//   or { deterministic: false, firstDivergenceAt: 4 }

If the verdict is deterministic: true, the replay is byte-identical. If it diverges, firstDivergenceAt is the step where the chain slipped — walk back through the five contracts in Determinism.

Regression eval across versions

The fingerprint includes pleachVersion. A turn recorded in 1.1.0 will not cache-hit in 1.2.0 automatically — the new version invalidates the bucket.

The regression workflow: replay the recorded turn against the new version without the cache, capture the new output, and diff.

// 1. Record under the old version.
const recordRuntime = new SessionRuntime({
  /* ... */ runtimeMode: "production",  // pleachVersion = "1.1.0"
});
await recordRuntime.executeMessage(sessionId, prompt);

// 2. Re-run under the new version in eval-noncached mode.
//    (Update the package first; pleachVersion = "1.2.0" now.)
const evalRuntime = new SessionRuntime({
  /* ... */ runtimeMode: "eval-noncached",
});
await evalRuntime.executeMessage(newSessionId, prompt);

// 3. Diff the audit ledger rows.
const recorded = await ledger.listBySession(sessionId);
const evaluated = await ledger.listBySession(newSessionId);

const diffs = recorded.map((r, i) => ({
  callIndex:     i,
  modelChanged:  r.modelId !== evaluated[i]?.modelId,
  tokenDelta:    (evaluated[i]?.tokenUsage.out ?? 0) - r.tokenUsage.out,
  outcomeChanged: r.outcome.status !== evaluated[i]?.outcome.status,
}));

The diffs are the regression report. A clean migration produces zero outcomeChanged: true rows; otherwise you've found a behavior change to triage.

Behavioral eval with scorers

Behavioral eval scores model output for quality (helpfulness, accuracy, safety) rather than checking byte equality. Run the turn, then score the output:

import type { AuditableCall } from "@pleach/core/audit";

interface Scorer {
  name: string;
  score(call: AuditableCall, output: string): Promise<number>;
}

const scorers: Scorer[] = [
  {
    name: "factual-accuracy",
    async score(call, output) {
      // Use a judge model to score factual claims:
      const judge = await judgeModel.evaluate({ prompt, output });
      return judge.factualAccuracy;
    },
  },
  {
    name: "concision",
    async score(call, output) {
      return output.length < 1000 ? 1.0 : 1000 / output.length;
    },
  },
];

const synthRow = (await ledger.listBySession(sessionId))
  .find((r) => r.callClass === "synthesize");

const scores = await Promise.all(
  scorers.map(async (s) => ({
    name:  s.name,
    score: await s.score(synthRow, synthRow.payload.output as string),
  })),
);

The scorers are pluggable. Build a library matched to your domain — output-format compliance, citation accuracy, refusal calibration — and run them as a post-hoc pass over the recorded ledger.

The fingerprint pins the comparison

Every score is keyed on (fingerprint, scorer, version). Same input + same model + same scorer = same score. That's what makes "score drift" a real signal rather than measurement noise.

Forking from a checkpoint

Replay can branch from any checkpoint. The substrate ships the checkpoint envelope; the sibling @pleach/replay will ship a typed fork API when published. Today's pattern:

import { SessionRuntime } from "@pleach/core";

// 1. Pick a checkpoint to fork from.
const checkpoints = await runtime.listCheckpoints(sessionId);
const forkPoint = checkpoints.find((c) => c.stageId === "tool-loop");

// 2. Restore into a fresh session.
const forkedSession = await runtime.createSession({
  parentSessionId: sessionId,
  forkFromCheckpoint: forkPoint.id,
});

// 3. Replay (or branch with a different prompt).
for await (const event of runtime.executeMessage(forkedSession.id, "different prompt")) {
  // ...
}

The forked session's audit ledger carries parentSessionId and forkPoint so the lineage stays queryable through LineageTracker.

Eval CI

A typical eval-in-CI shape:

# .github/workflows/eval.yml
jobs:
  regression-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install
      - run: HARNESS_RUNTIME_MODE=eval-noncached npm run eval:replay
      - run: npm run eval:diff > eval-report.md
      - uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: eval-report.md

The eval:replay script runs every fixture against the current build; eval:diff compares the new ledger rows against the golden ledger rows committed in the repo. A PR is mergeable when the diff is empty (or all diffs are explained in the PR description).

What the sibling SKUs will add

When @pleach/eval and @pleach/replay ship their first cuts, they layer convenience on top of the primitives above:

• @pleach/eval — fixture format, scorer registry, diff rendering, CI integration. The workflow above expressed as typed config.
• @pleach/replay — forkFromCheckpoint(runtime, opts), divergence-detection on partial replays, replay-mode optimization for cold-cache scenarios.

Until they ship, the primitives are enough — the workflow above is the substrate's own test suite, with one fingerprint diff per turn the canonical signal.

Where to go next