@pleach/eval

How the garden grew this season — fixed seeds, measured shoots, comparable rows. @pleach/eval is the evaluation harness for @pleach/core agent runtimes. The package ships an EvalSuite class that accepts cases as a discriminated union, four built-in scorers, three report formatters (JSON, Markdown, JUnit XML), and a Phase B EvalRuntime factory with live bodies for batch execution, report rendering, structural diffReports, and statistical compareScored (bootstrap + Welch's t-test). Coupling to @pleach/replay is opt-in via dependency injection — the two SKUs install and adopt independently.

This page is the SKU reference. For the DIY workflow using @pleach/core primitives directly today (without this SKU), see Eval and replay.

Install

npm install @pleach/eval @pleach/core

pnpm add @pleach/eval @pleach/core

bun add @pleach/eval @pleach/core

@pleach/core is a peer dependency.

import { EvalSuite } from "@pleach/eval";
import { tokenF1 } from "@pleach/eval/scorers";
import { formatMarkdown } from "@pleach/eval/report";
import { NoopReplayClient } from "@pleach/eval/replay";

The import names above are illustrative. The authoritative export shape lives in the package README on npm — check it when pinning a version, since the surface evolves between Phase A and the 1.0 cut.

The EvalSuite class

EvalSuite is the entry point. Construct it with a suiteId (the identifier that surfaces in reports), register cases, then call run() to produce an EvalReport.

import { EvalSuite } from "@pleach/eval";

const suite = new EvalSuite({ suiteId: "my-eval" });

suite.register({
  kind: "expected",
  id: "greeting",
  input: "hello",
  expected: "hello",
});

const report = await suite.run();
console.log(report.summary);
// { total: 1, passed: 1, failed: 0, errored: 0 }

Cases run in registration order. The report carries a stable runId, the start and completion timestamps, every case result, and a summary count.

The constructor takes a runtime (any EvalSessionRuntimeLike — a real SessionRuntime from @pleach/core satisfies the shape), a cacheBackend (defaults per-suite — see below), and a cacheReadPolicy ("strict-mode" for research replication, "cross-mode-readable" for graded coursework). The runtime is optional in Phase A so the surface unit-tests against a mock; in production callers always provide a real runtime.

Case shapes

EvalCase is a discriminated union with three shapes. There is no global registry — scorers ride inline with the case.

expected — pass/fail equality

A string or RegExp equality check.

suite.register({
  kind: "expected",
  id: "section-cite",
  input: "the model output",
  expected: /section 4\.\d+/,
});

scored — numeric score from a scorer

A scorer returns a score in [0, 1]. Reference a built-in by id or ship a custom function inline.

suite.register({
  kind: "scored",
  id: "f1-against-golden",
  input: { expected: "the quick brown fox" },
  scorer: { kind: "builtin", id: "tokenF1" },
});

suite.register({
  kind: "scored",
  id: "length-bound",
  input: "...",
  scorer: {
    kind: "custom",
    fn: async (actual) => ({
      score: actual.length > 100 ? 1 : 0,
      explanation: `length=${actual.length}`,
    }),
  },
});

A scored case passes when its score is >= 1. Anything lower is a fail, but the numeric score still rolls into the report's mean.

judged — LLM-as-judge ensemble

A list of judge models that score the actual output. Each judge carries a model id, a rubric prompt, and a score schema ("binary", "likert-5", or "numeric").

suite.register({
  kind: "judged",
  id: "summary-accuracy",
  input: "summarize this paper: ...",
  judges: [
    {
      model: "anthropic/claude-opus-4-7",
      rubric: "Rate the summary's factual accuracy on a 1-5 Likert scale.",
      scoreSchema: "likert-5",
    },
  ],
});

Phase A ships the contract; judge dispatch is stubbed. Judged cases return a placeholder score and an explanation noting the stub. The Phase B landing wires dispatch through the runtime's provider seam so judges route through @pleach/gateway when present.

The four built-in scorers

Exported from @pleach/eval/scorers. All four implement the EvalScorerFn contract — (actual, ctx) => EvalScore — so they work as both inline values and EvalScorerRef references.

exactMatch, substringContains, and tokenF1 read the expected value from either a bare string input or an { expected } field on a record input. regexMatch reads { pattern, flags? } (or treats a bare string input as the pattern with no flags). When the expected value is missing, the scorer returns score 0 with an explanation naming the missing field — no thrown exceptions for misconfigured cases.

tokenF1 returns the F1 over the token multiset: 2PR / (P + R), where P is overlap divided by actual token count and R is overlap divided by expected token count. Both sides empty score 1 by convention; one side empty scores 0.

The three report formats

Exported from @pleach/eval/report. The same EvalReport feeds all three formatters — pick one (or several) depending on where the report lands.

import { formatJSON, formatMarkdown, formatJUnitXML } from "@pleach/eval/report";

const report = await suite.run();

console.log(formatJSON(report));        // pretty-printed JSON
console.log(formatMarkdown(report));    // human-readable summary
console.log(formatJUnitXML(report));    // CI consumer

formatJSON sorts case results by caseId and emits stable key ordering — two runs against identical input produce byte-identical JSON output, which makes the formatter usable as a golden-file target. Two-space indent matches jq defaults.

formatMarkdown emits a summary block plus a per-case table with status, score, and explanation. Pipe character escaping is handled.

formatJUnitXML produces a single <testsuite> element with one <testcase> per eval case. Errored cases include <error>, failed cases include <failure>. The shape matches the junit-xml convention used by mocha-junit-reporter and jest-junit, so the output plugs into GitHub Actions, GitLab CI, and Jenkins without a custom reporter.

Replay coupling via constructor DI

@pleach/eval does not import @pleach/replay. The two SKUs adopt independently and couple through a structural contract — EvalReplayClient — that consumers (or @pleach/replay itself) implement.

import { EvalSuite, type EvalReplayClient } from "@pleach/eval";
import { createReplayRuntime } from "@pleach/replay";

const suite = new EvalSuite({ suiteId: "replay-eval", runtime });

// @pleach/replay's turn-granular runtime, adapted to eval's structural
// `EvalReplayClient` contract (`replay(chatId) => { output }`).
const tenantId = "tenant_xyz";
const replayRuntime = createReplayRuntime({ tenantId, sessionRuntime: runtime });
const replayClient: EvalReplayClient = {
  async replay(chatId) {
    const { state } = await replayRuntime.replayTurn({ chatId, tenantId });
    return { output: String(state ?? "") };
  },
};
suite.setReplayClient(replayClient);

const report = await suite.replay("chat_01HKXJ...");

replay(chatId) reconstructs the turn's final output via the injected client and runs every registered case against it. Without a client, replay() throws with a message that names the missing DI.

@pleach/eval/replay ships two test stubs:

• NoopReplayClient — returns empty output for any chat id. Useful when unit-testing the replay() wiring without dragging in the real replay package.
• StaticReplayClient — returns a canned output for any chat id. Useful when unit-testing scorer dispatch against a known output.

import { NoopReplayClient, StaticReplayClient } from "@pleach/eval/replay";

suite.setReplayClient(new StaticReplayClient("the canned output"));

The stubs ship in the same subpath as the contract, so consumers that only need the structural type — not the real replay client — don't pick up a transitive dependency.

Per-suite cache backend

The cache backend defaults to a fresh in-memory backend per EvalSuite instance. That isolation is the load-bearing piece: parallel suites in the same process can't poison each other's cache, and a run() against a deterministic runtime produces byte-identical reports on repeat.

Override when you want the suite to share state with a production runtime:

import { createMemoryCacheBackend } from "@pleach/core/cache";

const suite = new EvalSuite({
  suiteId:          "shared-cache-eval",
  runtime,
  cacheBackend:     createMemoryCacheBackend({ maxEntries: 10_000 }),
  cacheReadPolicy:  "strict-mode",
});

cacheReadPolicy: "strict-mode" is the research-replication setting — a cache miss in strict mode is an error, not a silent fallthrough. "cross-mode-readable" is the graded-coursework setting where reads can cross between interactive / headless-eval / headless-replay buckets. See Cache for the policy contract.

Phase A status

The package shipped at 0.1.0 as Phase A of the SKU. The Phase A EvalSuite contract is in place; two pieces of behavior are stubbed and land in a follow-up:

• Judge dispatch. Judged cases record the ensemble shape but do not invoke any provider. Judge dispatch wires through runtime.providers so judges route through @pleach/gateway when it's configured.
• Default actual-output resolver. Phase A's default resolver echoes the case input back as the actual output. Production callers override via setActualResolver(resolver) to drive a real runtime turn. The default keeps the surface unit-testable; nothing about the report shape changes when the resolver is swapped.

Both behaviors are tracked against the package's published changelog on npm; pin a version with the change you need.

Phase B runtime — createEvalRuntime

The Phase B EvalRuntime factory (createEvalRuntime) ships alongside the Phase A surface and exposes the batch / report / comparison surface used for cross-config and cross-run analysis. At the current cut:

diffReports(prior, next)

Pure-function structural diff between two EvalReportOutput instances. Both inputs must carry the same version and format: "json" (the canonical JSON format is parsed and walked cell-by-cell). The returned EvalReportDiff carries per-cell status (added, removed, changed, unchanged) and a summary roll-up. Use it to verify two pinned report snapshots match, or to surface what changed between a pre- and post-refactor batch.

compareScored(prior, next, opts?)

Statistical comparison of two scored eval runs. Pairs cells by (config, row) identity, then runs bootstrap resampling and Welch's t-test on the paired score deltas — paired-sample comparison with unequal variances (Welch's t-test is the unequal-variance generalization of Student's t-test; bootstrap resampling gives a non-parametric confidence interval that holds without distributional assumptions on the score). The returned EvalComparisonResult carries the bootstrap CI, the t-test statistic + p-value, and per-row pairing diagnostics so a caller can reject "the new config is statistically better" claims that fail the significance bar.

Pair this with @pleach/replay when the eval batches you're comparing are themselves derived from chat replays — @pleach/eval consumes EvalReportOutputs, and @pleach/replay is the substrate that produces deterministic outputs to score.

Where to go next