Eval · Parity

@pleach/eval/parity ships two parity primitives that answer orthogonal questions about output stability:

Both share editDistance from packages/eval/src/divergence/distance.ts as the underlying string comparison primitive, and both lean on the same divergence aggregator shape. The split exists because the shape of the input is genuinely different: runSwarmParity consumes SwarmRunResult[] (N iterations of one configuration), while runParitySuite consumes ParitySuiteRun[] (M configurations on K scenarios).

runSwarmParity

The within-swarm stability runner. Given a swarm runner that produces SwarmRunResult from a scenario, runSwarmParity invokes the runner iterations times per scenario (default 3) and reports per-scenario divergence + outlier session IDs.

import { runSwarmParity } from "@pleach/eval/parity";

const report = await runSwarmParity({
  runner: {
    runScenario: async (scenario) => {
      // Spin up a swarm fan-out from the scenario; collect sub-agent
      // outputs in BFS order.
      return await mySwarmEntryPoint(scenario);
    },
  },
  scenarios: [scenarioA, scenarioB, scenarioC],
  iterations: 3, // default
});

console.log(report.scenarios[0].divergence.outlierSessionIds);

Iteration semantics

Two iterations is the practical minimum (anything less leaves "outlier" undefined); three is the default to surface a stable majority vs a single outlier. Consumers may pass higher iterations for high-variance workloads at proportional cost.

Output shape

interface SwarmParityReport {
  readonly scenarioCount: number;
  readonly iterations: number;
  readonly scenarios: ReadonlyArray<{
    readonly scenarioLabel: string;
    readonly divergence: SwarmDivergenceResult; // .outlierSessionIds lives here
    // ...
  }>;
  readonly meanPairwiseDistance: number;
  readonly outlierSessionIds: ReadonlyArray<string>; // union across scenarios
}

computeSwarmDivergence reduces N iterations into pairwise distance + the outlier-session-id list. Outliers are flagged at a 1.5× threshold against the cohort mean — the structural minimum for a 3-config single-outlier cohort.

runParitySuite

The across-configuration divergence runner. Given M ParitySuiteRun[] (each with its own id and runner(scenario)) and a list of scenarios, runParitySuite runs every configuration over every scenario and emits a scenario-keyed report.

import { runParitySuite } from "@pleach/eval/parity";

const report = await runParitySuite({
  runs: [
    { id: "claude-opus", runner: scenarioRunnerOpus },
    { id: "gpt-5", runner: scenarioRunnerGpt },
    { id: "gemini-2", runner: scenarioRunnerGemini },
  ],
  scenarios: [scenarioA, scenarioB, scenarioC],
  scenarioLabels: ["math-word-problem", "code-refactor", "summarization"],
  tolerance: { distance: 50 }, // characters of editDistance per scenario
});

console.log(report.divergentScenarioIds);
// ["code-refactor"] — the scenarios that exceeded tolerance

Tolerance semantics

tolerance.distance sets the maximum editDistance between any two configurations' outputs for one scenario before that scenario is flagged as divergent. Default: 0 (strict — any character-level difference flags). Per-scenario ScenarioParity.divergent is true when any pairwise distance exceeds the threshold.

report.divergentScenarioIds is the union of flagged scenario IDs — the top-level summary the consumer typically pipes into a CI gate.

Output shape

interface ParityReport {
  readonly runCount: number;
  readonly scenarioCount: number;
  readonly scenarios: ReadonlyArray<ScenarioParity>;
  readonly meanDistance: number;     // cohort mean
  readonly maxDistance: number;      // cohort worst pair
  readonly divergentScenarioIds: ReadonlyArray<string>;
}

interface ScenarioParity {
  readonly scenarioId: string;
  readonly outputs: ReadonlyArray<{ runId: string; output: string }>;
  readonly pairwise: ReadonlyArray<DivergencePair>;
  readonly maxDistance: number;
  readonly meanDistance: number;
  readonly divergent: boolean;
}

Outlier detection — the 1.3× threshold

The cohort outlier set (the configurations whose mean per-scenario distance to peers exceeds a multiplier of the cohort mean) is exposed via the shared computeDivergence aggregator and surfaced as outlierRunIds. The multiplier is 1.3× on the non-swarm runner.

The brief on choosing 1.3× rather than the swarm runner's 1.5×: a 3-config single-outlier cohort with editDistance distribution [10, 10, 30] has a cohort mean of (10 + 10 + 30) / 3 ≈ 16.67; the outlier's per-peer mean is 30, and 30 / 16.67 ≈ 1.8. A 1.5× multiplier is the structural minimum to flag this cohort's outlier; 1.3× is the more sensitive setting for configuration-level parity, where a single character-level divergence already matters more than swarm-level row-level noise.

Sequential dispatch by design

Both runners dispatch sequentially within and across iterations. Consumers own concurrency — wrap runScenario / runner in Promise.all if you want parallel fan-out, but be aware that:

• Deterministic test ordering is preserved by sequential dispatch.
• Rate-limit budget is easier to predict — N iterations × M scenarios maps directly to N × M provider calls.
• Replay safety is straightforward — the dispatch order matches the recorded event-log order.

Consumers who need parallel dispatch typically wrap the runner with their own concurrency adapter and accept that the report's ordering-sensitive fields become substrate-dependent.

Cited source

• packages/eval/src/parity/runSwarmParity.ts — within-swarm runner.
• packages/eval/src/parity/runParitySuite.ts — across-config runner.
• packages/eval/src/parity/swarmDivergenceMetrics.ts — computeSwarmDivergence aggregator.
• packages/eval/src/parity/divergenceMetrics.ts — computeDivergence aggregator.
• packages/eval/src/divergence/distance.ts — shared editDistance primitive.
• packages/eval/src/parity/index.ts — barrel re-export.

Where to go next