Cache & memoization

The runtime memoizes prepared LLM inputs through a CacheBackend contract — expanded system prompts, normalized message arrays, the work the seam does before it hands a request to the provider. This is runtime-side caching. It's distinct from provider-side prompt caching, which is the Anthropic/OpenAI server-side prefix-reuse mechanism the substrate also rides on.

Both layers cut latency. Only this one is under the substrate's direct control, which is why the contract is short and the correctness rules are explicit.

CacheBackend contract

One interface; three methods. Every implementation conforms.

interface CacheBackend {
  get(key: Fingerprint): Promise<CacheEntry | null>;
  set(key: Fingerprint, value: CacheEntry): Promise<void>;
  metricsSnapshot(): CacheMetricsSnapshot;
}

Two contract rules:

1. set is fire-and-forget. The seam invokes it without awaiting in the hot path. A backend that throws synchronously breaks the turn — see What not to do.
2. metricsSnapshot is O(1). Counters update on every mutation; the snapshot reads them, never recomputes.

The returned CacheMetricsSnapshot carries hit/miss counters, entry count, and byte usage:

interface CacheMetricsSnapshot {
  readonly hits: number;
  readonly misses: number;
  readonly entryCount: number;
  readonly sizeBytes: number;
}

A miss returns null rather than throwing. A backend outage under the best-effort policy degrades to a miss too — the seam falls through to the provider call instead of failing the turn.

CacheGetMode vs CacheReadPolicy

Two orthogonal policies, both on the read path. Don't conflate them.

strict-fail is correct for headless-replay (a missed hit breaks determinism). best-effort is correct for interactive production traffic (a cache outage must not break user-facing calls). The two policies compose — strict-mode + best-effort is the common production shape.

The default memoryCacheBackend

The substrate default. PA-2 C2 Phase 3 promoted memoryCacheBackend from "must be configured" to default-constructed in the SessionRuntime constructor. Every runtime ships with a live cache out of the box — no cacheBackend field on SessionRuntimeConfig is required.

Default cap: 1000 entries / 64 MB, whichever fills first. Eviction is LRU over insertion order — get() re-inserts on hit, so the oldest unread entry is the next eviction candidate.

import { SessionRuntime } from "@pleach/core";
import { createMemoryCacheBackend } from "@pleach/core/cache";

// Default — runtime constructs a memoryCacheBackend with 1000/64MB caps.
const runtime = new SessionRuntime({
  storage,
  userId: "user_123",
});

// Override at construction with a larger memory cache.
const runtime2 = new SessionRuntime({
  storage,
  userId: "user_123",
  cacheBackend: createMemoryCacheBackend({
    maxEntries: 10_000,
    maxBytes: 256 * 1024 * 1024,
  }),
});

// Opt out — pass `null` to disable caching entirely.
const runtime3 = new SessionRuntime({
  storage,
  userId: "user_123",
  cacheBackend: null,
});

A custom backend (Redis, Memcached, Cloudflare KV) implements the same CacheBackend interface and slots into the same field. The seam doesn't care which it got.

Reach: all four seam factories

The runtime threads cacheBackend through to every seam construction site. The four seam factories — synthesizeSeam, reasoningSeam, utilitySeam, converseSeam — accept cacheBackend?: CacheBackend on their Create<Class>SeamOptions and forward it to the inner createSeam(...). So the cache is live at every call class out of the gate; you don't need to wire it per-seam.

prepareCacheInputs adapter callback

Hosts hook this to canonicalize what enters the cache key. Without it, volatile fields (request timestamps, monotonic correlation ids, per-request trace headers) would pollute the fingerprint and depress the hit rate without changing the semantic input.

The callback receives the prepared input the seam is about to fingerprint and returns the canonicalized form. The substrate ships a no-op default — every input passes through unchanged. See src/cache/ for the exact signature; the shape is intentionally narrow so hosts can express "strip these fields" without subscribing to a larger preprocessing pipeline.

A typical implementation strips request-scoped metadata before fingerprinting, leaving the system prompt, the message array, the tool list, and the model id intact.

The four fingerprint gaps

The cache key includes four fields that consumers must be aware of:

Two reads with different values across any of those four miss the cache. By design. The substrate trades hit rate for correctness — a hit that returns the wrong response is worse than a miss that re-invokes the provider.

Two read modes

The CacheReadPolicy governs how strict the runtimeMode gap is at read time. Default is strict-mode.

strict-mode

A turn running in mode A cannot read a row written by mode B. The key includes runtimeMode and the lookup fails when the modes differ.

This is the safe default. live-mode turns may have invoked non-deterministic tools whose results shouldn't replay verbatim under headless-replay; isolating reads by mode keeps each mode's cache semantics independent.

cross-mode-readable

A turn may read a row written under a different mode when the prepared input is mode-independent. The seam evaluates the mode-independence claim at read time; the row is returned only when the claim holds.

Cross-mode reads stay in soak today. Hosts should keep the default until the cross-mode contract clears — see Versioning for how the gate moves.

Telemetry

metricsSnapshot() returns live counters. Two of the four fields matter for storage-volume monitoring on production:

const snapshot = runtime.getCacheBackend().metricsSnapshot();
console.log(
  `cache: ${snapshot.hits} hits / ${snapshot.misses} misses`,
  `(${snapshot.entryCount} entries, ${snapshot.sizeBytes} bytes)`,
);

The CI gate audit:c2-cache-hit-rate-clean is the canonical hit-rate signal in production. It accumulates per-canvas-batch samples of [UXParity:c2-cache-hit-rate] emissions and fails :strict until a 3-batch soak window shows hit-rate ≥ 5% and median hit-latency < 50 ms. The clock starts on the first production canvas batch after the default-promotion deploy; the ledger lives in-repo at scripts/audit/c2-cache-hit-rate-clean.soak-ledger.json.

Wire the snapshot into your existing metrics pipeline the same way you wire the audit ledger — see Observability for the per-call decorator pattern; cache metrics are coarser, so a periodic snapshot read into a gauge is enough.

What's not in scope today

Cross-mode read of cached state across runtimeMode transitions remains in soak. The contract isn't sealed yet — edge cases around tool-call replay and async-job resumption are still being walked.

Keep strict-mode until the gate clears. The cost of the stricter read is one extra provider call when a mode boundary is crossed; the cost of a premature cross-mode hit is a correctness bug that won't surface until the row replays. See Versioning for the gate-moves-the-mode rollout shape.

The in-memory backend is single-process. Each SessionRuntime instance owns its own Map; two replicas of the same host process don't share cache state. For multi-process or multi-replica deployments — autoscaling Vercel / Cloudflare Workers, fanned-out Kubernetes pods, anything beyond a single Node process — swap in a shared backend (Redis, Memcached, Cloudflare KV, Supabase) by implementing the same CacheBackend contract.

The substrate does not ship a Redis or Supabase cache backend today. Both are straightforward against the contract above — the in-tree memory backend is the reference implementation and the shape consumer adapters mirror.

What not to do

A few patterns that fight the substrate:

• Don't substitute the cache for an audit log. The cache is best-effort and evictable; the audit ledger is append-only and durable. A cache entry that aged out is gone; an audit row is not. Cost attribution, compliance evidence, and replay all read the ledger.
• Don't put PII into prepared inputs without scrubbing first. The cache key derives from the prepared input; a leaked field there shows up in cross-tenant hit-rate diagnostics later. Run inputs through scrubbers before the seam fingerprints them.
• Don't construct a SessionRuntime with a custom cache that throws synchronously. The seam invokes set without awaiting; a synchronous throw escapes the fire-and-forget contract and breaks the turn. Wrap I/O in try/catch inside the backend and degrade to a counter increment.
• Don't tune the fingerprint to chase hit rate. The four gaps are load-bearing for correctness. A higher hit rate that comes from collapsing tenantId or runtimeMode is a cross-tenant bug waiting to ship.

Where to go next