Prompt caching

A cache hit is determined entirely by the fingerprint. The variable surface the backend stores against it is the recorded response — content, toolCalls, finishReason, token usage, modelId, and (for streaming calls) the chunk sequence — plus recordedAt/recordedBy provenance and a sizeBytes accounting field the backend maintains on set.

The seam derives the fingerprint key from @pleach/core/fingerprint; this module is only about storage and retrieval semantics.

Public exports

CacheBackend

Memory and Supabase backends both implement this surface. The seam holds only this contract.

interface CacheBackend {
  readonly id: string;
  get(key: Fingerprint, options: CacheGetOptions): Promise<CacheEntry | null>;
  set(key: Fingerprint, entry: CacheEntry): Promise<void>;
  delete(key: Fingerprint): Promise<void>;
  list(prefix: Partial<Fingerprint>): AsyncIterable<Fingerprint>;
  stats(): Promise<CacheStats>;
}

Four invariants every implementation maintains:

1. get() returns a deep-frozen entry — callers must not mutate.
2. set() is idempotent on fingerprint (last-write-wins).
3. stats() is O(1) — counters update on every mutation.
4. list() is async-iterable even for memory implementations so callers handle pagination from day one.

CacheGetMode

The per-call failure-mode policy. Distinct from CacheReadPolicy in @pleach/core/fingerprint, which governs cross-RuntimeMode boundary reads.

Same backend instance serves both — the mode is a per-call decision, not a backend configuration.

CacheEntry

The recorded payload. Opaque to the backend; the seam owns shape validation.

CacheStats

interface CacheStats {
  readonly id: string;
  readonly entryCount: number;
  readonly sizeBytes: number;
  readonly hits: number;
  readonly misses: number;
  readonly hitRatio: number;
  readonly evictions: number;
}

hitRatio is hits / (hits + misses), or 0 when both are zero. Counters update on every mutation, so stats() is O(1).

Memory backend

import { createMemoryCacheBackend } from "@pleach/core/cache";

const cache = createMemoryCacheBackend({
  maxEntries: 1000,         // default 1000
  maxBytes: 64 * 1024 * 1024, // default 64 MB
  id: "memory",             // default "memory"
});

const entry = await cache.get(fingerprint, { mode: "best-effort" });
if (!entry) {
  const response = await seam.invoke(...);
  await cache.set(fingerprint, buildEntry(response));
}

The implementation rides on JavaScript's insertion-ordered Map: get() re-inserts on hit so the iterator's first entry is the LRU candidate. Eviction runs after every set() until both maxEntries and maxBytes caps are satisfied.

Wiring into a runtime

SessionRuntimeConfig.cacheBackend is the field every seam reads through. Pass the backend at construction and every provider call shares the same cache instance.

import { SessionRuntime } from "@pleach/core";
import { createMemoryCacheBackend } from "@pleach/core/cache";

const cacheBackend = createMemoryCacheBackend({ maxEntries: 5_000 });

const runtime = new SessionRuntime({
  storage,
  userId: "user_123",
  cacheBackend,
});

// Later, inspect counters:
const stats = await cacheBackend.stats();
console.log(stats.hits, stats.misses, stats.hitRatio);

Picking a mode per call

Same backend instance; different mode per get(). Production traffic reads best-effort so a backend outage degrades to a miss; replay reads strict-fail so a missed hit fails fast instead of re-invoking the provider.

// Interactive request — a cache outage must not break the user-facing call.
const cached = await cacheBackend.get(fingerprint, { mode: "best-effort" });
if (cached) return cached;
return await seam.invoke(...);

// Replay run — a missed hit means the cache is wrong, not the model.
const replayed = await cacheBackend.get(fingerprint, { mode: "strict-fail" });
if (!replayed) throw new Error(`replay miss: ${fingerprint.promptHash}`);
return replayed;

Where to go next