Performance

The runtime's overhead per call is small but not zero. This page documents what the substrate does for you, what it expects you to configure, and where to look when latency or throughput matter.

Where the time goes

A single LLM call through the runtime walks five distinct phases. Knowing the rough budget for each helps when you're profiling.

The provider call dominates. Everything else stays in the single-millisecond range. If your per-call overhead is multi-tens of ms, something custom is the cause — usually a slow plugin hook, a heavy runtime-aware prompt contribution, or a stream observer doing work that should be async.

Concretely: a turn that takes 4.2s end-to-end against a model that reports 4.1s of provider latency on the ledger row leaves ~100ms of runtime overhead — well within the table's budget (1–5ms prompt + <1ms fingerprint + <1ms channels per chunk + <1ms ledger), with the slack spent on the chunk-by-chunk channel writes during streaming. A turn that takes 4.5s against the same 4.1s provider latency leaves ~400ms unaccounted — that's the signal to profile plugin hooks, because the substrate's own phases don't add up to that number.

Fingerprint-based dedup

Every call computes a fingerprint. Two calls with the same fingerprint hash are interchangeable — the runtime can serve the second from the first's recorded result.

The cache layer isn't built in; it's a contract that lets you plug one in. The simplest pattern:

import { computeFingerprint, canonicalize, sha256 } from "@pleach/core/fingerprint";

const fp    = computeFingerprint(fingerprintInput);
// The Fingerprint carries typed component fields (promptHash, toolsHash, …);
// derive the canonical cache key with the same canonical-JSON + sha256 helpers
// the fingerprint uses internally, so keys join cleanly across processes.
const fpKey = sha256(canonicalize(fp));

const cached = await myCache.get(fpKey);
if (cached) return cached;

const result = await provider.execute(config);
await myCache.set(fpKey, result, { ttl: 3600 });
return result;

Wrap this in a provider decorator and you get cache-hit reuse without changing application code. The fingerprint contract guarantees the cache is sound — same hash means same output for the same model.

What survives a cache hit

A cache hit is byte-identical to the recorded result; the audit ledger still writes a row, but the row carries a cacheHit: true flag in the payload. Two implications:

1. Latency drops to a single network round trip (cache fetch + audit write).
2. The ledger remains complete — every call is still recorded, even cached ones. GROUP BY model rollups stay accurate.

What invalidates the cache

The fingerprint includes pleachVersion, so substrate upgrades invalidate automatically. Within a version, anything in the fingerprint's IN set (model, messages, tools, system prompt, temperature bucket, seed, family, call class, runtime mode, tenant, active safety policies) is the invalidation surface.

A new prompt contribution invalidates the cache; toggling a safety policy invalidates the cache; changing model id invalidates the cache. That's the contract.

Prompt caching (provider-side)

Distinct from the fingerprint cache: provider-side prompt caching is what Anthropic and OpenAI offer to discount the input-token bill for repeat prefixes. Two integration paths:

Native via AnthropicSdkProvider

The Anthropic provider passes through Anthropic's prompt-cache control headers. Mark cacheable sections with cache_control markers in the system prompt; the provider forwards them.

const runtime = new SessionRuntime({
  provider: new AnthropicSdkProvider({
    apiKey: process.env.ANTHROPIC_API_KEY!,
    model:  "claude-sonnet-4-5",
    // Prompt cache is on by default for cacheable content blocks.
  }),
  // ...
});

Via the AI SDK

The AI SDK's experimental_providerOptions field on streamText threads through AiSdkProvider.providerConfig. Use it to pass prompt-cache flags per call.

In both cases, prompt caching and fingerprint dedup compose: the fingerprint cache replays the whole response; prompt caching discounts the prefix tokens on a fresh provider call. Use both.

Batching tool calls

ToolBatchExecutor groups concurrent tool calls per the batching strategy declared on each tool. The default inferred strategy is conservative (serial); override on tools that are safe to parallelize.

// lib/tools/fetchDocument.ts
import { defineTool, Semaphore } from "@pleach/core";

export const fetchDocument = defineTool({
  name: "fetch_document",
  description: "Fetch one document by id.",
  inputSchema: z.object({ id: z.string() }),
  async execute(input, ctx) { /* ... */ },
  // @ts-expect-error — extended field
  batching: { strategy: "parallel", maxConcurrency: 5 },
});

Choosing strategy:

Use Semaphore inside execute for hand-rolled concurrency limits when the tool itself fans out further.

A search tool that fetches the top hits in parallel, capped at four in flight at once:

// lib/tools/searchCorpus.ts
import { defineTool, Semaphore } from "@pleach/core";

const gate = new Semaphore(4);

export const searchCorpus = defineTool({
  name: "search_corpus",
  description: "Search the knowledge base and hydrate the top hits.",
  inputSchema: z.object({ query: z.string(), topK: z.number().default(8) }),
  async execute({ query, topK }, ctx) {
    const hits = await index.search(query, { limit: topK, signal: ctx.signal });
    return Promise.all(
      hits.map(async (h) => {
        await gate.acquire();
        try {
          return await fetchDocument(h.id, ctx.signal);
        } finally {
          gate.release();
        }
      }),
    );
  },
});

The semaphore lives at module scope, so it caps concurrency across every concurrent invocation of search_corpus in the process — not just within one call.

Fan-out, fan-in subagents

When a turn needs three independent sub-analyses, spawn them in parallel and await the joined result. The runtime tracks each subagent under its own turnId; the audit ledger keeps the attribution clean:

async execute(input, ctx) {
  const [summary, facts, label] = await Promise.all([
    ctx.spawnSubagent({ tool: "summarize",      input: { docId: "doc-abc123" } }),
    ctx.spawnSubagent({ tool: "extract_facts",  input: { docId: "doc-abc123" } }),
    ctx.spawnSubagent({ tool: "classify",       input: { docId: "doc-abc123" } }),
  ]);
  return { summary, facts, label };
}

Three rows in harness_auditable_calls, three subagent.completed events, one parent turn — wall-clock latency is the slowest branch, not the sum.

Lazy module loading

Hosts mid-migration use setHarnessModuleLoader with dynamic imports. The runtime caches resolved modules — each key resolves once per runtime lifetime, then stays in memory.

For cold-start-sensitive deployments (Fluid Compute, Lambda, Cloudflare Workers), the first call after cold boot pays the dynamic-import cost for every key it hits. Two mitigations:

1. Warm critical paths in your handler init. Resolve the keys the first turn will need before the request lands.
2. Use the typed config surface where possible. provider, tools, plugins, metaToolNames, and the storage adapters never go through dynamic import — they're direct references.

Channel write overhead

Channel writes are O(1) for LastValue / EphemeralValue / NamedBarrier, O(N) for Topic / BinaryOperatorAggregate (N = size of the existing accumulator), and O(log N) for DataChannel (LRU map operations).

For high-throughput per-step writes, prefer Topic over BinaryOperatorAggregate — appending a single item is O(1) amortized, whereas a BinaryOperatorAggregate write applies the reducer over the existing value. A concrete case: a search_corpus result with 500 hits, fan-in through BinaryOperatorAggregate + appendReducer, applies the reducer once per write — the 500th write copies a 499-element array. The same result through a Topic lands each item in O(1). For a single 500-hit turn the difference is irrelevant; for a 5000-hit turn it's the difference between a turn that lands in 100ms and one that lands in 2s of pure channel-write overhead.

When the reducer dominates (e.g. a per-message dedup pass over many thousand messages), bucket the writes — accumulate in an ephemeral channel within a step, flush once at end-of-step.

Stream observer cost

Observers fire on every chunk. onChunk is sync-only by contract, so the cost is bounded; the runtime won't await an observer that returns a promise.

For observers that need to do something expensive (network call, DB write), pattern: observe synchronously, emit a named-channel envelope, handle the expensive work in a post-turn node.

contributeStreamObservers: () => [{
  name: "detect-citation",
  onChunk(chunk, ctx) {
    if (looksLikeCitation(chunk.text)) {
      ctx.emit("citations.detected", { fragment: chunk.text });
    }
    return { verdict: "continue" };
  },
}],

The post-turn node reads citations.detected and does the network resolution — async work happens off the hot path.

Audit ledger batching

The ProviderDecisionLedger contract permits batching: up to 50 ms or 32 records, flushed unconditionally at end-of-turn (writing one auditable call row per record). Reference Supabase adapter implementations batch by default.

For very high-throughput deployments (multi-thousand-RPS), the adapter is the natural place to add a queue + worker pattern — the contract is fire-and-forget, so a queued write that lands later still satisfies the invariants.

DevTools-driven profiling

When a turn is mysteriously slow in development:

__HARNESS_DEVTOOLS__.events({ types: ["step.start", "step.end"] });
// Walk pairs to see per-step latency.

__HARNESS_DEVTOOLS__.ledger().map((r) => ({
  call:    r.callClass,
  model:   r.modelId,
  latency: r.latencyMs,
}));
// Per-call latency breakdown.

The per-call latency on the audit row is provider-side latency. Subtract from observed end-to-end and you get a runtime overhead estimate. Typical overhead is single-millisecond; multi-tens of ms is the signal to look for a hot-path plugin. The DevTools ledger view also surfaces cacheHit — a turn with three rows all showing cacheHit: true should report effectively zero provider latency (the fingerprint replay path doesn't dial out), so a cache-hit row reporting 500ms of latencyMs is itself a bug signal (likely a stale recording that's slower than the live provider).

When to bring in @pleach/gateway

The gateway SKU is the right pick when:

• Per-tenant routing rules need to be policy-driven, not hardcoded.
• Cost attribution needs to roll up across providers (Anthropic rate-limited fallback to OpenAI, for example).
• Observability needs to feed an external sink (Datadog, Honeycomb) at call granularity.

It's a plugin against @pleach/core — no architectural change, just an extra contribution in your plugin set.

Where to go next