Troubleshooting

Common symptoms, likely causes, and fixes — runtime construction, streaming, tools, sync, storage, and provider failures.

A failure-mode reference. Each entry pairs an observed symptom with the most likely cause and the concrete fix. When a symptom has multiple causes, they're ordered by frequency.

If a failure isn't here, the Error codes catalog covers the structured 1xxx–9xxx ranges with recovery hints. CI-time audit:* gate failures are covered by Audit gates.

Runtime construction

`[UXParity:metaToolNames-config-missing]` at startup

Cause. SessionRuntimeConfig.metaToolNames not passed; the runtime falls back to an empty set and continuation / fabrication guards depending on the set silently disable.

Fix. Pass the set explicitly:

const runtime = new SessionRuntime({
  metaToolNames: new Set(["set_step_complete", "wait_for_jobs"]),
  // ...
});

See SessionRuntime.

`HarnessModuleLoaderUnregisteredError`

Cause. executeMessage was called before setHarnessModuleLoader(...) registered a loader; happens in hosts mid-migration from legacy orchestrator integration.

Fix. Register the loader at startup. The error message names the first key it tried to resolve — implement that arm at minimum. See Host adapter.

`UnknownSafetyPolicyError`

Cause. A policy id in enabledSafetyPolicies doesn't match any policy registered through contributeSafetyPolicies().

Fix. Check the spelling against the plugin that ships the policy. Programmer-error class — surfaces at construction, not on the first turn.

Bisecting "is the bug in the runtime or my tool?"

Set HARNESS_MOCK_MODE=true and re-run the failing input. Mock mode replaces the provider with a deterministic stub but leaves your tools, plugins, and runtime config untouched:

HARNESS_MOCK_MODE=true npm run dev
# In another shell:
curl -X POST http://localhost:3000/api/chat \
  -H 'content-type: application/json' \
  -d '{"sessionId":"session-018f-7a","message":"fetch doc-abc123"}'

If the bug reproduces under mock mode, it's in tool code or plugin code (the provider isn't running). If it disappears, the bug is in provider config or in the cascade. Never ship code with HARNESS_MOCK_MODE=true set — it's a debug-only switch.

Runtime works in dev, throws in production on first request

Cause. HARNESS_MOCK_MODE=true was set in dev but not in production; the runtime tried to construct a real provider / storage adapter with missing env vars.

Fix. Verify production env. The mock-mode env var should never be set in production; the real OPENROUTER_API_KEY (or ANTHROPIC_API_KEY / OPENAI_API_KEY) plus SUPABASE_URL / SUPABASE_SERVICE_ROLE_KEY should be.

OTel exports are empty, but the runtime is firing turns

Cause. The OTel SDK was started after new SessionRuntime(...) was constructed. The runtime reads the global tracer provider at construction time.

Fix. Call sdk.start() before constructing the runtime. If the runtime is built in a module-level constant, hoist the SDK start to a top-level side-effect import before any runtime import. See OTel observability.

Streaming

Stream silently disconnects after ~30 s in production

Cause. SSE response is being buffered by a reverse proxy or CDN. Most common with Vercel's default response handling when Cache-Control isn't set. The diagnostic: open the network panel and watch the response — if data: lines arrive in a single burst at ~30s instead of trickling, the proxy is buffering; if no bytes arrive at all before the timeout, the connection is being killed upstream. Buffered streams break the chunked rendering; the fix below addresses both.

Fix. Set headers on the SSE response:

return new Response(stream, {
  headers: {
    "Content-Type":  "text/event-stream",
    "Cache-Control": "no-cache, no-transform",
    "Connection":    "keep-alive",
    "X-Accel-Buffering": "no",  // nginx
  },
});

`tool.started` fires but `tool.completed` never does

Cause. Tool's execute is awaiting an external resource that never resolves (no timeout, no abort handling). The diagnostic step: pull the toolCallId from the tool.started payload and query the audit ledger for payload->>'toolCallId' = $1 — if no row exists, execute never returned at all (and the provider call was never made); if a row exists with outcome.status === "failed", the tool errored but the failure event was lost (likely durable flush wasn't registered).

Fix. Honor ToolContext.signal in every fetch / spawn:

async execute(input, ctx) {
  const res = await fetch(url, { signal: ctx.signal });
  // ...
}

The user pressing stop cancels the AbortController; tools that ignore the signal keep running and the stream looks frozen.

`message.delta` chunks arrive, then a `content.correction` replaces them

Cause. Not a bug. A post-stream fabrication guard detected phantom tool references or another correctable issue and rewrote the streamed content.

Fix. Treat correctedContent as the authoritative final text — replace the last assistant message wholesale. See Stream events.

Stream stops mid-token with `stream.truncated`

Cause. A degeneration guard (entropy collapse, phrase loop, substring repetition) detected runaway output and stopped the stream.

Fix. Surface the truncation reason to the user (entropy_collapse typically means the model lost coherence; phrase_loop means it got stuck repeating). The partialLength field tells you how much was emitted before truncation.

Tools

`Tool not found` (code `1001`)

Cause. Tool name in createSession({ tools: { enabled: [...] } }) doesn't match any registered tool, or registration ran after the session was created.

Fix. Register tools before constructing the runtime (or before any createSession call). Verify with DevTools:

__HARNESS_DEVTOOLS__.tools().map((t) => t.name);

Tool fires the wrong arguments

Cause. The model is producing arg shapes that don't match your Zod schema; the validator throws and tool.failed fires with the validation error.

Fix. Two paths.

Tighten the description. The model uses description to pick args; ambiguous descriptions produce ambiguous args.
Loosen the schema. Use z.string().or(z.number()) for fields where the model produces both, then normalize in execute.

Tool runs serially even though it's safe to parallelize

Cause. Default batching strategy is conservative; tools inherit serial unless overridden.

Fix. Mark explicitly:

defineTool({
  name: "fetch_paper",
  // @ts-expect-error — extended field
  batching: { strategy: "parallel", maxConcurrency: 5 },
  // ...
});

See Tools for the strategies.

Outbound HTTP from a tool doesn't carry the tenant header in a tenant-scoped runtime

Cause. The fetch client wasn't wrapped with withTenantHeader. The runtime stamps tenant_id on internal write sites; outbound HTTP is consumer-owned by default.

Fix. Wrap the fetch client at construction (withTenantHeader(fetch, runtime.tenant)) and pass the wrapped client to whatever performs the outbound call — the tool's fetch, the SDK client, the gateway adapter. See Tenant facet.

Storage and sync

`Version conflict` (code `3001`) on every write after a network blip

Cause. The client's version vector has drifted behind the server's; the conflict-detection layer is doing its job.

Fix. Pull the remote vector, merge, and re-push. The SyncCoordinator does this automatically; the symptom usually means a custom adapter is bypassing the coordinator.

`Session not found` (code `2001`) right after `createSession` returns

Cause. The runtime's userId doesn't match the RLS policy's expected scope. Common when constructing with an anon client and no auth context.

Fix. Either pass an authed session token (anon client path) or use a service-role client (server path). Verify with a direct DB query — the row exists; RLS is hiding it.

Sync conflicts surfacing constantly in multi-device setup

Cause. Two clients writing to the same session with overlapping fields and no merger configured.

Fix. Configure a custom merger in SyncCoordinatorConfig that picks remote for fields where the server is authoritative (timestamps, server-derived metadata). See Sync.

Outbox grows unbounded; never drains

Cause. Network is failing silently; the durable-flush retry budget is exhausted; or ConnectivityMonitor is reporting offline when the network is actually up. The diagnostic: check the most recent parked entry's lastError — a 3001 code means the network refused every retry (look at the probe URL); a 3003 means the server is rejecting on version mismatch (look at SyncCoordinator's merger configuration); a 3004 means the outbox cap is too small for the write rate.

Fix. Check __HARNESS_DEVTOOLS__.syncStatus() for the errors array. Forced sync surfaces the underlying error:

await __HARNESS_DEVTOOLS__.forceSync();
__HARNESS_DEVTOOLS__.syncStatus().errors;

A field that used to appear in event log payloads is now blank

Cause. A scrubber matched the field and redacted it before persistence. Scrubbers run at write time; nothing flows past them.

Fix. List the active scrubbers via the plugin contract surface to find which one matched. If the redaction is overly aggressive, override the matcher pattern — for KeyedRegex, narrow the regex; for the bundled scrubbers, register a more specific scrubber that runs first and tags the field as exempt. See Scrubbers.

A custom `GraphProjection<T>` returns stale data on a fresh deployment

Cause. The projection is folding against pre-stamping rows (rows written before the projection's substrate landed) that don't carry the event types the reducer expects.

Fix. Scope the projection's read with since: cutoverId where cutoverId is the first row written after the substrate upgrade. For shipped projections in soak, depend on GraphProjection<T> directly and treat the bundled implementations as references — their signatures may move between releases. See Event log projections.

React

`useHarness` returns stale messages

Cause. HarnessProvider re-mounted because the runtime prop was re-created on every render.

Fix. Wrap runtime construction in useMemo:

const runtime = useMemo(
  () => new SessionRuntime({/* ... */}),
  [userId],  // recreate only when scope changes
);

`useHarnessDevTools` doesn't expose `window.__HARNESS_DEVTOOLS__`

Cause. The hook ran before HarnessProvider mounted, or the hook is inside a conditional that didn't fire.

Fix. Call useHarnessDevTools() inside a component that's a descendant of HarnessProvider. The hook attaches via the runtime context.

Chat resets on tenant switch

Cause. useMemo dep array missing the tenant id; runtime gets reused across tenants.

Fix. Add the tenant to the deps:

const runtime = useMemo(
  () => new SessionRuntime({/* ... */}),
  [tenantId, userId],
);

This is also the right pattern for tenant isolation — see Multi-tenant.

Provider

`LLM unavailable` (code `5001`) on the first call

Cause. Provider credentials missing or invalid. The runtime's cascade walked every in-family rung and all failed.

Fix. Check the env var set per provider — OPENROUTER_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY. Run a smoke test against the provider SDK directly to verify credentials before involving the runtime.

`Rate limited` (code `5003`) under normal load

Cause. Provider-side rate limit hit on a single API key across all sessions; the cascade walks rungs but every rung shares the same upstream limit.

Fix. Two paths.

Multi-key routing. @pleach/gateway@0.1.0 ships the Phase A surface (GatewayClient.route, BYOK fingerprint, family-locked failover for anthropic). Other transports are deferred to Phase B per Gateway. For broader coverage today, wire your own router behind the provider adapter or use a third-party gateway (Portkey, LiteLLM proxy, OpenRouter).
Increase the budget. Contact provider; raise the limit.

`Token limit exceeded` (code `5005`) on long sessions

Cause. Context exceeded the model's window; the runtime emitted a context.summarized event and retried, but the retry also exceeded.

Fix. The runtime's context compaction is automatic but bounded. For sessions that hit it repeatedly, configure a more aggressive compaction strategy or split the session. The diagnostic step: query the ledger for the affected turnId and read payload.tokenUsage.in — a value approaching the model's window cap (200k for Claude Sonnet 4.5, for example) confirms the prompt itself is the bottleneck rather than the response; a value well under cap with the error firing means a tool result inflated the context mid-turn, and the fix is to truncate the tool result upstream rather than tune compaction.

Confirming a fallback fired (without trusting logs)

The audit row is the ground truth — logs may be rate-limited, the ledger is not. Query for the suspect turn:

SELECT
  created_at,
  payload->>'modelId'         AS model,
  payload->>'family'          AS family,
  payload->>'fallbackReason'  AS fallback_reason,
  outcome->>'status'          AS outcome
FROM harness_auditable_calls
WHERE payload->>'turnId' = 'turn-018f-7a-3'
ORDER BY created_at;

A row with non-null fallback_reason confirms the cascade advanced; multiple rows for one turnId show the full walk through the family.

`family-exhausted` event after a fallback cascade

Cause. Every rung in the locked family failed. The runtime emits this rather than silently switching families — see Architecture §4.

Fix. Surface a "pick a different model family" prompt to the user. Silent cross-family widening is structurally disallowed by design.

Audit ledger

`cacheHit: false` on every call, even ones that should be deduped

Cause. No fingerprint cache wired. The substrate ships the fingerprint but not the cache layer — that's a contract you implement.

Fix. Wrap your provider with a fingerprint-keyed cache decorator. See Performance.

Cache hit rate is unexpectedly low despite identical-looking prompts

Cause. One of the four fingerprint gaps differs — systemPrompt, temperature, runtimeMode, or tenantId. The cache key is structurally distinct across any gap; identical content with different gap values misses by design.

Fix. Print the fingerprint via cacheBackend.metricsSnapshot() and compare the gap values across the two calls you expected to share a cache row. If the gap difference is structural (cross-tenant pollution, mode change at turn start), that's correctness — the cache is doing its job. If it's accidental (a timestamp leaking into the system prompt), normalize the input via prepareCacheInputs. See Cache.

`audit:c8-event-type-allowlist-coverage` fails CI after adding a new event type

Cause. The new event type doesn't have a matching Scrubber allowlist entry. Every persisted event type needs scrubber coverage, even if it's a pass-through.

Fix. Register a Scrubber that allowlists the new event type (a pass-through scrubber is fine if no fields need redaction) via contributeScrubbers in the plugin that owns the event type. See Scrubbers, Plugin contract.

Hash-chain verification SQL returns the same row repeatedly

Cause. The recursive CTE's join condition is matching multiple successor rows when no prev_hash constraint is present — typically because the chain has a NULL gap mid-stream.

Fix. Filter to row_hash IS NOT NULL on both sides of the join and walk in record_id order. A gap of NULL rows is back-compat-expected; the verification resumes at the next stamped row. See Hash chain.

After adding `@pleach/compliance`, CI fails on `audit:tenant-scoping`

Cause. The package brings two CI gates with it. Adopting the package without first wiring runtime.tenant (so every write site has a tenant) makes the gate fail.

Fix. Configure tenantId (and optionally subTenantId) on SessionRuntimeConfig before enabling the compliance plugin. Run audit:tenant-scoping locally first; it lists the write sites that need backfill. See Compliance, Tenant facet.

Ledger rows missing after a crash

Cause. Durable flush wasn't registered with waitUntil on a serverless function; teardown lost the queued writes.

Fix. Register the durable-flush pipeline at handler entry:

import { setWaitUntilImpl } from "@pleach/core/eventLog";
setWaitUntilImpl(ctx.waitUntil.bind(ctx));

See Event log.

Troubleshooting

Error codes

Audit gates

DevTools

Deployment

On this page