@pleach/gateway

The gate in the hedge — every call walks through here, in family, on budget, attributable to one tenant. @pleach/gateway is the multi-tenant routing SKU layered over the @pleach/core model-family substrate. One client per tenant accepts a (family, callClass, model, prompt) call, picks a transport, walks the in-family cascade on transient failure, emits one cost event per successful call, and never silently widens across families.

The substrate's in-family cascade behavior — what locks at session start and how pickNextInFamily walks the column — lives in Family-locked routing. The model resolution matrix defines the (family × callClass) cells the cascade walks. This page is the multi-tenant routing client layered on top: per-tenant scoping, operator allowlist, BYOK fingerprinting, and the per-call cost event.

The package is intentionally thin. The routing math, family matrix, walk order, and BYOK transport resolution all live in @pleach/core's model-family module; gateway re-exports CallClass, ProviderFamily, and RoutingDecision so consumers don't need to import from both places, but it doesn't reimplement any of them.

Not a proxy

@pleach/gateway routes, refuses over-budget calls, attributes cost, and fails over — the same verbs a governed-proxy control plane (Portkey, Cloudflare AI Gateway, and the enterprise control planes above them) ships. The difference is where it sits. A proxy fences traffic with a one-URL swap, but routes 100% of your model calls through a vendor and holds the audit trail on its side. Gateway is an in-process routing client: it never opens a socket of its own, consumes a GatewayTransport the host supplies, and runs inside your process against transports you own. The out-of-family or over-budget call is denied before it touches the wire, and the cost event lands in a sink you own — no vendor in the data path. That's altitude, not timing: both enforce at runtime; only the substrate governs at execution-graph altitude instead of the HTTP boundary. See the comparison for the full axis.

Install

npm install @pleach/gateway

pnpm add @pleach/gateway

yarn add @pleach/gateway

bun add @pleach/gateway

import {
  GatewayClient,
  asTenantId,
  fingerprintByokKey,
  GatewayFamilyDeniedError,
  GatewayFamilyExhaustedError,
  GatewayTransportMissingError,
  type CostEvent,
  type GatewayResponse,
  type GatewayTransport,
} from "@pleach/gateway";

The package re-exports CallClass, ProviderFamily, and RoutingDecision from @pleach/core as well, so a consumer that talks only to the gateway doesn't need a second import line for the substrate types.

GatewayClient

One client per tenant is the canonical pattern. tenantId is required at construction; the constructor throws if it's missing or empty. Every cost event the client emits carries that tenant id, and the operator-supplied allowedFamilies allowlist is scoped to the client instance.

const gateway = new GatewayClient({
  tenantId: asTenantId("acme-corp"),
  transports: new Map([
    ["anthropic", anthropicTransport],
    ["openai",    openaiTransport],
  ]),
  allowedFamilies: new Set(["anthropic", "openai"]),
  costEventEmitter: {
    emit(event) {
      eventLog.append(event);
    },
  },
});

const response = await gateway.route({
  family:    "anthropic",
  callClass: "synthesize",
  model:     "claude-sonnet-4-6",
  prompt:    "Summarize the meeting notes …",
});

asTenantId(raw) is the helper that turns a plain string into the branded TenantId the constructor expects. It validates the input is non-empty so the silent-isolation case (an unset env var interpolated as "") becomes a load-bearing throw at init rather than a months-later billing incident.

Constructor options

route()

Returns a GatewayResponse on success. The response carries the final modelInvoked, the usage token counts, a cost block with the family and rate snapshot, the full routingDecision, the cascadeWalks history (empty when the first rung succeeded), and a familyExhausted boolean.

close()

Marks the client closed; subsequent route() calls reject. Phase A has no underlying resources to release — transports are caller-supplied — so close() is purely a state flag today.

The transport seam

GatewayClient doesn't ship concrete HTTP clients. It consumes a narrow GatewayTransport interface and a Map<family, transport> the host supplies at construction.

interface GatewayTransport {
  readonly kind:
    | "openrouter"
    | "native-anthropic"
    | "native-openai"
    | "native-google"
    | "bedrock"
    | "azure"
    | "vertex";
  invoke(req: GatewayTransportRequest): Promise<GatewayTransportResponse>;
}

The contract is intentionally minimal — the host wraps its existing provider client (OpenRouter HTTP, the Anthropic SDK, an in-house Bedrock adapter) in a GatewayTransport and hands the map to the constructor. Gateway never opens a socket of its own.

A transport map keyed by family is the Phase A shape. Region-aware routing ships separately through @pleach/gateway/region (see Region routing); (family, callClass) transport arbitration remains future work.

The chat-completion transport seam

GatewayChatTransport is a distinct seam from the Phase-A GatewayTransport above. Phase-A transports are callClass-shaped and cost-less, serving the low-level GatewayClient.route() path. GatewayChatTransport is the runtime-native shape GatewayRuntime.routeChatCompletion() dispatches through: messages in, content + usage out.

interface GatewayChatTransport {
  route(request: GatewayChatTransportRequest): Promise<GatewayChatTransportResult>;
}

The request carries the already-resolved route — provider, family, model, messages, tenantId, an optional region, and an optional abortSignal. The result carries content, a usage pair (promptTokens + completionTokens), an optional modelInvoked (the concrete id the transport actually hit), and an optional costUsd hint.

costUsd is a hint, not a requirement. The gateway is the single source of truth for cost attribution, so a failover counts only the successful transport's cost, once, at the gateway. A transport that already computed a cost may surface it; absent that, the runtime resolves cost from usage against the gateway-owned rate card.

createOpenAiCompatibleTransport is the first concrete GatewayChatTransport. Point its baseURL at any endpoint speaking the OpenAI Chat Completions dialect — OpenAI, OpenRouter, Together, Groq, a self-hosted vLLM — and supply an auth header callback. The OpenAI request encoding and response decoding are reused from the shape adapter, not reimplemented; the factory only maps the two contracts and threads the per-call abortSignal. It deliberately does not expose requestEncoder/responseDecoder overrides — its name sells the OpenAI dialect, so a non-OpenAI wire format uses the lower-level HTTP transport directly instead.

The cloud-IAM transports — Bedrock, Azure OpenAI, Vertex — implement the same GatewayChatTransport contract over an IAM-credential adapter in later cuts. See @pleach/transport-bedrock, @pleach/transport-azure-openai, and @pleach/transport-vertex for the per-cloud placement.

Per-call cost event

Every successful route() emits one event to the configured sink. The cadence is per-call, not batched: accuracy and reproducibility for compliance attestation outweigh the batching savings. A sink that wants its own batching can implement it internally; the gateway's contract with consumers is one event per call.

interface CostEvent {
  readonly type:             "domain.gateway.cost.recorded";
  readonly tenantId:         TenantId;
  readonly family:           ProviderFamily;
  readonly callClass:        CallClass;
  readonly modelInvoked:     string;
  readonly costUsd:          number;
  readonly promptTokens:     number;
  readonly completionTokens: number;
  readonly byokActive:       boolean;
  readonly byokKeyHash?:     string;
  readonly routingDecision:  RoutingDecision;
  readonly timestamp:        string;
}

The full routingDecision shape ships inside the payload so a downstream consumer — typically @pleach/compliance's attestation runtime — has the provenance to attest the call without a round-trip to the audit ledger. raw_provider_cost_usd and markup_pct live inside the decision separately from the marked costUsd, so a consumer that needs the pre-markup figure has it directly.

response.cost.usd is raw_provider_cost × (1 + markupPct). The 20% flat markup is sourced from MODEL_FAMILY_MATRIX; the gateway doesn't compute it independently.

BYOK is header-only

A BYOK key passed to the constructor or to an individual route() call is forwarded to the transport via the request payload and never written anywhere else. The gateway computes a 16-character sha256 fingerprint of the key for three purposes:

• The cache fingerprint key, so the same BYOK gets cache reuse without ever comparing raw keys.
• The byok_key_id field on the routing decision for audit attribution.
• The byokKeyHash field on the emitted cost event.

import { fingerprintByokKey, isSameByokKey } from "@pleach/gateway";

fingerprintByokKey("sk-ant-api03-abc…");
// → "8f3a2b1c4d5e6f78"

isSameByokKey(a, b);
// → true iff fingerprintByokKey(a) === fingerprintByokKey(b)

The 16-character hex slice is opaque — enough entropy to de-dup within a tenant, not enough to reverse. brandedFingerprintByokKey returns the same string under a branded ByokKeyFingerprint type for code that wants to distinguish gateway hashes from arbitrary hex strings at the type layer.

The caller is responsible for never logging the raw key. Once it's fingerprinted, gateway forgets the plain value.

Family-strict cascade on 503

On a transient transport failure (status 502 / 503 / 504, rate-limit 429, timeout, abort), GatewayClient walks the in-family rung ladder via pickNextInFamily(family, currentModel, attempted) from the substrate. The walk follows the same order the runtime uses — synthesize → reasoning → utility → converse — skipping any rung already tried this call. Each attempt produces a CascadeWalk entry on the response.

When every rung fails, the gateway throws GatewayFamilyExhaustedError with the attempted-model list attached. It does not widen across families. That's the gateway-side mirror of the runtime-side Family-Strict Cascade Pivot: cross-family is a consumer decision, made explicitly, not a silent behavior of the routing layer.

try {
  const response = await gateway.route({ family: "anthropic", … });
} catch (err) {
  if (err instanceof GatewayFamilyExhaustedError) {
    // err.family            — "anthropic"
    // err.attemptedModels   — ["claude-sonnet-4-6", "claude-haiku-4-5", …]
    // Consumer chooses: widen to "openai", surface to user, or fail.
  }
}

The graceful familyExhausted: true flag on the response is reserved for a future shape where a transport reports a degraded-but-non-failed rung; Phase A throws on hard exhaustion to keep the error contract honest.

The cascade is bounded at 16 steps so a pathological transport can't infinite-loop the gateway.

allowedFamilies governance

allowedFamilies is the operator-facing governance hook. When set, route() checks the requested family against the allowlist before any transport invocation and throws GatewayFamilyDeniedError on denial. The throw is synchronous-within-the-promise — denied calls never touch the wire, never count against rate budgets, and never emit a cost event.

const gateway = new GatewayClient({
  tenantId:        asTenantId("acme-corp"),
  transports:      transportMap,
  allowedFamilies: new Set(["anthropic", "openai"]),
});

await gateway.route({ family: "deepseek", … });
// → throws GatewayFamilyDeniedError
//   err.family            — "deepseek"
//   err.allowedFamilies   — Set { "anthropic", "openai" }

Use it to enforce per-tenant family policy ("acme-corp is on anthropic-only"), to block a family during a rollback, or to gate a preview family behind an explicit operator opt-in. The mechanism is structural — a tenant on the allowlist cannot reach a family that's not on it, regardless of what the model id string says.

Errors

Three error classes, three failure modes. All thrown via promise rejection from route(); the constructor's TypeError for invalid tenantId is the fourth.

GatewayFamilyDeniedError and GatewayTransportMissingError are configuration failures and never reach the wire. GatewayFamilyExhaustedError is a cascade outcome and carries the attempted-model list so the consumer can decide what (if anything) to widen to.

Tenant scoping

tenantId is required at construction and stamped on every emitted CostEvent. The gateway's tenant-scope contract aligns with the substrate's tenant facet and the multi-tenant deployment pattern — same field, same partition key. The gateway's cost numbers come from the transport's reported usage, not from reading the core audit ledger; the gateway never queries the ledger. A consumer that wants both surfaces partitioned the same way (GROUP BY tenantId) gets that because both stamp the same opaque tenantId, not because the gateway joins through the ledger.

The tenantId field is opaque the same way the audit row's is. A GatewayClient per end customer is the SaaS pattern; a GatewayClient per cost center is the internal-Enterprise pattern (one Anthropic Workspace or OpenAI Project sits behind the gateway, and the cost event partitions spend across teams). The cost-event sink reads the same GROUP BY tenantId either way.

One GatewayClient instance per tenant is the canonical usage pattern. Per-tenant API keys (rotating credentials scoped to the tenant, separate from BYOK) are deferred to a later phase.

Phase A status

What ships in the Phase A cut:

• GatewayClient with route() and close().
• Per-call domain.gateway.cost.recorded event emission with full RoutingDecision payload.
• BYOK fingerprinting (fingerprintByokKey, isSameByokKey, brandedFingerprintByokKey).
• Family-strict cascade-on-503 via pickNextInFamily.
• Operator allowedFamilies allowlist governance.
• Three typed error classes.
• Re-exports of CallClass, ProviderFamily, and RoutingDecision from @pleach/core.
• createGatewayRuntime() contract + routeChatCompletion() body. routeChatCompletion() validates the input shape (tenantId / family / messages) and the allowedProviders governance hook, resolves the family default provider, and returns a RouteChatCompletionOutput. When a transport is wired for the resolved provider (via the GatewayRuntimeConfig.transports map — e.g. createOpenAiCompatibleTransport), it invokes that transport and returns real content + usage (costUsd stays 0 until the rate-card slice lands). When no transport is wired it returns a lossless stub-shape response so callers can build against the contract before wiring a wire transport. routeEmbedding(), getProviderHealth(), and getTrafficStats() continue to throw the slice-1 sentinel.

What's not in Phase A — lands in subsequent passes:

• The full concrete-transport set. createOpenAiCompatibleTransport ships today as the first concrete GatewayChatTransport (OpenAI-dialect wire shape, exported from @pleach/gateway/transport) — wire it into GatewayRuntimeConfig.transports and routeChatCompletion() dispatches through it. The bundled native-Anthropic / native-OpenAI / native-Google adapters and the Bedrock / Azure / Vertex concrete transports with region-pin routing arrive in subsequent Phase B slices.
• OTEL llm.invocation span emission. The C7 telemetry rung that records every gateway call as an OTEL span lands once runtime.otel is consumable from @pleach/core's exported surface. Until then, callers thread their own span context if they need it.
• Native-vs-openrouter transport arbitration. Choosing between a native provider transport and an OpenRouter transport per (family, callClass) is v1.x work. Phase A's cascade is family-strict and walk-order-strict; there's no policy knob between them. (Note: cross-provider failover and region routing primitives now ship as experimental subpaths — see Failover and Region routing — but they are opt-in wrappers, not part of the default route() path.)

Tenant resolution

The base @pleach/gateway/auth subpath ships the TenantResolver contract plus createInMemoryTenantResolver for tests. Four production adapters resolve a tenantId from a request, each on its own subpath so the base stays dependency-free:

import { createJwtTenantResolver } from "@pleach/gateway/auth/adapters/jwt"

const resolver = createJwtTenantResolver({
  jwksUri: "https://issuer.example.com/.well-known/jwks.json",
  tenantClaim: "org_id",
})
const { tenantId } = await resolver.resolve({ headers })

Failover

@pleach/gateway/failover wraps an operator-supplied provider chain in a retry loop. It's vendor-neutral — you supply the chain and the transient-error classifier is purely structural (no provider SDKs).

• createFailoverPolicy({ chain, maxAttempts?, backoff? }) — a chain cursor. maxAttempts defaults to and clamps at chain.length; backoff defaults to defaultLinearBackoff.
• createFailoverMiddleware({ policy, onExhausted?, sleep?, isTransient? }) — wraps a call in runWithFailover(fn). Throws FailoverExhaustedError and emits a structural FailoverExhausted event when the chain runs out.

import {
  createFailoverPolicy,
  createFailoverMiddleware,
} from "@pleach/gateway/failover"

const mw = createFailoverMiddleware({
  policy: createFailoverPolicy({ chain: ["primary", "secondary"] }),
})
const res = await mw.runWithFailover(() => callProvider())

Region routing

@pleach/gateway/region augments a RoutingDecision with a region and enforces region allowlists:

• createRegionRouter({ resolver }) — augments a route with its resolved region.
• createStaticRegionResolver({ region }) — reference single-region resolver.
• createRegionAwareCascade(...) — filters cascade candidates to a region allowlist.
• createRegionDrPolicy({ primary, secondaries, mode? }) — primary→secondary DR ordering; mode defaults to "active-passive". Rejects with RegionNotPermittedError when no permitted region remains.

Identity providers

@pleach/gateway/identity resolves short-lived federated credentials per tenant. Multi-cloud deployments route LLM traffic without copying long-lived API keys into the gateway process. The base subpath ships the contract plus a credential cache:

• IdentityProvider — the resolution seam. Resolves credentials for an IdentityContext.
• IdentityContext — carries tenantId, targetFamily, and an optional region.
• ResolvedCredentials — kind: "static" (a stored key) or kind: "federated" (a bearer token with expiresAt).
• createTokenCache(...) — LRU + TTL cache that evicts on expiresAt before the upstream endpoint rejects a stale token.

Reference adapters resolve against a vendor client you inject — the adapter never imports the SDK:

Each factory consumes a duck-typed client matching the vendor SDK shape (STSClient.send, TokenCredential.getToken, GoogleAuth.getAccessToken). Federated resolution suits regulated, multi-cloud tenants — see the government use case.

Related SKUs

• @pleach/core — the family-locked matrix the gateway routes against. pickNextInFamily, MODEL_FAMILY_MATRIX, and the CallClass / ProviderFamily / RoutingDecision types all live there; gateway re-exports them for convenience.
• @pleach/compliance — downstream consumer of domain.gateway.cost.recorded events. Reads the full RoutingDecision payload for attestation provenance without a round-trip to the audit ledger.
• @pleach/observe — wires the gateway's cost events into OpenTelemetry / Datadog / Honeycomb spans.

For the full SKU map see Which SKU do I need?.

Where to go next