Gateway · Rate limiting

The gateway's rate-limit substrate is the seam between a host's shared quota store (Redis, Upstash, DynamoDB, Postgres) and the per-tenant / per-user / per-route throttling decision. Today the package ships a contract + one in-process reference adapter; production adapters follow on additive subpaths.

RateLimiter contract

interface RateLimitCheckOptions {
  readonly windowMs?: number
  readonly maxRequests?: number
}

interface RateLimitDecision {
  readonly allowed: boolean
  readonly remaining: number
  readonly resetAt: number       // epoch ms; window reset
  readonly retryAfterMs?: number // populated only when !allowed
}

interface RateLimiter {
  check(key: string, opts?: RateLimitCheckOptions): Promise<RateLimitDecision>
}

The contract is intentionally minimal: given a key plus optional per-call overrides, return a decision indicating whether the call is allowed, the remaining quota in the current window, and the reset time.

The decision shape mirrors what production adapters (Upstash, AWS Throttle Tokens) emit. Callers render HTTP 429 responses with accurate Retry-After + X-RateLimit-* headers without per-adapter glue.

Implementation requirements

• check() MUST be atomic — the count increment + comparison against the ceiling MUST happen as a single observable operation. The in-process reference adapter achieves this via the single-threaded JavaScript event loop; future Redis / Upstash adapters use Lua scripts or INCR+EXPIRE pipelines.
• check() MUST NOT throw under normal load — return a conservative allowed: false if the underlying substrate misbehaves and let the caller decide whether to fail open or closed.
• Distinct key values MUST be independent — a denied tenant does not affect a peer tenant's quota.

In-process reference adapter

import { createInMemoryRateLimiter } from "@pleach/gateway/rateLimit"

const limiter = createInMemoryRateLimiter({
  defaultWindowMs: 60_000,
  defaultMaxRequests: 100,
})

const decision = await limiter.check(`tenant:${tenantId}`)
if (!decision.allowed) {
  return new Response("Too Many Requests", {
    status: 429,
    headers: {
      "Retry-After": String(Math.ceil((decision.retryAfterMs ?? 0) / 1000)),
      "X-RateLimit-Remaining": "0",
      "X-RateLimit-Reset": String(Math.ceil(decision.resetAt / 1000)),
    },
  })
}

Configuration

interface InMemoryRateLimiterConfig {
  readonly defaultWindowMs?: number      // default 60_000
  readonly defaultMaxRequests?: number   // default 60
}

Per-call overrides are passed to check():

// Tighter limit for an expensive route, same limiter instance.
await limiter.check(`tenant:${tenantId}:expensive-llm`, {
  windowMs: 60_000,
  maxRequests: 5,
})

What it is

Fixed-window counter stored in an in-process Map. Simpler than a sliding-window log and sufficient for local dev, single-node deployments, and unit tests.

What it is NOT

Not distributed-safe. Counters live in the calling process's heap. Multiple Node.js instances behind a load balancer will each track an independent count, so a request might pass the limiter on instance A while instance B is at the ceiling. For multi-node production deployments use a shared substrate — see the roadmap below.

Memory bound is lazy. Idle keys are reaped only when a check() call observes that the stored window expired. A long tail of one-shot keys can accumulate; hosts that route via short-lived per-user keys should prefer a production adapter with explicit eviction policy.

Roadmap

Production adapters ship as additive subpaths so the base package stays dependency-free. None of these are in the current 0.x cut — they are planned for v1.x:

• @pleach/gateway/rateLimit/redis — ioredis / redis v4+. Atomic INCR+EXPIRE pipeline; sliding-window via Lua script.
• @pleach/gateway/rateLimit/upstash — @upstash/ratelimit. HTTP REST API, fits Workers / Edge runtime.
• @pleach/gateway/rateLimit/dynamodb — AWS SDK v3. Conditional writes + TTL-based window reset.

Per-tenant-call-class policy (different windows for utility vs reasoning vs synthesize calls inside one tenant) is also a v1.x item. Today, hosts that need this layer pass distinct keys at the call site:

// Workaround pre-v1.x: encode the call-class in the key.
await limiter.check(`tenant:${tenantId}:callClass:${callClass}`, {
  maxRequests: callClass === "synthesize" ? 10 : 100,
})

Cited source

• packages/gateway/src/rateLimit/types.ts — RateLimiter contract.
• packages/gateway/src/rateLimit/inMemoryLimiter.ts — fixed-window in-process adapter.

Where to go next