pleach
Cookbook

Swarm agent

An agent that spawns subagents at scale — deep sequential loops, parallel fan-out, supervisor/worker topologies, recursive delegation — with cost ceilings and a session-tree audit walk.

A swarm agent is the shape that pushes the runtime's recursive contract: one user turn fans out into dozens or hundreds of subagents, each of which may spawn its own. Kimi K2-style deep sequential loops, OpenAI Deep Research-style parallel fan-out, AutoGen / LangGraph supervisor topologies, Claude Code-style recursive delegation — different topologies, same buyer cohort, same runtime stress.

The load-bearing pain is the runaway-loop case. A buggy fan-out or a stuck sequential loop can spend a five-figure budget in one user turn. This page walks the cost ceiling at the root-turn boundary, the session-tree audit walk that attributes every descendant call back to the originating turn, and the concurrency primitives a swarm host reaches for.

Related shapes. Research agent for the canonical single-anchor / bounded-subagent pattern — the swarm shape inherits its primitives and pushes them further. Coding agent if the subagents run code in sandboxes. Multi-tenant SaaS agent if one runtime serves many customers, each driving their own swarm.

What you're building

A turn loop whose recursion is bounded by construction, not by hope:

  • Every subagent — including subagents-of-subagents — carries the originating user turn's turnId and its own subagentDepth. The whole tree is one query against the ledger.
  • A per-root-turn cost ceiling aborts the turn when cumulative spend crosses the line. The ledger records the cutoff.
  • A fan-out cap limits how many subagents one node can spawn at once. A recursion-depth cap limits how deep the tree can go.
  • Concurrency primitives — race, waitAll, waitN, bounded, backpressure — handle the structural cases a swarm host actually hits.

Cost ceiling at the root-turn boundary

The cost ceiling is the load-bearing protection for a swarm. Today @pleach/core ships the structural caps — maxDepth, maxConcurrent, maxPerTurn, timeoutMs — as hard ceilings the substrate enforces independently of consumer config. A per-root-turn USD ceiling now ships in @pleach/gateway/swarm as an inline pre-transport guard — see Gateway swarm governance below. Hosts that don't route through the gateway enforce the dollar cap out-of-band against the audit ledger.

import {
  SessionRuntime,
  SUBAGENT_LIMITS,
  AiSdkProvider,
  createSupabaseAdapter,
} from "@pleach/core";

const runtime = new SessionRuntime({
  storage: createSupabaseAdapter({ client: supabase }),
  userId:  "user_123",
  tenantId: "tenant_abc",

  enableSubagentConcurrency: true,
  // SUBAGENT_LIMITS.maxConcurrent is the substrate-enforced
  // ceiling (3); consumer values above the ceiling clamp down.
  maxConcurrentSubagents: SUBAGENT_LIMITS.maxConcurrent,

  orchestratorConfig: {
    provider: anchorProvider,
  },
});

// Out-of-band USD ceiling, polled by the host against the
// audit ledger. `@pleach/compliance/education` ships a
// per-session helper for the single-process case; for swarms
// that span multiple processes, back it with a tenant-scoped
// store and check on each turn boundary.
import { createCostCapPolicy } from "@pleach/compliance/education";

const costCap = createCostCapPolicy({
  ceilingUsd: 25,
  onBreach:   () => runtime.abort(),
});

SUBAGENT_LIMITS carries the substrate-side hard caps:

CapValueEffect when breached
maxDepth3Spawn rejected — fourth nested runtime never starts
maxConcurrent3New spawns queue until a slot frees
maxPerTurn5Sixth spawn in a turn is rejected
timeoutMs120_000Subagent emits subagent.failed with terminalStatus: "timeout"

The recursion is bounded structurally, not by per-subagent prompts. A subagent that doesn't know about the cap can't escape it.

Gateway swarm governance

@pleach/gateway/swarm ships the inline cost governance the out-of-band poll above approximates. The middleware guards each routing call and throws before transport when the root turn would cross its ceiling:

  • createRootTurnCostAggregator(...) — accumulates CostEvents by rootTurnId. In-memory by default; wire a shared RootTurnCostStore (Redis / Upstash / DynamoDB) so sub-agents on different processes accumulate to one counter.
  • createCostCeilingMiddleware({ aggregator, maxCostUsdPerRootTurn }) — wraps a routing call via guard({ rootTurnId, invoke }). A call that would push accumulated cost over the ceiling throws CostCeilingExceededError before the transport runs — the breaching call never touches the wire. Optional warnCostUsdPerRootTurn / warnThreshold fire onWarn at a soft threshold.
  • createCostRateLimitMiddleware({ windowMs?, maxRequests? }) — a per-rootTurnId request throttle. process() returns a { retryAfterMs } deferral once the window cap is met and fires onDefer at most once per window.
  • createFailoverCoordinator(...) — coordinates sibling sub-agents on a failover, with mode: "best-of-N" | "all-or-nothing".
  • classifySwarmTopology(events) — classifies a flat spawn-event list into tree | dag | ring, plus depth, fanOut, and subAgentCount.

Two structural events ride the callbacks for operator telemetry: CostBudgetWarned (soft threshold) and CostRateLimitDeferred (transient throttle). Both are data-shaped for the event log and OTEL attributes.

import {
  createRootTurnCostAggregator,
  createCostCeilingMiddleware,
  CostCeilingExceededError,
} from "@pleach/gateway/swarm";

const aggregator = createRootTurnCostAggregator();
const ceiling = createCostCeilingMiddleware({
  aggregator,
  maxCostUsdPerRootTurn: 25,
  warnThreshold: 0.8,
  onWarn: (e) => notifyOperator(e),
});

try {
  const answer = await ceiling.guard({
    rootTurnId,
    // invoke returns the call result plus its CostEvent so the
    // aggregator can roll spend up by rootTurnId.
    invoke: async () => {
      const { result, costEvent } = await runRoutedCall(decision);
      return { result, costEvent };
    },
  });
} catch (err) {
  if (err instanceof CostCeilingExceededError) {
    // root turn halted before transport — err carries accumulated + ceiling
  }
}

What today ships

Today, @pleach/core ships:

  • Recursive SessionRuntime sessions — every subagent is itself a SessionRuntime, with the same primitives the root has.
  • AuditableCall typed audit-ledger rows — every spawn writes a row that ties the child session back to the parent turn.
  • parent_turn_id and subagent_depth columns on the audit ledger — the tree walk is one recursive CTE.
  • SUBAGENT_LIMITS for depth, parallel fanout, and total subagent count.

For the canonical anchor → bounded subagent fan-out pattern, see Research agent.

Roadmap

The deeper swarm surface lands incrementally:

  • Per-root-turn cost ceiling. Shipped in @pleach/gateway/swarm as an inline pre-transport guard — see Gateway swarm governance. A first-class maxCostUsdPerTurn limit inside @pleach/core — enforced without routing through the gateway — is still on the roadmap. Until then, gateway-less hosts poll harness_auditable_calls filtered by parent_turn_id and call runtime.abort() when spend crosses the budget; @pleach/compliance/education ships a createCostCapPolicy({ ceilingUsd, onBreach }) helper for the per-session shape (single-process / in-memory by default; swap in a tenant-DB backing store for multi-process).
  • Concurrency primitives. A @pleach/core/spawn subpath with race, waitAll, waitN, bounded, and backpressure primitives that absorb the topology a swarm host already wrote in user space. Until it lands, hosts compose the same shapes with Promise.all, Promise.race, and a host-side semaphore.
  • Session-tree audit walker. Shipped as walkSessionTree in @pleach/compliance/swarm — it walks the tree the CTE above describes and verifies each node's hash chain. See Verifying the tree with walkSessionTree.
  • Deterministic spawn-order replay. A spawnIndex on SpawnTreeState that normalizes parallel-spawn ordering for byte-identical replay across the tree.
  • Per-subagent cost ceilings. A nested cap that lets one branch of the tree have its own budget without blocking the rest. Lands after the root-turn ceiling is in soak.

No dates. Track the upstream package READMEs and CHANGELOG for landing notices.

The session-tree audit walk

Every subagent's audit rows carry parent_turn_id and subagent_depth. The whole tree — sequential loops, parallel fan-out, recursive delegation — reads as one recursive CTE.

with recursive tree as (
  select turn_id, parent_turn_id, subagent_depth,
         tool_name, model_id, input_tokens, output_tokens
  from harness_auditable_calls
  where turn_id = $1
  union all
  select c.turn_id, c.parent_turn_id, c.subagent_depth,
         c.tool_name, c.model_id, c.input_tokens, c.output_tokens
  from harness_auditable_calls c
  join tree t on c.parent_turn_id = t.turn_id
)
select
  subagent_depth,
  count(*)                                  as calls,
  sum(input_tokens + output_tokens)         as tokens
from tree
group by subagent_depth
order by subagent_depth;

The query reads the whole investigation top-down: depth 0 is the anchor turn; depth N is the deepest spawn. A regulator asking "what did the agent do for user turn T" gets the full tree in one read.

Verifying the tree with walkSessionTree

The SQL above rolls the tree up for review. @pleach/compliance/swarm walks the same tree and verifies each node's hash chain in one pass:

  • walkSessionTree({ rootSessionId, eventStore, maxDepth? }) — BFS over the spawn tree from the root session. maxDepth defaults to 64. Returns a TreeWalkResult.
  • EventStoreLike — the duck-typed store the walker reads through. One method: listEvents(sessionId, filter?), returning rows in sequence_number ascending order.
  • TreeWalkResultrootSessionId, treeValid (the conjunction of every node's chainValid), nodes, totalRows, totalFailures.
  • TreeWalkNode — per session: sessionId, depth (0 at root), chainValid, rowCount, and failures (typed ChainVerificationErrors).

A per-node chain divergence surfaces via nodes[i].failures and treeValid: false — the walker only throws on a store-level error. A malformed tree that links back to an ancestor can't loop; visited session IDs are tracked in a set.

import { walkSessionTree } from "@pleach/compliance/swarm";

const result = await walkSessionTree({ rootSessionId, eventStore });
if (!result.treeValid) {
  // result.nodes[i].failures carries the per-node ChainVerificationErrors
}

Topology-neutral primitives

The runtime doesn't have a topology opinion. The same primitives — recursive sessions, the ledger tree, the cost ceiling, the spawn record — absorb four different shapes:

TopologyWhat the host writesWhat the runtime stamps
Deep sequential loopa long tool-loop in one subagentsubagent_depth = 1, many rows
Parallel fan-outN subagents spawned at the anchorsubagent_depth = 1, N siblings
Supervisor / workersupervisor spawns workers per tasksubagent_depth = 1 or 2, tree
Recursive delegationsubagent spawns its own subagentsubagent_depth = N, deep tree

A swarm host picks the topology; the runtime makes the audit walk identical across all four.

Where to go next

On this page