Constellation COO: scheduling AI agent fleets like a constrained optimization problem

Constellation COO (Constraint-Optimized Orchestration) is the flagship asset in the Full Stack Assets catalog: an algorithm and reference implementation for managing fleets of LLM agents under hard operating constraints — cost per task, latency SLA, and concurrency budget — instead of the ad-hoc heuristics most agent frameworks ship with by default.

The problem

Most multi-agent systems route work with something close to round-robin or greedy-cheapest selection: send the next task to whichever agent is free, or whichever model is cheapest per call. That works until the fleet is under real load. Cheap agents get overloaded and blow their latency SLA; fast agents sit idle while their more expensive siblings drown; and nobody is tracking the aggregate budget until the invoice arrives.

The failure mode isn't a crash — it's silent SLA erosion. Tasks still complete, just slower and less predictably, and the scheduler has no signal that tells it why.

The approach

Constellation treats agent routing as a multi-dimensional constrained optimization problem rather than a queueing heuristic. Every candidate agent is scored on a fitness function that blends:

• Cost fitness — remaining budget headroom against the task's expected cost
• Latency fitness — the agent's live p95 response time against the task's SLA window
• Load fitness — current concurrency against a soft ceiling, penalized non-linearly as it approaches saturation
• Affinity fitness — whether this agent has handled similar task shapes recently (cache and context warm-start effects)

The four fitness scores are combined into a single weighted objective, and the scheduler routes to the highest-scoring available agent. When no agent clears a minimum fitness threshold, Constellation spawns a new one — or, if the fleet is at its budget ceiling, gracefully defers or degrades the task rather than force-fitting it onto an overloaded agent.

Architecture

The reference implementation is a 13-module Python engine, covered by a 39-test pytest suite that exercises the fitness function, the spawn/retire lifecycle, and constraint-boundary edge cases (what happens exactly at budget zero, exactly at the concurrency ceiling, exactly at SLA expiry).

Lifecycle management follows three phases: route (score and assign incoming tasks), rebalance (periodically re-evaluate live agents against drifting load), and retire (shut down agents whose fitness has decayed below a hysteresis floor, avoiding flapping). This mirrors how a human on-call engineer would manage a fleet — not a single decision, but a continuous loop.

Results

Constellation was benchmarked against a round-robin baseline scheduler under identical synthetic load — same task mix, same agent pool size, same budget ceiling — across repeated trials with bootstrap confidence intervals to separate real effect from noise.

The SLA gain compounds with scale: the more heterogeneous the agent pool (mixed cost/latency profiles), the larger Constellation's advantage over uniform-treatment schedulers, because there's more signal for the fitness function to exploit.

Protecting the IP

Because the fitness-scoring approach to agent-fleet scheduling was genuinely novel against the prior art I reviewed, I drafted it into a provisional patent application with seven claims — covering the multi-dimensional fitness function, the spawn/retire hysteresis mechanism, and the constraint-aware deferral behavior specifically. That process is documented in more depth in Why I Patent My Own Algorithms as a Solo Engineer.

Lessons learned

The biggest surprise was how much of the win came from the retire phase, not the routing phase. Most orchestration discussions focus on where to send the next task; in practice, a scheduler that's slow to retire a degraded agent bleeds SLA quietly for minutes before anyone notices. Hysteresis on the retire threshold — waiting for sustained low fitness, not a single bad sample — turned out to matter more than the precision of the fitness weights themselves.