Constraint-optimized orchestration: rethinking how agent fleets get scheduled

I built Constellation COO because I kept watching the same failure pattern in agent frameworks — mine included. It never shows up as a crash. It shows up as a system that technically works, is technically "up," and is quietly getting worse at its job in a way no dashboard is built to catch. This is the version of that story worth generalizing, for anyone building or operating a fleet of LLM agents.

The two schedulers everyone starts with

If you've built more than one agent and needed to route work between them, you've probably reached for one of two defaults:

• Round-robin — send the next task to the next agent in line, cycling evenly. Simple, fair, and completely blind to the fact that "fair" and "efficient" are different goals.
• Greedy-cheapest — send every task to whichever agent (or model) is cheapest per call. Great for the invoice, terrible for latency, because cheap agents get hammered while expensive ones idle.

Both are schedulers with one dimension. Round-robin optimizes for even distribution. Greedy-cheapest optimizes for cost. Neither one knows about the other's constraint, and neither knows about latency at all. That's fine when load is low. It stops being fine the moment you have more concurrent tasks than your fastest, cheapest agents can absorb — which, if your product is working, is exactly when it starts mattering most.

Why the failure is invisible

The dangerous part isn't that a single-dimension scheduler fails badly. It's that it fails quietly. Tasks still complete. Nothing throws an exception. What actually happens is SLA erosion: p95 latency creeps from 4 seconds to 9, then to 14, over the course of an afternoon, and unless you're explicitly plotting that curve, the first signal you get is a support ticket or a churn event three weeks later.

A one-dimensional scheduler has no way to notice this, because it isn't tracking the thing that's degrading. Round-robin doesn't know an agent is slow — it just knows whose "turn" it is. Greedy-cheapest doesn't know an agent is overloaded — it just knows what the last invoice said.

Scheduling as constrained optimization

The reframe that actually fixed this for me: stop treating routing as "pick the next agent" and start treating it as a constrained optimization problem, evaluated fresh on every task. Every candidate agent gets scored on a small set of fitness dimensions, and the highest score wins — or, if nothing clears a floor, the system spawns capacity or defers gracefully instead of forcing a bad assignment.

In Constellation, that's four dimensions, blended into one score:

fitness(agent, task) =
    w_cost    * cost_headroom(agent, task)
  + w_latency * latency_margin(agent, task)
  + w_load    * (1 - saturation_penalty(agent))
  + w_affinity* warm_context_bonus(agent, task)

None of the individual terms is exotic. cost_headroom is just remaining budget minus expected task cost. latency_margin is the agent's live p95 against the task's SLA window. saturation_penalty is non-linear — it stays near zero until an agent approaches its concurrency ceiling, then rises sharply, because the cost of overloading an agent isn't linear either. The value isn't in any single term. It's in scoring all of them together, per task, in real time, instead of picking one axis and hoping the others stay out of the way.

The phase nobody designs for: retiring an agent

Here's the part that surprised me most when I benchmarked this against a round-robin baseline: the biggest SLA win didn't come from routing better. It came from retiring degraded agents faster — and doing it without flapping.

Most orchestration writing focuses entirely on the assignment decision: given a new task, where does it go? That's necessary but not sufficient. A fleet also needs a continuous rebalancing loop that asks, independent of any single task: is this agent still fit to be in the pool? An agent whose live latency has drifted up, or whose error rate has ticked up, needs to be pulled from rotation — but pulled based on sustained degradation, not a single bad sample, or you end up thrashing agents in and out of service on noise.

What it actually moved

Benchmarked against a round-robin baseline under identical synthetic load — same task mix, same pool size, same budget ceiling, repeated trials with bootstrap confidence intervals to separate signal from noise — the fitness-scored scheduler held 16.5% better SLA compliance and cut load variance across the fleet by 41%. The full benchmark methodology and results are in the Constellation COO case study.

The gain wasn't uniform, though — and this is the generalizable part. The advantage over a naive scheduler scaled with how heterogeneous the agent pool was. If every agent has near-identical cost and latency profiles, round-robin and fitness-scoring converge — there's nothing for the extra dimensions to exploit. The more your fleet mixes fast/expensive and slow/cheap agents, the more a constraint-aware scheduler earns its complexity.

If you're building this yourself

• Instrument before you optimize. You can't score fitness dimensions you're not measuring live. Latency, cost headroom, and saturation all need to be real-time signals, not end-of-day aggregates.
• Don't skip the retire/rebalance loop. Routing decisions get all the attention; the recurring health check on agents already in the pool is where a lot of the actual SLA protection lives.
• Add hysteresis before you add more dimensions. A five-dimension fitness function that flaps agents in and out of service on noisy samples is worse than a two-dimension one that's stable.
• Check whether your pool is even heterogeneous enough to need this. If every agent is functionally identical, a simpler scheduler may be the right call — constraint-optimization earns its keep on mixed fleets, not uniform ones.

This is also, not incidentally, why the approach ended up in a patent filing rather than just a blog post — the specifics are covered in why I patent my own algorithms as a solo engineer.