Auto-scaling: agents scale with load

Auto-scaling for agent fleets means that the compute infrastructure automatically adds or removes capacity in response to agent task load, without manual intervention. When a sprint kicks off and 50 developers simultaneously dispatch agent tasks, the fleet scales out to absorb the demand. When it is 2am and load is low, the fleet scales in to reduce costs. The scaling is driven by metrics that reflect actual agent demand: task queue depth, waiting task age, devbox utilization rate, and pre-warmed pool depletion rate.

Standard Kubernetes autoscaling (the Horizontal Pod Autoscaler) is designed for services with steady load, not for bursty task workloads. The right tool for agent fleet autoscaling is KEDA (Kubernetes Event-Driven Autoscaling), which can scale based on external queue metrics. KEDA reads the depth of the agent task queue (from SQS, Redis, RabbitMQ, or another queue backend) and scales the devbox pool size proportionally. When there are 50 tasks in the queue, KEDA creates 50 devboxes. When the queue is empty, KEDA scales down to the minimum pool size.

Agent task load has distinct patterns that make the scaling parameters different from typical web services. Agent load is bursty and correlated: developers tend to dispatch tasks at the start of work sessions (morning) and before major review cycles (end of sprint). These bursts can be 5-10x the average load. Burst absorption requires either maintaining a large standing pool (expensive) or scaling out quickly when burst demand arrives. The right approach combines a small standing pool (handles baseline demand immediately) with fast scale-out (adds capacity within 60-90 seconds when the queue grows).

At L5, scaling is not just about the compute nodes - it is about the full stack: the pre-warmed container pool scales with load (more pre-warmed containers when queue depth is high), the node pool scales with the container pool (more nodes when containers need more hosts), and the task queue scales with submission rate (more queue consumers when more tasks are arriving). Each layer has its own scaling policy, and the policies are coordinated to prevent cascading failures when load spikes.

• Cost efficiency across day/night cycles - agent workloads are strongly time-of-day dependent; auto-scaling that scales in at night reduces compute costs by 60-80% during low-demand periods without manual intervention
• Burst absorption without permanent over-provisioning - without auto-scaling, fleets must be sized for peak demand; with auto-scaling, fleets can be sized for average demand and scale out for bursts, significantly reducing standing compute costs
• Developer trust requires consistent availability - developers who try to dispatch agent tasks during a peak period and encounter queuing will stop using agents; auto-scaling that keeps queue depth near zero maintains developer trust in the infrastructure
• Enables load-responsive pre-warming - the pre-warmed container pool should grow when demand is rising and shrink when demand is falling; this requires the same scaling signals that drive node scaling, coordinated across both layers
• Autonomous agent workflows depend on capacity availability - automated agent pipelines (triggered by CI, alerts, or schedules) cannot predict when compute will be available; auto-scaling ensures capacity is always available when automation triggers agent tasks

1. Instrument your task queue for scaling signals - The primary scaling signal for agent fleets is task queue depth. Ensure your task queue (SQS, Redis, RabbitMQ) emits metrics that KEDA or your autoscaler can read: queue depth, oldest waiting task age, and consumer count. These metrics drive scaling decisions.
2. Deploy KEDA on your Kubernetes cluster - KEDA is a CNCF project that extends Kubernetes with event-driven scaling. Install KEDA with helm install keda kedacore/keda. Create a ScaledObject that points to your agent task queue and defines the scaling behavior: minimum replicas, maximum replicas, and the queue depth threshold that triggers scaling.
3. Configure the Cluster Autoscaler for node-level scaling - Install the Kubernetes Cluster Autoscaler configured for your cloud provider. The Cluster Autoscaler watches for unschedulable pods (devboxes that cannot find a node to run on) and triggers node provisioning. Set scale-out parameters: node provision time target (should be under 90 seconds for pre-baked AMIs), maximum node count, and scale-in delay (wait 10 minutes after load drops before removing nodes to avoid flapping).
4. Set scaling thresholds based on load measurement - Measure your typical load patterns over two weeks. Identify the average tasks-per-developer, the peak multiplier (peak load / average load), and the duration of typical peaks. Set your KEDA scaling target (one devbox per N queue items) to keep queue wait time under 30 seconds during typical peaks.
5. Implement scale-out pre-warming - When KEDA detects queue growth and starts scaling out the devbox pool, the new devboxes need to be pre-warmed before they can accept tasks. Implement a pipeline: KEDA scales out devbox pods, the devbox initialization process pre-warms the environment, a readiness probe signals when pre-warming is complete, and the task queue dispatcher assigns tasks only to ready devboxes.
6. Test scale-out under synthetic load - Before relying on auto-scaling in production, test it under synthetic load. Submit 50 tasks simultaneously and observe: how long before new nodes are available, how long before new devboxes are ready, what is the queue wait time for the last task submitted. This test reveals the actual scale-out latency and validates that the configuration meets your SLO targets.

Scaling based on CPU/memory rather than queue depth. Standard Kubernetes HPA scales on CPU and memory utilization. Agent devboxes often have low CPU while waiting for LLM API responses. A devbox that is 5% CPU utilization but waiting on an LLM response will not trigger CPU-based scale-out even if the task queue is deep. Use KEDA with queue depth as the primary scaling signal.

Not accounting for pre-warming time in scale-out latency. A node that provisions in 60 seconds and a devbox that warms up in 60 seconds gives you a 2-minute scale-out latency. This latency is hidden when looking at node provisioning time alone. Measure the full scale-out latency from "queue depth exceeds threshold" to "task starts executing" and set your scaling thresholds to ensure queue wait time stays within your SLO.

Scale-in that destroys pre-warmed containers too aggressively. When load drops, aggressive scale-in destroys pre-warmed containers. If load then spikes again immediately (common at sprint transition times), the fleet has no pre-warmed capacity and falls back to cold-start times. Set a scale-in cooldown that keeps pre-warmed containers for at least 10-15 minutes after demand drops, and scale in gradually rather than all at once.

No cost controls on maximum scale. An auto-scaling fleet with no maximum cap can scale to 500 nodes during an unexpected load event (a CI pipeline that spawns 500 agent tasks in parallel, for example) and produce a surprise cloud bill. Always set a maximum node count and a maximum simultaneous devbox count. When the maximum is reached, task queue depth increases but costs are bounded.

Treating auto-scaling as infrastructure set-and-forget. Load patterns change as teams grow, as new agent workflows are introduced, and as sprint cadences evolve. Scaling parameters that are correct for a 10-developer team may be wrong for a 100-developer team. Review scaling parameters quarterly and after any significant change in agent usage patterns.