Agent fleet on dedicated compute

An agent fleet on dedicated compute is the infrastructure pattern where AI agent workloads run on a distinct, purpose-built compute layer that is separate from developer laptops, CI infrastructure, and application workloads. Instead of agents sharing resources with other systems, there is a compute cluster specifically sized, configured, and optimized for running agent processes. The fleet is managed as a first-class infrastructure concern with its own capacity planning, monitoring, autoscaling, and operational runbooks.

At L5, agent workloads are substantial enough to warrant dedicated compute. A team running 50-100 simultaneous agent tasks, 24 hours a day, 7 days a week, needs compute that is always available, appropriately sized, and not subject to resource contention from unrelated workloads. Sharing this with CI (which has peak-and-trough load patterns) or with application servers (which have different resource profiles) creates interference that degrades agent reliability and response time.

The dedicated compute layer is typically Kubernetes-based, because Kubernetes provides the primitives needed for agent fleet management: pod isolation, resource quotas, namespace separation, and integration with autoscaling systems. Agent devboxes run as Kubernetes pods in a dedicated namespace, with node selectors that pin them to nodes in the agent node pool. The agent node pool uses instance types optimized for agent workloads: high memory (agents are memory-intensive), fast local NVMe storage (disk I/O is a key bottleneck for concurrent agent workloads), and high-bandwidth networking (large codebases move significant data).

The dedicated fleet is also the level at which hardware specialization becomes relevant. Cursor's engineering team documented that disk I/O is the hidden bottleneck when running hundreds of agents: each agent reads large numbers of files, and concurrent file reads on a shared disk system create I/O contention that serializes what should be parallel work. Dedicated NVMe SSDs with high IOPS ratings, or a distributed filesystem optimized for concurrent reads, is the solution at this scale.

• No resource contention from unrelated workloads - CI pipelines, application servers, and agent workloads have different resource profiles; mixing them on shared compute creates unpredictable performance characteristics that are hard to diagnose and fix
• Compute can be optimized for agent-specific workloads - agent processes are memory-heavy, disk-read-heavy, and episodically network-intensive; dedicated compute uses instance types and storage configurations matched to this profile rather than to general-purpose workloads
• Capacity planning becomes straightforward - with dedicated agent compute, you can measure utilization, forecast growth, and make capacity decisions independently of other workloads; shared compute makes capacity planning for agents nearly impossible
• Fleet-level observability enables optimization - when all agent workloads run on dedicated compute, fleet-level metrics (tasks per node, disk IOPS per task, memory per task, network bandwidth per task) become meaningful for optimization; this data does not exist when agents are scattered across shared infrastructure
• Reliability becomes a compute infrastructure SLO - a dedicated fleet can have an SLO (e.g., 99.9% task start within 30 seconds, 99.5% task completion without infrastructure failure) that is distinct from application SLOs; this SLO commitment drives infrastructure investment and is auditable

1. Size the initial fleet based on observed agent load - Before deploying dedicated compute, measure current agent workload: peak simultaneous tasks, average task duration, resource consumption per task (CPU, memory, disk IOPS). Use these measurements to size the initial node pool. A starting point: N simultaneous tasks requires approximately N * (2 CPU, 4 GB RAM, 100 GB NVMe) at baseline resource limits.
2. Create a Kubernetes namespace for agent workloads - Isolate agent workloads from other Kubernetes workloads with a dedicated namespace and resource quotas. Set namespace-level limits on total CPU, memory, and storage that prevent agent workloads from consuming resources needed by application pods.
3. Deploy a dedicated node pool for agents - Create a Kubernetes node pool (in EKS, GKE, or AKS) with nodes that are tainted for agent workloads only. Use node selectors or taints/tolerations to ensure agent pods run only on agent nodes and non-agent pods do not run on agent nodes.
4. Choose storage infrastructure for disk-I/O-bound workloads - For each agent node, provision NVMe-backed instance storage (e.g., AWS i3en or i4i instances, GCP n2-highmem with local SSD) rather than network-attached storage. Or deploy a distributed filesystem (e.g., Ceph, Weka) optimized for concurrent read workloads. Measure IOPS per node and per task to confirm the storage configuration meets the demand.
5. Deploy the devbox manager on the agent cluster - The devbox lifecycle manager (pre-warmed pool, task assignment, environment destruction) should run on the agent cluster, not on a shared management cluster. This co-location minimizes the network distance between the devbox manager and the devboxes it manages.
6. Establish fleet observability - Deploy Prometheus and Grafana on the agent cluster with dashboards for fleet-level metrics: nodes available, pods running, tasks per node, CPU/memory/disk utilization per node, task queue depth, and task completion rates. These dashboards are the operational interface for the fleet.

Under-sizing the initial fleet and then scaling reactively. An under-sized fleet that queues tasks trains developers to not trust the agent infrastructure. Start with more capacity than you think you need and scale down based on observed utilization. Over-provisioning by 50% for the first month is a reasonable approach.

Using network-attached storage when local NVMe is needed. Network-attached storage (EBS, Azure Disk, GCP persistent disk) has unpredictable IOPS under concurrent load. When 50 agents on the same node all read large codebases simultaneously, NAS IOPS limits become the bottleneck. Measure actual disk IOPS under your expected concurrent load before committing to a storage type.

Not separating the agent control plane from the agent data plane. The devbox manager, pool manager, and task queue are the control plane. The devboxes themselves are the data plane. Running them on the same nodes creates resource contention: the control plane components consume CPU and memory that the data plane needs for agent tasks. Use separate node pools for control and data plane components.

Treating the agent fleet as a cost center rather than a productivity center. Dedicated compute has a real cost. If the fleet is evaluated purely as an infrastructure expense, it will be undersized and underinvested. The correct evaluation is: how many developer-hours of productivity does the fleet enable? A fleet that costs $10,000/month and enables 500 developer-hours of agent productivity per month is delivering developer time at $20/hour - significantly cheaper than human labor.

No cost attribution to teams or projects. When agent compute is shared infrastructure with no allocation, teams do not see the cost of their agent usage and have no incentive to optimize. Implement cost attribution at the team or project level using Kubernetes labels and cloud cost allocation tools. Teams that see their agent compute costs make better decisions about task efficiency and cleanup.