"Kubernetes for agents" - centralized orchestration

"Kubernetes for agents" describes the centralized orchestration infrastructure that enables large-scale agent deployment across an organization - analogous to how Kubernetes manages containerized workloads. Just as Kubernetes abstracts away the complexity of running containers at scale (scheduling, scaling, failure recovery, resource allocation, service discovery), a centralized agent orchestration layer abstracts away the complexity of running agent workloads at scale: task queuing, context management, cost routing, model selection, failure handling, and audit logging.

At L5 (Autonomous), organizations are running hundreds or thousands of agent tasks concurrently across many teams. The ad-hoc fleet management practices of L4 do not scale to this level. What is needed is infrastructure: a centralized system that receives agent tasks from teams and developers, routes them to appropriate models based on cost and capability requirements, manages the context and tool access each agent needs, monitors execution, handles failures, and returns results. This infrastructure is what Gas Town is to a post-apocalyptic economy - the distribution layer that enables everything else to run.

The practical form of this infrastructure varies. Some organizations build it on top of existing agent frameworks (LangChain, AutoGen, custom orchestration on top of Claude's API). Others adopt emerging "agentic infrastructure" platforms. The consistent elements are: a central task queue that receives agent work requests from across the organization, model routing logic that selects the right model for each task (frontier models for complex reasoning, smaller models for routine tasks, cached responses for repeated patterns), an execution environment that manages tool access and sandboxing, and observability infrastructure that provides visibility into what agents are doing and what it costs.

This is L5 infrastructure, appropriate for organizations where agents are doing a substantial fraction of the engineering work and the economics of model inference are a material budget concern. Building this infrastructure before the organization is running agents at scale is premature - the requirements won't be clear and the investment won't pay off. But organizations that try to scale to L5 agent use without it will hit coordination, cost, and governance walls.

• Model routing is the key to economics at scale - using frontier models for every agent task at L5 scale is prohibitively expensive; centralized orchestration enables intelligent routing: frontier models for complex reasoning, smaller models for mechanical transformations, cached responses for repeated patterns; cost management at scale requires this routing layer
• Coordination between agents requires shared infrastructure - at L5, agents often depend on outputs from other agents, work on related parts of the same codebase, or need to avoid conflicting changes; coordination is not possible without shared state infrastructure that individual agent tooling doesn't provide
• Audit and compliance become critical at scale - when hundreds of agents are taking actions in production systems, the audit trail requirement is not optional; centralized orchestration provides the audit logging infrastructure that compliance requires
• Observability at fleet scale requires instrumentation - visibility into what hundreds of agents are doing cannot be achieved by watching individual agent sessions; it requires aggregate dashboards, anomaly detection, and cost tracking at the orchestration layer
• Failure recovery at scale requires systematic approaches - individual agent failures are handled case-by-case at L4; at L5, systematic failure policies (retry, escalate, abandon, rollback) managed by the orchestration layer are required for reliable operation

1. Map your current agent workload before designing the orchestration - Before building infrastructure, understand what you're orchestrating: what task types are agents being used for, what is the distribution of task complexity, what models are currently being used, what are the failure patterns, what is the cost breakdown. This map is the requirements document for the orchestration system.

2. Start with task queuing and cost tracking - The first centralized infrastructure to build is the simplest: a task queue that receives agent work requests and a cost tracking system that records what each task costs. These two capabilities make the rest of the system designable - you can see the workload and you can see the economics.

3. Implement model routing with a clear policy - Define the model selection policy before automating it: what criteria determine whether a task gets a frontier model, a mid-tier model, or a smaller model? Typical criteria: task complexity (measured by context size and step count), output quality requirements, latency requirements, and cost ceiling. Automate the policy once it's clear, not before.

4. Build the context management layer - Centralized orchestration needs to manage the context that each agent receives. This includes: fetching the relevant codebase sections, injecting the architectural documentation, providing the team conventions, and limiting the context to what is actually needed for the task. A context management layer that provides right-sized, well-curated context improves both output quality and cost efficiency.

5. Implement sandbox and permission management - At L5 scale, agents need sandboxed execution environments with carefully managed tool access. An agent that has unnecessary file system access or network permissions is a governance risk at any scale; at L5, it is a systematic risk. The orchestration layer should enforce permission boundaries consistently rather than relying on per-agent configuration.

6. Build aggregate observability before you need it - Cost dashboards, throughput dashboards, failure rate dashboards, and agent queue depth - all of these need to be in place before the scale that makes them necessary, because building them retroactively while under operational pressure is much harder. Instrument the orchestration layer for observability from the start.

Building the orchestration system before the organization is at L5. Centralized agent orchestration is significant infrastructure investment. Organizations at L3-L4 that build it prematurely are over-investing in infrastructure for workloads that don't require it yet. The right time to build centralized orchestration is when the operational problems (cost management, coordination failures, audit gaps) of L4 fleet management are consistently appearing.

Designing for current scale instead of next-scale. Orchestration systems built for "our current 50 concurrent agents" will need to be redesigned when the organization moves to 500 concurrent agents. Design the orchestration system for 10x your current scale. The incremental cost of designing for scale is small; the cost of redesigning under operational pressure is large.

Centralizing everything and eliminating team autonomy. Centralized orchestration should handle infrastructure concerns (scheduling, routing, cost management, audit logging) without eliminating team-level flexibility in how agents are used, what context they receive, and what workflows they execute. Overcentralization creates a bottleneck and a single point of failure.

Treating the orchestration layer as a product instead of infrastructure. Centralized orchestration should be invisible to developers - they submit tasks and receive results. If the orchestration layer requires developers to understand routing policies, context management details, or scheduling logic, the abstraction is wrong. Build the layer so that developers interact with a simple interface and the complexity is managed underneath.

Not planning for model transitions. The frontier model landscape is changing every few months. Orchestration systems built tightly around a specific model's API require significant rework when that model is superseded. Build the model routing layer with model abstraction - the policy selects from a class of capabilities, and the specific model providing those capabilities can be swapped without rearchitecting the orchestration system.