CI as Sandbox: 50 attempts in 5 min without blocking team

"CI as Sandbox" is a configuration pattern where the CI system is intentionally designed to support rapid, high-frequency iteration by AI agents, isolated from the normal developer CI workflow. Instead of running CI as a quality gate that blocks merging, agents run CI as an exploration environment - attempting up to 50 runs in a 5-minute window without consuming any of the team's normal CI capacity or affecting other developers' queue times.

The key architectural requirement is isolation: agent sandbox CI runs must be on separate infrastructure from team CI runs. When an agent attempts 50 CI runs in 5 minutes, those 50 runs need to complete against an agent-specific runner pool, not the shared team pool. Without this isolation, agents would consume all available CI capacity and queue every developer's PR for the duration of the agent's iteration session.

The "50 attempts in 5 minutes" target comes from the iteration rate that enables agents to solve complex problems autonomously. An agent working on a failing test or a type error needs to try an approach, observe the result, and try again. At 50 attempts per 5 minutes, that's one attempt every 6 seconds. This requires sub-minute CI (see the Sub-minute Feedback guide) on the agent's dedicated runners. The agent that can attempt and discard 50 approaches in 5 minutes has a fundamentally different problem-solving capability than one that's limited to 5 attempts in the same period.

This pattern is already present in several frontier engineering organizations. Stripe's internal "Minions" agent system spawns lightweight sandboxes that can run dozens of CI checks in parallel. GitHub Copilot Workspace's fix-and-verify loop is designed around rapid CI iteration. The pattern is becoming standard at L4-L5: agents don't just submit code and wait for CI; they iterate against CI as a feedback mechanism during the task itself.

• Agents solve harder problems autonomously - an agent with 50 attempts in 5 minutes can converge on a working solution through rapid iteration; an agent with 3 attempts must be much more precise in each attempt, limiting problem complexity
• Human developers are never blocked - dedicated agent sandbox runners mean agent burst activity doesn't affect human CI queue times; the team never notices agent iteration happening
• Total solution time compresses - a feature that requires 20 iterations to implement takes 2 minutes at "CI as sandbox" rates vs. 100 minutes at 5-minute CI; this makes complex autonomous tasks viable within a single session
• Enables TDD-style agent development - agents can write a failing test, attempt implementations against CI feedback, converge to green, and produce a clean test-driven implementation without human steering at each step
• Eliminates the "try, wait, give up" pattern - agents on slow CI often "give up" on an approach after 2-3 slow iterations because the cost per attempt is too high; high-frequency iteration changes agent behavior to "try, adjust, converge"

1. Provision a dedicated agent sandbox runner pool - This pool is distinct from the team's standard CI runners. Size it for peak concurrent agent count times 10x (agents running fast CI generate 10x the jobs of human developers). Use autoscaling to handle burst patterns. Tag these runners agent-sandbox and never let human developer workflows route to them.
2. Create a lightweight "agent sandbox" CI pipeline - This pipeline runs only the checks relevant to the agent's current task, not the full suite. Typically: lint, type checking, and the test files directly related to the changed code. Target: under 30 seconds. This is distinct from the full CI pipeline that runs on PR creation. The agent sandbox pipeline is for rapid iteration; the full pipeline is for quality gates.
3. Implement a concurrency limit for sandbox pipelines - An agent iterating at 50 attempts per 5 minutes without any throttling can overwhelm even a dedicated runner pool. Set a per-agent concurrency limit: one agent can have at most N sandbox runs in progress simultaneously (start with N=5). This ensures predictable resource consumption without blocking the agent's iteration speed.
4. Connect the CI sandbox to the agent's feedback loop - The agent must receive CI results before deciding on the next attempt. Configure the sandbox pipeline to push results to the agent via: GitHub status checks (which agents can poll), webhook notifications to an agent controller, or a results endpoint the agent queries. Sub-30-second pipelines make polling practical.
5. Define the "sandbox to gate" transition - The agent sandbox pipeline and the PR quality gate pipeline are different. When the agent is confident its solution is correct (sandbox pipeline green for N consecutive runs, or the agent signals completion), trigger the full quality gate pipeline on the resulting PR. This transition is the point where sandbox iteration ends and validation begins.
6. Monitor sandbox usage for cost and abuse - Track: total sandbox runs per agent per session, cost per sandbox run, and completion rate (what fraction of sandbox sessions end in a successful quality gate run). This data identifies agents that are iterating inefficiently (many attempts, low convergence rate) and informs prompt engineering or capability improvements.

Not isolating sandbox runners from team runners. If the sandbox runners are the same pool as the team's runners, 50 agent attempts in 5 minutes will block every developer's CI run for that period. Isolation is the core requirement, not an optimization. Without it, "CI as sandbox" is not viable.

Running the full test suite in the sandbox pipeline. The sandbox pipeline must be fast - under 30 seconds, ideally under 15. Running the full test suite in the sandbox defeats the purpose. Define a minimal, relevant subset of checks for the sandbox pipeline and save the full suite for the quality gate transition.

Allowing agents to run sandbox CI without human-defined task scope. An agent running sandbox CI without clear task scope may iterate on aspects of the codebase that weren't part of the original task, consuming sandbox resources on off-scope work. Define agent task scope explicitly (which files, which tests, which behavior) before initiating sandbox iteration sessions.

Treating sandbox success as equivalent to quality gate success. A sandbox pipeline that passes is not a substitute for the full quality gate. Teams that skip the quality gate step because "the agent's sandbox runs were green" will regress. The sandbox is for iteration speed; the quality gate is for correctness validation. Both are required.

Ignoring the cost of 50 CI runs per session. Even at cheap sandbox pipeline costs (30 seconds × 50 attempts × $0.001/second = $1.50 per session), high-frequency agent usage can generate significant compute costs. Monitor cost per agent session and set a budget alert. Agents that consistently need 50+ attempts before converging may need improved task specifications or agent capability, not more sandbox quota.