Remote execution (EngFlow)

Remote execution distributes build actions across a cluster of machines rather than running them locally.

·Advanced build system (Bazel, Buck2, or Pants) is adopted for primary codebase
·Remote execution (EngFlow or equivalent) distributes build steps across multiple machines
·Incremental builds run only changed targets (not full rebuild)

·BUILD file maintenance is assigned to specific team members or automated
·Remote cache hit rate exceeds 80%

Evidence

·Bazel/Buck2/Pants BUILD files in repository
·Remote execution configuration (EngFlow, BuildBuddy, or equivalent)
·Build log showing incremental target selection

What It Is

Remote execution distributes build actions across a cluster of machines rather than running them locally. When a Bazel build encounters a compilation step, instead of running it on the developer's laptop or CI runner, it sends the action to a remote execution cluster - potentially hundreds of workers - and receives the result back. Actions with no dependencies can run in parallel across the cluster simultaneously. A build that would take 10 minutes to run sequentially on a single machine can complete in 30 seconds when 1,000 actions are distributed across 200 workers.

EngFlow is a commercial remote execution service built by former Google engineers who built the original Blaze (the internal Bazel predecessor). BuildBuddy is an open-core alternative with a self-hosted option. Google Cloud's RBE (Remote Build Execution) is the cloud-native offering. All three implement the Bazel Remote Execution API (REAPI), which means they're interchangeable from Bazel's perspective - switching between them requires changing configuration, not rewriting BUILD files.

Remote execution combines two capabilities: a remote action cache and distributed workers. The remote action cache stores the output of every build action keyed by its inputs. If 10 developers each run a build that includes compiling the same unchanged library, the first developer's compilation result is cached and the next 9 get instant cache hits. The distributed workers run cache-miss actions in parallel across the cluster. The combination means that in a large team, most build actions are cache hits, and the remaining cache misses run in parallel across the cluster.

For AI agent workflows, remote execution changes the economics of parallel agents completely. Without remote execution, 50 parallel agents each doing a fresh build consume 50x the local CPU and produce 50x the compilation work. With remote execution, the first agent to compile a given library populates the remote cache; all subsequent agents get cache hits for that library regardless of whether they're running on the same machine, different machines, or CI runners. The shared remote cache means that the build cost for N parallel agents approaches the cost of 1 agent plus the marginal cost of the incremental changes each agent is making.

Why It Matters

Build time is bounded by critical path, not codebase size - with enough remote workers, a build that takes 20 minutes sequentially can complete in under 2 minutes; the limit is the longest sequential chain, not the total number of actions
Shared remote cache eliminates redundant work across all agents - once any agent or CI runner compiles a given target, that compiled output is available to every other agent and runner for free
Local machine is no longer the bottleneck - a developer with a 4-core laptop gets the same build performance as a developer with a 64-core workstation, because the heavy lifting happens on the remote cluster
CI and local builds share the same cache - a developer's local Bazel build can hit cache entries populated by CI, and vice versa; the entire team's build work contributes to a shared cache that benefits everyone
Scales linearly with demand - adding 10 more developers running agents doesn't slow down existing builds; the remote execution cluster scales to absorb the additional load

Getting Started

Enable the Bazel remote cache first, before remote execution - Start with remote caching only: configure a GCS or S3 bucket as a Bazel cache backend. This gives immediate cross-machine cache sharing without the operational complexity of remote execution. Measure cache hit rates for 2-4 weeks to establish a baseline before adding remote workers.
Sign up for EngFlow or BuildBuddy trial - Both offer free trials with remote execution. EngFlow has a free tier suitable for small teams. BuildBuddy has an open-source self-hosted option. Start with the managed service to evaluate fit before investing in self-hosted infrastructure.
Configure Bazel remote execution in .bazelrc - Add remote execution flags: --remote_executor=grpcs://your-endpoint, --remote_cache=grpcs://your-endpoint, --remote_instance_name=your-project. Test with a clean build and verify actions are being executed remotely via the EngFlow or BuildBuddy UI.
Enable remote caching for CI first - CI is the highest-value initial target for remote caching. Every CI run on a new runner starts cold; a remote cache turns cold CI starts into warm builds. Configure your CI pipeline to pass remote cache flags to every Bazel invocation.
Monitor cache hit rates and action distribution - EngFlow and BuildBuddy provide dashboards showing cache hit rates by target, action count, and execution time distribution. Target 85%+ cache hit rates for established codebases. Low cache hit rates indicate hermetic build violations or incorrect BUILD file configurations.
Implement remote execution for the developer workflow - Once CI remote execution is stable, roll out remote execution for local developer builds. This requires distributing the remote execution credentials to developers (use a service account, not personal credentials). The impact on local build times is often dramatic: 30-second incremental builds become 5-second builds as cache hits dominate.

Tip

Use bazel build //... --remote_download_minimal in CI environments where you only need to verify the build succeeds but don't need the output artifacts locally. This flag tells Bazel to skip downloading build outputs, leaving them in the remote cache. It can reduce CI build time by 20-30% by eliminating large artifact downloads.

6 steps to get from here to the next level

Common Pitfalls

Running remote execution before fixing hermetic build violations. Remote execution requires truly hermetic builds - if your build reads from the local file system, shells out to host tools, or depends on environment variables, remote execution will fail or produce incorrect results. Fix all hermetic violations before enabling remote execution. The diagnostic tool is bazel build --sandbox_debug.

Not configuring credentials correctly for developers. Remote execution credentials need to be distributed to all developers. Using personal credentials creates a support burden and a security risk. Use a service account per team or per project with appropriate IAM permissions. Manage the credentials through your secrets manager, not by hand.

Ignoring network latency to the remote execution cluster. Remote execution is most effective when the cluster is geographically close to the developers and CI runners. A remote cluster in us-east-1 serving developers in Europe will see 150ms+ round-trip latency per action, which significantly reduces the effective parallelism. Use a cluster in the same region as your primary development team.

Not setting upload/download bandwidth limits. Large build outputs can saturate developer network connections if not rate-limited. Set --remote_max_connections and configure per-build upload limits. Developers on slow connections may need to use remote execution only for CI and use local builds during development.

Treating EngFlow and BuildBuddy as commodity infrastructure. These services require ongoing operational attention: monitoring cache hit rates, managing cache eviction policies, right-sizing the execution cluster. Assign a specific owner for remote execution infrastructure and build it into your infrastructure engineering roadmap.

Mistakes teams actually make at this stage - and how to avoid them

How Different Roles See It

BobHead of Engineering

Bob approved the Bazel migration 6 months ago and the team has been running Bazel with a remote cache but not yet with remote execution. Incremental builds are 15-30 seconds locally and 90 seconds in CI for cache misses. His agents are running well but he's heard that Google-scale teams get sub-10-second builds across any codebase size. He's wondering if remote execution is worth the additional investment.

What Bob should do: Bob should have his infrastructure team run a cost-benefit analysis on remote execution. Input data: current CI cost, current agent iteration volume, average build time with cache misses, and the frequency of cache misses (which determines how often the slower path is hit). If 30% of agent CI runs are cache misses taking 90 seconds each, and agents generate 2,000 builds per day, that's 600 builds at 90 seconds = 15 hours of compute time daily. Remote execution with a large cluster can cut those 90-second cache-miss builds to under 20 seconds. At that volume, the cost savings in CI compute typically offset the EngFlow/BuildBuddy subscription within a quarter.

What Bob should do - role-specific action plan

SarahProductivity Lead

Sarah's DevEx dashboard shows that agent iteration time has two modes: 5-10 seconds for cache hits (85% of builds) and 90 seconds for cache misses (15% of builds). The cache miss tail is the remaining friction in agent iteration loops. Developers notice and complain about the occasional slow builds even though the average is good.

What Sarah should do: Sarah should focus first on understanding why 15% of builds are cache misses. Are they builds triggered by BUILD file changes? New branches with different commit ranges? First-time target compilations? Each cause has a specific mitigation. If cache misses are primarily first-build-on-new-branch scenarios, seeding the cache from the main branch build solves most of them. If they're BUILD file changes causing widespread invalidation, that's a BUILD file hygiene issue. Sarah should reduce the cache miss rate to under 5% before investing in remote execution to speed up the remaining misses.

What Sarah should do - role-specific action plan

VictorStaff Engineer - AI Champion

Victor has configured remote execution with EngFlow for his team and is seeing 8-second builds for all targets, including cache misses, because EngFlow distributes the compilation across 50 workers simultaneously. He's now running 8 parallel agents (expanded from his previous 5 limit) because the build system can absorb the additional CI load without degradation.

What Victor should do: Victor should quantify the concurrent agent scaling property: at what number of parallel agents does build performance start to degrade with remote execution, if ever? He should run scaling experiments: 5, 10, 20 parallel agents, each doing independent builds, and measure p50 and p95 build times at each level. If build time stays flat up to 20 parallel agents (it should, with a properly sized cluster), Victor has quantitative evidence that remote execution changes the agent scaling equation fundamentally. This data should go to Bob as the business case for making remote execution a team-wide standard rather than a Victor-specific setup.

What Victor should do - role-specific action plan