Production feedback → CI auto-adjusts test suite

"Production feedback drives CI test suite adjustment" is an L5 pattern where the CI test suite is not a static artifact maintained by engineers but a dynamic system that evolves based on what's actually going wrong in production. When a production incident occurs, the system automatically: identifies the code path that failed, generates a regression test covering that failure mode, adds the test to CI, and ensures that path is covered on every future change that touches the affected code. The test suite grows in response to real failures, not anticipated failures.

The pattern works in the opposite direction too: when production telemetry shows that a code path has never caused a problem and has not been changed in 6 months, the system can flag the tests covering it as candidates for deprioritization in the fast CI path - moving them to a weekly validation suite rather than running them on every commit. The test suite's composition is continuously optimized: more coverage where production failures occur, less coverage of stable code paths that are rarely exercised in production.

This is a natural extension of the production telemetry patterns that already exist in mature engineering organizations (distributed tracing, error tracking, anomaly detection). The new element is the feedback loop: production signals automatically trigger test suite changes rather than requiring a human to analyze an incident, decide to write a regression test, implement it, and add it to CI. At L5, this loop runs continuously and automatically, with human review for the generated tests but no human requirement to initiate the process.

Thoughtworks made this an explicit invariant in its operating model for enterprise AI agent reliability, published 2026-08-14: every production failure becomes a permanent regression test. The word doing the work is "permanent". A regression test that can be deleted by the next agent tidying up a test file is not a control, it is a comment. In the Thoughtworks model each of six layers - terminology, routing, agent intent, semantic context, execution, result - has a named owning team and an executable "truth contract", and the regression suite is where those contracts accumulate. If you are building this loop, decide up front which team owns the generated test and what it takes to remove one.

The mechanism typically involves: error tracking (Sentry, Datadog) that identifies failing code paths in production; an agent that reads the error and the relevant source code and generates a regression test; a CI integration that adds the test to the affected module's test suite; and a review step (which can be automated to skip human review for straightforward regression tests with high confidence). The loop closes when the regression test is added to CI and future changes to the affected code path run against it automatically.

The same loop now has to answer a harder question: where verification happens at all. CircleCI data across 28 million workflows, analysed by Paul Stack (AI broke the assumptions behind CI, 2026-08-26), shows throughput up 59% year on year while main-branch success fell to 70.8%, a five-year low, and recovery time rose 13%. More changes are arriving, more of them are breaking main, and fixing main takes longer. Stack's argument is that CI was built on the assumption that a human ran things locally before pushing, and agents broke that assumption. His proposal is to move verification before the pull request opens - the agent runs the relevant tests, including every production-derived regression test for the code it touched, and attaches the results as evidence - and to have CI check that evidence rather than re-run everything from scratch. In a shadow run over 35 pull requests, nothing slipped through. Thirty-five PRs is a small sample, and the idea drew public pushback (Jeremy Miller replied on 2026-08-31), so treat it as a direction to pilot rather than a settled result. But it fits this pattern exactly: the production-derived suite is the most valuable thing an agent can run before it opens a PR, and CI's job shifts from repeating that run to verifying it happened, against the right commit, with the right tests.

This is also where the capacity problem lands. The martinfowler.com write-up "An Accidental Blackboard" (2026-09-02), in which ten engineers built an airline disruption-management system in four days with many agents, reports that the agents strained the build pipelines. A CI that re-runs every agent's full suite on every push will not keep up; a CI that checks signed evidence and re-runs only what is missing or suspicious has a chance.

• Test coverage grows where it matters most - tests are added for code paths that fail in production, which is the most reliable signal of where coverage is needed; coverage grows in response to real risk rather than developer intuition
• Regression prevention becomes automatic - every production incident generates a regression test; the same bug cannot ship again without failing CI; the test suite becomes an automatically maintained safety net
• CI test suite stays relevant as codebases evolve - code paths that are never exercised in production and never changed are deprioritized in CI, keeping the test suite proportional to actual risk; the suite doesn't grow unboundedly
• Eliminates "we should write a regression test" backlog items - production incidents generate regression tests immediately and automatically; there's no backlog of "we should have a test for this" items because the system creates them
• Permanence is the whole point - Thoughtworks' operating model treats every production failure as a permanent regression test with a named owning team; a suite that agents can quietly prune is a suite that will regress to the mean
• Main is breaking more often as agent volume rises - across 28M CircleCI workflows, throughput rose 59% while main-branch success fell to a five-year low of 70.8% and recovery time grew 13%; catching failures before the PR opens, with production-derived tests, is cheaper than repairing main after
• Demonstrates that CI and production are a continuous loop, not separate stages - production feedback flowing back to CI represents the highest level of CI maturity: the pipeline learns from reality rather than being a static artifact

1. Establish production error tracking with code path attribution - Before auto-generating tests, you need production errors attributed to specific code paths. Sentry, Datadog APM, and Honeycomb all provide stack trace analysis that maps errors to source code locations. Configure your error tracking to capture full stack traces and correlate them with your current deployed version's source map or symbol table.
2. Build a "production failure to test spec" translation layer - Create a process (initially human-run, eventually agent-automated) that takes a production error with its stack trace and converts it into a test specification: what inputs triggered the error, what was the expected behavior, what was the actual behavior. This can start as a structured post-incident template and evolve toward automated generation.
3. Implement an agent-based regression test generator - Use Claude Code or a similar agent to take the test specification (error description, failing code path, relevant context) and generate a regression test in the appropriate test framework. The agent needs access to: the error details, the source code of the failing function, examples of existing tests in the same file as style reference, and the test framework's assertion patterns.
4. Add CI integration for automatically-generated tests - Generated tests should be added to the test file for the failing module with a metadata tag (a comment or test attribute) identifying them as production-feedback-generated. This allows tracking them separately and reviewing them as a group. Configure CI to run these tests with the same priority as manually written tests.
5. Implement test coverage-to-production-path correlation - Use code coverage data (Istanbul/nyc for JavaScript, coverage.py for Python, JaCoCo for Java) correlated with production code paths (from APM traces) to identify over-tested stable code and under-tested risky code. This correlation is the basis for deprioritizing stable tests and prioritizing risky ones.
6. Gate the generated tests on mutation strength, not on their existence - a generated regression test can pass, add a line to the coverage report and assert nothing meaningful. Sławomir Radzymiński's August write-up found one component at 100% line coverage and 61% mutation strength. Apply mutation testing selectively to the high-risk areas the generated tests land in - auth, business rules, payments - and treat the mutation score, not the test count, as the signal the loop is working.
7. Make the generated test permanent and owned - tag it with the incident it came from, assign it to the team that owns the failing layer, and require an explicit justification to delete it. Otherwise the next tidy-up pass removes exactly the tests you paid an outage for.
8. Move verification before the PR opens, and make CI check the evidence - require agents to run the affected tests, including every production-derived regression test for the touched code paths, before opening a PR, and to attach the results (commit hash, test list, outcomes) to it. Have CI verify that evidence matches the head commit and covers the required tests, and re-run only what is missing, stale or on a sampled audit basis. Pilot it in shadow mode first - run the old full suite alongside and count what the evidence-check would have let through - as Stack did over 35 PRs.
9. Start with human review of generated tests before automation - For the first 3 months, have a human engineer review each auto-generated regression test before it's added to CI. This validates the quality of the generation and builds the team's confidence in the process. After reviewing 50-100 tests, you'll know the false positive rate well enough to decide which categories of auto-generated tests can be added without review.

Generating tests that test implementation details rather than behavior. Auto-generated tests that directly assert on internal function structure rather than observable behavior will break every time the implementation changes, even when behavior is correct. Validate generated tests against the behavior-specification principle: do they test inputs and outputs, not implementation?

Adding auto-generated tests without fixing the underlying bug. A regression test that fails on the production-reproducing input is valuable. But if it's added to CI without also fixing the bug that caused the production failure, every CI run will fail until someone fixes it. The workflow must be: fix the bug first, then add the regression test to prevent recurrence. Not the other way around.

Deprioritizing tests based on coverage data alone. Coverage data says which code paths are executed by tests, not which code paths are important. A path that has 100% test coverage but is never exercised in production (a rarely-used admin feature, a deprecated API endpoint) may be safe to deprioritize. But a path with 100% test coverage that handles critical production traffic should not be deprioritized even if it's stable. Use production traffic data alongside coverage data for deprioritization decisions.

Not attributing auto-generated tests in version control. Generated tests should be clearly marked as auto-generated in their file header or with test metadata. This allows engineers to understand the test's origin, helps during code review ("this test was auto-generated from production error XYZ"), and enables bulk operations (find all auto-generated tests for a module, review their quality as a batch).

Creating a feedback loop that generates tests faster than the team can maintain them. An organization with many production incidents generating automatic regression tests could accumulate thousands of tests quickly. Monitor test suite growth rate and set a threshold that triggers review: if more than X tests are generated per week, review the generation quality and consider tightening the generation criteria. SlopCodeBench's July 2026 run is the cautionary number here: ~29,000 source lines of which 51% were test code, with complexity and duplication rising at every checkpoint for every model. Test-suite mass is a warning sign, not a maturity signal.

Trusting agent-supplied evidence without checking it. Evidence-based CI only works if the evidence is bound to the exact commit and the exact test set, produced by a runner the agent cannot tamper with, and spot-checked by re-running a sample. An agent that reports "all tests passed" in a PR description has produced a claim, not evidence.

Reading a green auto-generated suite as proof of coverage. Mutation testing is the check that catches this, and it is now the recommended monitor for agent-written code generally - Birgitta Böckeler's August experiment on TDD inside the agent loop ends with exactly that recommendation, having found that agents told to TDD produced worse-judged solutions at 3-8x the tokens. Her result rests on five batches of greenfield business logic judged largely by a model, so carry it as a provocation rather than a settled finding; the mutation-testing advice stands on its own either way.