Capability 04 of 04 · Development
Testing Strategy
How tests are written, maintained, and validated in an AI-assisted workflow. A test "oracle" is the source of truth for what a test's correct result should be.
5
maturity levels
15
practices
15
guides
L1 · Stage 01Ad-hoc
Practices - what it looks like3 guides
- Tests written by handThe baseline testing state at L1 - manual test writing, chronic under-coverage, and the compounding debt that makes AI-generated code increasingly risky to ship.guide→
- Flaky tests are a recurring costFlaky tests silently consume a sixth of your engineering capacity - Google's internal research quantified the cost, and at L1 they're accepted as normal instead of eliminated.guide→
- AI tests treat current output as "correct" - no independent oracle for the intended resultWhen AI generates tests by reading your implementation, it encodes existing behavior as correct - including bugs - giving you coverage numbers that feel like safety but provide none.guide→
Criteria - what to measure
- 01An automated test suite exists and runs
- 02The team writes and maintains its own tests
- 03Team is aware of flaky test impact (16% of dev time per Google data)
- 04AI-generated tests are reviewed for circular testing (testing what code does, not what it should do)
L2 · Stage 02Guided
Practices - what it looks like3 guides
- Agent-generated unit tests + human acceptance testsA hybrid testing strategy at L2 that uses AI to generate unit test scaffolding at scale while keeping business-behavior verification in human hands.guide→
- Flaky test quarantineA systematic L2 process for removing flaky tests from the main CI signal without deleting them - creating accountability to fix them while keeping builds reliable.guide→
- Humans define expected results for key paths (acceptance tests are the oracle)Fixing flaky tests at the root by replacing fragile implementation-coupled assertions with stable, behavior-level oracles that reliably distinguish real failures from noise.guide→
Criteria - what to measure
- 01Agents generate unit tests; humans write acceptance tests
- 02Flaky test quarantine process is active (flaky tests are isolated, not deleted)
- 03Humans define the expected results for important paths (not just snapshotting current output)
- 04Flaky test count is tracked and reported weekly
- 05Quarantined tests have a resolution SLA (e.g., fix or delete within 30 days)
L3 · Stage 03Systematic
Practices - what it looks like3 guides
- Expected results come from requirements (tickets/specs are the oracle, not the code)TORS quantifies what percentage of test failures are real bugs - at L3, achieving 90%+ is the prerequisite for trusting automated quality gates and enabling AI-driven testing workflows.guide→
- Acceptance tests from tickets (Autonomous Requirements)At L3, AI agents read requirements tickets and generate failing acceptance tests before implementation begins - making requirements machine-executable and eliminating circular testing at scale.guide→
- Incremental test selection (only changed paths)Running only the tests affected by a given code change - using dependency graph analysis - so CI feedback stays fast as the codebase grows to millions of lines.guide→
Criteria - what to measure
- 01Expected results are derived from requirements/specs (the requirement is the oracle, not the code)
- 02Acceptance tests are auto-generated from ticket requirements (Autonomous Requirements pipeline)
- 03Incremental test selection runs only tests affected by changed code paths
- 04Oracle reliability is reviewed per service, not just overall
- 05Test generation from tickets includes edge cases, not just happy paths
L4 · Stage 04OptimizedMost teams aim here
Practices - what it looks like3 guides
- Held-out oracles the agent never sees gate releases ("Building to the Test": with oracle access agents ship dead code passing all 222 tests); property-based testing + fuzzing over LLM-written testsAt L4, raising the Test Oracle Reliability Score to 95%+ is the prerequisite for trusting automated merge decisions - where 1-in-20 false positives is the maximum the system can tolerate.guide→
- Agent iterates tests to green in sandbox (doesn't block team CI)AI agents fix failing tests in isolated sandboxes - running their own private CI loop - so agent work-in-progress never pollutes the shared pipeline or slows the team.guide→
- Mutation testing agent validationAt L4, AI agents use mutation testing to verify that agent-generated test suites actually catch bugs - not just execute code - before those tests enter the shared codebase.guide→
Criteria - what to measure
- 01A failing test reliably indicates a real defect (oracle false-positives are rare)
- 02Agents iterate tests to green in isolated sandbox CI without blocking team CI queue
- 03Mutation testing validates that tests catch real defects (not just achieve coverage)
- 04Sandbox CI iteration count per PR is tracked (ITS target: 1-3)
- 05Mutation testing kill rate exceeds 80%
L5 · Stage 05Autonomous
Practices - what it looks like3 guides
- Self-healing test suiteAt L5, AI agents continuously maintain the test suite - detecting and fixing flaky tests, updating tests broken by intentional refactors, and generating tests for uncovered paths - so humans set quality policy rather than doing quality work.guide→
- Production logs → auto-generated regression testsAt L5, agents mine production errors to automatically generate regression tests - capturing the exact inputs that caused real failures so they can never reach production again.guide→
- Agent detects edge case → writes test → fixes bug → shipsThe fully autonomous quality loop at L5: an agent finds an edge case, writes a failing test, fixes the bug, verifies all tests pass, and submits the PR without any human involvement in the cycle.guide→
Criteria - what to measure
- 01Test suite is self-healing (agent detects broken tests, diagnoses root cause, fixes without human input)
- 02Production logs automatically generate regression tests for observed failures
- 03Agents detect edge cases, write tests, fix bugs, and ship - full autonomous loop
- 04Self-healing test updates are validated by mutation testing before merge
- 05Production-to-test pipeline latency is under 1 hour (failure observed to regression test committed)
From the Field
Recent releases, projects and discussions the AI Radar classified into this capability.
release
trycua/cua
article
An Autonomous RL Agent Methodology for Dynamic Web UI Testing in a BDD Framework
release
langwatch/langwatch
article
Meta Reports 4x Higher Bug Detection with Just-in-Time Testing
article
EvanFlow – A TDD driven feedback loop for Claude Code
release
langchain-ai/langchain
release
promptfoo/promptfoo
article
How we know if our agent is right
Which level is your team at in Testing Strategy?
The criteria above are what we check in a facilitated assessment. Run it yourself to place this capability, see which gates you have passed, and compare it against the other 3 in Development.
Start the assessment →