Capability 04 of 04 · Development
Testing Strategy
How tests are written, maintained, and validated in an AI-assisted workflow. A test "oracle" is the source of truth for what a test's correct result should be.
5
maturity levels
15
practices
15
guides
L1 · Stage 01Ad-hoc
Practices - what it looks like3 guides
- Tests written by handThe baseline testing state at L1 - manual test writing, chronic under-coverage, and the compounding debt that makes AI-generated code increasingly risky to ship.guide→
- Flaky tests are a recurring costFlaky tests silently consume a sixth of your engineering capacity - Google's internal research quantified the cost, and at L1 they're accepted as normal instead of eliminated.guide→
- AI tests treat current output as "correct" - no independent oracle for the intended resultWhen AI generates tests by reading your implementation, it encodes existing behavior as correct - including bugs - giving you coverage numbers that feel like safety but provide none.guide→
Criteria - what to measure
- 01An automated test suite exists and runs
- 02The team writes and maintains its own tests
- 03Team is aware of flaky test impact (16% of dev time per Google data)
- 04AI-generated tests are reviewed for circular testing (testing what code does, not what it should do)
L2 · Stage 02Guided
Practices - what it looks like3 guides
- Agent-generated unit tests + human acceptance testsA hybrid testing strategy at L2 that uses AI to generate unit test scaffolding at scale while keeping business-behavior verification in human hands.guide→
- Flaky test quarantineA systematic L2 process for removing flaky tests from the main CI signal without deleting them - creating accountability to fix them while keeping builds reliable.guide→
- Humans define expected results for key paths (acceptance tests are the oracle)Fixing flaky tests at the root by replacing fragile implementation-coupled assertions with stable, behavior-level oracles that reliably distinguish real failures from noise.guide→
Criteria - what to measure
- 01Agents generate unit tests; humans write acceptance tests
- 02Flaky test quarantine process is active (flaky tests are isolated, not deleted)
- 03Humans define the expected results for important paths (not just snapshotting current output)
- 04Flaky test count is tracked and reported weekly
- 05Quarantined tests have a resolution SLA (e.g., fix or delete within 30 days)
L3 · Stage 03Systematic
Practices - what it looks like3 guides
- Expected results come from requirements (tickets/specs are the oracle, not the code)TORS quantifies what percentage of test failures are real bugs - at L3, achieving 90%+ is the prerequisite for trusting automated quality gates and enabling AI-driven testing workflows.guide→
- Acceptance tests from tickets (Autonomous Requirements)At L3, AI agents read requirements tickets and generate failing acceptance tests before implementation begins - making requirements machine-executable and eliminating circular testing at scale.guide→
- Incremental test selection (only changed paths)Running only the tests affected by a given code change - using dependency graph analysis - so CI feedback stays fast as the codebase grows to millions of lines.guide→
Criteria - what to measure
- 01Expected results are derived from requirements/specs (the requirement is the oracle, not the code)
- 02Acceptance tests are auto-generated from ticket requirements (Autonomous Requirements pipeline)
- 03Incremental test selection runs only tests affected by changed code paths
- 04Oracle reliability is reviewed per service, not just overall
- 05Test generation from tickets includes edge cases, not just happy paths
L4 · Stage 04OptimizedMost teams aim here
Practices - what it looks like3 guides
- A failing test reliably means a real defect; held-out oracles the agent never sees gate releases (SpecBench: validation lies as LOC grows; Kong fixed 12/15 flaky tests with orchestrator+verifier)At L4, raising the Test Oracle Reliability Score to 95%+ is the prerequisite for trusting automated merge decisions - where 1-in-20 false positives is the maximum the system can tolerate.guide→
- Agent iterates tests to green in sandbox (doesn't block team CI)AI agents fix failing tests in isolated sandboxes - running their own private CI loop - so agent work-in-progress never pollutes the shared pipeline or slows the team.guide→
- Mutation testing agent validationAt L4, AI agents use mutation testing to verify that agent-generated test suites actually catch bugs - not just execute code - before those tests enter the shared codebase.guide→
Criteria - what to measure
- 01A failing test reliably indicates a real defect (oracle false-positives are rare)
- 02Agents iterate tests to green in isolated sandbox CI without blocking team CI queue
- 03Mutation testing validates that tests catch real defects (not just achieve coverage)
- 04Sandbox CI iteration count per PR is tracked (ITS target: 1-3)
- 05Mutation testing kill rate exceeds 80%
L5 · Stage 05Autonomous
Practices - what it looks like3 guides
- Self-healing test suiteAt L5, AI agents continuously maintain the test suite - detecting and fixing flaky tests, updating tests broken by intentional refactors, and generating tests for uncovered paths - so humans set quality policy rather than doing quality work.guide→
- Production logs → auto-generated regression testsAt L5, agents mine production errors to automatically generate regression tests - capturing the exact inputs that caused real failures so they can never reach production again.guide→
- Agent detects edge case → writes test → fixes bug → shipsThe fully autonomous quality loop at L5: an agent finds an edge case, writes a failing test, fixes the bug, verifies all tests pass, and submits the PR without any human involvement in the cycle.guide→
Criteria - what to measure
- 01Test suite is self-healing (agent detects broken tests, diagnoses root cause, fixes without human input)
- 02Production logs automatically generate regression tests for observed failures
- 03Agents detect edge cases, write tests, fix bugs, and ship - full autonomous loop
- 04Self-healing test updates are validated by mutation testing before merge
- 05Production-to-test pipeline latency is under 1 hour (failure observed to regression test committed)
From the Field
Recent releases, projects and discussions the AI Radar classified into this capability.
release
core-v0.3.1
article
An Autonomous RL Agent Methodology for Dynamic Web UI Testing in a BDD Framework
release
[email protected]
article
Meta Reports 4x Higher Bug Detection with Just-in-Time Testing
article
EvanFlow – A TDD driven feedback loop for Claude Code
release
release
article
How we know if our agent is right
Which level is your team at in Testing Strategy?
The criteria above are what we check in a facilitated assessment. Run it yourself to place this capability, see which gates you have passed, and compare it against the other 3 in Development.
Start the assessment →