Capability 03 of 04 · Delivery Management
Metrics
What you measure to understand AI-assisted engineering productivity and quality.
5
maturity levels
15
practices
15
guides
L1 · Stage 01Assisted
Practices - what it looks like3 guides
- Delivery performance measured at all (DORA, SPACE or an equivalent set), if trackedAt L1 (Assisted), most engineering teams track DORA metrics inconsistently or not at all.guide→
- Standard delivery metrics (not yet AI-specific)At L1, engineering teams that have adopted AI tools - GitHub Copilot, Cursor, Claude Code - are tracking those tools with zero AI-specific metrics.guide→
- ROI of AI not yet measured"How much did we save with AI?" is the question every engineering leader eventually faces from finance, from the CTO, or from the board.guide→
Criteria - what to measure
- 01The team can say how often it ships and how often a change fails
- 02The team looks at these metrics at least once a month
- 03Team acknowledges that its existing delivery metrics do not capture AI-assisted work
- 04Basic deployment frequency is at least known (even if not dashboarded)
L2 · Stage 02Delegated
Practices - what it looks like3 guides
- A delivery-performance baseline plus basic AI tracking; per-session token spend; input tokens (context), not output, drive spend - watch power users well above the medianAt L2 (Delegated), teams have moved past the L1 silence on metrics.guide→
- Licenses vs usage rate - but never token spend or seat activity as an adoption target; both are gamed within weeks, and Meta scrapped its 85k-employee token leaderboard and pulled AI usage from performance reviewsThe first uncomfortable AI metric is that 30-50% of the AI coding licenses an organization pays for sit unused in any given month - a diagnostic worth measuring, and a target that is gamed within weeks of being set.guide→
- PR throughput per dev as a proxy, never a target - and never suggestion acceptance rate, of which 31% is deleted within 15 minutesPRs merged per developer per week is a crude proxy, since PRs vary wildly in size, but it is the first output signal available without real instrumentation - useful as a diagnostic, ruinous as a target, and far better than suggestion acceptance rate.guide→
Criteria - what to measure
- 01A delivery-performance baseline (throughput, lead time, change failure rate, restore time - DORA, SPACE or an equivalent set) is on a dashboard the team can open
- 02AI tool license count vs. active usage rate is measured
- 03PR throughput per developer is tracked
- 04Cost per merged PR is measured per tool (acceptance rate is not used as a quality signal)
- 05Metrics are reviewed in team retrospectives at least monthly
L3 · Stage 03Systematic
Practices - what it looks like3 guides
- Cost per iteration (CPI) measured per task, not per token: cheaper models can cost more per task (Gemini 3.8 Flash: same token price, $0.40 -> $0.58 per task), and cheaper tokens made sessions 3.3x longer; include CI compute and the production compute the generated code burns (+5-8%)Cost-per-Iteration (CPI) measures what a single agent CI attempt costs across three components - model tokens, CI compute, and the compute the generated code goes on to burn in production - with cost per merged PR tracked as a trend.guide→
- Iterations to success (ITS): counted per change, limit set by the teamIterations-to-Success (ITS) is an AI-native metric that measures how many CI attempts it takes for an agent's PR to pass.guide→
- Push-to-result time: median recorded per reporting periodCI Feedback Latency is the time from when an agent pushes a commit to when CI produces a result (pass or fail) that the agent can act on.guide→
Criteria - what to measure
- 01The team measures the cost of one agent iteration (cost per iteration, CPI). The cost includes model tokens and CI compute.
- 02The team counts the iterations that one change needs to pass CI (iterations to success, ITS).
- 03The team records the median time from a push to the CI result, for each reporting period.
- 04The team sets a limit for the cost per iteration and for the iteration count.
- 05The team reports each measurement per team and per repository.
- 06The team attributes cost to a delivered change, not only to a pull request.
L4 · Stage 04GovernedMost teams aim here
Practices - what it looks like4 guides
- Test-oracle reliability tracked as a metric, alongside model-regression signals (thinking length, files read before edit)The Test Oracle Reliability Score is the share of CI failures that are real defects rather than flake. Above 95% is what makes auto-merge trustworthy.guide→
- Every agent run terminates in a classified state - success / flawed / blocked / manual - and each non-success class routes to a different fix; only the classification rate justifies expanding the automation boundaryEvery agent run terminates in one of four classified states - success, flawed, blocked or manual - only success ships, each of the other three routes to a different fix, and the classification rate is what justifies expanding the automation boundary.guide→
- Agent Autonomy Score: % tasks without human interventionThe share of tasks an agent takes from assignment to merge with no human intervention: no clarifying questions, no course corrections, no manual fixes.guide→
- Merge Queue Wait < 10 minMerge Queue Wait is the time a PR spends waiting in the merge queue after all gates pass (CI green, reviews approved, policy rules satisfied) before it is actually merged.guide→
Criteria - what to measure
- 01Test-oracle reliability is measured and tracked on a dashboard
- 02Every agent run is recorded with a terminal state (success / flawed / blocked / manual), and the distribution is reviewed
- 03Each non-success state has a named owner and routes to a different remedy
- 04Agent Autonomy Score (% of tasks completed without human intervention) is measured and broken down by task type
- 05Metrics trigger automated alerts when thresholds are breached (e.g., test-oracle reliability drops)
L5 · Stage 05Self-improving
Practices - what it looks like2 guides
- Cost-per-feature (not cost-per-PR); CFO scorecard: Useful Work, Cost per Successful Task, Return on Compute - paired with incidents-per-merged-change and firefighting hours, which move in the opposite directionCost-per-feature is the total cost - AI compute, CI infrastructure, human review and product management time - to deliver a complete user-facing feature, reported on a CFO scorecard of Useful Work, Cost per Successful Task and Return on Compute, and always paired with incidents-per-merged-change and firefighting hours.guide→
- Business value throughput, not activity metricsOnce agents make engineering activity cheap, PRs merged stops measuring anything useful and revenue, churn and conversion become the primary metrics.guide→
Criteria - what to measure
- 01Cost-per-feature is tracked (not cost-per-PR) - aggregating all agent, CI, and review costs per delivered feature
- 02Business value throughput is the primary metric (features delivered per week, not PRs merged per week)
- 03Metrics system auto-detects vanity metrics (high activity, low value delivery) and flags them
- 04Cost-per-feature trend is declining quarter-over-quarter
From the Field
Recent releases, projects and discussions the AI Radar classified into this capability.
discussion
How to understand if AI is adding value?
discovered
Deep analytics and session insights for your Claude Code usage
article
Dynamic Languages Faster and Cheaper in 13-Language Claude Code Benchmark
article
Agent SLOs: Grounding autonomous agents in metrics that matter
article
Show HN: Kerf-CLI – SQLite-backed cost analytics for Claude Code
discovered
Platform Evolution Engine. Distill the best from the combinatorial mess.
discovered
See where your AI coding tokens go. Interactive TUI dashboard for Claude Code cost observability.
discovered
See where Claude Code is burning tokens - turn raw JSONL transcripts into local cost analytics, hotspot views, and session-level usage insight.
Which level is your team at in Metrics?
The criteria above are what we check in a facilitated assessment. Run it yourself to place this capability, see which gates you have passed, and compare it against the other 3 in Delivery Management.
Start the assessment →