Capability 04 of 04 · Infrastructure
Observability & Feedback Loop
Monitoring agent behavior, costs, and outcomes to close the improvement loop.
5
maturity levels
15
practices
15
guides
L1 · Stage 01Assisted
Practices - what it looks like3 guides
- Basic loggingThe most primitive form of production visibility: unstructured print statements scattered wherever somebody once needed to debug something.guide→
- Alerting on errorsAlerting on errors is the practice of automatically notifying a human when something goes wrong in production - before a customer reports it.guide→
- Prod feedback and token-cost visibility not yet wired to dev"No connection: prod to dev feedback" describes the state where production incidents have no automatic path back to the developer or agent that caused them.guide→
Criteria - what to measure
- 01The application writes logs a developer can read
- 02Alerting fires on application errors
- 03Logs are searchable (centralized logging, not just local files)
- 04Production issues do not yet feed back into dev priorities
L2 · Stage 02Delegated
Practices - what it looks like3 guides
- Structured loggingStructured logging replaces free-form text log output with machine-parseable records - typically JSON - where every field has a defined name and type.guide→
- OpenTelemetry basicOpenTelemetry (OTel) is the open standard for collecting and exporting telemetry data - traces, metrics, and logs - from distributed systems.guide→
- Post-deploy monitoring; per-session token cost as table stakes; input tokens (context) drive spend - track them, not outputThe window right after a deploy becomes its own monitoring phase, with tighter thresholds on error rates and latency, instead of ship and move on.guide→
Criteria - what to measure
- 01Structured logging is implemented (JSON logs with consistent fields)
- 02The application emits traces and metrics, not only logs
- 03Post-deploy monitoring checks run after each deployment
- 04Traces are correlated across services
- 05Post-deploy checks include automated smoke tests
L3 · Stage 03Systematic
Practices - what it looks like3 guides
- Full observability stack (OTel + Grafana)A full observability stack means having all three telemetry pillars - metrics, traces, and logs - collected, correlated, and queryable in a unified system.guide→
- Production metrics → dashboards; agent telemetry through a governed gateway (self-hosted control plane: identity, policy, telemetry - Claude Apps Gateway model; catch shadow AI via proxy)Production metrics dashboards are the operational nerve center of a mature engineering team: real-time, continuously updated views into the health and behavior of every production service.guide→
- Incident data available for context; usage-truth reconciliation (client-reported tokens vs agent-claimed work)Past incidents, runbooks and metric baselines are reachable programmatically, rather than buried in Confluence pages, Slack threads and people's memories.guide→
Criteria - what to measure
- 01Traces, metrics and logs from production are queryable in one place
- 02Production metrics feed into dashboards accessible to all developers
- 03Incident data (post-mortems, error patterns) is available as agent context
- 04SLOs are defined and tracked for key services
- 05Incident data is structured for machine consumption (not just human-readable post-mortem docs)
L4 · Stage 04GovernedMost teams aim here
Practices - what it looks like3 guides
- Production anomaly → auto-ticket → agent investigation; incident rate and firefighting hours tracked against change volume, because the two move in opposite directionsThe production anomaly to auto-ticket to agent investigation pipeline automates the first phase of incident response, and tracks incident rate and firefighting hours against change volume, because the two move in opposite directions.guide→
- Self-healing basic: known patterns auto-fixed, with diagnosis kept human - agents are superhuman at observing and unreliable at orienting, where correlation gets mistaken for causationSelf-healing for known patterns means specific, well-understood failure conditions are remediated automatically, with diagnosis kept human, because agents are superhuman at observing and unreliable at orienting.guide→
- Infrastructure recommends code changes back to the team, and agent sessions are audited for quality regression over timeInfrastructure stops merely running the code and starts reading it: it watches production behavior and hands back concrete code changes to make.guide→
Criteria - what to measure
- 01Production anomaly detection auto-creates tickets and triggers agent investigation
- 02Self-healing for known patterns: agent detects known error pattern, applies known fix, deploys, and verifies
- 03Infrastructure recommends code changes based on production data (Vercel SDI model)
- 04Auto-created tickets include full context (traces, logs, affected users, similar past incidents)
- 05Self-healing success rate is tracked (% of auto-fixes that resolve the issue without human intervention)
L5 · Stage 05Self-improving
Practices - what it looks like3 guides
- Full production → agent loopThe full production-to-agent loop is the L5 realization of observability as an agent input channel.guide→
- Anomaly → investigate → fix → test → deploy autonomousEnd-to-end incident response with nobody in the loop: the anomaly fires, an agent finds the cause, writes the fix, tests it and ships it behind a canary.guide→
- Infrastructure self-drives: code defines infra, production informs code"Infrastructure self-drives" describes the fully realized bidirectional relationship between code and infrastructure at L5.guide→
Criteria - what to measure
- 01Full production-to-agent loop operates autonomously: anomaly detected, investigated, fixed, tested, deployed
- 02Infrastructure self-drives: code defines infrastructure, production performance informs code changes
- 03Anomaly-to-deploy cycle completes without human intervention for 80%+ of known issue categories
- 04Novel anomalies (not matching known patterns) are escalated to humans with full investigation context
- 05Mean time from anomaly detection to autonomous fix deployment is under 15 minutes
From the Field
Recent releases, projects and discussions the AI Radar classified into this capability.
Which level is your team at in Observability & Feedback Loop?
The criteria above are what we check in a facilitated assessment. Run it yourself to place this capability, see which gates you have passed, and compare it against the other 3 in Infrastructure.
Start the assessment →