Canary/progressive deployment auto

Automated canary and progressive deployment is the practice of rolling out changes to a small percentage of production traffic first, automatically monitoring key metrics during the rollout, and either automatically promoting to full traffic (if metrics look good) or automatically rolling back (if metrics degrade). The entire lifecycle - from "deploy to 5%" to "promoted to 100%" or "rolled back to previous" - is automated without human intervention in the happy path.

Manual canary deployments existed before AI-assisted development. The L4 upgrade is the automation: removing the human who watches the canary and makes the promotion/rollback decision. At 50+ PRs/day, manually watching each canary deployment is not feasible. Automated analysis - comparing error rates, latency distributions, and key business metrics between canary and baseline using statistical analysis - makes the promotion/rollback decision reliably without human attention.

Flagger (built on top of Kubernetes) and Argo Rollouts are the primary implementations. Both support analysis templates: define the metrics to compare, the comparison method (Prometheus queries, Datadog metrics, custom HTTP checks), and the thresholds. The operator handles the traffic splitting, metric collection, comparison, and promotion/rollback. Spinnaker's automated canary analysis (ACA) provides similar capability for non-Kubernetes environments.

At L4, progressive deployment is the default path for all production changes. The deployment doesn't go to 100% of traffic immediately - it starts at 5-10%, waits for the analysis window (typically 10-15 minutes), and advances to 25%, 50%, then 100% if each phase passes. This staged approach limits blast radius for every deployment while maintaining deployment frequency.

• Limits blast radius for every deployment - at 50+ PRs/day, some percentage of deployments will introduce regressions; progressive rollout limits each regression to 5-10% of users for the first 10-15 minutes, dramatically reducing incident scope
• Eliminates manual canary watching - manually monitoring a canary at high deployment frequency is a full-time job; automated analysis removes this overhead while providing more rigorous and consistent evaluation than human monitoring
• Enables safe auto-merge → auto-deploy - the green = auto-merge → auto-deploy pattern is only safe with automated progressive deployment as the deployment strategy; full-traffic auto-deploy without canary is reckless at L4 volume
• Generates deployment performance data - every canary analysis run produces a record of which metrics were compared, what values were observed, and what the decision was; this data makes deployment patterns visible and improvable
• Supports agent-driven development at scale - agents that produce PRs need deployment infrastructure that can handle their output safely; automated canary is the deployment safety net that makes autonomous agent workflows viable

1. Instrument your key metrics in Prometheus or Datadog - canary analysis requires observable metrics. At minimum: HTTP error rate (5xx responses), p50/p95/p99 latency, and 1-3 business-level metrics (checkout conversion, login success rate, API success rate). These must be queryable by the canary analysis tool.
2. Install Flagger or Argo Rollouts - if using Kubernetes, Flagger integrates with your service mesh (Istio, Linkerd) or ingress (NGINX, Traefik) for traffic splitting. Install Flagger with your preferred provider and configure it for your cluster.
3. Define a canary analysis template - create a Flagger MetricTemplate or Argo Rollouts AnalysisTemplate that defines: which metrics to compare, what thresholds indicate regression, and how long the analysis window should be. Start with error rate (fail if canary error rate > 2x baseline) and latency (fail if p99 latency > 20% above baseline).
4. Configure your first canary rollout - annotate one service for canary deployment. Start with a conservative schedule: 5% traffic for 10 minutes, then 25% for 10 minutes, then 50% for 10 minutes, then 100%. Use maxSurge and maxUnavailable settings to control pod counts during rollout.
5. Test rollback explicitly - deploy a known-bad version (one that generates 5xx errors at a 5% rate) and verify that Flagger detects the error rate increase and automatically rolls back within the configured window. Rollback must be tested before it's needed.
6. Add deployment events to your team dashboard - canary promotions, rollbacks, and in-progress deployments should be visible in a shared dashboard. "Canary at 25%, metrics nominal" tells the team that an auto-deploy is in progress without requiring anyone to watch it actively.

Using canary analysis with insufficient traffic. Canary analysis comparing 5% traffic to 95% baseline requires statistically significant traffic volume. If your service handles 100 requests/hour, 5% is 5 requests/hour - not enough to detect a 2% error rate increase with confidence. For low-traffic services, use blue-green deployment (100% traffic switch with instant rollback) rather than canary, or use a minimum request count threshold before canary promotion.

Analysis templates that are too strict or too lenient. A template that fails on any 1% error rate increase will cause constant rollbacks from normal production noise. A template that requires 10% error rate increase won't catch real regressions. Calibrate thresholds by analyzing your current production metric variance: a "failure" threshold should be outside normal variance by 2-3 standard deviations. Start lenient and tighten over time.

Not accounting for time-based traffic patterns. A canary deployed at 3am Sunday sees different traffic (and different baseline behavior) than one deployed at 2pm Tuesday. Some analysis templates use fixed numeric thresholds that become inappropriate at low-traffic times. Use relative thresholds (canary error rate vs. concurrent baseline error rate) rather than absolute thresholds to handle traffic pattern variance.

Canary that can't actually roll back. Some services have one-way state changes: database migrations, event streams, feature flags that once flipped can't be flipped back cleanly. Canary + automated rollback only works for stateless deployments or deployments with backward-compatible state changes. For breaking changes, progressive deployment must be decoupled from feature activation (feature flags) and use a blue-green strategy instead.

Forgetting about canary in your monitoring. When a canary is running, your monitoring shows mixed signals: some users are on the new version, some are on the old. Alerts that fire on aggregate error rate can be triggered by canary issues that are already being handled by rollback automation. Ensure your monitoring is canary-aware: distinguish between "canary pod errors" (handled by automation) and "production pod errors" (requires human response).