iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Testing in production can reveal problems that preproduction tests miss, but it exposes real users to the change. Reduce that risk by keeping automated checks, releasing to a small and representative cohort, comparing its user-visible health with a control, and expanding only when the evidence supports it. Feature flags can control whether a feature is exposed; canary deployment controls which version receives traffic. Neither makes a release safe on its own.
What testing in production means—and what it does not
Testing in production means evaluating a change against real workloads after it has reached the live environment. A canary is a partial, time-limited deployment: a limited share of traffic receives the change while the rest of the service provides a control for comparison. Real inputs can reveal issues that unit tests or load tests did not exercise, but users in the canary cohort can be affected if the change fails. Google SRE’s canarying guidance treats canarying as an early deployment safety step, not a substitute for preproduction testing.
Continuous delivery and continuous deployment are related but distinct. DORA defines continuous delivery as keeping changes deployable so the team can release them quickly, safely, and sustainably when needed. Continuous deployment goes further: the practice is to try to deploy each code change to production as soon as possible. Continuous delivery does not require automatically exposing every change to all users.
How feature flags and canary deployment work together
A feature flag separates releasing code from exposing the feature. A binary can contain a feature that remains disabled until the team enables it for a chosen cohort or under specified conditions. If the feature behaves unexpectedly, the team may be able to disable it without waiting for a new binary deployment. That capability depends on the system’s flag design and operational setup; a flag is not a substitute for testing or monitoring.
Canary deployment controls the version or traffic receiving a change. A flag controls feature exposure. These controls can be combined: deploy a new version to a limited share of traffic, keep a feature disabled for most users, then enable it for a selected cohort while comparing outcomes. Keeping the two controls conceptually separate helps teams diagnose whether a problem comes from the new version, the feature itself, or both. Google SRE describes flags as a way to decouple feature launch from binary release.
A production canary process that can inform a decision
- Keep preproduction safeguards in the pipeline. Run automated tests and security checks, and verify the build and deployment process before exposing a change to production. DORA’s continuous delivery capability guidance includes test automation, deployment automation, comprehensive testing, and observability as relevant practices.
- Make the change small and containable. Deploy a self-contained change behind a feature flag or to a limited traffic cohort. Use a flag to control feature exposure independently of the binary release where the system supports it.
- Choose the control and success criteria before rollout. Compare the canary with a control group and select a small set of signals connected to user-visible service health. Error rate and latency are useful examples in Google SRE’s guidance. Set decision criteria before deployment; merely shifting some traffic does not establish whether the release is healthy.
- Give the canary a representative sample. Allow enough traffic and time to encounter relevant workloads and user behavior. There is no universal safe percentage or duration: traffic volume, workload diversity, metric behavior, and release cadence all affect what evidence is meaningful.
- Pause or reverse when signals deteriorate. Investigate adverse changes, disable the feature if that is the appropriate containment action, or roll back the release. Make sure the chosen recovery path is operationally safe. Promote traffic in stages only when the observed evidence supports doing so; Google Cloud’s deployment strategies describe progressive traffic shifting with rollback and cancellation controls.
- Keep monitoring after promotion. Some defects emerge only after a delay or under particular circumstances. Continue watching production health after the rollout, as advised in Google Cloud’s approach to change.
Choose signals that can distinguish a release problem
Use metrics that matter to users and can be compared meaningfully between the canary and control. Errors and latency are common examples, but a metric is useful only if it reflects the service’s health and the comparison is not confounded by unrelated differences between the groups. For example, if the canary cohort handles a different workload mix, a change in latency may not be attributable to the release.
Thresholds need judgment: a gate that is too strict can block healthy changes, while one that is too loose can miss a problem. Define what change warrants pausing, investigate the context when a signal moves, and avoid treating a single metric as proof that the rollout is safe. A canary provides evidence, not a guarantee.
Separate feature exposure decisions from traffic rollout decisions
When choosing an implementation, assess feature flags and deployment controls as separate layers. The relevant trade-offs are:
- Feature exposure: whether flag rules can target the intended users or cohort and whether that cohort represents the behavior you need to evaluate.
- Version and traffic control: whether the deployment method can route a limited share of traffic to the new version and promote it progressively.
- Evaluation quality: whether the chosen signals reflect user-visible health and allow a fair comparison with the control.
- Recovery: how quickly and safely the team can disable the feature or roll back the release.
- Operational fit: whether the team can diagnose results and manage the added complexity within its platform and runtime.
Google Cloud Deploy documents progressive canary traffic splitting for Cloud Run. On a first deployment to a target, it skips canary phases because there is no existing version with which to split traffic. This is a platform-specific example, not a universal requirement or a recommendation that every team use the same deployment product.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Why production exposure still requires caution
A canary’s defining strength is also its risk: it handles real user traffic. Adrian Hilton, a Customer Reliability Engineer in SRE, wrote in a Google Cloud Blog post published March 31, 2017: “The fact that the canary handles real user traffic is key: if it breaks, real users get affected, so canarying should be the first step in your deployment process, as opposed to the last step in testing.” That is why a production canary belongs after automated safeguards and before broad exposure, with monitoring and a practiced recovery path in place.
Quick Recap
Best Value
Rank #4
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
