Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

Start by identifying which layer is reporting failure. An Argo CD OutOfSync status means the live resources differ from the rendered target state; it does not prove that a Deployment rollout failed. A Deployment that is stuck, unavailable, or repeatedly restarting is a Kubernetes workload problem, even when GitOps reports the application as synced. Diagnose the reporting controller first, then verify the other layers.

Record the cluster, namespace, application, resource name, Git revision or image, promotion stage, and first-failure time. Compare those same fields with a known-good cluster. This separates a rendering or reconciliation problem from a cluster-specific quota, permission, image, admission, configuration, or probe problem.

Understand what “drift” and “failed promotion” actually mean

“Promotion failed” is an outcome, not a single Kubernetes condition. A promotion workflow can fail while rendering manifests, applying them, waiting for health, or deciding whether to advance a canary. The first useful question is: which component declared failure, and what evidence did it use?

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Signal What it means Where to investigate first
OutOfSync in Argo CD Argo CD calculates that live resources differ from the target state it rendered. Rendered manifests, live-versus-target diff, sync operation, and resource ownership.
Sync operation failed The controller could not complete an apply or another sync step. Operation error, affected resource, admission response, permissions, and retry settings.
Deployment is unavailable or stalled The Deployment controller cannot make the new ReplicaSet meet its progress requirements. Deployment conditions, ReplicaSet and Pod events, rollout status, probes, images, quota, and configuration.
Promotion controller is green but users see errors The promotion decision succeeded, but application readiness or runtime behavior may still be wrong. Service endpoints, Pod readiness, application health, and resource-level status.

Argo CD’s overview documentation defines an application whose live state deviates from its target state as OutOfSync. That is a state comparison, not a statement that the target is healthy. Conversely, Kubernetes reports Deployment progress through conditions and rollout status. Do not repeatedly synchronize an application when the actual problem is a failed Pod startup.

Capture a comparable incident baseline

Before changing anything, capture a small, time-stamped baseline from the failing cluster and a known-good cluster. Use the same namespace and resource names where possible. This prevents a later comparison from mixing different revisions or inputs.

  • Cluster context and Kubernetes release supported by that cluster.
  • Namespace, application, Deployment, ReplicaSet, Service, and canary resource names.
  • Git revision, chart or Kustomize revision, image digest or tag, and promotion stage.
  • Rendered desired manifests and the live-versus-target difference.
  • Argo CD sync status, operation result, health status, retry policy, and self-heal setting.
  • Deployment conditions, current and desired replica counts, rollout revision, and recent events.
  • Pod phase, container state, readiness and liveness results, image-pull messages, and termination reasons.
  • Quota, LimitRange, service-account permissions, admission decisions, image-pull access, and environment-specific configuration.
  • Time of the first error and whether another rollout or configuration change began during observation.

These comparison axes are an operational checklist, not a Kubernetes-defined cross-cluster standard. Their value is that they align the GitOps decision, Kubernetes controller state, and environment differences on one timeline.

Separate desired-state drift from a failed rollout

When Argo CD reports OutOfSync

Inspect the rendered target first. Confirm that the expected image, values, namespace, labels, replica counts, probes, and configuration data were actually produced for the failing cluster. Then inspect the live-versus-target diff. A difference can result from a failed apply, a legitimate controller mutation, an admission default, or another tool changing fields after synchronization.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Next inspect the sync operation itself. Identify the first resource that failed, the exact API error, and whether later resources were skipped. A successful-looking application page can still conceal an unhealthy child resource; open the resource tree and inspect the Deployment, ReplicaSet, Pods, Services, and any custom resources involved in promotion.

Check for ownership collisions

Argo CD uses an application instance label to identify resources belonging to an application. Its FAQ documents a failure mode in which another tool writes app.kubernetes.io/instance, replacing the value Argo CD expects. The resources can remain OutOfSync after a sync because the controller no longer associates them correctly.

When the diff and sync operation do not explain a persistent state, check who owns and mutates the relevant labels. If this collision is present, configure Argo CD to use a different application instance label key, such as argocd.argoproj.io/instance, according to the deployed Argo CD version and configuration method. Resynchronize only after the ownership key is consistent across the application and its resources.

When resources are synced but unhealthy

Stop treating synchronization as the next fix. If the target and live objects match but the Deployment has unavailable replicas, inspect Kubernetes conditions, events, and Pods. A sync can be completely successful while the application fails to start, fails readiness checks, or cannot pull its image.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Diagnose reconciliation, retries, and timing

Argo CD automated sync normally acts when an application is OutOfSync, but its documented behavior matters during an incident: a failed sync is not automatically attempted again for the same commit SHA and application parameters unless a retry policy is configured. A retry policy can apply exponential backoff. Self-heal, when enabled, addresses live-state drift; it is not a guarantee that a failed promotion or broken application will become healthy.

  1. Verify the revision and parameters Argo CD is evaluating. A promotion dashboard may show a newer revision while the application still points at an older one.
  2. Read the sync error and identify whether it is a rendering, authorization, admission, apply, or health-wait failure.
  3. Check the configured retry policy and its current attempt or backoff interval.
  4. Confirm that the desired revision actually changed if you expect a new automated attempt for the same application.
  5. Check whether self-heal is enabled and whether another controller is continuously changing the same field.

The stable Argo CD documentation describes a default reconciliation period of 120 seconds plus up to 60 seconds of jitter, for a maximum interval of three minutes. This is a configurable default, not a timing guarantee. During an incident, inspect the deployed Argo CD version and configuration rather than assuming that waiting exactly three minutes proves reconciliation is broken.

Investigate a Deployment that will not progress

Kubernetes identifies a stalled Deployment through conditions such as ProgressDeadlineExceeded or FailedCreate. The controller’s progressDeadlineSeconds setting controls when insufficient progress is marked as stalled; it does not make an unhealthy application healthy.

Read the rollout and conditions

Use the documented rollout command to observe progress:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
kubectl rollout status deployment/<name>

A nonzero result means the rollout has not succeeded. If another rollout might start while you are watching, pin the observation to the intended revision:

kubectl rollout status deployment/<name> --revision=N

Then inspect the Deployment conditions and events, followed by the newest ReplicaSet and its Pods. The condition identifies the controller’s conclusion; the events and container state usually explain why.

Check the common concrete causes

  • Insufficient quota: The namespace cannot create the requested number of Pods or requested CPU and memory. Compare the failing cluster’s quota usage and limits with the working cluster.
  • Readiness failure: Containers run but never become eligible for Service traffic. Verify probe path, port, host, headers, startup duration, credentials, and dependencies.
  • Image-pull failure: The tag or digest is unavailable, the registry cannot be reached, or the service account lacks pull credentials. Distinguish an unknown tag from a registry authentication or network error.
  • Insufficient permissions: The service account, admission policy, or identity used by the workload cannot perform a required operation.
  • LimitRange conflict: Namespace defaults or maximums can reject a Pod whose resource requests or limits are missing or outside policy.
  • Runtime misconfiguration: Environment variables, mounted configuration, secrets, arguments, certificates, feature flags, or external endpoints differ between clusters.
  • Scheduling or node constraints: Taints, tolerations, affinity, topology rules, or unavailable node capacity prevent placement even though the manifest applied.

Do not infer the cause from the Deployment name or image tag alone. The first useful evidence is normally the Pod event or container state associated with the newest ReplicaSet.

Use readiness and liveness probes as separate signals

Readiness answers whether a Pod should receive traffic. A Pod is not Ready when its node is not Ready, a readiness gate is false, or one or more containers are not ready. Kubernetes removes not-ready Pods from Service load balancers, so a rollout can have running containers but no usable endpoints.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Liveness answers whether a container should be restarted. A liveness failure does not merely remove a Pod from traffic; it can restart the process. The check should represent an unrecoverable failure, not temporary overload, a slow dependency, or a startup phase. An aggressive liveness probe can create a restart loop and make a promotion appear to fail across an otherwise healthy cluster.

Compare probe behavior between clusters

  • Confirm the exact probe type, path, port, scheme, timeout, period, failure threshold, and initial delay in the rendered manifest.
  • Check whether a startup probe or application warm-up time differs by environment.
  • Verify that the probe endpoint does not require credentials or a dependency unavailable during startup.
  • Look at probe-failure events and container restarts together; readiness failures and liveness restarts imply different remediation paths.
  • Confirm that a readiness gate supplied by another controller is present and becoming true in the failing cluster.

Do not equate a green promotion controller with a healthy application

A promotion controller may report that its decision or synchronization step completed while Kubernetes still has unavailable Pods, stale Service endpoints, or application-level errors. Treat the promotion result as one signal in a chain:

  1. Was the intended revision rendered?
  2. Was it synchronized to the intended cluster and namespace?
  3. Did the Deployment create the expected ReplicaSet and Pods?
  4. Did Pods become Ready and enter Service endpoints?
  5. Did application health checks and user-facing requests succeed?

Check the resource health and rollout state after a promotion reports success. If the controller’s health model differs from Kubernetes readiness, document that difference and use the stricter signal for traffic decisions.

Contain a failed canary without destroying evidence

When monitoring identifies a canary problem, protect stable capacity first. Scale the canary Deployment to zero rather than deleting it. This stops canary traffic and preserves its configuration, Pod template, events, and revision for inspection.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  1. Record the canary revision, image digest, rendered configuration, events, logs, and probe results.
  2. Scale the canary Deployment to zero using the normal operational change path.
  3. Restore stable replicas if the stable workload was reduced during the experiment or if capacity is at risk.
  4. Verify Service endpoints and error rates for the stable workload.
  5. Investigate the canary evidence before attempting another promotion; do not erase the only failing revision by applying an unrelated replacement.

Scaling to zero is containment, not a diagnosis. Keep the canary definition available until the evidence has been captured and the owning source has been corrected.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Roll back from the system that owns desired state

Directly managed Kubernetes Deployment

For a Deployment managed directly through Kubernetes, kubectl rollout undo can restore an earlier Pod-template revision:

kubectl rollout undo deployment/<name>

You can target a particular revision when the operational procedure requires it. Kubernetes creates revisions when the Pod template changes; scaling-only changes do not create a new revision. A rollback restores the Pod template, not every related object or external configuration. Recheck Services, ConfigMaps, Secrets, policies, and dependent resources after the rollback.

Argo CD with automated sync enabled

Argo CD documents that rollback cannot be performed against an application while automated sync is enabled. An imperative rollback made only in the live cluster can be overwritten by reconciliation. Correct or revert the declared configuration in Git through the normal deployment workflow, then confirm that Argo CD renders, synchronizes, and reports the workload healthy.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

This source-of-truth rule also applies to emergency changes: if you must make a temporary live-cluster change to protect availability, record it immediately and reconcile the declared configuration so the system does not silently return to the failed state.

Compare a working and failing cluster systematically

Comparison axis Questions to answer Evidence to capture
Target revision Are both clusters using the same commit, chart, values, and image digest? Git revision, rendered manifests, Helm/Kustomize inputs, image reference.
GitOps reconciliation Did both applications render and sync the same resources? Sync status, operation result, diff, retry policy, self-heal setting, timestamps.
Resource health Do both Deployments, ReplicaSets, Pods, and Services have the same conditions? Conditions, rollout revision, replica counts, Pod phases, endpoint membership.
Cluster policy and capacity Can the failing cluster admit, schedule, and run the same Pod? Quota, LimitRange, admission decisions, node capacity, taints, permissions.
Images and secrets Can the failing cluster pull the exact image and read required configuration? Pull events, service-account credentials, Secret and ConfigMap references, registry access.
Probe behavior Are the checks identical, and do dependencies respond the same way? Rendered probe settings, readiness gates, probe events, restart counts, endpoint changes.

If the rendered target differs, fix the promotion inputs or environment overlay. If the target matches but sync differs, investigate controller permissions, ownership, admission, or API availability. If sync matches but health differs, focus on the cluster and workload evidence rather than GitOps retry behavior.

A practical decision path for the next incident

  1. Name the reporter: Argo CD, a promotion controller, a Deployment, a Service monitor, or an application check.
  2. Freeze the comparison point: record cluster, namespace, revision, image, stage, and first-failure time.
  3. Check rendered versus live state: inspect the target, diff, sync operation, and ownership labels.
  4. Follow the resource tree: identify the first unhealthy Deployment, ReplicaSet, Pod, or custom resource.
  5. Read conditions and events: look specifically for ProgressDeadlineExceeded, FailedCreate, quota, image, permission, scheduling, and probe errors.
  6. Verify traffic eligibility: confirm readiness, readiness gates, Service endpoints, and stable capacity.
  7. Contain before experimenting: scale a failed canary to zero and restore stable replicas when necessary.
  8. Change the owner’s source: use Git for Argo CD-managed desired state; use a Kubernetes rollback only for a Deployment whose source is managed directly.
  9. Confirm convergence: wait for the configured reconciliation behavior, verify the intended revision, and recheck rollout and application health.

Prevent recurring cross-cluster promotion failures

  • Render manifests for every cluster overlay in CI and compare the resulting Pod template, probes, images, labels, and resource settings before promotion.
  • Make the target cluster, namespace, revision, and image digest visible in promotion records.
  • Alert separately on Argo CD drift, sync failure, Deployment unavailability, probe failures, and application errors.
  • Document which controller owns labels, replicas, image fields, and other mutable fields; avoid multiple tools writing the same ownership key.
  • Set retry and backoff policies deliberately, and expose the current attempt so operators do not mistake waiting for progress.
  • Test quota, image-pull credentials, admission policy, readiness gates, and probe behavior in each environment.
  • Keep rollback changes in the same source-of-truth workflow as forward changes.
  • Retain failed canary revisions, events, and rendered configuration long enough to compare them with the working cluster.

Use the Kubernetes and Argo CD documentation for the versions deployed in your environment when checking command flags, reconciliation defaults, rollback behavior, and controller-specific health semantics. Defaults and status behavior can change between releases.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.