How to Debug a Kubernetes Deployment Rollout

Start with the Deployment status

kubectl rollout status deployment/api -n production
kubectl get deployment api -n production
kubectl describe deployment api -n production

Compare desired replicas with updated, ready, available, and unavailable counts. A Deployment can have running pods without enough available replicas. Conditions such as Progressing and Available, plus their reasons and messages, tell you what the controller currently believes.

ProgressDeadlineExceeded means progress did not complete within the configured deadline. It does not identify the failing container or setting, so continue down to the new ReplicaSet and pods.

Find the new ReplicaSet

List ReplicaSets owned by the Deployment and sort or inspect their creation time and revision. During a rolling update, old and new sets can coexist. The new set may have zero ready replicas while the old set continues serving traffic.

kubectl get rs -n production -l app=api
kubectl rollout history deployment/api -n production

Labels are convenient but not always unique enough; owner references provide the actual controller relationship. Compare the pod template, image, environment sources, service account, volumes, and probes between the working and failing revisions.

Inspect pods before changing the Deployment

Pending pods usually point toward scheduling, quota, affinity, taints, volume binding, or image retrieval. Running but unready pods point toward readiness checks or application health. Restarting pods require container state, previous logs, and probe analysis.

Read pod events in time order. Messages such as FailedScheduling, FailedMount, ErrImagePull, ImagePullBackOff, and Unhealthy narrow the search quickly. Check whether all new pods fail in the same way or only those on one node.

Check image identity and pull behaviour

Verify registry, repository, tag, and ideally digest. A mutable tag such as latest makes it harder to know what is running and can behave differently with image pull policy and node caches. A digest gives an exact image identity.

Image pull failures can come from a missing image, wrong architecture, registry outage, rate limit, DNS issue, or bad image-pull credentials. Do not restart healthy old replicas until the new image can be pulled and started.

Separate startup from readiness

A container can run while the pod remains unready. Inspect readiness failures and test the endpoint inside the relevant network context. The application may listen on the wrong port, return the wrong status, require more startup time, or depend on a service that is unavailable.

Use a startup probe for legitimately slow initialization so liveness does not kill the process too early. Avoid making readiness depend on every downstream service; one failing dependency can remove all replicas and amplify an outage.

Understand the rollout strategy

maxSurge controls how many extra pods may be created during a rolling update. maxUnavailable controls how many desired pods may be unavailable. Cluster capacity, quota, anti-affinity, and disruption rules can prevent the surge pods from scheduling. A strategy that works in a spacious staging cluster may stall in a full production cluster.

Check termination grace periods and readiness timing too. A rollout can appear slow because old pods take a long time to exit or new pods need a long minimum ready period.

Rollback is a risk decision, not a reflex

If the previous revision is known to be compatible with current database schemas, configuration, and external dependencies, a rollback may restore service quickly. If the release ran irreversible migrations or changed a shared contract, blindly undoing the Deployment can make things worse. Follow the release runbook and capture evidence first.

kubectl rollout history deployment/api -n production
kubectl rollout undo deployment/api -n production --to-revision=12

Close the loop

After the fix, watch until the rollout completes. Confirm stable readiness, expected replica counts, flat restart counts, correct image digests, and normal application metrics. Then record the root cause and add a test or deployment check that would catch it earlier.

Clusterdeck exposes workload status, ReplicaSets, pods, events, YAML, rollout history, live logs, and rollout actions in connected views. It is useful for following the controller chain; keep durable configuration changes in the source of truth used by your team.