ClusterDeck
1. Identify the container that is failing
A pod can contain several containers, and only one may be crashing. Start with the pod detail rather than a broad cluster restart. Look at each container's current state, last state, restart count, reason, and exit code. The waiting reason may say CrashLoopBackOff, while the last terminated state gives the more useful reason such as Error, OOMKilled, or Completed.
kubectl get pod api-7d9c8f6c9b-k2x4m -n production
kubectl describe pod api-7d9c8f6c9b-k2x4m -n productionIf a container exits cleanly with code 0 but the pod expects it to stay running, check the image entrypoint, command, and workload type. A short batch task may belong in a Job rather than a Deployment.
2. Read events in chronological context
The Events section can reveal failed mounts, missing Secrets, image pull failures, probe failures, and scheduling trouble. Read the timestamps and repetition count. An event saying a liveness probe failed after the process started tells a different story from a container that exits before any probe runs.
Events expire and can repeat in compressed form, so capture important evidence during the incident. Also inspect the owning Deployment or StatefulSet. The pod is usually a replaceable result of a higher-level workload, not the configuration you should edit directly.
3. Read the previous container log
When a container restarts, the current log may contain only the new startup attempt. Kubernetes can expose the log from the preceding instance with --previous. Always specify the container in a multi-container pod.
kubectl logs api-7d9c8f6c9b-k2x4m -n production -c api --previous
kubectl logs api-7d9c8f6c9b-k2x4m -n production -c api --tail=200Look for the first meaningful error, not only the final stack trace. A missing environment variable, refused database connection, migration failure, invalid flag, or unreadable file often appears earlier.
4. Treat OOMKilled as a measurement problem
OOMKilled normally means the container exceeded its memory limit. Compare current and historical memory use if metrics are available. Check whether the limit is unrealistic, whether a new release changed memory behaviour, or whether the process has an unbounded cache or leak. Raising the limit can restore service, but it should not end the investigation.
Memory requests affect scheduling; memory limits constrain the container. CPU limits behave differently and can cause throttling rather than an out-of-memory termination. Do not copy a memory fix into CPU settings without understanding the resource.
5. Check configuration and mounted data
Compare the pod spec with a healthy revision. Verify environment variables, Secret and ConfigMap names, volume mounts, file paths, service account, image tag or digest, and the effective command and arguments. A ConfigMap can exist while containing the wrong key. A Secret can mount successfully while its value is stale.
A pod shell is useful only if the container stays alive long enough. If it does not, use an ephemeral debug container or create a copy with a changed command, following your organisation's incident and security procedures.
6. Review startup, readiness, and liveness probes
A liveness probe restarts a container when Kubernetes considers it unhealthy. A readiness probe removes it from Service endpoints without restarting it. A startup probe delays liveness and readiness checks until slow initialization succeeds. If a healthy but slow application is killed during startup, changing the liveness path alone may hide the real timing problem.
Inspect the path, port, scheme, headers, initial delay, period, timeout, and failure threshold. Then test the same endpoint from the relevant network context. Confirm that the probe checks process health rather than a fragile downstream dependency.
7. Fix the owning configuration and watch the rollout
Make the durable correction in the Deployment manifest, Helm values, Kustomize overlay, or GitOps source. Watch the new ReplicaSet and confirm that readiness stabilizes and restart counts stay flat. Avoid repeatedly deleting the pod: its controller will recreate the same broken configuration, and deletion can remove useful evidence.
Clusterdeck puts pod state, events, YAML, descriptions, container logs, and owning workload navigation in one Mac workspace. It can shorten the investigation, but the same diagnostic order still matters: state, events, previous logs, configuration, then a controlled fix.