Kubernetes forensics

A pod is in CrashLoopBackOff. Walk me through it.

kubectl describe pod api-7d9f -n prod          # Events + Last State + exit code
kubectl logs api-7d9f -n prod --previous       # the CRASHED container, not the restarting one
kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

--previous is the whole trick: the current container may have just started and have no useful output. The exit code narrows it immediately:

• 1 / 2 — the application threw on startup. Almost always config: a missing environment variable, an unreachable database, a bad migration, a malformed secret.
• 137 — SIGKILL, which is 128 + 9. Either OOMKilled (check State.Reason) or a liveness probe killing it.
• 139 — segfault, 128 + 11. Native dependency or architecture mismatch (an amd64 image on arm64 nodes, or the reverse).
• 143 — SIGTERM, 128 + 15; something asked it to stop.
• 0 — the process completed successfully and exited. A Deployment expects a long-running process; if your command runs and returns, Kubernetes restarts it forever. That should be a Job.

The commonest real causes, roughly in order: a config or secret that does not exist in this namespace; a dependency not ready at startup with no retry (the app should retry, not assume ordering); a liveness probe that is too aggressive — its initialDelaySeconds is shorter than the app's real startup time, so kubelet kills it mid-boot, forever; and an image built for the wrong architecture.

A pod shows OOMKilled. The app's heap looks fine. What is happening?

OOMKilled means the cgroup exceeded resources.limits.memory and the kernel killed the largest process. Causes beyond an actual leak:

1. The runtime does not know about the limit. A JVM without -XX:MaxRAMPercentage (older JVMs) sizes its heap from the host's memory and happily grows past a 512 MiB container limit. Node needs --max-old-space-size below the limit. Go needs GOMEMLIMIT. Python has no cap at all. The runtime never gets to run a full GC because the kernel kills it first.
2. Non-heap memory: thread stacks, native allocations, memory-mapped files, buffer pools, glibc arena fragmentation in multi-threaded services (often fixed by MALLOC_ARENA_MAX=2 or jemalloc).
3. Page cache and tmpfs counted against the cgroup — writing large temp files to an emptyDir backed by memory consumes the limit.
4. Sidecars. The limit is per container, but the pod's total is what gets scheduled; a logging or mesh sidecar with a low limit can be the one that dies while your app looks fine.
5. The limit was simply set from a guess and the app's real working set is larger under production traffic.

kubectl describe pod x | grep -A5 "Last State"     # Reason: OOMKilled, exit 137
kubectl top pod x --containers                     # current usage per container

A pod is stuck in Pending. Nothing is wrong with the image.

Pending means no node has been selected. kubectl describe pod prints the scheduler's exact reason under Events — read it rather than guessing:

• Insufficient cpu / Insufficient memory — no node has enough unreserved capacity for your requests. Note it is requests, not current usage: a cluster at 30% actual utilisation can be 100% requested. The fix is right-sizing requests, adding nodes, or a cluster autoscaler.
• node(s) had untolerated taint — nodes are tainted (control plane, GPU pool, a node being drained) and your pod has no matching toleration.
• node(s) didn't match Pod's node affinity/selector — a nodeSelector for a label or zone that does not exist any more.
• pod has unbound immediate PersistentVolumeClaims — no PV matches, the StorageClass is missing, or — very commonly — the PVC is in a different availability zone than the only node with capacity. Block storage is zonal, so the pod must schedule where the volume is.
• too many pods — the node's pod limit (110 by default) or IP exhaustion in the subnet.
• 0/5 nodes are available: 5 node(s) didn't match pod topology spread constraints — your own anti-affinity rules cannot be satisfied, common when replicas exceed available zones.

kubectl describe pod x | tail -20
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
kubectl describe node <node> | grep -A8 "Allocated resources"

A rollout has been "progressing" for twenty minutes and never completes.

kubectl rollout status deploy/api -n prod
kubectl get rs -n prod                       # old and new ReplicaSets, desired vs ready
kubectl describe deploy api -n prod          # conditions: Progressing / Available
kubectl get pods -l app=api -o wide          # which pods are not Ready

The mechanism: with maxSurge/maxUnavailable, the Deployment creates new pods and waits for them to become Ready before scaling down old ones. If the new pods never pass their readiness probe, the rollout parks there. Causes:

• The readiness probe fails — wrong path or port, the app binds 127.0.0.1 rather than 0.0.0.0, TLS expected on the probe port, or the probe hits a route requiring auth.
• The new pods are Pending — no capacity for the surge, which is why maxSurge: 25% on a tight cluster can deadlock a rollout.
• A PodDisruptionBudget prevents scaling down the old ReplicaSet.
• The image is wrong — ImagePullBackOff on the new ReplicaSet only.
• progressDeadlineSeconds (default 600) eventually marks it failed, but Kubernetes does not roll back automatically — someone must.

kubectl rollout undo deploy/api reverts to the previous ReplicaSet. Do that first if users are affected, then diagnose. And note the failure mode that matters for correctness: during a rolling update, old and new code run simultaneously, so the schema and the API contract must be compatible with both.

ImagePullBackOff in production, but the image pulls fine on my laptop.

kubectl describe pod gives the specific failure:

• unauthorized / denied — the node cannot authenticate. Either imagePullSecrets is missing from the pod spec or the service account, the secret is in a different namespace (they are namespaced), or the cloud IAM role attached to the node group lacks registry read permission. Your laptop works because you have a personal docker login.
• manifest unknown / not found — the tag does not exist. Usually CI pushed to a different tag, or the deployment references :latest while CI pushes SHA tags.
• no match for platform — an arm64 image (built on an Apple Silicon laptop) on amd64 nodes. Build multi-arch with docker buildx --platform linux/amd64,linux/arm64.
• Timeouts — a private registry unreachable from the node subnet, or missing egress/NAT. This is a networking problem wearing a registry costume.
• Rate limits — anonymous Docker Hub pulls are throttled per IP, so a large cluster behind one NAT gateway hits the limit and pulls fail intermittently. Mirror to a private registry.

A Service returns nothing, but the pods are healthy and I can curl them directly by pod IP.

A Service selects pods by label selector and maintains an Endpoints/EndpointSlice object. If that object is empty, traffic goes nowhere and you get a connection refused or a timeout with no useful error.

kubectl get endpoints my-svc -n prod          # EMPTY is the answer 80% of the time
kubectl get pods -n prod --show-labels
kubectl describe svc my-svc -n prod           # compare Selector to the pods' labels

Causes, in order of frequency:

1. Selector/label mismatch — a typo, or the Deployment's pod template labels were changed without updating the Service. Endpoints is empty.
2. Pods are not Ready. Only Ready pods are added to endpoints, by design. So a failing readiness probe presents as "the service is down" while the pods look fine in get pods (until you read the READY column).
3. targetPort is wrong — the Service forwards to a port the container is not listening on. port is the Service's port; targetPort is the container's. Named ports avoid this.
4. The app binds 127.0.0.1 inside the container, so it is unreachable from outside the pod's loopback. Must bind 0.0.0.0.
5. A NetworkPolicy denies the traffic — especially in a namespace with a default-deny policy where someone added an ingress rule but no egress rule on the caller's side.
6. DNS: the client resolved my-svc in the wrong namespace. Cross-namespace needs my-svc.other-ns.svc.cluster.local.

kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash
# then: nslookup my-svc.prod.svc.cluster.local ; curl -v my-svc.prod:8080