↓ Skip to main content
  1. Runbook/
  2. Kubernetes/

Kubernetes Pod and Workload Troubleshooting

Kubernetes Pods Workloads Logs Troubleshooting

Time guide

  • Initial pod triage: approximately 5 to 10 minutes
  • Restart or scheduling investigation: approximately 15 to 30 minutes
  • Dependency or storage investigation: approximately 30 to 60 minutes

Purpose
#

Use this entry to investigate pods in Pending, Failed, Unknown, CrashLoopBackOff, ImagePullBackOff or repeated restart states.

Locate unhealthy workloads
#

kubectl get pods -A -o wide
kubectl get pods -A \
  --field-selector=status.phase!=Running,status.phase!=Succeeded

Review restart counts:

kubectl get pods -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,PHASE:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,NODE:.spec.nodeName'

Describe the pod
#

kubectl describe pod <pod-name> -n <namespace>

Pay attention to:

  • Container state and reason
  • Last termination state
  • Readiness and liveness probes
  • Volumes and mounts
  • Node assignment
  • Conditions
  • Events

Logs
#

Current logs:

kubectl logs <pod-name> \
  -n <namespace> \
  --all-containers=true \
  --tail=200

Previous container instance:

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --previous \
  --tail=200

Follow logs:

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --follow

Stop with Ctrl+C.

Events
#

kubectl get events -n <namespace> \
  --sort-by=.metadata.creationTimestamp
kubectl get events -n <namespace> \
  --field-selector=type=Warning \
  --sort-by=.metadata.creationTimestamp

Pending pods
#

Check scheduling details:

kubectl describe pod <pod-name> -n <namespace>

Review nodes and taints:

kubectl get nodes
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'

Review claims:

kubectl get pvc -n <namespace>

Common causes:

  • Insufficient CPU or memory
  • Node selectors or affinity
  • Taints without tolerations
  • Unbound persistent volume claims
  • Missing storage classes
  • Admission-policy rejection

CrashLoopBackOff
#

kubectl get pod <pod-name> -n <namespace> -o yaml
kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --previous \
  --tail=200

Look for:

  • Application configuration faults
  • Missing Secrets or ConfigMaps
  • Permission errors
  • Probe failures
  • OOMKilled
  • Failed mounts
  • Dependency failures

ImagePullBackOff
#

kubectl describe pod <pod-name> -n <namespace>

Check the configured image:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.spec.containers[*].image}{"\n"}'

Check:

  • Image name and tag
  • Registry availability
  • Architecture compatibility
  • Pull Secret references
  • Registry credentials
  • Rate limits

Workload controllers
#

kubectl get deployments,statefulsets,daemonsets -n <namespace>
kubectl describe deployment <deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>

rollout status is read-only and waits for status until completion or timeout.

Dependencies
#

kubectl get configmaps,secrets,services,pvc -n <namespace>

List Secret keys without values:

kubectl get secret <secret-name> \
  -n <namespace> \
  -o go-template='{{range $key, $value := .data}}{{$key}}{{"\n"}}{{end}}'

Safety notes
#

This entry excludes delete, restart, scale, patch and edit commands. Recreating a pod can temporarily hide evidence. Capture descriptions, logs and events before taking a state-changing action.

Related entries#

  • Kubernetes Cluster Health Checks
  • Kubernetes Node Health and Resource Pressure
  • Services and Ingress Troubleshooting
  • Persistent Volume Checks
  • Kubernetes Events and Diagnostics