↓ Skip to main content
  1. Runbook/
  2. Kubernetes/

Kubernetes Node Health and Resource Pressure

Kubernetes K3s Nodes Capacity Troubleshooting

Time guide

  • Initial node triage: approximately 5 to 10 minutes
  • Resource-pressure investigation: approximately 15 to 30 minutes
  • Host-level investigation: approximately 20 to 45 minutes

Purpose
#

Use this entry when a node is NotReady, workloads cannot schedule, resource pressure is suspected, or platform capacity needs review.

Cluster node summary
#

kubectl get nodes -o wide
kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,MEMORY:.status.conditions[?(@.type=="MemoryPressure")].status,DISK:.status.conditions[?(@.type=="DiskPressure")].status,PID:.status.conditions[?(@.type=="PIDPressure")].status,UNSCHEDULABLE:.spec.unschedulable'

Healthy nodes normally show Ready=True and all pressure conditions as False.

Capacity and allocatable resources
#

kubectl describe node <node-name>
kubectl get node <node-name> \
  -o jsonpath='{.status.capacity}{"\n"}{.status.allocatable}{"\n"}'
kubectl top nodes

If metrics are unavailable:

kubectl get apiservice v1beta1.metrics.k8s.io

Taints and scheduling
#

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'
kubectl get node <node-name> \
  -o jsonpath='{.spec.unschedulable}{"\n"}'

Review workloads assigned to a node:

kubectl get pods -A \
  --field-selector spec.nodeName=<node-name> \
  -o wide

Recent node events
#

kubectl get events -A \
  --field-selector involvedObject.kind=Node,involvedObject.name=<node-name> \
  --sort-by=.metadata.creationTimestamp

Host-level checks
#

Run on the affected node:

uptime
free -h
df -h
df -ih
systemctl --failed

For a k3s server:

sudo systemctl status k3s --no-pager --full
sudo journalctl -u k3s --since "30 minutes ago" --no-pager

For a k3s agent:

sudo systemctl status k3s-agent --no-pager --full
sudo journalctl -u k3s-agent --since "30 minutes ago" --no-pager

Check kernel warnings:

sudo journalctl -k -p warning --since "30 minutes ago" --no-pager

Common findings
#

MemoryPressure
#

Check memory and high-usage workloads:

free -h
kubectl top pods -A --sort-by=memory

DiskPressure
#

df -h
df -ih
sudo du -xhd1 /var/lib 2>/dev/null | sort -h

The du command can take time on large filesystems but is read-only.

PIDPressure
#

ps -e --no-headers | wc -l
ps aux --sort=-%cpu | head

NodeNotReady
#

Check:

  • Network connectivity
  • k3s service state
  • Disk availability
  • Memory pressure
  • Time synchronisation
  • Container runtime logs
  • Recent kernel faults

Safety notes
#

This entry intentionally excludes cordon, drain, restart and reboot commands. Those actions can move or interrupt workloads and should only follow diagnosis, capacity review and a rollback plan.

Related entries#

  • Kubernetes Cluster Health Checks
  • Pod and Workload Troubleshooting
  • Kubernetes Resource Requests and Limits
  • k3s Server and Agent Administration