Kubernetes Cluster Health Checks

Time guide
Initial cluster triage: approximately 10 to 20 minutes
Standard health review: approximately 30 minutes
Full platform, storage and host review: approximately 30 to 60 minutes
The actual duration depends on cluster size, access availability and whether any checks identify faults that need further investigation.

Purpose
#

This Runbook entry provides a structured set of read-only checks for reviewing the health of a Kubernetes or k3s cluster.

The checks move from the broadest cluster-level view toward individual nodes, workloads, services, storage and k3s host services.

The commands are intended for:

Routine platform checks
Initial incident triage
Investigating failed workloads
Checking node availability
Reviewing recent warning events
Confirming storage and networking objects
Gathering information before making any changes

These commands are primarily read-only. Commands that could generate a large amount of output or require access to a cluster node are clearly identified.

Requirements
#

A working kubectl installation
A valid kubeconfig
Network access to the Kubernetes API server
Permission to read the required cluster resources
SSH access to a k3s node for host-level service checks
sudo access for systemd journal checks where required

Confirm the current kubectl client and server versions:

kubectl version

Confirm the active context:

kubectl config current-context

List all configured contexts:

kubectl config get-contexts

Display the selected cluster endpoint without showing credential contents:

kubectl config view --minify \
  -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

Treat kubeconfig files as sensitive. A kubeconfig may contain credentials or references that provide access to a cluster. Do not publish the file or paste its contents into a public Runbook.

Quick cluster overview
#

Begin with the main cluster endpoints and services:

kubectl cluster-info

Check whether the API server responds:

kubectl get --raw='/readyz?verbose'

Check the API server live endpoint:

kubectl get --raw='/livez?verbose'

A healthy API server should return successful checks rather than connection, authentication or internal server errors.

Display general cluster information:

kubectl get componentstatuses

componentstatuses is deprecated in newer Kubernetes releases and may not provide useful results on every cluster. Prefer the API readiness endpoints and platform monitoring where available.

Node health
#

List all nodes with their status, roles, age and Kubernetes version:

kubectl get nodes -o wide

Expected result:

All expected nodes are present
Each node reports Ready
Internal addresses are correct
Kubernetes versions are within the expected range
No unexpected node has joined the cluster

Display node resource usage:

kubectl top nodes

This command requires Metrics Server or another compatible metrics API.

If kubectl top nodes fails, check whether the metrics API is available:

kubectl get apiservice v1beta1.metrics.k8s.io

Describe a particular node:

kubectl describe node <node-name>

Review the conditions of every node:

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,MEMORY:.status.conditions[?(@.type=="MemoryPressure")].status,DISK:.status.conditions[?(@.type=="DiskPressure")].status,PID:.status.conditions[?(@.type=="PIDPressure")].status'

Healthy nodes should normally show:

READY   True
MEMORY  False
DISK    False
PID     False

List node taints:

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'

List nodes that are currently cordoned:

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,UNSCHEDULABLE:.spec.unschedulable'

A value of true means the node is marked unschedulable.

Workload overview
#

List all pods across every namespace:

kubectl get pods -A -o wide

List pods that are not currently in the Running or Succeeded phases:

kubectl get pods -A \
  --field-selector=status.phase!=Running,status.phase!=Succeeded

This filter is useful for finding pods in phases such as:

Pending
Failed
Unknown

Some container-level states such as CrashLoopBackOff may still appear inside a pod whose overall phase is Running. For a broader visual check, use:

kubectl get pods -A

List deployments:

kubectl get deployments -A

List StatefulSets:

kubectl get statefulsets -A

List DaemonSets:

kubectl get daemonsets -A

List jobs:

kubectl get jobs -A

List CronJobs:

kubectl get cronjobs -A

A healthy result should generally show:

Expected replicas are available
StatefulSet ready counts match desired counts
DaemonSet desired and ready counts match
Jobs are completing successfully
CronJobs are not unexpectedly suspended

Failed and restarting containers
#

List pod restart counts:

kubectl get pods -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount'

Sort pods by creation time:

kubectl get pods -A \
  --sort-by=.metadata.creationTimestamp

Describe a problem pod:

kubectl describe pod <pod-name> -n <namespace>

Review current logs from all containers in a pod:

kubectl logs <pod-name> \
  -n <namespace> \
  --all-containers=true \
  --tail=200

Review the logs from the previous instance of a restarted container:

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --previous \
  --tail=200

Follow current logs:

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --follow

Press Ctrl+C to stop following logs.

Recent events
#

List events across all namespaces in creation order:

kubectl get events -A \
  --sort-by=.metadata.creationTimestamp

Show warning events only:

kubectl get events -A \
  --field-selector=type=Warning \
  --sort-by=.metadata.creationTimestamp

Show events in a particular namespace:

kubectl get events \
  -n <namespace> \
  --sort-by=.metadata.creationTimestamp

Common event reasons to investigate include:

FailedScheduling
FailedMount
FailedAttachVolume
FailedCreate
FailedPull
BackOff
Unhealthy
Evicted
NodeNotReady

Events are retained for a limited period and should not be treated as a permanent audit history.

Resource usage
#

Display pod CPU and memory usage across all namespaces:

kubectl top pods -A

Sort pod usage by CPU:

kubectl top pods -A \
  --sort-by=cpu

Sort pod usage by memory:

kubectl top pods -A \
  --sort-by=memory

Display usage for all containers:

kubectl top pods -A \
  --containers

Check resource requests and limits:

kubectl get pods -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,CPU_REQUESTS:.spec.containers[*].resources.requests.cpu,CPU_LIMITS:.spec.containers[*].resources.limits.cpu,MEMORY_REQUESTS:.spec.containers[*].resources.requests.memory,MEMORY_LIMITS:.spec.containers[*].resources.limits.memory'

Missing requests or limits are not automatically a fault, but the result can help explain scheduling, contention and capacity problems.

Services and ingress
#

List services:

kubectl get services -A

List endpoint slices:

kubectl get endpointslices -A

List ingress resources:

kubectl get ingress -A

Describe a particular service:

kubectl describe service <service-name> \
  -n <namespace>

Check the selector used by a service:

kubectl get service <service-name> \
  -n <namespace> \
  -o jsonpath='{.spec.selector}{"\n"}'

List the pods matching a known selector:

kubectl get pods \
  -n <namespace> \
  -l 'app=<application-label>' \
  -o wide

If a service has no ready endpoints, check:

Service selectors
Pod labels
Pod readiness
Container ports
Service targetPort
Network policies
Application listening addresses

DNS checks
#

Check the CoreDNS deployment:

kubectl get deployment coredns \
  -n kube-system

List CoreDNS pods:

kubectl get pods \
  -n kube-system \
  -l k8s-app=kube-dns \
  -o wide

Review CoreDNS logs:

kubectl logs \
  -n kube-system \
  -l k8s-app=kube-dns \
  --tail=200

Display the cluster DNS service:

kubectl get service kube-dns \
  -n kube-system

If DNS resolution fails inside workloads, also inspect:

The pod’s /etc/resolv.conf
CoreDNS pod health
CoreDNS ConfigMap
Upstream resolvers
Network policies
Node resolver configuration

Persistent storage
#

List persistent volumes:

kubectl get persistentvolumes

List persistent volume claims:

kubectl get persistentvolumeclaims -A

List storage classes:

kubectl get storageclasses

List volume attachments:

kubectl get volumeattachments

Find claims that are not bound:

kubectl get persistentvolumeclaims -A \
  --field-selector=status.phase!=Bound

Describe a problem claim:

kubectl describe persistentvolumeclaim <claim-name> \
  -n <namespace>

Describe the related persistent volume:

kubectl describe persistentvolume <volume-name>

Storage issues may appear as:

Pending
FailedMount
FailedAttachVolume
Multi-Attach error
Volume not ready
Insufficient storage

Longhorn checks
#

List Longhorn namespaces:

kubectl get namespaces | grep longhorn

Check Longhorn workloads:

kubectl get pods \
  -n longhorn-system \
  -o wide

List Longhorn volumes:

kubectl get volumes.longhorn.io \
  -n longhorn-system

List Longhorn replicas:

kubectl get replicas.longhorn.io \
  -n longhorn-system

List Longhorn engines:

kubectl get engines.longhorn.io \
  -n longhorn-system

List Longhorn nodes:

kubectl get nodes.longhorn.io \
  -n longhorn-system

Display selected Longhorn volume health information:

kubectl get volumes.longhorn.io \
  -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,SCHEDULED:.status.conditions[?(@.type=="scheduled")].status'

Review warning events in the Longhorn namespace:

kubectl get events \
  -n longhorn-system \
  --field-selector=type=Warning \
  --sort-by=.metadata.creationTimestamp

Longhorn custom-resource fields can vary between versions. If a custom-column field is blank, inspect the resource directly:

kubectl get volumes.longhorn.io <volume-name> \
  -n longhorn-system \
  -o yaml

This command is read-only, but the output may contain environment-specific information and should be sanitised before publication or sharing.

Ingress controller checks
#

List Traefik workloads:

kubectl get pods \
  -n kube-system \
  -l app.kubernetes.io/name=traefik \
  -o wide

List Traefik services:

kubectl get services \
  -n kube-system \
  -l app.kubernetes.io/name=traefik

Review Traefik logs:

kubectl logs \
  -n kube-system \
  -l app.kubernetes.io/name=traefik \
  --tail=200

Check ingress resources and assigned addresses:

kubectl get ingress -A -o wide

If Traefik was deployed into another namespace, replace kube-system with the correct namespace.

k3s service health
#

Run these commands over SSH on a k3s server node.

Check the server service:

sudo systemctl status k3s \
  --no-pager \
  --full

Check whether the server service is active:

sudo systemctl is-active k3s

Review recent k3s server logs:

sudo journalctl \
  -u k3s \
  --since "30 minutes ago" \
  --no-pager

Follow the server logs:

sudo journalctl \
  -u k3s \
  --follow

Press Ctrl+C to stop following logs.

On an agent node, use:

sudo systemctl status k3s-agent \
  --no-pager \
  --full

Review recent agent logs:

sudo journalctl \
  -u k3s-agent \
  --since "30 minutes ago" \
  --no-pager

K3s systems using systemd send service logs to Journald. Pod logs are normally stored under /var/log/pods, while the embedded containerd log can be found beneath /var/lib/rancher/k3s/agent/containerd/. These paths should be treated as host-level diagnostic sources rather than files to modify directly.

k3s host checks
#

Display the installed k3s version:

k3s --version

Run the k3s host configuration check:

sudo k3s check-config

Check host disk capacity:

df -h

Check inode usage:

df -ih

Check memory:

free -h

Check system load and uptime:

uptime

Check failed systemd services:

systemctl --failed

Display recent high-priority system journal entries:

sudo journalctl \
  -p warning \
  --since "30 minutes ago" \
  --no-pager

These host checks can help identify:

Full filesystems
Exhausted inodes
Memory pressure
Failed system services
Excessive load
Kernel or storage warnings

Namespace-focused investigation
#

Set a reusable namespace variable:

NAMESPACE="<namespace>"

List the namespace’s main resources:

kubectl get all \
  -n "${NAMESPACE}"

List ConfigMaps:

kubectl get configmaps \
  -n "${NAMESPACE}"

List Secrets by name and type without displaying secret values:

kubectl get secrets \
  -n "${NAMESPACE}"

List persistent volume claims:

kubectl get persistentvolumeclaims \
  -n "${NAMESPACE}"

List recent events:

kubectl get events \
  -n "${NAMESPACE}" \
  --sort-by=.metadata.creationTimestamp

Do not use this command in public screenshots:

kubectl get secrets -n "${NAMESPACE}" -o yaml

Although Kubernetes Secret values are base64 encoded, base64 is not encryption and the values may be recoverable.

Optional cluster information bundle
#

Kubernetes can generate a broad diagnostic dump:

kubectl cluster-info dump \
  --output-directory="./cluster-dump"

This command is read-only against the cluster, but it writes a substantial amount of information locally.

The resulting files may contain:

Internal addresses
Resource names
Configuration details
Pod specifications
Environment variables
Log messages
Infrastructure identifiers

Review and sanitise the complete output before sharing it.

Delete the local dump only after confirming it is no longer required:

rm -rf ./cluster-dump

The removal command is destructive to the specified local directory. Confirm the path before running it.

Expected result
#

A healthy cluster should generally show:

The API server is reachable
All expected nodes are present
Nodes report Ready
No unexpected resource-pressure conditions are active
Required deployments have available replicas
StatefulSets have the expected ready replicas
DaemonSets are scheduled on suitable nodes
Most application pods are Running
Completed jobs report Completed
Restart counts are stable
Warning events do not indicate persistent faults
Services have appropriate endpoints
Ingress resources have the expected routing information
Persistent volume claims are Bound
Longhorn volumes report healthy or expected states
k3s server and agent services are active
Host disks have sufficient free space

A cluster can still have a valid reason for exceptions. For example:

Completed job pods may display Succeeded
Maintenance workloads may be intentionally suspended
A node may be cordoned for planned maintenance
A deployment may intentionally have zero replicas
A test namespace may contain failed workloads

Interpret results in the context of the intended state.

Troubleshooting
#

kubectl cannot connect to the cluster
#

Example error:

Unable to connect to the server

Check the active context:

kubectl config current-context

Check the cluster endpoint:

kubectl config view --minify \
  -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

Confirm basic network connectivity to the API endpoint using an appropriate network diagnostic tool.

Possible causes include:

Incorrect kubeconfig
Wrong context
Unavailable API server
DNS failure
VPN disconnection
Firewall or routing issue
Expired credentials
Incorrect certificate information

A node is NotReady
#

Check the node:

kubectl describe node <node-name>

Review the relevant k3s service on that node:

sudo systemctl status k3s \
  --no-pager \
  --full

Or, for an agent:

sudo systemctl status k3s-agent \
  --no-pager \
  --full

Review recent service logs:

sudo journalctl \
  -u k3s-agent \
  --since "30 minutes ago" \
  --no-pager

Also check:

df -h
free -h
systemctl --failed

A pod is Pending
#

Describe the pod:

kubectl describe pod <pod-name> \
  -n <namespace>

Look for event messages relating to:

Insufficient CPU or memory
Node selectors
Affinity rules
Taints and tolerations
Unbound persistent volume claims
Missing storage classes
Scheduling restrictions

A pod is restarting
#

Check current state and restart count:

kubectl get pod <pod-name> \
  -n <namespace>

Review the previous container logs:

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --previous \
  --tail=200

Describe the pod:

kubectl describe pod <pod-name> \
  -n <namespace>

Look for:

CrashLoopBackOff
OOMKilled
Probe failures
Missing files or configuration
Permission errors
Failed volume mounts
Application startup errors

A service has no endpoints
#

Check the service selector:

kubectl get service <service-name> \
  -n <namespace> \
  -o jsonpath='{.spec.selector}{"\n"}'

List matching pods:

kubectl get pods \
  -n <namespace> \
  --show-labels

Check endpoint slices:

kubectl get endpointslices \
  -n <namespace>

Confirm:

Service selectors match pod labels
Pods are ready
The exposed service port is correct
The service targetPort matches the application port

A persistent volume claim is Pending
#

Describe the claim:

kubectl describe persistentvolumeclaim <claim-name> \
  -n <namespace>

Check storage classes:

kubectl get storageclasses

Check recent events:

kubectl get events \
  -n <namespace> \
  --sort-by=.metadata.creationTimestamp

Check the relevant storage-system namespace, such as Longhorn:

kubectl get pods \
  -n longhorn-system

Safety notes
#

Most commands in this Runbook entry are read-only.

The following commands require particular care:

Log following
#

Commands using --follow continue running until stopped:

kubectl logs --follow
sudo journalctl --follow

Stop them with:

Ctrl+C

Cluster information dump
#

This command writes potentially sensitive diagnostic information to the local filesystem:

kubectl cluster-info dump

Review and sanitise its output before uploading or sharing it.

YAML output
#

Full YAML output can reveal:

Internal addresses
Resource names
Environment variables
Volume configuration
Infrastructure labels
Secret references

Never publish Kubernetes Secret values or unsanitised diagnostic output.

No direct changes
#

This Runbook intentionally avoids commands such as:

kubectl delete
kubectl apply
kubectl edit
kubectl patch
kubectl scale
kubectl rollout restart
systemctl restart

Those commands modify state and should only be used after the fault has been understood, the intended change has been reviewed, and a rollback plan is available.

Related entries
#

Planned related Runbook entries:

Kubernetes Workload Troubleshooting
Kubernetes Storage Health Checks
Longhorn Volume Troubleshooting
Traefik Ingress Troubleshooting
Flux Reconciliation Commands
Kubernetes Resource and Capacity Checks
k3s Server and Agent Troubleshooting

Purpose#

Requirements#

Quick cluster overview#

Node health#

Workload overview#

Failed and restarting containers#

Recent events#

Resource usage#

Services and ingress#

DNS checks#

Persistent storage#

Longhorn checks#

Ingress controller checks#

k3s service health#

k3s host checks#

Namespace-focused investigation#

Optional cluster information bundle#

Expected result#

Troubleshooting#

kubectl cannot connect to the cluster#

A node is NotReady#

A pod is Pending#

A pod is restarting#

A service has no endpoints#

A persistent volume claim is Pending#

Safety notes#

Log following#

Cluster information dump#

YAML output#

No direct changes#

Related entries#

References#