↓ Skip to main content
  1. Runbook/
  2. Monitoring/

Prometheus Target Health Checks

Prometheus ServiceMonitor Observability

Time guide

  • Initial checks: approximately 10 to 20 minutes
  • Detailed investigation: approximately 20 to 60 minutes

Purpose
#

Investigate Prometheus targets, ServiceMonitor discovery and scrape configuration.

Investigation workflow
#

  1. Confirm the correct environment and context.
  2. Capture the current state before making changes.
  3. Run the read-only checks below.
  4. Review events, logs and dependencies.
  5. If a change is required, update the Git-managed desired state where applicable.
  6. Render, dry-run and review the change.
  7. Deploy or reconcile, then complete post-change validation.

Read-only checks
#

kubectl get pods -n <monitoring-namespace>
kubectl get servicemonitors -A
kubectl get prometheusrules -A
kubectl get services -A --show-labels

Expected result
#

The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.

Troubleshooting
#

If a check fails:

  • Confirm names, namespaces, contexts and permissions.
  • Review recent events and service logs.
  • Compare the live state with the documented desired state.
  • Check storage, DNS, network and dependent services.
  • Preserve evidence before restarting, deleting or recreating anything.

Safety notes
#

These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.

Related entries#

  • Kubernetes Cluster Health Checks
  • Flux Reconciliation Commands
  • GitOps Repository Structure
  • Security Incident Triage