[{"content":"A collection of practical technology case studies. Each project explains the problem, design choices, implementation approach, validation, operational outcome and lessons learned.\nExamples are public and sanitised. Credentials, private addressing, customer information and sensitive infrastructure detail are excluded.\n","date":"31 August 2026","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"A collection of practical technology case studies. Each project explains the problem, design choices, implementation approach, validation, operational outcome and lessons learned.\nExamples are public and sanitised. Credentials, private addressing, customer information and sensitive infrastructure detail are excluded.\n","title":"Projects","type":"projects"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/categories/artificial-intelligence/","section":"Categories","summary":"","title":"Artificial Intelligence","type":"categories"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/tags/artificial-intelligence/","section":"Tags","summary":"","title":"Artificial Intelligence","type":"tags"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/tags/case-study/","section":"Tags","summary":"","title":"Case Study","type":"tags"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":" Project snapshot # Status: In Development Project type: Local AI, language-model engineering and controlled automation Role: Architecture, implementation, testing, model evaluation and operational ownership Initial delivery: Locally hosted conversational assistant Long-term objective: A privacy-conscious whole-house intelligence platform Model strategy: Proven open model first, original Lintech foundation-model research in parallel Processing strategy: Dedicated server resources for inference and training workloads Platform strategy: Kubernetes for APIs, orchestration, portal integration and future controlled actions Overview # The Lintech Local AI Platform is the next stage of a wider homelab engineering programme. The long-term objective is to create a locally operated whole-house intelligence system that can assist with technical operations, household services, information retrieval and carefully governed automation.\nThe project is not beginning with an autonomous agent or unrestricted infrastructure access. The first milestone is a conventional chatbot running against a proven, locally deployable language model. This provides a stable baseline for evaluating response quality, local inference performance, privacy, conversation handling and integration with the existing Lintech Intelligence Platform.\nAlongside that useful first system, a second track will establish the datasets, tokenizer, architecture experiments, training pipelines, evaluations and governance required to research, design and train an original Lintech language model from the foundation up. The initial open model may help propose candidate examples, analyse failures and support experiments, but reviewed data, architectural decisions, evaluation criteria and model promotion will remain under human control.\nEnvironment and operating context # The existing homelab already provides the platform services needed to support an AI system responsibly. Kubernetes hosts applications, GitOps controls deployment, Prometheus and Grafana provide observability, Longhorn supplies persistent storage, Wazuh and CrowdSec provide security visibility, and the Lintech Intelligence Platform provides a unified operational interface.\nA dedicated Dell PowerEdge server will supply the heavier compute and memory resources required for model inference, dataset processing, evaluation and future fine-tuning work. Kubernetes will host the lighter but security-sensitive control layer, including the chatbot API, policy enforcement, portal integration and any future tightly controlled action services.\nThis division keeps resource-intensive model processing away from the power-efficient Kubernetes nodes while preventing the model runtime from becoming a privileged infrastructure controller.\nProject objective # The immediate objective is to deliver a useful local chatbot that can provide general technical assistance without claiming access to systems or information that have not been connected.\nThe longer-term objective is to develop a Lintech-specific assistant that can:\nFollow an evidence-led troubleshooting process. Understand the terminology and operating practices used across the homelab. Retrieve approved project and Runbook content with citations. Request current state through narrow read-only tools. Distinguish live observations from documented or planned information. Produce recommendations with impact, validation and rollback considerations. Propose tightly bounded actions for explicit human approval. Support future household services without becoming an unrestricted automation engine. Why two development tracks are being used # The project deliberately separates immediate usefulness from long-term model development.\nTrack A: Build a useful assistant # Track A begins with a proven locally deployable instruction model. The model will be hosted locally and exposed through an internal, OpenAI-compatible interface. A separate Lintech AI API will manage conversations, enforce policy, apply request limits and later broker access to approved information sources.\nThis approach provides a usable chatbot early in the project and creates a stable baseline against which later models can be measured.\nTrack B: Develop an original Lintech foundation model # Track B establishes the research, data, training and evaluation foundations required to design and train an original Lintech language model:\nA versioned evaluation suite. A curated instruction dataset. Sensitive-data filtering and review procedures. Governed pre-training dataset development. Tokenizer design and evaluation. Model architecture experiments. Small-scale pre-training runs. Training and validation pipelines. Model checkpoints and reproducibility records. Optional LoRA comparison experiments. Candidate model cards. Open-model-versus-Lintech-model comparison reports. Safety and evidence-boundary tests. A controlled model promotion and rollback process. The initial operational assistant will use a proven open foundation model so that the surrounding platform, evaluation framework, safety boundaries and user experience can be established against a reliable baseline. In parallel, the longer-term research objective is to design and train an original Lintech language model from the foundation up, including its architecture, tokenizer, governed dataset, training process and evaluation framework. LoRA adaptation of an existing model may also be explored as an intermediate comparison track, but it is not the final objective of the foundation-model programme.\nOperational value and risk reduction # Before the project # The homelab contains strong operational data and documentation, but no conversational layer connects that information into a guided investigation experience. General-purpose AI services can help with individual questions, but sending detailed internal operational context to an external provider would introduce privacy, governance and availability concerns.\nAfter the initial release # The first release will provide a locally operated chatbot with a defined security boundary, a versioned deployment process and a known model baseline. It will not have infrastructure tools, persistent memory or private repository access.\nIntended risk reduction # The architecture is intended to reduce several risks:\nSensitive operational context can remain within the local environment. Model credentials and provider access remain outside browser code. The model runtime receives no general Kubernetes administrator credentials. Infrastructure-changing capabilities are separated from language generation. Candidate models cannot replace the stable model without evaluation and approval. Live system state will come from approved tools rather than model memory. Changing facts will not be embedded permanently through fine-tuning. These are architectural objectives. Their effectiveness will be validated as each phase is implemented.\nProposed architecture # User | v Lintech Intelligence Platform | v Lintech AI API on Kubernetes |-- conversation validation |-- policy enforcement |-- model selection |-- streaming responses |-- read-only tools later `-- approval workflow later | v Model service on dedicated server resources |-- baseline model inference |-- embeddings later |-- evaluation runs |-- fine-tuning experiments `-- candidate model storage The model service generates language. Kubernetes-hosted services determine which information or action interfaces are available. The model will not receive a general-purpose shell, unrestricted Kubernetes credentials or direct administrative access to infrastructure platforms.\nArchitectural decisions and alternatives # Hosted external model API # A hosted provider could reduce the time required to launch the first chatbot and might offer stronger performance than the initial CPU-only environment. The trade-off is that prompts and contextual data would leave the homelab, creating additional privacy, cost and governance requirements.\nModel runtime directly inside Kubernetes # Running inference on the Kubernetes nodes would simplify service discovery, but it would compete with operational workloads and place heavy processing on the power-efficient ARM64 cluster. The nodes are better suited to orchestration, APIs and controlled tools.\nDedicated model-processing VM # A dedicated virtual machine on the server provides substantial memory, isolated model storage and a clean API boundary. It can support CPU-based prototyping immediately and GPU acceleration later without changing the portal or orchestration architecture.\nTraining a complete foundation model immediately # Training a competitive general-purpose language model from scratch would require a much larger dataset and considerably more accelerator capacity. It would delay the useful assistant while creating significant engineering and evaluation complexity.\nSelected approach # The selected approach combines a proven open model, a dedicated processing environment, a Kubernetes-hosted control layer and a parallel foundation-model research programme. This provides useful capability early while preserving a credible route towards an original Lintech model whose architecture, tokenizer, training data and checkpoints are developed under the project’s own governance.\nImplementation plan # Create the model-processing environment. A dedicated virtual machine will be provisioned on the server with separate operating-system and model storage. The initial deployment will use CPU inference so the architecture can be validated before any GPU investment. Install a local model runtime. An OpenAI-compatible runtime will host one small test model and one stronger baseline model. The service will remain internal and will not be published directly to the Internet. Benchmark the baseline models. Model load time, time to first token, generation rate, memory use, context handling and answer quality will be recorded. The largest model will not automatically be selected if its latency makes the chatbot impractical. Build the Lintech AI API. A Python and FastAPI service will validate requests, apply the system policy, forward conversations to the model runtime, stream results and return safe structured errors. Deploy the API through GitOps. The service will be containerised, built for the required architecture, published through CI and reconciled to Kubernetes by Flux. Credentials and configuration will use SOPS-encrypted Kubernetes Secrets. Add the chatbot interface. The Lintech Intelligence Platform will gain an Intelligence Assistant page with conversation history, streaming responses, connection status, retry behaviour and a clear advisory boundary. Establish the evaluation baseline. A reviewed evaluation suite will test general assistance, Kubernetes, GitOps, storage, security, evidence handling, live-state boundaries and action safety against the original model. Create governed research datasets. Candidate material may be derived from approved Runbooks, project content and suitably licensed public sources, but every dataset component will require provenance, licence, quality and sensitive-data review before use. Design and evaluate a tokenizer. Tokenizer experiments will measure vocabulary coverage, sequence efficiency and handling of technical language before a tokenizer is selected for model training. Run small architecture experiments. Small transformer configurations will be trained first to validate the data pipeline, loss behaviour, checkpointing, evaluation and recovery process without presenting those experiments as production-capable models. Train the first original Lintech candidate. A selected architecture will be pre-trained from an initial checkpoint using the governed dataset and reproducible configuration. Optional LoRA work may be used as a comparison track, not as the final foundation-model objective. Compare the open and original models. The open-model assistant and Lintech candidates will receive the same held-out tests. Technical correctness, hallucination rate, evidence discipline, clarity, latency, safety compliance and resource use will be compared. Use shadow evaluation before replacement. A candidate may process selected prompts alongside the stable model without becoming the user-facing response. This allows comparison without risking the working assistant. Promote only after approval. An original candidate will replace the open-model baseline only if evaluation demonstrates a meaningful improvement without unacceptable regression. The stable open model will remain available as a rollback option. Add read-only context. Approved portal summary tools will provide live system state with timestamps and source links. The model will request these tools rather than memorising changing infrastructure facts. Add documentation retrieval. Approved Runbooks and architecture material will be indexed separately from model training and returned with citations and document review dates. Add controlled proposals and actions later. The assistant may eventually propose narrow actions, but execution will require an independent Kubernetes service, deterministic validation, visible impact, rollback information, human approval and post-action verification. Model-development strategy # The original Lintech model will be trained to learn general language and reasoning patterns from governed data while avoiding permanent encoding of changing environment state. Behavioural adaptation through LoRA may be evaluated separately, but the foundation-model track covers tokenizer selection, architecture design, pre-training, checkpointing and evaluation from an initial model state.\nSuitable fine-tuning targets include:\nEvidence-aware language. Safe admission of uncertainty. Troubleshooting order. Runbook structure. Recommendation quality. Tool-selection decisions. Approval and rollback language. Lintech-specific terminology. Changing facts will remain outside the model:\nCurrent node readiness. Current Pod counts. Current alerts. Current image digests. Current service availability. Current storage health. The separation is intentional:\nFine-tuning = how the assistant behaves Retrieval = what approved documents say Live tools = what is true now Actions = what may change after approval Dataset and evaluation governance # The baseline model may help propose examples, categorise failures and generate question variations, but the model will not approve its own training data or control its own promotion.\nThe dataset lifecycle will be:\nProposed example | v Sensitive-data and secret scan | v Human review | +-- approved dataset `-- rejected examples Full chat histories will not be added automatically to training. Only selected, sanitised and reviewed examples will be eligible.\nEvery candidate release will record:\nBase model and exact revision. Base-model licence. Dataset version. Evaluation-suite version. Tokenizer and vocabulary version. Model architecture and parameter count. Pre-training configuration. Optional fine-tuning configuration. Known strengths and weaknesses. Safety-test results. Artifact checksum. Approval status. Rollback model. Evidence and success criteria # The project is currently in development, so future capability is not presented as completed work. Initial success will be measured through repeatable technical evidence.\nPhase 1 acceptance criteria # Model-processing VM provisioned and internally reachable. Baseline model responds through an OpenAI-compatible API. Memory use and response latency recorded. Chat API returns a healthy status. Streaming responses work correctly. Invalid roles and oversized requests are rejected. No conversation content is persisted by default. No infrastructure tools are enabled. Kubernetes deployment is reconciled through Flux. The LIP chatbot renders and handles provider failure safely. Foundation-model acceptance criteria # Open-model baseline evaluation results recorded before original-model training. Dataset provenance, licensing and sensitive-data reviews recorded. Training, validation and held-out test data separated. Approved datasets contain no detected credentials or private keys. Tokenizer selection is supported by recorded coverage and efficiency tests. Small-scale training validates the full pipeline before a larger run. Every training run is reproducible from versioned data and configuration. Candidate checkpoints are compared against the stable open-model baseline. Safety and evidence-boundary tests do not regress. Stable-model rollback requires only a controlled configuration change. No performance improvement, accuracy percentage or response-speed claim will be published until measured.\nSecurity and action boundaries # The initial assistant will have:\nNo Kubernetes credentials. No shell or SSH access. No Proxmox administrative access. No access to Kubernetes Secrets. No persistent personal memory. No unrestricted repository access. No infrastructure tools. No ability to claim that an action was completed. Future action handling will remain outside the model runtime. A safe action flow would be:\nUser request | Model creates structured proposal | Policy service validates scope | Portal displays evidence, impact and rollback | Human approval | Narrow action service executes | Post-action validation and audit result The model will never receive a general command-execution interface.\nLimitations and residual risks # Initial CPU inference may have a noticeable response delay. A larger model may fit into available memory but still be too slow for interactive use. Future GPU support will require hardware, power, cooling and passthrough validation. An original model may underperform the established open-model baseline for a considerable period. Scaling an original model beyond research size may exceed the available compute, storage, power or training-time budget. Training data can transfer errors or outdated assumptions into the model if review is weak. Local operation improves data control but does not automatically guarantee privacy or security. Prompt injection remains relevant when retrieval and tools are introduced. Model output remains advisory and may be incorrect. Controlled actions will increase risk and are intentionally deferred. Optional LoRA candidates will remain dependent on the licence and characteristics of their base models. Original-model training data must remain compatible with recorded licences and permitted uses. Plain-English glossary # Locally deployable model A model that can run within privately controlled infrastructure under its applicable licence. This does not mean the Lintech project or its source code will be published. Inference Running a trained model to generate an answer. Pre-training Training a model from an initial parameter state on a broad governed dataset so that the model learns language patterns and representations. Fine-tuning Continuing training on a targeted dataset to adapt an existing model’s behaviour or specialised capability. LoRA A parameter-efficient fine-tuning method that trains smaller adapter matrices while leaving the base model largely unchanged. Embedding A numerical representation of text used for semantic search and retrieval. Retrieval Finding relevant approved documents at question time and supplying them to the model as evidence. Tool A fixed software function that can return live information or request a narrow action. Shadow evaluation Running a candidate model alongside the stable model for comparison without making the candidate user-facing. Source availability # The publication and licensing model for the Lintech AI source code, datasets, adapters and trained artifacts has not yet been decided. This case study describes the engineering direction only and should not be interpreted as a commitment to release the project as open source.\nCurrent status # In Development.\nThe existing homelab, GitOps, observability, recovery, security and Lintech Intelligence Platform foundations are operational. The dedicated AI processing virtual machine, model runtime, Lintech AI API and chatbot page are the next implementation milestones.\nNo live portal tools, persistent memory, custom fine-tuned model or infrastructure actions are currently claimed as complete.\nPlanned milestones # Provision the AI model-processing VM. Install and benchmark the first open model. Build the standalone Lintech AI API. Deploy the API to Kubernetes through Flux. Add the LIP chatbot interface. Publish the first baseline evaluation report. Build and review the first governed pre-training dataset. Design and evaluate the first Lintech tokenizer. Run small-scale transformer architecture and training experiments. Train the first original Lintech model candidate. Compare original candidates against the open-model baseline. Explore LoRA as an optional comparison track. Add read-only portal tools and approved retrieval. Design, but do not yet enable, controlled action workflows. Skills being developed # Local LLM deployment Model-runtime benchmarking Python and FastAPI development Streaming chatbot interfaces MLOps and model versioning Dataset design and governance Evaluation engineering Tokenizer design Transformer architecture experiments Foundation-model pre-training Checkpointing and training recovery Optional LoRA fine-tuning Retrieval-augmented generation Tool and policy design Kubernetes AI orchestration Security boundaries for AI systems Human approval and auditable automation Project outlook # The Lintech Local AI Platform is intended to become the intelligence layer above an already operational homelab. Starting with a proven open model provides immediate value and a measurable baseline. Developing a separate foundation-model track creates a structured path towards an original Lintech model, with project-owned architecture decisions, tokenizer experiments, governed training data, reproducible checkpoints and independent evaluation.\nThe objective is not to create an assistant that changes infrastructure without supervision. The objective is to build a trustworthy system that can explain evidence, support investigation, retrieve approved knowledge and eventually participate in carefully controlled operations where every action remains visible, validated and reversible.\nFollow the project # The project will be documented as the model VM, chatbot, evaluation suite and first original Lintech model experiments are completed. Related case studies cover the Kubernetes platform, Flux delivery model, Lintech Intelligence Platform, security monitoring and recovery architecture that provide the foundation for this work.\n","date":"8 September 2026","externalUrl":null,"permalink":"/projects/lintech-local-ai-platform/","section":"Projects","summary":"Building a local-first AI platform that begins with a proven open model while establishing the data, architecture, training and evaluation foundations for an original Lintech language model.","title":"Developing the Lintech Local AI Platform","type":"projects"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/tags/kubernetes/","section":"Tags","summary":"","title":"Kubernetes","type":"tags"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/tags/llm/","section":"Tags","summary":"","title":"LLM","type":"tags"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/tags/local-ai/","section":"Tags","summary":"","title":"Local AI","type":"tags"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/tags/mlops/","section":"Tags","summary":"","title":"MLOps","type":"tags"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/tags/project/","section":"Tags","summary":"","title":"Project","type":"tags"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/","section":"Richard Lintern","summary":"","title":"Richard Lintern","type":"page"},{"content":"","date":"8 September 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/age/","section":"Tags","summary":"","title":"Age","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/antivirus/","section":"Tags","summary":"","title":"Antivirus","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/apple-diagnostics/","section":"Tags","summary":"","title":"Apple Diagnostics","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Plan, render, validate and reconcile a Kubernetes application through Flux GitOps.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl kustomize apps/\u0026lt;application-name\u0026gt; \u0026gt; /tmp/rendered.yaml kubectl apply --dry-run=client -f /tmp/rendered.yaml kubectl apply --server-side --dry-run=server -f /tmp/rendered.yaml flux reconcile kustomization \u0026lt;application-name\u0026gt; -n \u0026lt;flux-namespace\u0026gt; --with-source Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/application-onboarding/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Plan, render, validate and reconcile a Kubernetes application through Flux GitOps.\n","title":"Application GitOps Onboarding","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/applications/","section":"Tags","summary":"","title":"Applications","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/backup/","section":"Tags","summary":"","title":"Backup","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/bash/","section":"Tags","summary":"","title":"Bash","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/battery/","section":"Tags","summary":"","title":"Battery","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/bitlocker/","section":"Tags","summary":"","title":"BitLocker","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only checks for encryption, protection, TPM and protector state. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # Get-BitLockerVolume manage-bde -status Get-Tpm (Get-BitLockerVolume -MountPoint \u0026#34;C:\u0026#34;).KeyProtector | Select KeyProtectorType,KeyProtectorId Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/bitlocker-status-protection/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only checks for encryption, protection, TPM and protector state. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"BitLocker Status and Protection Checks","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for identifying why BitLocker protection is off or suspended and whether maintenance caused it. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-BitLockerVolume -MountPoint \u0026#34;C:\u0026#34; | Format-List * manage-bde -status C: Get-WinEvent -LogName \u0026#34;Microsoft-Windows-BitLocker/BitLocker Management\u0026#34; -MaxEvents 100 Get-WinEvent -FilterHashtable @{LogName=\u0026#34;Setup\u0026#34;; StartTime=(Get-Date).AddDays(-7)} Resume-BitLocker -MountPoint \u0026#34;C:\u0026#34; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/bitlocker-suspension-investigation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for identifying why BitLocker protection is off or suspended and whether maintenance caused it. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"BitLocker Suspension Investigation","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/capacity/","section":"Tags","summary":"","title":"Capacity","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/case-studies/","section":"Tags","summary":"","title":"Case Studies","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/cmd/","section":"Tags","summary":"","title":"CMD","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/compliance/","section":"Tags","summary":"","title":"Compliance","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Separate DNS, route, TCP, TLS and application-layer connectivity faults.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # getent hosts \u0026lt;hostname\u0026gt; dig \u0026lt;hostname\u0026gt; ip route nc -vz \u0026lt;hostname\u0026gt; \u0026lt;port\u0026gt; curl -Iv https://\u0026lt;hostname\u0026gt; Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/connectivity-port-testing/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Separate DNS, route, TCP, TLS and application-layer connectivity faults.\n","title":"Connectivity and Port Testing","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/controllers/","section":"Tags","summary":"","title":"Controllers","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/defender-for-endpoint/","section":"Tags","summary":"","title":"Defender for Endpoint","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Check Defender for Endpoint sensor services, onboarding indicators and relevant event channels. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-Service Sense,WinDefend,WdNisSvc Get-MpComputerStatus Get-WinEvent -LogName \u0026#34;Microsoft-Windows-SENSE/Operational\u0026#34; -MaxEvents 100 Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/defender-endpoint-sensor-health/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Check Defender for Endpoint sensor services, onboarding indicators and relevant event channels. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Defender for Endpoint Sensor Health","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/detection-engineering/","section":"Tags","summary":"","title":"Detection Engineering","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/device-registration/","section":"Tags","summary":"","title":"Device Registration","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/dhcp/","section":"Tags","summary":"","title":"DHCP","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for address assignment, lease, gateway, DNS options and conflict symptoms. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # ipconfig /all ipconfig /release ipconfig /renew Get-NetIPConfiguration networksetup -getinfo \u0026#34;\u0026lt;network-service\u0026gt;\u0026#34; nmcli device show Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/dhcp-client-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for address assignment, lease, gateway, DNS options and conflict symptoms. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"DHCP Client Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/disaster-recovery/","section":"Tags","summary":"","title":"Disaster Recovery","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/disk-cleanup/","section":"Tags","summary":"","title":"Disk Cleanup","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/dism/","section":"Tags","summary":"","title":"DISM","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for checking and repairing the Windows component store and protected system files. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # DISM.exe /Online /Cleanup-Image /CheckHealth DISM.exe /Online /Cleanup-Image /ScanHealth DISM.exe /Online /Cleanup-Image /RestoreHealth sfc /scannow Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/dism-sfc-repair/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for checking and repairing the Windows component store and protected system files. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"DISM and System File Checker","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/dns/","section":"Tags","summary":"","title":"DNS","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Plan and validate internal DNS, public DNS, ingress and protected service exposure.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get svc,ingress -n \u0026lt;namespace\u0026gt; kubectl get endpointslices -n \u0026lt;namespace\u0026gt; dig \u0026lt;hostname\u0026gt; @\u0026lt;dns-server\u0026gt; curl -I https://\u0026lt;hostname\u0026gt; Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/dns-exposure-management/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Plan and validate internal DNS, public DNS, ingress and protected service exposure.\n","title":"DNS and Service Exposure Management","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Cross-platform workflow for resolver, record, cache, suffix and upstream DNS faults. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Resolve-DnsName \u0026lt;hostname\u0026gt; nslookup \u0026lt;hostname\u0026gt; \u0026lt;dns-server\u0026gt; dig \u0026lt;hostname\u0026gt; @\u0026lt;dns-server\u0026gt; scutil --dns ipconfig /displaydns Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/dns-resolution-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Cross-platform workflow for resolver, record, cache, suffix and upstream DNS faults. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"DNS Resolution Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/drift/","section":"Tags","summary":"","title":"Drift","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/dsregcmd/","section":"Tags","summary":"","title":"Dsregcmd","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/encryption/","section":"Tags","summary":"","title":"Encryption","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/endpoint/","section":"Tags","summary":"","title":"Endpoint","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/endpoint-management/","section":"Tags","summary":"","title":"Endpoint Management","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/endpoint-security/","section":"Tags","summary":"","title":"Endpoint Security","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Interpret core dsregcmd device join, registration, SSO and MDM indicators. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # dsregcmd /status Get-ScheduledTask -TaskPath \u0026#34;\\Microsoft\\Windows\\Workplace Join\\\u0026#34; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/entra-device-registration/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Interpret core dsregcmd device join, registration, SSO and MDM indicators. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Entra Device Registration Checks","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/entra-id/","section":"Tags","summary":"","title":"Entra ID","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only Microsoft Graph checks for users, groups, licenses and membership. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # Connect-MgGraph -Scopes User.Read.All,Group.Read.All,Directory.Read.All Get-MgUser -UserId \u0026lt;user-id\u0026gt; Get-MgUserMemberOf -UserId \u0026lt;user-id\u0026gt; Get-MgGroupMember -GroupId \u0026lt;group-id\u0026gt; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/entra-user-group-checks/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only Microsoft Graph checks for users, groups, licenses and membership. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Entra ID User and Group Checks","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/event-logs/","section":"Tags","summary":"","title":"Event Logs","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/exchange-online/","section":"Tags","summary":"","title":"Exchange Online","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Install, connect, verify and disconnect Exchange Online PowerShell safely. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # Install-Module ExchangeOnlineManagement -Scope CurrentUser Import-Module ExchangeOnlineManagement Connect-ExchangeOnline -UserPrincipalName \u0026lt;admin-account\u0026gt; Get-ConnectionInformation Disconnect-ExchangeOnline -Confirm:$false Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/exchange-online-connections/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Install, connect, verify and disconnect Exchange Online PowerShell safely. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Exchange Online PowerShell Connections","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only checks for mailbox type, aliases, forwarding, permissions and delivery settings. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-EXOMailbox -Identity \u0026lt;user\u0026gt; Get-EXORecipient -Identity \u0026lt;user\u0026gt; Get-Mailbox \u0026lt;user\u0026gt; | Select ForwardingAddress,ForwardingSmtpAddress,DeliverToMailboxAndForward Get-InboxRule -Mailbox \u0026lt;user\u0026gt; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/exchange-recipient-investigation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only checks for mailbox type, aliases, forwarding, permissions and delivery settings. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Exchange Online Recipient Investigation","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/filesystem/","section":"Tags","summary":"","title":"Filesystem","type":"tags"},{"content":"Time guide\nBasic controller health check: approximately 5 to 10 minutes Source or reconciliation investigation: approximately 15 to 30 minutes Purpose # Use this entry when Flux resources are not reconciling, a source is stale, or one of the controllers appears unhealthy.\nInstallation checks # flux version flux check Controller workloads # kubectl get deployments -n flux-system kubectl get pods -n flux-system -o wide kubectl get deployments -n flux-system \\ -o custom-columns=\u0026#39;NAME:.metadata.name,READY:.status.readyReplicas,AVAILABLE:.status.availableReplicas,DESIRED:.spec.replicas\u0026#39; Flux resource health # flux get all -A flux get sources all -A flux get kustomizations -A flux get helmreleases -A Suspended resources # kubectl get kustomizations.kustomize.toolkit.fluxcd.io -A \\ -o custom-columns=\u0026#39;NAMESPACE:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend\u0026#39; kubectl get gitrepositories.source.toolkit.fluxcd.io -A \\ -o custom-columns=\u0026#39;NAMESPACE:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend\u0026#39; Events # flux events -A kubectl get events -n flux-system \\ --sort-by=.metadata.creationTimestamp kubectl get events -n flux-system \\ --field-selector=type=Warning \\ --sort-by=.metadata.creationTimestamp Controller logs # kubectl logs -n flux-system deployment/source-controller --tail=200 kubectl logs -n flux-system deployment/kustomize-controller --tail=200 kubectl logs -n flux-system deployment/helm-controller --tail=200 kubectl logs -n flux-system deployment/notification-controller --tail=200 Search recent errors:\nkubectl logs -n flux-system deployment/kustomize-controller --since=30m \\ | grep -iE \u0026#39;error|failed|timeout\u0026#39; Common findings # Authentication or repository access failure Invalid source reference Stale or unavailable source artifact Kustomize build failure SOPS decryption failure Dependency not ready Helm install or upgrade failure Admission-policy rejection Unhealthy target workload Safety notes # All commands in this entry are read-only. Use the separate Flux Reconciliation entry only after confirming the cluster context, intended source revision and likely fault domain.\nRelated entries # Flux Reconciliation Commands GitRepository Troubleshooting Kustomization Troubleshooting HelmRelease Troubleshooting SOPS and age Secret Workflow ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/flux-controller-health/","section":"Runbook","summary":"Time guide\nBasic controller health check: approximately 5 to 10 minutes Source or reconciliation investigation: approximately 15 to 30 minutes Purpose # Use this entry when Flux resources are not reconciling, a source is stale, or one of the controllers appears unhealthy.\n","title":"Flux Controller Health Checks","type":"lab-notes"},{"content":"Time guide\nInitial Kustomization checks: approximately 5 to 10 minutes Build or dependency investigation: approximately 15 to 30 minutes Render, dry-run and reconciliation validation: approximately 20 to 45 minutes Purpose # Use this entry when a Flux Kustomization is not ready, reports a build failure, waits on a dependency, or applies an unexpected revision.\nInspect status # flux get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; kubectl describe kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; kubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o custom-columns=\u0026#39;NAME:.metadata.name,PATH:.spec.path,SOURCE:.spec.sourceRef.name,PRUNE:.spec.prune,SUSPENDED:.spec.suspend,REVISION:.status.lastAppliedRevision\u0026#39; Readiness message # kubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].status}{\u0026#34;\\n\u0026#34;}{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].reason}{\u0026#34;\\n\u0026#34;}{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].message}{\u0026#34;\\n\u0026#34;}\u0026#39; Source health # kubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.sourceRef.kind}{\u0026#34;/\u0026#34;}{.spec.sourceRef.name}{\u0026#34;\\n\u0026#34;}\u0026#39; flux get sources all -A Dependencies # kubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.dependsOn}{\u0026#34;\\n\u0026#34;}\u0026#39; flux get kustomizations -A Investigate any dependency that is not ready before reconciling the dependent object.\nLocal render # From the repository root:\nkubectl kustomize \u0026lt;path-to-kustomization\u0026gt; \\ \u0026gt; /tmp/\u0026lt;kustomization-name\u0026gt;-rendered.yaml test -s /tmp/\u0026lt;kustomization-name\u0026gt;-rendered.yaml grep \u0026#39;^kind:\u0026#39; /tmp/\u0026lt;kustomization-name\u0026gt;-rendered.yaml Dry runs # kubectl apply \\ --dry-run=client \\ -f /tmp/\u0026lt;kustomization-name\u0026gt;-rendered.yaml After confirming context and permissions:\nkubectl apply \\ --server-side \\ --dry-run=server \\ -f /tmp/\u0026lt;kustomization-name\u0026gt;-rendered.yaml Controller logs and events # flux events \\ --for Kustomization/\u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; kubectl logs \\ -n flux-system \\ deployment/kustomize-controller \\ --since=30m Common reasons # BuildFailed # Check for:\nMissing resource paths Invalid YAML Duplicate resources Invalid patches Unsupported API versions SOPS decryption failures Missing Helm chart inputs DependencyNotReady # Inspect spec.dependsOn and the readiness of each dependency.\nHealthCheckFailed # Check the target workloads:\nkubectl get deployments,statefulsets,daemonsets -n \u0026lt;workload-namespace\u0026gt; kubectl get pods -n \u0026lt;workload-namespace\u0026gt; kubectl get events -n \u0026lt;workload-namespace\u0026gt; --sort-by=.metadata.creationTimestamp Reconcile after validation # This can create, update or prune resources according to Git.\nflux reconcile kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ --with-source Post-reconciliation validation # flux get kustomization \u0026lt;kustomization-name\u0026gt; -n \u0026lt;namespace\u0026gt; kubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.lastAppliedRevision}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl get pods -n \u0026lt;workload-namespace\u0026gt; kubectl get events -n \u0026lt;workload-namespace\u0026gt; --sort-by=.metadata.creationTimestamp Remove the temporary render:\nrm /tmp/\u0026lt;kustomization-name\u0026gt;-rendered.yaml Safety notes # A Kustomization with pruning enabled may delete resources absent from the desired state. Always review the rendered output, Git diff, cluster context and target path before manual reconciliation.\nRelated entries # Flux Reconciliation Commands Flux Controller Health Checks GitOps Repository Structure SOPS and age Secret Workflow Kubernetes Pod and Workload Troubleshooting ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/kustomization-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial Kustomization checks: approximately 5 to 10 minutes Build or dependency investigation: approximately 15 to 30 minutes Render, dry-run and reconciliation validation: approximately 20 to 45 minutes Purpose # Use this entry when a Flux Kustomization is not ready, reports a build failure, waits on a dependency, or applies an unexpected revision.\n","title":"Flux Kustomization Troubleshooting","type":"lab-notes"},{"content":"Time guide\nBasic Flux health check: approximately 5 to 10 minutes Failed reconciliation investigation: approximately 10 to 30 minutes Render, dry-run, reconciliation and validation: approximately 15 to 45 minutes The actual duration depends on repository size, deployment dependencies, controller health and the cause of any reconciliation failure.\nPurpose # This Runbook entry provides commands for checking and troubleshooting a Kubernetes environment managed by Flux.\nThe workflow follows a read-only-first approach:\nConfirm the Kubernetes context. Confirm Flux CLI and controller health. Inspect sources and managed resources. Review events and controller logs. Validate local manifests. Trigger reconciliation only when required. Confirm the resulting Git and cluster state. Flux normally reconciles resources automatically according to their configured intervals. Manual reconciliation is useful after pushing an important change or while investigating a delayed or failed deployment.\nRequirements # A working kubectl installation The Flux CLI A valid Kubernetes kubeconfig Access to the Flux-managed cluster Permission to read Flux custom resources Permission to reconcile resources when using the mutating commands A local clone of the GitOps repository for rendering and validation Check the active Kubernetes context:\nkubectl config current-context List available contexts:\nkubectl config get-contexts Check the Flux CLI version:\nflux version --client Display the client and installed controller versions:\nflux version Flux uses the current Kubernetes context. Confirm the context before running any reconciliation command.\nRead-only Flux health checks # Check whether Flux is installed and healthy:\nflux check Run installation prerequisite checks:\nflux check --pre List Flux controllers:\nkubectl get deployments \\ -n flux-system List controller pods:\nkubectl get pods \\ -n flux-system \\ -o wide Expected controllers commonly include:\nsource-controller kustomize-controller helm-controller notification-controller Image automation installations may also include:\nimage-reflector-controller image-automation-controller Check whether all Flux deployments have available replicas:\nkubectl get deployments \\ -n flux-system \\ -o custom-columns=\u0026#39;NAME:.metadata.name,READY:.status.readyReplicas,AVAILABLE:.status.availableReplicas,DESIRED:.spec.replicas\u0026#39; Flux resource overview # List all Flux resources across all namespaces:\nflux get all -A List all sources:\nflux get sources all -A List Git sources:\nflux get sources git -A List OCI sources:\nflux get sources oci -A List Helm repositories:\nflux get sources helm -A List Kustomizations:\nflux get kustomizations -A List Helm releases:\nflux get helmreleases -A List image automation resources:\nflux get images all -A Healthy resources should normally show:\nREADY True The status message should include the current applied or fetched revision.\nInspect a Git source # List GitRepository resources:\nkubectl get gitrepositories.source.toolkit.fluxcd.io -A Inspect a particular Git source:\nflux get source git \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Describe the source:\nkubectl describe gitrepository \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Display the fetched revision:\nkubectl get gitrepository \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.artifact.revision}{\u0026#34;\\n\u0026#34;}\u0026#39; Display the source readiness condition:\nkubectl get gitrepository \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].status}{\u0026#34;\\n\u0026#34;}\u0026#39; Display the latest source status message:\nkubectl get gitrepository \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].message}{\u0026#34;\\n\u0026#34;}\u0026#39; Possible source problems include:\nRepository authentication failure DNS or network connectivity problems Invalid repository URL Missing branch or tag TLS certificate failure Repository rate limiting Invalid or inaccessible artifact Inspect a Kustomization # List Kustomizations:\nkubectl get kustomizations.kustomize.toolkit.fluxcd.io -A Inspect a particular Kustomization:\nflux get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Describe it:\nkubectl describe kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Display the last applied revision:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.lastAppliedRevision}{\u0026#34;\\n\u0026#34;}\u0026#39; Display its readiness state:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].status}{\u0026#34;\\n\u0026#34;}\u0026#39; Display the latest readiness message:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].message}{\u0026#34;\\n\u0026#34;}\u0026#39; Check its configured source and path:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o custom-columns=\u0026#39;NAME:.metadata.name,SOURCE_KIND:.spec.sourceRef.kind,SOURCE_NAME:.spec.sourceRef.name,PATH:.spec.path,PRUNE:.spec.prune,INTERVAL:.spec.interval\u0026#39; Inspect a HelmRelease # List Helm releases:\nkubectl get helmreleases.helm.toolkit.fluxcd.io -A Inspect a particular release:\nflux get helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Describe the release:\nkubectl describe helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Display its readiness state:\nkubectl get helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].status}{\u0026#34;\\n\u0026#34;}\u0026#39; Display its latest readiness message:\nkubectl get helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].message}{\u0026#34;\\n\u0026#34;}\u0026#39; Display the currently applied chart revision:\nkubectl get helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.history[0].chartVersion}{\u0026#34;\\n\u0026#34;}\u0026#39; The exact status fields available can vary between Flux versions. If a JSONPath returns no value, inspect the resource:\nkubectl get helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o yaml Sanitise YAML output before sharing it publicly.\nFind suspended resources # List suspended Kustomizations:\nkubectl get kustomizations.kustomize.toolkit.fluxcd.io -A \\ -o custom-columns=\u0026#39;NAMESPACE:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend\u0026#39; | grep true List suspended Helm releases:\nkubectl get helmreleases.helm.toolkit.fluxcd.io -A \\ -o custom-columns=\u0026#39;NAMESPACE:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend\u0026#39; | grep true List suspended Git sources:\nkubectl get gitrepositories.source.toolkit.fluxcd.io -A \\ -o custom-columns=\u0026#39;NAMESPACE:.metadata.namespace,NAME:.metadata.name,SUSPENDED:.spec.suspend\u0026#39; | grep true A suspended resource will not reconcile until resumed.\nDo not resume a resource until the reason for suspension is understood.\nFlux events # Display recent Flux events:\nflux events -A Display events for a particular resource:\nflux events \\ --for Kustomization/\u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Display Kubernetes events in the Flux namespace:\nkubectl get events \\ -n flux-system \\ --sort-by=.metadata.creationTimestamp Display warning events:\nkubectl get events -A \\ --field-selector=type=Warning \\ --sort-by=.metadata.creationTimestamp Investigate messages relating to:\nArtifactFailed BuildFailed DependencyNotReady GitOperationFailed HealthCheckFailed ReconciliationFailed SourceUnavailable StorageOperationFailed UpgradeFailed InstallFailed Controller logs # Review source-controller logs:\nkubectl logs \\ -n flux-system \\ deployment/source-controller \\ --tail=200 Review kustomize-controller logs:\nkubectl logs \\ -n flux-system \\ deployment/kustomize-controller \\ --tail=200 Review helm-controller logs:\nkubectl logs \\ -n flux-system \\ deployment/helm-controller \\ --tail=200 Review notification-controller logs:\nkubectl logs \\ -n flux-system \\ deployment/notification-controller \\ --tail=200 Follow logs from a controller:\nkubectl logs \\ -n flux-system \\ deployment/kustomize-controller \\ --follow Stop log following with:\nCtrl+C Search recent kustomize-controller logs for errors:\nkubectl logs \\ -n flux-system \\ deployment/kustomize-controller \\ --since=30m | grep -iE \u0026#39;error|failed|timeout\u0026#39; Compare Git and Flux revisions # Display the latest local Git commit:\ngit log -1 \\ --oneline Display the current local branch:\ngit branch --show-current Check the working tree:\ngit status Fetch remote changes:\ngit fetch --prune Compare the local branch with its upstream:\ngit status -sb Display the Flux Git source revision:\nkubectl get gitrepository \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.artifact.revision}{\u0026#34;\\n\u0026#34;}\u0026#39; Display the Kustomization\u0026rsquo;s applied revision:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.lastAppliedRevision}{\u0026#34;\\n\u0026#34;}\u0026#39; These values help determine whether:\nGit contains the intended change The Flux source has fetched the latest revision The Kustomization applied that revision The intended resources exist in the cluster Local Kustomize render # Change into the GitOps repository:\ncd \u0026lt;path-to-gitops-repository\u0026gt; Render a Kustomize path using kubectl:\nkubectl kustomize \u0026lt;path-to-kustomization\u0026gt; Save the rendered output for inspection:\nkubectl kustomize \u0026lt;path-to-kustomization\u0026gt; \\ \u0026gt; /tmp/rendered-manifests.yaml Check that the output file is not empty:\ntest -s /tmp/rendered-manifests.yaml Review the output:\nless /tmp/rendered-manifests.yaml Search for expected resource kinds:\ngrep \u0026#39;^kind:\u0026#39; /tmp/rendered-manifests.yaml Remove the temporary file after review:\nrm /tmp/rendered-manifests.yaml Confirm the path before removing files.\nClient-side Kubernetes dry run # Render and perform a client-side validation:\nkubectl kustomize \u0026lt;path-to-kustomization\u0026gt; | kubectl apply \\ --dry-run=client \\ -f - This does not submit the resources to the Kubernetes API server.\nIt can identify issues such as:\nInvalid YAML Missing required fields Unknown local object structure Invalid generated resources Duplicate resource definitions Client-side dry runs do not validate all server-side policies or admission behaviour.\nServer-side dry run # Where supported and permitted, perform a server-side dry run:\nkubectl kustomize \u0026lt;path-to-kustomization\u0026gt; | kubectl apply \\ --server-side \\ --dry-run=server \\ -f - This submits the resources to the API server for validation without persisting the changes.\nA server-side dry run may evaluate:\nKubernetes API validation Custom resource schemas Admission policies Webhooks Server-side field management A server-side dry run communicates with the real cluster. Confirm the Kubernetes context before running it.\nFlux build validation # Build a Kustomization using Flux:\nflux build kustomization \u0026lt;kustomization-name\u0026gt; \\ --path \u0026lt;local-path\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Save the rendered Flux output:\nflux build kustomization \u0026lt;kustomization-name\u0026gt; \\ --path \u0026lt;local-path\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ \u0026gt; /tmp/flux-rendered.yaml Review it:\nless /tmp/flux-rendered.yaml Remove the temporary output after review:\nrm /tmp/flux-rendered.yaml Check the command options available in the installed Flux version:\nflux build kustomization --help Manual Git source reconciliation # State-changing operation\nThis command annotates the Flux resource and requests an immediate source reconciliation. It does not modify the Git repository.\nReconcile a Git source:\nflux reconcile source git \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Example using the standard Flux source:\nflux reconcile source git flux-system \\ -n flux-system Specify a timeout:\nflux reconcile source git \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ --timeout=2m After reconciliation, inspect the fetched revision:\nflux get source git \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Manual Kustomization reconciliation # State-changing operation\nThis command requests an immediate Kustomization reconciliation. Flux may create, update or delete cluster resources according to the declared configuration and pruning settings.\nReconcile a Kustomization:\nflux reconcile kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Reconcile the source first, then the Kustomization:\nflux reconcile kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ --with-source Specify a timeout:\nflux reconcile kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ --with-source \\ --timeout=5m Use --with-source when Flux should fetch the latest source revision before applying the Kustomization.\nManual HelmRelease reconciliation # State-changing operation\nThis command requests an immediate Helm reconciliation. Helm may install, upgrade, roll back or remediate a release according to the HelmRelease configuration.\nReconcile a HelmRelease:\nflux reconcile helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Reconcile its source first:\nflux reconcile helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ --with-source Specify a timeout:\nflux reconcile helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ --with-source \\ --timeout=10m After reconciliation:\nflux get helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Recommended reconciliation workflow # 1. Confirm the cluster context # kubectl config current-context 2. Confirm the Git working tree # git status 3. Confirm the intended commit exists # git log -1 \\ --oneline 4. Render the manifests # kubectl kustomize \u0026lt;path-to-kustomization\u0026gt; \\ \u0026gt; /tmp/rendered-manifests.yaml 5. Perform client-side validation # kubectl apply \\ --dry-run=client \\ -f /tmp/rendered-manifests.yaml 6. Perform server-side validation where appropriate # kubectl apply \\ --server-side \\ --dry-run=server \\ -f /tmp/rendered-manifests.yaml 7. Confirm the change has been pushed # git status -sb 8. Reconcile the source # flux reconcile source git \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; 9. Reconcile the Kustomization # flux reconcile kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; 10. Check the resulting status # flux get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; 11. Check affected workloads # kubectl get pods \\ -n \u0026lt;workload-namespace\u0026gt; \\ -o wide 12. Review recent events # kubectl get events \\ -n \u0026lt;workload-namespace\u0026gt; \\ --sort-by=.metadata.creationTimestamp 13. Compare the applied revision # kubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.lastAppliedRevision}{\u0026#34;\\n\u0026#34;}\u0026#39; 14. Remove the temporary rendered file # rm /tmp/rendered-manifests.yaml Troubleshooting # GitRepository is not ready # Inspect the source:\nkubectl describe gitrepository \u0026lt;source-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Check source-controller logs:\nkubectl logs \\ -n flux-system \\ deployment/source-controller \\ --since=30m Check for warning events:\nkubectl get events \\ -n \u0026lt;namespace\u0026gt; \\ --field-selector=type=Warning \\ --sort-by=.metadata.creationTimestamp Common causes include:\nInvalid Git URL Missing branch or tag Invalid credentials Expired deploy key DNS failure Network timeout Repository access restriction TLS validation error Do not display repository credential Secrets in public output.\nKustomization reports BuildFailed # Inspect it:\nkubectl describe kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Render the configured local path:\nkubectl kustomize \u0026lt;path-to-kustomization\u0026gt; Common causes include:\nInvalid YAML Missing resource file Incorrect relative path Duplicate resource Invalid patch target Unavailable Helm chart SOPS decryption failure Unsupported API version Kustomization reports DependencyNotReady # Display its dependencies:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.dependsOn}{\u0026#34;\\n\u0026#34;}\u0026#39; Check all Kustomizations:\nflux get kustomizations -A Investigate the dependency before reconciling the dependent resource.\nKustomization reports HealthCheckFailed # Describe the Kustomization:\nkubectl describe kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; List the target workloads:\nkubectl get deployments,statefulsets,daemonsets \\ -n \u0026lt;workload-namespace\u0026gt; Review the relevant pods:\nkubectl get pods \\ -n \u0026lt;workload-namespace\u0026gt; Review recent warning events:\nkubectl get events \\ -n \u0026lt;workload-namespace\u0026gt; \\ --field-selector=type=Warning \\ --sort-by=.metadata.creationTimestamp HelmRelease reconciliation fails # Inspect the release:\nkubectl describe helmrelease \u0026lt;release-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Review helm-controller logs:\nkubectl logs \\ -n flux-system \\ deployment/helm-controller \\ --since=30m Check the Helm source:\nflux get sources helm -A Check the resulting workload:\nkubectl get all \\ -n \u0026lt;workload-namespace\u0026gt; Common causes include:\nChart repository unavailable Invalid chart version Invalid values Failed Kubernetes hooks Unhealthy workloads Insufficient resources Storage provisioning failure Admission-policy rejection Flux fetched the latest commit but the application did not change # Check the Kustomization revision:\nflux get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Check whether the expected path is configured:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.path}{\u0026#34;\\n\u0026#34;}\u0026#39; Check dependencies:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.dependsOn}{\u0026#34;\\n\u0026#34;}\u0026#39; If the application is controlled by a HelmRelease, reconcile and inspect the HelmRelease rather than assuming that source reconciliation alone will recreate or upgrade the release.\nPost-reconciliation checks # Check all Flux resources:\nflux get all -A Check Flux controller pods:\nkubectl get pods \\ -n flux-system Check the affected namespace:\nkubectl get all \\ -n \u0026lt;workload-namespace\u0026gt; Check persistent volume claims:\nkubectl get persistentvolumeclaims \\ -n \u0026lt;workload-namespace\u0026gt; Check ingress:\nkubectl get ingress \\ -n \u0026lt;workload-namespace\u0026gt; Check warning events:\nkubectl get events \\ -n \u0026lt;workload-namespace\u0026gt; \\ --field-selector=type=Warning \\ --sort-by=.metadata.creationTimestamp Confirm the applied Git revision:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.status.lastAppliedRevision}{\u0026#34;\\n\u0026#34;}\u0026#39; Safety notes # Read-only commands in this entry include:\nflux check flux get flux events kubectl get kubectl describe kubectl logs kubectl kustomize git status git log git fetch The following commands initiate reconciliation and may change cluster resources:\nflux reconcile source git flux reconcile kustomization flux reconcile helmrelease Kustomizations with pruning enabled may delete managed resources that no longer exist in Git.\nHelmRelease reconciliation may install, upgrade, remediate or roll back workloads according to its declared configuration.\nBefore manual reconciliation:\nConfirm the Kubernetes context Confirm the intended Git branch Review the Git diff Render the relevant manifests Run client-side validation Use server-side dry run where appropriate Confirm the commit has been pushed Understand pruning and dependency behaviour Prepare a rollback plan for consequential changes Do not expose:\nGit credentials Deploy keys Personal access tokens SOPS private keys Kubernetes Secrets Private repository URLs Internal hostnames and addresses Related entries # Kubernetes Cluster Health Checks Kubernetes Workload Troubleshooting Kustomize Render Validation SOPS and age Reference HelmRelease Troubleshooting Flux Controller Health Checks Git Branch and Revision Checks References # Flux documentation Flux reconcile command ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/flux-reconciliation/","section":"Runbook","summary":"Time guide\nBasic Flux health check: approximately 5 to 10 minutes Failed reconciliation investigation: approximately 10 to 30 minutes Render, dry-run, reconciliation and validation: approximately 15 to 45 minutes The actual duration depends on repository size, deployment dependencies, controller health and the cause of any reconciliation failure.\n","title":"Flux Reconciliation Commands","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/fluxcd/","section":"Tags","summary":"","title":"FluxCD","type":"tags"},{"content":"Flux, Kustomize and SOPS references for validating and operating Git-managed Kubernetes environments.\nThe collection covers reconciliation, controller health, repository structure, application onboarding, rendering, dry runs, drift investigation and encrypted secrets.\nManual reconciliation is clearly identified because it may change Kubernetes resources. ``\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/","section":"Runbook","summary":"Flux, Kustomize and SOPS references for validating and operating Git-managed Kubernetes environments.\nThe collection covers reconciliation, controller health, repository structure, application onboarding, rendering, dry runs, drift investigation and encrypted secrets.\n","title":"GitOps","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/gitops/","section":"Tags","summary":"","title":"GitOps","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Compare Git, Flux revisions and live Kubernetes resources when desired and actual state differ.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # git status -sb git log -1 --oneline flux get sources all -A flux get kustomizations -A kubectl get kustomization \u0026lt;name\u0026gt; -n \u0026lt;namespace\u0026gt; -o jsonpath=\u0026#34;{.status.lastAppliedRevision}{\\n}\u0026#34; Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/gitops-drift-investigation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Compare Git, Flux revisions and live Kubernetes resources when desired and actual state differ.\n","title":"GitOps Drift Investigation","type":"lab-notes"},{"content":"Time guide\nRepository orientation: approximately 5 to 10 minutes Planning a new component: approximately 15 to 30 minutes Reviewing an existing layout: approximately 20 to 45 minutes Purpose # This entry describes a practical GitOps repository structure that separates desired state, cluster registration, operational documentation and historical evidence.\nThe layout is intended to make changes easier to review, render, reconcile, troubleshoot and recover.\nRecommended top-level structure # apps/ clusters/ docs/ infrastructure/ monitoring/ networking/ security/ Use each area consistently:\napps/ contains application-specific Kubernetes resources. clusters/ contains Flux entry points and cluster-level Kustomizations. docs/ contains architecture, decisions, Runbooks and implementation evidence. infrastructure/ contains shared platform components. monitoring/ contains observability resources. networking/ contains ingress, DNS and network-platform resources. security/ contains security platforms, controls and detection content. Application structure # A simple application can use:\napps/\u0026lt;application-name\u0026gt;/ ├── namespace.yaml ├── deployment.yaml ├── service.yaml ├── ingress.yaml └── kustomization.yaml A Helm-managed application can use:\napps/\u0026lt;application-name\u0026gt;/ ├── helmrepository.yaml ├── helmrelease.yaml ├── secret.enc.yaml └── kustomization.yaml Only include files that the application actually requires. Avoid empty placeholder files because they create false confidence that a component is fully documented or managed.\nCluster registration # Register applications and infrastructure through a cluster-specific directory:\nclusters/\u0026lt;cluster-name\u0026gt;/ ├── flux-system/ ├── applications.yaml ├── infrastructure.yaml └── kustomization.yaml A component-level Flux Kustomization should identify:\nThe source The repository path The reconciliation interval Pruning behaviour Dependencies Health checks where appropriate SOPS decryption where encrypted resources are present Standard change flow # Edit the source manifests. Render the affected Kustomization locally. Review the Git diff. Scan staged content for plaintext secrets. Commit and push the change. Wait for Flux or trigger reconciliation. Validate Flux status and Kubernetes resources. Record reusable procedures or implementation evidence. Render validation # Render a component before committing:\nkubectl kustomize apps/\u0026lt;application-name\u0026gt;/ \\ \u0026gt; /tmp/\u0026lt;application-name\u0026gt;-rendered.yaml Confirm output exists:\ntest -s /tmp/\u0026lt;application-name\u0026gt;-rendered.yaml Review resource kinds:\ngrep \u0026#39;^kind:\u0026#39; /tmp/\u0026lt;application-name\u0026gt;-rendered.yaml Run a client-side dry run:\nkubectl apply \\ --dry-run=client \\ -f /tmp/\u0026lt;application-name\u0026gt;-rendered.yaml Where appropriate, run a server-side dry run after confirming the cluster context:\nkubectl apply \\ --server-side \\ --dry-run=server \\ -f /tmp/\u0026lt;application-name\u0026gt;-rendered.yaml Remove the temporary render after review:\nrm /tmp/\u0026lt;application-name\u0026gt;-rendered.yaml Git review # Check the working tree:\ngit status --short Review unstaged changes:\ngit diff Review staged changes:\ngit diff --cached Search for common secret indicators while excluding encrypted secret files:\ngrep -R -nEi \\ \u0026#39;password:|token:|api[_-]?key:|BEGIN .*PRIVATE KEY\u0026#39; . \\ --exclude-dir=.git \\ --exclude=\u0026#39;*.enc.yaml\u0026#39; Review every match manually. Documentation and environment-variable references can produce legitimate matches.\nReconciliation checks # flux get sources all -A flux get kustomizations -A flux get helmreleases -A kubectl get pods -A Documentation structure # docs/ ├── architecture/ ├── decisions/ ├── phases/ ├── runbooks/ ├── service-catalogue/ └── disaster-recovery/ Use:\nArchitecture documents for current design. Decision records for why a technology or pattern was selected. Phase documents for historical implementation evidence. Runbooks for reusable operations. Service catalogue pages for ownership and dependencies. Disaster-recovery documents for rebuild and restore procedures. Safety notes # Never commit plaintext credentials. Do not assume an encrypted file contains correct plaintext. Preserve the final desired state in Git after an emergency direct change. Treat rendered manifests as potentially sensitive. Review pruning before reconciling a Kustomization. Keep private keys and recovery credentials outside Git. Related entries # Flux Reconciliation Commands Kustomize Render and Dry-Run Validation SOPS and age Secret Workflow Application GitOps Onboarding GitOps Drift Investigation ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/gitops-repository-structure/","section":"Runbook","summary":"Time guide\nRepository orientation: approximately 5 to 10 minutes Planning a new component: approximately 15 to 30 minutes Reviewing an existing layout: approximately 20 to 45 minutes Purpose # This entry describes a practical GitOps repository structure that separates desired state, cluster registration, operational documentation and historical evidence.\n","title":"GitOps Repository Structure","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/gpresult/","section":"Tags","summary":"","title":"Gpresult","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/grafana/","section":"Tags","summary":"","title":"Grafana","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Validate Grafana workload health and connectivity to Prometheus and Loki data sources.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get pods,svc,ingress -n \u0026lt;grafana-namespace\u0026gt; kubectl logs -n \u0026lt;grafana-namespace\u0026gt; deployment/\u0026lt;grafana-deployment\u0026gt; --tail=200 curl -fsS http://\u0026lt;prometheus-service\u0026gt;.\u0026lt;namespace\u0026gt;.svc:\u0026lt;port\u0026gt;/-/ready Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/monitoring/grafana-data-source-validation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Validate Grafana workload health and connectivity to Prometheus and Loki data sources.\n","title":"Grafana Data Source Validation","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/group-policy/","section":"Tags","summary":"","title":"Group Policy","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Capture applied computer and user policy and investigate processing failures. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # gpresult /r gpresult /h \u0026#34;$env:TEMP\\gpresult.html\u0026#34; Get-WinEvent -LogName \u0026#34;Microsoft-Windows-GroupPolicy/Operational\u0026#34; -MaxEvents 100 Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/group-policy-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Capture applied computer and user policy and investigate processing failures. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Group Policy Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/identity/","section":"Tags","summary":"","title":"Identity","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/incident-response/","section":"Tags","summary":"","title":"Incident Response","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/incident-triage/","section":"Tags","summary":"","title":"Incident Triage","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/ingress/","section":"Tags","summary":"","title":"Ingress","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/intune/","section":"Tags","summary":"","title":"Intune","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for enrollment, compliance, sync, ownership and managed-device investigation. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Connect-MgGraph -Scopes DeviceManagementManagedDevices.Read.All Get-MgDeviceManagementManagedDevice -Filter \u0026#34;deviceName eq \u0026#39;\u0026lt;device-name\u0026gt;\u0026#39;\u0026#34; Get-MgDeviceManagementManagedDevice -ManagedDeviceId \u0026lt;device-id\u0026gt; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/intune-device-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for enrollment, compliance, sync, ownership and managed-device investigation. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Intune Device Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/investigation/","section":"Tags","summary":"","title":"Investigation","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/k3s/","section":"Tags","summary":"","title":"K3s","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/kasten/","section":"Tags","summary":"","title":"Kasten","type":"tags"},{"content":"Operational and troubleshooting references for Kubernetes and k3s environments.\nThe collection covers cluster health, nodes, workloads, resource usage, persistent storage and Longhorn.\nThe comprehensive cluster-health entry provides initial triage, while focused entries support deeper investigation.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/kubernetes/","section":"Runbook","summary":"Operational and troubleshooting references for Kubernetes and k3s environments.\nThe collection covers cluster health, nodes, workloads, resource usage, persistent storage and Longhorn.\nThe comprehensive cluster-health entry provides initial triage, while focused entries support deeper investigation.\n","title":"Kubernetes","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review layered recovery using Git, etcd, Longhorn and application-aware backups.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # git status --short sudo k3s etcd-snapshot ls kubectl get pvc -A kubectl get backups.longhorn.io -n longhorn-system kubectl get policies.actions.kio.kasten.io -A Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/virtualisation/backup-restore-operations/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review layered recovery using Git, etcd, Longhorn and application-aware backups.\n","title":"Kubernetes Backup and Restore Operations","type":"lab-notes"},{"content":"Time guide\nInitial node triage: approximately 5 to 10 minutes Resource-pressure investigation: approximately 15 to 30 minutes Host-level investigation: approximately 20 to 45 minutes Purpose # Use this entry when a node is NotReady, workloads cannot schedule, resource pressure is suspected, or platform capacity needs review.\nCluster node summary # kubectl get nodes -o wide kubectl get nodes \\ -o custom-columns=\u0026#39;NAME:.metadata.name,READY:.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].status,MEMORY:.status.conditions[?(@.type==\u0026#34;MemoryPressure\u0026#34;)].status,DISK:.status.conditions[?(@.type==\u0026#34;DiskPressure\u0026#34;)].status,PID:.status.conditions[?(@.type==\u0026#34;PIDPressure\u0026#34;)].status,UNSCHEDULABLE:.spec.unschedulable\u0026#39; Healthy nodes normally show Ready=True and all pressure conditions as False.\nCapacity and allocatable resources # kubectl describe node \u0026lt;node-name\u0026gt; kubectl get node \u0026lt;node-name\u0026gt; \\ -o jsonpath=\u0026#39;{.status.capacity}{\u0026#34;\\n\u0026#34;}{.status.allocatable}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl top nodes If metrics are unavailable:\nkubectl get apiservice v1beta1.metrics.k8s.io Taints and scheduling # kubectl get nodes \\ -o custom-columns=\u0026#39;NAME:.metadata.name,TAINTS:.spec.taints\u0026#39; kubectl get node \u0026lt;node-name\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.unschedulable}{\u0026#34;\\n\u0026#34;}\u0026#39; Review workloads assigned to a node:\nkubectl get pods -A \\ --field-selector spec.nodeName=\u0026lt;node-name\u0026gt; \\ -o wide Recent node events # kubectl get events -A \\ --field-selector involvedObject.kind=Node,involvedObject.name=\u0026lt;node-name\u0026gt; \\ --sort-by=.metadata.creationTimestamp Host-level checks # Run on the affected node:\nuptime free -h df -h df -ih systemctl --failed For a k3s server:\nsudo systemctl status k3s --no-pager --full sudo journalctl -u k3s --since \u0026#34;30 minutes ago\u0026#34; --no-pager For a k3s agent:\nsudo systemctl status k3s-agent --no-pager --full sudo journalctl -u k3s-agent --since \u0026#34;30 minutes ago\u0026#34; --no-pager Check kernel warnings:\nsudo journalctl -k -p warning --since \u0026#34;30 minutes ago\u0026#34; --no-pager Common findings # MemoryPressure # Check memory and high-usage workloads:\nfree -h kubectl top pods -A --sort-by=memory DiskPressure # df -h df -ih sudo du -xhd1 /var/lib 2\u0026gt;/dev/null | sort -h The du command can take time on large filesystems but is read-only.\nPIDPressure # ps -e --no-headers | wc -l ps aux --sort=-%cpu | head NodeNotReady # Check:\nNetwork connectivity k3s service state Disk availability Memory pressure Time synchronisation Container runtime logs Recent kernel faults Safety notes # This entry intentionally excludes cordon, drain, restart and reboot commands. Those actions can move or interrupt workloads and should only follow diagnosis, capacity review and a rollback plan.\nRelated entries # Kubernetes Cluster Health Checks Pod and Workload Troubleshooting Kubernetes Resource Requests and Limits k3s Server and Agent Administration ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/kubernetes/node-health-resource-pressure/","section":"Runbook","summary":"Time guide\nInitial node triage: approximately 5 to 10 minutes Resource-pressure investigation: approximately 15 to 30 minutes Host-level investigation: approximately 20 to 45 minutes Purpose # Use this entry when a node is NotReady, workloads cannot schedule, resource pressure is suspected, or platform capacity needs review.\n","title":"Kubernetes Node Health and Resource Pressure","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate PersistentVolumes, claims, storage classes, attachments and mount failures.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get storageclasses kubectl get pv kubectl get pvc -A kubectl get volumeattachments kubectl get events -A --field-selector=type=Warning --sort-by=.metadata.creationTimestamp Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/kubernetes/persistent-volume-checks/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate PersistentVolumes, claims, storage classes, attachments and mount failures.\n","title":"Kubernetes Persistent Volume Checks","type":"lab-notes"},{"content":"Time guide\nInitial pod triage: approximately 5 to 10 minutes Restart or scheduling investigation: approximately 15 to 30 minutes Dependency or storage investigation: approximately 30 to 60 minutes Purpose # Use this entry to investigate pods in Pending, Failed, Unknown, CrashLoopBackOff, ImagePullBackOff or repeated restart states.\nLocate unhealthy workloads # kubectl get pods -A -o wide kubectl get pods -A \\ --field-selector=status.phase!=Running,status.phase!=Succeeded Review restart counts:\nkubectl get pods -A \\ -o custom-columns=\u0026#39;NAMESPACE:.metadata.namespace,POD:.metadata.name,PHASE:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,NODE:.spec.nodeName\u0026#39; Describe the pod # kubectl describe pod \u0026lt;pod-name\u0026gt; -n \u0026lt;namespace\u0026gt; Pay attention to:\nContainer state and reason Last termination state Readiness and liveness probes Volumes and mounts Node assignment Conditions Events Logs # Current logs:\nkubectl logs \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ --all-containers=true \\ --tail=200 Previous container instance:\nkubectl logs \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -c \u0026lt;container-name\u0026gt; \\ --previous \\ --tail=200 Follow logs:\nkubectl logs \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -c \u0026lt;container-name\u0026gt; \\ --follow Stop with Ctrl+C.\nEvents # kubectl get events -n \u0026lt;namespace\u0026gt; \\ --sort-by=.metadata.creationTimestamp kubectl get events -n \u0026lt;namespace\u0026gt; \\ --field-selector=type=Warning \\ --sort-by=.metadata.creationTimestamp Pending pods # Check scheduling details:\nkubectl describe pod \u0026lt;pod-name\u0026gt; -n \u0026lt;namespace\u0026gt; Review nodes and taints:\nkubectl get nodes kubectl get nodes -o custom-columns=\u0026#39;NAME:.metadata.name,TAINTS:.spec.taints\u0026#39; Review claims:\nkubectl get pvc -n \u0026lt;namespace\u0026gt; Common causes:\nInsufficient CPU or memory Node selectors or affinity Taints without tolerations Unbound persistent volume claims Missing storage classes Admission-policy rejection CrashLoopBackOff # kubectl get pod \u0026lt;pod-name\u0026gt; -n \u0026lt;namespace\u0026gt; -o yaml kubectl logs \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -c \u0026lt;container-name\u0026gt; \\ --previous \\ --tail=200 Look for:\nApplication configuration faults Missing Secrets or ConfigMaps Permission errors Probe failures OOMKilled Failed mounts Dependency failures ImagePullBackOff # kubectl describe pod \u0026lt;pod-name\u0026gt; -n \u0026lt;namespace\u0026gt; Check the configured image:\nkubectl get pod \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.containers[*].image}{\u0026#34;\\n\u0026#34;}\u0026#39; Check:\nImage name and tag Registry availability Architecture compatibility Pull Secret references Registry credentials Rate limits Workload controllers # kubectl get deployments,statefulsets,daemonsets -n \u0026lt;namespace\u0026gt; kubectl describe deployment \u0026lt;deployment-name\u0026gt; -n \u0026lt;namespace\u0026gt; kubectl rollout status deployment/\u0026lt;deployment-name\u0026gt; -n \u0026lt;namespace\u0026gt; rollout status is read-only and waits for status until completion or timeout.\nDependencies # kubectl get configmaps,secrets,services,pvc -n \u0026lt;namespace\u0026gt; List Secret keys without values:\nkubectl get secret \u0026lt;secret-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o go-template=\u0026#39;{{range $key, $value := .data}}{{$key}}{{\u0026#34;\\n\u0026#34;}}{{end}}\u0026#39; Safety notes # This entry excludes delete, restart, scale, patch and edit commands. Recreating a pod can temporarily hide evidence. Capture descriptions, logs and events before taking a state-changing action.\nRelated entries # Kubernetes Cluster Health Checks Kubernetes Node Health and Resource Pressure Services and Ingress Troubleshooting Persistent Volume Checks Kubernetes Events and Diagnostics ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/kubernetes/pod-workload-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial pod triage: approximately 5 to 10 minutes Restart or scheduling investigation: approximately 15 to 30 minutes Dependency or storage investigation: approximately 30 to 60 minutes Purpose # Use this entry to investigate pods in Pending, Failed, Unknown, CrashLoopBackOff, ImagePullBackOff or repeated restart states.\n","title":"Kubernetes Pod and Workload Troubleshooting","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review Kubernetes CPU and memory requests, limits, live usage and scheduling impact.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get pods -A -o custom-columns=\u0026#34;NAMESPACE:.metadata.namespace,POD:.metadata.name,CPU_REQUEST:.spec.containers[*].resources.requests.cpu,CPU_LIMIT:.spec.containers[*].resources.limits.cpu,MEMORY_REQUEST:.spec.containers[*].resources.requests.memory,MEMORY_LIMIT:.spec.containers[*].resources.limits.memory\u0026#34; kubectl top nodes kubectl top pods -A --containers Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/kubernetes/resource-requests-limits/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review Kubernetes CPU and memory requests, limits, live usage and scheduling impact.\n","title":"Kubernetes Resource Requests and Limits","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/kustomize/","section":"Tags","summary":"","title":"Kustomize","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Validate desired-state manifests locally and against the Kubernetes API without persisting changes.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl kustomize \u0026lt;path\u0026gt; \u0026gt; /tmp/rendered.yaml test -s /tmp/rendered.yaml kubectl apply --dry-run=client -f /tmp/rendered.yaml kubectl apply --server-side --dry-run=server -f /tmp/rendered.yaml Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/render-dry-run-validation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Validate desired-state manifests locally and against the Kubernetes API without persisting changes.\n","title":"Kustomize Render and Dry-Run Validation","type":"lab-notes"},{"content":"Linux support references covering system health, storage, filesystems, networking, DNS, SSH, services, logs and common command-line operations.\nThe collection includes both quick-reference pages and structured troubleshooting workflows.\nPotentially disruptive administration commands are separated from read-only checks.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/linux/","section":"Runbook","summary":"Linux support references covering system health, storage, filesystems, networking, DNS, SSH, services, logs and common command-line operations.\nThe collection includes both quick-reference pages and structured troubleshooting workflows.\nPotentially disruptive administration commands are separated from read-only checks.\n","title":"Linux","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/linux/","section":"Tags","summary":"","title":"Linux","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Quick reference for files, processes, storage, services, networking, users and logs. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # uname -a uptime free -h lsblk -f df -h systemctl --failed journalctl -p warning --since \u0026#34;30 minutes ago\u0026#34; --no-pager ip address ip route Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/linux/linux-command-quick-reference/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Quick reference for files, processes, storage, services, networking, users and logs. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Linux Command Quick Reference","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only checks for devices, filesystems, mounts, capacity, inodes and high-usage paths. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # lsblk -f findmnt df -h df -ih sudo du -xhd1 /var 2\u0026gt;/dev/null | sort -h Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/linux/disk-filesystem-management/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only checks for devices, filesystems, mounts, capacity, inodes and high-usage paths. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Linux Disk and Filesystem Checks","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for links, addressing, routes, resolvers, DNS and TCP connectivity. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # ip link ip address ip route resolvectl status getent hosts \u0026lt;hostname\u0026gt; dig \u0026lt;hostname\u0026gt; nc -vz \u0026lt;hostname\u0026gt; \u0026lt;port\u0026gt; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/linux/linux-network-dns/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for links, addressing, routes, resolvers, DNS and TCP connectivity. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Linux Network and DNS Troubleshooting","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review uptime, CPU, memory, storage, services, logs and network state on Linux.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # uptime free -h df -h df -ih systemctl --failed journalctl -p warning --since \u0026#34;30 minutes ago\u0026#34; --no-pager Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/linux/linux-system-health/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review uptime, CPU, memory, storage, services, logs and network state on Linux.\n","title":"Linux System Health Checks","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/logging/","section":"Tags","summary":"","title":"Logging","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/logs/","section":"Tags","summary":"","title":"Logs","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/loki/","section":"Tags","summary":"","title":"Loki","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate missing logs, ingestion failures, DaemonSet coverage and Loki readiness.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get pods,svc,pvc -n \u0026lt;logging-namespace\u0026gt; kubectl get daemonset -n \u0026lt;logging-namespace\u0026gt; kubectl logs -n \u0026lt;logging-namespace\u0026gt; \u0026lt;loki-pod\u0026gt; --tail=200 kubectl logs -n \u0026lt;logging-namespace\u0026gt; \u0026lt;promtail-pod\u0026gt; --tail=200 Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/monitoring/loki-promtail-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate missing logs, ingestion failures, DaemonSet coverage and Loki readiness.\n","title":"Loki and Promtail Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/longhorn/","section":"Tags","summary":"","title":"Longhorn","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Validate Longhorn ServiceMonitor discovery and storage dashboard data.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get svc -n longhorn-system --show-labels kubectl get servicemonitors -A | grep -i longhorn kubectl get endpointslices -n longhorn-system Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/monitoring/longhorn-metrics-validation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Validate Longhorn ServiceMonitor discovery and storage dashboard data.\n","title":"Longhorn Metrics Validation","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Confirm Longhorn recurring jobs, snapshots, backups and backup-target health.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get recurringjobs.longhorn.io -n longhorn-system kubectl get backupvolumes.longhorn.io -n longhorn-system kubectl get backups.longhorn.io -n longhorn-system kubectl logs -n longhorn-system -l app=longhorn-manager --tail=200 Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/virtualisation/longhorn-snapshot-backup-checks/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Confirm Longhorn recurring jobs, snapshots, backups and backup-target health.\n","title":"Longhorn Snapshot and Backup Checks","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Check Longhorn workloads, volumes, replicas, nodes, capacity and warning events.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get pods -n longhorn-system -o wide kubectl get volumes.longhorn.io -n longhorn-system kubectl get replicas.longhorn.io -n longhorn-system kubectl get nodes.longhorn.io -n longhorn-system kubectl get events -n longhorn-system --field-selector=type=Warning --sort-by=.metadata.creationTimestamp Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/kubernetes/longhorn-storage-health/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Check Longhorn workloads, volumes, replicas, nodes, capacity and warning events.\n","title":"Longhorn Storage Health Checks","type":"lab-notes"},{"content":"macOS workplace-support references using Terminal, unified logging and Apple diagnostic tools.\nThe collection covers system health, hardware, storage, networking, DNS, processes, updates and diagnostic logging.\nSystem identifiers, usernames and diagnostic output should be sanitised before sharing.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/macos/","section":"Runbook","summary":"macOS workplace-support references using Terminal, unified logging and Apple diagnostic tools.\nThe collection covers system health, hardware, storage, networking, DNS, processes, updates and diagnostic logging.\nSystem identifiers, usernames and diagnostic output should be sanitised before sharing.\n","title":"macOS","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/macos/","section":"Tags","summary":"","title":"MacOS","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Use unified logging and Apple Diagnostics to investigate software and hardware faults. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # log show --last 30m --style compact log show --predicate \u0026#39;process == \u0026#34;\u0026lt;process-name\u0026gt;\u0026#34;\u0026#39; --last 1h system_profiler SPDiagnosticsDataType Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/macos/macos-logs-diagnostics/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Use unified logging and Apple Diagnostics to investigate software and hardware faults. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"macOS Logs and Apple Diagnostics","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for interfaces, services, routes, DNS, Wi-Fi and TCP connectivity. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # networksetup -listallnetworkservices networksetup -getinfo \u0026#34;\u0026lt;network-service\u0026gt;\u0026#34; ifconfig route -n get default scutil --dns dscacheutil -q host -a name \u0026lt;hostname\u0026gt; nc -vz \u0026lt;hostname\u0026gt; \u0026lt;port\u0026gt; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/macos/macos-network-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for interfaces, services, routes, DNS, Wi-Fi and TCP connectivity. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"macOS Network and DNS Troubleshooting","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Quick reference for macOS system, disk, network, process, update and log checks. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # sw_vers system_profiler SPSoftwareDataType SPHardwareDataType diskutil list df -h networksetup -listallhardwareports scutil --dns route -n get default log show --last 30m --style compact Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/macos/macos-support-command-reference/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Quick reference for macOS system, disk, network, process, update and log checks. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"macOS Support Command Reference","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only checks for platform version, hardware, storage, uptime, processes and updates. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # sw_vers uptime system_profiler SPHardwareDataType diskutil info / df -h top -l 1 softwareupdate --list Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/macos/macos-system-health/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only checks for platform version, hardware, storage, uptime, processes and updates. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"macOS System Health Checks","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/mail-flow/","section":"Tags","summary":"","title":"Mail Flow","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for investigating delayed, rejected or missing Exchange Online messages. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-MessageTraceV2 -SenderAddress \u0026lt;sender\u0026gt; -StartDate (Get-Date).AddDays(-2) -EndDate (Get-Date) Get-MessageTraceDetailV2 -MessageTraceId \u0026lt;trace-id\u0026gt; -RecipientAddress \u0026lt;recipient\u0026gt; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/mail-flow-message-trace/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for investigating delayed, rejected or missing Exchange Online messages. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Mail Flow and Message Trace","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/mailbox/","section":"Tags","summary":"","title":"Mailbox","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review Full Access, Send As, Send on Behalf and shared mailbox delegation. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-MailboxPermission -Identity \u0026lt;mailbox\u0026gt; Get-RecipientPermission -Identity \u0026lt;mailbox\u0026gt; Get-Mailbox \u0026lt;mailbox\u0026gt; | Select -Expand GrantSendOnBehalfTo Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/mailbox-permissions-delegation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review Full Access, Send As, Send on Behalf and shared mailbox delegation. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Mailbox Permissions and Delegation","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/message-trace/","section":"Tags","summary":"","title":"Message Trace","type":"tags"},{"content":"Commands and structured workflows for supporting Microsoft 365 services and identities.\nThe collection covers Exchange Online, mail flow, mailbox permissions, Entra ID, Intune, Microsoft Defender, Microsoft Graph and service-health investigation.\nAll examples use placeholder users, domains, devices and tenant information.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/","section":"Runbook","summary":"Commands and structured workflows for supporting Microsoft 365 services and identities.\nThe collection covers Exchange Online, mail flow, mailbox permissions, Entra ID, Intune, Microsoft Defender, Microsoft Graph and service-health investigation.\n","title":"Microsoft 365","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/microsoft-365/","section":"Tags","summary":"","title":"Microsoft 365","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review tenant service health, advisories and workload-specific diagnostics. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # Connect-MgGraph -Scopes ServiceHealth.Read.All Get-MgServiceAnnouncementHealthOverview Get-MgServiceAnnouncementIssue Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/service-health-checks/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Review tenant service health, advisories and workload-specific diagnostics. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Microsoft 365 Service Health Checks","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/microsoft-defender/","section":"Tags","summary":"","title":"Microsoft Defender","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Check Microsoft Defender Antivirus health, versions, services and security intelligence freshness. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # Get-MpComputerStatus Get-Service WinDefend,WdNisSvc,Sense Update-MpSignature \u0026amp; \u0026#34;$env:ProgramFiles\\Windows Defender\\MpCmdRun.exe\u0026#34; -SignatureUpdate Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/defender-health-updates/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Check Microsoft Defender Antivirus health, versions, services and security intelligence freshness. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Microsoft Defender Health and Updates","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # A read-only first workflow for triaging Microsoft Defender alerts and affected entities. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Connect-MgGraph -Scopes SecurityAlert.Read.All,SecurityIncident.Read.All Get-MgSecurityAlertV2 -SecurityAlertId \u0026lt;alert-id\u0026gt; Get-MgSecurityIncident -SecurityIncidentId \u0026lt;incident-id\u0026gt; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/microsoft-365/defender-initial-investigation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # A read-only first workflow for triaging Microsoft Defender alerts and affected entities. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Microsoft Defender Initial Investigation","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Run controlled scans and review Defender threats and detections. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-MpThreat Get-MpThreatDetection Start-MpScan -ScanType QuickScan Get-MpPreference Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/defender-scan-detection/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Run controlled scans and review Defender threats and detections. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Microsoft Defender Scan and Detection Checks","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/microsoft-graph/","section":"Tags","summary":"","title":"Microsoft Graph","type":"tags"},{"content":"Monitoring and observability references for metrics, dashboards, logs and storage visibility.\nThe collection covers Prometheus, Grafana, Loki, Promtail and Longhorn metrics.\nA working dashboard does not alone prove that targets and data sources are current, so entries include source-level validation.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/monitoring/","section":"Runbook","summary":"Monitoring and observability references for metrics, dashboards, logs and storage visibility.\nThe collection covers Prometheus, Grafana, Loki, Promtail and Longhorn metrics.\nA working dashboard does not alone prove that targets and data sources are current, so entries include source-level validation.\n","title":"Monitoring","type":"lab-notes"},{"content":"Cross-platform workflows for diagnosing DNS, DHCP, Wi-Fi, VPN, routing and service-connectivity problems.\nThe collection includes general workplace-support commands alongside UniFi, ingress and infrastructure-focused references.\nOnly systems and destinations that the operator is authorised to test should be investigated.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/","section":"Runbook","summary":"Cross-platform workflows for diagnosing DNS, DHCP, Wi-Fi, VPN, routing and service-connectivity problems.\nThe collection includes general workplace-support commands alongside UniFi, ingress and infrastructure-focused references.\nOnly systems and destinations that the operator is authorised to test should be investigated.\n","title":"Networking","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/networking/","section":"Tags","summary":"","title":"Networking","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/nodes/","section":"Tags","summary":"","title":"Nodes","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/observability/","section":"Tags","summary":"","title":"Observability","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/onboarding/","section":"Tags","summary":"","title":"Onboarding","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/operations/","section":"Tags","summary":"","title":"Operations","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/permissions/","section":"Tags","summary":"","title":"Permissions","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/pods/","section":"Tags","summary":"","title":"Pods","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/power/","section":"Tags","summary":"","title":"Power","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/powershell/","section":"Tags","summary":"","title":"PowerShell","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/projects/","section":"Tags","summary":"","title":"Projects","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/prometheus/","section":"Tags","summary":"","title":"Prometheus","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate Prometheus targets, ServiceMonitor discovery and scrape configuration.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get pods -n \u0026lt;monitoring-namespace\u0026gt; kubectl get servicemonitors -A kubectl get prometheusrules -A kubectl get services -A --show-labels Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/monitoring/prometheus-target-health/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate Prometheus targets, ServiceMonitor discovery and scrape configuration.\n","title":"Prometheus Target Health Checks","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/promtail/","section":"Tags","summary":"","title":"Promtail","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/pvc/","section":"Tags","summary":"","title":"PVC","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/reconciliation/","section":"Tags","summary":"","title":"Reconciliation","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/recovery/","section":"Tags","summary":"","title":"Recovery","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/remote-access/","section":"Tags","summary":"","title":"Remote Access","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/repository-design/","section":"Tags","summary":"","title":"Repository Design","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/resources/","section":"Tags","summary":"","title":"Resources","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/rules/","section":"Tags","summary":"","title":"Rules","type":"tags"},{"content":"A personal technical reference containing reusable commands, procedures, troubleshooting notes and lessons learned.\nContent will be organised into practical subject areas, including:\nKubernetes and k3s FluxCD and GitOps Linux administration PowerShell Microsoft 365 Networking Proxmox Wazuh and security Monitoring and observability Troubleshooting Only sanitised information suitable for public sharing will be published.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/","section":"Runbook","summary":"A personal technical reference containing reusable commands, procedures, troubleshooting notes and lessons learned.\nContent will be organised into practical subject areas, including:\nKubernetes and k3s FluxCD and GitOps Linux administration PowerShell Microsoft 365 Networking Proxmox Wazuh and security Monitoring and observability Troubleshooting Only sanitised information suitable for public sharing will be published.\n","title":"Runbook","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/secrets/","section":"Tags","summary":"","title":"Secrets","type":"tags"},{"content":"Security-operations references for endpoint monitoring, alert triage, detection validation and incident investigation.\nThe collection covers Wazuh onboarding, agent troubleshooting, detection tuning, alert severity and structured incident triage.\nSensitive events, identities, credentials and security configurations are excluded from public examples.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/security/","section":"Runbook","summary":"Security-operations references for endpoint monitoring, alert triage, detection validation and incident investigation.\nThe collection covers Wazuh onboarding, agent troubleshooting, detection tuning, alert severity and structured incident triage.\nSensitive events, identities, credentials and security configurations are excluded from public examples.\n","title":"Security","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Classify informational, low, medium, high and critical alerts consistently.\nSeverity model # Informational: useful context with no immediate security impact. Low: limited risk or low-confidence suspicious activity. Medium: credible suspicious activity or a meaningful control failure. High: strong evidence of malicious activity, privilege abuse or material service risk. Critical: active or confirmed compromise with severe impact or urgent threat. Read-only checks # Use the authorised security consoles and read-only queries available for the affected platform. Record every query and preserve timestamps.\nExpected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/security/alert-severity-classification/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Classify informational, low, medium, high and critical alerts consistently.\n","title":"Security Alert Severity Classification","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Validate alerts, establish scope, classify severity and preserve investigation evidence.\nTriage workflow # Record the detection time, source and identifier. Validate source health and timestamp accuracy. Establish the affected account, asset and scope using sanitised references. Review related identity, endpoint, network and change evidence. Classify the outcome and confidence. Escalate, contain or close according to authorised procedures. Record lessons learned and detection improvements. Read-only checks # Use the authorised security consoles and read-only queries available for the affected platform. Record every query and preserve timestamps.\nExpected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/security/security-incident-triage/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Validate alerts, establish scope, classify severity and preserve investigation evidence.\n","title":"Security Incident Triage","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/security-intelligence/","section":"Tags","summary":"","title":"Security Intelligence","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/security-operations/","section":"Tags","summary":"","title":"Security Operations","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/sense/","section":"Tags","summary":"","title":"Sense","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/service-health/","section":"Tags","summary":"","title":"Service Health","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/servicemonitor/","section":"Tags","summary":"","title":"ServiceMonitor","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/services/","section":"Tags","summary":"","title":"Services","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/severity/","section":"Tags","summary":"","title":"Severity","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/sfc/","section":"Tags","summary":"","title":"SFC","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/shared-mailbox/","section":"Tags","summary":"","title":"Shared Mailbox","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/soc/","section":"Tags","summary":"","title":"SOC","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/sops/","section":"Tags","summary":"","title":"SOPS","type":"tags"},{"content":"Time guide\nValidate an existing encrypted Secret: approximately 5 to 10 minutes Create and reconcile a new encrypted Secret: approximately 15 to 30 minutes Investigate a decryption fault: approximately 20 to 45 minutes Purpose # This entry describes a safe workflow for storing encrypted Kubernetes Secret manifests in Git using SOPS and age, with Flux performing decryption during reconciliation.\nSecurity principles # Never commit an age private key. Never commit a plaintext Kubernetes Secret manifest. Commit only encrypted Secret manifests. Keep protected recovery copies of the age private key outside Git. Validate plaintext before encryption and the resulting Secret after reconciliation. Remove temporary plaintext files immediately. Requirements # sops --version age --version flux version kubectl version --client Confirm the active cluster context:\nkubectl config current-context Confirm SOPS configuration # cat .sops.yaml The configuration should target the intended encrypted-secret paths and use the expected public age recipient.\nThe age recipient is public. The age identity or private key must remain protected outside Git.\nValidate an existing encrypted file # sops filestatus path/to/\u0026lt;secret-name\u0026gt;.enc.yaml Test decryption without printing plaintext:\nsops -d path/to/\u0026lt;secret-name\u0026gt;.enc.yaml \u0026gt; /dev/null Confirm the encrypted file does not contain an expected plaintext marker:\n! grep -q \u0026#39;\u0026lt;known-plaintext-marker\u0026gt;\u0026#39; path/to/\u0026lt;secret-name\u0026gt;.enc.yaml Create a new encrypted Secret # Create the plaintext file outside the repository where practical:\numask 077 cat \u0026gt; /tmp/\u0026lt;secret-name\u0026gt;.yaml \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; apiVersion: v1 kind: Secret metadata: name: \u0026lt;secret-name\u0026gt; namespace: \u0026lt;namespace\u0026gt; type: Opaque stringData: \u0026lt;key-name\u0026gt;: \u0026lt;secret-value\u0026gt; EOF Encrypt directly into the repository:\nsops --encrypt \\ /tmp/\u0026lt;secret-name\u0026gt;.yaml \\ \u0026gt; path/to/\u0026lt;secret-name\u0026gt;.enc.yaml Remove the plaintext file:\nshred -u /tmp/\u0026lt;secret-name\u0026gt;.yaml If shred is not suitable for the underlying storage, securely remove the file according to the platform and storage characteristics.\nValidate before commit # sops filestatus path/to/\u0026lt;secret-name\u0026gt;.enc.yaml sops -d path/to/\u0026lt;secret-name\u0026gt;.enc.yaml \u0026gt; /dev/null git status --short git diff -- path/to/\u0026lt;secret-name\u0026gt;.enc.yaml Confirm no plaintext copy exists:\nfind . -type f \\ ! -name \u0026#39;*.enc.yaml\u0026#39; \\ ! -path \u0026#39;./.git/*\u0026#39; \\ -exec grep -Il \u0026#39;\u0026lt;known-plaintext-marker\u0026gt;\u0026#39; {} + Expected result: no output.\nReference the encrypted file # Add the encrypted manifest to the relevant kustomization.yaml:\nresources: - \u0026lt;secret-name\u0026gt;.enc.yaml Confirm the Flux Kustomization is configured for SOPS decryption. Inspect it without printing Secret values:\nkubectl get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;flux-namespace\u0026gt; \\ -o yaml Render and dry-run considerations # A normal Kustomize render preserves SOPS-encrypted content because Kustomize does not decrypt it automatically:\nkubectl kustomize path/to/component Use SOPS-aware validation for plaintext structure when necessary, but do not write decrypted output into the repository.\nCommit and reconcile # git add .sops.yaml path/to/kustomization.yaml path/to/\u0026lt;secret-name\u0026gt;.enc.yaml git diff --cached --check git diff --cached git commit -m \u0026#34;Add encrypted \u0026lt;component\u0026gt; secret\u0026#34; git push Reconcile the affected resource:\nflux reconcile kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;flux-namespace\u0026gt; \\ --with-source Post-reconciliation validation # flux get kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;flux-namespace\u0026gt; Confirm the Secret exists without printing data:\nkubectl get secret \u0026lt;secret-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Confirm expected keys exist without decoding values:\nkubectl get secret \u0026lt;secret-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o go-template=\u0026#39;{{range $key, $value := .data}}{{$key}}{{\u0026#34;\\n\u0026#34;}}{{end}}\u0026#39; Validate the consuming workload:\nkubectl get pods -n \u0026lt;namespace\u0026gt; kubectl get events -n \u0026lt;namespace\u0026gt; --sort-by=.metadata.creationTimestamp Troubleshooting # Decryption failed # kubectl describe kustomization \u0026lt;kustomization-name\u0026gt; \\ -n \u0026lt;flux-namespace\u0026gt; kubectl logs \\ -n \u0026lt;flux-namespace\u0026gt; \\ deployment/kustomize-controller \\ --since=30m Check:\nThe encrypted file matches a SOPS creation rule. The expected age recipient is present in file metadata. The Flux decryption identity exists. The Kustomization has SOPS decryption enabled. The file has not been encrypted twice. The Kubernetes Secret manifest is structurally valid. Secret exists but the workload fails # kubectl describe pod \u0026lt;pod-name\u0026gt; -n \u0026lt;namespace\u0026gt; kubectl get events -n \u0026lt;namespace\u0026gt; --sort-by=.metadata.creationTimestamp Confirm:\nThe Secret name matches the workload reference. Expected keys exist. Environment-variable and volume references are correct. The plaintext value was correct before encryption. Recovery # Recovery requires the protected age private key and sufficient cluster access to restore the Flux decryption identity.\nDo not publish:\nPrivate-key paths Private-key contents Recovery copy locations Exact Secret names used for privileged systems Decoded Secret values Safety notes # Commands that decode or decrypt values can expose credentials to terminal history, process output, logs, screenshots or copied text. Prefer validation that confirms successful decryption without displaying plaintext.\nRelated entries # GitOps Repository Structure Flux Reconciliation Commands Secret Leakage Checks Kustomization Troubleshooting Disaster-Recovery Critical Assets ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/gitops/sops-age-workflow/","section":"Runbook","summary":"Time guide\nValidate an existing encrypted Secret: approximately 5 to 10 minutes Create and reconcile a new encrypted Secret: approximately 15 to 30 minutes Investigate a decryption fault: approximately 20 to 45 minutes Purpose # This entry describes a safe workflow for storing encrypted Kubernetes Secret manifests in Git using SOPS and age, with Flux performing decryption during reconciliation.\n","title":"SOPS and age Secret Workflow","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/ssh/","section":"Tags","summary":"","title":"SSH","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for name resolution, reachability, host keys, authentication and server logs. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # getent hosts \u0026lt;hostname\u0026gt; nc -vz \u0026lt;hostname\u0026gt; 22 ssh -vvv \u0026lt;user\u0026gt;@\u0026lt;hostname\u0026gt; sudo journalctl -u ssh --since \u0026#34;30 minutes ago\u0026#34; --no-pager Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/linux/ssh-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for name resolution, reachability, host keys, authentication and server logs. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"SSH Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/storage/","section":"Tags","summary":"","title":"Storage","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/systemd/","section":"Tags","summary":"","title":"Systemd","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate failed or unhealthy Linux services using systemctl and Journald.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # systemctl status \u0026lt;service-name\u0026gt; --no-pager --full systemctl is-enabled \u0026lt;service-name\u0026gt; systemctl is-active \u0026lt;service-name\u0026gt; journalctl -u \u0026lt;service-name\u0026gt; --since \u0026#34;30 minutes ago\u0026#34; --no-pager Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/linux/systemd-service-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate failed or unhealthy Linux services using systemctl and Journald.\n","title":"systemd Service Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/tcp/","section":"Tags","summary":"","title":"TCP","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/terminal/","section":"Tags","summary":"","title":"Terminal","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/tpm/","section":"Tags","summary":"","title":"TPM","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/triage/","section":"Tags","summary":"","title":"Triage","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/troubleshooting/","section":"Tags","summary":"","title":"Troubleshooting","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/unifi/","section":"Tags","summary":"","title":"UniFi","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Support workflow for association, VLAN, lease, DNS, roaming and access problems. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Review the client record, access point, signal, retries, VLAN, lease, DNS and roaming history in the authorised UniFi console. Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/unifi-client-connectivity/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Support workflow for association, VLAN, lease, DNS, roaming and access problems. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"UniFi Client Connectivity Checks","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate UniFi network DHCP scopes, leases, gateway and DNS options. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Review the virtual network DHCP server, available lease range, gateway, DNS server, lease time and conflict-detection settings in the authorised UniFi console. Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/unifi-dhcp-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate UniFi network DHCP scopes, leases, gateway and DNS options. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"UniFi DHCP Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/unified-logging/","section":"Tags","summary":"","title":"Unified Logging","type":"tags"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/validation/","section":"Tags","summary":"","title":"Validation","type":"tags"},{"content":"Operational references for virtualisation, backup and recovery platforms.\nThe current collection focuses on layered Kubernetes recovery, Longhorn backups and restore validation. Additional Proxmox, VMware and Hyper-V references can be added over time.\nBackup availability should never be considered proven until a restore has been tested.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/virtualisation/","section":"Runbook","summary":"Operational references for virtualisation, backup and recovery platforms.\nThe current collection focuses on layered Kubernetes recovery, Longhorn backups and restore validation. Additional Proxmox, VMware and Hyper-V references can be added over time.\n","title":"Virtualisation","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/vpn/","section":"Tags","summary":"","title":"VPN","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for client state, DNS, routes, policy, authentication and service reachability. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-VpnConnection Get-NetRoute Test-NetConnection \u0026lt;internal-hostname\u0026gt; -Port \u0026lt;port\u0026gt; route -n get default ip route Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/vpn-connectivity-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for client state, DNS, routes, policy, authentication and service reachability. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"VPN Connectivity Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/wazuh/","section":"Tags","summary":"","title":"Wazuh","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Pilot, enroll, group and validate a Wazuh endpoint agent using a controlled rollout.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # kubectl get pods -n \u0026lt;wazuh-namespace\u0026gt; sudo systemctl status wazuh-agent --no-pager --full sudo journalctl -u wazuh-agent --since \u0026#34;30 minutes ago\u0026#34; --no-pager Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/security/wazuh-agent-onboarding/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Pilot, enroll, group and validate a Wazuh endpoint agent using a controlled rollout.\n","title":"Wazuh Agent Onboarding","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate Wazuh agent service state, connectivity, enrollment and telemetry flow.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # sudo systemctl status wazuh-agent --no-pager --full sudo journalctl -u wazuh-agent --since \u0026#34;30 minutes ago\u0026#34; --no-pager getent hosts \u0026lt;manager-hostname\u0026gt; nc -vz \u0026lt;manager-hostname\u0026gt; \u0026lt;agent-port\u0026gt; Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/security/wazuh-agent-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate Wazuh agent service state, connectivity, enrollment and telemetry flow.\n","title":"Wazuh Agent Troubleshooting","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Test custom Wazuh rules, record evidence and tune detections safely.\nInvestigation workflow # Confirm the correct environment and context. Capture the current state before making changes. Run the read-only checks below. Review events, logs and dependencies. If a change is required, update the Git-managed desired state where applicable. Render, dry-run and review the change. Deploy or reconcile, then complete post-change validation. Read-only checks # flux get kustomizations -A kubectl get pods -n \u0026lt;wazuh-namespace\u0026gt; kubectl get events -n \u0026lt;wazuh-namespace\u0026gt; --sort-by=.metadata.creationTimestamp Expected result # The relevant services and resources should report healthy or ready states, expected dependencies should be present, and recent warning events should not show a persistent unresolved fault.\nTroubleshooting # If a check fails:\nConfirm names, namespaces, contexts and permissions. Review recent events and service logs. Compare the live state with the documented desired state. Check storage, DNS, network and dependent services. Preserve evidence before restarting, deleting or recreating anything. Safety notes # These entries begin with read-only checks. Treat restart, delete, restore, reconcile, exposure and security-containment actions as state-changing operations. Confirm scope, impact, authorisation and rollback before proceeding. Replace all private names, addresses, identifiers and credentials with placeholders before publication or sharing.\nRelated entries # Kubernetes Cluster Health Checks Flux Reconciliation Commands GitOps Repository Structure Security Incident Triage ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/security/wazuh-rule-validation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Test custom Wazuh rules, record evidence and tune detections safely.\n","title":"Wazuh Rule Validation and Tuning","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/wi-fi/","section":"Tags","summary":"","title":"Wi-Fi","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for signal, association, DHCP, DNS, gateway and internet access. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # netsh wlan show interfaces netsh wlan show wlanreport networksetup -getairportnetwork en0 nmcli device wifi list Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/networking/wifi-connectivity-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for signal, association, DHCP, DNS, gateway and internet access. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Wi-Fi Connectivity Troubleshooting","type":"lab-notes"},{"content":"Practical Windows references for workplace support, maintenance and troubleshooting.\nThe collection covers system health, Windows Update, DISM and System File Checker, disk cleanup, event logs, services, networking, Microsoft Defender, Defender for Endpoint, BitLocker, Entra device registration, Group Policy and power diagnostics.\nArticles begin with read-only investigation where possible. Repair, security and state-changing commands are clearly identified.\n","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/","section":"Runbook","summary":"Practical Windows references for workplace support, maintenance and troubleshooting.\nThe collection covers system health, Windows Update, DISM and System File Checker, disk cleanup, event logs, services, networking, Microsoft Defender, Defender for Endpoint, BitLocker, Entra device registration, Group Policy and power diagnostics.\n","title":"Windows","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/windows/","section":"Tags","summary":"","title":"Windows","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate low disk capacity and apply controlled cleanup actions. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-Volume Get-ChildItem C:\\ -Force -ErrorAction SilentlyContinue | Sort Length -Descending | Select -First 20 cleanmgr.exe DISM.exe /Online /Cleanup-Image /StartComponentCleanup Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/disk-space-cleanup/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate low disk capacity and apply controlled cleanup actions. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Windows Disk Space and Cleanup","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # PowerShell and command-line techniques for filtering and exporting relevant Windows events. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-WinEvent -ListLog * | Where-Object RecordCount -gt 0 Get-WinEvent -FilterHashtable @{LogName=\u0026#34;System\u0026#34;; StartTime=(Get-Date).AddHours(-4)} wevtutil el Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/event-log-investigation/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # PowerShell and command-line techniques for filtering and exporting relevant Windows events. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Windows Event Log Investigation","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only first checks for operating system, uptime, storage, services, reliability and recent faults. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # Get-ComputerInfo -Property WindowsProductName,WindowsVersion,OsBuildNumber,OsArchitecture Get-CimInstance Win32_OperatingSystem | Select Caption,Version,LastBootUpTime,FreePhysicalMemory Get-Volume Get-PhysicalDisk | Select FriendlyName,HealthStatus,OperationalStatus,Size Get-Service | Where-Object Status -eq Stopped | Sort Name Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/windows-initial-health-checks/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Read-only first checks for operating system, uptime, storage, services, reliability and recent faults. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Windows Initial Health Checks","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for adapter, address, DNS, route, proxy and TCP connectivity faults. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # ipconfig /all Get-NetAdapter Get-NetIPAddress Get-DnsClientServerAddress Get-NetRoute -AddressFamily IPv4 Resolve-DnsName \u0026lt;hostname\u0026gt; Test-NetConnection \u0026lt;hostname\u0026gt; -Port \u0026lt;port\u0026gt; netsh winhttp show proxy Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/windows-network-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for adapter, address, DNS, route, proxy and TCP connectivity faults. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Windows Network Troubleshooting","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Check battery health, sleep blockers, wake sources and power configuration. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # powercfg /batteryreport powercfg /requests powercfg /lastwake powercfg /waketimers Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/power-battery-diagnostics/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Check battery health, sleep blockers, wake sources and power configuration. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Windows Power and Battery Diagnostics","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate stopped, failing or misconfigured Windows services. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-Service \u0026lt;service-name\u0026gt; Get-CimInstance Win32_Service -Filter \u0026#34;Name=\u0026#39;\u0026lt;service-name\u0026gt;\u0026#39;\u0026#34; Get-WinEvent -FilterHashtable @{LogName=\u0026#34;System\u0026#34;; ProviderName=\u0026#34;Service Control Manager\u0026#34;; StartTime=(Get-Date).AddDays(-1)} Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/windows-services-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Investigate stopped, failing or misconfigured Windows services. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Windows Services Troubleshooting","type":"lab-notes"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Quick reference for common Windows support, identity, networking, service and system checks. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nCommands and checks # Get-ComputerInfo systeminfo Get-CimInstance Win32_OperatingSystem Get-Volume Get-Service | Where-Object Status -ne Running ipconfig /all Get-NetIPAddress Test-NetConnection \u0026lt;hostname\u0026gt; -Port \u0026lt;port\u0026gt; Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/windows-support-command-reference/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Quick reference for common Windows support, identity, networking, service and system checks. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Windows Support Command Reference","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/windows-update/","section":"Tags","summary":"","title":"Windows Update","type":"tags"},{"content":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for update history, services, policy, logs, pending reboots and component repair. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\nStandard workflow # Confirm the affected user, device, service and time window. Capture current state before changes. Run read-only checks. Identify whether the fault is device, identity, policy, network, application, storage or security related. Review logs, events and recent changes. Apply the least disruptive controlled resolution. Validate the original issue and check for side effects. Record the cause, action and follow-up. Commands and checks # Get-HotFix | Sort-Object InstalledOn -Descending Get-Service wuauserv,bits,cryptsvc,msiserver Get-WinEvent -LogName \u0026#34;Microsoft-Windows-WindowsUpdateClient/Operational\u0026#34; -MaxEvents 100 DISM.exe /Online /Cleanup-Image /ScanHealth Interpretation # Capture timestamps and exact errors. Compare the result with policy, expected configuration and recent changes. Confirm whether a central management platform can override local settings. Preserve relevant logs before a repair, restart or re-enrollment action. Safety notes # Read-only checks appear before repair actions where possible. Commands that update signatures, scan a device, repair Windows, renew DHCP, resume encryption or alter managed services require appropriate authorisation and impact review. Never paste credentials, recovery keys, tenant identifiers, personal information or unsanitised diagnostic output into public records.\nRelated entries # Windows Support Command Reference Windows Network Troubleshooting Security Incident Triage Connectivity and Port Testing ","date":"31 August 2026","externalUrl":null,"permalink":"/lab-notes/windows/windows-update-troubleshooting/","section":"Runbook","summary":"Time guide\nInitial checks: approximately 10 to 20 minutes Detailed investigation: approximately 20 to 60 minutes Purpose # Workflow for update history, services, policy, logs, pending reboots and component repair. This public reference consolidates professional support experience developed since March 2022 and uses generic placeholders rather than employer or customer information.\n","title":"Windows Update Troubleshooting","type":"lab-notes"},{"content":"","date":"31 August 2026","externalUrl":null,"permalink":"/tags/workloads/","section":"Tags","summary":"","title":"Workloads","type":"tags"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/tags/blowfish/","section":"Tags","summary":"","title":"Blowfish","type":"tags"},{"content":"Project date: 2026-08-30 Role: Architecture, implementation, validation and operation\nA searchable Hugo and Git documentation platform containing 78 runbooks across ten categories and eight evidence-backed project case studies.\nEnvironment and operating context # The documentation began as university notes, moved through workplace OneNote collections and later into Obsidian. Each move improved parts of the organisation, but the information remained difficult to publish consistently, version alongside technical work and connect to related projects.\nAs the homelab expanded across Kubernetes, Microsoft 365, networking, security, virtualisation and recovery, documentation needed to become a maintained platform rather than a folder of notes.\nObjective # The goal was to create a searchable public technical reference that separated portfolio case studies from repeatable runbooks, retained Git history and enforced a boundary between publishable knowledge and restricted operational detail.\nOperational value and risk reduction # Before: Knowledge was fragmented across university notes, OneNote and Obsidian, with inconsistent structure and weak linkage to implementation evidence.\nAfter: A searchable Git-backed Hugo site contains 78 runbooks across ten categories and eight evidence-backed case studies.\nRisk reduced: Operational knowledge is less dependent on one note application or memory, while public-safety rules keep restricted material out of the published site.\nArchitectural decisions and alternatives # Continue with OneNote Convenient for notes, but weaker for Git history, static publishing and content-as-code workflows. Continue with Obsidian only Strong local knowledge management, but public publishing and theme integration required additional workflow. Hosted documentation SaaS Could reduce maintenance, but offers less control and does not align as closely with the self-hosted portfolio objective. Selected design Hugo and Blowfish provide fast static output, Git-controlled source, search, aliases and a clear public/private boundary. Technical challenges # The source material varied in age, structure and purpose. University learning notes, workplace procedures and homelab runbooks could not simply be published unchanged. Content needed common metadata, category ownership, public-safety review, aliases for moved pages and a design that worked on desktop and mobile.\nProposed solution # Hugo Extended 0.165.0 generates the site using the Blowfish theme. Home output includes HTML, RSS and JSON, enabling built-in search. Authored Markdown or HTML remains the source; the generated public directory is build output. Projects explain engineering decisions and outcomes, while Lab Notes contain operational procedures.\nInformation architecture # Runbook source files 78 Runbook categories 10 Individual project case studies 8 Generated pages 341 Aliases 113 Hugo build time 2.645 seconds Search Enabled Theme Blowfish Windows: 17 documents GitOps: 9 Microsoft 365: 9 Networking: 9 Kubernetes: 7 Linux: 7 Security: 6 macOS: 5 Monitoring: 5 Virtualisation: 3 Implementation # The website provides Home, About, Journey, Projects, Homelab and Runbook navigation. Category landing pages organise the 78 runbooks, project pages link to supporting procedures, and Hugo generates tables of contents, metadata, aliases and search data.\nThe eight project articles were rewritten because the original pages had correct layout but insufficient engineering evidence. The revised standard includes context, objectives, constraints, architecture, responsibilities, implementation, validation, measurable outcomes and lessons learned.\nContent governance # Git-controlled authored source Generated HTML not edited directly Public, internal and restricted information separated No credentials, private keys or raw security events Claims classified as implemented, validated, observed, documented or planned No fabricated customer testimonials or commercial KPIs Runbooks linked to the projects they support Validation and measurable outcomes # A clean Hugo build generated 341 pages, three paginator pages, seven static files and 113 aliases with no reported build error. Search is enabled in the active configuration, the public base URL is https://lintech.uk/, and the source is organised into ten practical categories.\nOutcome # The documentation is now a versioned, searchable technical platform rather than material tied to one note-taking application. It presents both breadth, through 78 runbooks, and depth, through evidence-backed project case studies.\nSkills demonstrated # Technical writing and information architecture Hugo and Blowfish Metadata, search and aliases Git-based content management Public-safety review Runbook and case-study design Lessons learned # More documentation is not automatically better documentation. A useful knowledge platform needs clear audiences, evidence standards, lifecycle status and a deliberate distinction between narrative case studies and executable procedures.\nCurrent status and next steps # Maintained and in pre-release content remediation. The next step is final route, link and responsive testing before the redesigned site replaces the existing public website.\nExplore the work # Browse the runbook library or view all project case studies.\nHow it was done # Consolidated the source material. Notes that had moved from university documents to workplace OneNote and then Obsidian were reviewed and moved into a Git-controlled website repository. Separated content types. Projects were defined as engineering narratives covering decisions and outcomes, while Lab Notes became operational runbooks focused on repeatable procedures. Created the taxonomy. Runbooks were organised into GitOps, Kubernetes, Linux, macOS, Microsoft 365, Monitoring, Networking, Security, Virtualisation and Windows categories. Selected the publishing platform. Hugo Extended and the Blowfish theme were used to generate a fast static site with responsive layouts, taxonomies, tables of contents, aliases and JSON search output. Standardised article metadata. Titles, summaries, dates, categories, tags, publication status and related links were made consistent so content could be discovered and maintained. Established public-safety rules. Credentials, private keys, raw security events and overly specific recovery details remain outside the public site. Internal documentation stays in private repositories. Preserved moved URLs. Hugo aliases were used to retain navigation paths as material changed location, reducing broken links during the migration. Enabled search and navigation. Home output includes JSON for search, and custom desktop and mobile navigation provides direct access to Projects, Homelab and Runbooks. Reworked the project portfolio. Eight short placeholder pages were replaced with evidence-backed case studies that include context, process, validation, measurable outcomes and lessons learned. Validated the build. A clean Hugo build generated 341 pages, three paginator pages, seven static files and 113 aliases in 2.645 seconds without a reported build failure. Evidence methodology # The figures in this case study were derived from reproducible source and runtime checks rather than estimates.\nRunbook and project totals were counted from authored files under content/lab-notes and content/projects, excluding generated public output. Category totals were calculated from the ten top-level Lab Notes directories. A clean Hugo build supplied the generated page, alias and duration evidence. Limitations and residual risks # The redesigned site remains pre-release until final content and deployment validation are complete. File and page counts measure scope, not documentation quality. Public runbooks intentionally omit sensitive internal detail. Documentation can become stale and still requires review after platform changes. Plain-English glossary # Hugo A static-site generator that builds deployable pages from content and templates. Alias A generated redirect that preserves access to a page after its URL changes. Content as code Managing documentation through source files, version control and repeatable builds. Independent validation # This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.\n","date":"30 August 2026","externalUrl":null,"permalink":"/projects/runbook-knowledge-platform/","section":"Projects","summary":"A searchable Hugo and Git documentation platform containing 78 runbooks across ten categories and eight evidence-backed project case studies.","title":"Building a Searchable Operational Runbook Platform","type":"projects"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/categories/documentation/","section":"Categories","summary":"","title":"Documentation","type":"categories"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/tags/git/","section":"Tags","summary":"","title":"Git","type":"tags"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/tags/hugo/","section":"Tags","summary":"","title":"Hugo","type":"tags"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/tags/platform-engineering/","section":"Tags","summary":"","title":"Platform Engineering","type":"tags"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/tags/runbooks/","section":"Tags","summary":"","title":"Runbooks","type":"tags"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/tags/search/","section":"Tags","summary":"","title":"Search","type":"tags"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/tags/cluster-health/","section":"Tags","summary":"","title":"Cluster Health","type":"tags"},{"content":"","date":"30 August 2026","externalUrl":null,"permalink":"/tags/kubectl/","section":"Tags","summary":"","title":"Kubectl","type":"tags"},{"content":"Time guide\nInitial cluster triage: approximately 10 to 20 minutes Standard health review: approximately 30 minutes Full platform, storage and host review: approximately 30 to 60 minutes The actual duration depends on cluster size, access availability and whether any checks identify faults that need further investigation.\nPurpose # This Runbook entry provides a structured set of read-only checks for reviewing the health of a Kubernetes or k3s cluster.\nThe checks move from the broadest cluster-level view toward individual nodes, workloads, services, storage and k3s host services.\nThe commands are intended for:\nRoutine platform checks Initial incident triage Investigating failed workloads Checking node availability Reviewing recent warning events Confirming storage and networking objects Gathering information before making any changes These commands are primarily read-only. Commands that could generate a large amount of output or require access to a cluster node are clearly identified.\nRequirements # A working kubectl installation A valid kubeconfig Network access to the Kubernetes API server Permission to read the required cluster resources SSH access to a k3s node for host-level service checks sudo access for systemd journal checks where required Confirm the current kubectl client and server versions:\nkubectl version Confirm the active context:\nkubectl config current-context List all configured contexts:\nkubectl config get-contexts Display the selected cluster endpoint without showing credential contents:\nkubectl config view --minify \\ -o jsonpath=\u0026#39;{.clusters[0].cluster.server}{\u0026#34;\\n\u0026#34;}\u0026#39; Treat kubeconfig files as sensitive. A kubeconfig may contain credentials or references that provide access to a cluster. Do not publish the file or paste its contents into a public Runbook.\nQuick cluster overview # Begin with the main cluster endpoints and services:\nkubectl cluster-info Check whether the API server responds:\nkubectl get --raw=\u0026#39;/readyz?verbose\u0026#39; Check the API server live endpoint:\nkubectl get --raw=\u0026#39;/livez?verbose\u0026#39; A healthy API server should return successful checks rather than connection, authentication or internal server errors.\nDisplay general cluster information:\nkubectl get componentstatuses componentstatuses is deprecated in newer Kubernetes releases and may not provide useful results on every cluster. Prefer the API readiness endpoints and platform monitoring where available.\nNode health # List all nodes with their status, roles, age and Kubernetes version:\nkubectl get nodes -o wide Expected result:\nAll expected nodes are present Each node reports Ready Internal addresses are correct Kubernetes versions are within the expected range No unexpected node has joined the cluster Display node resource usage:\nkubectl top nodes This command requires Metrics Server or another compatible metrics API.\nIf kubectl top nodes fails, check whether the metrics API is available:\nkubectl get apiservice v1beta1.metrics.k8s.io Describe a particular node:\nkubectl describe node \u0026lt;node-name\u0026gt; Review the conditions of every node:\nkubectl get nodes \\ -o custom-columns=\u0026#39;NAME:.metadata.name,READY:.status.conditions[?(@.type==\u0026#34;Ready\u0026#34;)].status,MEMORY:.status.conditions[?(@.type==\u0026#34;MemoryPressure\u0026#34;)].status,DISK:.status.conditions[?(@.type==\u0026#34;DiskPressure\u0026#34;)].status,PID:.status.conditions[?(@.type==\u0026#34;PIDPressure\u0026#34;)].status\u0026#39; Healthy nodes should normally show:\nREADY True MEMORY False DISK False PID False List node taints:\nkubectl get nodes \\ -o custom-columns=\u0026#39;NAME:.metadata.name,TAINTS:.spec.taints\u0026#39; List nodes that are currently cordoned:\nkubectl get nodes \\ -o custom-columns=\u0026#39;NAME:.metadata.name,UNSCHEDULABLE:.spec.unschedulable\u0026#39; A value of true means the node is marked unschedulable.\nWorkload overview # List all pods across every namespace:\nkubectl get pods -A -o wide List pods that are not currently in the Running or Succeeded phases:\nkubectl get pods -A \\ --field-selector=status.phase!=Running,status.phase!=Succeeded This filter is useful for finding pods in phases such as:\nPending Failed Unknown Some container-level states such as CrashLoopBackOff may still appear inside a pod whose overall phase is Running. For a broader visual check, use:\nkubectl get pods -A List deployments:\nkubectl get deployments -A List StatefulSets:\nkubectl get statefulsets -A List DaemonSets:\nkubectl get daemonsets -A List jobs:\nkubectl get jobs -A List CronJobs:\nkubectl get cronjobs -A A healthy result should generally show:\nExpected replicas are available StatefulSet ready counts match desired counts DaemonSet desired and ready counts match Jobs are completing successfully CronJobs are not unexpectedly suspended Failed and restarting containers # List pod restart counts:\nkubectl get pods -A \\ -o custom-columns=\u0026#39;NAMESPACE:.metadata.namespace,POD:.metadata.name,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount\u0026#39; Sort pods by creation time:\nkubectl get pods -A \\ --sort-by=.metadata.creationTimestamp Describe a problem pod:\nkubectl describe pod \u0026lt;pod-name\u0026gt; -n \u0026lt;namespace\u0026gt; Review current logs from all containers in a pod:\nkubectl logs \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ --all-containers=true \\ --tail=200 Review the logs from the previous instance of a restarted container:\nkubectl logs \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -c \u0026lt;container-name\u0026gt; \\ --previous \\ --tail=200 Follow current logs:\nkubectl logs \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -c \u0026lt;container-name\u0026gt; \\ --follow Press Ctrl+C to stop following logs.\nRecent events # List events across all namespaces in creation order:\nkubectl get events -A \\ --sort-by=.metadata.creationTimestamp Show warning events only:\nkubectl get events -A \\ --field-selector=type=Warning \\ --sort-by=.metadata.creationTimestamp Show events in a particular namespace:\nkubectl get events \\ -n \u0026lt;namespace\u0026gt; \\ --sort-by=.metadata.creationTimestamp Common event reasons to investigate include:\nFailedScheduling FailedMount FailedAttachVolume FailedCreate FailedPull BackOff Unhealthy Evicted NodeNotReady Events are retained for a limited period and should not be treated as a permanent audit history.\nResource usage # Display pod CPU and memory usage across all namespaces:\nkubectl top pods -A Sort pod usage by CPU:\nkubectl top pods -A \\ --sort-by=cpu Sort pod usage by memory:\nkubectl top pods -A \\ --sort-by=memory Display usage for all containers:\nkubectl top pods -A \\ --containers Check resource requests and limits:\nkubectl get pods -A \\ -o custom-columns=\u0026#39;NAMESPACE:.metadata.namespace,POD:.metadata.name,CPU_REQUESTS:.spec.containers[*].resources.requests.cpu,CPU_LIMITS:.spec.containers[*].resources.limits.cpu,MEMORY_REQUESTS:.spec.containers[*].resources.requests.memory,MEMORY_LIMITS:.spec.containers[*].resources.limits.memory\u0026#39; Missing requests or limits are not automatically a fault, but the result can help explain scheduling, contention and capacity problems.\nServices and ingress # List services:\nkubectl get services -A List endpoint slices:\nkubectl get endpointslices -A List ingress resources:\nkubectl get ingress -A Describe a particular service:\nkubectl describe service \u0026lt;service-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Check the selector used by a service:\nkubectl get service \u0026lt;service-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.selector}{\u0026#34;\\n\u0026#34;}\u0026#39; List the pods matching a known selector:\nkubectl get pods \\ -n \u0026lt;namespace\u0026gt; \\ -l \u0026#39;app=\u0026lt;application-label\u0026gt;\u0026#39; \\ -o wide If a service has no ready endpoints, check:\nService selectors Pod labels Pod readiness Container ports Service targetPort Network policies Application listening addresses DNS checks # Check the CoreDNS deployment:\nkubectl get deployment coredns \\ -n kube-system List CoreDNS pods:\nkubectl get pods \\ -n kube-system \\ -l k8s-app=kube-dns \\ -o wide Review CoreDNS logs:\nkubectl logs \\ -n kube-system \\ -l k8s-app=kube-dns \\ --tail=200 Display the cluster DNS service:\nkubectl get service kube-dns \\ -n kube-system If DNS resolution fails inside workloads, also inspect:\nThe pod\u0026rsquo;s /etc/resolv.conf CoreDNS pod health CoreDNS ConfigMap Upstream resolvers Network policies Node resolver configuration Persistent storage # List persistent volumes:\nkubectl get persistentvolumes List persistent volume claims:\nkubectl get persistentvolumeclaims -A List storage classes:\nkubectl get storageclasses List volume attachments:\nkubectl get volumeattachments Find claims that are not bound:\nkubectl get persistentvolumeclaims -A \\ --field-selector=status.phase!=Bound Describe a problem claim:\nkubectl describe persistentvolumeclaim \u0026lt;claim-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Describe the related persistent volume:\nkubectl describe persistentvolume \u0026lt;volume-name\u0026gt; Storage issues may appear as:\nPending FailedMount FailedAttachVolume Multi-Attach error Volume not ready Insufficient storage Longhorn checks # List Longhorn namespaces:\nkubectl get namespaces | grep longhorn Check Longhorn workloads:\nkubectl get pods \\ -n longhorn-system \\ -o wide List Longhorn volumes:\nkubectl get volumes.longhorn.io \\ -n longhorn-system List Longhorn replicas:\nkubectl get replicas.longhorn.io \\ -n longhorn-system List Longhorn engines:\nkubectl get engines.longhorn.io \\ -n longhorn-system List Longhorn nodes:\nkubectl get nodes.longhorn.io \\ -n longhorn-system Display selected Longhorn volume health information:\nkubectl get volumes.longhorn.io \\ -n longhorn-system \\ -o custom-columns=\u0026#39;NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,SCHEDULED:.status.conditions[?(@.type==\u0026#34;scheduled\u0026#34;)].status\u0026#39; Review warning events in the Longhorn namespace:\nkubectl get events \\ -n longhorn-system \\ --field-selector=type=Warning \\ --sort-by=.metadata.creationTimestamp Longhorn custom-resource fields can vary between versions. If a custom-column field is blank, inspect the resource directly:\nkubectl get volumes.longhorn.io \u0026lt;volume-name\u0026gt; \\ -n longhorn-system \\ -o yaml This command is read-only, but the output may contain environment-specific information and should be sanitised before publication or sharing.\nIngress controller checks # List Traefik workloads:\nkubectl get pods \\ -n kube-system \\ -l app.kubernetes.io/name=traefik \\ -o wide List Traefik services:\nkubectl get services \\ -n kube-system \\ -l app.kubernetes.io/name=traefik Review Traefik logs:\nkubectl logs \\ -n kube-system \\ -l app.kubernetes.io/name=traefik \\ --tail=200 Check ingress resources and assigned addresses:\nkubectl get ingress -A -o wide If Traefik was deployed into another namespace, replace kube-system with the correct namespace.\nk3s service health # Run these commands over SSH on a k3s server node.\nCheck the server service:\nsudo systemctl status k3s \\ --no-pager \\ --full Check whether the server service is active:\nsudo systemctl is-active k3s Review recent k3s server logs:\nsudo journalctl \\ -u k3s \\ --since \u0026#34;30 minutes ago\u0026#34; \\ --no-pager Follow the server logs:\nsudo journalctl \\ -u k3s \\ --follow Press Ctrl+C to stop following logs.\nOn an agent node, use:\nsudo systemctl status k3s-agent \\ --no-pager \\ --full Review recent agent logs:\nsudo journalctl \\ -u k3s-agent \\ --since \u0026#34;30 minutes ago\u0026#34; \\ --no-pager K3s systems using systemd send service logs to Journald. Pod logs are normally stored under /var/log/pods, while the embedded containerd log can be found beneath /var/lib/rancher/k3s/agent/containerd/. These paths should be treated as host-level diagnostic sources rather than files to modify directly.\nk3s host checks # Display the installed k3s version:\nk3s --version Run the k3s host configuration check:\nsudo k3s check-config Check host disk capacity:\ndf -h Check inode usage:\ndf -ih Check memory:\nfree -h Check system load and uptime:\nuptime Check failed systemd services:\nsystemctl --failed Display recent high-priority system journal entries:\nsudo journalctl \\ -p warning \\ --since \u0026#34;30 minutes ago\u0026#34; \\ --no-pager These host checks can help identify:\nFull filesystems Exhausted inodes Memory pressure Failed system services Excessive load Kernel or storage warnings Namespace-focused investigation # Set a reusable namespace variable:\nNAMESPACE=\u0026#34;\u0026lt;namespace\u0026gt;\u0026#34; List the namespace\u0026rsquo;s main resources:\nkubectl get all \\ -n \u0026#34;${NAMESPACE}\u0026#34; List ConfigMaps:\nkubectl get configmaps \\ -n \u0026#34;${NAMESPACE}\u0026#34; List Secrets by name and type without displaying secret values:\nkubectl get secrets \\ -n \u0026#34;${NAMESPACE}\u0026#34; List persistent volume claims:\nkubectl get persistentvolumeclaims \\ -n \u0026#34;${NAMESPACE}\u0026#34; List recent events:\nkubectl get events \\ -n \u0026#34;${NAMESPACE}\u0026#34; \\ --sort-by=.metadata.creationTimestamp Do not use this command in public screenshots:\nkubectl get secrets -n \u0026#34;${NAMESPACE}\u0026#34; -o yaml Although Kubernetes Secret values are base64 encoded, base64 is not encryption and the values may be recoverable.\nOptional cluster information bundle # Kubernetes can generate a broad diagnostic dump:\nkubectl cluster-info dump \\ --output-directory=\u0026#34;./cluster-dump\u0026#34; This command is read-only against the cluster, but it writes a substantial amount of information locally.\nThe resulting files may contain:\nInternal addresses Resource names Configuration details Pod specifications Environment variables Log messages Infrastructure identifiers Review and sanitise the complete output before sharing it.\nDelete the local dump only after confirming it is no longer required:\nrm -rf ./cluster-dump The removal command is destructive to the specified local directory. Confirm the path before running it.\nExpected result # A healthy cluster should generally show:\nThe API server is reachable All expected nodes are present Nodes report Ready No unexpected resource-pressure conditions are active Required deployments have available replicas StatefulSets have the expected ready replicas DaemonSets are scheduled on suitable nodes Most application pods are Running Completed jobs report Completed Restart counts are stable Warning events do not indicate persistent faults Services have appropriate endpoints Ingress resources have the expected routing information Persistent volume claims are Bound Longhorn volumes report healthy or expected states k3s server and agent services are active Host disks have sufficient free space A cluster can still have a valid reason for exceptions. For example:\nCompleted job pods may display Succeeded Maintenance workloads may be intentionally suspended A node may be cordoned for planned maintenance A deployment may intentionally have zero replicas A test namespace may contain failed workloads Interpret results in the context of the intended state.\nTroubleshooting # kubectl cannot connect to the cluster # Example error:\nUnable to connect to the server Check the active context:\nkubectl config current-context Check the cluster endpoint:\nkubectl config view --minify \\ -o jsonpath=\u0026#39;{.clusters[0].cluster.server}{\u0026#34;\\n\u0026#34;}\u0026#39; Confirm basic network connectivity to the API endpoint using an appropriate network diagnostic tool.\nPossible causes include:\nIncorrect kubeconfig Wrong context Unavailable API server DNS failure VPN disconnection Firewall or routing issue Expired credentials Incorrect certificate information A node is NotReady # Check the node:\nkubectl describe node \u0026lt;node-name\u0026gt; Review the relevant k3s service on that node:\nsudo systemctl status k3s \\ --no-pager \\ --full Or, for an agent:\nsudo systemctl status k3s-agent \\ --no-pager \\ --full Review recent service logs:\nsudo journalctl \\ -u k3s-agent \\ --since \u0026#34;30 minutes ago\u0026#34; \\ --no-pager Also check:\ndf -h free -h systemctl --failed A pod is Pending # Describe the pod:\nkubectl describe pod \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Look for event messages relating to:\nInsufficient CPU or memory Node selectors Affinity rules Taints and tolerations Unbound persistent volume claims Missing storage classes Scheduling restrictions A pod is restarting # Check current state and restart count:\nkubectl get pod \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Review the previous container logs:\nkubectl logs \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -c \u0026lt;container-name\u0026gt; \\ --previous \\ --tail=200 Describe the pod:\nkubectl describe pod \u0026lt;pod-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Look for:\nCrashLoopBackOff OOMKilled Probe failures Missing files or configuration Permission errors Failed volume mounts Application startup errors A service has no endpoints # Check the service selector:\nkubectl get service \u0026lt;service-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; \\ -o jsonpath=\u0026#39;{.spec.selector}{\u0026#34;\\n\u0026#34;}\u0026#39; List matching pods:\nkubectl get pods \\ -n \u0026lt;namespace\u0026gt; \\ --show-labels Check endpoint slices:\nkubectl get endpointslices \\ -n \u0026lt;namespace\u0026gt; Confirm:\nService selectors match pod labels Pods are ready The exposed service port is correct The service targetPort matches the application port A persistent volume claim is Pending # Describe the claim:\nkubectl describe persistentvolumeclaim \u0026lt;claim-name\u0026gt; \\ -n \u0026lt;namespace\u0026gt; Check storage classes:\nkubectl get storageclasses Check recent events:\nkubectl get events \\ -n \u0026lt;namespace\u0026gt; \\ --sort-by=.metadata.creationTimestamp Check the relevant storage-system namespace, such as Longhorn:\nkubectl get pods \\ -n longhorn-system Safety notes # Most commands in this Runbook entry are read-only.\nThe following commands require particular care:\nLog following # Commands using --follow continue running until stopped:\nkubectl logs --follow sudo journalctl --follow Stop them with:\nCtrl+C Cluster information dump # This command writes potentially sensitive diagnostic information to the local filesystem:\nkubectl cluster-info dump Review and sanitise its output before uploading or sharing it.\nYAML output # Full YAML output can reveal:\nInternal addresses Resource names Environment variables Volume configuration Infrastructure labels Secret references Never publish Kubernetes Secret values or unsanitised diagnostic output.\nNo direct changes # This Runbook intentionally avoids commands such as:\nkubectl delete kubectl apply kubectl edit kubectl patch kubectl scale kubectl rollout restart systemctl restart Those commands modify state and should only be used after the fault has been understood, the intended change has been reviewed, and a rollback plan is available.\nRelated entries # Planned related Runbook entries:\nKubernetes Workload Troubleshooting Kubernetes Storage Health Checks Longhorn Volume Troubleshooting Traefik Ingress Troubleshooting Flux Reconciliation Commands Kubernetes Resource and Capacity Checks k3s Server and Agent Troubleshooting References # Kubernetes: Troubleshooting clusters Kubernetes: Troubleshooting kubectl K3s FAQ ","date":"30 August 2026","externalUrl":null,"permalink":"/lab-notes/kubernetes/cluster-health-checks/","section":"Runbook","summary":"Time guide\nInitial cluster triage: approximately 10 to 20 minutes Standard health review: approximately 30 minutes Full platform, storage and host review: approximately 30 to 60 minutes The actual duration depends on cluster size, access availability and whether any checks identify faults that need further investigation.\n","title":"Kubernetes Cluster Health Checks","type":"lab-notes"},{"content":"","date":"28 August 2026","externalUrl":null,"permalink":"/tags/cloudflare-tunnel/","section":"Tags","summary":"","title":"Cloudflare Tunnel","type":"tags"},{"content":"","date":"28 August 2026","externalUrl":null,"permalink":"/tags/crowdsec/","section":"Tags","summary":"","title":"CrowdSec","type":"tags"},{"content":"Project date: 2026-08-28 Role: Architecture, implementation, validation and operation\nA tunnel-based public access design with two cloudflared replicas, selective edge controls and CrowdSec-informed enforcement.\nEnvironment and operating context # The Kubernetes cluster hosts both internal and externally reachable applications. Publishing every ingress directly would unnecessarily expose the origin and mix public routing with internal service discovery. The edge project created a selective path through Cloudflare while retaining Traefik and application authentication inside the cluster.\nObjective # The design needed to avoid broad inbound application exposure, support public and protected services, preserve clear rollback, and integrate behavioural decisions without embedding Cloudflare credentials in source.\nOperational value and risk reduction # Before: Kubernetes ingress and external publication risked being treated as the same thing, making exposure boundaries harder to reason about.\nAfter: Two controlled tunnel replicas provide an outbound publication path, while CrowdSec decisions feed a dedicated Cloudflare bouncer.\nRisk reduced: Selected public services do not require broad direct inbound origin exposure, and connector credentials are encrypted rather than embedded in manifests.\nArchitectural decisions and alternatives # Direct port forwarding Simple, but exposes the origin path and increases dependence on perimeter firewall rules. Single tunnel connector Lower resource use, but introduces a connector-level single point of failure. Separate tunnel routes directly to every service Possible, but duplicates the routing responsibility already owned by Traefik. Selected design Two cloudflared replicas feed the existing Traefik ingress layer, with central CrowdSec decisions and encrypted credentials. Technical challenges # A tunnel being connected does not prove that an application is healthy. DNS, edge policy, the tunnel, Traefik and the application can fail independently. The cluster also contains 23 ingress hostnames, but many are internal, so the exposure model had to distinguish ingress from Internet publication.\nProposed solution # Two cloudflared replicas run on separate k3s nodes and establish outbound tunnel connections. Automatic connector updates are disabled so image changes remain controlled; Prometheus-compatible metrics are exposed on port 2000. Public traffic passes through Cloudflare to Traefik, which then routes to the selected Kubernetes service.\nInternet -\u0026gt; Cloudflare DNS and controls -\u0026gt; two cloudflared replicas -\u0026gt; Traefik ingress -\u0026gt; selected Kubernetes service CrowdSec integration # CrowdSec was selected for its central LAPI model: multiple telemetry sources can contribute to one decision core, and enforcement components can consume those decisions. The Cloudflare Worker bouncer runs as a dedicated ready Deployment and receives Cloudflare and LAPI values through environment references backed by a SOPS-encrypted Secret.\nThe available evidence proves the central decision and enforcement architecture. It does not prove a deliberately validated fail-open scenario, so the case study does not make that stronger claim.\nImplementation and controls # Two cloudflared replicas, both ready Replicas distributed across nodes 1 and 4 Immutable cloudflared image digest --no-autoupdate for controlled releases Tunnel metrics available on port 2000 SOPS-encrypted tunnel token SOPS-encrypted bouncer credentials Traefik retained as internal ingress authority Validation and measurable outcomes # Cloudflared replicas 2 desired, 2 ready, 2 available Replica restarts 0 in evidence snapshot Total Kubernetes ingress hosts 23 internal and external CrowdSec Cloudflare bouncer 1 of 1 ready Encrypted Cloudflare Secret manifests 2 The 23-host count is deliberately not described as 23 public services. It records the size of the Kubernetes ingress estate and demonstrates why selective edge classification is necessary.\nOutcome # Selected applications can use an encrypted outbound tunnel rather than general direct inbound exposure. The connector is redundant across nodes, its release is controlled through an immutable image, and behavioural decisions can be distributed through the CrowdSec core.\nSkills demonstrated # Cloudflare Tunnel Zero Trust edge design Traefik integration DNS and ingress classification CrowdSec LAPI and bouncer architecture Secret management Availability and failure-boundary analysis Lessons learned # A connected tunnel is only one link in the service path. Useful validation must separate connector health, edge policy, ingress routing and application health. Internal ingress records must also never be assumed to be publicly exposed.\nCurrent status and next steps # Operational. Exact public-route and Cloudflare Access counts are intentionally omitted because the collected Kubernetes evidence cannot establish dashboard-side policy state.\nExplore the implementation # Browse the networking runbooks or review the CrowdSec and Wazuh design.\nHow it was done # Classified ingress routes. Kubernetes ingress names were reviewed as internal or candidates for external publication. The presence of an ingress was not treated as evidence that a service was Internet-accessible. Selected an outbound tunnel model. Cloudflare Tunnel was used so published services could be reached without creating a general inbound application path to the Kubernetes origin. Deployed redundant connectors. Cloudflared was deployed with two replicas scheduled on separate k3s nodes. Automatic updates were disabled, and the container image was controlled through an immutable digest. Connected the tunnel to Traefik. Cloudflare traffic reaches the existing Traefik ingress layer, preserving one internal routing authority rather than creating separate application routes inside the tunnel connector. Protected tunnel credentials. The connector token was stored in a SOPS-encrypted Kubernetes Secret and supplied at runtime rather than embedded in source. Enabled operational metrics. Cloudflared exposes metrics on port 2000 so tunnel connector state can be monitored independently from application state. Centralised behavioural decisions. CrowdSec LAPI provides one decision core that can accept connected telemetry sources and distribute decisions to enforcement components. Added the Cloudflare Worker bouncer. A dedicated bouncer Deployment consumes Cloudflare account, zone, API and CrowdSec values through protected environment references. Validated each path separately. Connector readiness, Kubernetes ingress, Traefik routing and application reachability were checked as separate layers so a healthy tunnel could not mask an unhealthy application. Recorded the evidence boundary. The cluster showed 23 internal and external ingress hostnames, two ready tunnel replicas and one ready bouncer. The article does not incorrectly present all 23 ingresses as public routes. Evidence methodology # The figures in this case study were derived from reproducible source and runtime checks rather than estimates.\nDeployment replica, readiness, restart, image and placement evidence came from the Kubernetes API. Ingress hosts were counted from Kubernetes resources and deliberately classified only as internal-or-external ingress, not as public Cloudflare routes. Secret manifests were checked for SOPS markers, and bouncer configuration was inspected by variable name without exposing values. Limitations and residual risks # Kubernetes evidence cannot prove the exact number of Cloudflare dashboard routes or Access policies. A ready tunnel does not prove that the application behind it is healthy. A tested CrowdSec fail-open scenario is not claimed. Cloudflare remains an external dependency for published-service reachability. Plain-English glossary # Cloudflare Tunnel An outbound connector that carries selected traffic between Cloudflare and an internal service without a general inbound listener. Traefik The Kubernetes ingress controller that maps host and path rules to services. Bouncer A CrowdSec enforcement component that consumes central decisions and applies them to another platform. Independent validation # This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.\n","date":"28 August 2026","externalUrl":null,"permalink":"/projects/cloudflare-edge-hardening/","section":"Projects","summary":"A tunnel-based public access design with two cloudflared replicas, selective edge controls and CrowdSec-informed enforcement.","title":"Hardening Public Services with Cloudflare Zero Trust","type":"projects"},{"content":"","date":"28 August 2026","externalUrl":null,"permalink":"/categories/security/","section":"Categories","summary":"","title":"Security","type":"categories"},{"content":"","date":"28 August 2026","externalUrl":null,"permalink":"/tags/traefik/","section":"Tags","summary":"","title":"Traefik","type":"tags"},{"content":"","date":"28 August 2026","externalUrl":null,"permalink":"/tags/zero-trust/","section":"Tags","summary":"","title":"Zero Trust","type":"tags"},{"content":"Project date: 2026-08-26 Role: Architecture, implementation, validation and operation\nA detection and investigation platform combining Wazuh, Microsoft 365 audit data, CrowdSec, MISP and a least-privilege SOC portal.\nEnvironment and operating context # The homelab includes externally reachable services, Kubernetes workloads, personal endpoints and Microsoft 365 activity. Security-relevant evidence therefore existed across network logs, endpoint agents, cloud audit events and specialist platforms. The project created one operational model without pretending that every alert represented a confirmed incident.\nObjective # The goal was to detect meaningful behaviours, retain evidence for investigation, support threat-intelligence lookup and expose a concise SOC view without introducing unapproved automated response.\nOperational value and risk reduction # Before: Endpoint, cloud-audit, behavioural and threat-intelligence evidence existed in separate tools without one validation and presentation model.\nAfter: Wazuh, CrowdSec, MISP and LIP provide an integrated detection and investigation path with nine tested local rules.\nRisk reduced: High-value Microsoft 365 activities and disconnected endpoints are visible sooner, while least-privilege readers prevent the portal from becoming a response or administration channel.\nArchitectural decisions and alternatives # Wazuh alone Provides SIEM and endpoint capability, but does not replace distributed behavioural enforcement or analyst threat-intelligence workflows. CrowdSec alone Strong for behavioural blocking, but not an endpoint and Microsoft 365 detection platform. Automatic response from the portal Could reduce response time, but creates unacceptable risk without approval, rollback and audit controls. Selected design Wazuh for endpoint and cloud detections, CrowdSec for shared behavioural decisions, MISP for analyst enrichment, and LIP as a read-only summary layer. Technical challenges # Wazuh Manager and Indexer required separate access models. Microsoft 365 events needed local rule logic and reproducible tests. Internal certificates lacked the service DNS identities used by Kubernetes. Alert volume also required clear severity and evidence boundaries so that thousands of rule matches did not become thousands of claimed incidents.\nProposed solution # Wazuh Manager master and worker components collect and process data, while the Indexer stores searchable alerts and the Dashboard remains authoritative for detailed analysis. CrowdSec centralises behavioural decisions and supplies the Cloudflare Worker bouncer. MISP Core and Modules provide analyst-operated enrichment. LIP consumes bounded read-only summaries.\nDetection engineering # The local catalogue contains one pipeline-validation rule and eight Microsoft 365 behavioural detections. Richard implemented and owns the catalogue. Some concepts were adapted from pinned SOCFortress material; others were developed locally with AI-assisted research. Every adopted rule was placed in the local rule-ID scheme and supported by documentation, positive fixtures, negative fixtures and expected-result files.\nFixtures use a mixture of sanitised event structures and synthetic validation data. This provides repeatable tests without publishing raw production events. Rule matches are described as evidence requiring investigation, not proof of malicious activity.\nSecurity controls # Separate read-only Wazuh Manager and Indexer identities Indexer search limited to alert indices All-index access tested and denied SOPS-encrypted connector credentials NetworkPolicy limited to Wazuh API and Indexer ports No active-response permissions in LIP MISP enrichment remains analyst-operated Source data and rule descriptions separated from analyst conclusions Validation and measurable outcomes # Wazuh version 4.14.7 Enrolled endpoints 8 Active endpoints 7 Disconnected endpoints 1 Observed 24-hour alerts 4,312 Recent alerts returned 10 Custom rules represented 9 Custom-rule matches 9 Portal egress ports 55000 and 9200 The 4,312-alert figure is a dated observation, not a permanent KPI. Validation also confirmed that Wazuh Dashboard, Indexer, Manager master, Manager worker, CrowdSec LAPI, the Cloudflare bouncer and four MISP components were ready.\nOutcome # Security monitoring now has an evidence-led workflow rather than a collection of unconnected dashboards. Endpoint state, Microsoft 365 detections, behavioural enforcement and threat-intelligence lookup are available through defined boundaries, while disruptive action stays in the authoritative platforms.\nSkills demonstrated # Wazuh deployment and RBAC OpenSearch index permissions Microsoft 365 audit ingestion Detection engineering and fixtures CrowdSec architecture MISP integration TLS troubleshooting SOC interface and triage design Lessons learned # Detection ownership includes provenance and testing. Adapting an upstream concept is professional when the source is pinned, local behaviour is understood and both positive and negative cases are tested.\nCurrent status and next steps # Operational. The platform remains detection and investigation focused. Automated response is deliberately excluded until narrow actions, human approval and audit controls are designed.\nExplore the implementation # Browse the public security runbooks.\nHow it was done # Defined the evidence sources. Endpoint and host data were assigned to Wazuh, behavioural service activity to CrowdSec, Microsoft 365 activity to unified audit ingestion, and threat-intelligence lookup to MISP. Deployed the Wazuh stack. Manager master and worker StatefulSets, the Indexer and Dashboard were installed and validated as separate but connected components. Enrolled endpoints. Agents were added and monitored for active and disconnected state. The final evidence snapshot contained eight enrolled endpoints, seven active and one disconnected. Connected Microsoft 365 telemetry. Protected credentials were supplied through SOPS-encrypted Secrets and audit events entered the Wazuh processing pipeline. Built the detection catalogue. One pipeline-validation rule and eight operational Microsoft 365 detections were assigned local rule IDs. Some concepts were adapted from pinned SOCFortress material and others were developed locally with AI-assisted research. Created repeatable tests. Positive fixtures, negative-operation fixtures, negative-result fixtures and expected-result files were added using a mixture of sanitised event structures and synthetic test data. Deployed CrowdSec and MISP. CrowdSec LAPI centralised behavioural decisions for connected agents and bouncers. MISP Core and Modules were deployed for analyst-operated enrichment. Created least-privilege portal identities. Separate Wazuh Manager and Indexer readers were limited to the data required by LIP. Unrelated Indexer catalogue access was tested and denied. Restricted network access. Portal egress was limited to Wazuh Manager port 55000 and Indexer port 9200. No active-response permissions were provided. Validated the live SOC view. The production API returned Wazuh 4.14.7, endpoint status, 4,312 alerts in the observed 24-hour window, ten recent alerts and nine custom-rule matches. Evidence methodology # The figures in this case study were derived from reproducible source and runtime checks rather than estimates.\nEndpoint and alert measurements came from the production Wazuh Manager and Indexer through the read-only SOC API. Each custom rule has source, documentation, positive fixtures, negative fixtures and expected-result files in Git. Least privilege was tested by allowing required Wazuh alert searches while denying unrelated all-index access. Limitations and residual risks # Alert volume is not equivalent to incident count. One endpoint was disconnected in the evidence snapshot and requires interpretation, not an automatic compromise conclusion. MISP enrichment remains analyst-operated. Some rule concepts originated upstream and are locally adapted rather than all being original from first principles. Plain-English glossary # SIEM A platform that collects, searches and correlates security event data. Detection fixture A controlled example event used to prove that a rule should or should not match. LAPI CrowdSec Local API, the central service that distributes decisions to connected components. Independent validation # This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.\n","date":"26 August 2026","externalUrl":null,"permalink":"/projects/security-monitoring-platform/","section":"Projects","summary":"A detection and investigation platform combining Wazuh, Microsoft 365 audit data, CrowdSec, MISP and a least-privilege SOC portal.","title":"Building a Homelab Security Monitoring Platform","type":"projects"},{"content":"","date":"26 August 2026","externalUrl":null,"permalink":"/tags/misp/","section":"Tags","summary":"","title":"MISP","type":"tags"},{"content":"","date":"26 August 2026","externalUrl":null,"permalink":"/tags/opensearch/","section":"Tags","summary":"","title":"OpenSearch","type":"tags"},{"content":"Project date: 2026-08-24 Role: Architecture, implementation, validation and operation\nA website-like internal operations portal that unifies live telemetry from eleven validated APIs without replacing authoritative tools.\nEnvironment and operating context # Grafana and specialist platforms provided valuable views, but understanding the whole homelab still required moving between Kubernetes, Proxmox, Longhorn, UniFi, Wazuh, Flux and several dashboards. The goal was a customisable interface that looked and behaved like a coherent website, offered straightforward access to every domain and could evolve into the front end for future AI-assisted operations.\nObjective # LIP was designed as a read-only intelligence and navigation layer. It needed to summarise current state, preserve source ownership, distinguish missing data from healthy zero values, keep credentials on the server side and run on the ARM64 cluster through the same GitOps process as other applications.\nOperational value and risk reduction # Before: Establishing overall health required visiting multiple specialist interfaces and interpreting each data model separately.\nAfter: One website-like portal presents current evidence from eleven validated APIs and links back to authoritative systems.\nRisk reduced: Partial connector failures are exposed explicitly, credentials remain server-side, and the convenience layer has no infrastructure-changing permissions.\nArchitectural decisions and alternatives # Grafana only Excellent for metric dashboards, but less suited to mixed inventory, navigation, documented states and custom cross-system workflows. Direct links page Simple, but provides no current evidence or normalised exception handling. Buy a commercial operations platform Potentially broader, but unnecessary cost and less control over a deeply customised homelab model. Selected design A Next.js portal with server-side read-only connectors provided the required website experience and extension path for AI. Technical challenges # The hardest integration was SOC. Wazuh Manager and Indexer use different authentication models, and the internal certificates did not match the Kubernetes service DNS names. Certificate material also had to be exported and imported safely across volume contexts. Multi-source dashboards introduced another challenge: one unavailable connector could not be allowed to crash an otherwise useful page.\nProposed solution # A Next.js application provides static page shells and server-side API routes. Each route makes fixed, bounded queries to its source, normalises the response and exposes only the data required by the browser. The portal calls Prometheus, Proxmox telemetry, UniFi Network and Protect, Wazuh, storage and Flux evidence without accepting arbitrary infrastructure queries.\nBrowser -\u0026gt; Next.js UI -\u0026gt; server-side read-only APIs |-- Prometheus / Kubernetes / Flux |-- Proxmox / storage |-- UniFi Network / Protect `-- Wazuh Manager / Indexer Implementation # The portal includes Overview, Administration, Infrastructure, Inventory, Kubernetes, Monitoring, Network, Proxmox, Services, SOC, Storage and CCTV pages. Knowledge content was deliberately removed so Git repositories remain the documentation authority.\nGitHub Actions builds AMD64 and ARM64 images and publishes them to GHCR. Flux image automation selects the main image digest and updates the GitOps repository. Kubernetes then reconciles the Deployment and its encrypted connector credentials.\nSecurity and operational controls # Server-side connector credentials only SOPS-encrypted Kubernetes Secrets Dedicated Wazuh Manager and Indexer readers NetworkPolicy restricted to ports 55000 and 9200 No arbitrary PromQL from users No restart, reconciliation, response or backup actions Source-system authentication remains authoritative Partial connector failures render as unavailable rather than zero Validation and measurable outcomes # Production APIs tested 11 Successful HTTP responses 11 Failed responses 0 Portal pod readiness 1 of 1 Pod restarts 0 Flux Kustomization Ready Container reference Immutable SHA-256 digest Image architectures AMD64 and ARM64 Acceptance validation also checked the production build, TypeScript, Kubernetes rollout, mounted credentials and CA bundle, application logs and browser rendering. The Wazuh certificate issue was resolved with pinned trust and narrowly scoped certificate handling rather than globally disabling TLS verification.\nOutcome # LIP now provides one coherent operational surface across the homelab while preserving the depth and authority of the specialist platforms. It shortens the path from “something may be wrong” to the relevant evidence and management tool.\nSkills demonstrated # Next.js and TypeScript API integration and normalisation Prometheus and Kubernetes telemetry GitHub Actions and multi-architecture containers Flux deployment TLS and certificate troubleshooting Least privilege and NetworkPolicy Operational user-experience design Lessons learned # A trustworthy portal is defined by failure behaviour as much as successful data. An unavailable connector must never become a reassuring zero, and a ready Deployment must not be described as an end-to-end application check.\nCurrent status and next steps # Operational. The next planned capability is a standalone advisory chatbot, followed later by tightly bounded read-only portal tools and approved documentation retrieval.\nExplore the implementation # View the portal source on GitHub or read the SOC integration case study.\nHow it was done # Defined the portal boundary. LIP was designed as a read-only operational interface, not a replacement control plane. Proxmox, Kubernetes, Grafana, Wazuh, UniFi and other source systems retained authentication and administrative ownership. Created the Next.js application shell. A consistent layout, Sidebar, responsive page structure, status cards, shared tables and navigation were built before live integrations were added. Implemented the first telemetry routes. Server-side API routes queried Prometheus and exposed bounded summaries for Kubernetes, infrastructure and monitoring without accepting arbitrary PromQL from the browser. Added platform-specific connectors. Proxmox, storage, UniFi Network, UniFi Protect and Flux data were introduced through dedicated normalisation routes and dashboard components. Built the SOC integration. Separate least-privilege Wazuh Manager and Indexer identities were created. Certificate material was transferred across storage boundaries, pinned in a mounted trust bundle and restricted by NetworkPolicy. Hardened partial-failure behaviour. Home and Inventory were changed so a 503 response from one optional connector produced an unavailable card rather than a JavaScript exception or a false healthy value. Containerised the application. GitHub Actions validates the Next.js build and publishes AMD64 and ARM64 images to GHCR, allowing development on Windows and deployment to ARM64 k3s. Connected the release to Flux. Image automation detects the new portal image, records an immutable digest in GitOps and reconciles the Kubernetes Deployment. Standardised the source. Compressed generated code was reformatted for maintenance, while graphical Sidebar icons were represented with JavaScript Unicode escapes to avoid Windows encoding corruption. Completed production acceptance. The pod, mounted credentials, CA bundle, logs, browser pages and all eleven API endpoints were checked. Every production endpoint returned HTTP 200. Evidence methodology # The figures in this case study were derived from reproducible source and runtime checks rather than estimates.\nEvery production API route was requested over the deployed ingress and its HTTP status recorded. Eleven of eleven returned HTTP 200. Kubernetes readiness and restart counts were collected for the live portal pod, followed by a recent-error log scan. Build, route, image-digest, Flux reconciliation and browser checks were part of release acceptance. Limitations and residual risks # LIP reports and correlates evidence but is not an authoritative management interface. Some Services entries indicate workload readiness rather than a full transaction-level health test. NetBox is excluded from trusted totals until its records are reconciled. The Wazuh connector currently uses narrowly scoped certificate hostname handling pending certificate reissue. Plain-English glossary # Normalised API A portal endpoint that converts source-specific data into a consistent response for the user interface. Read-only connector An integration identity and route that can retrieve evidence but cannot change the source system. Partial failure A condition where one data source is unavailable while the remainder of the page stays usable. Independent validation # This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.\n","date":"24 August 2026","externalUrl":null,"permalink":"/projects/observability-platform/","section":"Projects","summary":"A website-like internal operations portal that unifies live telemetry from eleven validated APIs without replacing authoritative tools.","title":"Building the Lintech Intelligence Platform","type":"projects"},{"content":"","date":"24 August 2026","externalUrl":null,"permalink":"/tags/flux/","section":"Tags","summary":"","title":"Flux","type":"tags"},{"content":"","date":"24 August 2026","externalUrl":null,"permalink":"/tags/next.js/","section":"Tags","summary":"","title":"Next.js","type":"tags"},{"content":"","date":"24 August 2026","externalUrl":null,"permalink":"/categories/operations/","section":"Categories","summary":"","title":"Operations","type":"categories"},{"content":"Project date: 2026-08-21 Role: Architecture, implementation, validation and operation\nA layered recovery system combining Longhorn replication, daily snapshots, weekly QNAP backups and Veeam tape archival.\nEnvironment and operating context # The cluster runs stateful databases, monitoring systems and security services. Longhorn replication protects availability during normal node disruption, but replication cannot protect against every form of deletion, corruption or administrative error. The recovery design therefore separates availability, short-path recovery, off-cluster backup and archive.\nObjective # The project aimed to provide fast operational recovery, retain independent backup copies and use existing QNAP and Dell tape infrastructure without presenting untested controls as complete.\nOperational value and risk reduction # Before: Storage replication existed as an availability control, but recovery layers and their validation boundaries were less explicit.\nAfter: Replication, 14 daily snapshots, four weekly QNAP backups and the Veeam tape write path have defined responsibilities.\nRisk reduced: Recent deletion, loss of an active volume and loss of online cluster storage no longer depend on one recovery mechanism.\nArchitectural decisions and alternatives # Rely on Longhorn replicas only Supports node failure but can replicate deletion or corruption. NAS-only persistent volumes Centralises data and simplifies some backups, but creates a different availability dependency. Kasten Considered previously, but not configured and therefore excluded from the validated design. Selected design Longhorn for active storage and operational recovery, QNAP for off-cluster backup, and PowerVault tape for a separate archival path. Technical challenges # The platform needed enough free disk for three-replica Longhorn volumes and replica rebuilding. Backup jobs also had to cross system boundaries: Kubernetes writes to QNAP, then a dedicated Proxmox VM uses Veeam and a SAS-presented PowerVault TL2000 to handle tape. Expected states, such as the Backup VM being powered off, had to be kept distinct from failures.\nProposed solution # Daily Longhorn snapshots provide short-path recovery and retain fourteen points. Weekly full backups run at 01:00 every Saturday and retain four copies on QNAP. Completed backup material then enters the documented Veeam tape workflow. There is no separate monthly Longhorn RecurringJob in the observed configuration, so no monthly implementation is claimed.\nThree-replica Longhorn volume |-- Daily snapshots, retain 14 `-- Weekly full backup, retain 4 -\u0026gt; QNAP `-\u0026gt; Veeam -\u0026gt; PowerVault TL2000 Implementation # Longhorn allocates approximately 1.88 TiB on each of four nodes and uses a default replica count of three. The observed 24 volumes totalled 456.1 GiB of provisioned capacity, leaving substantial raw space for replicas, rebuilding and growth.\nVM 101, named Backup, receives the PowerVault TL2000 through SAS passthrough and is started only when required. This reduces unnecessary runtime and makes its powered-off condition an expected operating state.\nRecovery validation # Wazuh data was used to exercise the Longhorn recovery workflow. The task required persistent application content and certificate material to remain usable after restoration, providing a more meaningful test than checking only a backup-job status. The evidence supports a successful Longhorn recovery and an implemented tape write path. It does not conclusively prove an end-to-end restoration from tape, so that claim is deliberately excluded.\nMeasurable outcomes # Longhorn volumes 24 Healthy and attached 24 of 24 Provisioned capacity 456.1 GiB Default replica count 3 Daily snapshots retained 14 Weekly backups retained 4 Weekly schedule Saturday at 01:00 Backup destination QNAP Validated recovery workload Wazuh Outcome # The environment now has clearly separated recovery layers with documented ownership and validation. Storage replication supports availability, snapshots address recent operational mistakes, QNAP holds off-cluster full backups, and tape provides a separate archive path.\nSkills demonstrated # Longhorn capacity and replica planning Kubernetes recovery QNAP integration Proxmox passthrough Veeam and tape workflow design Recovery validation and evidence classification Lessons learned # Replication is not backup, and successful backup creation is not the same as successful restoration. Recovery documentation also needs to state what has not been tested, especially for offline media.\nCurrent status and next steps # Operational. Daily snapshots, weekly backups and Wazuh recovery are evidenced. A full tape restoration test remains an appropriate future validation milestone.\nExplore the implementation # Review the Kubernetes platform that hosts these volumes.\nHow it was done # Separated availability from backup. The design began by treating Longhorn replicas, snapshots, external backups and tape archives as different controls with different failure boundaries. Deployed Longhorn storage. Each of the four k3s nodes contributed approximately 1.88 TiB of NVMe-backed capacity. A default replica count of three was selected for Kubernetes volumes. Assigned storage classes. General Longhorn, static Longhorn, workload-specific Wazuh, SMB and local storage classes were retained so applications could use the appropriate persistence model. Created the snapshot schedule. A recurring job takes daily snapshots at 02:00 and retains fourteen local recovery points for recent operational mistakes. Created the backup schedule. A second recurring job performs a full Longhorn backup at 01:00 each Saturday and retains four copies on the QNAP backup target. Monitored capacity and replica health. Volume state, robustness, attachment and free node capacity were reviewed to ensure replica rebuilding and recovery remained possible. Performed a real recovery exercise. Wazuh persistent data was selected because the workload includes state and certificate dependencies. Restored material was inspected to confirm the required application data remained available. Integrated the tape workflow. Proxmox VM 101, named Backup, was configured to access the SAS-connected Dell PowerVault TL2000. Veeam handles completed backup material and writes it to tape on the following day. Documented expected states. The Backup VM is intentionally powered off outside backup operations, and temporary tape cleaning or maintenance states are not treated automatically as hardware failure. Recorded validation boundaries. Longhorn recovery and the tape write path are evidenced. A separate monthly Longhorn job and a completed end-to-end tape restoration are not claimed. Evidence methodology # The figures in this case study were derived from reproducible source and runtime checks rather than estimates.\nVolume totals, robustness, attachment and provisioned bytes were collected from Longhorn volume custom resources. RecurringJob resources supplied the live snapshot and backup schedules and retention values. Recovery evidence came from a Wazuh volume exercise that confirmed required persistent data and certificate material remained usable. Limitations and residual risks # No separate monthly Longhorn RecurringJob was present in the evidence snapshot. The tape write path is implemented, but a complete end-to-end tape restore is not claimed as validated. No measured recovery-time or recovery-point objective is claimed. Longhorn replicas share one physical site and do not protect against total site loss. Plain-English glossary # Replica An additional copy of a Longhorn volume maintained on another storage node. Snapshot A point-in-time recovery reference associated with the active storage system. Backup A copy moved to an external backup target so recovery does not depend only on active cluster storage. Independent validation # This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.\n","date":"21 August 2026","externalUrl":null,"permalink":"/projects/longhorn-storage-recovery/","section":"Projects","summary":"A layered recovery system combining Longhorn replication, daily snapshots, weekly QNAP backups and Veeam tape archival.","title":"Designing Layered Kubernetes Backup and Recovery","type":"projects"},{"content":"","date":"21 August 2026","externalUrl":null,"permalink":"/tags/powervault/","section":"Tags","summary":"","title":"PowerVault","type":"tags"},{"content":"","date":"21 August 2026","externalUrl":null,"permalink":"/tags/qnap/","section":"Tags","summary":"","title":"QNAP","type":"tags"},{"content":"","date":"21 August 2026","externalUrl":null,"permalink":"/categories/recovery/","section":"Categories","summary":"","title":"Recovery","type":"categories"},{"content":"","date":"21 August 2026","externalUrl":null,"permalink":"/tags/veeam/","section":"Tags","summary":"","title":"Veeam","type":"tags"},{"content":"","date":"18 August 2026","externalUrl":null,"permalink":"/categories/gitops/","section":"Categories","summary":"","title":"GitOps","type":"categories"},{"content":"Project date: 2026-08-18 Role: Architecture, implementation, validation and operation\nEncryption and recovery controls for 31 Git-managed Kubernetes Secret manifests using SOPS, age and Flux.\nEnvironment and operating context # GitOps needed complete application definitions, but many workloads depended on database credentials, tunnel tokens, storage access, monitoring accounts and security API keys. Leaving those resources outside Git would preserve manual deployment gaps; storing them in plaintext would create an unacceptable exposure.\nObjective # The objective was to store Secret manifests beside their applications while ensuring that usable values never entered Git history. The design also needed a tested recovery path, because encryption without recoverable key custody would create a new single point of failure.\nOperational value and risk reduction # Before: A complete rebuild required credentials to be recreated or supplied through steps outside Git.\nAfter: Thirty-one encrypted Secret manifests can be reconciled with their workloads, and the protected decryption identity has been recovery-tested.\nRisk reduced: Usable credentials are no longer stored in plaintext source, while disaster recovery no longer depends on remembering how to reconstruct every Secret manually.\nArchitectural decisions and alternatives # Sealed Secrets Strong Kubernetes-focused option, but encryption and recovery are tied more closely to the cluster-side controller and key lifecycle. External Secrets Operator Useful when an external secret store already exists, but would introduce another runtime dependency and secret backend. HashiCorp Vault Powerful central secret platform, but operationally heavier than required for this single-operator GitOps environment. Selected design SOPS with age kept encrypted values beside workload definitions, supported local review and worked directly with Flux decryption. Technical challenges # Base64 encoding is not encryption, and simply excluding Secret resources would make a rebuild incomplete. The project needed selective field encryption, consistent file naming, Flux decryption, temporary-file controls and a protected off-Git age identity.\nProposed solution # SOPS encrypts only Kubernetes data and stringData values, leaving metadata reviewable. Three path-based rules cover cluster Secrets, applications and infrastructure. The public age recipient can encrypt; the private identity is protected separately and has been used successfully to validate recovery.\nTemporary Secret -\u0026gt; SOPS encryption -\u0026gt; Git | Protected age identity -\u0026gt; Flux decryption | Kubernetes Secret Implementation # The migration covered application databases, Grafana, the LIP SOC connector, Mealie, media VPN and SMB credentials, n8n, NetBox, Cloudflare Tunnel, WordPress, CrowdSec, MISP, Proxmox and UniFi monitoring, Wazuh certificates and credentials, Microsoft 365 audit ingestion and threat-intelligence keys.\nA validation Secret was retained as historical evidence that the encryption and decryption workflow functioned before production credentials were migrated. Temporary plaintext material is handled outside Git and removed after the encrypted result has been checked.\nValidation and measurable outcomes # SOPS-encrypted manifests 31 Files with AES-256-GCM markers 31 Marker coverage 31 of 31 Encryption rule scopes 3 Protected recovery identity Tested successfully A repository scan found no tracked private-key block. One documentation line contained the search phrase BEGIN PRIVATE KEY; this was a defensive command example, not key material. Flux has also reconciled encrypted Secrets into running applications, including the SOC connector.\nSecurity controls # Private age identity remains outside Git Only sensitive fields are encrypted Dedicated .enc.yaml naming Least-privilege source-system credentials No infrastructure credentials delivered to browser code Repository scans before commit Recovery copy tested, not merely stored Outcome # Git can represent complete workload deployments without containing usable credentials. The result closes a major recovery gap in the GitOps model and makes Secret changes attributable while maintaining confidentiality.\nSkills demonstrated # SOPS and age Kubernetes Secret design Flux decryption Key custody and recovery Secure migration and validation Git history and plaintext hygiene Lessons learned # Secret management is not solved by choosing an encryption tool. Recovery, rotation, least privilege, temporary files and evidence that every eligible file is encrypted are equally important.\nCurrent status and next steps # Operational. All 31 evidenced encrypted manifests contained SOPS AES-GCM markers, and the protected recovery identity has been tested successfully.\nExplore the implementation # Read the Flux delivery case study.\nHow it was done # Identified Secret-bearing workloads. Database accounts, application credentials, tunnel tokens, monitoring users, storage access and security integration keys were catalogued before migration. Generated an age identity. The public recipient was used for encryption, while the private identity was kept outside Git and protected as recovery material. Created path-based SOPS rules. A repository-level policy was defined for cluster Secrets, application manifests and infrastructure manifests. Only Kubernetes data and stringData fields were encrypted. Tested the workflow safely. A non-production test Secret was encrypted, decrypted with authorised recovery material and reconciled through Flux before production credentials were migrated. Configured Flux decryption. Flux was given access to the protected cluster-side age identity so encrypted manifests could be decrypted only during reconciliation. Migrated existing credentials. Secret resources were created using temporary protected files, encrypted into .enc.yaml files and added to the relevant application or infrastructure Kustomization. Changed workloads to Secret references. Deployments consumed credentials through Kubernetes Secret references instead of embedding values in manifests or browser-delivered code. Removed temporary plaintext. Temporary source files were deleted after verifying the encrypted result. Repository searches were used to check for private-key markers and exposed values. Validated application consumption. Reconciled Secrets were checked for expected key names without showing values, and consuming applications were rolled out and tested. Verified coverage and recovery. All 31 encrypted manifests contained AES-256-GCM markers, and the protected recovery identity was tested successfully. Evidence methodology # The figures in this case study were derived from reproducible source and runtime checks rather than estimates.\nEncrypted manifests were counted with a repository search for files ending in .enc.yaml. Each counted file was checked for an ENC[AES256_GCM] marker; 31 files were found and all 31 contained the marker. The SOPS policy was inspected to confirm encryption of Kubernetes data and stringData fields across cluster, application and infrastructure paths. Recovery was tested using the protected age identity, and Flux consumption was validated through successfully reconciled workloads. Limitations and residual risks # The protected age identity remains a critical recovery dependency and requires continued custody and testing. Secrets are encrypted in Git but exist decrypted in Kubernetes and application memory at runtime. Encryption does not replace credential rotation, least privilege or source-system revocation. Marker coverage proves that files contain encrypted values; it does not by itself prove that every possible credential in the wider environment has been migrated. Plain-English glossary # SOPS A tool that encrypts selected values inside structured files while leaving non-sensitive metadata readable. age identity The private key material used to decrypt content encrypted to an age recipient. Flux decryption Flux reads encrypted manifests and decrypts them inside the cluster immediately before applying the Kubernetes resource. Independent validation # This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.\n","date":"18 August 2026","externalUrl":null,"permalink":"/projects/sops-age-secret-management/","section":"Projects","summary":"Encryption and recovery controls for 31 Git-managed Kubernetes Secret manifests using SOPS, age and Flux.","title":"Securing GitOps Secrets with SOPS and age","type":"projects"},{"content":"","date":"9 August 2026","externalUrl":null,"permalink":"/tags/github/","section":"Tags","summary":"","title":"GitHub","type":"tags"},{"content":"","date":"9 August 2026","externalUrl":null,"permalink":"/tags/helm/","section":"Tags","summary":"","title":"Helm","type":"tags"},{"content":"Project date: 2026-08-09 Role: Architecture, implementation, validation and operation\nA declarative delivery system using Flux, Kustomize, Helm, image automation and encrypted Git-managed secrets.\nEnvironment and operating context # The Kubernetes platform grew from manually managed resources into a service environment with databases, media applications, networking, monitoring, security and internal tools. Early deployments relied on locally held YAML and manual Kubernetes or Helm operations. That approach was useful for experimentation but made it difficult to prove that source control and the live cluster represented the same system.\nBusiness and technical objective # The objective was to make Git the authoritative desired state, preserve reviewable history and allow the cluster to converge automatically. A successful design also needed to support Helm-based applications, direct Kubernetes resources, encrypted secrets and automated image updates without making a mutable image tag the only production reference.\nOperational value and risk reduction # Before: Deployments were performed from locally held manifests and Helm commands, leaving more operational knowledge outside the cluster definition.\nAfter: Git records application, infrastructure, image and encrypted Secret state, while Flux continuously reconciles the cluster.\nRisk reduced: Configuration drift and undocumented rebuild steps are reduced because the deployed state has a versioned source and rollback coordinate.\nArchitectural decisions and alternatives # Continue manual kubectl and Helm Fast for isolated changes, but weak for drift control, reviewability and disaster reconstruction. CI pipeline pushes directly to the cluster Automates deployment, but grants the pipeline write access and does not provide continuous reconciliation. Argo CD A credible alternative with a strong UI, but Flux aligned well with the repository-driven, controller-based and image-automation requirements. Selected design Flux with Kustomize, HelmRelease, SOPS and image automation provided a compact Git-native operating model. Technical challenges # The hard part was migrating stateful, already-running applications without losing configuration or replacing resources unexpectedly. Dependencies between namespaces, storage, Secrets, Helm releases and monitoring had to be represented explicitly. Image delivery also crossed two repositories: application source produced the image, while the GitOps repository recorded the immutable digest.\nProposed solution # The repository was divided into application, infrastructure and cluster-composition layers. Kustomize manages direct resources and composition, while HelmRelease objects manage upstream charts. Flux ImageRepository and ImagePolicy resources discover and select images; ImageUpdateAutomation writes the selected digest into Git. SOPS decrypts eligible Secret manifests only during reconciliation.\nSource commit -\u0026gt; CI build -\u0026gt; GHCR image | v ImageRepository -\u0026gt; ImagePolicy -\u0026gt; Git digest update | v Flux reconciliation Implementation # Applications were migrated incrementally. Each migration identified the namespace, service account, configuration, storage, network exposure, Secret references and dependency order. Existing resources were rendered and compared before Flux assumed ownership.\nThe current repository contains Kustomizations for applications such as databases, Grafana, Headlamp, LIP, Mealie, media services, n8n, NetBox, networking, Ombi, Vaultwarden and WordPress, plus infrastructure layers for CrowdSec, logging, MISP, monitoring and Traefik.\nValidation and measurable outcomes # GitRepository 1 of 1 ready Kustomizations 19 of 19 ready ImageRepositories 17 of 17 ready ImagePolicies 17 of 17 ready ImageUpdateAutomations 1 of 1 ready Suspended Flux resources 0 Observed applied revision main@sha1:227ff0ae Readiness was collected directly from Flux custom resources. Image policies resolved to immutable SHA-256 digests, including the ARM64-compatible LIP image. The evidence demonstrates the whole delivery chain rather than only a successful kubectl apply.\nOperational controls # Small reviewable commits Kustomization dependency ordering Render and server-side validation SOPS decryption during reconciliation Immutable digest recording Rollout and API checks after deployment Git history as the rollback coordinate Outcome # The cluster can now be reconstructed and maintained from source-controlled desired state. Image changes are attributable, reconciliation health is measurable, and emergency changes cannot remain an undocumented alternative state.\nSkills demonstrated # Flux controllers and custom resources Kustomize and HelmRelease design Container image automation Git-based release management Stateful workload migration Dependency and rollback planning Lessons learned # Installing Flux is straightforward; adopting GitOps is an operating-model change. Repository structure, dependency ownership, Secret handling and post-deployment validation determine whether reconciliation is genuinely reliable.\nCurrent status and next steps # Operational, with all observed Flux resources ready. The existing direct-to-main workflow is effective for a single operator; pull-request policy can be added when collaboration or stronger approval requirements justify it.\nExplore the implementation # See how secrets were brought into GitOps safely.\nHow it was done # Assessed the existing deployment model. Applications previously managed through locally held YAML, manual Kubernetes commands and Helm operations were identified, together with their storage, configuration, Secret and networking dependencies. Designed the repository layout. Workloads were separated into apps, shared platform services into infrastructure, and cluster composition into clusters. This made ownership and reconciliation order easier to understand. Bootstrapped Flux. Flux controllers were installed and connected to the Git repository through a GitRepository source. The initial cluster Kustomization established continuous reconciliation. Created platform Kustomizations. Separate Kustomizations were introduced for databases, monitoring, logging, networking, security and individual applications. Dependencies were used where one layer required another. Converted chart-based services. Applications best maintained through upstream Helm charts were represented as HelmRelease resources, while direct Kubernetes resources remained under Kustomize. Migrated applications in small batches. Existing resources were rendered and compared before Flux took ownership. Each migration was followed by rollout, storage, ingress and application checks. Integrated encrypted Secrets. SOPS and age were introduced so Secret manifests could be committed alongside workloads without placing usable credentials in Git. Added image automation. ImageRepository resources scan registries, ImagePolicies select acceptable images, and ImageUpdateAutomation records selected immutable digests in the GitOps repository. Defined the release workflow. Application source changes trigger CI builds, multi-architecture images are published to GHCR, Flux discovers the new image and Kubernetes reconciles the digest recorded in Git. Validated the complete chain. The final snapshot showed one ready Git source, nineteen ready Kustomizations, seventeen ready image repositories, seventeen ready policies, one ready automation and no suspended resources. Evidence methodology # The figures in this case study were derived from reproducible source and runtime checks rather than estimates.\nCounts were taken directly from Flux GitRepository, Kustomization, ImageRepository, ImagePolicy and ImageUpdateAutomation custom resources. A resource counted as ready only when its Ready condition was True. Suspended state was checked separately. Delivery validation combined Flux readiness with image-digest selection, Kubernetes rollout checks and application/API verification. Limitations and residual risks # Direct commits to main are appropriate for the current single-operator model but provide less separation than pull-request approval. GitOps can reproduce declared state, but cannot recover application data without the storage and backup design. A valid manifest can still contain a poor operational decision, so review and post-deployment testing remain necessary. Plain-English glossary # Reconciliation A controller repeatedly compares desired Git state with live cluster state and corrects differences. Kustomization A Flux resource that builds and applies a defined set of Kubernetes manifests. ImagePolicy A Flux resource that selects an acceptable image version or digest from a scanned repository. Independent validation # This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.\n","date":"9 August 2026","externalUrl":null,"permalink":"/projects/flux-gitops-platform/","section":"Projects","summary":"A declarative delivery system using Flux, Kustomize, Helm, image automation and encrypted Git-managed secrets.","title":"Implementing GitOps Delivery with Flux","type":"projects"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/arm64/","section":"Tags","summary":"","title":"ARM64","type":"tags"},{"content":"Project date: 2026-07-12 Role: Architecture, implementation, validation and operation\nA power-efficient, highly available four-node Kubernetes platform built with Raspberry Pi 5 hardware, k3s, Longhorn and GitOps.\nEnvironment and operating context # The homelab is the technical foundation for a longer-term goal of building a locally operated whole-house AI platform. Before any intelligence layer could be credible, the environment needed dependable compute, persistent storage, networking, monitoring, security and recovery. The Kubernetes project established that foundation.\nThe platform consists of four identical Raspberry Pi 5 Model B Rev 1.1 systems. Each node provides four ARM64 CPU cores, approximately 16 GB of memory, a 1.9 TB NVMe device and 1 Gbit Ethernet. Debian 13 and k3s provide a compact, power-conscious platform capable of running continuously without adding another high-power rack server.\nBusiness and technical objective # The objective was not simply to demonstrate that Kubernetes could run on Raspberry Pi hardware. The goal was to create an operable environment that could survive an individual node outage, reschedule applications, protect persistent data, expose services consistently and be rebuilt from documented desired state.\nMaintain an odd-numbered etcd quorum. Use all four nodes effectively while retaining a dedicated worker. Provide replicated persistent volumes and external SMB storage. Support ingress, load-balanced addresses and internal DNS. Make readiness, workloads and failures observable. Create a platform suitable for later automation and AI services. Operational value and risk reduction # Before: Services depended more heavily on individual hosts and manual deployment knowledge.\nAfter: Four standardised ARM64 nodes provide a three-member control plane, reschedulable workloads, replicated storage and Git-managed operations.\nRisk reduced: Loss of one server no longer means loss of the Kubernetes control plane, provided etcd quorum and the remaining dependencies stay healthy.\nArchitectural decisions and alternatives # Single-node k3s Lower cost and simpler operation, but no control-plane resilience and limited failure testing. One server with three workers More worker capacity, but the server would remain a control-plane single point of failure. Additional rack-server virtual machines More raw capacity, but higher power use and less alignment with the low-power, always-on objective. Selected design Three embedded-etcd server nodes and one worker on identical Raspberry Pi 5 hardware balanced quorum, efficiency and practical learning. Technical challenges # ARM64 compatibility, resource limits and distributed state introduced constraints not present in a single Docker host. High availability also required more than adding nodes: control-plane quorum, storage replicas, pod scheduling, application design and network reachability all had to work together.\nThe most difficult engineering problem was achieving useful high availability across the platform so that a node outage did not leave applications tied permanently to one device. This required three embedded-etcd server nodes, schedulable control-plane capacity, a dedicated worker and Longhorn volumes able to remain available as pods moved.\nProposed solution and architecture # Three nodes were configured as k3s server and embedded-etcd members, giving the control plane an odd-numbered quorum. Node 3 operates as the dedicated worker, while server nodes remain schedulable so their resources are not wasted. Traefik handles ingress, MetalLB provides selected service addresses, and Longhorn supplies three-replica Kubernetes storage.\nGit and Flux | Three k3s server/etcd nodes + one worker |-- Traefik and MetalLB |-- Longhorn replicated storage |-- Prometheus, Grafana and Loki |-- Wazuh, CrowdSec and MISP `-- Application workloads Implementation # The build proceeded in controlled layers: operating-system and network preparation, k3s bootstrap, server and worker enrolment, ingress, load balancing, storage, monitoring, logging, security and finally application migration. This limited the number of variables introduced at once.\nLonghorn uses an approximately 1.88 TiB storage allocation on each node and a default replica count of three. Kubernetes also has separate local-path, static Longhorn, SMB and workload-specific storage classes, allowing storage behaviour to match application requirements. Flux now owns application and infrastructure reconciliation.\nValidation and measurable outcomes # Nodes ready 4 of 4 Control-plane and etcd members 3 Dedicated workers 1 Deployments 66, all ready StatefulSets 6 DaemonSets 10 Running pods 122 StorageClasses 6 Longhorn volumes 24 of 24 healthy and attached Provisioned Longhorn capacity 456.1 GiB Default Longhorn replicas 3 The evidence snapshot was collected from the live cluster using Kubernetes and Longhorn APIs. All Deployments reported their desired ready replica count, and all four nodes ran the same k3s version on Debian 13 ARM64. These figures are point-in-time operational evidence, not synthetic portfolio metrics.\nOutcome # The result is a compact cloud-native platform that supports real personal and household services while remaining power efficient. Applications are no longer constrained to one node, persistent data has defined replication and recovery layers, and the entire environment exposes enough operational evidence to diagnose rather than guess.\nSkills demonstrated # Kubernetes and k3s architecture ARM64 platform engineering etcd quorum and scheduling Traefik, MetalLB and service networking Longhorn and SMB storage Observability and security integration Failure analysis and operational validation Lessons learned # High availability is a chain rather than a checkbox. A healthy control plane does not make an application highly available if its storage, scheduling or dependencies cannot move. The project reinforced the value of validating each layer and describing limitations honestly.\nCurrent status and next steps # Operational. The platform now provides the runtime for LIP and the security, monitoring and documentation services. Its next major workload is the planned chatbot and locally oriented AI platform.\nExplore the implementation # Read the Flux GitOps case study or review the recovery design.\nHow it was done # Defined the target architecture. The cluster was designed around four power-efficient Raspberry Pi 5 systems. Three nodes were assigned as k3s server and embedded-etcd members to maintain quorum, while the fourth became a dedicated worker. Server nodes remained schedulable so their capacity could still support workloads. Standardised the node build. Each node was prepared with the same Raspberry Pi 5 platform, approximately 16 GB of memory, a 1.9 TB NVMe device, Debian 13 and gigabit Ethernet. Consistent hardware and operating-system configuration reduced node-specific troubleshooting. Bootstrapped the control plane. The first k3s server was installed and used to initialise the embedded-etcd cluster. Two additional server nodes were joined to establish the three-member control plane, followed by the worker node. Validated cluster membership. Node roles, versions, internal connectivity and readiness were checked before application services were introduced. This established a known-good control-plane baseline. Added service networking. Traefik was used for ingress routing, MetalLB supplied addresses for selected services, and internal DNS records provided stable names for applications and management interfaces. Introduced persistent storage. Longhorn was deployed across all four nodes with a default replica count of three. Separate storage classes were retained for local, Longhorn, SMB and workload-specific storage requirements. Built the operating layer. Prometheus, Grafana, Loki, Wazuh, CrowdSec and supporting tools were added so that node health, workloads, logs and security events could be inspected rather than inferred. Migrated applications incrementally. Databases, media services, networking, monitoring, security and internal applications were moved in controlled groups, with readiness and persistence checked after each change. Moved ownership to GitOps. Flux became responsible for application and infrastructure reconciliation, ensuring that the working cluster could be reproduced from source-controlled desired state. Performed acceptance validation. The final evidence snapshot confirmed four ready nodes, 66 ready Deployments, six StatefulSets, ten DaemonSets, 122 running pods and 24 healthy attached Longhorn volumes. Evidence methodology # The figures in this case study were derived from reproducible source and runtime checks rather than estimates.\nNode and workload totals were collected with Kubernetes API commands including kubectl get nodes, Deployments, StatefulSets, DaemonSets and running Pods. Longhorn volume health, attachment, capacity and replica settings were collected from Longhorn custom resources. Readiness was accepted only when all four nodes were Ready and every Deployment reported its desired ready-replica count. Limitations and residual risks # Three control-plane nodes protect quorum, but do not make every application highly available. Stateful resilience still depends on Longhorn replicas, scheduling, application architecture and external dependencies. All four nodes share the same power, network and physical location, so site-level faults remain outside the design. The evidence is a dated operational snapshot rather than a formal availability SLA. Plain-English glossary # k3s A lightweight, conformant Kubernetes distribution suited to edge and resource-conscious environments. etcd quorum The majority of control-plane database members required to continue making consistent cluster decisions. Longhorn Kubernetes-native distributed block storage that maintains replicas across nodes. Independent validation # This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.\n","date":"12 July 2026","externalUrl":null,"permalink":"/projects/k3s-cluster/","section":"Projects","summary":"A power-efficient, highly available four-node Kubernetes platform built with Raspberry Pi 5 hardware, k3s, Longhorn and GitOps.","title":"Building a Four-Node ARM64 Kubernetes Platform","type":"projects"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/categories/infrastructure/","section":"Categories","summary":"","title":"Infrastructure","type":"categories"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/metallb/","section":"Tags","summary":"","title":"MetalLB","type":"tags"},{"content":"I work across technical support, platform operations and infrastructure engineering, combining day-to-day troubleshooting discipline with a strong interest in automation, resilience and security.\n","externalUrl":null,"permalink":"/about/","section":"About","summary":"I work across technical support, platform operations and infrastructure engineering, combining day-to-day troubleshooting discipline with a strong interest in automation, resilience and security.\n","title":"About","type":"about"},{"content":"The Lintech homelab is a four-node K3s platform used to practise GitOps, storage, observability, security operations, backup and recovery.\n","externalUrl":null,"permalink":"/homelab/","section":"Homelab","summary":"The Lintech homelab is a four-node K3s platform used to practise GitOps, storage, observability, security operations, backup and recovery.\n","title":"Homelab","type":"homelab"},{"content":"My progression connects operational ownership, workplace support and platform engineering. Each stage has strengthened the same core approach: understand impact, gather evidence, make controlled changes and validate outcomes.\n","externalUrl":null,"permalink":"/journey/","section":"Journey","summary":"My progression connects operational ownership, workplace support and platform engineering. Each stage has strengthened the same core approach: understand impact, gather evidence, make controlled changes and validate outcomes.\n","title":"Journey","type":"journey"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]