Designing Layered Kubernetes Backup and Recovery

Project date: 2026-08-21 Role: Architecture, implementation, validation and operation

A layered recovery system combining Longhorn replication, daily snapshots, weekly QNAP backups and Veeam tape archival.

Environment and operating context

#

The cluster runs stateful databases, monitoring systems and security services. Longhorn replication protects availability during normal node disruption, but replication cannot protect against every form of deletion, corruption or administrative error. The recovery design therefore separates availability, short-path recovery, off-cluster backup and archive.

Objective

#

The project aimed to provide fast operational recovery, retain independent backup copies and use existing QNAP and Dell tape infrastructure without presenting untested controls as complete.

Operational value and risk reduction

#

Before: Storage replication existed as an availability control, but recovery layers and their validation boundaries were less explicit.

After: Replication, 14 daily snapshots, four weekly QNAP backups and the Veeam tape write path have defined responsibilities.

Risk reduced: Recent deletion, loss of an active volume and loss of online cluster storage no longer depend on one recovery mechanism.

Architectural decisions and alternatives

#

Rely on Longhorn replicas only

Supports node failure but can replicate deletion or corruption.

NAS-only persistent volumes

Centralises data and simplifies some backups, but creates a different availability dependency.

Kasten

Considered previously, but not configured and therefore excluded from the validated design.

Selected design

Longhorn for active storage and operational recovery, QNAP for off-cluster backup, and PowerVault tape for a separate archival path.

Technical challenges

#

The platform needed enough free disk for three-replica Longhorn volumes and replica rebuilding. Backup jobs also had to cross system boundaries: Kubernetes writes to QNAP, then a dedicated Proxmox VM uses Veeam and a SAS-presented PowerVault TL2000 to handle tape. Expected states, such as the Backup VM being powered off, had to be kept distinct from failures.

Proposed solution

#

Daily Longhorn snapshots provide short-path recovery and retain fourteen points. Weekly full backups run at 01:00 every Saturday and retain four copies on QNAP. Completed backup material then enters the documented Veeam tape workflow. There is no separate monthly Longhorn RecurringJob in the observed configuration, so no monthly implementation is claimed.

Three-replica Longhorn volume
  |-- Daily snapshots, retain 14
  `-- Weekly full backup, retain 4 -> QNAP
                                         `-> Veeam -> PowerVault TL2000

Implementation

#

Longhorn allocates approximately 1.88 TiB on each of four nodes and uses a default replica count of three. The observed 24 volumes totalled 456.1 GiB of provisioned capacity, leaving substantial raw space for replicas, rebuilding and growth.

VM 101, named Backup, receives the PowerVault TL2000 through SAS passthrough and is started only when required. This reduces unnecessary runtime and makes its powered-off condition an expected operating state.

Recovery validation

#

Wazuh data was used to exercise the Longhorn recovery workflow. The task required persistent application content and certificate material to remain usable after restoration, providing a more meaningful test than checking only a backup-job status. The evidence supports a successful Longhorn recovery and an implemented tape write path. It does not conclusively prove an end-to-end restoration from tape, so that claim is deliberately excluded.

Measurable outcomes

#

Outcome

#

The environment now has clearly separated recovery layers with documented ownership and validation. Storage replication supports availability, snapshots address recent operational mistakes, QNAP holds off-cluster full backups, and tape provides a separate archive path.

Skills demonstrated

#

• Longhorn capacity and replica planning
• Kubernetes recovery
• QNAP integration
• Proxmox passthrough
• Veeam and tape workflow design
• Recovery validation and evidence classification

Lessons learned

#

Replication is not backup, and successful backup creation is not the same as successful restoration. Recovery documentation also needs to state what has not been tested, especially for offline media.

Current status and next steps

#

Operational. Daily snapshots, weekly backups and Wazuh recovery are evidenced. A full tape restoration test remains an appropriate future validation milestone.

Explore the implementation

#

Review the Kubernetes platform that hosts these volumes.

How it was done

#

1. Separated availability from backup. The design began by treating Longhorn replicas, snapshots, external backups and tape archives as different controls with different failure boundaries.
2. Deployed Longhorn storage. Each of the four k3s nodes contributed approximately 1.88 TiB of NVMe-backed capacity. A default replica count of three was selected for Kubernetes volumes.
3. Assigned storage classes. General Longhorn, static Longhorn, workload-specific Wazuh, SMB and local storage classes were retained so applications could use the appropriate persistence model.
4. Created the snapshot schedule. A recurring job takes daily snapshots at 02:00 and retains fourteen local recovery points for recent operational mistakes.
5. Created the backup schedule. A second recurring job performs a full Longhorn backup at 01:00 each Saturday and retains four copies on the QNAP backup target.
6. Monitored capacity and replica health. Volume state, robustness, attachment and free node capacity were reviewed to ensure replica rebuilding and recovery remained possible.
7. Performed a real recovery exercise. Wazuh persistent data was selected because the workload includes state and certificate dependencies. Restored material was inspected to confirm the required application data remained available.
8. Integrated the tape workflow. Proxmox VM 101, named Backup, was configured to access the SAS-connected Dell PowerVault TL2000. Veeam handles completed backup material and writes it to tape on the following day.
9. Documented expected states. The Backup VM is intentionally powered off outside backup operations, and temporary tape cleaning or maintenance states are not treated automatically as hardware failure.
10. Recorded validation boundaries. Longhorn recovery and the tape write path are evidenced. A separate monthly Longhorn job and a completed end-to-end tape restoration are not claimed.

Evidence methodology

#

The figures in this case study were derived from reproducible source and runtime checks rather than estimates.

1. Volume totals, robustness, attachment and provisioned bytes were collected from Longhorn volume custom resources.
2. RecurringJob resources supplied the live snapshot and backup schedules and retention values.
3. Recovery evidence came from a Wazuh volume exercise that confirmed required persistent data and certificate material remained usable.

Limitations and residual risks

#

• No separate monthly Longhorn RecurringJob was present in the evidence snapshot.
• The tape write path is implemented, but a complete end-to-end tape restore is not claimed as validated.
• No measured recovery-time or recovery-point objective is claimed.
• Longhorn replicas share one physical site and do not protect against total site loss.

Plain-English glossary

#

Replica

An additional copy of a Longhorn volume maintained on another storage node.

Snapshot

A point-in-time recovery reference associated with the active storage system.

Backup

A copy moved to an external backup target so recovery does not depend only on active cluster storage.

Independent validation

#

This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.