Securing GitOps Secrets with SOPS and age

Project date: 2026-08-18 Role: Architecture, implementation, validation and operation

Encryption and recovery controls for 31 Git-managed Kubernetes Secret manifests using SOPS, age and Flux.

Environment and operating context

#

GitOps needed complete application definitions, but many workloads depended on database credentials, tunnel tokens, storage access, monitoring accounts and security API keys. Leaving those resources outside Git would preserve manual deployment gaps; storing them in plaintext would create an unacceptable exposure.

Objective

#

The objective was to store Secret manifests beside their applications while ensuring that usable values never entered Git history. The design also needed a tested recovery path, because encryption without recoverable key custody would create a new single point of failure.

Operational value and risk reduction

#

Before: A complete rebuild required credentials to be recreated or supplied through steps outside Git.

After: Thirty-one encrypted Secret manifests can be reconciled with their workloads, and the protected decryption identity has been recovery-tested.

Risk reduced: Usable credentials are no longer stored in plaintext source, while disaster recovery no longer depends on remembering how to reconstruct every Secret manually.

Architectural decisions and alternatives

#

Sealed Secrets

Strong Kubernetes-focused option, but encryption and recovery are tied more closely to the cluster-side controller and key lifecycle.

External Secrets Operator

Useful when an external secret store already exists, but would introduce another runtime dependency and secret backend.

HashiCorp Vault

Powerful central secret platform, but operationally heavier than required for this single-operator GitOps environment.

Selected design

SOPS with age kept encrypted values beside workload definitions, supported local review and worked directly with Flux decryption.

Technical challenges

#

Base64 encoding is not encryption, and simply excluding Secret resources would make a rebuild incomplete. The project needed selective field encryption, consistent file naming, Flux decryption, temporary-file controls and a protected off-Git age identity.

Proposed solution

#

SOPS encrypts only Kubernetes data and stringData values, leaving metadata reviewable. Three path-based rules cover cluster Secrets, applications and infrastructure. The public age recipient can encrypt; the private identity is protected separately and has been used successfully to validate recovery.

Temporary Secret -> SOPS encryption -> Git
                                      |
Protected age identity -> Flux decryption
                                      |
                              Kubernetes Secret

Implementation

#

The migration covered application databases, Grafana, the LIP SOC connector, Mealie, media VPN and SMB credentials, n8n, NetBox, Cloudflare Tunnel, WordPress, CrowdSec, MISP, Proxmox and UniFi monitoring, Wazuh certificates and credentials, Microsoft 365 audit ingestion and threat-intelligence keys.

A validation Secret was retained as historical evidence that the encryption and decryption workflow functioned before production credentials were migrated. Temporary plaintext material is handled outside Git and removed after the encrypted result has been checked.

Validation and measurable outcomes

#

A repository scan found no tracked private-key block. One documentation line contained the search phrase BEGIN PRIVATE KEY; this was a defensive command example, not key material. Flux has also reconciled encrypted Secrets into running applications, including the SOC connector.

Security controls

#

• Private age identity remains outside Git
• Only sensitive fields are encrypted
• Dedicated .enc.yaml naming
• Least-privilege source-system credentials
• No infrastructure credentials delivered to browser code
• Repository scans before commit
• Recovery copy tested, not merely stored

Outcome

#

Git can represent complete workload deployments without containing usable credentials. The result closes a major recovery gap in the GitOps model and makes Secret changes attributable while maintaining confidentiality.

Skills demonstrated

#

• SOPS and age
• Kubernetes Secret design
• Flux decryption
• Key custody and recovery
• Secure migration and validation
• Git history and plaintext hygiene

Lessons learned

#

Secret management is not solved by choosing an encryption tool. Recovery, rotation, least privilege, temporary files and evidence that every eligible file is encrypted are equally important.

Current status and next steps

#

Operational. All 31 evidenced encrypted manifests contained SOPS AES-GCM markers, and the protected recovery identity has been tested successfully.

Explore the implementation

#

Read the Flux delivery case study.

How it was done

#

1. Identified Secret-bearing workloads. Database accounts, application credentials, tunnel tokens, monitoring users, storage access and security integration keys were catalogued before migration.
2. Generated an age identity. The public recipient was used for encryption, while the private identity was kept outside Git and protected as recovery material.
3. Created path-based SOPS rules. A repository-level policy was defined for cluster Secrets, application manifests and infrastructure manifests. Only Kubernetes data and stringData fields were encrypted.
4. Tested the workflow safely. A non-production test Secret was encrypted, decrypted with authorised recovery material and reconciled through Flux before production credentials were migrated.
5. Configured Flux decryption. Flux was given access to the protected cluster-side age identity so encrypted manifests could be decrypted only during reconciliation.
6. Migrated existing credentials. Secret resources were created using temporary protected files, encrypted into .enc.yaml files and added to the relevant application or infrastructure Kustomization.
7. Changed workloads to Secret references. Deployments consumed credentials through Kubernetes Secret references instead of embedding values in manifests or browser-delivered code.
8. Removed temporary plaintext. Temporary source files were deleted after verifying the encrypted result. Repository searches were used to check for private-key markers and exposed values.
9. Validated application consumption. Reconciled Secrets were checked for expected key names without showing values, and consuming applications were rolled out and tested.
10. Verified coverage and recovery. All 31 encrypted manifests contained AES-256-GCM markers, and the protected recovery identity was tested successfully.

Evidence methodology

#

The figures in this case study were derived from reproducible source and runtime checks rather than estimates.

1. Encrypted manifests were counted with a repository search for files ending in .enc.yaml.
2. Each counted file was checked for an ENC[AES256_GCM] marker; 31 files were found and all 31 contained the marker.
3. The SOPS policy was inspected to confirm encryption of Kubernetes data and stringData fields across cluster, application and infrastructure paths.
4. Recovery was tested using the protected age identity, and Flux consumption was validated through successfully reconciled workloads.

Limitations and residual risks

#

• The protected age identity remains a critical recovery dependency and requires continued custody and testing.
• Secrets are encrypted in Git but exist decrypted in Kubernetes and application memory at runtime.
• Encryption does not replace credential rotation, least privilege or source-system revocation.
• Marker coverage proves that files contain encrypted values; it does not by itself prove that every possible credential in the wider environment has been migrated.

Plain-English glossary

#

SOPS

A tool that encrypts selected values inside structured files while leaving non-sensitive metadata readable.

age identity

The private key material used to decrypt content encrypted to an age recipient.

Flux decryption

Flux reads encrypted manifests and decrypts them inside the cluster immediately before applying the Kubernetes resource.

Independent validation

#

This was a self-directed engineering project rather than a client engagement. Credibility is established through deployed configuration, source history, successful builds, operational measurements, restoration tests and repeatable validation rather than a client testimonial.