↓ Skip to main content
  1. Runbook/
  2. GitOps/

SOPS and age Secret Workflow

SOPS Age FluxCD Kubernetes Secrets

Time guide

  • Validate an existing encrypted Secret: approximately 5 to 10 minutes
  • Create and reconcile a new encrypted Secret: approximately 15 to 30 minutes
  • Investigate a decryption fault: approximately 20 to 45 minutes

Purpose
#

This entry describes a safe workflow for storing encrypted Kubernetes Secret manifests in Git using SOPS and age, with Flux performing decryption during reconciliation.

Security principles
#

  • Never commit an age private key.
  • Never commit a plaintext Kubernetes Secret manifest.
  • Commit only encrypted Secret manifests.
  • Keep protected recovery copies of the age private key outside Git.
  • Validate plaintext before encryption and the resulting Secret after reconciliation.
  • Remove temporary plaintext files immediately.

Requirements
#

sops --version
age --version
flux version
kubectl version --client

Confirm the active cluster context:

kubectl config current-context

Confirm SOPS configuration
#

cat .sops.yaml

The configuration should target the intended encrypted-secret paths and use the expected public age recipient.

The age recipient is public. The age identity or private key must remain protected outside Git.

Validate an existing encrypted file
#

sops filestatus path/to/<secret-name>.enc.yaml

Test decryption without printing plaintext:

sops -d path/to/<secret-name>.enc.yaml > /dev/null

Confirm the encrypted file does not contain an expected plaintext marker:

! grep -q '<known-plaintext-marker>' path/to/<secret-name>.enc.yaml

Create a new encrypted Secret
#

Create the plaintext file outside the repository where practical:

umask 077
cat > /tmp/<secret-name>.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: <secret-name>
  namespace: <namespace>
type: Opaque
stringData:
  <key-name>: <secret-value>
EOF

Encrypt directly into the repository:

sops --encrypt \
  /tmp/<secret-name>.yaml \
  > path/to/<secret-name>.enc.yaml

Remove the plaintext file:

shred -u /tmp/<secret-name>.yaml

If shred is not suitable for the underlying storage, securely remove the file according to the platform and storage characteristics.

Validate before commit
#

sops filestatus path/to/<secret-name>.enc.yaml
sops -d path/to/<secret-name>.enc.yaml > /dev/null
git status --short
git diff -- path/to/<secret-name>.enc.yaml

Confirm no plaintext copy exists:

find . -type f \
  ! -name '*.enc.yaml' \
  ! -path './.git/*' \
  -exec grep -Il '<known-plaintext-marker>' {} +

Expected result: no output.

Reference the encrypted file
#

Add the encrypted manifest to the relevant kustomization.yaml:

resources:
  - <secret-name>.enc.yaml

Confirm the Flux Kustomization is configured for SOPS decryption. Inspect it without printing Secret values:

kubectl get kustomization <kustomization-name> \
  -n <flux-namespace> \
  -o yaml

Render and dry-run considerations
#

A normal Kustomize render preserves SOPS-encrypted content because Kustomize does not decrypt it automatically:

kubectl kustomize path/to/component

Use SOPS-aware validation for plaintext structure when necessary, but do not write decrypted output into the repository.

Commit and reconcile
#

git add .sops.yaml path/to/kustomization.yaml path/to/<secret-name>.enc.yaml
git diff --cached --check
git diff --cached
git commit -m "Add encrypted <component> secret"
git push

Reconcile the affected resource:

flux reconcile kustomization <kustomization-name> \
  -n <flux-namespace> \
  --with-source

Post-reconciliation validation
#

flux get kustomization <kustomization-name> \
  -n <flux-namespace>

Confirm the Secret exists without printing data:

kubectl get secret <secret-name> \
  -n <namespace>

Confirm expected keys exist without decoding values:

kubectl get secret <secret-name> \
  -n <namespace> \
  -o go-template='{{range $key, $value := .data}}{{$key}}{{"\n"}}{{end}}'

Validate the consuming workload:

kubectl get pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp

Troubleshooting
#

Decryption failed
#

kubectl describe kustomization <kustomization-name> \
  -n <flux-namespace>
kubectl logs \
  -n <flux-namespace> \
  deployment/kustomize-controller \
  --since=30m

Check:

  • The encrypted file matches a SOPS creation rule.
  • The expected age recipient is present in file metadata.
  • The Flux decryption identity exists.
  • The Kustomization has SOPS decryption enabled.
  • The file has not been encrypted twice.
  • The Kubernetes Secret manifest is structurally valid.

Secret exists but the workload fails
#

kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp

Confirm:

  • The Secret name matches the workload reference.
  • Expected keys exist.
  • Environment-variable and volume references are correct.
  • The plaintext value was correct before encryption.

Recovery
#

Recovery requires the protected age private key and sufficient cluster access to restore the Flux decryption identity.

Do not publish:

  • Private-key paths
  • Private-key contents
  • Recovery copy locations
  • Exact Secret names used for privileged systems
  • Decoded Secret values

Safety notes
#

Commands that decode or decrypt values can expose credentials to terminal history, process output, logs, screenshots or copied text. Prefer validation that confirms successful decryption without displaying plaintext.

Related entries#

  • GitOps Repository Structure
  • Flux Reconciliation Commands
  • Secret Leakage Checks
  • Kustomization Troubleshooting
  • Disaster-Recovery Critical Assets