Skip to content
Documentation Background

Secrets management

GitOps puts the desired state of a cluster in Git, and a cluster needs secrets: database passwords, API tokens, TLS keys, registry credentials. A Kubernetes Secret is only base64-encoded, so committing one publishes the value to everyone with read access, forever, because Git history cannot be unwritten. Every GitOps setup therefore needs a deliberate answer to one question: how does a secret reach the cluster when its value must not be in Git?


ModelWhat Git holdsWhere the value is resolvedExamples
Reference an external storeA pointer to a secret in a vault or cloud secret managerIn the cluster, by an operator that creates the SecretExternal Secrets Operator, Vault Secrets Operator, Secrets Store CSI Driver
Encrypt in Git, decrypt in the clusterThe encrypted secretIn the cluster, by a controller that holds the private keySealed Secrets
Decrypt or inject at render timeAn encrypted file or a placeholderIn the GitOps tool, while it renders manifestsSOPS with a Kustomize or Argo CD plugin, argocd-vault-plugin

The first two keep plaintext inside the destination cluster. The third puts it inside the delivery tool, which is the reason it is the model to avoid when a choice exists.


The source of truth for the value is a managed secret store. Git holds only a declaration of which secret to fetch and which Kubernetes Secret to create.

With External Secrets Operator, a store object says how to reach the provider, and an ExternalSecret says what to pull:

platform/secret-store.yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: aws-secrets-manager
spec:
provider:
aws:
service: SecretsManager
region: eu-west-1
# no auth block: the operator's own IAM role (IRSA or EKS Pod Identity) is used
apps/catalog-api/base/external-secret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: catalog-db
namespace: catalog
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: aws-secrets-manager
target:
name: catalog-db # the Kubernetes Secret to create
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: prod/catalog/db # the secret in the external store
property: password

The operator creates and refreshes the catalog-db Secret, and the Deployment mounts it by name. Nothing sensitive is in Git, and rotating the value in the store reaches the cluster at the next refresh.

Strengths

  • One system of record, with its own access control, audit log, and rotation.
  • Secret changes are decoupled from deployments, so an unrelated release never carries a surprise credential change.
  • Argo CD never reads the value.

Costs

  • A dependency on the external store being reachable.
  • Another operator to run, upgrade, and give IAM or vault access to.

Sealed Secrets lets a team commit encrypted secrets. A controller in the cluster holds the private key, and only that controller can decrypt. The developer encrypts with the public key, using the kubeseal CLI:

Terminal window
kubectl create secret generic catalog-db \
--namespace catalog \
--from-literal=password='<value>' \
--dry-run=client -o yaml \
| kubeseal --format yaml \
--controller-name sealed-secrets --controller-namespace kube-system \
> apps/catalog-api/overlays/prod/catalog-db-sealed.yaml

The result is a SealedSecret that is safe to commit. The controller decrypts it into a regular Secret in the same namespace. By default a sealed value is bound to its name and namespace, so copying it to another namespace does not work.

Strengths: no external system, works offline, and the encrypted file is reviewed and versioned with everything else.

Costs

  • Losing the controller’s private key means losing the ability to decrypt, so back it up, and plan for key rotation.
  • Rotating a secret means re-sealing and committing it.
  • A secret cannot be read back from Git, which makes audits harder than with a store.
  • It does not fit the rendered manifests pattern, because the sealed form must stay sealed.

SOPS encrypts the values inside a YAML file and leaves the keys readable, so diffs stay meaningful. Rules live in a .sops.yaml file, and the key can come from age, a cloud KMS, or a vault:

.sops.yaml
creation_rules:
- path_regex: .*/secrets/.*\.yaml$
encrypted_regex: ^(data|stringData)$
age: age1examplepublickey... # public key, safe to commit
Terminal window
sops --encrypt --in-place apps/catalog-api/overlays/prod/secrets/catalog-db.yaml

Something has to decrypt it before apply. With Argo CD that is a Kustomize plugin such as KSOPS, or a config management plugin, running in the repo server with access to the key. This is the render-time model, so the decrypted manifest is cached in Redis, and the repo server holds a long-lived key.

SOPS suits teams that want no extra operator in the cluster and accept that trade. Where an external store or Sealed Secrets is possible, prefer those. If SOPS is chosen, restrict network access to Redis and the repo server with network policies, and consider running Argo CD on a cluster of its own.


The generated Secret is not in Git, so it is not pruned. A Secret created by an operator carries no Argo CD tracking annotation, and Argo CD ignores it. The ExternalSecret or SealedSecret is what Git defines and what Argo CD manages.

Ordering matters. The store and the operator’s CRDs must exist before the first ExternalSecret is applied. Put the operator in an earlier sync wave, or in a separate Application that the app of apps orders first. Argo CD ships a health check for ExternalSecret, so a failing fetch shows as a degraded resource instead of a silent success.

Operator-managed fields can cause drift. If a controller mutates a field that Git also sets, add an ignoreDifferences rule. See diffing and ignore rules.

Argo CD’s own secrets need the same care. Repository credentials, cluster Secrets, and the SSO client secret are Kubernetes Secrets in the argocd namespace. They can be produced the same way, for example an ExternalSecret whose template adds the label argocd.argoproj.io/secret-type: repository. See TLS and repository credentials and single sign-on.


  • Never place secret values in Helm values.yaml or Kustomize patches in Git. Charts that accept existingSecret are the safe form. Pass a reference, not a value.
  • Scan for leaks before they merge. Run a secret scanner, such as gitleaks, as a pre-commit hook and a required pull request check.
  • Treat a committed secret as compromised. Rewriting history does not undo exposure. Rotate the credential first, then clean up.
  • Limit who can read the cluster’s Secrets. The encryption protects Git, but kubectl get secret still reveals the value to anyone with access. Use Kubernetes RBAC and enable encryption at rest in etcd.
  • Scope access narrowly. Give each store or controller only the paths it needs, and one identity per environment.
  • Plan rotation. Prefer short-lived credentials and automated refresh. Applications must reload a changed Secret, either by restarting or by watching the file.
  • Have a break-glass path. Know how to recover if the secret store, or the sealing key, is unavailable.

SituationChoice
A cloud or vault secret manager already existsExternal Secrets Operator or the vault’s own operator
No external store, and the team wants everything in GitSealed Secrets
Secrets must be readable in code review diffs by key, without valuesSOPS, with the render-time trade-off accepted
Pods should receive secrets as files without a Kubernetes SecretSecrets Store CSI Driver
The manifests are rendered into Git before deploymentA store-based operator, since sealed and render-time secrets conflict with the pattern

When in doubt, use a store-based operator. It keeps Argo CD out of the secret path, gives one place to rotate and audit, and works with every other practice on this page.