Secrets management
GitOps puts the desired state of a cluster in Git, and a cluster needs secrets: database passwords, API tokens, TLS keys, registry credentials. A Kubernetes Secret is only base64-encoded, so committing one publishes the value to everyone with read access, forever, because Git history cannot be unwritten. Every GitOps setup therefore needs a deliberate answer to one question: how does a secret reach the cluster when its value must not be in Git?
Three models
Section titled “Three models”| Model | What Git holds | Where the value is resolved | Examples |
|---|---|---|---|
| Reference an external store | A pointer to a secret in a vault or cloud secret manager | In the cluster, by an operator that creates the Secret | External Secrets Operator, Vault Secrets Operator, Secrets Store CSI Driver |
| Encrypt in Git, decrypt in the cluster | The encrypted secret | In the cluster, by a controller that holds the private key | Sealed Secrets |
| Decrypt or inject at render time | An encrypted file or a placeholder | In the GitOps tool, while it renders manifests | SOPS with a Kustomize or Argo CD plugin, argocd-vault-plugin |
The first two keep plaintext inside the destination cluster. The third puts it inside the delivery tool, which is the reason it is the model to avoid when a choice exists.
Reference an external store
Section titled “Reference an external store”The source of truth for the value is a managed secret store. Git holds only a declaration of which secret to fetch and which Kubernetes Secret to create.
With External Secrets Operator, a store object says how to reach the provider, and an ExternalSecret says what to pull:
apiVersion: external-secrets.io/v1kind: ClusterSecretStoremetadata: name: aws-secrets-managerspec: provider: aws: service: SecretsManager region: eu-west-1 # no auth block: the operator's own IAM role (IRSA or EKS Pod Identity) is usedapiVersion: external-secrets.io/v1kind: ExternalSecretmetadata: name: catalog-db namespace: catalogspec: refreshInterval: 1h secretStoreRef: kind: ClusterSecretStore name: aws-secrets-manager target: name: catalog-db # the Kubernetes Secret to create creationPolicy: Owner data: - secretKey: password remoteRef: key: prod/catalog/db # the secret in the external store property: passwordThe operator creates and refreshes the catalog-db Secret, and the Deployment mounts it by name. Nothing sensitive is in Git, and rotating the value in the store reaches the cluster at the next refresh.
Strengths
- One system of record, with its own access control, audit log, and rotation.
- Secret changes are decoupled from deployments, so an unrelated release never carries a surprise credential change.
- Argo CD never reads the value.
Costs
- A dependency on the external store being reachable.
- Another operator to run, upgrade, and give IAM or vault access to.
Encrypt in Git with Sealed Secrets
Section titled “Encrypt in Git with Sealed Secrets”Sealed Secrets lets a team commit encrypted secrets. A controller in the cluster holds the private key, and only that controller can decrypt. The developer encrypts with the public key, using the kubeseal CLI:
kubectl create secret generic catalog-db \ --namespace catalog \ --from-literal=password='<value>' \ --dry-run=client -o yaml \ | kubeseal --format yaml \ --controller-name sealed-secrets --controller-namespace kube-system \ > apps/catalog-api/overlays/prod/catalog-db-sealed.yamlThe result is a SealedSecret that is safe to commit. The controller decrypts it into a regular Secret in the same namespace. By default a sealed value is bound to its name and namespace, so copying it to another namespace does not work.
Strengths: no external system, works offline, and the encrypted file is reviewed and versioned with everything else.
Costs
- Losing the controller’s private key means losing the ability to decrypt, so back it up, and plan for key rotation.
- Rotating a secret means re-sealing and committing it.
- A secret cannot be read back from Git, which makes audits harder than with a store.
- It does not fit the rendered manifests pattern, because the sealed form must stay sealed.
Decrypt at render time with SOPS
Section titled “Decrypt at render time with SOPS”SOPS encrypts the values inside a YAML file and leaves the keys readable, so diffs stay meaningful. Rules live in a .sops.yaml file, and the key can come from age, a cloud KMS, or a vault:
creation_rules:- path_regex: .*/secrets/.*\.yaml$ encrypted_regex: ^(data|stringData)$ age: age1examplepublickey... # public key, safe to commitsops --encrypt --in-place apps/catalog-api/overlays/prod/secrets/catalog-db.yamlSomething has to decrypt it before apply. With Argo CD that is a Kustomize plugin such as KSOPS, or a config management plugin, running in the repo server with access to the key. This is the render-time model, so the decrypted manifest is cached in Redis, and the repo server holds a long-lived key.
SOPS suits teams that want no extra operator in the cluster and accept that trade. Where an external store or Sealed Secrets is possible, prefer those. If SOPS is chosen, restrict network access to Redis and the repo server with network policies, and consider running Argo CD on a cluster of its own.
How secrets behave inside Argo CD
Section titled “How secrets behave inside Argo CD”The generated Secret is not in Git, so it is not pruned. A Secret created by an operator carries no Argo CD tracking annotation, and Argo CD ignores it. The ExternalSecret or SealedSecret is what Git defines and what Argo CD manages.
Ordering matters. The store and the operator’s CRDs must exist before the first ExternalSecret is applied. Put the operator in an earlier sync wave, or in a separate Application that the app of apps orders first. Argo CD ships a health check for ExternalSecret, so a failing fetch shows as a degraded resource instead of a silent success.
Operator-managed fields can cause drift. If a controller mutates a field that Git also sets, add an ignoreDifferences rule. See diffing and ignore rules.
Argo CD’s own secrets need the same care. Repository credentials, cluster Secrets, and the SSO client secret are Kubernetes Secrets in the argocd namespace. They can be produced the same way, for example an ExternalSecret whose template adds the label argocd.argoproj.io/secret-type: repository. See TLS and repository credentials and single sign-on.
Operating rules
Section titled “Operating rules”- Never place secret values in Helm
values.yamlor Kustomize patches in Git. Charts that acceptexistingSecretare the safe form. Pass a reference, not a value. - Scan for leaks before they merge. Run a secret scanner, such as gitleaks, as a pre-commit hook and a required pull request check.
- Treat a committed secret as compromised. Rewriting history does not undo exposure. Rotate the credential first, then clean up.
- Limit who can read the cluster’s Secrets. The encryption protects Git, but
kubectl get secretstill reveals the value to anyone with access. Use Kubernetes RBAC and enable encryption at rest in etcd. - Scope access narrowly. Give each store or controller only the paths it needs, and one identity per environment.
- Plan rotation. Prefer short-lived credentials and automated refresh. Applications must reload a changed Secret, either by restarting or by watching the file.
- Have a break-glass path. Know how to recover if the secret store, or the sealing key, is unavailable.
Choosing an approach
Section titled “Choosing an approach”| Situation | Choice |
|---|---|
| A cloud or vault secret manager already exists | External Secrets Operator or the vault’s own operator |
| No external store, and the team wants everything in Git | Sealed Secrets |
| Secrets must be readable in code review diffs by key, without values | SOPS, with the render-time trade-off accepted |
Pods should receive secrets as files without a Kubernetes Secret | Secrets Store CSI Driver |
| The manifests are rendered into Git before deployment | A store-based operator, since sealed and render-time secrets conflict with the pattern |
When in doubt, use a store-based operator. It keeps Argo CD out of the secret path, gives one place to rotate and audit, and works with every other practice on this page.