Skip to content
Documentation Background

GitOps best practices

The GitOps principles say what the model is. They do not say how to arrange a repository, how to avoid copying YAML into every environment, or how a change travels from a build to production. Those decisions decide whether a GitOps setup stays pleasant at ten applications and survives at a thousand. This page collects the practices that hold up in production.


Keep application source and deployment manifests in different repositories.

ReasonEffect of mixing them
Different lifecyclesCode changes with features. Manifests change with scaling, limits, and platform integration. One history for both is noisy
Needless rebuildsChanging a replica count in the same repository triggers a full build and test run for code that did not change
Different approvalsEnvironment changes follow platform review gates, which should not block or be blocked by application pull requests
Different accessDevelopers rarely edit platform configuration, and platform engineers rarely edit application code

The cost is one more repository and a handoff step. The CI side of that handoff is covered in CI integration and webhooks.


Repository layout follows the organization. Teams, ownership, and access rules shape it more than any template does, so there is no layout that fits everyone. Two shapes cover most cases.

LayoutWhat it looks likeStrengthsLimits
MonorepoAll manifests for all applications and clusters in one repositoryOne place for governance and review, simple to startSlows Argo CD as it grows, and every team shares one permission model
PolyrepoA repository per responsibilityScales across teams, access matches ownershipMore repositories, and releases that span several of them need coordination

A common polyrepo start uses two repositories:

  • A platform repository owned by the platform team. It holds Argo CD’s own configuration (projects, Applications, settings) and organization-wide components such as policy engines, a service mesh, and a secrets operator.
  • Application deployment repositories owned by development teams. They hold workloads: Deployments, Services, ConfigMaps, and Ingress or routes.

Whichever shape is chosen, model environments as directories, not long-lived branches, and keep the layout predictable so that tooling can rely on it. A large monorepo needs the tuning described in high availability and scaling.


Repeating nearly identical YAML for every environment guarantees drift. Store the shared part once and capture only the differences. Argo CD supports two tools natively, and plugins cover the rest (rendering tools).

ToolBest whenMechanism
KustomizeThe differences are known when the manifest is writtenA base, plus patching overlays per environment
HelmA value depends on where the application lands, and is known only at deploy time, such as a hostname per clusterA template, plus a values file per environment

Many teams use both. Kustomize overlays keep applications they write themselves small and readable, and Helm packages third-party software and templates the fields that vary per cluster. In Argo CD one Application can combine a chart with values from a separate Git repository, using multiple sources.


Teams used to Git Flow often carry it into configuration repositories: long-lived dev, staging, and prod branches with merges between them. It works poorly for manifests.

  • Each environment has settings that must not travel: its own ConfigMaps, hostnames, and secret references. Merging a branch drags them along, and the answer becomes cherry-picking, which is slow and error-prone.
  • What moves between environments is a version, not a code change.

Instead, keep one main branch as the single source of truth. Environment differences live in overlays or values files, and promotion is a small commit or pull request that edits one of them. Use trunk-based development for the configuration repository even when the application repository uses another workflow.


Helm and Kustomize keep source DRY, but they hide what is applied. Argo CD renders the templates at sync time, so a one-line change in Git can produce a sweeping change in the cluster. Bumping a chart version from 58.6.1 to 61.9.0 is a one-line diff and may alter hundreds of objects. The reviewer approves a line, and the cluster receives something else.

The rendered manifests pattern (also called hydrated manifests) removes the ambiguity. The desired state in Git is the final YAML, treated like a container image: immutable, reviewed in full, and applied unchanged.

flowchart LR
    A["Dry source<br/>Helm values, Kustomize overlays<br/>on main"] --> B["Render<br/>CI or Argo CD"]
    B --> C["Hydrated YAML<br/>environment branch"]
    C --> D["Argo CD syncs<br/>plain manifests"]
    D --> E["Cluster"]

Benefits

  • Pull requests show the exact resources added, changed, or removed.
  • Argo CD does no rendering at sync time, which lowers load.
  • What was reviewed is what runs.
  • Each environment branch can carry its own protection rules and approvals.

Costs

  • Rendering and committing add logic to the pipeline.
  • Tools that depend on sealing or render-time substitution fit badly. Use a store-based secrets operator instead.

The environment branches in this pattern are generated release artifacts. Nobody merges them by hand, so they are not Git Flow. Source stays on main.

Argo CD 3.5 can run the pattern itself. The source hydrator is a beta feature that renders a dry source and pushes the result to a Git branch before syncing it. It is off by default. Enable it with hydrator.enabled: "true" in argocd-cmd-params-cm, run the commit server component, and restart the controller and API server. The upstream *-install-with-hydrator.yaml manifests include the extra component.

It needs two credentials for the repository: a normal one to read, and one labeled argocd.argoproj.io/secret-type: repository-write to push, so that reading and writing stay isolated.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: catalog-api-prod
namespace: argocd
spec:
project: shop
destination:
name: prod-eu
namespace: catalog
sourceHydrator:
drySource:
repoURL: https://git.example.com/shop/deploy-config.git
targetRevision: main
path: apps/catalog-api/overlays/prod # Kustomize, Helm, or plain manifests
syncSource:
targetBranch: environments/prod # hydrated output is pushed here
path: catalog-api # must not be the repository root
hydrateTo:
targetBranch: environments/prod-next # optional staging branch

With hydrateTo set, Argo CD pushes to the staging branch and does not sync it. A pull request from environments/prod-next to environments/prod then becomes the human approval gate, showing the full rendered diff. Argo CD does not open that pull request, so use CI or a promotion tool to do it.

The hydrator also records commit trailers that link each hydrated commit back to the code commit that caused it, so the trail from a running resource to a source change stays intact. Because it is beta, review the upstream documentation before adopting it for critical environments.


Promotion is the movement of an artifact version through environments, such as a new image from staging to production. The simplest form is a pipeline that edits the image tag in an overlay and opens a pull request. As the number of services grows, scripts get fragile, and dedicated tools help.

ToolWhat it doesFits when
Argo CD Image UpdaterWatches registries for new image tags, then updates Applications through the API or by committing the tag to Git. It is configured with an ImageUpdater custom resource and works with Kustomize, Helm, and plugin sourcesImage bumps are the only automation needed
KargoOrchestrates promotion across stages, tracking images, Helm charts, and Git together under declarative rules, and commits the resulting changesMany services, multiple stages, and a real release process
TelefonistkaOpens pull requests that copy a component from one environment directory to the next, according to a configured promotion order. Created at Wayfair and now maintained under commercetoolsEnvironments are directories, and promotion should be a reviewed pull request

They overlap and can be combined. Image Updater handles one narrow job, Kargo is an orchestration layer above the GitOps controller, and Telefonistka brings pull-request governance to directory layouts. Check the maturity and support status of any of these before depending on it, since all three move quickly.

Whatever the tool, keep the rule that a promotion is a commit: reviewed, attributable, and revertible.


Continuous delivery gets changes to production quickly. Progressive delivery limits the damage of a bad one by exposing it gradually and judging it with data.

StrategyHow it works
Blue-greenThe new version runs beside the old. It is tested, then traffic switches in one step
CanaryA small share of traffic goes to the new version, grows in steps while metrics stay healthy, and rolls back automatically if they do not

Argo CD syncs state and does not shift traffic by itself. Argo Rollouts is the companion controller that does. A Rollout replaces a Deployment, declares the strategy, and works with ingress controllers and service meshes for traffic splitting. Argo CD manages the Rollout like any other resource, and shows its health.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: catalog-api
namespace: catalog
spec:
replicas: 4
selector:
matchLabels:
app.kubernetes.io/name: catalog-api
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 10m }
template:
metadata:
labels:
app.kubernetes.io/name: catalog-api
spec:
containers:
- name: api
image: registry.example.com/shop/catalog-api:1.4.2

Without a traffic provider, replica counts approximate the weights. With one, such as a gateway or mesh, the split is exact. Add automated analysis against metrics to promote or abort without a human in the loop.


  • Source code and configuration live in separate repositories.
  • One main branch per configuration repository, with environments as directories.
  • Shared YAML is written once, with Kustomize, Helm, or both.
  • Environment promotion is a reviewed commit.
  • Rendered output is reviewed when the template layer is deep, ideally through hydration.
  • Secrets are references or sealed, never plain text. See secrets management.
  • Branch protection and code owners guard the configuration repository, because merge rights are deploy rights.
  • Risky releases use canary or blue-green, with metrics deciding.