GitOps best practices
The GitOps principles say what the model is. They do not say how to arrange a repository, how to avoid copying YAML into every environment, or how a change travels from a build to production. Those decisions decide whether a GitOps setup stays pleasant at ten applications and survives at a thousand. This page collects the practices that hold up in production.
Separate source code from configuration
Section titled “Separate source code from configuration”Keep application source and deployment manifests in different repositories.
| Reason | Effect of mixing them |
|---|---|
| Different lifecycles | Code changes with features. Manifests change with scaling, limits, and platform integration. One history for both is noisy |
| Needless rebuilds | Changing a replica count in the same repository triggers a full build and test run for code that did not change |
| Different approvals | Environment changes follow platform review gates, which should not block or be blocked by application pull requests |
| Different access | Developers rarely edit platform configuration, and platform engineers rarely edit application code |
The cost is one more repository and a handoff step. The CI side of that handoff is covered in CI integration and webhooks.
Choose a repository layout
Section titled “Choose a repository layout”Repository layout follows the organization. Teams, ownership, and access rules shape it more than any template does, so there is no layout that fits everyone. Two shapes cover most cases.
| Layout | What it looks like | Strengths | Limits |
|---|---|---|---|
| Monorepo | All manifests for all applications and clusters in one repository | One place for governance and review, simple to start | Slows Argo CD as it grows, and every team shares one permission model |
| Polyrepo | A repository per responsibility | Scales across teams, access matches ownership | More repositories, and releases that span several of them need coordination |
A common polyrepo start uses two repositories:
- A platform repository owned by the platform team. It holds Argo CD’s own configuration (projects, Applications, settings) and organization-wide components such as policy engines, a service mesh, and a secrets operator.
- Application deployment repositories owned by development teams. They hold workloads: Deployments, Services, ConfigMaps, and Ingress or routes.
Whichever shape is chosen, model environments as directories, not long-lived branches, and keep the layout predictable so that tooling can rely on it. A large monorepo needs the tuning described in high availability and scaling.
Keep configuration DRY
Section titled “Keep configuration DRY”Repeating nearly identical YAML for every environment guarantees drift. Store the shared part once and capture only the differences. Argo CD supports two tools natively, and plugins cover the rest (rendering tools).
| Tool | Best when | Mechanism |
|---|---|---|
| Kustomize | The differences are known when the manifest is written | A base, plus patching overlays per environment |
| Helm | A value depends on where the application lands, and is known only at deploy time, such as a hostname per cluster | A template, plus a values file per environment |
Many teams use both. Kustomize overlays keep applications they write themselves small and readable, and Helm packages third-party software and templates the fields that vary per cluster. In Argo CD one Application can combine a chart with values from a separate Git repository, using multiple sources.
Use trunk-based development
Section titled “Use trunk-based development”Teams used to Git Flow often carry it into configuration repositories: long-lived dev, staging, and prod branches with merges between them. It works poorly for manifests.
- Each environment has settings that must not travel: its own ConfigMaps, hostnames, and secret references. Merging a branch drags them along, and the answer becomes cherry-picking, which is slow and error-prone.
- What moves between environments is a version, not a code change.
Instead, keep one main branch as the single source of truth. Environment differences live in overlays or values files, and promotion is a small commit or pull request that edits one of them. Use trunk-based development for the configuration repository even when the application repository uses another workflow.
Rendered manifests
Section titled “Rendered manifests”Helm and Kustomize keep source DRY, but they hide what is applied. Argo CD renders the templates at sync time, so a one-line change in Git can produce a sweeping change in the cluster. Bumping a chart version from 58.6.1 to 61.9.0 is a one-line diff and may alter hundreds of objects. The reviewer approves a line, and the cluster receives something else.
The rendered manifests pattern (also called hydrated manifests) removes the ambiguity. The desired state in Git is the final YAML, treated like a container image: immutable, reviewed in full, and applied unchanged.
flowchart LR
A["Dry source<br/>Helm values, Kustomize overlays<br/>on main"] --> B["Render<br/>CI or Argo CD"]
B --> C["Hydrated YAML<br/>environment branch"]
C --> D["Argo CD syncs<br/>plain manifests"]
D --> E["Cluster"]
Benefits
- Pull requests show the exact resources added, changed, or removed.
- Argo CD does no rendering at sync time, which lowers load.
- What was reviewed is what runs.
- Each environment branch can carry its own protection rules and approvals.
Costs
- Rendering and committing add logic to the pipeline.
- Tools that depend on sealing or render-time substitution fit badly. Use a store-based secrets operator instead.
The environment branches in this pattern are generated release artifacts. Nobody merges them by hand, so they are not Git Flow. Source stays on main.
Argo CD source hydrator
Section titled “Argo CD source hydrator”Argo CD 3.5 can run the pattern itself. The source hydrator is a beta feature that renders a dry source and pushes the result to a Git branch before syncing it. It is off by default. Enable it with hydrator.enabled: "true" in argocd-cmd-params-cm, run the commit server component, and restart the controller and API server. The upstream *-install-with-hydrator.yaml manifests include the extra component.
It needs two credentials for the repository: a normal one to read, and one labeled argocd.argoproj.io/secret-type: repository-write to push, so that reading and writing stay isolated.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: catalog-api-prod namespace: argocdspec: project: shop destination: name: prod-eu namespace: catalog sourceHydrator: drySource: repoURL: https://git.example.com/shop/deploy-config.git targetRevision: main path: apps/catalog-api/overlays/prod # Kustomize, Helm, or plain manifests syncSource: targetBranch: environments/prod # hydrated output is pushed here path: catalog-api # must not be the repository root hydrateTo: targetBranch: environments/prod-next # optional staging branchWith hydrateTo set, Argo CD pushes to the staging branch and does not sync it. A pull request from environments/prod-next to environments/prod then becomes the human approval gate, showing the full rendered diff. Argo CD does not open that pull request, so use CI or a promotion tool to do it.
The hydrator also records commit trailers that link each hydrated commit back to the code commit that caused it, so the trail from a running resource to a source change stays intact. Because it is beta, review the upstream documentation before adopting it for critical environments.
Promote changes between environments
Section titled “Promote changes between environments”Promotion is the movement of an artifact version through environments, such as a new image from staging to production. The simplest form is a pipeline that edits the image tag in an overlay and opens a pull request. As the number of services grows, scripts get fragile, and dedicated tools help.
| Tool | What it does | Fits when |
|---|---|---|
| Argo CD Image Updater | Watches registries for new image tags, then updates Applications through the API or by committing the tag to Git. It is configured with an ImageUpdater custom resource and works with Kustomize, Helm, and plugin sources | Image bumps are the only automation needed |
| Kargo | Orchestrates promotion across stages, tracking images, Helm charts, and Git together under declarative rules, and commits the resulting changes | Many services, multiple stages, and a real release process |
| Telefonistka | Opens pull requests that copy a component from one environment directory to the next, according to a configured promotion order. Created at Wayfair and now maintained under commercetools | Environments are directories, and promotion should be a reviewed pull request |
They overlap and can be combined. Image Updater handles one narrow job, Kargo is an orchestration layer above the GitOps controller, and Telefonistka brings pull-request governance to directory layouts. Check the maturity and support status of any of these before depending on it, since all three move quickly.
Whatever the tool, keep the rule that a promotion is a commit: reviewed, attributable, and revertible.
Progressive delivery
Section titled “Progressive delivery”Continuous delivery gets changes to production quickly. Progressive delivery limits the damage of a bad one by exposing it gradually and judging it with data.
| Strategy | How it works |
|---|---|
| Blue-green | The new version runs beside the old. It is tested, then traffic switches in one step |
| Canary | A small share of traffic goes to the new version, grows in steps while metrics stay healthy, and rolls back automatically if they do not |
Argo CD syncs state and does not shift traffic by itself. Argo Rollouts is the companion controller that does. A Rollout replaces a Deployment, declares the strategy, and works with ingress controllers and service meshes for traffic splitting. Argo CD manages the Rollout like any other resource, and shows its health.
apiVersion: argoproj.io/v1alpha1kind: Rolloutmetadata: name: catalog-api namespace: catalogspec: replicas: 4 selector: matchLabels: app.kubernetes.io/name: catalog-api strategy: canary: steps: - setWeight: 10 - pause: { duration: 5m } - setWeight: 50 - pause: { duration: 10m } template: metadata: labels: app.kubernetes.io/name: catalog-api spec: containers: - name: api image: registry.example.com/shop/catalog-api:1.4.2Without a traffic provider, replica counts approximate the weights. With one, such as a gateway or mesh, the split is exact. Add automated analysis against metrics to promote or abort without a human in the loop.
A short checklist
Section titled “A short checklist”- Source code and configuration live in separate repositories.
- One
mainbranch per configuration repository, with environments as directories. - Shared YAML is written once, with Kustomize, Helm, or both.
- Environment promotion is a reviewed commit.
- Rendered output is reviewed when the template layer is deep, ideally through hydration.
- Secrets are references or sealed, never plain text. See secrets management.
- Branch protection and code owners guard the configuration repository, because merge rights are deploy rights.
- Risky releases use canary or blue-green, with metrics deciding.