Skip to content
Documentation Background

Walkthrough: from commit to production

This walkthrough connects the pieces from the rest of the section into one delivery path for a service called catalog-api. A merge to the application repository builds an image, deploys it to staging, verifies it, and prepares a reviewed promotion to production. Every change to a cluster is a commit, and nothing pushes into a cluster.


flowchart LR
    A["Merge to main<br/>catalog-api repo"] --> B["CI: build, test,<br/>push image"]
    B --> C["Commit new tag to<br/>staging overlay"]
    C --> D["Argo CD syncs staging<br/>automatically"]
    D --> E{"Healthy?"}
    E -->|"yes"| F["Pull request:<br/>prod overlay"]
    E -->|"no"| G["Pipeline fails,<br/>alert sent"]
    F --> H["Review and merge"]
    H --> I["Pipeline syncs prod<br/>at the merged commit"]

Two environments are used: staging-eu syncs automatically, and prod-eu syncs only when a pipeline asks, after a human approved the change.

RepositoryPurposeOwner
shop/catalog-apiApplication source and the CI workflowDevelopment team
shop/deploy-configKubernetes manifests, Argo CD ApplicationsDevelopment team, with platform review

Prerequisites: a running Argo CD with both clusters registered, a catalog namespace already created in each, and webhooks set up so changes are noticed within seconds.


Step 1: Lay out the configuration repository

Section titled “Step 1: Lay out the configuration repository”
deploy-config/
├── argocd/
│ ├── project-shop.yaml
│ ├── catalog-api-staging.yaml
│ └── catalog-api-prod.yaml
└── apps/
└── catalog-api/
├── base/
│ ├── kustomization.yaml
│ ├── deployment.yaml
│ ├── service.yaml
│ └── migrate-job.yaml
└── overlays/
├── staging/kustomization.yaml
└── prod/kustomization.yaml

The base and overlays follow the pattern in Application organisation and GitOps. One more file belongs in the base: a database migration that runs before each release, written as a PreSync hook.

apps/catalog-api/base/migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: catalog-api-migrate
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry.example.com/shop/catalog-api:1.4.2
command: ["/app/migrate", "up"]
envFrom:
- secretRef:
name: catalog-db

Both overlays set the image tag with an images: entry. The pipeline will edit that one line. If the migration fails, the sync stops before any workload changes, and the old version keeps running.


The project limits what this team can deploy and where, and defines the role the pipeline will use.

argocd/project-shop.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: shop
namespace: argocd
spec:
description: Shop team
sourceRepos:
- https://git.example.com/shop/deploy-config.git
destinations:
- name: staging-eu
namespace: catalog
- name: prod-eu
namespace: catalog
roles:
- name: ci-deployer
description: Used by the delivery pipeline
policies:
- p, proj:shop:ci-deployer, applications, get, shop/*, allow
- p, proj:shop:ci-deployer, applications, sync, shop/catalog-api-prod, allow
- p, proj:shop:ci-deployer, applications, override, shop/catalog-api-prod, allow

Create a token for the role and store it as a CI secret named ARGOCD_AUTH_TOKEN:

Terminal window
argocd proj role create-token shop ci-deployer --expires-in 90d

The role can read every application in the project, and sync production, including at a pinned revision (the override action). Staging syncs itself, so the pipeline never needs to sync it. See multi-tenancy and projects for the project settings.


argocd/catalog-api-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: catalog-api-staging
namespace: argocd
annotations:
argocd.argoproj.io/manifest-generate-paths: .;../../base
notifications.argoproj.io/subscribe.on-sync-failed.slack: shop-deploys
spec:
project: shop
source:
repoURL: https://git.example.com/shop/deploy-config.git
targetRevision: main
path: apps/catalog-api/overlays/staging
destination:
name: staging-eu
namespace: catalog
syncPolicy:
automated:
prune: true
selfHeal: true
retry:
limit: 3
backoff: { duration: 10s, factor: 2, maxDuration: 3m }

Production has the same shape with two differences: no automated block, so a sync happens only on request, and a subscription to both outcomes.

argocd/catalog-api-prod.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: catalog-api-prod
namespace: argocd
annotations:
argocd.argoproj.io/manifest-generate-paths: .;../../base
notifications.argoproj.io/subscribe.on-sync-succeeded.slack: shop-deploys
notifications.argoproj.io/subscribe.on-sync-failed.slack: shop-deploys
spec:
project: shop
source:
repoURL: https://git.example.com/shop/deploy-config.git
targetRevision: main
path: apps/catalog-api/overlays/prod
destination:
name: prod-eu
namespace: catalog

The manifest-generate-paths annotation limits refreshes to commits that touch this Application’s overlay or the shared base, which matters once the repository holds many services. The notification triggers and Slack service are configured as described in monitoring and notifications.

Bootstrap once by applying a parent Application that watches the argocd/ directory, so later changes to these files go through Git as well (app-of-apps):

Terminal window
kubectl apply -f deploy-config/bootstrap/shop-root.yaml

Step 4: Build, deploy to staging, and verify

Section titled “Step 4: Build, deploy to staging, and verify”

The workflow in the application repository does the CI work, then hands over through Git. It never talks to a cluster, and its only Argo CD call is a read-only wait.

# .github/workflows/release.yaml in shop/catalog-api
name: release
on:
push:
branches: [main]
env:
IMAGE: registry.example.com/shop/catalog-api
ARGOCD_SERVER: argocd.example.com
ARGOCD_VERSION: v3.5.7
jobs:
build:
runs-on: ubuntu-latest
outputs:
tag: ${{ steps.vars.outputs.tag }}
steps:
- uses: actions/checkout@v4
- id: vars
run: echo "tag=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
- uses: docker/login-action@v3
with:
registry: registry.example.com
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_TOKEN }}
- run: |
docker build -t "$IMAGE:${{ steps.vars.outputs.tag }}" .
docker push "$IMAGE:${{ steps.vars.outputs.tag }}"
staging:
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
repository: shop/deploy-config
token: ${{ secrets.CONFIG_REPO_TOKEN }}
- name: Set the staging image tag
run: |
cd apps/catalog-api/overlays/staging
kustomize edit set image "$IMAGE=$IMAGE:${{ needs.build.outputs.tag }}"
git config user.name "release-bot"
git config user.email "[email protected]"
git commit -am "catalog-api: staging ${{ needs.build.outputs.tag }}"
git push origin HEAD:main
- name: Install the Argo CD CLI
run: |
curl -sSL -o argocd "https://github.com/argoproj/argo-cd/releases/download/$ARGOCD_VERSION/argocd-linux-amd64"
sudo install -m 0755 argocd /usr/local/bin/argocd
- name: Wait for staging to be synced and healthy
env:
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
run: |
argocd app get catalog-api-staging --refresh --grpc-web > /dev/null
argocd app wait catalog-api-staging --sync --health --timeout 300 --grpc-web

What happens:

  1. The commit to deploy-config triggers a webhook, and Argo CD refreshes catalog-api-staging. The explicit --refresh covers a missed webhook.
  2. Argo CD runs the PreSync migration, then applies the Deployment and waits for it to become healthy.
  3. argocd app wait returns when the Application is synced and healthy. If it times out, or the Application turns Degraded, the step fails, the pipeline stops, and the Slack subscription alerts the team.

When staging is healthy, the same workflow opens a pull request that changes only the production tag:

promote:
needs: [build, staging]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
repository: shop/deploy-config
token: ${{ secrets.CONFIG_REPO_TOKEN }}
- name: Open the production pull request
env:
GH_TOKEN: ${{ secrets.CONFIG_REPO_TOKEN }}
run: |
TAG="${{ needs.build.outputs.tag }}"
git checkout -b "promote/catalog-api-$TAG"
(cd apps/catalog-api/overlays/prod && kustomize edit set image "$IMAGE=$IMAGE:$TAG")
git config user.name "release-bot"
git config user.email "[email protected]"
git commit -am "catalog-api: promote $TAG to prod"
git push origin HEAD
gh pr create --title "catalog-api: promote $TAG to prod" \
--body "Verified healthy in staging. Source commit: $GITHUB_SHA" --base main

A reviewer sees a one-line diff and the staging result. Branch protection and code owners on deploy-config make that review mandatory. Merging changes the production overlay on main, but production does not move yet, because its sync is manual.


Step 6: Sync production at the merged commit

Section titled “Step 6: Sync production at the merged commit”

A small workflow in deploy-config runs when the production overlay changes on main, and syncs that exact commit.

# .github/workflows/deploy-prod.yaml in shop/deploy-config
name: deploy-prod
on:
push:
branches: [main]
paths: ["apps/catalog-api/overlays/prod/**"]
jobs:
sync:
runs-on: ubuntu-latest
environment: production # an approval gate, if configured on the environment
env:
ARGOCD_SERVER: argocd.example.com
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
steps:
- name: Install the Argo CD CLI
run: |
curl -sSL -o argocd "https://github.com/argoproj/argo-cd/releases/download/v3.5.7/argocd-linux-amd64"
sudo install -m 0755 argocd /usr/local/bin/argocd
- run: argocd app sync catalog-api-prod --revision "$GITHUB_SHA" --grpc-web
- run: argocd app wait catalog-api-prod --sync --health --timeout 600 --grpc-web

--revision pins the sync to the commit that was reviewed, so a commit merged a minute later cannot slip in. Syncing a revision other than the Application’s target is an override action when application.sync.requireOverridePrivilegeForRevisionSync: "true" is set in argocd-cm. Turn that on and grant override only to this pipeline role, as in step 2, so that nobody else can deploy an arbitrary revision. See CI integration and webhooks.


During a release, the Application tree in the UI shows the migration hook, then the Deployment rolling. From the CLI:

Terminal window
argocd app get catalog-api-prod
argocd app history catalog-api-prod

The Slack channel receives the outcome from the notification subscription, and metrics record sync status and duration for dashboards and alerts.


The preferred rollback is a revert, because it keeps Git as the record:

Terminal window
git revert <promotion-commit>
git push origin main

For staging, Argo CD applies the revert automatically. For production, the deploy-prod workflow runs again because the production overlay changed, and syncs the reverted commit.

In an emergency, argocd app rollback catalog-api-prod <id> redeploys an earlier entry from the history list and works because production has no automated sync. It leaves the cluster ahead of Git, so follow it at once with a revert commit to realign them.


SymptomLikely causeFix
argocd app wait returns immediately, before the new versionThe Application had not refreshed yetKeep the --refresh call before the wait, and configure the webhook
Staging never becomes healthy and the wait times outThe migration failed, or the new Pods fail their readiness probeOpen the Application, then read the hook Job and Pod logs
permission denied on syncThe role lacks sync, or override for a pinned revisionAdd the policy to ci-deployer
Production syncs a different commit than expectedThe sync used the branch tip, not --revisionPin the revision in the workflow
The promotion pull request cannot be createdThe token lacks write access to deploy-configUse a token or app installation scoped to that repository
Every service’s commits refresh every ApplicationNo manifest-generate-paths annotationsAdd them, as in step 3

More causes and fixes are on the troubleshooting page.


  • CI and delivery are separate. CI produces an artifact and a commit. Argo CD does the deployment and owns the cluster credentials.
  • Staging is automatic and production is deliberate. The same Git history serves both.
  • Every release is reviewable. An image bump is one line, a promotion is a pull request, and a rollback is a revert.
  • The pipeline holds a narrow credential. A project role can read, and sync one Application, and nothing else.

From here, strengthen it with commit signing, progressive delivery for riskier services, and a secrets operator for the catalog-db Secret.