Walkthrough: from commit to production
This walkthrough connects the pieces from the rest of the section into one delivery path for a service called catalog-api. A merge to the application repository builds an image, deploys it to staging, verifies it, and prepares a reviewed promotion to production. Every change to a cluster is a commit, and nothing pushes into a cluster.
The target flow
Section titled “The target flow”flowchart LR
A["Merge to main<br/>catalog-api repo"] --> B["CI: build, test,<br/>push image"]
B --> C["Commit new tag to<br/>staging overlay"]
C --> D["Argo CD syncs staging<br/>automatically"]
D --> E{"Healthy?"}
E -->|"yes"| F["Pull request:<br/>prod overlay"]
E -->|"no"| G["Pipeline fails,<br/>alert sent"]
F --> H["Review and merge"]
H --> I["Pipeline syncs prod<br/>at the merged commit"]
Two environments are used: staging-eu syncs automatically, and prod-eu syncs only when a pipeline asks, after a human approved the change.
| Repository | Purpose | Owner |
|---|---|---|
shop/catalog-api | Application source and the CI workflow | Development team |
shop/deploy-config | Kubernetes manifests, Argo CD Applications | Development team, with platform review |
Prerequisites: a running Argo CD with both clusters registered, a catalog namespace already created in each, and webhooks set up so changes are noticed within seconds.
Step 1: Lay out the configuration repository
Section titled “Step 1: Lay out the configuration repository”deploy-config/├── argocd/│ ├── project-shop.yaml│ ├── catalog-api-staging.yaml│ └── catalog-api-prod.yaml└── apps/ └── catalog-api/ ├── base/ │ ├── kustomization.yaml │ ├── deployment.yaml │ ├── service.yaml │ └── migrate-job.yaml └── overlays/ ├── staging/kustomization.yaml └── prod/kustomization.yamlThe base and overlays follow the pattern in Application organisation and GitOps. One more file belongs in the base: a database migration that runs before each release, written as a PreSync hook.
apiVersion: batch/v1kind: Jobmetadata: name: catalog-api-migrate annotations: argocd.argoproj.io/hook: PreSync argocd.argoproj.io/hook-delete-policy: BeforeHookCreationspec: backoffLimit: 2 template: spec: restartPolicy: Never containers: - name: migrate image: registry.example.com/shop/catalog-api:1.4.2 command: ["/app/migrate", "up"] envFrom: - secretRef: name: catalog-dbBoth overlays set the image tag with an images: entry. The pipeline will edit that one line. If the migration fails, the sync stops before any workload changes, and the old version keeps running.
Step 2: Create the project and a CI role
Section titled “Step 2: Create the project and a CI role”The project limits what this team can deploy and where, and defines the role the pipeline will use.
apiVersion: argoproj.io/v1alpha1kind: AppProjectmetadata: name: shop namespace: argocdspec: description: Shop team sourceRepos: - https://git.example.com/shop/deploy-config.git destinations: - name: staging-eu namespace: catalog - name: prod-eu namespace: catalog roles: - name: ci-deployer description: Used by the delivery pipeline policies: - p, proj:shop:ci-deployer, applications, get, shop/*, allow - p, proj:shop:ci-deployer, applications, sync, shop/catalog-api-prod, allow - p, proj:shop:ci-deployer, applications, override, shop/catalog-api-prod, allowCreate a token for the role and store it as a CI secret named ARGOCD_AUTH_TOKEN:
argocd proj role create-token shop ci-deployer --expires-in 90dThe role can read every application in the project, and sync production, including at a pinned revision (the override action). Staging syncs itself, so the pipeline never needs to sync it. See multi-tenancy and projects for the project settings.
Step 3: Define the two Applications
Section titled “Step 3: Define the two Applications”apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: catalog-api-staging namespace: argocd annotations: argocd.argoproj.io/manifest-generate-paths: .;../../base notifications.argoproj.io/subscribe.on-sync-failed.slack: shop-deploysspec: project: shop source: repoURL: https://git.example.com/shop/deploy-config.git targetRevision: main path: apps/catalog-api/overlays/staging destination: name: staging-eu namespace: catalog syncPolicy: automated: prune: true selfHeal: true retry: limit: 3 backoff: { duration: 10s, factor: 2, maxDuration: 3m }Production has the same shape with two differences: no automated block, so a sync happens only on request, and a subscription to both outcomes.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: catalog-api-prod namespace: argocd annotations: argocd.argoproj.io/manifest-generate-paths: .;../../base notifications.argoproj.io/subscribe.on-sync-succeeded.slack: shop-deploys notifications.argoproj.io/subscribe.on-sync-failed.slack: shop-deploysspec: project: shop source: repoURL: https://git.example.com/shop/deploy-config.git targetRevision: main path: apps/catalog-api/overlays/prod destination: name: prod-eu namespace: catalogThe manifest-generate-paths annotation limits refreshes to commits that touch this Application’s overlay or the shared base, which matters once the repository holds many services. The notification triggers and Slack service are configured as described in monitoring and notifications.
Bootstrap once by applying a parent Application that watches the argocd/ directory, so later changes to these files go through Git as well (app-of-apps):
kubectl apply -f deploy-config/bootstrap/shop-root.yamlStep 4: Build, deploy to staging, and verify
Section titled “Step 4: Build, deploy to staging, and verify”The workflow in the application repository does the CI work, then hands over through Git. It never talks to a cluster, and its only Argo CD call is a read-only wait.
# .github/workflows/release.yaml in shop/catalog-apiname: releaseon: push: branches: [main]
env: IMAGE: registry.example.com/shop/catalog-api ARGOCD_SERVER: argocd.example.com ARGOCD_VERSION: v3.5.7
jobs: build: runs-on: ubuntu-latest outputs: tag: ${{ steps.vars.outputs.tag }} steps: - uses: actions/checkout@v4 - id: vars run: echo "tag=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT" - uses: docker/login-action@v3 with: registry: registry.example.com username: ${{ secrets.REGISTRY_USER }} password: ${{ secrets.REGISTRY_TOKEN }} - run: | docker build -t "$IMAGE:${{ steps.vars.outputs.tag }}" . docker push "$IMAGE:${{ steps.vars.outputs.tag }}"
staging: needs: build runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: repository: shop/deploy-config token: ${{ secrets.CONFIG_REPO_TOKEN }} - name: Set the staging image tag run: | cd apps/catalog-api/overlays/staging kustomize edit set image "$IMAGE=$IMAGE:${{ needs.build.outputs.tag }}" git config user.name "release-bot" git config user.email "[email protected]" git commit -am "catalog-api: staging ${{ needs.build.outputs.tag }}" git push origin HEAD:main - name: Install the Argo CD CLI run: | curl -sSL -o argocd "https://github.com/argoproj/argo-cd/releases/download/$ARGOCD_VERSION/argocd-linux-amd64" sudo install -m 0755 argocd /usr/local/bin/argocd - name: Wait for staging to be synced and healthy env: ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }} run: | argocd app get catalog-api-staging --refresh --grpc-web > /dev/null argocd app wait catalog-api-staging --sync --health --timeout 300 --grpc-webWhat happens:
- The commit to
deploy-configtriggers a webhook, and Argo CD refreshescatalog-api-staging. The explicit--refreshcovers a missed webhook. - Argo CD runs the
PreSyncmigration, then applies the Deployment and waits for it to become healthy. argocd app waitreturns when the Application is synced and healthy. If it times out, or the Application turnsDegraded, the step fails, the pipeline stops, and the Slack subscription alerts the team.
Step 5: Promote through a pull request
Section titled “Step 5: Promote through a pull request”When staging is healthy, the same workflow opens a pull request that changes only the production tag:
promote: needs: [build, staging] runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: repository: shop/deploy-config token: ${{ secrets.CONFIG_REPO_TOKEN }} - name: Open the production pull request env: GH_TOKEN: ${{ secrets.CONFIG_REPO_TOKEN }} run: | TAG="${{ needs.build.outputs.tag }}" git checkout -b "promote/catalog-api-$TAG" (cd apps/catalog-api/overlays/prod && kustomize edit set image "$IMAGE=$IMAGE:$TAG") git config user.name "release-bot" git config user.email "[email protected]" git commit -am "catalog-api: promote $TAG to prod" git push origin HEAD gh pr create --title "catalog-api: promote $TAG to prod" \ --body "Verified healthy in staging. Source commit: $GITHUB_SHA" --base mainA reviewer sees a one-line diff and the staging result. Branch protection and code owners on deploy-config make that review mandatory. Merging changes the production overlay on main, but production does not move yet, because its sync is manual.
Step 6: Sync production at the merged commit
Section titled “Step 6: Sync production at the merged commit”A small workflow in deploy-config runs when the production overlay changes on main, and syncs that exact commit.
# .github/workflows/deploy-prod.yaml in shop/deploy-configname: deploy-prodon: push: branches: [main] paths: ["apps/catalog-api/overlays/prod/**"]
jobs: sync: runs-on: ubuntu-latest environment: production # an approval gate, if configured on the environment env: ARGOCD_SERVER: argocd.example.com ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }} steps: - name: Install the Argo CD CLI run: | curl -sSL -o argocd "https://github.com/argoproj/argo-cd/releases/download/v3.5.7/argocd-linux-amd64" sudo install -m 0755 argocd /usr/local/bin/argocd - run: argocd app sync catalog-api-prod --revision "$GITHUB_SHA" --grpc-web - run: argocd app wait catalog-api-prod --sync --health --timeout 600 --grpc-web--revision pins the sync to the commit that was reviewed, so a commit merged a minute later cannot slip in. Syncing a revision other than the Application’s target is an override action when application.sync.requireOverridePrivilegeForRevisionSync: "true" is set in argocd-cm. Turn that on and grant override only to this pipeline role, as in step 2, so that nobody else can deploy an arbitrary revision. See CI integration and webhooks.
Step 7: Observe
Section titled “Step 7: Observe”During a release, the Application tree in the UI shows the migration hook, then the Deployment rolling. From the CLI:
argocd app get catalog-api-prodargocd app history catalog-api-prodThe Slack channel receives the outcome from the notification subscription, and metrics record sync status and duration for dashboards and alerts.
Rolling back
Section titled “Rolling back”The preferred rollback is a revert, because it keeps Git as the record:
git revert <promotion-commit>git push origin mainFor staging, Argo CD applies the revert automatically. For production, the deploy-prod workflow runs again because the production overlay changed, and syncs the reverted commit.
In an emergency, argocd app rollback catalog-api-prod <id> redeploys an earlier entry from the history list and works because production has no automated sync. It leaves the cluster ahead of Git, so follow it at once with a revert commit to realign them.
What can go wrong
Section titled “What can go wrong”| Symptom | Likely cause | Fix |
|---|---|---|
argocd app wait returns immediately, before the new version | The Application had not refreshed yet | Keep the --refresh call before the wait, and configure the webhook |
| Staging never becomes healthy and the wait times out | The migration failed, or the new Pods fail their readiness probe | Open the Application, then read the hook Job and Pod logs |
permission denied on sync | The role lacks sync, or override for a pinned revision | Add the policy to ci-deployer |
| Production syncs a different commit than expected | The sync used the branch tip, not --revision | Pin the revision in the workflow |
| The promotion pull request cannot be created | The token lacks write access to deploy-config | Use a token or app installation scoped to that repository |
| Every service’s commits refresh every Application | No manifest-generate-paths annotations | Add them, as in step 3 |
More causes and fixes are on the troubleshooting page.
What you have built
Section titled “What you have built”- CI and delivery are separate. CI produces an artifact and a commit. Argo CD does the deployment and owns the cluster credentials.
- Staging is automatic and production is deliberate. The same Git history serves both.
- Every release is reviewable. An image bump is one line, a promotion is a pull request, and a rollback is a revert.
- The pipeline holds a narrow credential. A project role can read, and sync one Application, and nothing else.
From here, strengthen it with commit signing, progressive delivery for riskier services, and a secrets operator for the catalog-db Secret.