Walkthrough: database schema with waves and hooks
Applying a database schema is the textbook use for hooks. The schema job needs a running database, the web application may need the schema, and none of it should start before the storage exists. This example combines sync waves, hooks, probes, and a retry policy to deploy a catalog application in the correct order.
The application
Section titled “The application”catalog has four parts: a MySQL database with a volume, a Service for it, a web Deployment with its Service, and a Job that creates the schema. All manifests live under manifests/ in one Git repository, tied together with a kustomization.yaml.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: catalog namespace: argocdspec: project: default destination: name: in-cluster namespace: catalog source: repoURL: https://git.example.com/org/catalog-config.git targetRevision: main path: manifests syncPolicy: automated: prune: true # remove resources deleted from Git selfHeal: true # revert manual changes syncOptions: - CreateNamespace=true # the catalog namespace does not exist yet retry: limit: 5 # retry transient failures backoff: duration: 5s factor: 2 maxDuration: 3mThe retry policy matters here: a dependency that is a few seconds late should not fail the sync permanently.
Ordering with waves
Section titled “Ordering with waves”Each manifest carries a wave annotation, and the schema Job is a PostSync hook.
| Phase and wave | Manifest | What Argo CD waits for |
|---|---|---|
Sync wave 1 | PersistentVolumeClaim and database Deployment | Claim bound and the database Pod ready |
Sync wave 2 | Database Service | Service created |
Sync wave 3 | Web Deployment | Web Pod ready |
Sync wave 4 | Web Service | Service created |
PostSync wave 0 | Schema Job | Job complete |
# database Deployment and its claimmetadata: annotations: argocd.argoproj.io/sync-wave: "1"The schema Job:
apiVersion: batch/v1kind: Jobmetadata: name: catalog-schema annotations: argocd.argoproj.io/hook: PostSync # after the Sync phase is healthy argocd.argoproj.io/hook-delete-policy: BeforeHookCreation # remove the previous run first argocd.argoproj.io/sync-wave: "0" # wave scoped inside PostSyncBeforeHookCreation is what lets a Job with a fixed name run again on the next sync. Jobs are immutable, so without it the old object would block the new run.
Probes make the waves honest. The database uses a tcpSocket probe on port 3306 and the web application uses an httpGet readiness probe on /, as shown on the application health page. Without them, wave 1 would complete the instant the MySQL container started.
What happens on sync
Section titled “What happens on sync”- Argo CD applies wave 1. The claim binds and the database Pod starts. Everything else is
Missing. - The database readiness probe passes on port 3306, so wave 1 is healthy and wave 2 begins: the database Service is created.
- Wave 3 applies the web Deployment. Its readiness probe must return HTTP 200 before the wave completes.
- Wave 4 creates the web Service.
- The
Syncphase is complete and every resource isHealthy, so thePostSyncphase starts and the schema Job runs. - When the Job completes, the Application shows
HealthyandSynced.
The Job appears in the UI resource tree with an anchor icon, the mark of a hook. To follow it from the command line:
kubectl get pods -n catalog -w
# Expected output (abridged):# catalog-db-<hash> 0/1 Running 0 5s# catalog-db-<hash> 1/1 Running 0 25s <- wave 1 complete# catalog-web-<hash> 1/1 Running 0 10s <- wave 3 complete# catalog-schema-<hash> 0/1 Completed 0 8s <- PostSync hookChoosing where the schema step runs
Section titled “Choosing where the schema step runs”The same Job could run in other places. The right choice depends on where the database lives and on what the application tolerates.
| Placement | Runs | Choose when | Watch out for |
|---|---|---|---|
PreSync hook | Before any manifest is applied | The database already exists outside this Application. The schema must be ready before new application code starts | If the database is part of this same Application, it does not exist yet and the hook fails |
Sync hook in an intermediate wave | Between ordinary resources, for example wave 2 | The database is part of the Application and the web tier needs the schema before it starts | The wave must come after the database is healthy and before the web Deployment |
PostSync hook | After everything is applied and healthy | The database is part of the Application and the web tier can start without a schema | The web application is Healthy before the schema exists, so it must tolerate an empty database for a short time |
A failing hook in any placement stops the sync, so a broken migration fails visibly instead of leaving a half-deployed release.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Likely cause | Fix |
|---|---|---|
| The schema Job fails with a connection error | It started before the database was accepting connections | Check the database readiness probe. The retry policy covers brief delays |
| The Job does not run again on the next sync | The old Job object still exists and blocks the new one | Set hook-delete-policy: BeforeHookCreation |
| Sync stays in progress with the hook running | The Job has not completed | Read the Job Pod’s logs, since failed hook Pods are kept unless HookFailed is set |
| The web Pods start before the schema exists | The Job is a PostSync hook | Accept that, or move the Job to a Sync hook between the database and web waves |
| The Job never ran | An earlier phase failed or a resource is not Healthy | Fix the earlier failure, since PostSync waits for a healthy Sync phase |