App-of-apps and ApplicationSets
One Application deploys one source to one destination. A real platform has dozens of them: add-ons on every cluster, a service per team, the same stack in each environment, a database that must exist before the API. Two patterns create many Applications from one place. App-of-apps uses an Application to manage other Application manifests. ApplicationSet is a controller that generates Applications from a template and a data source.
Applications are independent
Section titled “Applications are independent”Each Application is reconciled on its own. The Application resource has no field for “deploy after X”, and an Application does not see the health or sync state of any other. Sync waves order resources, but only inside one Application.
That is a problem when a stack is split by component (database, API, frontend) or by layer (service mesh before workloads, operators before the custom resources they own). Four approaches cover it, and they combine freely:
| Approach | How it orders deployments |
|---|---|
| One Application | Put the whole stack in one source and use sync waves and hooks |
| App-of-apps with waves | A parent applies child Applications in wave order and waits for each wave to be healthy |
| ApplicationSet progressive sync | The controller updates generated Applications in labelled steps |
| Eventual consistency | Retries until prerequisites exist, with no explicit order |
Every approach that waits for an Application depends on that Application’s health meaning something. See application health for readiness probes and for the Application health check, which a default installation lacks.
App-of-apps
Section titled “App-of-apps”A parent Application points at a Git directory whose contents are Application manifests. Syncing the parent creates the children, and Argo CD tracks each child like any other Application. The UI shows the parent with a tree of children beneath it.
The pattern started as a way to bootstrap Argo CD itself. It also serves disaster recovery (recreating a control plane means applying one manifest) and organizes deployments across many clusters.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: shop namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io # deleting the parent deletes the children and their resourcesspec: project: platform source: repoURL: https://git.example.com/platform/apps.git targetRevision: main path: shop/applications # a directory of child Application manifests destination: server: https://kubernetes.default.svc namespace: argocd # Application objects live in the Argo CD namespace syncPolicy: automated: prune: true selfHeal: truekubectl apply -n argocd -f shop-parent.yamlkubectl get applications -n argocd
# Expected output:# NAME SYNC STATUS HEALTH STATUS# shop Synced Healthy# shop-db Synced Healthy# shop-api Synced Healthy# shop-frontend Synced HealthyOrdering children with waves
Section titled “Ordering children with waves”Put a wave annotation on each child Application manifest. The parent applies the lowest wave, waits until those Applications are healthy, then moves on.
metadata: name: shop-db annotations: argocd.argoproj.io/sync-wave: "1" # first---# shop/applications/shop-api.yamlmetadata: name: shop-api annotations: argocd.argoproj.io/sync-wave: "2" # after the database is healthy---# shop/applications/shop-frontend.yamlmetadata: name: shop-frontend annotations: argocd.argoproj.io/sync-wave: "3" # lastThe sequence is wave 1 (database) syncing while the parent shows Progressing, then wave 2 (API), then wave 3 (frontend), and finally the parent turns Healthy. Two prerequisites make this reliable:
- Readiness probes in the child workloads, so a child is not healthy before it can serve
- The
Applicationhealth check, so the parent sees the child’s real health instead of treating it as healthy on creation
Without them the waves fire back to back.
The cost is manifests: one for the parent and one per child. Deleting the parent removes everything through finalizers, which is useful and dangerous in equal measure.
Templating children with Helm
Section titled “Templating children with Helm”When the same Application is needed for several clusters or environments, differing in a field or two, wrap the Application manifests in a Helm chart and feed a values file per target:
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: catalog-{{ .Values.environment }} namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.iospec: project: shop source: repoURL: https://git.example.com/shop/deploy-config.git path: services/catalog/overlays/{{ .Values.environment }} targetRevision: main destination: namespace: catalog server: {{ .Values.spec.destination.server }}environment: prodspec: destination: server: https://kubernetes.default.svcThis predates the ApplicationSet controller and remains supported. Where ordering is not needed, ApplicationSets do the same job with less ceremony.
ApplicationSets
Section titled “ApplicationSets”The ApplicationSet controller ships with Argo CD and reconciles ApplicationSet resources. One ApplicationSet is a factory for Applications: generators produce sets of parameters, and a template turns each set into an Application.
apiVersion: argoproj.io/v1alpha1kind: ApplicationSetmetadata: name: catalog namespace: argocdspec: goTemplate: true goTemplateOptions: ["missingkey=error"] # fail on a typo instead of rendering an empty value generators: - list: elements: - cluster: dev-eu url: https://dev-eu.k8s.example.com - cluster: prod-eu url: https://prod-eu.k8s.example.com template: metadata: name: '{{.cluster}}-catalog' spec: project: shop source: repoURL: https://git.example.com/shop/cluster-deployments.git targetRevision: main path: 'catalog/{{.cluster}}' destination: server: '{{.url}}' namespace: catalogThis produces two Applications, dev-eu-catalog and prod-eu-catalog. When the list changes, the controller creates, updates, and deletes Applications to match. Set goTemplate: true to use Go templates (with the Sprig functions) in place of the older {{param}} substitution.
Generators
Section titled “Generators”| Generator | Source of parameters | Typical use |
|---|---|---|
| List | A literal list in the manifest | A small, fixed set of targets |
| Cluster | Clusters registered in Argo CD | One Application per cluster, or per labelled group of clusters |
| Git | Directories or JSON and YAML files in a repository | One Application per directory, config-file-driven environments |
| Matrix | The cross product of two generators | Every Git directory on every cluster |
| Merge | Several generators merged on a key, later ones override | Defaults with per-target overrides |
| SCM Provider | Repositories in a GitHub, GitLab, or similar organization | One Application per repository |
| Pull Request | Open pull requests in a repository | Preview environments per pull request |
| Cluster Decision Resource | A custom resource that decides on the clusters | Placement engines |
| Plugin | An HTTP service you write | Anything else |
A post selector can filter the output of any generator.
Cluster generator. It exposes name, server, project, and the Secret’s labels and annotations as metadata.labels.<key> and metadata.annotations.<key>. Applications appear when a cluster is registered and disappear when it is removed. A label selector narrows the clusters:
generators:- clusters: selector: matchLabels: env: prod region: euThe local cluster has no Secret, so a selector on argocd.argoproj.io/secret-type: cluster excludes it. To include it, create an explicit in-cluster Secret as described in cluster management.
Git directory generator. Each directory matching a pattern becomes an Application, with path.basename and path.segments available to the template. Adding a directory to the repository adds an Application.
generators:- git: repoURL: https://git.example.com/platform/addons.git revision: main directories: - path: addons/* - path: addons/legacy exclude: true # excludes always beat includesGit file generator. Each matching JSON or YAML file supplies parameters, which suits an environment described by a config file (cluster, namespace, replica count) kept next to its manifests.
Matrix generator. Combining a Git directory generator with a cluster generator deploys every add-on to every matching cluster:
generators:- matrix: generators: - git: repoURL: https://git.example.com/platform/addons.git revision: main directories: - path: addons/* - clusters: selector: matchLabels: type: workloadLifecycle and deletion
Section titled “Lifecycle and deletion”Each generated Application has the ApplicationSet as its owner. Deleting the ApplicationSet deletes its Applications, and, through the Application finalizer, their deployed resources. Two settings change that:
syncPolicy.preserveResourcesOnDeletion: true- the Applications get no finalizer, so deleting them leaves the deployed resources in placekubectl delete applicationset <name> --cascade=orphan- deletes only the ApplicationSet. If the finalizer is still on the Applications, deleting one later still removes its resources
The applicationsSync policy limits what the controller does to existing Applications: create-only never modifies or deletes them, create-update never deletes them, create-delete never modifies them, and sync (the default) does everything. The controller flag --policy sets it for the whole instance.
Security
Section titled “Security”Progressive syncs
Section titled “Progressive syncs”A plain ApplicationSet updates all of its Applications at once. Progressive syncs put the update in stages, such as dev, then staging, then production, and wait for each stage to be healthy. The feature is beta since Argo CD 3.3 and is off until enabled in argocd-cmd-params-cm:
data: applicationsetcontroller.enable.progressive.syncs: "true"Restart the ApplicationSet controller after changing it. The strategy goes under spec.strategy, and the template adds the labels the steps select on:
spec: strategy: type: RollingSync deletionOrder: Reverse # delete in reverse step order rollingSync: steps: - matchExpressions: - key: env operator: In values: [dev] - matchExpressions: - key: env operator: In values: [qa] maxUpdate: 0 # 0 holds the step until someone syncs it by hand - matchExpressions: - key: env operator: In values: [prod] maxUpdate: 10% # sync a tenth of the matching Applications at a time template: metadata: name: '{{.cluster}}-catalog' labels: env: '{{.env}}' # the label the steps match onHow it behaves:
- Each step waits for every selected Application to become
Healthybefore the next step starts. maxUpdatelimits how many Applications in a step sync at once, as an integer or a percentage, which avoids a burst of work against the clusters and the controller when thousands of Applications change together.- The controller forces automated sync off on generated Applications and triggers syncs itself, so sync windows and each Application’s retry policy still apply.
- An Application matched by no step is not synced by the strategy and needs a manual sync.
deletionOrder: Reverseremoves Applications from the last step to the first, so a frontend goes before its backend.
Because the steps depend on Application health, the same prerequisites apply: readiness probes in the workloads and the Application health check.
Eventual consistency
Section titled “Eventual consistency”The lightest option is to skip explicit ordering. Argo CD keeps retrying, and Kubernetes converges as prerequisites appear. Add a retry policy so a failed sync is attempted again with exponential backoff, and mark resources whose CRDs may not exist yet so their dry run is skipped:
# on the Applicationspec: syncPolicy: retry: limit: 5 backoff: duration: 5s factor: 2 maxDuration: 3m# on a resource whose CRD arrives from another Applicationmetadata: annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=trueThis needs no parent, no ApplicationSet strategy, and no health checks. It is passive, though: it cannot guarantee a sequence, only that things work out after enough attempts, and a failure can take several backoff intervals to resolve. See sync policies and options.
Choosing a pattern
Section titled “Choosing a pattern”The patterns are not exclusive. A common large-scale layout is an app of ApplicationSets: a parent Application manages ApplicationSet manifests, waves on those ApplicationSets order the layers (cluster add-ons before application workloads), and each ApplicationSet uses RollingSync for its own stages. Retries sit underneath as a safety net.
| Need | Use |
|---|---|
| A fixed set of Applications with a strict order | App-of-apps with waves |
| The same application on many clusters or environments | ApplicationSet with a cluster or list generator |
| One Application per directory, repository, or pull request | ApplicationSet with a Git, SCM, or pull request generator |
| Staged rollout across generated Applications | ApplicationSet with RollingSync |
| Bootstrapping a new cluster | Parent Application applying ApplicationSets |
| Ordering of layers plus templating at scale | App of ApplicationSets |
| Tolerating missing prerequisites without ordering | Retries with SkipDryRunOnMissingResource |
A common repository layout puts parent and child Application manifests (or ApplicationSets) in a directory such as argocd/, separate from the application sources they point at, so changing the platform and changing a service are different reviews.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Likely cause | Fix |
|---|---|---|
| Child waves run back to back | No readiness probes, or no Application health check | Add probes and the health customization |
Parent stays Progressing | A child never becomes healthy | Open the child, fix its failing resource |
| ApplicationSet creates no Applications | The generator returned nothing, or the template is invalid | Check the controller logs and the ApplicationSet status conditions |
| Template renders empty values | A parameter name is misspelled and missingkey=error is off | Enable goTemplateOptions: ["missingkey=error"] |
| Applications are rejected with a project error | The templated project does not allow the source or destination | Fix the project or the template |
| An ApplicationSet change overwrites a manual edit to a generated Application | The controller owns the Application spec | Change the template instead, or use the create-only policy |
| Deleting an ApplicationSet removed workloads | Cascading deletion with finalizers on the Applications | Set preserveResourcesOnDeletion: true before deleting |
| Progressive sync does nothing | The feature is not enabled, or the controller was not restarted | Set the parameter and restart the ApplicationSet controller |
| Some Applications never sync under RollingSync | No step’s matchExpressions selects them | Add them to a step, or sync them manually |