Skip to content
Documentation Background

App-of-apps and ApplicationSets

One Application deploys one source to one destination. A real platform has dozens of them: add-ons on every cluster, a service per team, the same stack in each environment, a database that must exist before the API. Two patterns create many Applications from one place. App-of-apps uses an Application to manage other Application manifests. ApplicationSet is a controller that generates Applications from a template and a data source.


Each Application is reconciled on its own. The Application resource has no field for “deploy after X”, and an Application does not see the health or sync state of any other. Sync waves order resources, but only inside one Application.

That is a problem when a stack is split by component (database, API, frontend) or by layer (service mesh before workloads, operators before the custom resources they own). Four approaches cover it, and they combine freely:

ApproachHow it orders deployments
One ApplicationPut the whole stack in one source and use sync waves and hooks
App-of-apps with wavesA parent applies child Applications in wave order and waits for each wave to be healthy
ApplicationSet progressive syncThe controller updates generated Applications in labelled steps
Eventual consistencyRetries until prerequisites exist, with no explicit order

Every approach that waits for an Application depends on that Application’s health meaning something. See application health for readiness probes and for the Application health check, which a default installation lacks.


A parent Application points at a Git directory whose contents are Application manifests. Syncing the parent creates the children, and Argo CD tracks each child like any other Application. The UI shows the parent with a tree of children beneath it.

The pattern started as a way to bootstrap Argo CD itself. It also serves disaster recovery (recreating a control plane means applying one manifest) and organizes deployments across many clusters.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: shop
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io # deleting the parent deletes the children and their resources
spec:
project: platform
source:
repoURL: https://git.example.com/platform/apps.git
targetRevision: main
path: shop/applications # a directory of child Application manifests
destination:
server: https://kubernetes.default.svc
namespace: argocd # Application objects live in the Argo CD namespace
syncPolicy:
automated:
prune: true
selfHeal: true
Terminal window
kubectl apply -n argocd -f shop-parent.yaml
kubectl get applications -n argocd
# Expected output:
# NAME SYNC STATUS HEALTH STATUS
# shop Synced Healthy
# shop-db Synced Healthy
# shop-api Synced Healthy
# shop-frontend Synced Healthy

Put a wave annotation on each child Application manifest. The parent applies the lowest wave, waits until those Applications are healthy, then moves on.

shop/applications/shop-db.yaml
metadata:
name: shop-db
annotations:
argocd.argoproj.io/sync-wave: "1" # first
---
# shop/applications/shop-api.yaml
metadata:
name: shop-api
annotations:
argocd.argoproj.io/sync-wave: "2" # after the database is healthy
---
# shop/applications/shop-frontend.yaml
metadata:
name: shop-frontend
annotations:
argocd.argoproj.io/sync-wave: "3" # last

The sequence is wave 1 (database) syncing while the parent shows Progressing, then wave 2 (API), then wave 3 (frontend), and finally the parent turns Healthy. Two prerequisites make this reliable:

  • Readiness probes in the child workloads, so a child is not healthy before it can serve
  • The Application health check, so the parent sees the child’s real health instead of treating it as healthy on creation

Without them the waves fire back to back.

The cost is manifests: one for the parent and one per child. Deleting the parent removes everything through finalizers, which is useful and dangerous in equal measure.

When the same Application is needed for several clusters or environments, differing in a field or two, wrap the Application manifests in a Helm chart and feed a values file per target:

templates/catalog.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: catalog-{{ .Values.environment }}
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: shop
source:
repoURL: https://git.example.com/shop/deploy-config.git
path: services/catalog/overlays/{{ .Values.environment }}
targetRevision: main
destination:
namespace: catalog
server: {{ .Values.spec.destination.server }}
values.yaml
environment: prod
spec:
destination:
server: https://kubernetes.default.svc

This predates the ApplicationSet controller and remains supported. Where ordering is not needed, ApplicationSets do the same job with less ceremony.


The ApplicationSet controller ships with Argo CD and reconciles ApplicationSet resources. One ApplicationSet is a factory for Applications: generators produce sets of parameters, and a template turns each set into an Application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: catalog
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"] # fail on a typo instead of rendering an empty value
generators:
- list:
elements:
- cluster: dev-eu
url: https://dev-eu.k8s.example.com
- cluster: prod-eu
url: https://prod-eu.k8s.example.com
template:
metadata:
name: '{{.cluster}}-catalog'
spec:
project: shop
source:
repoURL: https://git.example.com/shop/cluster-deployments.git
targetRevision: main
path: 'catalog/{{.cluster}}'
destination:
server: '{{.url}}'
namespace: catalog

This produces two Applications, dev-eu-catalog and prod-eu-catalog. When the list changes, the controller creates, updates, and deletes Applications to match. Set goTemplate: true to use Go templates (with the Sprig functions) in place of the older {{param}} substitution.

GeneratorSource of parametersTypical use
ListA literal list in the manifestA small, fixed set of targets
ClusterClusters registered in Argo CDOne Application per cluster, or per labelled group of clusters
GitDirectories or JSON and YAML files in a repositoryOne Application per directory, config-file-driven environments
MatrixThe cross product of two generatorsEvery Git directory on every cluster
MergeSeveral generators merged on a key, later ones overrideDefaults with per-target overrides
SCM ProviderRepositories in a GitHub, GitLab, or similar organizationOne Application per repository
Pull RequestOpen pull requests in a repositoryPreview environments per pull request
Cluster Decision ResourceA custom resource that decides on the clustersPlacement engines
PluginAn HTTP service you writeAnything else

A post selector can filter the output of any generator.

Cluster generator. It exposes name, server, project, and the Secret’s labels and annotations as metadata.labels.<key> and metadata.annotations.<key>. Applications appear when a cluster is registered and disappear when it is removed. A label selector narrows the clusters:

generators:
- clusters:
selector:
matchLabels:
env: prod
region: eu

The local cluster has no Secret, so a selector on argocd.argoproj.io/secret-type: cluster excludes it. To include it, create an explicit in-cluster Secret as described in cluster management.

Git directory generator. Each directory matching a pattern becomes an Application, with path.basename and path.segments available to the template. Adding a directory to the repository adds an Application.

generators:
- git:
repoURL: https://git.example.com/platform/addons.git
revision: main
directories:
- path: addons/*
- path: addons/legacy
exclude: true # excludes always beat includes

Git file generator. Each matching JSON or YAML file supplies parameters, which suits an environment described by a config file (cluster, namespace, replica count) kept next to its manifests.

Matrix generator. Combining a Git directory generator with a cluster generator deploys every add-on to every matching cluster:

generators:
- matrix:
generators:
- git:
repoURL: https://git.example.com/platform/addons.git
revision: main
directories:
- path: addons/*
- clusters:
selector:
matchLabels:
type: workload

Each generated Application has the ApplicationSet as its owner. Deleting the ApplicationSet deletes its Applications, and, through the Application finalizer, their deployed resources. Two settings change that:

  • syncPolicy.preserveResourcesOnDeletion: true - the Applications get no finalizer, so deleting them leaves the deployed resources in place
  • kubectl delete applicationset <name> --cascade=orphan - deletes only the ApplicationSet. If the finalizer is still on the Applications, deleting one later still removes its resources

The applicationsSync policy limits what the controller does to existing Applications: create-only never modifies or deletes them, create-update never deletes them, create-delete never modifies them, and sync (the default) does everything. The controller flag --policy sets it for the whole instance.


A plain ApplicationSet updates all of its Applications at once. Progressive syncs put the update in stages, such as dev, then staging, then production, and wait for each stage to be healthy. The feature is beta since Argo CD 3.3 and is off until enabled in argocd-cmd-params-cm:

data:
applicationsetcontroller.enable.progressive.syncs: "true"

Restart the ApplicationSet controller after changing it. The strategy goes under spec.strategy, and the template adds the labels the steps select on:

spec:
strategy:
type: RollingSync
deletionOrder: Reverse # delete in reverse step order
rollingSync:
steps:
- matchExpressions:
- key: env
operator: In
values: [dev]
- matchExpressions:
- key: env
operator: In
values: [qa]
maxUpdate: 0 # 0 holds the step until someone syncs it by hand
- matchExpressions:
- key: env
operator: In
values: [prod]
maxUpdate: 10% # sync a tenth of the matching Applications at a time
template:
metadata:
name: '{{.cluster}}-catalog'
labels:
env: '{{.env}}' # the label the steps match on

How it behaves:

  • Each step waits for every selected Application to become Healthy before the next step starts.
  • maxUpdate limits how many Applications in a step sync at once, as an integer or a percentage, which avoids a burst of work against the clusters and the controller when thousands of Applications change together.
  • The controller forces automated sync off on generated Applications and triggers syncs itself, so sync windows and each Application’s retry policy still apply.
  • An Application matched by no step is not synced by the strategy and needs a manual sync.
  • deletionOrder: Reverse removes Applications from the last step to the first, so a frontend goes before its backend.

Because the steps depend on Application health, the same prerequisites apply: readiness probes in the workloads and the Application health check.


The lightest option is to skip explicit ordering. Argo CD keeps retrying, and Kubernetes converges as prerequisites appear. Add a retry policy so a failed sync is attempted again with exponential backoff, and mark resources whose CRDs may not exist yet so their dry run is skipped:

# on the Application
spec:
syncPolicy:
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
# on a resource whose CRD arrives from another Application
metadata:
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

This needs no parent, no ApplicationSet strategy, and no health checks. It is passive, though: it cannot guarantee a sequence, only that things work out after enough attempts, and a failure can take several backoff intervals to resolve. See sync policies and options.


The patterns are not exclusive. A common large-scale layout is an app of ApplicationSets: a parent Application manages ApplicationSet manifests, waves on those ApplicationSets order the layers (cluster add-ons before application workloads), and each ApplicationSet uses RollingSync for its own stages. Retries sit underneath as a safety net.

NeedUse
A fixed set of Applications with a strict orderApp-of-apps with waves
The same application on many clusters or environmentsApplicationSet with a cluster or list generator
One Application per directory, repository, or pull requestApplicationSet with a Git, SCM, or pull request generator
Staged rollout across generated ApplicationsApplicationSet with RollingSync
Bootstrapping a new clusterParent Application applying ApplicationSets
Ordering of layers plus templating at scaleApp of ApplicationSets
Tolerating missing prerequisites without orderingRetries with SkipDryRunOnMissingResource

A common repository layout puts parent and child Application manifests (or ApplicationSets) in a directory such as argocd/, separate from the application sources they point at, so changing the platform and changing a service are different reviews.


SymptomLikely causeFix
Child waves run back to backNo readiness probes, or no Application health checkAdd probes and the health customization
Parent stays ProgressingA child never becomes healthyOpen the child, fix its failing resource
ApplicationSet creates no ApplicationsThe generator returned nothing, or the template is invalidCheck the controller logs and the ApplicationSet status conditions
Template renders empty valuesA parameter name is misspelled and missingkey=error is offEnable goTemplateOptions: ["missingkey=error"]
Applications are rejected with a project errorThe templated project does not allow the source or destinationFix the project or the template
An ApplicationSet change overwrites a manual edit to a generated ApplicationThe controller owns the Application specChange the template instead, or use the create-only policy
Deleting an ApplicationSet removed workloadsCascading deletion with finalizers on the ApplicationsSet preserveResourcesOnDeletion: true before deleting
Progressive sync does nothingThe feature is not enabled, or the controller was not restartedSet the parameter and restart the ApplicationSet controller
Some Applications never sync under RollingSyncNo step’s matchExpressions selects themAdd them to a step, or sync them manually