High availability and scaling
Argo CD’s default installation runs one replica of each component. That works for a lab, and it makes a node failure or an upgrade a short outage of the delivery system. As the installation grows, the same components also run out of CPU, memory, or time. High availability keeps Argo CD running through failures, and scaling keeps it fast as the number of Applications and clusters grows. The two overlap but are not the same work.
Why Argo CD is easy to make available
Section titled “Why Argo CD is easy to make available”Argo CD keeps no state of its own that cannot be rebuilt:
- Configuration and desired state - Applications, AppProjects, ApplicationSets, ConfigMaps, and Secrets are Kubernetes objects stored in etcd. Availability of that data is the availability of the Kubernetes control plane.
- Operation status - sync and health results live in each Application’s
statusfield. - Redis - a disposable cache for repository files, generated manifests, and the live resource tree. Losing it costs time to rebuild the cache, never data.
Because no component holds local state, a Pod can be killed and rescheduled without loss. On a highly available Kubernetes cluster, Argo CD already survives a node failure, with a short gap while Pods restart. The HA configuration removes that gap by running several replicas at once.
HA installation
Section titled “HA installation”Two paths install the HA topology.
- Manifests -
manifests/ha/install.yaml(cluster-scoped) andmanifests/ha/namespace-install.yaml(namespace-scoped), as covered in installation and access - Helm chart - values that raise replica counts and enable the Redis HA subchart
Fixed replicas:
redis-ha: enabled: true # replaces the single Redis Deployment with a Redis HA StatefulSet
controller: replicas: 1 # more than 1 turns on sharding, see below
server: replicas: 2
repoServer: replicas: 2
applicationSet: replicas: 2Autoscaling for the stateless services:
redis-ha: enabled: true
controller: replicas: 1
server: autoscaling: enabled: true minReplicas: 2
repoServer: autoscaling: enabled: true minReplicas: 2
applicationSet: replicas: 2helm upgrade -i argo-cd argo/argo-cd -n argocd --reuse-values -f argocd-ha-values.yaml| Component | HA approach | Why |
|---|---|---|
| API server | Two or more replicas, or an HPA | Stateless. Three or more avoids downtime during rolling upgrades, and ARGOCD_API_SERVER_REPLICAS should match so that the login-concurrency limit is divided among them |
| Repo server | Two or more replicas, or an HPA | Stateless, and each replica keeps its own repository clones |
| ApplicationSet controller | Two replicas | Stateless |
| Redis | Redis HA: three servers and three HAProxy Pods | Quorum and failover for the cache |
| Application controller | One replica per shard | It scales by sharding, not by standby copies |
| Dex | One replica | Its database is in memory, so replicas would hold inconsistent data |
Setting explicit anti-affinity or topology spread constraints on the scaled services spreads their replicas across nodes and zones too, so a single node loss cannot take out both replicas of the API server.
Raising every replica count blindly does not make Argo CD faster. The application controller does not gain from standby copies of itself. Scale the component that the metrics show is saturated.
Sizing component resources
Section titled “Sizing component resources”Scaling up, giving a component more CPU and memory, is the quickest gain. Each component ships with requests and limits that suit a modest installation. Adjust them from monitored usage (monitoring and notifications covers the dashboard), not from guesses.
| Component | Grow its resources when |
|---|---|
| Redis | Many large repositories, or a high volume of Kubernetes requests |
| Application controller | It manages many Applications or large clusters, and status and health evaluation is slow. Cache size grows with the number of objects in each managed cluster |
| API server | UI and CLI requests are slow on a busy multi-tenant instance |
| Repo server | Many repositories, large monorepos, or heavy manifest generation |
| ApplicationSet controller | Many ApplicationSets, or generators that produce many Applications |
Resources go in the Helm values per component:
controller: resources: requests: { cpu: 500m, memory: 1Gi } limits: { memory: 2Gi }repoServer: resources: requests: { cpu: 250m, memory: 512Mi } limits: { memory: 1Gi }These numbers are placeholders for the shape of the configuration. Real values depend on the installation, which is why measuring comes first. A memory limit that an out-of-memory spike exceeds restarts the Pod and drops its work, so leave headroom above the observed peak.
Sharding the application controller
Section titled “Sharding the application controller”One application controller watches every managed cluster and reconciles every Application. Its memory grows with the number of resources in the clusters it caches. When one controller cannot keep up, sharding splits the managed clusters across several controller replicas, and each replica handles only its own clusters.
The controller runs as a StatefulSet, and the Pod ordinal is the shard number: argocd-application-controller-0 is shard 0, -1 is shard 1. Increase controller.replicas to enable it. The chart also writes the count to the ARGOCD_CONTROLLER_REPLICAS environment variable.
controller: replicas: 2Check how clusters are spread:
argocd admin cluster stats -n argocdA shard handles all Applications whose destination cluster it owns, so sharding balances by cluster. One huge cluster still lands on one shard.
Distribution algorithms
Section titled “Distribution algorithms”Set the algorithm with controller.sharding.algorithm in argocd-cmd-params-cm:
| Algorithm | Behavior | Status |
|---|---|---|
legacy | Hashes the cluster’s identifier modulo the shard count. Distribution is not even, and a shard can end up with a disproportionate load | Default |
round-robin | Spreads clusters evenly across shards. Removing the cluster with the lowest rank can reshuffle every cluster | Alpha |
consistent-hashing | Consistent hashing with bounded loads: near-even distribution and little reshuffling when clusters or shards change | Alpha |
Assigning shards by hand
Section titled “Assigning shards by hand”To fix an imbalance, pin a cluster to a shard with the shard key in its cluster Secret:
kubectl patch secret prod-eu -n argocd --patch '{"stringData": {"shard": "1"}}'argocd admin cluster stats -n argocdA common production layout gives each busy production cluster its own shard, and shares two or three shards among all non-production clusters. Choose the split from CPU, memory, and argocd_app_reconcile data per shard.
Dynamic cluster distribution
Section titled “Dynamic cluster distribution”Normally the shard count is fixed by ARGOCD_CONTROLLER_REPLICAS, and changing it restarts the controller Pods. Dynamic cluster distribution (alpha since 2.9) runs the controller as a Deployment instead and reads the replica count from it, so adding or removing replicas redistributes clusters without restarting the rest. Enable it in the chart with controller.dynamicClusterDistribution: true.
Each controller Pod records its shard and a heartbeat in the argocd-app-controller-shard-cm ConfigMap, refreshed every 10 seconds by default (controller.heartbeatTime). A Pod whose heartbeat is stale for three intervals fails its readiness probe, and a change in the replica count triggers a fresh distribution of clusters across the remaining shards.
Tuning throughput
Section titled “Tuning throughput”Before adding replicas, check whether a setting is the limit.
Application controller
- It uses two work queues, for reconciliation and for sync operations.
controller.status.processors(20 by default) andcontroller.operation.processors(10 by default) set how many workers serve them. One thousand Applications is a typical point to raise them to 50 and 25. controller.kubectl.parallelism.limitcaps concurrentkubectlexecutions.- Manifest generation is the slowest step. A
Context deadline exceededduring reconciliation means generation exceededcontroller.repo.server.timeout.seconds. Raise it, and scale the repo server. - The controller caches only the preferred API version of each resource. If the manifests in Git use an older version that cannot be converted, it falls back to slow API queries (
argocd_app_k8s_request_totalshows it). Use the preferred versions in Git. - A resource that updates itself constantly, such as an operator rewriting a status timestamp, triggers a reconcile for its Application each time.
resource.customizations.ignoreResourceUpdates.<group>_<Kind>inargocd-cmignores updates to named fields. The controller log lineRequesting app refresh caused by object updatereveals which kinds cause the churn.
Repo server
--parallelismlimitcaps concurrent manifest generations and prevents out-of-memory kills.- Repositories are cloned into
/tmp, so a large number of repositories can fill the Pod’s disk. Mount a persistent volume if needed. - Generated manifests are cached for 24 hours by default. If a Helm chart or a Kustomize remote base changes without a new commit, lower
--repo-cache-expiration(for example to1h) at the cost of more rendering. - Tool commands time out after 90 seconds (
ARGOCD_EXEC_TIMEOUT). - Set
ARGOCD_GIT_ATTEMPTS_COUNTto retry transientgit ls-remotefailures.
Monorepos. Many Applications in one repository strain the repo server, because each new commit invalidates the manifest cache for all of them.
- Use fully qualified references in
targetRevision, such asrefs/heads/main, so that Argo CD does not scan every branch and tag to resolve a short name. - Add the
argocd.argoproj.io/manifest-generate-pathsannotation to each Application, as described in CI integration and webhooks. - Helm applications render in parallel by default. Custom plugins that write temporary files can create an
.argocd-allow-concurrencyfile, or use the sidecar plugin form, to avoid serializing.
API server. With 3000 or more Applications, raise ARGOCD_GRPC_MAX_SIZE_MB (200 by default) if large responses fail. server.glob.cache.size (10000 by default) controls how many compiled RBAC patterns are cached.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Likely cause | Fix |
|---|---|---|
Redis HA Pods stay Pending | Fewer than three nodes, or anti-affinity cannot be met | Add nodes, or run the non-HA topology |
| The UI is unreachable during upgrades | A single API server replica | Run at least two, three to avoid any gap |
| One controller shard is at its memory limit while others idle | A heavy cluster was hashed to that shard | Pin clusters to shards with the shard key |
| Controller memory keeps growing | Large clusters with many objects in the cache | Shard, raise memory, and ignore noisy resource updates |
Reconciliation fails with Context deadline exceeded | Manifest generation takes longer than the timeout | Raise the repo server timeout on the controller, and scale or tune the repo server |
| Repo server restarts with out-of-memory kills | Too many concurrent generations | Lower --parallelismlimit or raise memory |
| Applications take long to refresh after many commits to a monorepo | Every commit invalidates every Application’s cache | Add manifest paths annotations and use fully qualified revisions |
| Controller Pod reports not ready with dynamic distribution | Its heartbeat in the shard ConfigMap is stale | Check the controller logs and connectivity to the Kubernetes API |