Skip to content
Documentation Background

High availability and scaling

Argo CD’s default installation runs one replica of each component. That works for a lab, and it makes a node failure or an upgrade a short outage of the delivery system. As the installation grows, the same components also run out of CPU, memory, or time. High availability keeps Argo CD running through failures, and scaling keeps it fast as the number of Applications and clusters grows. The two overlap but are not the same work.


Argo CD keeps no state of its own that cannot be rebuilt:

  • Configuration and desired state - Applications, AppProjects, ApplicationSets, ConfigMaps, and Secrets are Kubernetes objects stored in etcd. Availability of that data is the availability of the Kubernetes control plane.
  • Operation status - sync and health results live in each Application’s status field.
  • Redis - a disposable cache for repository files, generated manifests, and the live resource tree. Losing it costs time to rebuild the cache, never data.

Because no component holds local state, a Pod can be killed and rescheduled without loss. On a highly available Kubernetes cluster, Argo CD already survives a node failure, with a short gap while Pods restart. The HA configuration removes that gap by running several replicas at once.


Two paths install the HA topology.

  • Manifests - manifests/ha/install.yaml (cluster-scoped) and manifests/ha/namespace-install.yaml (namespace-scoped), as covered in installation and access
  • Helm chart - values that raise replica counts and enable the Redis HA subchart

Fixed replicas:

redis-ha:
enabled: true # replaces the single Redis Deployment with a Redis HA StatefulSet
controller:
replicas: 1 # more than 1 turns on sharding, see below
server:
replicas: 2
repoServer:
replicas: 2
applicationSet:
replicas: 2

Autoscaling for the stateless services:

redis-ha:
enabled: true
controller:
replicas: 1
server:
autoscaling:
enabled: true
minReplicas: 2
repoServer:
autoscaling:
enabled: true
minReplicas: 2
applicationSet:
replicas: 2
Terminal window
helm upgrade -i argo-cd argo/argo-cd -n argocd --reuse-values -f argocd-ha-values.yaml
ComponentHA approachWhy
API serverTwo or more replicas, or an HPAStateless. Three or more avoids downtime during rolling upgrades, and ARGOCD_API_SERVER_REPLICAS should match so that the login-concurrency limit is divided among them
Repo serverTwo or more replicas, or an HPAStateless, and each replica keeps its own repository clones
ApplicationSet controllerTwo replicasStateless
RedisRedis HA: three servers and three HAProxy PodsQuorum and failover for the cache
Application controllerOne replica per shardIt scales by sharding, not by standby copies
DexOne replicaIts database is in memory, so replicas would hold inconsistent data

Setting explicit anti-affinity or topology spread constraints on the scaled services spreads their replicas across nodes and zones too, so a single node loss cannot take out both replicas of the API server.

Raising every replica count blindly does not make Argo CD faster. The application controller does not gain from standby copies of itself. Scale the component that the metrics show is saturated.


Scaling up, giving a component more CPU and memory, is the quickest gain. Each component ships with requests and limits that suit a modest installation. Adjust them from monitored usage (monitoring and notifications covers the dashboard), not from guesses.

ComponentGrow its resources when
RedisMany large repositories, or a high volume of Kubernetes requests
Application controllerIt manages many Applications or large clusters, and status and health evaluation is slow. Cache size grows with the number of objects in each managed cluster
API serverUI and CLI requests are slow on a busy multi-tenant instance
Repo serverMany repositories, large monorepos, or heavy manifest generation
ApplicationSet controllerMany ApplicationSets, or generators that produce many Applications

Resources go in the Helm values per component:

controller:
resources:
requests: { cpu: 500m, memory: 1Gi }
limits: { memory: 2Gi }
repoServer:
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { memory: 1Gi }

These numbers are placeholders for the shape of the configuration. Real values depend on the installation, which is why measuring comes first. A memory limit that an out-of-memory spike exceeds restarts the Pod and drops its work, so leave headroom above the observed peak.


One application controller watches every managed cluster and reconciles every Application. Its memory grows with the number of resources in the clusters it caches. When one controller cannot keep up, sharding splits the managed clusters across several controller replicas, and each replica handles only its own clusters.

The controller runs as a StatefulSet, and the Pod ordinal is the shard number: argocd-application-controller-0 is shard 0, -1 is shard 1. Increase controller.replicas to enable it. The chart also writes the count to the ARGOCD_CONTROLLER_REPLICAS environment variable.

controller:
replicas: 2

Check how clusters are spread:

Terminal window
argocd admin cluster stats -n argocd

A shard handles all Applications whose destination cluster it owns, so sharding balances by cluster. One huge cluster still lands on one shard.

Set the algorithm with controller.sharding.algorithm in argocd-cmd-params-cm:

AlgorithmBehaviorStatus
legacyHashes the cluster’s identifier modulo the shard count. Distribution is not even, and a shard can end up with a disproportionate loadDefault
round-robinSpreads clusters evenly across shards. Removing the cluster with the lowest rank can reshuffle every clusterAlpha
consistent-hashingConsistent hashing with bounded loads: near-even distribution and little reshuffling when clusters or shards changeAlpha

To fix an imbalance, pin a cluster to a shard with the shard key in its cluster Secret:

Terminal window
kubectl patch secret prod-eu -n argocd --patch '{"stringData": {"shard": "1"}}'
argocd admin cluster stats -n argocd

A common production layout gives each busy production cluster its own shard, and shares two or three shards among all non-production clusters. Choose the split from CPU, memory, and argocd_app_reconcile data per shard.

Normally the shard count is fixed by ARGOCD_CONTROLLER_REPLICAS, and changing it restarts the controller Pods. Dynamic cluster distribution (alpha since 2.9) runs the controller as a Deployment instead and reads the replica count from it, so adding or removing replicas redistributes clusters without restarting the rest. Enable it in the chart with controller.dynamicClusterDistribution: true.

Each controller Pod records its shard and a heartbeat in the argocd-app-controller-shard-cm ConfigMap, refreshed every 10 seconds by default (controller.heartbeatTime). A Pod whose heartbeat is stale for three intervals fails its readiness probe, and a change in the replica count triggers a fresh distribution of clusters across the remaining shards.


Before adding replicas, check whether a setting is the limit.

Application controller

  • It uses two work queues, for reconciliation and for sync operations. controller.status.processors (20 by default) and controller.operation.processors (10 by default) set how many workers serve them. One thousand Applications is a typical point to raise them to 50 and 25.
  • controller.kubectl.parallelism.limit caps concurrent kubectl executions.
  • Manifest generation is the slowest step. A Context deadline exceeded during reconciliation means generation exceeded controller.repo.server.timeout.seconds. Raise it, and scale the repo server.
  • The controller caches only the preferred API version of each resource. If the manifests in Git use an older version that cannot be converted, it falls back to slow API queries (argocd_app_k8s_request_total shows it). Use the preferred versions in Git.
  • A resource that updates itself constantly, such as an operator rewriting a status timestamp, triggers a reconcile for its Application each time. resource.customizations.ignoreResourceUpdates.<group>_<Kind> in argocd-cm ignores updates to named fields. The controller log line Requesting app refresh caused by object update reveals which kinds cause the churn.

Repo server

  • --parallelismlimit caps concurrent manifest generations and prevents out-of-memory kills.
  • Repositories are cloned into /tmp, so a large number of repositories can fill the Pod’s disk. Mount a persistent volume if needed.
  • Generated manifests are cached for 24 hours by default. If a Helm chart or a Kustomize remote base changes without a new commit, lower --repo-cache-expiration (for example to 1h) at the cost of more rendering.
  • Tool commands time out after 90 seconds (ARGOCD_EXEC_TIMEOUT).
  • Set ARGOCD_GIT_ATTEMPTS_COUNT to retry transient git ls-remote failures.

Monorepos. Many Applications in one repository strain the repo server, because each new commit invalidates the manifest cache for all of them.

  • Use fully qualified references in targetRevision, such as refs/heads/main, so that Argo CD does not scan every branch and tag to resolve a short name.
  • Add the argocd.argoproj.io/manifest-generate-paths annotation to each Application, as described in CI integration and webhooks.
  • Helm applications render in parallel by default. Custom plugins that write temporary files can create an .argocd-allow-concurrency file, or use the sidecar plugin form, to avoid serializing.

API server. With 3000 or more Applications, raise ARGOCD_GRPC_MAX_SIZE_MB (200 by default) if large responses fail. server.glob.cache.size (10000 by default) controls how many compiled RBAC patterns are cached.


SymptomLikely causeFix
Redis HA Pods stay PendingFewer than three nodes, or anti-affinity cannot be metAdd nodes, or run the non-HA topology
The UI is unreachable during upgradesA single API server replicaRun at least two, three to avoid any gap
One controller shard is at its memory limit while others idleA heavy cluster was hashed to that shardPin clusters to shards with the shard key
Controller memory keeps growingLarge clusters with many objects in the cacheShard, raise memory, and ignore noisy resource updates
Reconciliation fails with Context deadline exceededManifest generation takes longer than the timeoutRaise the repo server timeout on the controller, and scale or tune the repo server
Repo server restarts with out-of-memory killsToo many concurrent generationsLower --parallelismlimit or raise memory
Applications take long to refresh after many commits to a monorepoEvery commit invalidates every Application’s cacheAdd manifest paths annotations and use fully qualified revisions
Controller Pod reports not ready with dynamic distributionIts heartbeat in the shard ConfigMap is staleCheck the controller logs and connectivity to the Kubernetes API