Skip to content
Documentation Background

Cluster management

One Argo CD installation can deploy to the cluster it runs in and to any number of others. Every Application names exactly one destination, and Argo CD treats the local cluster and a remote one as the same thing: a Kubernetes API server it has credentials for. Cluster management is the work of registering those credentials and deciding how much each one is allowed to do.


Argo CD uses a push model. The control plane (the hub) renders manifests, caches the result, and connects out to the API server of each target cluster (the spokes) to apply them. Nothing needs to run on the spokes.

flowchart LR
    G["Git repository"] --> H["Argo CD hub"]
    H -->|"API calls"| L["Local cluster (in-cluster)"]
    H -->|"API calls"| R1["Remote cluster A"]
    H -->|"API calls"| R2["Remote cluster B"]

This shapes the operational requirements:

  • Network - the hub needs a route and firewall rules to every spoke’s API server, and the spokes never need to reach the hub
  • Credentials - the hub stores a credential for each spoke as a Secret in its own namespace, so those Secrets are among the most sensitive objects in the cluster
  • Blast radius - an attacker who controls the hub controls every spoke whose credential has write access, which is why sync impersonation and narrow credentials matter

A pull model, where an agent on each spoke fetches from Git, avoids inbound access to the spokes. Argo CD does not work that way, so a hub that reaches clusters it cannot route to needs a network design (VPN, private connectivity, or one Argo CD instance per network zone).


An Application sets spec.destination to one cluster, identified by its API server URL (server) or by the name it was registered under (name). Use one of them, not both.

spec:
destination:
name: prod-eu # or: server: https://prod-eu.example.com:6443
namespace: orders
FieldMatchesNotes
serverThe URL of the cluster’s API serverhttps://kubernetes.default.svc is the local cluster
nameThe name in the cluster Secretin-cluster is the local cluster. Names survive an endpoint change, so they keep manifests portable
namespaceThe namespace for namespaced resourcesApplied only to manifests that do not set their own metadata.namespace

A single Application has a single destination, even when it has several sources. Deploying the same application to many clusters therefore means one Application per cluster, which is what ApplicationSets generate, or what a Helm chart renders from a values file per cluster.


Argo CD can manage the cluster it runs on without any registration. It uses the in-cluster API address, the cluster’s CA, and the token of the application controller’s service account. kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=cluster returns nothing on a fresh install.

To refine the local cluster, create a cluster Secret named in-cluster. A common use is reserving the control plane for administrators by scoping it to one project:

apiVersion: v1
kind: Secret
metadata:
name: in-cluster
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
name: in-cluster
server: https://kubernetes.default.svc
project: sysadmin # only Applications in this project may deploy here
config: |
{
"tlsClientConfig": { "insecure": false }
}
Terminal window
argocd cluster get in-cluster -o json | jq -r .project
# Expected output: sysadmin

The cluster then also appears with a settings page in the UI under Settings, Clusters.


Each remote cluster is a Secret in the Argo CD namespace with the label argocd.argoproj.io/secret-type: cluster. There is one Secret per cluster, so plan the scope of each credential before creating it.

FieldPurpose
nameName used by destination.name
serverAPI server URL, used by destination.server
namespacesOptional comma-separated list of namespaces Argo CD may manage. When set, cluster-scoped resources are ignored unless clusterResources is "true"
clusterResourcesOptional "true" or "false". Read only when namespaces is set
projectOptional. Makes the cluster available only to Applications in that project
configJSON with the connection details

The config payload holds the credentials and TLS settings:

KeyContents
bearerTokenA service account token, in plain text (not base64 encoded)
tlsClientConfig.caDataThe cluster CA certificate, base64 encoded
tlsClientConfig.certData, keyDataA client certificate and key, base64 encoded
tlsClientConfig.insecuretrue skips server certificate verification. Avoid it outside tests
awsAuthConfig, execProviderConfigCloud IAM and external-command authentication, used for EKS, GKE, and AKS
proxyUrlA proxy for the connection to the API server
apiVersion: v1
kind: Secret
metadata:
name: prod-eu
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
env: prod # labels are selectable by the ApplicationSet cluster generator
region: eu
type: Opaque
stringData:
name: prod-eu
server: https://prod-eu.example.com:6443
namespaces: orders,payments # optional restriction
clusterResources: "false"
config: |
{
"bearerToken": "<service account token>",
"tlsClientConfig": {
"insecure": false,
"caData": "<base64 encoded CA certificate>"
}
}

Labels on the Secret have a second job: ApplicationSets read them to decide which clusters receive which applications. Setting env, region, and similar labels at registration is the basis of label-driven fleet management.


argocd cluster add takes a context name from a kubeconfig file and does the setup for you:

Terminal window
argocd cluster add kind-remote \
--kubeconfig ~/remote-cluster.config \
--name prod-eu \
--label env=prod --label region=eu \
--project sysadmin \
--yes

Using your kubeconfig credentials, the command creates these objects on the target cluster:

  1. A service account argocd-manager in the kube-system namespace (--system-namespace changes the namespace)
  2. A ClusterRole argocd-manager-role granting every verb on every resource
  3. A ClusterRoleBinding argocd-manager-role-binding
  4. A long-lived token for the service account

It then writes the cluster Secret to the Argo CD namespace with that token and the CA data. The effect is cluster-admin on the target. To narrow it, pass --namespace (repeatable) to restrict the cluster to specific namespaces, which creates namespaced Roles instead of a ClusterRole, and --cluster-resources to also allow cluster-scoped resources.

The other subcommands manage what exists:

Terminal window
argocd cluster list # server, name, version, status, project
argocd cluster get prod-eu -o yaml
argocd cluster set prod-eu --project sysadmin # change settings without editing the Secret
argocd cluster rm prod-eu

A new cluster shows Unknown status and the message Cluster has no applications and is not being monitored until an Application targets it. Running the commands requires clusters permissions in Argo CD RBAC, which lets platform teams delegate cluster work without handing out kubectl access to the hub.

The CLI is convenient, but a GitOps team wants clusters described as manifests. Repeat what the CLI does by hand, then apply the Secret.

Terminal window
# on the remote cluster
kubectl create sa argocd-manager -n kube-system
kubectl create clusterrolebinding argocd-manager-role-binding \
--clusterrole=cluster-admin --serviceaccount=kube-system:argocd-manager

Since Kubernetes 1.24, service accounts no longer get a token Secret automatically. Request one explicitly:

apiVersion: v1
kind: Secret
metadata:
name: argocd-manager-token
namespace: kube-system
annotations:
kubernetes.io/service-account.name: argocd-manager
type: kubernetes.io/service-account-token
Terminal window
TOKEN=$(kubectl get secret argocd-manager-token -n kube-system \
-o jsonpath='{.data.token}' | base64 -d)

Put the token and the cluster CA in a cluster Secret like the one above and apply it to the hub. A Role scoped to the namespaces the tenant needs is safer than cluster-admin here, since you choose the permissions.

A cluster Secret holds a bearer token, so a plain-text copy in a repository is a credential leak. Create cluster Secrets from a secret store instead, with a tool such as the External Secrets Operator or Sealed Secrets, and commit only the reference. Note that sealing a Secret removes its labels, so re-add argocd.argoproj.io/secret-type: cluster through the template.


Because a cluster is a Secret, any Kubernetes tool updates it: kubectl edit, kubectl patch, a re-applied manifest, or argocd cluster set. All of them change the same object. Use argocd cluster set when the person making the change has Argo CD RBAC but no access to the hub cluster.

Typical changes are rotating a service account token, renewing a CA certificate, changing an endpoint, and moving a cluster to a different project. In config, bearerToken stays plain text and caData stays base64 encoded. A mistake in this encoding produces authentication or certificate errors.

Annotating a cluster Secret with argocd.argoproj.io/skip-reconcile: "true" stops the application controller from reconciling every Application on that cluster while keeping it listed. It suits maintenance windows and clusters being decommissioned.


SymptomLikely causeFix
Cluster status stays UnknownNo Application targets it yetDeploy an Application, this status is normal for an unused cluster
argocd cluster add fails to connectThe context’s API server address is not reachable from where the command runs (for example 127.0.0.1)Use a kubeconfig whose server address is routable from the hub and from your machine
Applications show Unknown and the controller logs x509 errorsThe CA in caData is wrong, or not base64 encodedReplace caData with the base64 of the cluster CA
Unauthorized or Forbidden on the targetThe token expired or was revoked, or the service account lost its bindingIssue a new token, update bearerToken, and verify the ClusterRoleBinding
An Application cannot deploy to a registered clusterThe cluster has a project set to a different projectUse the right project, or change the cluster’s project
Cluster-scoped resources are skippednamespaces is set and clusterResources is not "true"Set clusterResources: "true" or remove the namespace restriction
The destination is rejected as not permittedThe project does not allow that server and namespaceAdd it to the AppProject destinations
The Application reports an unknown destination clusterdestination.name or server matches no cluster SecretCheck the spelling, and use name or server, not both