Skip to content
Documentation Background

Monitoring and notifications

A GitOps platform that nobody watches fails quietly: an Application stays OutOfSync for a week, a repo server runs out of memory, a sync fails at night. Monitoring answers “is Argo CD healthy and what is it doing”, and notifications push the events that need a person to act.


The web UI is built for triage. It abstracts Kubernetes primitives into Applications, shows sync and health at a glance, and gives quick access to diffs, history, and logs. A broken deployment is visible in seconds.

It does not cover everything:

  • It shows that something failed, and rarely why in depth
  • It describes one Argo CD instance and has no view across several
  • It has no platform metrics such as CPU, memory, queue depth, or sync duration trends

Prometheus and Grafana fill those gaps, and alert rules and notifications bring the signal to people who are not looking at the screen.


Every Argo CD component exposes Prometheus metrics on its own port:

ComponentServicePortExamples
Application controllerargocd-metrics8082Application sync and health, reconcile time, cluster cache, kubectl calls
API serverargocd-server-metrics8083gRPC request counts, login failures, Redis requests
Repo serverargocd-repo-server8084Git requests, manifest generation
ApplicationSet controllerargocd-applicationset-controller8080ApplicationSet count and reconcile time
Notifications controllerargocd-notifications-controller-metrics9001Deliveries and trigger evaluations

With the Prometheus Operator (the kube-prometheus-stack chart installs it), a ServiceMonitor tells Prometheus where to scrape. The Argo CD Helm chart creates them per component:

controller:
metrics:
enabled: true
serviceMonitor:
enabled: true
additionalLabels:
release: kube-prometheus-stack # must match the Prometheus release's selector
server:
metrics:
enabled: true
serviceMonitor:
enabled: true
additionalLabels:
release: kube-prometheus-stack
repoServer:
metrics:
enabled: true
serviceMonitor:
enabled: true
additionalLabels:
release: kube-prometheus-stack
applicationSet:
metrics:
enabled: true
serviceMonitor:
enabled: true
additionalLabels:
release: kube-prometheus-stack
notifications:
metrics:
enabled: true
serviceMonitor:
enabled: true
additionalLabels:
release: kube-prometheus-stack
Terminal window
helm upgrade -i argo-cd argo/argo-cd -n argocd --reuse-values -f argocd-metrics-values.yaml

The release label is the usual stumbling block. By default, a kube-prometheus-stack Prometheus selects only ServiceMonitor objects carrying its own release label, so a monitor with the wrong value is never scraped. --reuse-values keeps your earlier settings during the upgrade. Each monitor targets the component’s metrics port and /metrics path, with a 30 second interval by default.

To check that scraping works, port-forward Prometheus and open Status, Targets, or query argocd_app_info.

MetricUse
argocd_app_infoOne series per Application with sync_status, health_status, project, and destination labels, the basis of status dashboards and alerts
argocd_app_sync_totalCounter of sync operations by phase, for sync failure rates
argocd_app_reconcileHistogram of reconciliation time, to see when the controller is falling behind
argocd_app_conditionActive Application conditions such as comparison errors
argocd_cluster_connection_statusWhether the hub can reach each managed cluster
argocd_cluster_api_resource_objectsSize of each cluster cache, a predictor of controller memory use
argocd_git_request_totalGit requests by repository and type, to find chatty repositories and rate limit risk
argocd_redis_request_totalRedis requests with failure flag

Add your own Application labels, such as the owning team, to argocd_app_labels with controller.metrics.application.labels in argocd-cmd-params-cm, to route alerts by team. Each distinct value is a new time series, so use low-cardinality labels.

Two example alerts on argocd_app_info:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: argocd-alerts
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
groups:
- name: argocd
rules:
- alert: ArgoAppOutOfSync
expr: argocd_app_info{sync_status="OutOfSync"} == 1
for: 30m # drift that persists, not a sync in progress
labels:
severity: warning
annotations:
summary: "{{ $labels.name }} has been OutOfSync for 30 minutes"
- alert: ArgoAppDegraded
expr: argocd_app_info{health_status="Degraded"} == 1
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.name }} is Degraded"

The Argo CD repository publishes a ready-made Grafana dashboard at examples/dashboard.json. Import it, or let Grafana load it at install time. It shows application status, sync statistics, controller telemetry, cluster statistics, and repo server statistics, plus the resource metrics the UI lacks: memory, CPU, and goroutines. Use these panels to size the components, as covered in high availability and scaling.


The notifications controller watches Applications and sends a message when a condition becomes true. It ships with Argo CD, runs as its own Deployment, and works from three kinds of object:

  • Services - where a message goes: Slack, Mattermost, email, Microsoft Teams, Telegram, PagerDuty, Opsgenie, GitHub commit statuses, a generic webhook, and others
  • Triggers - when to send. A trigger has a condition expression over the Application and the names of templates to send
  • Templates - what to send. A template renders the message and any service-specific parts, such as Slack blocks

Configuration lives in the argocd-notifications-cm ConfigMap, and credentials in the argocd-notifications-secret Secret. An Application chooses what it hears about with subscription annotations.

Start from the catalog, a set of ready-made triggers and templates (on-sync-succeeded, on-sync-failed, on-health-degraded, and others):

Terminal window
kubectl apply -n argocd --server-side --force-conflicts \
-f https://raw.githubusercontent.com/argoproj/argo-cd/stable/notifications_catalog/install.yaml

The Helm chart leaves triggers and templates empty by default, so apply the catalog or define your own under notifications.triggers and notifications.templates.

Register a service and keep its credential in the Secret. Slack:

apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
namespace: argocd
data:
service.slack: |
token: $slack-token # resolved from argocd-notifications-secret
---
apiVersion: v1
kind: Secret
metadata:
name: argocd-notifications-secret
namespace: argocd
stringData:
slack-token: <bot token>

With Helm, use notifications.notifiers for the service definition and notifications.secret.items.slack-token for the token. Mattermost works the same way with service.mattermost holding apiURL and token (a bot account token), and it can reuse Slack-style attachments in templates.

Argo CD’s argocdUrl for links in messages is set with notifications.argocdUrl in the chart, or the context key in the ConfigMap.

The annotation key is notifications.argoproj.io/subscribe.<trigger>.<service>, and the value lists recipients, separated by semicolons:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: shop-api
namespace: argocd
annotations:
notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deployments;team-shop
notifications.argoproj.io/subscribe.on-sync-failed.slack: team-shop

The recipient is a channel name or ID for chat services and an address for email. Put the same annotation on an AppProject to subscribe every Application in the project. Subscribe everyone centrally with a subscriptions list in the ConfigMap, optionally filtered by a label selector, and set defaultTriggers so that a bare notifications.argoproj.io/subscribe.slack: <channel> annotation uses a standard set:

data:
subscriptions: |
- recipients: [slack:platform-alerts]
selector: tier=critical # only Applications with this label
triggers: [on-sync-failed, on-health-degraded]
defaultTriggers: |
- on-sync-failed

A trigger is a list of conditions, each with the templates it sends. Conditions use the expr language and can use the whole Application object:

data:
trigger.on-sync-succeeded: |
- description: Application sync succeeded
when: app.status.operationState.phase in ['Succeeded']
send: [app-sync-succeeded]
trigger.on-deployed: |
- when: app.status?.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
oncePer: app.status.sync.revision
send: [app-sync-succeeded]

Use ?. for fields that may be missing, such as status.operationState, which does not exist until an operation starts. oncePer stops a flapping condition from sending repeatedly: with app.status.sync.revision, one message is sent per revision. In a monorepo use app.status?.operationState.syncResult.revision so that each Application notifies only for its own revisions.

A template is a Go template with access to app, appProject, context, secrets, serviceType, and recipient:

data:
template.app-sync-succeeded: |
message: |
{{.app.metadata.name}} synced to {{.app.status.sync.revision}}.
Details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}
slack:
attachments: |
[{"title": "{{.app.metadata.name}}", "color": "#18be52"}]

message works for every service. Service-specific keys (slack, email, webhook, mattermost) add formatting where the service supports it.

MessageTypical destination
Sync succeeded, deployment startedA team chat channel, for awareness
Sync failed, health degradedThe team channel, and an on-call service such as PagerDuty or Opsgenie for production
Commit statusGitHub, so a pull request or commit shows whether the deployment worked
Anything customThe generic webhook service, which calls any HTTP endpoint

The notifications controller also exposes argocd_notifications_deliveries_total and argocd_notifications_trigger_eval_total, so a failing delivery shows up in the same dashboards.

The CLI can test a configuration without waiting for a real event:

Terminal window
argocd admin notifications template get app-sync-succeeded -o yaml
argocd admin notifications trigger run on-sync-succeeded shop-api
argocd admin notifications template notify app-sync-succeeded shop-api --recipient slack:team-shop

Run them with the -n argocd flag and a kubeconfig that can read the ConfigMap and Secret. trigger run reports whether a condition is true for an Application, and template notify sends a real message.


SymptomLikely causeFix
Prometheus shows no Argo CD targetsThe ServiceMonitor lacks the Prometheus release label, or metrics are not enabledSet additionalLabels.release and metrics.enabled for each component
Dashboard panels are empty for one componentThat component’s monitor is missingEnable serviceMonitor for it in the chart
argocd_app_labels has no dataNo labels are configuredSet controller.metrics.application.labels
High memory or series count in PrometheusMany deleted Applications or high-cardinality labelsSet --metrics-cache-expiration on the controller, and use fewer label values
No notification arrivesNo subscription annotation, a typo in the trigger or service name, or the condition is falseCheck the annotation and run trigger run
Delivery failsBad token, bot not in the channel, or the service is not defined in the ConfigMapCheck the controller logs and argocd_notifications_deliveries_total{succeeded="false"}
Duplicate messagesA flapping conditionAdd oncePer
Messages link to the wrong hostargocdUrl is unsetSet notifications.argocdUrl or context.argocdUrl