Monitoring and notifications
A GitOps platform that nobody watches fails quietly: an Application stays OutOfSync for a week, a repo server runs out of memory, a sync fails at night. Monitoring answers “is Argo CD healthy and what is it doing”, and notifications push the events that need a person to act.
What the UI covers
Section titled “What the UI covers”The web UI is built for triage. It abstracts Kubernetes primitives into Applications, shows sync and health at a glance, and gives quick access to diffs, history, and logs. A broken deployment is visible in seconds.
It does not cover everything:
- It shows that something failed, and rarely why in depth
- It describes one Argo CD instance and has no view across several
- It has no platform metrics such as CPU, memory, queue depth, or sync duration trends
Prometheus and Grafana fill those gaps, and alert rules and notifications bring the signal to people who are not looking at the screen.
Metrics with Prometheus
Section titled “Metrics with Prometheus”Every Argo CD component exposes Prometheus metrics on its own port:
| Component | Service | Port | Examples |
|---|---|---|---|
| Application controller | argocd-metrics | 8082 | Application sync and health, reconcile time, cluster cache, kubectl calls |
| API server | argocd-server-metrics | 8083 | gRPC request counts, login failures, Redis requests |
| Repo server | argocd-repo-server | 8084 | Git requests, manifest generation |
| ApplicationSet controller | argocd-applicationset-controller | 8080 | ApplicationSet count and reconcile time |
| Notifications controller | argocd-notifications-controller-metrics | 9001 | Deliveries and trigger evaluations |
Enabling scraping
Section titled “Enabling scraping”With the Prometheus Operator (the kube-prometheus-stack chart installs it), a ServiceMonitor tells Prometheus where to scrape. The Argo CD Helm chart creates them per component:
controller: metrics: enabled: true serviceMonitor: enabled: true additionalLabels: release: kube-prometheus-stack # must match the Prometheus release's selectorserver: metrics: enabled: true serviceMonitor: enabled: true additionalLabels: release: kube-prometheus-stackrepoServer: metrics: enabled: true serviceMonitor: enabled: true additionalLabels: release: kube-prometheus-stackapplicationSet: metrics: enabled: true serviceMonitor: enabled: true additionalLabels: release: kube-prometheus-stacknotifications: metrics: enabled: true serviceMonitor: enabled: true additionalLabels: release: kube-prometheus-stackhelm upgrade -i argo-cd argo/argo-cd -n argocd --reuse-values -f argocd-metrics-values.yamlThe release label is the usual stumbling block. By default, a kube-prometheus-stack Prometheus selects only ServiceMonitor objects carrying its own release label, so a monitor with the wrong value is never scraped. --reuse-values keeps your earlier settings during the upgrade. Each monitor targets the component’s metrics port and /metrics path, with a 30 second interval by default.
To check that scraping works, port-forward Prometheus and open Status, Targets, or query argocd_app_info.
Metrics worth knowing
Section titled “Metrics worth knowing”| Metric | Use |
|---|---|
argocd_app_info | One series per Application with sync_status, health_status, project, and destination labels, the basis of status dashboards and alerts |
argocd_app_sync_total | Counter of sync operations by phase, for sync failure rates |
argocd_app_reconcile | Histogram of reconciliation time, to see when the controller is falling behind |
argocd_app_condition | Active Application conditions such as comparison errors |
argocd_cluster_connection_status | Whether the hub can reach each managed cluster |
argocd_cluster_api_resource_objects | Size of each cluster cache, a predictor of controller memory use |
argocd_git_request_total | Git requests by repository and type, to find chatty repositories and rate limit risk |
argocd_redis_request_total | Redis requests with failure flag |
Add your own Application labels, such as the owning team, to argocd_app_labels with controller.metrics.application.labels in argocd-cmd-params-cm, to route alerts by team. Each distinct value is a new time series, so use low-cardinality labels.
Two example alerts on argocd_app_info:
apiVersion: monitoring.coreos.com/v1kind: PrometheusRulemetadata: name: argocd-alerts namespace: monitoring labels: release: kube-prometheus-stackspec: groups: - name: argocd rules: - alert: ArgoAppOutOfSync expr: argocd_app_info{sync_status="OutOfSync"} == 1 for: 30m # drift that persists, not a sync in progress labels: severity: warning annotations: summary: "{{ $labels.name }} has been OutOfSync for 30 minutes" - alert: ArgoAppDegraded expr: argocd_app_info{health_status="Degraded"} == 1 for: 10m labels: severity: critical annotations: summary: "{{ $labels.name }} is Degraded"Dashboards
Section titled “Dashboards”The Argo CD repository publishes a ready-made Grafana dashboard at examples/dashboard.json. Import it, or let Grafana load it at install time. It shows application status, sync statistics, controller telemetry, cluster statistics, and repo server statistics, plus the resource metrics the UI lacks: memory, CPU, and goroutines. Use these panels to size the components, as covered in high availability and scaling.
Notifications
Section titled “Notifications”The notifications controller watches Applications and sends a message when a condition becomes true. It ships with Argo CD, runs as its own Deployment, and works from three kinds of object:
- Services - where a message goes: Slack, Mattermost, email, Microsoft Teams, Telegram, PagerDuty, Opsgenie, GitHub commit statuses, a generic webhook, and others
- Triggers - when to send. A trigger has a condition expression over the Application and the names of templates to send
- Templates - what to send. A template renders the message and any service-specific parts, such as Slack blocks
Configuration lives in the argocd-notifications-cm ConfigMap, and credentials in the argocd-notifications-secret Secret. An Application chooses what it hears about with subscription annotations.
Setting it up
Section titled “Setting it up”Start from the catalog, a set of ready-made triggers and templates (on-sync-succeeded, on-sync-failed, on-health-degraded, and others):
kubectl apply -n argocd --server-side --force-conflicts \ -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/notifications_catalog/install.yamlThe Helm chart leaves triggers and templates empty by default, so apply the catalog or define your own under notifications.triggers and notifications.templates.
Register a service and keep its credential in the Secret. Slack:
apiVersion: v1kind: ConfigMapmetadata: name: argocd-notifications-cm namespace: argocddata: service.slack: | token: $slack-token # resolved from argocd-notifications-secret---apiVersion: v1kind: Secretmetadata: name: argocd-notifications-secret namespace: argocdstringData: slack-token: <bot token>With Helm, use notifications.notifiers for the service definition and notifications.secret.items.slack-token for the token. Mattermost works the same way with service.mattermost holding apiURL and token (a bot account token), and it can reuse Slack-style attachments in templates.
Argo CD’s argocdUrl for links in messages is set with notifications.argocdUrl in the chart, or the context key in the ConfigMap.
Subscribing an Application
Section titled “Subscribing an Application”The annotation key is notifications.argoproj.io/subscribe.<trigger>.<service>, and the value lists recipients, separated by semicolons:
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: shop-api namespace: argocd annotations: notifications.argoproj.io/subscribe.on-sync-succeeded.slack: deployments;team-shop notifications.argoproj.io/subscribe.on-sync-failed.slack: team-shopThe recipient is a channel name or ID for chat services and an address for email. Put the same annotation on an AppProject to subscribe every Application in the project. Subscribe everyone centrally with a subscriptions list in the ConfigMap, optionally filtered by a label selector, and set defaultTriggers so that a bare notifications.argoproj.io/subscribe.slack: <channel> annotation uses a standard set:
data: subscriptions: | - recipients: [slack:platform-alerts] selector: tier=critical # only Applications with this label triggers: [on-sync-failed, on-health-degraded] defaultTriggers: | - on-sync-failedTriggers and templates
Section titled “Triggers and templates”A trigger is a list of conditions, each with the templates it sends. Conditions use the expr language and can use the whole Application object:
data: trigger.on-sync-succeeded: | - description: Application sync succeeded when: app.status.operationState.phase in ['Succeeded'] send: [app-sync-succeeded] trigger.on-deployed: | - when: app.status?.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' oncePer: app.status.sync.revision send: [app-sync-succeeded]Use ?. for fields that may be missing, such as status.operationState, which does not exist until an operation starts. oncePer stops a flapping condition from sending repeatedly: with app.status.sync.revision, one message is sent per revision. In a monorepo use app.status?.operationState.syncResult.revision so that each Application notifies only for its own revisions.
A template is a Go template with access to app, appProject, context, secrets, serviceType, and recipient:
data: template.app-sync-succeeded: | message: | {{.app.metadata.name}} synced to {{.app.status.sync.revision}}. Details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}} slack: attachments: | [{"title": "{{.app.metadata.name}}", "color": "#18be52"}]message works for every service. Service-specific keys (slack, email, webhook, mattermost) add formatting where the service supports it.
Where to send what
Section titled “Where to send what”| Message | Typical destination |
|---|---|
| Sync succeeded, deployment started | A team chat channel, for awareness |
| Sync failed, health degraded | The team channel, and an on-call service such as PagerDuty or Opsgenie for production |
| Commit status | GitHub, so a pull request or commit shows whether the deployment worked |
| Anything custom | The generic webhook service, which calls any HTTP endpoint |
The notifications controller also exposes argocd_notifications_deliveries_total and argocd_notifications_trigger_eval_total, so a failing delivery shows up in the same dashboards.
Debugging
Section titled “Debugging”The CLI can test a configuration without waiting for a real event:
argocd admin notifications template get app-sync-succeeded -o yamlargocd admin notifications trigger run on-sync-succeeded shop-apiargocd admin notifications template notify app-sync-succeeded shop-api --recipient slack:team-shopRun them with the -n argocd flag and a kubeconfig that can read the ConfigMap and Secret. trigger run reports whether a condition is true for an Application, and template notify sends a real message.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Likely cause | Fix |
|---|---|---|
| Prometheus shows no Argo CD targets | The ServiceMonitor lacks the Prometheus release label, or metrics are not enabled | Set additionalLabels.release and metrics.enabled for each component |
| Dashboard panels are empty for one component | That component’s monitor is missing | Enable serviceMonitor for it in the chart |
argocd_app_labels has no data | No labels are configured | Set controller.metrics.application.labels |
| High memory or series count in Prometheus | Many deleted Applications or high-cardinality labels | Set --metrics-cache-expiration on the controller, and use fewer label values |
| No notification arrives | No subscription annotation, a typo in the trigger or service name, or the condition is false | Check the annotation and run trigger run |
| Delivery fails | Bad token, bot not in the channel, or the service is not defined in the ConfigMap | Check the controller logs and argocd_notifications_deliveries_total{succeeded="false"} |
| Duplicate messages | A flapping condition | Add oncePer |
| Messages link to the wrong host | argocdUrl is unset | Set notifications.argocdUrl or context.argocdUrl |