Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Prometheus evaluates alerting rules; Alertmanager routes, groups, suppresses, and delivers the resulting alerts. Following that path explains what happens before a notification arrives—and why a valid alert might not produce one. This is a conceptual walkthrough of the documented workflow, not a trace captured from a particular deployment.
1. Prometheus evaluates the alerting rule
An alert starts when a Prometheus alerting rule’s expression and any configured timing conditions indicate that its condition is active. Prometheus evaluates rules; Alertmanager handles notification management. Prometheus can send alert-state information to Alertmanager periodically. The Prometheus alerting rules documentation describes the division of labor: rules help determine what is broken, while Alertmanager manages notification dispatch. The broader alerting overview treats configuring Alertmanager, connecting Prometheus to it, and defining alerting rules as separate parts of setup.
2. Alertmanager matches the alert to a route
On arrival, an alert enters the configured top-level route, which must match all alerts. It then proceeds through matching child routes. A child route can match on alert labels; if it matches and continue is false, Alertmanager stops evaluating sibling routes. If continue is true, it can also consider later siblings. When no child matches, the alert is handled at the current route. Child routes inherit optional settings from their parent unless they override them. See the Alertmanager configuration reference for route behavior and version-sensitive configuration details.
In practice, route selection answers which handling path applies—not whether Prometheus detected the condition. When diagnosing a destination, check the alert’s labels against the route matchers and consider whether an earlier matching sibling stops evaluation.
#1 Best Overall
3. Grouping and timing determine notification cadence
Alertmanager can combine alerts of a similar nature into one notification. Grouping changes how alerts are presented together; it does not erase or disprove the underlying alert. Route settings control which labels form a group and when notifications are sent.
| Setting | What it controls | Documented default |
|---|---|---|
group_by |
Labels used to group alerts into notifications. | Not stated as a timing default in the configuration reference. |
group_wait |
Delay before the first notification for a new group, allowing related alerts or an inhibiting alert to arrive. | 30 seconds. |
group_interval |
Wait between notifications for updates to an existing group. | 5 minutes. |
repeat_interval |
Wait before repeating a notification for an alert that remains active. | 4 hours. |
These are defaults documented in the configuration reference, not universal recommendations. A longer group_wait can give related or inhibiting alerts time to arrive, but delays the initial page; if an alert resolves before the wait expires, Alertmanager sends no notification for it.
4. Silences and inhibition can suppress delivery
Before treating a missing notification as a routing problem, check whether a suppression rule applies:
- Silence: mutes alerts matching its configured criteria for a specified time.
- Inhibition: suppresses target alerts while a matching source alert exists and the configured equality labels agree.
Both are intentional Alertmanager behaviors. The Alertmanager concepts documentation explains aggregation, silencing, and inhibition; the configuration reference documents their configuration.
5. The selected receiver builds and sends the notification
A route selects a named receiver, whose configuration specifies one or more notification integrations. Alertmanager’s notification data can include the receiver name, status, alerts, group labels, common labels and annotations, route labels, and an Alertmanager backlink. Templates can display information about the group or iterate over individual alerts. Native integrations include destinations such as email, Slack, and webhooks; a webhook can connect Alertmanager to other notification mechanisms. Consult the notification template reference and integration reference for supported fields and integrations.
Tracing an alert is not the same as enabling distributed tracing
Alertmanager’s configuration reference describes distributed tracing controls, including client type (HTTP or gRPC), endpoint, sampling fraction, TLS or insecure behavior, custom HTTP headers, compression, and timeout. Those settings describe how tracing may be configured; they do not establish that a particular instance has tracing enabled, that a trace was sampled, or that a particular span corresponds to a specific alert. For deployment-specific conclusions, check the installed Alertmanager version and its actual configuration.
Rank #4
Alertmanager also documents an optional event recorder that can capture process start and shutdown, alert lifecycle transitions, silence creation, notification delivery, and mute or inhibition events, then send events to configured destinations. The current configuration reference says the recorder is gated by the event-recorder feature flag. It can provide a lifecycle timeline when enabled, but its availability in a specific installation must be verified. Both mechanisms are covered in the configuration reference; they are distinct from following the documented alert path conceptually.
High availability changes where Prometheus sends alerts
In a high-availability setup, Prometheus should send alerts to all Alertmanager instances rather than to a load balancer. This is a configuration consideration for the alert’s destination path, not a change to the rule, route, or receiver stages described above. See the Prometheus Authors’ high-availability guidance.
Best Value
A practical checklist for a missing notification
- Confirm the Prometheus alerting rule is active and that Prometheus is configured to communicate with Alertmanager.
- Check which Alertmanager instance receives the alert; in a high-availability deployment, verify Prometheus targets all instances rather than a load balancer.
- Inspect the alert labels and route tree: verify the intended child route matches and check whether
continuestops sibling evaluation. - Check the route’s grouping labels and timing settings, especially whether the alert resolved before
group_waitelapsed. - Look for an active matching silence or an inhibition rule whose source alert and equality labels apply.
- Verify the selected receiver and its integration configuration, then inspect the notification template fields it uses.
- If you need a deployment-specific timeline, verify whether distributed tracing or the event recorder is actually configured and enabled for the installed version.
For matcher syntax and other version-sensitive details, check the configuration reference against the Alertmanager version you run; the documentation notes UTF-8 matcher parser changes for Alertmanager 0.27 and later.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
