Skip to content

Too many notifications

The opposite of the previous problem: the inbox or Slack channel is overflowing — during a larger outage, with an unstable check, or just in normal operation. Five levers, which you can combine.

1. Check grouping

If 50 checks go critical at the same time, you want one summary message, not 50 individual mails. Grouping is on by default and configured per tenant (if it is switched off there, every message is sent individually): Tenants → expand tenant → Alert grouping. Three knobs:

  • Group by — host (default), host + severity or service name.
  • Wait time — after the first alert, collect this long, then send one message (default 30 s).
  • Minimum interval — an update for the same group at the earliest this long after the last message (default 5 min). If updates arrive too densely, this is the knob.

If several hosts fail at the same time, the same wait time bundles them into one "Hosts unreachable" message. When they come back, a start-up phase begins (default 30 min, Admin → Notifications → E-mail & Messages): problems that only start with the restart — event log errors, "Unexpected Shutdown", services still starting — are collected for that long. Whatever resolves by then is not reported; the rest arrives as one "Alert after restart" message for all returning hosts. Reminders for open problems, on the other hand, are configured per alert rule. Also check whether grouping was switched off for individual hosts (Manage host → Alert grouping) — their messages are sent individually.

Escalation stages are also grouped per tick by (tenant, stage, channel set) — if multiple services hit the same escalation stage at the same time, you get one summary message, not one mail per service.

Details: Alert rules → Grouping.

2. Raise the per-status wait time

Does a rule fire too fast on short blips (e.g. a 20-second network hiccup)? Raise the wait time for the affected status (Critical/Warning/No Data/Unknown) — a problem then has to persist continuously for longer before a notification is due at all. A short outlier that resolves before that no longer triggers a mail at all.

Details: Alert rules → Wait time per status.

3. Detect flapping instead of fighting it

If a check bounces very quickly between two statuses several times (flapping — e.g. a Wi-Fi access point that keeps dropping out briefly and coming back), there is no cascade of alert and recovery mails: Vesana sends "Flapping" once and later "Stable again". Clicking the flapping badge on the check shows whether a message goes out for it and to which channels.

If you still get many individual mails for an unstable check: if it reaches the rule's wait time in one stretch every time, it really failed several times from the rule's point of view. A longer wait time (point 2) or a look at the rule's repeat mode (point 4) helps then.

4. Check repeats and escalation

  • Repeat messages: the rule's mode under "Repeat messages for an open alert" decides whether an open problem reports again. The default is "Only when it gets worse"; "On every breach" and "Remind regularly" report additionally — no more often than the "No more often than" interval. If several rules match the same check, the chattiest mode applies.
  • Escalation: too many, too tightly spaced steps become a source themselves. Each step waits at least 1 minute after the first notification, and the delays must strictly increase. Check whether the first step really needs to inform the next circle after just a few minutes. Escalation pauses as soon as the alert is acknowledged and ends when the problem resolves — a timely ACK is often the fastest way to prevent further steps.

Details: Alert rules → Repeat messages, Alert rules → Escalation.

5. Maintenance windows instead of muting manually

If a planned reboot, a migration or a maintenance window is coming up where several checks are known to go red temporarily: create a downtime instead of enduring the mail flood afterwards or temporarily disabling rules. During an active downtime, alerts are not sent at all; if a problem persists after the window ends, you are informed then.

Presets (15 min up to 1 month, incl. "until end of day" and "until tomorrow 08:00"), several devices or the whole tenant in one window, and recurring windows make this quick for larger maintenance too.

Details: Downtimes.

Additionally: check the rule scope

Sometimes the actual cause is a rule that is too broad — e.g. scope "All" with all four statuses, matching far more checks than intended. The match preview in the rule editor ("Currently matches N checks", with examples) shows this while editing; individual hosts, collectors or checks are taken out via the exceptions.

Next