Skip to content

Too many notifications

The opposite of the previous problem: the inbox or Slack channel is overflowing — during a larger outage, with an unstable check, or just in normal operation. Five levers, which you can combine.

1. Check grouping

If 50 checks go critical at the same time, you want one summary message, not 50 individual mails. Grouping is on by default and configured per tenant: Tenants → expand tenant → Alert grouping. Three knobs:

  • Group by — host (default), host + severity or service name.
  • Wait time — after the first alert, collect this long, then send one message (default 30 s).
  • Minimum interval — an update for the same group at the earliest this long after the last message (default 5 min). If updates arrive too densely, this is the knob.

If several hosts fail at the same time, the same wait time bundles them into one "Hosts unreachable" message. Reminders for open problems, on the other hand, are configured per alert rule. Also check whether grouping was switched off for individual hosts (Manage host → Alert grouping) — their messages are sent individually.

Escalation stages are also grouped per tick by (tenant, stage, channel set) — if multiple services hit the same escalation stage at the same time, you get one summary message, not one mail per service.

Details: Alert rules → Grouping.

2. Raise the per-status wait time

Does a rule fire too fast on short blips (e.g. a 20-second network hiccup)? Raise the wait time for the affected status (Critical/Warning/No Data/Unknown) — a problem then has to persist continuously for longer before a notification is due at all. A short outlier that resolves before that no longer triggers a mail at all.

Details: Alert rules → Wait time per status.

3. Detect flapping instead of fighting it

If a service rapidly switches back and forth between two statuses multiple times (flapping — e.g. a Wi-Fi access point that keeps briefly dropping and coming back), that normally produces a cascade of alert and recovery mails. Vesana detects this pattern automatically and instead sends one "flapping" message, later a "flapping resolved" message — the normal alert/recovery cycle stays paused for the duration of the flapping.

If you still get many individual mails for an unstable service: check whether the check's interval is too long (flapping detection needs a certain number of status changes in a short time) or whether the rule reacts to a very narrow time window.

4. Limit escalation

A multi-stage escalation can itself become a source if too many stages are spaced too tightly:

escalation:
  - after: 0min
    channels: [ops-email]
  - after: 15min
    channels: [ops-push, manager-email]
  - after: 60min
    channels: [oncall-pager, ceo-sms]

Check whether the first stage really needs to fire immediately (after: 0min), or whether a small delay is already enough to absorb short-lived problems before the manager gets involved. Escalation stops automatically once the alert is acknowledged or resolves — a timely ACK is often the fastest way to prevent further escalation stages.

5. Maintenance window instead of manually muting

If a planned reboot, a migration, or a maintenance window is coming up where you know several checks will turn red temporarily: create a downtime instead of enduring the mail flood afterward or temporarily disabling rules. While a downtime is active, alerts aren't sent at all — not even delayed afterward.

Presets (1 h/4 h/8 h/24 h/until end of day/1 week) and bulk creation via tags make this fast even for larger maintenance windows.

Details: Downtimes.

Also: check the rule's scope

Sometimes the actual cause is a rule that's too broad — e.g. a rule without a host/tag filter that accidentally matches far more services than intended. Use the Test button in the rule editor before saving a new or changed rule — it shows which hosts/services would actually be affected.

Next