Skip to content

Alert rules

An alert rule defines when and to whom a notification is sent. Without a matching, enabled rule Vesana never notifies — a red check on its own sends nothing.

Rules live under Alerting → Rules (sidebar: Configuration → Alerting). The list can be searched, filtered by tenant and enabled/disabled, and paged; every row has an on/off toggle plus Test, Edit and Delete.

Creating a rule

Alerting → Rules → New Rule. The editor has two columns: "when and where" on the left, "who gets notified" on the right.

Name and tenant

The name appears in messages and in the alarm timeline. The tenant determines which hosts and channels can be selected — channels can only be assigned from the same tenant.

Status filter

Four chips: Critical · Warning · No Data · Unknown. The rule reports only the selected statuses; at least one must be selected. New rules start with Critical and Unknown.

Per-status timing (the send gate)

Section "Notify once the problem persists for". A problem is reported only after it has existed continuously for this long.

Field Meaning
Critical wait time in seconds for CRITICAL (default 60)
Warning wait time for WARNING (default 120)
No Data blank = "like Critical"
Unknown blank = "like Critical"

0 explicitly means "immediately" — the message goes out without a batching window. The quick picks (immediately · 30 s · 1 min · 5 min · 15 min) set Critical and Warning together.

If the problem clears before the wait time elapses, the rule sends nothing at all for that episode — not late, but never. That is intended: an outlier that fixes itself should not alarm anyone. The display in the error overview and host detail reacts immediately regardless, see Alerting overview.

Every rule evaluates its own wait time. If several rules match the same check, each notifies its own channels after its own time.

New or reactivated rules don't blast retroactively

If you create a rule, or re-enable a disabled one, while a problem is already running somewhere, the rule only applies to problems that start afterwards. A problem already running stays silent for this rule until it has recovered once and reappears — escalation does not start for it either. Under "Active alarms" this shows as "never sent — the problem was already running when the rule was armed".

When does a check turn NO_DATA?

A check that was last OK is treated as NO_DATA after roughly 3× its interval without a new result (5-minute check: after about 15 minutes of silence). A check that was already WARNING or CRITICAL gets more tolerance — NO_DATA only after 10× its interval or 30 minutes, whichever is longer. That keeps an already broken check from bouncing between CRITICAL and NO_DATA on every small delay.

For agent hosts whose agent stops reporting, the host additionally counts as unreachable — then one host message is sent instead of one per check, see If-then reference.

Scope

Mode Meaning
All all checks of the tenant
Specific Hosts pick from the host list (searchable)
Specific Services pick by check (profile check); applies on every host that has it
Host + Services explicit host → check assignments, with a search box across hosts and checks

Exceptions

Under "Configure exceptions" you exclude individual hosts or individual checks — in every scope mode, e.g. "everything in the tenant, except this host". An entire host as exception also mutes this rule's "Host unreachable" message; single check exceptions deliberately let it through. The search also finds individual checks and expands their host; the section can be collapsed, the counter stays visible.

Match preview

While you change the scope, the editor shows live "Currently matches N checks" with examples and the number of checks excluded by exception. At 0 the editor warns: this rule will never notify. From v1.9.437 invisible agent telemetry checks are no longer counted.

Anomaly detection

Only visible when anomaly detection is enabled instance-wide: "Anomaly Notifications" (unusual metrics) and "Capacity Warnings" (disk full in under 7 days). Details: Anomaly detection.

First notification

The channels that receive the first message once the wait time has elapsed. Add channels via the "+ channel name" pills; only active channels of the tenant are offered. Channel configuration: Notification channels.

Per-channel fine-tuning

Per channel and per step you define right in the rule what goes through it:

  • Status chips — which of the rule's selected statuses this channel receives (e.g. push only on Critical, email on everything). A grey chip is deselected; the last status cannot be deselected — remove the channel instead.
  • Recovery — whether this channel gets the "resolved" message.

If no channel in any step receives the recovery, the editor warns: this rule reports problems only. Channel-level filters no longer exist — what a channel receives is decided by the rule alone.

Escalation

Delayed steps for "the operator doesn't react, then the manager, then the team". Per step: Wait after the first notification (presets 5 min · 15 min · 30 min · 1 h · 2 h · 4 h or Custom, to the minute, at least 1 min) plus its own channels with the same fine-tuning. Steps need at least one channel and strictly increasing delays.

Steps count from the first notification — not from the start of the problem if its wait time was still running. When the alert is acknowledged, escalation pauses; when the problem is resolved, it ends. If several checks hit the same step at once, one bundled message goes out per step and channel set.

A server restart loses no escalation: running alerts continue from where they were.

The preview line under the steps summarises the schedule: "After 60 s to ops-mail → 30 min later to ops-push".

Repeat messages

Section "Repeat messages for an open alert" — what happens while the problem stays open:

Mode Behaviour
Only when it gets worse (default) Re-notify only when the severity rises above the last reported one or another check joins. A value oscillating around its threshold stays quiet.
On every breach Additionally every time the threshold is crossed again — no sooner than the configured gap.
Remind regularly Additionally at a fixed interval, even when nothing changes.

The latter two modes share one frequency "No more often than" (presets 5 min to 1 d or Custom; pre-filled 30 min). Reminders are therefore opt-in per rule; there is no tenant-wide repeat interval any more. If several rules match the same check, the chattiest mode wins.

Grouping

When 50 checks fail at once you want one bundled message, not 50 single mails. Grouping is not configured per rule but per tenant under Tenants → expand tenant → Alert grouping. There are exactly three knobs:

Field Meaning
Group by Host (all alerts of a host in one message, default) · Host + Severity (like host, but separated by status) · Service name (same check name across all hosts; caution: if one host with many checks fails, several messages result — one per check name)
Wait time After the first alert, wait this long for more, then send one message with everything so far (default 30 s, 0 = immediately)
Minimum interval An update for the same group arrives at the earliest this long after the last message — more checks, higher severity or a partial recovery (default 5 min, at least 1 min)

Below the controls an example sentence translates the three values into what the recipient will actually experience.

What grouping does beyond that:

  • Several hosts unreachable at the same time (site without power, collector network gone): all hosts that fail within the wait time appear in one message "Hosts unreachable: srv01, srv02, srv03" — and one recovery when they come back. A single host still gets its familiar individual message.
  • Bundled recoveries: if several bundles recover shortly after one another, one bundled recovery is sent within the wait time.
  • Rules with wait time 0 send their first message immediately but remain part of the bundle — whatever joins afterwards follows as an update.
  • Grouping can be switched off per host (Manage host → Alert grouping): its messages are then sent individually, without a collection window.

Reminders for problems that stay open do not belong here but to the rule (see above). How bundles, updates and reminders interact in detail: If-then reference.

Enabling and disabling

The toggle sits in the rule list (and in the Alerting overview). A disabled rule is really silent: no alerts, no escalation steps and no recoveries any more — not even for problems that were already running before it was disabled. If another enabled rule hits the same channel, that rule takes precedence. On re-enabling, the rule applies only to problems that start afterwards (see above).

A deleted rule, by contrast, still delivers open recoveries so that no alert is left "hanging".

Test and probe alert

Two different things:

  • Test (in the rule row): sends a message marked as test directly to the rule's active channels — without wait time, suppression or delivery journal. Only checks whether the channels are reachable.
  • Probe alert (button above the list): pick a host and send an alert marked TEST ALERT through the real chain — rule matching, suppression, delivery journal, channels. The result shows status and error per channel. If the table stays empty, no enabled rule matches this host or a suppression applies — details in the diagnostic log, category "Suppressed".

Recoveries

A recovery goes to exactly the channels that were actually notified for the alert — even if you changed or deleted the rule afterwards. Nobody gets a recovery for an alert they never saw, and nobody misses out just because the rule looks different now. Which channels receive recoveries at all is set in the per-channel fine-tuning.

Worsening and flapping

If an already reported check jumps from WARNING to CRITICAL, the rule reports again with the higher severity — in every repeat mode. The yardstick is the last reported severity: a value oscillating around its threshold (CRITICAL → WARNING → CRITICAL) does not produce a new alert mail.

If a check bounces between two statuses very quickly several times (flapping), there is no cascade of alert and recovery: Vesana sends "Flapping" once and later "Stable again". Clicking the flapping badge on the check shows whether a message goes out for it (with channels) — or why not: downtime, ACK, info mode, no matching rule, status filter or no active channel.

Permissions

Viewing rules: alert_rules.view (Operator and Viewer). Creating, editing (incl. test and probe alert) and deleting: alert_rules.create, alert_rules.edit, alert_rules.delete — in the built-in roles from Admin upwards.

Audit

Rule changes land in the audit log with a diff — who set the wait time from 300 to 60 seconds?

Next