Skip to content

Status & state model

The five status values (plus PENDING as a display case)

Status Code Color Meaning
OK 0 green Everything within thresholds
WARNING 1 yellow Warn threshold crossed, not yet critical
CRITICAL 2 red Crit threshold crossed or check failed
NO_DATA 3 orange No data within expected interval
UNKNOWN 4 gray Check ran, result not interpretable (e.g. plugin exit code 3)
PENDING – blue "Awaiting data" — special case of NO_DATA, see below; counts as its own blue share in the health score, never as a problem
Info – teal Display status for checks in info mode: the real status stays stored and is visible on click in the check detail ("Real status"); lists, counters and filters treat Info as its own category

NO_DATA and UNKNOWN are often confused. Rule of thumb:

  • NO_DATA = we never heard the check (agent offline, collector dead, brand-new service)
  • UNKNOWN = the check did run, but the result isn't unambiguous (parser failure, missing OID on SNMP device)

PENDING — "Awaiting data"

PENDING isn't its own database status — it's a read-time derivation of NO_DATA: a freshly created (or reactivated/restored) service shows blue PENDING instead of orange NO_DATA at first, as long as last_check_at is still NULL, i.e. no result has ever arrived. There is no time window: PENDING only ends through an event — the first real result, or a verdict by the system (an offline agent or collector sets NO_DATA with a timestamp). A check whose agent is installed twenty minutes later stays blue until then, not orange.

Rationale: a service that was just created isn't an outage — it's simply waiting for its first result. Every creation path (create host, add check, restore from trash, reactivate service) sets last_state_change_at for this reason; otherwise the PENDING derivation doesn't kick in and the service would incorrectly show orange NO_DATA.

PENDING only affects the UI display — the underlying severity order and the alerting_state remain NO_DATA.

Severity order

Important for aggregation (tenant status, host status, alert groups):

CRITICAL > WARNING > NO_DATA > UNKNOWN > OK

If a host has 3 services — OK, NO_DATA, WARNING — its rolled-up status is WARNING, not NO_DATA. A concrete problem (WARNING) is more relevant than a data gap (NO_DATA).

Three-layer state model (since v0.48)

Vesana separates operator visibility from notification trigger into two layers. The old soft/hard model with max_check_attempts and retry_interval_seconds was fully removed in migration 147.

Layer Column For Latency
UI state current_status.ui_state What the operator sees in the error overview ~15-30 s
Alerting state current_status.alerting_state What drives notifications/SLA sustained ≥ alert_after_seconds (60-120 s)

UI-state values (ui_state_kind enum)

Value Meaning
ok Check is green
probing First failure, confirmation running (confirmation retry)
degraded Failure confirmed — but no push notification yet, because the sustained threshold hasn't been reached
alerting Failure sustained → push goes out, operator alerted
no_data No check result received (agent/collector offline, new service)
unknown Check ran, result not interpretable

Alerting-state values (alerting_state_kind enum)

Only two values — ok or alerting. This field alone drives notifications + status pages + SLA. degraded triggers no alert; only the promotion to alerting after the sustained threshold does.

Phase sequence

stateDiagram-v2
    [*] --> ok: Check ok
    ok --> probing: Check critical (first contact)
    probing --> ok: Recovery (never alerted)
    probing --> degraded: Confirmation confirmed (1 retry default)
    degraded --> ok: Recovery before sustained threshold (never alerted)
    degraded --> alerting: Sustained >= 60s (CRITICAL) or 120s (WARNING)
    alerting --> ok: Recovery (recovery notification goes out)
Phase Concretely
ok → probing first failing response from the check
probing → degraded after confirmation_attempts × confirmation_interval_seconds (default 1×10 s)
degraded → alerting after alert_after_seconds sustained — default 60 s for CRITICAL, 120 s for WARNING
alerting → ok check returns green again, recovery notification fires

Active-mode fast poll

During probing, the worker scheduler sets the next next_check_at to confirmation_interval_seconds (default 10 s); during degraded to recovery_poll_interval_seconds (default 15 s). This confirms and recovers active checks faster. Passive/agent checks still depend on a heartbeat push.

Acknowledgement during degraded

If the operator sets an ACK during degraded, it blocks the promotion to alerting — no push spam for something the operator has already seen. Recovery clears the ACK automatically.

Long check intervals

At intervals > 2× the sustained threshold (e.g. a 5 min active check with a 60 s threshold), the promotion is based on stale data — the service may already have recovered without us knowing yet. Detection latency = interval_seconds + confirmation_interval. Rule of thumb: don't set the main interval much larger than the sustained threshold.

Reading the pills in the error overview

The error overview shows a composite pill next to each service — on the left the status (CRITICAL/WARNING/NO_DATA/UNKNOWN), on the right the UI state (PROBING/DEGRADED/ALERT). Together they tell you at a glance: what's the problem? And: where is it in the escalation path?

Pill Visual Meaning
CRITICAL + PROBING gray pulsing First contact, confirmation running — no alert planned
CRITICAL + DEGRADED red pulsing Failure confirmed, no push yet — live countdown to alerting visible
CRITICAL + ALERT red solid + bell Push sent, operator alerted
WARNING + DEGRADED yellow pulsing Warn threshold confirmed, sustained running (default 120 s)
WARNING + ALERT yellow solid + bell Warn push sent
NO_DATA orange No check result — agent or collector offline / service new
UNKNOWN gray Check ran, result not interpretable

What you can conclude

  • Pill pulses gray (PROBING): nothing to do yet, wait — confirmation is running.
  • Pill pulses red/yellow (DEGRADED): look now. If you're on it, press ACK → the promotion to ALERT (and the push to all channels) is stopped.
  • Pill solid + bell (ALERT): the push already went out. Other operators know. ACK anyway if you're handling it.

What the small indicators mean

  • Pulsing = state is "in motion", either confirmation or sustained is running.
  • Bell = at least one notification went out.
  • Backoff (5 / 15 / 30 min) on long-down services — up to 10 min recovery lag, because we don't poll dead endpoints permanently.

When is each status set?

Creating a new service

New host_service → initial status NO_DATA, not UNKNOWN. This makes it explicit: "awaiting first result" rather than "check is broken".

On incoming check result

Worker translates the plugin result into a status:

  • Exit code 0, value below threshold_warn → OK
  • Value ≥ threshold_warn and < threshold_crit → WARNING
  • Value ≥ threshold_crit or exit code 2 → CRITICAL
  • Exit code 3 / unparseable result → UNKNOWN

When data is missing

Watcher loops set NO_DATA:

Watcher Trigger
dead_agent_watcher Agent silent for > agent_dead_after (default 3 × heartbeat interval, ~3 min)
dead_collector_watcher Collector silent for > 3 min
service_overdue_watcher Service result older than interval_seconds × 1.5

Once a result arrives, the status flips to the actual outcome.

Configuration per profile-check

Field Meaning Default
interval_seconds Main interval 60 s
confirmation_attempts Confirmation retries in probing before degraded 1
confirmation_interval_seconds Gap between confirmation retries 10 s
recovery_poll_interval_seconds Active poll interval in degraded (recovery) 15 s
recovery_poll_window_seconds Window for recovery fast poll 300 s

The sustained thresholds (alert_after_critical_seconds, alert_after_warning_seconds) live in system_settings and default globally to 60 s / 120 s.

At the host-service level there are override fields (confirmation_attempts_override, confirmation_interval_override, recovery_poll_interval_override, recovery_poll_window_override) — resolved via COALESCE.

Acknowledged — "in progress" vs. accepted exception

A service can be acknowledged (ACK) — the operator has seen it, further notifications are suppressed:

alerting   + ACK → no further notifications (inhibition)
degraded   + ACK → promotion to alerting blocked (no push at first)
+ recovery       → status OK, ACK auto-cleared

ACK is ONE mechanism with two flavors, chosen via the classification in the ACK dialog — and named the same everywhere (pills, dashboard, filters, tooltips):

Classification Meaning Counts as a problem?
ACK (marked as error) — "in progress" (default) Operator is handling it, but it remains a real problem Yes — stays visible in rollups/dashboards
ACK (marked as no error) — accepted exception Status stays red/true, but is deliberately not counted as a problem (e.g. known false alarm, permanently tolerated behavior) No — suppressed

An accepted exception can carry an expiry (acknowledged_until) — after which it automatically falls back to "open". If an accepted check gets worse than at ACK time (e.g. WARNING → CRITICAL), the marker breaks automatically — the check becomes a real, visible problem again.

Suppression rule

A check counts in no problem rollup (health %, host/tenant color, problem counter, bell) if it's suppressed:

suppressed ⟺ in downtime OR (accepted AND NOT "is a problem") OR (info mode AND status ≠ OK)

"In progress" (acked, but "is a problem" stays on) still counts as a problem — ACK alone hides nothing, only the deliberate exception does.

The error overview deliberately does NOT apply this rule — there you see everything on purpose: open, in-progress, and accepted problems side by side, with filtering available. The only exception: info-mode checks are hidden there by default, but can be shown.

Info mode (permanently configured vs. state-bound accepted)

A check can additionally be set to permanent info mode — pure statistics, the real status stays visible, but it never reaches alerting_state='alerting' (no notifications, escalation, or inhibition). The difference from an accepted exception: info mode is a permanent configuration on the check, while an accepted exception is state-bound (breaks on worsening, ends on recovery).

Details: Alerting → Acknowledgements.

Downtime

During an active downtime the check runs as normal, but alert rules see it as "in maintenance". Notifications aren't sent, the status appears in the UI with a maintenance badge.

Maintenance != ACK: ACK is "I've seen it", downtime is "this is expected".

Details: Alerting → Downtimes.

Aggregated status

Host status is the worst severity of its services (with inhibition applied). Tenant status is the worst severity of its hosts.

Inhibition can reduce the effective status: if the parent host is alerting, its dependent services appear grayed out in the UI and don't count toward tenant status. Inhibition only applies from parent alerting, not already from parent degraded — otherwise child alerts would be silently suppressed during brief parent glitches.

Details: Alerting → Dependencies & inhibition.

Status pages

Public status pages hardcode a filter on alerting_state='alerting'. That way the visitor never sees "server briefly red then green" — only problems the operator really needs to see.

Next