Skip to content

Collector overloaded — what to do

A collector is a small Linux VM that runs SNMP/SSH/Ping/HTTP checks in parallel. With too many hosts, too short intervals or an unlucky check mix it can no longer keep up: check results arrive late, some don't arrive at all.

This page describes the symptoms, how to diagnose them and which remedies make sense in which order. The same applies to the Active Collector; in addition, its admin page shows "Cannot keep up" and you can add more active collectors.

How much runs in parallel

The collector runs its checks in the background; config fetch, newly due checks and jobs do not wait for the slowest check. The number of concurrent checks follows the VM's cores: up to 3 cores 16, above that 16 per core, at most 256. A single target never gets more than 16 concurrent checks, script and plugin checks (one process each) run at most 16 at a time. More cores therefore help an overloaded collector directly.

Every check has a hard deadline (twice its interval, at least 30 s, at most 90 s; internet line 120 s). If it expires, the check reports UNKNOWN and the collector carries on.

Symptoms

Symptom Likely cause
Processing delay (pipeline) notice in the UI Worker backlog on the server (see Scaling) — rarely the collector
Many checks of one collector on "No Data" at once Collector offline or overloaded
The collector's "last seen" older than ~1 minute Collector has connection problems or hangs (heartbeat every 30 s, offline after 3 minutes)
Discovery scans run for a long time Large subnet or slow targets (deadline per scan 15 minutes)
High CPU or memory on the collector VM Too many concurrent walks/SSH sessions
Log often shows check abandoned (deadline exceeded) Checks run into their deadline — slow targets or a busy collector

Step 1 — Diagnose on the collector VM

# Service status
systemctl status vesana-collector

# Live log
journalctl -u vesana-collector -f

# CPU/memory
top -p $(pgrep -d',' vesana-collector)

Useful log lines:

  • running checks hosts=… due=… and checks complete count=… — how many checks are due per cycle and how many complete.
  • check abandoned (deadline exceeded) — a check exceeded its deadline.
  • failed to send results / send attempt failed — results do not reach the server (network, certificate, server load).
  • watchdog: no check progress for too long, exiting for restart — a real hang; systemd restarts the collector.

On the Vesana server (psql):

-- Heartbeat lag
SELECT id, name, last_seen_at, NOW() - last_seen_at AS lag
  FROM collectors WHERE deleted_at IS NULL ORDER BY lag DESC;

-- How many hosts are attached to which collector
SELECT c.name, COUNT(h.id) AS hosts
  FROM collectors c LEFT JOIN hosts h ON h.collector_id = c.id AND h.deleted_at IS NULL
  WHERE c.deleted_at IS NULL GROUP BY c.id, c.name ORDER BY hosts DESC;

Step 2 — Remedies, from simple to complex

A) Raise check intervals (immediate relief)

Double the interval of non-time-critical checks:

  • Raise the interval on the profile check (e.g. 60 s → 180 s) — applies to all hosts with that profile.
  • For a single host: edit the check → Interval.

Rule of thumb: SNMP walks every 5 min are enough for most switches. Disk usage every 2 min is enough. Ping every 60 s — shorter rarely helps.

B) Thin out profile checks

If a profile adds 30 checks per host, that multiplies: 100 hosts with that profile are 3,000 checks.

  • Profiles → open the profile — check which checks are really needed.
  • Stop adding rarely needed checks automatically on the profile; deactivate existing checks of individual hosts in the host detail view.
  • Point walk checks at the tables/OIDs you actually want instead of querying whole trees.

C) Tame discovery

Discovery scans put a heavy temporary load on the collector.

  • Schedule automatic scans less often, or only scan manually when something changes.
  • Keep subnets that should not be scanned out permanently with Exclude subnet in Discovery.
  • Several collectors: each scans its own network instead of everything.

D) Split hosts onto a second collector

With very many hosts per collector or a very mixed check mix, a second collector is worthwhile.

  1. Create a new collector: Tenants → tenant → Collectors → New Collector.
  2. Install it on the desired VM (see Collector).
  3. Move hosts: Host detail → Manage host → Connection → Collector (passive checks). The assignment is per host.

Splitting heuristics:

  • By tenant — every customer has their own collectors anyway (a collector belongs to exactly one tenant).
  • By subnet — collector A for 10.10.0.0/16, collector B for 10.20.0.0/16.
  • By check mix — one collector for SNMP-heavy switches, one for servers (Ping/SSH/HTTP).

E) Upgrade the collector VM

Give it more resources if CPU or memory usage stays high. From 4 cores, parallelism grows automatically (see above).

Hosts per collector Recommended VM
< 50 1 vCPU, 1 GB RAM
50–200 2 vCPU, 2 GB RAM
200–500 4 vCPU, 4 GB RAM
> 500 better add a second collector

These are guidelines; the check mix (many SSH and walk checks) matters more than the host count.

F) Slow targets

Check timeouts are built in and cannot be set per check: SNMP 5 s with one retry (walks 8 s with two retries), SSH connection setup 10 s. An HTTP check has its own timeout field (default 10 s), as does a plugin check (default 30 s). If a target is constantly at the limit, the device or the network is usually the problem — update the firmware or query less.

G) Server-side emergency switch

The parallelism can be fixed for all collectors (collector_check_concurrency in system_settings, 16 = behaviour of earlier versions). Normally not needed.

When it's not the collector

  • Pipeline delay but the collector is idle → server side: too few workers. See Scaling.
  • A single host hangs → the target device answers slowly (often the SNMP daemon). Update the firmware or query fewer OIDs.
  • Discovery scan runs long → normal for large subnets (/16 = 65,536 addresses). Restrict to smaller subnets.

Monitoring the collector itself

  1. Create a host for the collector VM (IP of the VM, a Linux profile).
  2. Easiest: install the Vesana agent on the collector VM — it delivers CPU, memory and disk without extra effort.
  3. Without an agent: check the VM from another collector or the Active Collector — if it checks itself, nobody notices when it's gone.

If a collector fails, Vesana reports it anyway as "Collector offline" with the number of unobserved hosts and checks.