Skip to content

Scaling

Moved

The concrete sizing profiles (Small/Medium/Large/XL), the pipeline traffic light, and applying new worker/Postgres/Redis values now live under Admin → System → Resources → Performance — see Resources. This page remains as a conceptual guide: when is a bigger profile enough, when do you need more?

When to scale?

Signs that the current size no longer suffices:

  • The pipeline traffic light under Resources → Performance is yellow or red
  • The RAM distribution under Resources → RAM shows one service (usually the database) permanently close to the server limit
  • New check results visibly arrive delayed in the frontend

First reaction: pick a higher sizing profile (see Resources → Performance). That covers most cases without touching individual .env variables by hand.

Vertical vs. horizontal

Path Pro / con
Vertical — bigger sizing profile, more RAM/CPU on the server Simple (one click + container restart), no architecture change, eventually hardware-limited
Horizontal — multiple API containers, external database/Redis More complexity, but growth far beyond a single server

Recommendation: scale vertically first, up to the largest sensible server configuration (XL profile, 16 GB+ RAM). Only then think horizontally.

Externalizing (Postgres / Redis)

For very large setups it can make sense to take Postgres or Redis out of the compose stack and use a dedicated or managed service. Rough steps:

  1. Set up the external instance (same Postgres major version, TimescaleDB extension installed)
  2. Migrate the data (restore the built-in backup's dump into it)
  3. Switch the connection variable in .env to the external instance
  4. Remove the corresponding service from your own compose stack

This path isn't officially tested end-to-end — try it out on a test instance first, including a full restore drill.

Connections between nginx and API/receiver

Since October 2026 nginx keeps its connections to the API and the receiver open and reuses them for further requests (keepalive) instead of opening a new TCP connection per request. With many agents and collectors this saves noticeable CPU time. When a container is recreated (update, restart), nginx picks up its new address on its own within a few seconds.

Switching back (no downgrade, no change to the compose file):

cd /opt/vesana
docker compose -f docker-compose.prod.yml exec nginx touch /etc/nginx/ssl/vesana-upstream-keepalive.off
docker compose -f docker-compose.prod.yml restart nginx
# if enabled, likewise: agent-gateway, nginx-admin

nginx then opens a separate connection per request again, as before October 2026 — the container log shows Upstreams: direkt. To turn it back on, delete the file (rm instead of touch) and restart the containers. The file lives in the ssl volume shared by all nginx containers; it survives updates. This works on every installation.

With the compose file from October 2026 on, the .env works as an alternative: add VESANA_NGINX_KEEPALIVE=off and run docker compose up -d. On installations with an older compose file the variable never reaches the container — only the file switch above works there.

When to switch back: if occasional 502 responses appear after an update although the API and receiver are running, or if your own setup between nginx and the services (proxy, firewall) cuts idle connections.

If nginx does not start with the keepalive settings (for example a self-built image with nginx older than 1.27.3), the container falls back to the previous variant on its own and logs a warning.

Throughput improvements and their emergency switches

Since October 2026 Vesana processes more results per core. Each of these changes can be reverted to the previous behaviour without a downgrade and without editing .env — via SQL in the instance database:

INSERT INTO system_settings (key, value) VALUES ('<key>', '<value>')
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;
Key Value for "as before" What it does When to switch back
ingest_batch_transaction_enabled false The worker writes the results of one read batch in a single database transaction instead of one per request. Takes effect within one minute. If the diagnose log often shows result_batch_fallback or results arrive noticeably late.
worker_processes 1 Number of consumer processes in the worker container (default: one per 4 cores, at most 4). Takes effect on the next start of the worker container. If the server is short on memory or the database reports "too many clients".
receiver_auth_cache_enabled false The receiver remembers valid agent and collector credentials for up to 60 seconds instead of checking every request against the database. A revocation (token removed, host or tenant deleted or deactivated) still takes effect within one second. The switch takes effect within 30 seconds. If a revoked agent or collector keeps delivering results.
receiver_collector_rate_limit_by_results false A collector is limited by the number of its results (default 120,000 per minute) instead of by requests (120 per minute). Agents stay at 120 requests per minute. Takes effect within 30 seconds. If a single collector overloads the receiver.
receiver_collector_results_max number Maximum results per collector and minute when limiting by results. Takes effect within 30 seconds. To throttle collectors without switching the limit back entirely.
collector_check_concurrency 16 How many checks a collector (including the Active Collector) runs at the same time. Unset or 0 = by cores (16 below 4 cores, otherwise 16 per core, at most 256); at most 1024. Independently of this, at most 16 checks run at the same time per target device and for local scripts (api_script, plugins). Takes effect on the collector's next config fetch (Active Collector within seconds, otherwise within a minute), once the collector runs the version with this change. If target devices suffer from concurrent queries or the collector machine is short on memory.
anzeige_teilergebnisse_enabled false Home page, tenant overview, hosts page, map, sites and the tenant list compute their numbers from per-tenant intermediate results that all users and open tabs share. Every change (new result, ACK, downtime, new check, host deactivated …) invalidates the intermediate result of its tenant immediately — after your own action the page shows the new state right away, as before. When refreshing automatically, the hosts page and the map only ask whether anything changed instead of loading all devices every time. Switched off, every request computes from scratch again and the hosts page reloads all devices on every refresh. Takes effect immediately, no restart. If an overview shows a different number than the error overview or the host detail, if a change on the hosts page only appears after reloading, or if the diagnose log shows status_teilergebnis_abweichung.
anzeige_teilergebnisse_max_age_seconds number, 1–30 (default 5) An intermediate result is valid for at most this many seconds even if nothing changed — a safety bound, not a refresh interval. Rarely needed; lower it if, after status_teilergebnis_abweichung, every number should be as fresh as possible until the cause is fixed (or switch off the key above).

Independently of that, the worker now only removes results from the queue that it has already processed. After a worker restart or outage, waiting results are no longer lost. An entry that keeps failing for more than ten minutes is moved to the dead letters, visibly, instead of blocking the queue.

Background checks with many checks

Since October 2026 the background checks — overdue checks ("No Data"), maintenance windows, device reachability, agents offline, replayed alerts and escalations — only look at the affected checks instead of all of them on every run. The result is the same. Vesana verifies this itself: on the first run after a start and hourly afterwards it also computes the previous way. If anything differs, that run uses the previous result and writes watcher_abgleich_abweichung to the diagnose log.

Each group can be switched back on its own (same SQL form as above, value false, takes effect on the next run, i.e. within one minute):

Key Affects
watcher_index_stale_enabled Checks without new results become "No Data"
watcher_index_downtime_enabled Maintenance windows start and end (expiry of time-limited ACKs is independent of it)
watcher_index_reachability_enabled Device reachability (ping hints, agent heartbeat)
watcher_index_dead_agent_enabled Agents that stopped reporting
watcher_index_suppressed_replay_enabled Suppressed alerts that are replayed
watcher_index_escalation_backfill_enabled Escalations for alerts that already existed before a restart

When to switch back: if the diagnose log shows watcher_abgleich_abweichung (the entry names the group) or a check is visibly wrong — it stays green although no results arrive, or stays in "Downtime" after its maintenance window ended. Set that group's switch to false and report the case via the feedback button.

Check right away, without changing anything:

docker compose exec api python -m api.app.cli.watcher_abgleich

The output states per group whether the previous and the new way agree; on a difference the command exits with code 1.

Three more diagnose-log entries belong to this: downtime_flag_reconciled (a check was in "Downtime" without a running maintenance window and was reset), stale_eingaben_repaired (a check's stored "No Data" threshold no longer matched its interval and was corrected) and watcher_takt_ueber_sperre (a run took longer than intended — a sign that the database is reaching its limit).

When updating to this version the database creates four small indexes, two columns and a small table and fills the columns once; with very many checks the update takes a few seconds longer.

Configuration only when it changed

Agents, collectors and the Active Collector fetch their configuration from the server at fixed intervals (agent every 5 minutes, collector every minute, Active Collector every 5 seconds). Until now the server rebuilt it completely every time. Since October 2026 it answers "unchanged" as long as nothing changed for that agent or collector — jobs (network scans, network discovery, "Check now") still come with every answer. A change to a check, device, profile, script or to the permissions still reaches the agent or collector with its next fetch; a change to the agent's own device reaches it with its next heartbeat already (within 20 seconds).

This applies to agents and collectors running the new version only — older ones keep receiving the full configuration as before.

As a safety net every agent and collector fetches its full configuration once an hour. If it differs from what it had although the server said "unchanged", the diagnose log shows a config_drift entry (naming the affected checks or devices) and the admin overview names the reason. The fresh configuration applies immediately from then on.

Key Value for "as before" What it does When to switch back
config_delta_enabled false Switched off, the server answers every fetch with the full configuration again. Takes effect within 5 seconds, no restart. If the diagnose log shows config_drift or config_version_failed, or a change does not reach an agent or collector.

Agent bundles unchanged results

Until now an agent sent its results every 5 seconds whenever one was pending — with many agents the largest share of all requests to the server. Since October 2026 it still sends everything that means something right away (at the usual 5-second pace): every result that is not OK, every status change, the first result of a check, "Collect now", checks with a short interval and checks whose status the server judges itself. Only unchanged OK results wait for up to 30 seconds and travel together. Alerts and recoveries therefore do not arrive later; an unchanged value shows up in graphs and lists at most 30 seconds later (in the history at its measurement time).

In this mode an agent sends logs every 30 seconds instead of every 5 seconds. When you open the live view of a device's logs, its agent goes back to the 5-second pace while the view is open — the first lines arrive within 30 seconds of opening it, after that as usual.

This applies to agents running the new version only; older ones keep sending as before. If a status change nevertheless arrives bundled (later than 10 seconds after the measurement), the diagnose log shows a result_change_late entry. A single one right after vesana-agent test on the device is explained (the test run sends past the running instance).

Key Value for "as before" What it does When to switch back
result_batching_window_s 0 How long an unchanged OK result waits at most (default 30, at most 60). 0 = every result at the 5-second pace as before. Takes effect with the agent's next config fetch (within 5 minutes; immediately with "Collect now"), no restart. If the diagnose log repeatedly shows result_change_late or values appear noticeably delayed.
agent_log_flush_s 0 Pace at which agents send logs (default 30 seconds, at least 5). 0 = the pace of the log configuration as before (5 seconds). Takes effect with the agent's next config fetch. If logs appear too late without the live view, or the live view does not speed up.
### Heartbeats of agents and collectors

Agents report to the server every 20 seconds, collectors every 30 seconds ("heartbeat"). Until now every heartbeat wrote several rows to the database. Since October 2026 the server keeps the heartbeat in its fast cache (Redis) and writes it to the database in one batch every 15 seconds. Nothing changes for you: "last seen", offline notifications, host-down alerts and the availability bars show the same as before. The only new thing: an ongoing outage of an agent or collector appears in the availability bar after a little more than five minutes, not up to an hour later.

If Redis is unreachable or was emptied (for example after a restart without saved data), the server temporarily writes the heartbeats straight to the database again — no agent or collector is considered offline because of it. The diagnose log then shows edge_liveness_redis_unavailable or edge_liveness_restored.

Once an hour the server additionally recomputes the outages the previous way and compares both results. If anything differs, the diagnose log shows edge_outages_schatten_abweichung and the admin overview names the reason. If syncing with the database stalls for more than two minutes, it reports edge_liveness_mirror_stale (a failed sync run: edge_liveness_mirror_failed) — heartbeats are then written one by one again until it recovers, nothing is lost.

Collectors from this version on follow the heartbeat interval set by the server (30 seconds today); older ones stay at 30 seconds.

Key Value for "as before" What it does When to switch back
edge_liveness_redis false Switched off, every heartbeat is written straight to the database again and outages are determined hourly again. Takes effect within 10 seconds, no restart; switching off and on again creates no false outages. If the diagnose log shows edge_outages_schatten_abweichung or edge_liveness_mirror_stale, if agents or collectors are wrongly considered offline, or if Redis is permanently short on memory.

Several Active Collectors

When the cores of one Active Collector are no longer enough, several share the active checks — each check belongs to exactly one, and if one fails the others take over after 2 minutes. How to add one and what the primary does in addition: Active Collector.

Key Value for "as before" What it does When to switch back
active_collector_sharing_enabled false Off: only the primary Active Collector runs checks, additional ones get none. Takes effect within seconds. When checks do not work on another machine or the diagnose log shows ac_result_not_responsible.
collector_native_ping_enabled false Ping checks of all collectors use an ICMP socket instead of the ping program. Off: back to the program. Takes effect with the next config fetch. When ping values look different after the update.

Several API processes and the background work

Since October 2026 the web API uses several processes when the machine allows it: one per 4 cores and per 4 GB of memory, at most 8. Below 8 cores or below 8 GB it stays at one process as before. The API's database connections (DB_POOL_SIZE + DB_MAX_OVERFLOW, 30 by default) stay the same in total: the processes share them, and the process running the background work gets a reserve of 10 out of them.

Everything that runs in the background — alert promotion, escalation, outage detection, maintenance, report schedules, status pages, backups — is still run by exactly one process, the "leader". The processes agree on it via Redis; if the leader fails, another takes over within about 17 seconds, on an orderly restart immediately. This also holds with the compose file of an older version and in split-container mode (the admin container never runs background work).

The admin overview shows whether background work is running in the Background work tile: which process is the leader, how many API processes are running and whether a task is stalled. If a task stalls for longer than its usual pace (for example alert promotion for more than two minutes) or no process leads at all, the overview names the reason. The diagnose log (category maintenance) then shows hintergrund_stillstand, hintergrund_ohne_leiter, leiter_verloren or hintergrund_aufgabe_abgestuerzt; a crashed task restarts by itself after one minute.

Login, 2FA and step-up limits count in Redis for all processes together. If Redis is unreachable, the same limits apply in the memory of each process (diagnose log rate_limit_speicher_ausgefallen) — the protection stays.

Key Value for "as before" What it does When to switch back
api_processes 1 Number of API processes. Not set or auto = by cores and memory as above. 1 means exactly one process without leader election; background work starts immediately as before this version. A number from 2 forces that many processes. Takes effect on the next start of the API container (docker compose restart api). With the compose file from this version on, the same works via API_WORKERS=1 in the .env and docker compose up -d (an older compose file does not pass the variable into the container). If the admin overview reports hintergrund_ohne_leiter or repeatedly leiter_verloren, if the server is short on memory, or if the database reports "too many clients".

Next