Skip to content

Storage keeps growing

The database or the server's disk is filling up faster than expected. Almost always the cause is a combination of retention settings, an overly chatty check, or enabled log collection — rarely an actual leak.

1. Look before you act

Admin → Maintenance → Resources → "Storage" subtab is the central place for everything around disk and DB usage:

  • Usage — how much of the partition is used (warning from 85 %)
  • Growth per day — after compression and raw; a large gap between the two shows that compression is working, but the raw data rate itself is high
  • Data class table — per hypertable, size, compression state, row count, oldest data point
  • "Where the space goes" — Docker breakdown (images, containers, volumes, build cache) including the immediately reclaimable share
  • "Biggest data producers" — which checks wrote the most in the last 24 hours

Details: Resources.

2. Identify the biggest data producer

Before touching retention, it's worth looking at "biggest data producers" — often a single check is the cause: an interval that's too short, a check that writes many values at once (e.g. an SNMP walk with many sensors), or a newly created check without a sensible interval. The fix is usually simpler than a retention change: raise that check's interval.

3. Logs as a common driver

Log collection (agent_script/file tailing/journald/Windows Event Log via the agent) can noticeably add to storage size at a high log rate — much more than plain metric checks. Check:

  • Is log collection running on hosts where it isn't really needed?
  • Is the retention for logs (default: compress after 3 days, keep 30 days) sensible for your volume, or too generous?
  • Is the log source unnecessarily pulling in chatty sources (e.g. debug-level logs nobody evaluates)?

Details: Logs.

4. Check and adjust retention

Configurable per data class under Admin → Maintenance → Resources → Storage:

Data class Default compress after Default keep
Metrics (check results, detail) 3 days 90 days
Logs from monitored systems 3 days 30 days
Diagnostic events (system log) 7 days 90 days
Collector/agent heartbeats — 30 days

Limits: compress between 1 and 60 days, keep between 7 and 3650 days — or 0 for unlimited. Important: saving takes effect immediately and permanently — there's no undo for already-deleted raw data.

What happens behind the scenes

Even though the detailed metrics (check_results) get deleted after the configured period, the derived 5-minute, hourly, and daily aggregates remain for their own, much longer retention (daily values unlimited). Long-term trends in the history view don't disappear along with raw-data retention.

5. "Clean up now"

If the storage page shows a visible backlog (compression or retention is lagging), a "Clean up now" button appears. It triggers the maintenance run immediately instead of waiting for the next scheduled run (every 6 hours), and reports the result in plain text, e.g. "12 chunk(s) compressed, 4 removed by retention".

6. Compression doesn't seem to be working

If the data class table shows a "compression not taking effect" warning even though compression is configured — that's usually a sign the maintenance run hasn't gone through yet (up to 6 hours delay) or that freshly written chunks are naturally still uncompressed (compression only kicks in after the configured age, e.g. 3 days). No cause for concern as long as the value moves over several hours.

7. Disk space from Docker itself

Not all growth comes from the database — old, unused Docker images and build cache accumulate over time, especially after several updates. The "where the space goes" panel shows the reclaimable share directly; a docker system prune (manual, outside the GUI) cleans this up additionally, but should be run outside a running update.

Checklist

  1. Open Resources → Storage, look at usage + growth/day
  2. Check biggest data producers — is a single check responsible?
  3. Is log collection sensibly limited?
  4. Is retention per data class matched to actual need?
  5. Use "Clean up now" instead of waiting for the next 6-hour run
  6. Factor in Docker leftovers (images/build cache) too

Next