Insights — anomaly detection

/insights shows metrics that are statistically outside their normal distribution. It is not machine learning and we don’t claim it is anywhere — it’s a z-score over a 7-day baseline.

Method

For every (agent, metric) in {CPU%, memory%, network total bps}:

baseline = AVG(metric) ± STDDEV_POP(metric) over [NOW-7d, NOW-1d]
current  = AVG(metric) over [NOW-15min, NOW]
z = (current - baseline_mean) / baseline_stddev

A metric is flagged anomalous when |z| > 2.5. That maps to ~0.6% false-positive rate if the metric is normally distributed (in practice the FP rate is higher because metrics are rarely truly Gaussian — but it’s good enough to surface the real outliers).

Only agents with ≥60 baseline samples (~1 day at 15s metrics) qualify. Otherwise a newly registered agent would always look “abnormal”.

What you see in the UI

Four KPI cards at the top:

• Anomalies (live) — count of current flags
• Agents covered — agents online in the last hour
• Metrics (24h) — total datapoints analysed
• Method — “z-score, threshold |z| > 2.5”

Below the KPIs: table of top-50 anomalies, sorted by |z|. Columns: agent (clickable → agent detail), metric, current value, baseline mean, stddev, z-score, direction (↑ above / ↓ below).

What this does WELL

• Sudden spikes where the system is 4× its normal usage
• Sudden dips (“why is this webserver’s CPU suddenly at 1%?”)
• Pattern breaks: ranges you didn’t have to program explicitly

What this does NOT do

• Seasonal patterns — if your nightly batch pegs CPU at 80% every 02:00, that stays in the baseline and stops being flagged. Intentional.
• Cross-metric correlations — a cross-metric ML would do that; this method doesn’t.
• Prediction — no “this server will fall over in 3 hours”. For predictive growth see Capacity planning.

How to use it operationally

1. Check /insights once per shift as part of your status check
2. Click through a row to see the agent and what’s going on
3. If it’s an expected metric (“nightly batch”) → working as intended, ignore
4. If it’s an unexpected metric → investigate

Not an alert replacement

Insights is passive observation, not a replacement for alert rules. An alert rule fires push notifications + audit log + webhooks; Insights asks you to look proactively. For pages that should wake you at night: use /alerts → Rules with severity=critical.