AI Explain — how it works, what it is not

The Verklaar dit / Explain this button on every alert, log line and detection event runs a locally-hosted Llama 3.1 8B model — no external AI provider is on the path. This page documents how it works, what it is good at, where it can be wrong, and what we did to reduce that risk.

Architecture

operator clicks "Explain"
    ↓
dashboard POSTs /api/v1/ai/explain   (cookie-authed, tenant-pinned)
    ↓
hub builds a STRUCTURED prompt:
    ├─ host metadata    (latest inventory snapshot: OS, kernel, hostname)
    ├─ grounding row    (looked up from ai_explain_grounding by alert
    │                    category / detection rule_kind / integrity kind
    │                    / log keyword — whichever matches)
    ├─ the line itself
    └─ a strict rule:   "if uncertain, say 'onbekend' instead of
                         inventing"
    ↓
POST http://127.0.0.1:11434/api/generate  (ollama container, llama3.1:8b)
    ↓
dashboard renders:
    ├─ model output (left)
    ├─ honesty banner ("AI-generated, verify")
    ├─ source-doc link if grounding was found
    └─ Verify panel (right) — REAL recent telemetry, no AI

Sovereignty: Ollama runs on the same VPS as the hub. No outbound connection to OpenAI/Anthropic/Google/etc. is made on the explain path. The container is named monsys-ollama and binds only to 127.0.0.1:11434.

What the model is good at (~80% of typical logs)

• Standard syslog patterns it has seen in training: sshd: Failed password, kernel: Out of memory, systemd: Failed to start, segfault, EAGAIN/EPIPE/etc.
• Concepts: signals, syscalls, OOM-killer behaviour, common firewall rules, what a TLS handshake does.
• Generic Linux/Windows admin tasks where the relevant command is in the standard toolset (systemctl, journalctl, lsof, …).

What the model is bad at — and how we mitigate

Pitfall	What can go wrong	Mitigation
Distro-specific commands	Suggests apt install on a Rocky 9 host because it pattern-matches on “Linux”.	Host metadata block in the prompt injects exact OS+version from the latest inventory snapshot, so the model picks the right package manager.
Monsys-specific terms	No idea what dep.maintainer_changed, mtls_cn_mismatch, kernel_module_suspicious mean — they are not in its training data.	RAG: ai_explain_grounding stores authoritative content per alert category / rule_kind / integrity kind. The model paraphrases that content instead of guessing.
Post-cutoff CVEs	Hallucinates about advisories published after Llama 3.1 was trained (e.g. PinTheft / PSA-2026-00022-1).	Grounding row for the relevant detection rule includes the advisory ID and remediation pointer.
Confidence inflation	The 8B model phrases everything with the same authority, even when 100% guessing.	Strict prompt rule: “if you’re not 100% sure about a specific package, path or command, answer ‘onbekend’ instead of inventing”. + Honesty banner in the UI.
Single point of failure on bad advice	Operator follows the AI advice blindly in a hurry.	Verify panel renders the actual last-24h alerts, detections and anomalies on the same agent next to the model output. Source-doc link when grounding fired.

How to verify on your own host

After the next agent inventory cycle:

# 1. Look at the grounding table — what does the hub seed?
sudo docker exec monsys-postgres psql -U monsys -d monsys -c \
  "SELECT match_kind, match_key, source_url FROM ai_explain_grounding ORDER BY match_kind, match_key;"

# 2. Confirm the hub is hitting only localhost ollama on explain calls:
sudo journalctl -u monsys-hub --since "10 min ago" | grep -i ollama

# 3. Confirm no outbound AI provider traffic:
sudo ss -tnp | grep -E "(openai|anthropic|google|cohere)"   # should be empty

When you should NOT trust the answer

• Anything involving a specific patched version number (“upgrade to X.Y.Z”). The 8B model often guesses these. Cross-check with the source-doc link or the CVE pipeline tab.
• Anything mentioning a path the model couldn’t see (“the config is at /etc/foo/bar.conf”). It pattern-matched a similar app — validate locally first.
• Confidence claims about a one-shot exploit. The model can’t tell active exploitation from log noise. Always look at the rest of the detection event (src_ip, count, window_secs) before pressing the EAT mitigation button.

Configuration

To add a grounding row (e.g. for a new alert category your tenant ships):

INSERT INTO ai_explain_grounding
  (match_kind, match_key, content_nl, content_fr, content_en, source_url, priority)
VALUES (
  'alert_category', 'your.category',
  'NL-tekst die het model moet parafraseren…',
  'Texte FR…',
  'EN text…',
  'https://docs.your-tenant.com/article',
  100
);

Match-kind values: alert_category, detection_rule, log_keyword, integrity_kind. Higher priority wins when multiple rows match.

Roadmap (not yet shipped)

• Larger model option: 70B / mixtral 8x7B for tenants with the VPS-RAM to afford it. Same Ollama API; just OLLAMA_MODEL=....
• Confidence-sampling: re-prompt with a different temperature and flag inconsistent answers as “low confidence”.
• Per-tenant grounding overrides: lets a tenant inject their own runbook content for an alert category they care about.