AI observability quick-start

monsys.ai ships a passive, audit-grade observability layer for your LLM applications. This guide takes you from an empty account to your first signed evidence pack in about fifteen minutes.

What you’ll get

• An ingest token scoped to one application.
• An SDK (Python, Node or Go, ~150 LOC) wrapping each LLM call into a trace.
• PII redaction at the source (IBAN-BE, RRN, BTW, KBO, email, phone).
• Per-request cost and token counts in the dashboard.
• Anomaly alerts (cost spikes, refusal spikes, PII-leak rate…).
• An Ed25519-signed evidence pack you can download per period, with an offline verifier for your auditor.

0. Prerequisite

Your tenant needs ai_observability_enabled = true. This is off by default. Request it via info@be-hosted.be or via your beta onboarding ticket — we flip the flag within 24h after signing the AI addendum to the DPA.

1. Mint an ingest token

In the dashboard:

1. Left nav → AI
2. Apps → New app
3. Fill in name (e.g. support-bot)
4. Pick a redaction_level:
   • off — no redaction (sandbox/test only)
   • hash-only — only content hashes are stored, no raw content
   • pattern-redact — IBAN/RRN/BTW etc. replaced by tokens, rest preserved (default, recommended)
   • full-content-strip — empty strings for all free-text fields
5. Click Create — the token (aiv_...) is shown once. Save it now; only the SHA256 is kept.

2. Install an SDK

Python

# No pip install — copy monsys_ai.py from /sdk/python/ into your project
export MONSYS_AI_ENDPOINT=https://api.monsys.ai/api/v1/ai/ingest
export MONSYS_AI_TOKEN=aiv_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

from monsys_ai import Tracer

tracer = Tracer()

with tracer.trace("rag.chat") as t:
    with t.span("openai.chat", provider="openai", model="gpt-4o") as s:
        s.prompt = user_msg
        resp = client.chat.completions.create(...)
        s.completion = resp.choices[0].message.content
        s.input_tokens = resp.usage.prompt_tokens
        s.output_tokens = resp.usage.completion_tokens

One HTTP POST per trace, no batching, no background threads. Failures log; they never throw.

Node (TypeScript)

import { Tracer } from "./monsys-ai";

const tracer = new Tracer();
await tracer.trace("rag.chat", async (t) => {
  await t.span("openai.chat", { provider: "openai", model: "gpt-4o" }, async (s) => {
    const resp = await openai.chat.completions.create({...});
    s.record({
      prompt: userMsg,
      completion: resp.choices[0].message.content!,
      inputTokens: resp.usage!.prompt_tokens,
      outputTokens: resp.usage!.completion_tokens,
    });
  });
});

Go

tracer, _ := monsysai.New(monsysai.Options{})
_ = tracer.Trace(ctx, "rag.chat", func(t *monsysai.Trace) error {
    return t.Span(ctx, "openai.chat",
        monsysai.SpanOpts{Provider: "openai", Model: "gpt-4o"},
        func(s *monsysai.Span) error {
            s.Prompt = userMsg
            // ... call your LLM ...
            return nil
        })
})

3. Verify in the dashboard

Within seconds you’ll see:

• AI → Traces — chronological list, click into one for a span tree with provider, model, tokens, latency, cost, PII hits.
• AI → Apps → support-bot — totals per app.
• AI → Summary — KPIs over the last 24h (call count, cost, refusal rate, PII hit rate).

4. Unlock content (TOTP-gated)

If your tenant uses redaction_level=hash-only or stricter, raw content isn’t visible in the dashboard. For incident investigation an admin can unlock the content of one specific span:

1. Open the trace.
2. Click a span → Unlock content.
3. Confirm with your 6-digit TOTP code + a reason (audit-logged).
4. Original (redacted) content shows for 5 minutes.

Every unlock is recorded in your audit log with user_id, span_id, reason and timestamp.

5. Configure alert rules

AI → Alert rules → New rule. Examples:

• Cost spike: cost_per_minute > 1.00 EUR for app=support-bot
• Refusal spike: refusal_rate_pct > 30% over 15min
• PII leak: pii_hit_rate_pct > 5%
• Tool-call anomaly: tool_call_zscore > 3 (vs 7-day baseline)

Alerts go to your ntfy channel and/or webhooks. Webhook payloads contain only content hashes, no raw text — see ADR-0009 invariant 9.

6. Export an evidence pack

AI → Evidence packs → New pack:

• Period: e.g. 2026-04-01 to 2026-04-30
• App: all or one specific
• include_blobs: on (for full evidence including redacted content)

Click Create. You get a download URL for a pack-N.tar.gz.

Bundle layout:

manifest.json    — metadata + content hashes (Ed25519-signed)
manifest.sig     — Ed25519 signature (base64)
traces.jsonl     — one line per trace
spans.jsonl      — one line per span
blobs/HASH.txt   — redacted content per hash

7. Offline verification (for your auditor)

Hand your auditor:

1. The pack-N.tar.gz
2. The script tools/evidence-pack-verify.py
3. (Optional) our public signing key from https://monsys.ai/security/keys

The auditor runs:

pip install cryptography
python3 evidence-pack-verify.py pack-N.tar.gz

Exit code:

• 0 — manifest signature valid + all artifact hashes match
• 1 — mismatch (tampering or corruption)

The auditor needs no monsys account. The script uses the public signing key embedded in the manifest itself; you cross-verify that against what monsys publishes publicly.

Next steps

• Read the three hard invariants (passive, PII at source, signed packs).
• Read the envelope spec if you’re building your own SDK.
• Read the evidence pack format for your auditor.