AI Observability Cheatsheet: What to Log, Alert, and Ignore

For: A backend or ML engineer at a 30–150-person SaaS company who shipped an LLM-backed feature three months ago and is now drowning in Datadog noise — every spike pages someone, but the actual regressions (drift, silent bad outputs, cost blowouts) go undetected until a customer reports them

If your LLM feature is paging you on p99 latency spikes but you're finding out about output regressions from customer support tickets, your observability stack is inverted. Log four things religiously (output quality signal, token spend per request, retrieval hit quality, prompt/model version), alert on rate-of-change in those four plus a hard cost ceiling, and stop paging on infrastructure metrics that don't correlate with user-visible quality. Everything below is the specific version of that.

The core mistake: treating AI observability as a superset of APM. It isn't. The signals that matter most — output quality distribution, token burn rate, retrieval relevance, prompt version regression — are invisible to Datadog, New Relic, and every infra tool you already pay for. They have to be instrumented at the application layer, deliberately, on the request path. By the time your p99 moves, the model has been misbehaving for days.

What to log on every LLM request

Log these fields per inference call. Structured JSON, one row per request, sampled at 100% until volume forces you down to 10%.

The quality signal — pick at least one

Every request needs a numeric proxy for "was this output good." Options, cheapest to most expensive:

• Implicit user feedback: thumbs-up/down, copy-to-clipboard, follow-up question rate, session abandonment within 30s of response
• Structural checks: JSON schema validity, expected fields present, output length within a sane range for the task
• Reference-free scores: perplexity of the output under a small local model, semantic similarity to retrieved context (RAG faithfulness)
• LLM-as-judge on a sample: 1–5% of traffic, cheap model, rubric-scored async — cheap enough to run continuously
• Golden set replay: nightly run of 200–500 curated prompts, compare to reference outputs

What to alert on

Alerts must correspond to user-visible or business-visible harm. If it doesn't, it's a dashboard, not a page.

Two thresholds matter more than the rest: cost burn rate and quality drift. Everything else is secondary. If you only instrument two alerts this quarter, do those.

What to ignore (or downgrade to dashboard)

• Total request latency p99 alone. LLM latency is bimodal and noisy. Alert on TTFT and only when sustained.
• CPU/memory on inference gateways. You're I/O bound on provider APIs. This tells you nothing.
• Individual 429s and transient 5xxs. Retry-with-backoff should absorb these. Alert on the rate, not the events.
• Token count per request as a standalone metric. Only meaningful in aggregate against budget.
• Cache hit rate below a hard floor. Useful as a dashboard, rarely a page.
• Any metric without a baseline. If you can't state "normal is X ± Y," you can't alert on it.

The four dashboards you actually need

1. Cost & budget: daily spend by feature/tenant/model, projected month-end, top 10 most expensive prompt patterns
2. Quality: judge scores, thumbs ratio, golden set pass rate, refusal rate — all sliced by prompt_version and model_version
3. Retrieval health (if RAG): hit rate, mean top-k score, doc freshness, queries with zero relevant hits
4. Reliability: TTFT, error rate by provider, fallback engagement rate, timeout rate

Instrumentation traps

• Logging full prompts and outputs unbounded. PII risk plus storage cost. Hash, truncate, or route to a separate encrypted store with short retention.
• Sampling before you have volume. Log everything until you're above ~100k requests/day. You can't debug what you didn't capture.
• No prompt version discipline. If prompts live in code, tag them. If they live in a CMS, checksum them. Otherwise quality regressions are unattributable.
• Judge model = production model. Same failure modes get scored as passes. Use a different family.
• Alerting on averages. Quality regressions hide in tails. Alert on p10 of judge scores, not mean.

How CodeNicely can help

If you're at the point where the observability gap has become a business problem — customer complaints leading the roadmap, unpredictable monthly bills, no confidence when you ship a prompt change — that's the work we do. On HealthPotli, an e-pharmacy platform with an AI drug interaction feature, output correctness wasn't optional — a bad response has clinical consequences. We built the quality-signal and golden-set infrastructure to catch regressions before pharmacists did, not after. The specific pattern — instrumenting output quality at the application layer, versioning prompts, and wiring cost ceilings to real budgets — carries across domains. More on our approach at CodeNicely AI Studio.

Frequently Asked Questions

What's the single most important metric for LLM monitoring in production?

A per-request quality signal — even a crude one like thumbs-up/down or JSON schema validity. Without it, every other metric is proxy noise. Latency, cost, and error rate can all look healthy while your model quietly gets worse.

How do I detect prompt regressions before customers do?

Maintain a golden set of 200–500 curated prompt/expected-output pairs, run it on every prompt or model change, and again nightly. Fail the deploy if pass rate drops more than a fixed threshold. This catches the majority of regressions the same day they land.

Should I use an LLM-as-judge for continuous quality monitoring?

Yes, on a small sample (1–5% of traffic) with a different model family than production. It's cheap enough to run continuously and catches drift that user feedback misses because most users don't flag bad outputs — they just leave.

What alerting thresholds should I set for token cost?

Set two: a hard monthly ceiling that pages when burn rate projects above 120% of budget, and a per-tenant anomaly alert for any account whose daily spend goes 3x its 7-day baseline. Everything else is a dashboard.

Do I need a specialized LLM observability tool, or can I extend Datadog?

You can start by pushing structured logs into whatever you already have — Datadog, Grafana, BigQuery. Specialized tools (Langfuse, Arize, Helicone) pay off once you need trace-level prompt/response inspection, judge orchestration, or evaluation UI. Instrument the fields first; pick the tool second.