Paul RukwaroSoftware engineering

2 min readAI observability

Detecting AI answer-quality regressions that don't throw errors

A simulation of the gap between request-level monitoring and answer-quality monitoring, comparing aggregate and topic-sliced drift detectors on deterministic, non-semantic features under a modeled operating budget.

Independent technical analysis, with implementation references below.

From the project. This note follows from Silent-Failure Observability (project 50). Read the source or the design documentation for the full implementation and decision records this note draws on.

The gap application monitoring can't see

Standard request monitoring — status codes, latency, error rates — says nothing about whether the content an AI system returns is still correct. A quality regression can live entirely inside a successful response payload, invisible to conventional APM. Project 50 is a simulation built to study that gap directly, not a production monitoring deployment.

A feature representation without semantics

The features driving detection are deterministic character 4-gram hashes projected into a 256-dimensional space — not a semantic embedding model, and not connected to any live OpenTelemetry trace pipeline. That keeps the simulation fully reproducible, but it also means the detectors are reacting to lexical and statistical shifts in text, not to meaning.

Slicing helps, and it costs something

A population-stability-index detector run per topic slice caught a localized regression affecting about 8% of simulated traffic that an aggregate, whole-population detector missed entirely.

The same slicing approach was slower on a different scenario: a diffuse, population-wide regression was detected roughly 12 days later by the sliced detector than by the aggregate one in that scenario. That is a real sensitivity-versus-timeliness trade-off inside this simulation, not a general argument that slicing is always better or worse.

Pricing the detector set, not just the detector

Under a modeled operating-cost assumption — a "quality canary" detector priced at roughly 24 calls a day — the canary was not part of the cheapest combination of detectors that still covered every simulated failure scenario; two lower-cost detectors covered the same ground and more. This used simulated budget assumptions and a modeled call volume, not real provider billing or usage.

What would have to change for production use

There is no real model provider behind this, no live trace data, and no actual customer traffic. Before any of this informed a real observability system, it would need privacy-preserving telemetry, genuine quality labels in place of simulated ones, correlation with provider and model version, and an actual on-call routing path — none of which exist here.