Your AI request returns 200 OK. How do you know the runtime was actually healthy?

While building WAIL, one problem kept coming up: infrastructure-level failures are easy to recognize when a provider returns a timeout or 5xx. The harder cases are requests that technically succeed.

A request can return 200 OK while first-token latency has increased significantly, throughput has degraded, or the stream has become unstable. From the application's perspective the call succeeded, but operationally the execution may already be unhealthy.

The approach I ended up taking with WAIL was to establish runtime baselines, evaluate degradation while execution is happening, and separate observation from control. Depending on the situation, WAIL can observe, retry, or reroute, while recording signed evidence of the decision and outcome. It runs alongside the provider client rather than requiring a gateway.

I'm curious how other teams running AI systems in production draw this line:

When a model call succeeds technically but deviates significantly from its normal runtime behavior, what determines whether you simply observe it, retry it, or route subsequent traffic elsewhere?

Are you using fixed SLOs, rolling baselines, provider-level metrics, or something else?