Not model quality — the plumbing around it. These are the failures that pass every test and then appear the week after launch. For each: how to check it in your own codebase, and what the fix looks like.
A prompt tweak, a model version bump, or a retrieval change quietly degrades output. Nobody notices until users complain — and by then you can't tell which change did it.
This is now the dominant cause of production LLM incidents, having displaced prompt injection in mature deployments. One documented case: a support agent's groundedness fell 11 points over 18 minutes, traced to a prompt template stitching in an empty retrieval block.
Check itIs there any test that runs the AI path with fixed inputs and asserts something about the output? Not "does it return 200" — something about the content. If the answer is no, you are shipping prompt changes blind.
FixA small set of golden cases in CI. Ten fixed inputs with asserted properties beats zero by an enormous margin. You are not trying to measure quality — you are trying to notice a cliff.
The same request arrives twice — a client retry, a webhook redelivery, a queue redrive — and something happens twice. A charge, an email, a credit deducted, a row inserted.
Check itgrep -rn "webhook\|/callback\|retry\|idempot" --include=*.ts --include=*.pyFor each handler, ask of every side effect: is this keyed on something stable? Look for an idempotency key, a unique constraint, or an upsert. A bare INSERT, or a call to a payment or email API with no key, is the bug.
A unique key on the operation, checked inside the same transaction as the write. Not checked before the write — inside it, or the race just moves.
Be honest about the limit: that makes your own state exactly-once. It does not make an external call exactly-once — if the provider charged before your process died, no amount of local fencing un-charges it. What good design buys you is that the duplicate becomes detectable and reconcilable rather than silent. Anyone selling you exactly-once across a network boundary is overselling.
No ceiling per user, per session, or globally. One retry loop, one enthusiastic user, or one bad actor and the bill is four figures by morning. Almost nobody has this on launch day.
Check itFind any spend or token budget enforced before a call is made. Then compute the worst case out loud: how many calls can one user trigger in an hour, at what token count, at current pricing? That number is usually the moment the room goes quiet.
FixA budget per request and per user per period, checked before the call rather than measured after. Metering is not a cap.
The model takes 60 seconds. The request hangs, the worker is tied up, and under load the pool exhausts and unrelated parts of your service start failing too.
Check itLook at every LLM client construction. Is a timeout set? Most SDKs default to none, or to something far longer than you'd choose. Then check it against the caller's deadline — a 120s client timeout behind a 30s gateway is the same as no timeout, except now you also can't see it.
FixAn explicit deadline per call, shorter than the caller's, with defined behaviour on expiry. Fail fast, don't hang.
Rate limit or outage, and the feature just breaks. This is not hypothetical: OpenAI had a service disruption in February 2026 and a global outage in April 2026 that took down ChatGPT, Codex and API access together.
Check itRead the error handling around model calls. Is there backoff? A second provider? A cheaper or cached path? A user-facing message that isn't a stack trace?
Error classification, backoff with jitter on the retryable kind, clean failure on the rest.
Tool calls tool, model re-plans, repeat. Cost and latency both without a ceiling, and the failure mode is a bill rather than an error.
Check itFind the agent loop. Is there a bound on turns, on total tokens, and on wall-clock time? All three, not one. Is there detection for the same tool being called with the same arguments repeatedly?
FixHard caps on all three axes and a defined behaviour when one is hit — return a partial result with the reason, rather than truncating silently.
The process dies between the external call and the database write. Now the record says one thing and reality says another: paid but not fulfilled, charged but not delivered, started and stuck forever.
Check itFind every multi-step operation that touches an external service and your own storage. Ask: if the process is killed between step two and step three, what does the record look like, and can anything recover it?
FixDurable state with explicit statuses, and claims that are leased so a crashed attempt can be retried — and fenced so a slow one can't overwrite the attempt that replaced it.
The connection drops mid-stream. The partial answer is persisted and presented to the user as finished — often with no visible difference.
Check itFind where streamed output is written. Is it saved on a clean terminal event, or accumulated and stored regardless of how the stream ended? Is there a "complete" flag distinct from "has content"?
FixOnly mark complete on a clean finish. Store partials as partial and let the UI say so.
A user upload, a scraped page, an email body reaches a model that can send mail, write to the database, or spend money.
Check itTrace every input that reaches a prompt and mark which are attacker-controlled. Then list what the model can actually do. The intersection is your surface.
Least-privilege tools, explicit confirmation for irreversible actions, and treating model output as untrusted input to whatever consumes it next.
It breaks at 3am and the log line says Error: request failed. Nobody can answer which user, which prompt, which provider, how long it took, or what it cost.
Read your catch blocks around model calls. Then ask the question that decides it: when this misbehaves, who notices first — you, or the customer? If it's the customer, this is your first finding regardless of what else is wrong.
FixStructured logs with a correlation id on every model call — user, model, latency, tokens, cost — and one alert on the failure that costs money.
Send it to me. $250, fixed price: I reproduce it, fix it, and hand back a regression test that fails without the fix, plus a written account of what was actually wrong. Half up front, and if I can't reproduce it you pay nothing and keep the write-up of what I ruled out.
If you'd rather have the whole feature reviewed against all ten before committing to anything, I do that as a two-day written audit for $500 — credited against the fix if you go ahead.
Fix one bug — $250, fixed price