Rate Limits, Latency, and Fallbacks: Running LLMs in Production

An LLM call that works perfectly in a notebook behaves differently the moment it sits on a real request path with real traffic. Three failure modes show up almost immediately, and none of them are about prompt quality.

Rate limits are a capacity problem, not an error to catch

Provider-side rate limits aren't an edge case — at any real volume, hitting them is a certainty, not a possibility. Catching a 429 and retrying immediately just moves the problem a few seconds later. The fix that actually holds up is a request queue with backoff and, where the workload allows it, batching multiple requests together instead of firing them independently. If a feature's usage pattern is bursty (a bulk import, a scheduled job), that queue needs its own concurrency limit separate from the rest of the application's traffic.

Latency has to be designed for, not hidden

A model call commonly takes anywhere from several hundred milliseconds to several seconds, and that variance is wide. Anything synchronous in a request-response cycle needs an explicit timeout and a defined behavior for what happens when it's hit — a cached previous answer, a simpler non-AI fallback, or an honest "still working" state in the UI. What doesn't work is leaving the default client timeout in place and hoping it's rarely hit; at scale, "rarely" becomes a support ticket every day.

Fallbacks need to be planned before the feature ships, not after an outage

Every provider has outages and degraded-performance windows. A feature that hard-depends on one model with no fallback path will go down when that provider does. The fallback doesn't have to be another LLM — often the right fallback for a summarization or classification feature is a simpler rule-based version that's worse but available. Deciding what "worse but available" looks like during a design review is a much better experience than deciding it during an incident.

What this actually costs to build

None of this is exotic engineering — a queue, a timeout policy, and a fallback path are patterns most backend teams already know. The work is remembering to apply them to an LLM call the same way you would to any other external dependency you don't control, instead of treating it as a special case that gets skipped past code review.