TokelyTHE SHORT ANSWER
LLM fallback sends a failed request to an approved alternative when the original route cannot complete it. A production policy needs eligible models, explicit failure conditions, a total deadline and a retry budget. Switching providers after every error can increase costs, repeat work and produce inconsistent results.
What is the difference between provider and model fallback?
Provider fallback changes the service delivering a model. Model fallback changes the model itself. A second provider may preserve the requested model identity, while a replacement model can change reasoning, formatting and tool behavior.
For an invoice extractor, both candidates must accept the document format and produce the required schema. For an agent, the replacement must understand the conversation and tool-result format. A model with a larger context window is not automatically a compatible substitute.
Which errors should trigger fallback?
Temporary unavailability, selected server errors and capacity limits can justify another attempt. Invalid credentials, an unsupported parameter and a malformed request usually need correction. A policy refusal should not become an automatic attempt to evade safeguards through another service.
Classify the failure before deciding. Also inspect whether the request may already have been accepted. A network timeout tells you that your client stopped waiting; it does not prove that the upstream stopped working.
How do you keep retries under control?
Give the entire operation one deadline. As an illustrative policy, a 20-second interactive operation might allow one primary attempt and one alternative, leaving time for validation and delivery. Those numbers need to match your product’s latency budget.
Avoid multiplying retries across layers. Three application attempts, each making three SDK attempts, can produce nine submissions before accounting for any gateway retries. Select the layer that owns retry policy and configure the others deliberately.
LiteLLM’s fallback documentation illustrates explicit fallback chains. The useful architectural principle is to define candidates and exceptions in configuration rather than scattering them through product code.
What happens if streaming has already started?
Do not silently append a second model’s answer to a partial first answer. Treat partial output as an incomplete attempt. Your interface can offer a restart, or your backend can buffer the response before presenting it if the extra delay is acceptable.
Tool execution needs separate protection. Use application-level operation IDs for actions such as creating a ticket or charging an account. A repeated model request must not repeat an irreversible tool action.
How should fallback be measured?
Record the original request ID, attempt count, selected model, failure category, elapsed time and total cost. Monitor successful recovery separately from first-attempt success. An apparently healthy application may be spending twice as much because its primary route is failing.
Tokely’s routing documentation explains eligible supplier fallback and possible route-dependent charges. A single-route model has no supplier fallback. Build application-level model substitution only after checking the capabilities of every candidate in the model catalog.