ANKIT YADAV

LLM Engineering · 19 Nov 2026 · 6 min read

Multi-LLM Fallback Architecture

A practical way to call a second model when the first one times out or returns an unusable response, without repeating the same side effect.

Introduction

A model provider is a dependency with a status page. Timeouts, rate limits, and a response that does not match your schema are normal. A product that has exactly one provider will show the user that status page, translated into a spinner that never ends.

Fallback is the boring design that keeps the turn moving: classify the failure, retry only the failures that are worth retrying, then call the next provider with the same intent. It is not a contest between model brands, and it is not a license to run the same write operation three times.

This note describes one reasonable chain. It does not report a latency number, because a number from someone else's traffic would be fiction here.

Problem

The failure modes are different and people often handle them as one catch.

A timeout means you do not know whether the provider started the request. A 429 means they asked you to slow down. A 5xx means their server failed. A 200 with malformed tool arguments means the transport worked and the content did not. A refusal on safety grounds means another provider might also refuse, and hammering the chain will not create a policy exception you are allowed to have.

If you retry all of those the same way, you do two harmful things. You spend the user's wait on errors that will not improve, and you repeat any side effect that sat in front of the model call. Sending a WhatsApp message and then retrying the whole handler is how a customer gets two replies.

Technical explanation

Give every outbound model call a budget: a timeout, a maximum number of attempts on that provider, and a maximum number of providers. A common starting point is one retry on the same provider for timeout and 429, then a single attempt on the next provider. Schema failures get one repair turn against the same provider, which is a different mechanism from transport fallback and should be counted separately so a bad schema does not burn the whole chain.

Classify before you branch.

  • Timeout and network error: retry once if the call was read-only, or if you have an idempotency key. Then fall through.
  • 429: honor a short backoff if you have time left in the user's budget. Otherwise fall through. Do not build a hidden sleep that exceeds the chat timeout.
  • 5xx: fall through. A second try on a server that just failed is optional and usually not worth the wait.
  • Schema or tool-argument failure: one repair, then a clean error. Another provider will often make the same mistake if your schema description is the thing that is unclear.
  • Safety refusal: stop. Do not shop the same blocked request around until some provider agrees.

Normalize the request before the first call. Tool schemas, system instructions, and the message list should be in your own types. Each provider adapter translates that into the vendor's body. If the adapter cannot express a feature, it should fail visibly for that provider rather than dropping the tool list and hoping the prose is enough.

The user-facing result is one of three: a normal reply, a handoff, or a short failure that tells the person what to do next. Internal errors stay in the log.

Architecture

Diagram

The chain is ordered on purpose. Put the provider you have tested your prompts against first. The fallback is there for availability, not because you measured it as smarter. If you have not run the same tool schema through the fallback provider, the fallback is a hope.

Log the provider id, the error class, the attempt index, and the latency. Keep prompt text and document text out of that log unless you already have a place those are allowed to live. A fallback system that copies customer content into a general log is a new incident, not a reliability feature.

Implementation approach

Wrap each vendor in an adapter with one method, generate(request, signal), that throws a typed error. The chain catches that type and nothing else. An unexpected exception is a bug in your code and should not be quietly retried as if the vendor had blinked.

Use AbortSignal so the user's deadline cancels the attempt that is still running when you move on. A fallback that does not abort the first call will have two responses in flight. If the first one was a write, you now have a race.

Idempotency sits outside the model. For a chat reply, the idempotency key can be the inbound message id from the channel. For a tool that sends mail or updates a claim, the key is created when the action is decided and stored before the call. The fallback may generate text again. It may not perform the action again if the key has already been used.

Circuit breaking is optional and easy to overbuild. A simple version counts recent failures per provider in memory and skips a provider for a minute after a burst. On a serverless host that memory is per instance, so it will be wrong and still useful during one process's bad minute. A shared store is better when you already have one. It is not a prerequisite for having a second provider.

Do not fall back from a successful tool execution into a second model just because you dislike the prose. The tool already ran. A style preference is not an error class.

Code examples

fallback-chain.ts
type ErrorClass = "timeout" | "rate_limit" | "server" | "schema" | "refusal" | "other";
 
type Attempt = {
  provider: string;
  run: (signal: AbortSignal) => Promise<string>;
};
 
async function withTimeout<T>(work: (signal: AbortSignal) => Promise<T>, ms: number) {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), ms);
  try {
    return await work(controller.signal);
  } finally {
    clearTimeout(timer);
  }
}
 
function classify(error: unknown): ErrorClass {
  if (error instanceof Error && error.name === "AbortError") return "timeout";
  const status = typeof error === "object" && error && "status" in error ? Number(error.status) : 0;
  if (status === 429) return "rate_limit";
  if (status >= 500) return "server";
  return "other";
}
 
async function runChain(attempts: Attempt[], budgetMs: number) {
  let last: ErrorClass = "other";
  for (const attempt of attempts) {
    try {
      return await withTimeout(attempt.run, budgetMs);
    } catch (error) {
      last = classify(error);
      if (last === "refusal" || last === "schema") break;
    }
  }
  throw new Error(`fallback_exhausted:${last}`);
}

The classifier in a real adapter should read the vendor's error, not a guess about status. The shape of the function is what matters: typed class in, decision out, no string matching on a paragraph the provider might reword.

Python's httpx with an explicit timeout, or asyncio.wait_for, is the same control flow. Catch the timeout you raised. Do not catch Exception and call it a fallback.

Lessons learned

Write the error classes down before you pick the second vendor. Teams buy a fallback provider and then discover their prompts use a tool feature the second API does not have. The chain looks highly available and fails every request that needs the tool.

Measure the chain with failures you inject: abort the socket, return a 429, return an empty tool call. A fallback you have never seen fire is a config flag.

Keep the primary provider's success path free of fallback code. The happy path should read as one call. The chain belongs in the error path, where a reviewer can see the budget.

Tell the user almost nothing about which provider answered. "Trying another model" is noise in a support chat. The operator log is the place that records the hop. If every provider fails, the message should offer the human path or the direct email, not a paragraph of status codes.

Conclusion

Multi-LLM fallback is a retry policy with more than one implementation behind it. Classify the error, retry only what is idempotent, abort the attempt you are leaving, and stop on refusals and on schema problems that another brand will not fix. Availability is the goal. A second copy of a side effect is not a success.

Tool calling needs this chain around both model calls, not only the first. Retrieval adds a different failure: the model can succeed and still answer from nothing. That is a prompt and product rule, covered separately, not a provider outage.

Related articles

LLM Engineering · 19 Nov 2026 · 6 min

Understanding LLM Tool Calling

How an application asks a model for a structured tool call, checks the arguments, runs the tool itself, and sends the result back to the model.