What is LLM failover?
LLM failover automatically retries a failed request on a backup so a single model or backend outage doesn't surface to your users as an error.
Models and their backends have outages, rate limits, and slow spells. Without failover, any of these becomes a user-facing error and your application has to implement its own retry-and-fallback logic. Failover moves that resilience into the gateway.
A robust failover chain has several layers: retry the same model, then try an equivalent model on a different backend, then fall back to a quality-equivalent model. Done well, the user sees a slightly slower response instead of an error, and you write none of the retry logic.
How Kyma does it
Kyma uses a multi-layer never-die chain — retry, equivalent-model swap, then a parallel race across quality-tier peers — so user-facing errors stay near zero. When a fallback serves, response headers tell you which model and which layer handled it.