LiteLLM Proxy with Fallbacks and Load Balancing

An OpenAI-compatible gateway that load-balances deployments and falls back across providers on errors.

litellmproxyfallbacksopenai-compatible

Setup time30-60 minutes (estimate)

Estimated costLiteLLM is open source; you pay only your providers' token prices plus hosting for the proxy.

ToolsLiteLLM, Docker

VerificationLast verified 2026-10-14

config.yaml

model_list:
  - model_name: chat-default
    litellm_params:
      model: openai/gpt-6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: chat-default            # same name = load-balanced group
    litellm_params:
      model: azure/my-gpt4o-mini-deployment
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
  - model_name: chat-backup
    litellm_params:
      model: anthropic/claude-haiku-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30
  fallbacks:
    - chat-default: [chat-backup]

Run

pip install 'litellm[proxy]'
litellm --config config.yaml --port 4000
curl http://localhost:4000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"chat-default","messages":[{"role":"user","content":"hi"}]}'

Point any OpenAI SDK at http://localhost:4000 as the base URL. Check model identifiers against your providers' current model lists.

Failure modes

Fallbacks hide outages; log which model actually answered.
Backup models format structured output differently; test them on the same prompts.
Retries multiply cost and latency; keep num_retries low and set a timeout.

Markdown version: /blueprints/litellm-fallback-router.md