AgentBlueprints Weekly · Issue #1

Your LLM app shouldn't die when one provider does

Welcome to issue #1. Each week we ship one blueprint that's fully built: the config, how to run it, what it costs, and how it breaks. Then you vote on the next one.

This week's blueprint: an LLM gateway with automatic fallbacks

The problem. Your app calls one model provider. That provider rate-limits you or has an outage, and your app goes down with it.

The build. Run a LiteLLM proxy in front of your app. It speaks the OpenAI API, so your code only changes its base URL. Behind it, you group two deployments under one name for load balancing and name a backup model on a different provider.

model_list:
  - model_name: chat-default
    litellm_params: { model: openai/<your-model>, api_key: os.environ/OPENAI_API_KEY }
  - model_name: chat-default
    litellm_params: { model: azure/<your-deployment>, api_base: os.environ/AZURE_API_BASE, api_key: os.environ/AZURE_API_KEY }
  - model_name: chat-backup
    litellm_params: { model: anthropic/<your-model>, api_key: os.environ/ANTHROPIC_API_KEY }
router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30
  fallbacks:
    - chat-default: [chat-backup]

pip install 'litellm[proxy]'
litellm --config config.yaml --port 4000

Setup time. 30-60 minutes (estimate). Last verified 2026-10-14. Cost. LiteLLM is open source; you pay only your providers' token prices plus hosting for the proxy.

How it breaks

Fallbacks hide problems. Log which model actually answered, or you won't notice you've been on the backup for a week.
Different providers give different outputs. If you parse structured output, test the backup model against the same prompts.
Retries multiply cost and latency on a bad day. Keep num_retries low and set a timeout.
Model names change. Check identifiers against each provider's current list.

Read the full blueprint

Vote: what should we build next?

Click one and we'll build the winner for next week's issue.

That's it. Email [email protected] and tell us what broke in your agent stack this week; it'll shape what we build.

The AgentBlueprints team Built by humans with help from AI agents.

---

Want the next issue in your inbox? Subscribe free. Sponsorships start from $79/mo: see sponsor options.