Semantic Response Cache in Front of LLM Calls
Reuse prior answers for near-duplicate prompts using embedding similarity with a strict threshold and TTL.
Steps
- Normalize the prompt (trim, lowercase for matching only) and include model + system prompt hash in the cache key scope.
- Exact-match lookup first (hash). On miss, embed the prompt.
- Nearest-neighbor search within the same scope; accept only if cosine similarity ≥ threshold.
- On miss, call the model and store
(embedding, prompt, response, created_at). - Expire entries by TTL; never cache responses that depend on time or user-specific data.
Config
semantic_cache:
exact_match_first: true
similarity_threshold: 0.95 # start strict; lower only after reviewing false hits
ttl_hours: 24
scope_by: [model, system_prompt_hash, tool_set_hash]
skip_if: [has_tools_with_side_effects, temperature_gt_0_5, user_specific_context]LiteLLM and GPTCache both offer built-in semantic caching if you prefer not to build it.
Failure modes
- Too-low similarity threshold returns wrong cached answers.
- Cache not scoped by model and system prompt serves stale behavior.
- Personal or time-sensitive answers cached across users; exclude them.
Markdown version: /blueprints/semantic-cache.md
Want a deeper build of this?
Vote for an extended, fully tested version (more tools, a full workflow file). Votes decide what we build next.
Get one fully built blueprint a week
A weekly email: one blueprint with config, cost, setup time and failure modes, plus a vote on what we build next.
Related blueprints
Complexity-Based Cheap/Strong Model Router
Route easy requests to a cheap model and hard ones to a strong model using a lightweight classifier plus escalation.
LiteLLM Proxy with Fallbacks and Load Balancing
An OpenAI-compatible gateway that load-balances deployments and falls back across providers on errors.
Local-First Routing with Ollama and Cloud Fallback
Serve requests from a local Ollama model, falling back to a cloud model only when local fails or the task needs it.