Semantic Cache (LLM Systems)
A caching layer that matches queries by meaning rather than exact text, using vector embeddings and similarity search to return stored LLM responses for paraphrased prompts.
A semantic cache is a response-caching layer for large language model systems that matches incoming queries to prior queries by semantic similarity rather than exact string equality. Each query is converted to a vector embedding and compared against embeddings of previously cached queries using cosine similarity; if a stored entry exceeds a configurable similarity threshold, its cached response is returned without calling the underlying model. Semantic caches sit above the model API, in contrast to prompt caching which reuses internal model state for exact prefix matches and to the KV cache which operates inside the transformer during a single generation. The semantic layer is the only one that tolerates paraphrase: differently worded queries with the same meaning produce nearby embeddings and can share a single cached answer. Production deployments report hit rates roughly between 30% and 70% on agent workloads and customer-support traffic, with cache lookups typically completing in single-digit milliseconds versus hundreds to thousands for a live model call. Open-source implementations include GPTCache; commercial offerings layer atop vector databases like Redis or Milvus. The principal risks are false positives — returning a stored answer to a query that is semantically similar but materially different — and staleness when underlying facts change without invalidating the cached embedding entry.