Semantic Cache

A semantic cache stores earlier requests with their answers and, for a new request, returns the stored answer of the nearest earlier request if the two are similar enough. It extends caching from identical requests to paraphrases: “Can I return this within 30 days?” and “Is a return possible inside 30 days?” can share one stored answer even though no string matches.

How it works

1. Each request is converted to an embedding, a vector that places texts with similar meaning close together.
2. The cache looks up the nearest stored embedding, usually by cosine similarity, \(\cos(a,b) = \dfrac{a\cdot b}{\|a\|\,\|b\|}\), which is 1 for vectors pointing the same way.
3. If the similarity is at least a threshold \(\theta\), the stored answer is returned and no model call is made. Otherwise the model answers and the pair is stored.

With a key that covers all answer dependencies and a still-valid stored answer, an exact match reuses that answer or misses. If permissions, versions, or context are missing from the key, an exact hit can be wrong too. A semantic cache replaces that with a score, so a hit can be wrong.

Why wrong hits happen

Embeddings capture topical closeness, not whether two questions have the same answer. With hand-made three-dimensional toy vectors (not from a real embedding model), the similarities to “Can I return this within 30 days?” are:

other request	cosine similarity
Is a return possible inside 30 days?	0.998
Can I return this after 30 days?	0.995
What is your shipping fee?	0.487

The paraphrase and the opposite question (“after” instead of “within”) are nearly equally close. The vectors were chosen to show the shape of the problem, which real embedding models share: negation, numbers, dates, and qualifiers move the vector little but change the answer completely.

Threshold trade-off with arithmetic

Suppose (hypothetically) 10,000 lookups, of which 80 percent have a nearest neighbor that is a true paraphrase and 20 percent have a neighbor that needs a different answer. At the chosen \(\theta\), 70 percent of paraphrases and 12 percent of the different-answer neighbors pass. Then hits are 5,600 correct plus 240 wrong, so 4.1 percent of served answers are wrong. Raising \(\theta\) lowers both numbers: fewer wrong answers, fewer saved model calls. The wrong-hit rate rarely reaches zero because the two similarity distributions overlap.

Other risks

• Staleness. A stored answer outlives the policy, price, or document it was based on. Entries need a time limit or a version key tied to the data the answer depends on.
• Authorization. An answer produced for one user’s data must not be served to another user. The cache key has to include whatever determines who may see the answer, such as tenant or permission scope.
• Context dependence. In a multi-turn chat, the same final question has different meanings after different histories. Caching on the last message alone mixes them.
• Invisible failure. A wrong hit raises no error and no model sees it, so it appears in no logs unless one looks.

Testing

Measure the false-hit rate instead of assuming it. Take a labeled sample of lookups, each with a stored candidate and a judgment of whether the stored answer is correct for the new request, and compute hit rate and wrong-hit rate at several thresholds. Do this per request class, since the cost of a wrong answer differs: a wrong answer about opening hours and one about eligibility are not equivalent. Choose a threshold per class and exclude classes where an error is expensive. When zero wrong hits are found in \(n\) audited hits, the rule of three puts the 95 percent upper bound on the error rate near \(3/n\), so 300 clean audited hits still allow an error rate of about 1 percent.

Neighbors

A prefix or context cache, offered by some model providers, reuses the processed form of an identical prompt prefix; the model still runs, so the answer is fresh. A semantic cache skips the model entirely, which is where both its saving and its risk come from.