9.1 Semantic Caching for LLM Responses

A support assistant receives two questions within one minute: "How do I reset my password?" and "I forgot my password, how can I reset it?" A cache keyed on the exact text of the question (an exact-match cache) treats them as two different keys. Both requests miss, so both pay for a new call to the large language model (LLM), the text-generating model behind the assistant.

A semantic cache keys on meaning instead of text