Semantic Caching

1. The Incident

The support assistant's inference bill runs ~$5,000/day, and an analysis of a week's queries produces the finding that launches this lesson: 40% of all questions are rephrasings of the same ~200 questions. "Where's my order?" arrives in a thousand costumes: "order status pls," "when will my package arrive," "has my stuff shipped": each one paying full generation price, ~$0.002 and 2.5 seconds, to produce an answer the fleet has already written thousands of times today.

The team does the Module 4 reflex: cache-aside on the query string