Semantic Caching

1. The Incident

The support assistant's inference bill runs about $2,000/day. An analysis of a week's queries produces the finding that launches this lesson. 40% of all questions are rephrasings of the same 200 or so questions. "Where's my order?" arrives in a thousand phrasings: "order status pls," "when will my package arrive," "has my stuff shipped." Each one pays full generation price, about $0.002 and 2.5 seconds. Each produces an answer the fleet has already written thousands of times today.