Serving Optimizations

1. The Menu, Organized by Where It Works

Interviewers ask some version of "this is too slow and too expensive, what do you do?" What they want back is an organized list, not a monologue. That list has three groups. The first group holds optimizations that avoid calling the big model at all. The second group holds optimizations inside the serving engine, the software that actually runs the model. The third group holds optimizations that change what the model computes. Learn them in that order, first group first, because avoiding a call is always better than making one faster.