Grokking the AI System Design Interview
Vote

0% completed

​

Serving Optimizations

The Menu, Organized by Where It Works

When asked how to reduce latency and cost, organize the options by the part of the system they change.

Use three groups.

The first group avoids or reduces large-model work. The second improves the serving engine. The third changes the model representation or decoding method.

Start with the largest measured source of cost or latency. Avoiding a call helps only if the replacement still meets quality, freshness, and safety requirements.

Group One: Do Not Call the Big Model

**Semantic caching

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content