Grokking Modern API Design Interview
Vote

0% completed

​

Streaming LLM APIs

A person submits a question. The page remains empty for nine seconds. Then the entire answer appears at once.

The total work may be acceptable. The experience is not. The model produced useful output before the ninth second, but the API hid it until completion.

Streaming changes the response from one body into a sequence of typed events. That makes the first output visible sooner, and it also creates new contract questions: when is the response complete, how does failure appear after 200 has already been sent, what may the caller do with partial output, and when does charging stop?

.....

.....

.....

Like the course? Get enrolled and start learning!