Grokking the Modern Frontend System Design Interview
Vote

0% completed

​

Design an AI Chat Interface

A streaming response is an answer that the server sends in small pieces while it is still producing the rest. A language model writes its answer one token at a time, where a token is a short piece of text, about four characters of English on average. A streaming interface shows each piece as soon as it arrives.

The difference is large for the user. Suppose the model starts after 0.8 seconds and then produces 50 tokens per second, and the answer is 600 tokens long. Without streaming, the user watches a spinner for 0.8 + 600 / 50 = 12.8 seconds

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content