Design ChatGPT

This lesson designs ChatGPT, a website and app where people type a message and a language model writes back, one word at a time. The web part of this system is ordinary. The hard part is that each answer needs a scarce and expensive GPU for many seconds, so most decisions in this design are about using those GPUs well.

1. Problem Definition and Scope

We are designing a global chat system where users talk to a Large Language Model (LLM), which is a program trained on a very large amount of text to predict the next word