System Design Interview Crash Course

0% completed

​

Design ChatGPT

Here is a step-by-step system design for ChatGPT.

1. Problem Definition and Scope

We are designing a global-scale conversational AI system where users can chat with a Large Language Model (LLM). The system must manage conversation state, persistent history, and real-time text streaming.

  • Users: 100 Million+ Daily Active Users (DAU), split into Free and Plus (Paid) tiers.

  • Main Actions: Starting a chat, sending messages, receiving streaming responses, and viewing history.

  • Scope:

.....

.....

.....

Like the course? Get enrolled and start learning!
Rohit Bhanot

Rohit Bhanot

· 3 days ago

This part is very shallow and needs more details ?

  1. How does the Model GW streams back to Chat Service ?
  2. How and where does Chat Service is maintaining the connection state of clients? For Chat service to stream response back to Client via SSE it needs client connection details.
  3. While Chat Service is streaming back to Client, where are the response being stored temporarily ? You mentioned write to Dynamo only happens when Model GW sends "Done", so till then those msgs need to be stored somewhere ? Are these being stored in Context Cache and when "Done" is sent they are written to DB ?
  4. Is the Model GW talking directly to Chat Service ? That seems problematic, say if the Chat Service is down then we lose model response !

So can you please more details into the content ?