Threads in Distributed Systems: How Servers Handle Many Requests at Once

A thread is a single stream of execution inside a process. Threads in the same process share memory, so passing data between them costs nothing. That is what makes them cheap compared with separate processes.

In a distributed system, threads solve a local problem. One node has many clients waiting on it. Threads let that one node work on many requests at the same time.

Here is the number that explains their value. Suppose a request spends 100 ms waiting on a database. One thread can then serve 10 requests per second. A pool of 200 threads serves about 2,000 requests per second on the same machine.

Nothing about that is distributed. The distributed part is what the threads do. Most of them are waiting for another machine to reply.

Threads, Processes and Nodes

These three words are often mixed up. They describe different levels.

LevelShares memoryWhat a failure affectsTypical cost
ThreadYes, with its processCan corrupt the whole processAbout 1 MB of stack
ProcessNoOne machine, one serviceTens of MB
NodeNoOne machineA full server

A thread crash is not contained. If one thread corrupts shared memory, every thread in that process is affected. This is the main reason some systems keep risky work in a separate process.

How a Multithreaded Server Works

There are three common designs, and each suits a different workload.

Thread per request creates a new thread for every incoming call. It is simple and it breaks under load. Creating 10,000 threads costs about 10 GB of stack memory.

A thread pool creates a fixed number of threads at start and reuses them. Requests wait in a queue when all threads are busy. This is the standard server design.

An event loop uses one thread per CPU core and never blocks. Work is handed back through callbacks. It handles many idle connections cheaply, but one slow piece of code stalls everything on that core.

Most server frameworks use a pool. The hard question is how large to make it.

For work that is mostly waiting, the pool can be large. For work that is mostly computing, the pool should be near the core count. A pool of 500 threads on 8 cores spends most of its time switching between threads, not running them.

Design Issues Threads Create in Distributed Computing

Threads bring four problems that show up in almost every design review.

Race conditions come first. Two threads write the same variable and the result depends on timing. The fix is a lock, and locks bring their own costs.

Deadlock comes second. Thread A holds lock 1 and wants lock 2. Thread B holds lock 2 and wants lock 1. Neither moves. The usual fix is to always take locks in the same order.

Thread pool exhaustion comes third, and it is the most common outage. A downstream service slows from 50 ms to 5 seconds. Every thread ends up waiting on it. The server stops answering healthy requests too.

Two guards prevent that. Set a timeout on every remote call. Give each downstream service its own small pool, so one slow service cannot take all the threads.

Debugging is the fourth problem. A bug that depends on timing may appear once in a million runs, and it will not appear on your laptop.

Threads Are Not Distributed Coordination

A lock inside a process only protects that process. Two nodes running the same code do not see each other's locks.

To protect a resource across machines, you need a distributed lock. That is a network call, so it is thousands of times slower than a local lock, and it can fail.

Treat local threading and distributed coordination as two separate topics. Mixing them is a common mistake in interviews.

How to Prepare

  • Define the term first. A thread is a stream of execution inside a process that shares memory with its siblings.
  • Size a pool out loud. Say whether the work waits or computes, then give a number and the reason.
  • Name thread pool exhaustion. Timeouts and separate pools per dependency are the answers interviewers listen for.
  • Reuse instead of recreate. The same idea drives connection pooling for databases.
  • Put threads in the wider picture. Read the challenges of designing a distributed system, then study Grokking System Design Fundamentals for the parts that surround a server.
TAGS
System Design Interview
System Design Fundamentals
CONTRIBUTOR
Arslan Ahmad
Arslan Ahmad
ex-FAANG engineering manager and author or Grokking series.

GET YOUR FREE

Coding Questions Catalog

Design Gurus Newsletter - Latest from our Blog
Boost your coding skills with our essential coding questions catalog.
Take a step towards a better tech career now!
Explore Answers
What Is the Toast Interview Process Like? (Round by Round)
Every stage of the Toast software engineer interview: recruiter screen, coding phone round, and a five-interview virtual onsite.
What is deadlock in OS?
What Are Time-series Database Best Practices?
Learn the best practices for time-series databases, including use cases, trade-offs, and pitfalls. Perfect for system design interview prep and real-world applications.
What is a technical skill example?
Which method is used to identify a thread?
Why Is Twilio Calling Me? And Who Is Twilio Ireland Limited?
Twilio does not call you itself; a business that uses Twilio's phone numbers is calling, and the name Twilio Ireland Limited is the account that owns the number in Europe.
Related Courses
New
Grokking Agentic Design Patterns course cover
Grokking Agentic Design Patterns
Design agents that act safely, recover from failure, and justify their cost.
4.9
(561 learners)
Discounted price for Your Region

$98

New
Grokking the AI System Design Interview course cover
Grokking the AI System Design Interview
Learn to design AI systems the way interviewers expect: classic ML products, LLM and RAG architectures, and agentic systems, all through the lens of the system design interview.
4.6
(3,192 learners)
Discounted price for Your Region

$98

Grokking the Coding Interview: Patterns for Coding Questions course cover
Grokking the Coding Interview: Patterns for Coding Questions
The 24 essential patterns behind every coding interview question. Available in Java, Python, JavaScript, C++, C#, and Go. The most comprehensive coding interview course with 543 lessons. A smarter alternative to grinding LeetCode.
4.6
Discounted price for Your Region

$197

Design Gurus logo
One-Stop Portal For Tech Interviews.
About Us
Our Team
Careers
Partners
Contact Us
Become Affiliate
Copyright © 2026 Design Gurus, LLC. All rights reserved.