0% completed
The Four Models and How to Size Them
On This Page
The Four Models
Sizing Waiting Work: Little's Law
Sizing Computing Work: Amdahl's Law
A First Choice
In Production
Key Takeaways
Practice Questions
Two short calculations decide how many workers a pool needs. If Harvest must fetch 40 pages per second and each page takes 0.25 seconds to arrive, then 40 x 0.25 = 10 fetches must be in progress at every moment. No pool smaller than 10 workers can meet that rate. If 95 percent of an indexing job can run in parallel, then 16 workers can make it at most 9.1 times faster, because the remaining 5 percent runs on one worker.
One input to these calculations is often wrong: the count of CPUs that a machine reports is not always the number of workers that can compute at the same time.
The Four Models
Python has four ways to run work concurrently.
asyncio runs many tasks on one thread, using an event loop. An event loop is a loop that runs one task until the task has to wait, and then runs another task that is ready. Because it uses one thread, asyncio overlaps waiting but never runs Python code in parallel.
Threads are run by the operating system inside one process and share its memory. On the standard build they overlap waiting, and they run in parallel only inside C code that releases the GIL. On the free-threaded build they also run Python code in parallel.
Processes each run their own interpreter with its own GIL, so they run Python code in parallel on either build.
Subinterpreters are separate Python interpreters inside one process, each with its own modules, objects, and GIL. They run Python code in parallel like processes, while sharing one process like threads. The concurrent.interpreters module and InterpreterPoolExecutor were added in Python 3.14.
| Model | Runs Python code in parallel | Shares memory | Main cost |
|---|---|---|---|
asyncio | No | Yes, one thread | One blocking call stops every task |
| Threads | Free-threaded build only | Yes | Races on shared data |
| Processes | Yes | No, data is copied | Start-up time, memory, copying |
| Subinterpreters | Yes | No, data is passed between interpreters | Python 3.14 or later; some C extensions do not support them |
Choosing among them starts with the classification from IO Bound and CPU Bound Work. The worker count then comes from a different law for each kind of work.
Sizing Waiting Work: Little's Law
Little's law states that the average number of items in a system equals the rate at which they arrive multiplied by the average time each one spends in the system. For Harvest's fetching, a page is in flight from the moment it is requested until its reply has fully arrived. Pages in flight therefore equal pages per second multiplied by seconds per page. Each worker keeps at most one fetch in flight, so the worker count must be at least that number.
The program little.py fetches 32 pages from the test site, which answers each request after 250 milliseconds. It ran with 1, 4, 8, and 16 threads, and then with 16 asyncio tasks. It prints the median of three runs, with the number in flight that Little's law gives from the throughput and the time per page. All output in this lesson comes from Python 3.15.2 on a cloud virtual machine running Linux with 2 virtual CPUs (one physical core):
Output:
$ python3.15 little.py 32 pages, each answered after 250 ms 1 threads 8.20 s 3.9 pages/s 256 ms each in flight 1.0 4 threads 2.07 s 15.4 pages/s 258 ms each in flight 4.0 8 threads 1.08 s 29.6 pages/s 265 ms each in flight 7.8 16 threads 0.61 s 52.2 pages/s 288 ms each in flight 15.1 16 asyncio tasks 0.67 s 47.9 pages/s 329 ms each in flight 15.7
The number in flight followed the number of workers, and throughput followed the number in flight. At 16 workers the time per page rose to 288 milliseconds, so throughput grew a little less than the worker count.
Sixteen asyncio tasks kept 15.7 pages in flight on one thread. Each page took longer, 329 milliseconds, so throughput was slightly lower than with 16 threads. For waiting work, the number of fetches in flight sets the throughput, whichever model keeps them in flight.
Little's law gives a lower limit, not a target: the remote site, memory, and the CPU time per reply set other limits.
Sizing Computing Work: Amdahl's Law
Amdahl's law states that if a fraction p of a job can run in parallel, N workers can speed it up by at most 1 / ((1 - p) + p / N). With p equal to 0.95, the limit is 1.9 times for 2 workers, 3.5 for 4, 5.9 for 8, and 9.1 for 16.
In Concurrency and Parallelism in Python, eight processes indexed Harvest's pages 5.4 times faster on 8 cores. Amdahl's law counts every loss as serial work, so the indexing behaved as if about 7 percent of it ran on one worker, which includes copying pages to the workers and results back.
N is the number of workers that can really execute at the same time. For Python code, that is close to the number of physical cores, not the logical CPUs that os.cpu_count() reports. Simultaneous multithreading, called hyperthreading by Intel, lets one physical core execute two instruction streams at once by sharing its execution units between them. The operating system counts each stream as a logical CPU, and the second stream gains only from the parts of the core that the first leaves idle.
The Linux machine shows this directly:
Output:
$ lscpu CPU(s): 2 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1
Output:
$ python3.15 cores.py os.cpu_count() = 2 index, one at a time 1.68 s index, 2 processes 1.44 s speedup 1.17x index, 4 processes 1.45 s speedup 1.16x
The 2 logical CPUs are the two instruction streams of one physical core. Two processes gave a speedup of 1.17 times rather than 2. Four added nothing more, because the core can execute only two streams at once.
os.process_cpu_count(), added in Python 3.13, returns how many logical CPUs the current process may use. Like os.cpu_count(), it does not report physical cores, so check them with lscpu on Linux, or measure one worker against two as cores.py does.
A First Choice
The two laws give a starting point; measure every choice on the real workload.
| Kind of work | Model to start with | How many workers |
|---|---|---|
| IO bound, many waits | asyncio, or threads | Target rate times time per request (Little's law) |
| CPU bound, Python code | Processes, or threads on the free-threaded build | About the number of physical cores, then measure (Amdahl's law) |
| CPU bound, C code that releases the GIL | Threads | About the number of physical cores |
| Mixed | Separate the steps, and size each one on its own | Each step by its own law |
In Production
| What you see | What it means | How to check |
|---|---|---|
A process pool sized to os.cpu_count() is barely faster than half as many workers | The logical CPUs are hyperthreads, and only the physical cores add real capacity | Run lscpu; measure 1 worker against 2 |
| A fetch pool stops getting faster when you add threads | The remote service, the network, or the CPU that handles each reply is the limit | Print the time per request; if it grows with the worker count, one of those limits is saturated |
| A service misses its target rate although its CPU is idle | Too few requests are in flight | Multiply the target rate by the time per request to find the workers it needs |
Key Takeaways
asyncioand threads overlap waiting; processes, subinterpreters, and threads on the free-threaded build also run Python code in parallel.- Little's law sizes waiting work: workers needed equal the target rate times the time per request.
- Harvest's fetch throughput grew with the number in flight, from 3.9 pages per second with 1 thread to 52.2 with 16.
- Amdahl's law caps the speedup of computing work at a limit set by its serial part.
os.cpu_count()counts logical CPUs, but two processes on one core with two hyperthreads were only 1.17 times faster, so check the hardware or measure.
Threads appeared in every lesson of this chapter, but only as tools for a measurement, never as code that must start, stop, and fail cleanly. The next lesson, Thread Lifecycle, shows how a thread is started, named, and joined, and what happens when one fails.
Practice Questions
Try each question first, then open the answer.
1. Harvest must fetch 60 pages per second from a site that answers each request in 0.4 seconds. What is the smallest thread pool that could do it?
<details> <summary>Show answer</summary>24 threads. Little's law gives 60 x 0.4 = 24 fetches in flight, and each thread keeps one in flight. Measure it, because the time per request can grow with the pool.
</details>2. With 16 asyncio tasks on one thread, Harvest kept 15.7 pages in flight. How can one thread do that?
<details> <summary>Show answer</summary>A fetch spends almost all of its time waiting, and waiting needs no thread. The event loop runs other tasks while each reply is pending.
</details>3. A job is 90 percent parallel. What is the most that 4 workers and 16 workers can speed it up?
<details> <summary>Show answer</summary>About 3.1 times and 6.4 times. Amdahl's law gives 1 / (0.1 + 0.225) for 4 workers and 1 / (0.1 + 0.05625) for 16. The serial 10 percent caps the speedup at 10 times.
</details>4. A CPU-bound Python service runs on a server with 16 logical CPUs and 8 cores. A teammate sizes its process pool with os.cpu_count(). What would you do first?
Measure the pool at 8 and at 16 workers. CPU-bound Python code gets most of its speedup from the 8 physical cores. If 16 are not clearly faster, use 8, which needs less memory.
</details>Reading Progress
0%
On This Page
The Four Models
Sizing Waiting Work: Little's Law
Sizing Computing Work: Amdahl's Law
A First Choice
In Production
Key Takeaways
Practice Questions