Grokking Concurrency in Python
Vote

0% completed

​

The Four Models and How to Size Them

The Four Models

Sizing Waiting Work: Little's Law

Sizing Computing Work: Amdahl's Law

A First Choice

In Production

Key Takeaways

Practice Questions

The Four Models and How to Size Them
The Four Models and How to Size Them

Two short calculations decide how many workers a pool needs. If Harvest must fetch 40 pages per second and each page takes 0.25 seconds to arrive, then 40 x 0.25 = 10 fetches must be in progress at every moment. No pool smaller than 10 workers can meet that rate. If 95 percent of an indexing job can run in parallel, then 16 workers can make it at most 9.1 times faster, because the remaining 5 percent runs on one worker.

One input to these calculations is often wrong: the count of CPUs that a machine reports is not always the number of workers that can compute at the same time.

The Four Models

Python has four ways to run work concurrently.

asyncio runs many tasks on one thread, using an event loop. An event loop is a loop that runs one task until the task has to wait, and then runs another task that is ready. Because it uses one thread, asyncio overlaps waiting but never runs Python code in parallel.

Threads are run by the operating system inside one process and share its memory. On the standard build they overlap waiting, and they run in parallel only inside C code that releases the GIL. On the free-threaded build they also run Python code in parallel.

Processes each run their own interpreter with its own GIL, so they run Python code in parallel on either build.

Subinterpreters are separate Python interpreters inside one process, each with its own modules, objects, and GIL. They run Python code in parallel like processes, while sharing one process like threads. The concurrent.interpreters module and InterpreterPoolExecutor were added in Python 3.14.

ModelRuns Python code in parallelShares memoryMain cost
asyncioNoYes, one threadOne blocking call stops every task
ThreadsFree-threaded build onlyYesRaces on shared data
ProcessesYesNo, data is copiedStart-up time, memory, copying
SubinterpretersYesNo, data is passed between interpretersPython 3.14 or later; some C extensions do not support them

Choosing among them starts with the classification from IO Bound and CPU Bound Work. The worker count then comes from a different law for each kind of work.

Sizing Waiting Work: Little's Law

Little's law states that the average number of items in a system equals the rate at which they arrive multiplied by the average time each one spends in the system. For Harvest's fetching, a page is in flight from the moment it is requested until its reply has fully arrived. Pages in flight therefore equal pages per second multiplied by seconds per page. Each worker keeps at most one fetch in flight, so the worker count must be at least that number.

The program little.py fetches 32 pages from the test site, which answers each request after 250 milliseconds. It ran with 1, 4, 8, and 16 threads, and then with 16 asyncio tasks. It prints the median of three runs, with the number in flight that Little's law gives from the throughput and the time per page. All output in this lesson comes from Python 3.15.2 on a cloud virtual machine running Linux with 2 virtual CPUs (one physical core):

Output:

$ python3.15 little.py 32 pages, each answered after 250 ms 1 threads 8.20 s 3.9 pages/s 256 ms each in flight 1.0 4 threads 2.07 s 15.4 pages/s 258 ms each in flight 4.0 8 threads 1.08 s 29.6 pages/s 265 ms each in flight 7.8 16 threads 0.61 s 52.2 pages/s 288 ms each in flight 15.1 16 asyncio tasks 0.67 s 47.9 pages/s 329 ms each in flight 15.7

The number in flight followed the number of workers, and throughput followed the number in flight. At 16 workers the time per page rose to 288 milliseconds, so throughput grew a little less than the worker count.

Sixteen asyncio tasks kept 15.7 pages in flight on one thread. Each page took longer, 329 milliseconds, so throughput was slightly lower than with 16 threads. For waiting work, the number of fetches in flight sets the throughput, whichever model keeps them in flight.

Little's law on Harvest's fetches
Little's law on Harvest's fetches

Little's law gives a lower limit, not a target: the remote site, memory, and the CPU time per reply set other limits.

Sizing Computing Work: Amdahl's Law

Amdahl's law states that if a fraction p of a job can run in parallel, N workers can speed it up by at most 1 / ((1 - p) + p / N). With p equal to 0.95, the limit is 1.9 times for 2 workers, 3.5 for 4, 5.9 for 8, and 9.1 for 16.

In Concurrency and Parallelism in Python, eight processes indexed Harvest's pages 5.4 times faster on 8 cores. Amdahl's law counts every loss as serial work, so the indexing behaved as if about 7 percent of it ran on one worker, which includes copying pages to the workers and results back.

N is the number of workers that can really execute at the same time. For Python code, that is close to the number of physical cores, not the logical CPUs that os.cpu_count() reports. Simultaneous multithreading, called hyperthreading by Intel, lets one physical core execute two instruction streams at once by sharing its execution units between them. The operating system counts each stream as a logical CPU, and the second stream gains only from the parts of the core that the first leaves idle.

The Linux machine shows this directly:

Output:

$ lscpu CPU(s): 2 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1

Output:

$ python3.15 cores.py os.cpu_count() = 2 index, one at a time 1.68 s index, 2 processes 1.44 s speedup 1.17x index, 4 processes 1.45 s speedup 1.16x

The 2 logical CPUs are the two instruction streams of one physical core. Two processes gave a speedup of 1.17 times rather than 2. Four added nothing more, because the core can execute only two streams at once.

Two logical CPUs, one core
Two logical CPUs, one core

os.process_cpu_count(), added in Python 3.13, returns how many logical CPUs the current process may use. Like os.cpu_count(), it does not report physical cores, so check them with lscpu on Linux, or measure one worker against two as cores.py does.

A First Choice

The two laws give a starting point; measure every choice on the real workload.

Kind of workModel to start withHow many workers
IO bound, many waitsasyncio, or threadsTarget rate times time per request (Little's law)
CPU bound, Python codeProcesses, or threads on the free-threaded buildAbout the number of physical cores, then measure (Amdahl's law)
CPU bound, C code that releases the GILThreadsAbout the number of physical cores
MixedSeparate the steps, and size each one on its ownEach step by its own law

In Production

What you seeWhat it meansHow to check
A process pool sized to os.cpu_count() is barely faster than half as many workersThe logical CPUs are hyperthreads, and only the physical cores add real capacityRun lscpu; measure 1 worker against 2
A fetch pool stops getting faster when you add threadsThe remote service, the network, or the CPU that handles each reply is the limitPrint the time per request; if it grows with the worker count, one of those limits is saturated
A service misses its target rate although its CPU is idleToo few requests are in flightMultiply the target rate by the time per request to find the workers it needs

Key Takeaways

  • asyncio and threads overlap waiting; processes, subinterpreters, and threads on the free-threaded build also run Python code in parallel.
  • Little's law sizes waiting work: workers needed equal the target rate times the time per request.
  • Harvest's fetch throughput grew with the number in flight, from 3.9 pages per second with 1 thread to 52.2 with 16.
  • Amdahl's law caps the speedup of computing work at a limit set by its serial part.
  • os.cpu_count() counts logical CPUs, but two processes on one core with two hyperthreads were only 1.17 times faster, so check the hardware or measure.

Threads appeared in every lesson of this chapter, but only as tools for a measurement, never as code that must start, stop, and fail cleanly. The next lesson, Thread Lifecycle, shows how a thread is started, named, and joined, and what happens when one fails.

Practice Questions

Try each question first, then open the answer.

1. Harvest must fetch 60 pages per second from a site that answers each request in 0.4 seconds. What is the smallest thread pool that could do it?

<details> <summary>Show answer</summary>

24 threads. Little's law gives 60 x 0.4 = 24 fetches in flight, and each thread keeps one in flight. Measure it, because the time per request can grow with the pool.

</details>

2. With 16 asyncio tasks on one thread, Harvest kept 15.7 pages in flight. How can one thread do that?

<details> <summary>Show answer</summary>

A fetch spends almost all of its time waiting, and waiting needs no thread. The event loop runs other tasks while each reply is pending.

</details>

3. A job is 90 percent parallel. What is the most that 4 workers and 16 workers can speed it up?

<details> <summary>Show answer</summary>

About 3.1 times and 6.4 times. Amdahl's law gives 1 / (0.1 + 0.225) for 4 workers and 1 / (0.1 + 0.05625) for 16. The serial 10 percent caps the speedup at 10 times.

</details>

4. A CPU-bound Python service runs on a server with 16 logical CPUs and 8 cores. A teammate sizes its process pool with os.cpu_count(). What would you do first?

<details> <summary>Show answer</summary>

Measure the pool at 8 and at 16 workers. CPU-bound Python code gets most of its speedup from the 8 physical cores. If 16 are not clearly faster, use 8, which needs less memory.

</details>

Reading Progress

0%


Vote for new content

On This Page

The Four Models

Sizing Waiting Work: Little's Law

Sizing Computing Work: Amdahl's Law

A First Choice

In Production

Key Takeaways

Practice Questions