0% completed
IO Bound and CPU Bound Work
On This Page
Two Clocks
Classifying Harvest's Steps
What the Classification Predicts
Measuring Real Code
In Production
Key Takeaways
Practice Questions
Will threads make a given piece of Python code faster? You can answer this before you write any concurrent code. The answer depends on how the code spends its time: waiting for something outside the program, or executing instructions on the CPU.
Harvest's crawl step, which fetches a page and then indexes it, shows why this matters. Eight threads made that step faster, but far less than eight times faster. Two clocks read around the same code predict both the gain and its limit.
Two Clocks
Wall time is the time that passes on a clock while the code runs, including every moment it spends waiting. In Python you read it with time.perf_counter(), which is monotonic: it never goes backward, even when the system clock is corrected. time.time() can jump when the system clock is adjusted, so it is the wrong tool for durations.
CPU time is the time that processors spend executing a program's instructions, including system calls made for it. time.process_time() returns it for the whole process, added up over all of its threads. It does not advance while the process sleeps or waits, because no instruction runs during a wait. time.thread_time() returns the same measure for the calling thread only.
The CPU share of a piece of work is its CPU time divided by its wall time. A share near 0 means the work spent its time waiting. A share near 100 percent means one CPU was busy for the whole time.
This function measures both clocks around one piece of work:
def measure(work): wall, cpu = time.perf_counter(), time.process_time() work() return time.perf_counter() - wall, time.process_time() - cpu
The program classify.py calls it three times for each step of Harvest and prints the medians. The steps are the fetch and index functions from Concurrency and Parallelism in Python, with the same test site.
Classifying Harvest's Steps
A piece of work is IO bound when most of its time is spent waiting for input or output, such as a network reply, a disk read, or a database query. A piece of work is CPU bound when most of its time is spent executing instructions. The program labels a share under 20 percent as IO bound and over 80 percent as CPU bound; those limits are its own choice, not a standard. The run used Python 3.15.2 on a cloud virtual machine running Linux with 2 virtual CPUs (one physical core):
Output:
$ python3.15 classify.py fetch 1 page wall 0.26 s CPU 0.01 s CPU share 3% IO bound index 1 page wall 0.25 s CPU 0.25 s CPU share 100% CPU bound sleep 0.25 s wall 0.25 s CPU 0.00 s CPU share 0% IO bound crawl 8 pages wall 4.03 s CPU 2.02 s CPU share 50% mixed
Fetching was almost entirely the 250 millisecond wait for the reply. Indexing used the CPU for its whole wall time, because it never waits. The sleep line is a control: it shows that the CPU clock really stops during a wait.
The full crawl of eight pages has a share of 50 percent, and neither label fits it. It is two very different steps placed one after the other. Classify each step, not the whole program, because each step may need a different tool.
What the Classification Predicts
On the standard build, threads can overlap their waiting, but only one of them runs Python code at a time. Threads can therefore remove the waiting, but not overlap CPU time spent in Python code. Almost all of the crawl's CPU time, about 2.0 seconds, is spent in index, so about 2.0 seconds is the lowest wall time that threads can reach.
The program mixed.py runs the same crawl step one page at a time and then with a pool of eight threads:
def crawl(n): return index(fetch(n))
Output:
$ python3.15 mixed.py one at a time wall 4.04 s CPU 2.03 s CPU share 50% 8 threads wall 2.34 s CPU 2.11 s CPU share 90%
With eight threads, the crawl took 2.34 seconds, close to the predicted limit. The eight waits now happened during the same period, so nearly all of the waiting disappeared. The CPU work did not shrink.
The CPU share moved from 50 percent to 90 percent. Adding concurrency changed the classification. The threaded crawl is now CPU bound, so more threads will not help on the standard build. The next gain needs parallel indexing, through processes or the free-threaded build.
Measuring Real Code
Real code is rarely divided as cleanly as Harvest's steps, so measure at the level where you would make a decision:
- Time each stage of a request, such as the call to another service, the parsing, and the database write, rather than the whole handler.
- Repeat each measurement and use the median.
- In a threaded program, measure a step inside one thread with
thread_time(), becauseprocess_time()adds up every thread.
A CPU share can exceed 100 percent when several threads or processes compute in parallel. The ratio then becomes the average number of busy CPUs.
| CPU share of a step | Kind of work | What can make it faster |
|---|---|---|
| Under about 20 percent | IO bound | Overlapping the waits: threads or asyncio |
| About 20 to 80 percent | Mixed | Overlap the waits first, then measure again |
| Over about 80 percent | CPU bound | Parallel execution: processes, or threads on the free-threaded build |
In Production
| What you see | What it means | How to check |
|---|---|---|
A worker shows about 100 percent CPU in top, and more threads do not raise throughput | The worker is CPU bound, and its threads share one CPU on the standard build | A CPU share near 100 percent for one request confirms it |
| A service shows low CPU use, but requests are slow | The time goes to waiting for another service, a database, or a lock | Time each stage of one request with perf_counter() |
| A job became faster after adding threads, then stopped improving | The waits are overlapped, and the remaining CPU work sets the limit | The job's CPU share has risen toward 100 percent |
Key Takeaways
- Wall time, from
time.perf_counter(), counts waiting. CPU time, fromtime.process_time(), counts only executed instructions. - The CPU share classifies a step: Harvest's fetch was 3 percent and its indexing was 100 percent.
- Classify each step separately, because a 50 percent share for a whole program can hide an IO bound half and a CPU bound half.
- On the standard build, CPU time spent in Python code is the lowest wall time threads can reach: the crawl fell from 4.04 to 2.34 seconds, against 2.03 seconds of CPU time.
- Measure again after each change, because the kind of work can change with it.
The prediction above depended on one fact about the standard build: only one thread runs Python code at a time. The next lesson, What the GIL Protects, explains why CPython has that lock, what it keeps safe, and what it does not keep safe in your own code.
Practice Questions
Try each question first, then open the answer.
1. A function takes 1.20 seconds of wall time and 0.06 seconds of CPU time. Is it IO bound or CPU bound, and which tools could make it faster?
<details> <summary>Show answer</summary>It is IO bound. Its CPU share is 5 percent, so it spends almost all of its time waiting. Threads or asyncio can overlap the waits of several calls.
2. A batch job spends 6 seconds of wall time on each of 10 records: 2 seconds of CPU time in Python code, and the rest waiting for an API. On the standard build, with 10 threads, about how long should the batch take at best?
<details> <summary>Show answer</summary>About 20 seconds. The threads can overlap the waiting, but not the CPU work. The 10 records need 20 seconds of CPU time in total, against 60 seconds one at a time.
</details>3. After adding a thread pool, a job's CPU share rose from 50 percent to 92 percent. A teammate proposes doubling the number of threads. What do you expect on the standard build?
<details> <summary>Show answer</summary>Little or no improvement. The job is now CPU bound, and more threads only take turns on that work. The next gain needs a process pool or the free-threaded build.
</details>4. In a program with 8 threads, you time one thread's parsing step with time.process_time() and get a CPU time larger than the step's wall time. What went wrong?
process_time() adds up the CPU time of all threads in the process, so the other seven threads were counted too. Use time.thread_time() inside the worker thread instead.
Reading Progress
0%
On This Page
Two Clocks
Classifying Harvest's Steps
What the Classification Predicts
Measuring Real Code
In Production
Key Takeaways
Practice Questions