Grokking Concurrency in Python
Vote

0% completed

​

Producer and Consumer With a Queue

Producers, Consumers, and a Queue

A Bounded Queue

Ending With Sentinels

Queue.shutdown()

Joining in the Right Order

In Production

Key Takeaways

Practice Questions

Producer and Consumer With a Queue
Producer and Consumer With a Queue
q = queue.Queue() if mode == "unbounded" else queue.Queue(maxsize=4)

That line sets how much memory Harvest's pipeline can use. With queue.Queue(), up to 33 fetched pages sat in memory at once, waiting to be indexed. With queue.Queue(maxsize=4), never more than 4 did, and the crawl took the same time: 2.18 seconds against 2.19.

The queue also decides how the threads stop, and the wrong number of stop messages left one thread waiting forever.

Producers, Consumers, and a Queue

A producer is a thread that creates work items, and a consumer is a thread that processes them. In Harvest, eight fetch threads produce pages and two index threads consume them. queue.Queue is a first-in, first-out queue that many threads can use at once: put() adds an item and get() removes the oldest one, waiting until an item is available. Inside, it is a collections.deque, a sequence that is fast to change at both ends, protected by one lock and three condition variables, the tools from Conditions, Events, Semaphores and Barriers, already written and tested.

def fetcher(q, numbers): for n in numbers: q.put(fetch(n)) note_depth(q) def indexer_sentinel(q): while True: page = q.get() if page is None: return save(index(page))

note_depth records the largest queue size seen, and save stands in for writing the counts to a search store, with a 100 millisecond pause. The demos ran with Python 3.15.2 on a Linux virtual machine with 2 virtual CPUs (one physical core). The test site answers after 50 milliseconds, so fetching is much faster than indexing and saving. The crawl covers 39 pages, because page 5 is missing.

A Bounded Queue

A bounded queue has a maximum size, and put() waits while the queue is full. Backpressure is a slow stage making a fast stage wait, so that unfinished work does not accumulate between them.

Output:

$ python3.15 pipeline.py unbounded unbounded indexed 39 of 39 pages in 2.18 s, queue peak 33 exit status: 0 $ python3.15 pipeline.py bounded bounded indexed 39 of 39 pages in 2.19 s, queue peak 4 exit status: 0

Without a bound, the fetchers ran far ahead of the two indexers. With it, they waited at put() instead, and the crawl took the same time, because the indexers were the slow stage either way. On a crawl of millions of pages, an unbounded queue keeps growing and can use up the process's memory.

A bounded queue between fetch and index
A bounded queue between fetch and index

Choose the size from the memory each item uses and how long the producers may pause. A few items per consumer is usually enough.

Ending With Sentinels

get() waits whenever the queue is empty, so an index thread needs a message that says no more pages will come. A sentinel is a special value, such as None, that a producer puts on the queue to tell a consumer to stop. Each consumer stops after reading one sentinel, and get() removes that sentinel from the queue, so the main thread must put exactly one per consumer, after all producers have finished.

The third run put one sentinel too few, for two index threads:

Output:

$ python3.15 pipeline.py one-sentinel-short one-sentinel-short indexed 39 of 39 pages in 3.88 s, queue peak 4, still running: ['index-0'] exit status: 1

Every page was indexed, but index-0 was still waiting in get(). The demo made the index threads daemon threads and joined them with a two-second timeout, as described in Thread Lifecycle, so it could report the stuck thread; otherwise the program would hang at its end. Adding a consumer, or a consumer that exits early on an error, changes the number of sentinels needed.

Queue.shutdown()

Queue.shutdown(), added in Python 3.13, closes a queue: later calls to put() raise queue.ShutDown, and get() returns the items still in the queue, then raises queue.ShutDown once it is empty. Every consumer stops, however many there are:

def indexer_shutdown(q): while True: try: page = q.get() except queue.ShutDown: return save(index(page))

Output:

$ python3.15 pipeline.py shutdown shutdown indexed 39 of 39 pages in 2.18 s, queue peak 4 exit status: 0

shutdown(immediate=True) also removes the items still in the queue, so get() raises queue.ShutDown at once. That suits a stop button. The demo stopped the crawl after 0.3 seconds, with fetchers that end when put() raises:

def fetcher_stoppable(q, numbers): for n in numbers: try: q.put(fetch(n)) note_depth(q) except queue.ShutDown: return

Output:

$ python3.15 pipeline.py immediate immediate indexed 6 of 39 pages in 0.38 s, queue peak 4, 4 queued pages dropped exit status: 0

The four queued pages were dropped. The index threads finished the pages they already held, so 6 pages were indexed. Dropped pages must be fetched again later if they matter.

Joining in the Right Order

The main thread of the pipeline follows one order:

  1. Start the consumers and the producers.
  2. Join every producer, so no more items will be put.
  3. Signal the end: q.shutdown(), or one sentinel per consumer.
  4. Join every consumer, so all items have been processed.

Signaling before step 2 loses items: after shutdown(), the producers' put() calls raise queue.ShutDown, and with sentinels, late items stay unprocessed in the queue, where on a bounded queue the producers then wait in put() forever. Skipping step 3 leaves the consumers waiting in get(). A consumer can also call task_done() after each item, and q.join() then waits until every item has been marked done. That call does not stop the consumer threads, so step 3 is still needed.

In Production

What you seeWhat it meansHow to check
Memory grows during a long job, then the process is killedAn unbounded queue sits in front of a slow stageLog q.qsize() every few seconds; give the queue a maxsize
A job finishes its work but the process never exitsA consumer is still waiting in get()List live threads at exit; use q.shutdown() instead of counting sentinels
Items are lost during a normal shutdownThe end was signaled before all producers were joinedJoin producers before shutdown() or the sentinels
Producers spend most of their time in put()Backpressure: the consumers are the slow stageAdd consumers or make them faster; a bigger queue only delays the wait

Key Takeaways

  • A queue.Queue connects producers and consumers without any locks in your own code.
  • A bounded queue cut the peak from 33 pages to 4 at the same total time of about 2.2 seconds.
  • A sentinel stops one consumer, and one too few left a thread waiting forever.
  • Queue.shutdown() stops every consumer once the queue is empty, and immediate=True also drops queued items.
  • Join the producers, then signal the end, then join the consumers.

A queue of work with a fixed set of threads taking items from it is the core of a thread pool. The next lesson, Build a Thread Pool, builds a small pool from a queue and worker threads, then compares it with the pool in the standard library.

Practice Questions

Try each question first, then open the answer.

1. Harvest's bounded and unbounded pipelines both took about 2.2 seconds. What did the bound change, and why did it not slow the crawl?

<details> <summary>Show answer</summary>

It cut the peak number of waiting pages from 33 to 4. The indexers were the slow stage, so fetching ahead of them only filled memory.

</details>

2. A pipeline has three consumers and ends with sentinels. A teammate adds a fourth consumer, and the program stops exiting. Why, and what prevents it?

<details> <summary>Show answer</summary>

The main thread still puts three sentinels, so the fourth consumer waits in get() forever. With q.shutdown(), every consumer gets queue.ShutDown once the queue is empty, however many there are.

</details>

3. A main thread calls q.shutdown() right after starting its producers, then joins everything. What goes wrong?

<details> <summary>Show answer</summary>

The producers' later put() calls raise queue.ShutDown, so those pages never reach the consumers. Signal the end only after every producer has been joined.

</details>

4. Producers spend most of their time blocked in put() on a queue of size 10. Will raising the size to 10,000 make the service faster?

<details> <summary>Show answer</summary>

No. The consumers are slower, and a bigger queue only lets more items accumulate before the same wait begins. Add consumers or make each one faster.

</details>

Reading Progress

0%


Vote for new content

On This Page

Producers, Consumers, and a Queue

A Bounded Queue

Ending With Sentinels

Queue.shutdown()

Joining in the Right Order

In Production

Key Takeaways

Practice Questions