Grokking Concurrency in Python
Vote

0% completed

​

Locks and Lock Ordering

Lock

RLock

Deadlock From Two Locks

A Fixed Lock Order

A Timeout as a Detector

In Production

Key Takeaways

Practice Questions

Locks and Lock Ordering
Locks and Lock Ordering

Two people share one pen and one notebook, and each needs both to write a note. One picks up the pen and waits for the notebook, while the other picks up the notebook and waits for the pen. Neither puts down what they hold, so both wait forever.

Threads that need two locks can stop in exactly this way, with no exception and no error in the log. The threading module has two lock types, Lock and RLock, and one rule about acquiring them prevents this failure.

Lock

A threading.Lock is an object with two states, locked and unlocked: acquire() locks it, waiting first if another thread holds it, and release() unlocks it. The usual way to use a lock is the with statement, which releases the lock at the end of the block even when the block raises an exception. Calling acquire() and release() by hand needs a try and finally for the same guarantee, or the first exception leaves the lock locked forever.

The demo added 1 to a counter a million times, first without a lock and then inside with lock:. The runs used Python 3.15.2 on a Linux virtual machine with 2 virtual CPUs (one physical core):

Output:

$ python3.15 locks.py cost 1,000,000 increments: 0.012 s bare, 0.043 s with a lock, 30 ns extra per increment $ python3.15t locks.py cost 1,000,000 increments: 0.013 s bare, 0.045 s with a lock, 32 ns extra per increment

About 30 nanoseconds per acquire and release is far less than any page fetch. The real cost of a lock is waiting. Lock contention is the time threads spend waiting for a lock that another thread holds. It grows with the time each thread holds the lock, which is why Race Conditions in Python Code kept fetching and counting outside the locked block.

RLock

Harvest's index has two methods that protect the word counts with one lock. add_page holds the lock while it calls add_word, which tries to take the same lock:

class Index: def __init__(self, lock): self.lock = lock self.counts = {} def add_word(self, key): with self.lock: self.counts[key] = self.counts.get(key, 0) + 1 def add_page(self, text): with self.lock: for key in index(text): self.add_word(key)

The demo ran add_page in a thread, once with each type of lock, and waited one second for it:

Output:

$ python3.15 locks.py reentry Lock: after 1.0 s the indexer is still blocked RLock: after 1.0 s the indexer is finished, 3 words indexed

A Lock does not record which thread acquired it, so the second acquire() waits like any other, for a release that only the waiting thread could perform. An RLock, or reentrant lock, is a lock that the thread holding it can acquire again; it unlocks only after that thread has released it as many times as it acquired it.

A cleaner design is often a private method, such as _add_word, that assumes the caller already holds the lock, with the lock taken only in the public methods. Use RLock when that split is not practical.

Deadlock From Two Locks

Harvest now has visited_lock, which protects the set of visited pages, and index_lock, which protects the index. The crawler takes visited_lock, fetches a page, and then takes index_lock. A compactor thread, which cleans up the index, takes the two locks in the opposite order:

def crawler(): with visited_lock: say("has visited_lock, fetching page 1") text = fetch(1) say("wants index_lock") with index_lock: index(text) say("done") def compactor(): with index_lock: say("has index_lock, fetching page 2") text = fetch(2) say("wants visited_lock") with visited_lock: index(text) say("done")

A deadlock is a state in which each thread in a group waits for a lock held by another thread in the same group, so none of them can ever continue.

Output:

$ python3.15 locks.py deadlock 0.00 s crawler has visited_lock, fetching page 1 0.00 s compactor has index_lock, fetching page 2 0.25 s crawler wants index_lock 0.26 s compactor wants visited_lock after 2.00 s, threads still waiting: ['crawler', 'compactor']

The two wants lines were the last lines the threads ever printed. The demo made them daemon threads only so that it could stop waiting after 2.00 seconds and exit; in a real service they would wait forever. Holding a lock during a 250 millisecond fetch made the deadlock certain here. With shorter work inside the locks, the same code would deadlock only now and then, which makes it much harder to find.

Two threads, two locks, one cycle
Two threads, two locks, one cycle

A Fixed Lock Order

A deadlock needs a cycle: thread A waits for B's lock while B waits for A's lock. Lock ordering means that every thread which needs more than one lock acquires them in the same agreed order. If every thread takes visited_lock before index_lock, no thread can hold index_lock while it waits for visited_lock.

def compactor_ordered(): with visited_lock: say("has visited_lock, fetching page 2") text = fetch(2) say("wants index_lock") with index_lock: index(text) say("done")

Output:

$ python3.15 locks.py ordered 0.00 s crawler has visited_lock, fetching page 1 0.25 s crawler wants index_lock 0.26 s crawler done 0.26 s compactor has visited_lock, fetching page 2 0.51 s compactor wants index_lock 0.52 s compactor done after 0.52 s, threads still waiting: none

The compactor waited at its first lock until the crawler was done, and both finished in 0.52 seconds. The two fetches ran one after the other because each was inside a lock; moving fetch() out would let them overlap.

When there are many locks of one kind, such as one lock per website, the order must come from the data. Sort the locks by a stable key, such as the site name, and write the rule down next to the locks, because it works only if every thread follows it.

A Timeout as a Detector

acquire() takes a timeout and returns False if the lock did not become free in time. The crawler below uses it for its second lock:

if not index_lock.acquire(timeout=1.0): say("gave up on index_lock after 1.0 s") return try: index(text) finally: index_lock.release()

Output:

$ python3.15 locks.py timeout 0.00 s crawler has visited_lock, fetching page 1 0.00 s compactor has index_lock, fetching page 2 0.25 s compactor wants visited_lock 0.25 s crawler wants index_lock 1.25 s crawler gave up on index_lock after 1.0 s 1.26 s compactor done after 1.26 s, threads still waiting: none

The program no longer hangs. But page 1 was never indexed, and the compactor finished only because the crawler gave up and released visited_lock. A timeout turns a silent deadlock into a visible error, and the repair is still a fixed lock order.

In Production

What you seeWhat it meansHow to check
A service stops responding, its threads are alive, and CPU use is near zeroThreads are deadlocked on each other's locksTake a thread dump and look for threads waiting in acquire or with on different locks
A thread hangs in a method that calls another method of the same objectOne thread acquires a plain Lock twiceFind the nested with self.lock:; split out a private method or use RLock
Adding threads makes a service slowerLock contention from slow work inside a lockTime how long the lock is held; move I/O and heavy computation out

Key Takeaways

  • Use a lock through with, so it is released even when the block raises.
  • An uncontended lock costs about 30 nanoseconds; the real cost is contention.
  • A plain Lock acquired twice by the same thread blocks forever. An RLock allows it.
  • Two threads that took two locks in opposite orders were deadlocked after 0.26 seconds. Acquire locks in one agreed order everywhere.
  • acquire(timeout=...) makes a deadlock visible, but only a fixed order prevents it.

A lock answers one question: may this thread touch the data now? Harvest also needs threads to wait for a page to arrive, a stop request, or a free slot on a busy website. The next lesson, Conditions, Events, Semaphores and Barriers, covers the four tools in threading built for those waits.

Practice Questions

Try each question first, then open the answer.

1. A method holds self.lock and calls a helper that also uses with self.lock:. The thread hangs. Why, and what are two fixes?

<details> <summary>Show answer</summary>

A plain Lock does not know which thread holds it, so the second acquire waits for a release that only the waiting thread could perform. Use an RLock, or make the helper a private method that assumes the caller holds the lock.

</details>

2. Thread A takes lock X, then Y. Thread B takes Y, then X. Tests never hang, but production hangs about once a week. Why is it rare, and what removes it?

<details> <summary>Show answer</summary>

A deadlock needs both threads to take their first lock before either takes its second, and with short locked sections that timing is rare. Making both threads take X before Y removes the cycle completely.

</details>

3. A job that moves links between two websites must hold both sites' locks. How do you choose the order?

<details> <summary>Show answer</summary>

Use an order that comes from the data, such as sorting the two locks by site name. Jobs moving links from A to B and from B to A then both take A's lock first.

</details>

4. A teammate adds acquire(timeout=1.0) to one thread, and the hangs stop. Why is this not a complete fix?

<details> <summary>Show answer</summary>

The thread that gives up does not finish its work, and the opposite lock order is still in the code. In the demo, page 1 was never indexed. Only a fixed lock order removes the cycle.

</details>

Reading Progress

0%


Vote for new content

On This Page

Lock

RLock

Deadlock From Two Locks

A Fixed Lock Order

A Timeout as a Detector

In Production

Key Takeaways

Practice Questions