Repository navigation
Description
When a stop-the-world is triggered, most commonly in a GC cycle, it can be
slow to take effect because it falls back to the timeout on its notification
event.
Stop-the-world has the following high-level steps before confirming all
threads have stopped:
- Set the
requestedflag in the interpreter state and count the threads
that need to stop. - Scan the threads. DETACHED threads are suspended and taken off the count;
ATTACHED threads have a stop bit set in theireval_breaker. - Wait on
PyEvent_WaitTimed(&stw->stop_event, 1 ms), then rescan on timeout.
The event is set when all threads have suspended, via a countdown mechanism
indecrement_stoptheworld_countdown.
A worker thread that is ATTACHED at step 2 and then detaches does not call
decrement_stoptheworld_countdown, so the stop_event wait times out.
queue.Queue is particularly at risk of this, since a thread blocking in
Queue.get() spends a fairly long time attached, with no eval-breaker check,
before it detaches. This makes it likely to still be ATTACHED at step 2.
import queue
import statistics
import sys
import threading
import time
done = False
def ping_pong(inbox, outbox):
while not done:
try:
outbox.put(inbox.get(timeout=0.1))
except queue.Empty:
pass
a, b = queue.Queue(), queue.Queue()
threads = [threading.Thread(target=ping_pong, args=(a, b)),
threading.Thread(target=ping_pong, args=(b, a))]
for t in threads:
t.start()
a.put("token")
limit = sys.getrecursionlimit()
times_us = []
for _ in range(1000):
t0 = time.perf_counter_ns()
sys.setrecursionlimit(limit) # stops the world
times_us.append((time.perf_counter_ns() - t0) / 1000)
time.sleep(0.0002)
done = True
for t in threads:
t.join()
slow = sum(t >= 900 for t in times_us)
print(f"GIL enabled: {sys._is_gil_enabled()}")
print(f"median stop: {statistics.median(times_us):7.1f} us")
print(f"stops that took the 1 ms poll: {slow} of {len(times_us)}")
GIL enabled: False
median stop: 1263.6 us
stops that took the 1 ms poll: 880 of 1000
One proposed solution is for a thread that detaches while a stop-the-world is
requested to suspend itself and call decrement_stoptheworld_countdown().
Detach is a critical code path, so I am unsure whether this was intentionally
left out. The cost when no stop is requested would be an atomic read of the
requested flag. In a prototype, the example above gives:
GIL enabled: False
median stop: 15.4 us
stops that took the 1 ms poll: 0 of 1000
Another option is to accept the current wait but tune the timeout, as the code
comment alludes to ("1ms (arbitrary, may need tuning)"). The trade-off is that
a shorter timeout means more wake-ups and rescans of the thread list while
threads are still running.
Activity