Skip to content

gh-83069: Use efficient event-driven subprocess.Popen.wait() on Linux / macOS / BSD #144047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vstinner merged 68 commits into from
Jan 28, 2026
Merged

gh-83069: Use efficient event-driven subprocess.Popen.wait() on Linux / macOS / BSD #144047

Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
288e4db
Use pidfd_open() and save the FD as an attribute
giampaolo Jan 18, 2026
6e494ca
Create _busy_wait() and move code in there
giampaolo Jan 18, 2026
aaf193c
Move existing code in new _blocking_wait() method
giampaolo Jan 18, 2026
7cbb0ad
Use _wait_pidfd()
giampaolo Jan 18, 2026
d9760c3
Use pidfd_open() not at class level but at method level
giampaolo Jan 18, 2026
5fd8eec
Add kqueue() implementation for macOS and BSD
giampaolo Jan 18, 2026
100b111
Add docstrings
giampaolo Jan 18, 2026
74ac2f4
Be conservative and check for specific error codes
giampaolo Jan 18, 2026
fc0cfd6
Document possible failures
giampaolo Jan 18, 2026
f282459
Add missing import
giampaolo Jan 18, 2026
6125976
Write test for pidfd_open() failing
giampaolo Jan 18, 2026
41dc6c0
Write test case for pidfd_open() / kqueue failing. Assert fallback is…
giampaolo Jan 18, 2026
73ba380
Write test case for kqueue failing. Assert fallback is used.
giampaolo Jan 18, 2026
3c8e603
Move tests in their own class
giampaolo Jan 18, 2026
932ae58
Add test for terminated PID race
giampaolo Jan 18, 2026
bb5080a
Add test_kqueue_race()
giampaolo Jan 19, 2026
f067073
Add test_kqueue_control_error
giampaolo Jan 19, 2026
4d96c2f
Add docstring
giampaolo Jan 19, 2026
fe05acc
Guard against possible slow test
giampaolo Jan 19, 2026
adb444e
Timeout: use math.ceil to avoid truncation
giampaolo Jan 19, 2026
4ec17c1
Remove unused exception var
giampaolo Jan 19, 2026
e807ba9
Timeout: use math.ceil to avoid truncation
giampaolo Jan 19, 2026
1ddc52b
Replace _can_use_kqueue() -> _CAN_USE_KQUEUE
giampaolo Jan 19, 2026
645ef6c
Shorten code
giampaolo Jan 19, 2026
6ee771b
Use waitpid() + WNOHANG even if process exited to avoid rare race
giampaolo Jan 19, 2026
61c6b99
Revert prev change
giampaolo Jan 19, 2026
527646d
Use ceil(timeout) to avoid truncation
giampaolo Jan 19, 2026
0a8a1b2
Remove check for timeout < 0 + rm test case which didn't make sense.
giampaolo Jan 19, 2026
73b97dc
Don't use _blocking_wait() as it has a while loop
giampaolo Jan 19, 2026
2d3c3f7
Add docstring
giampaolo Jan 19, 2026
4eac42f
Rm _busy_wait()
giampaolo Jan 19, 2026
3c156a9
Add comment
giampaolo Jan 19, 2026
dac7d3b
Add assert
giampaolo Jan 19, 2026
5c7ec2f
Update comments
giampaolo Jan 19, 2026
5c29144
Add test for timeout=0
giampaolo Jan 19, 2026
4359b07
Update comment about PID reuse race
giampaolo Jan 19, 2026
81275c8
Update comment
giampaolo Jan 19, 2026
43b500f
Handle rare case where poll() says we're done, but waitpid() doesn't
giampaolo Jan 19, 2026
b64e42b
Update Doc/library/subprocess.rst
giampaolo Jan 19, 2026
a101406
Add news entry
giampaolo Jan 19, 2026
452f8c4
Add entry in Doc/whatsnew/3.15.rst
giampaolo Jan 19, 2026
b0c9890
Fix typo
giampaolo Jan 19, 2026
27b7c9f
Re-wording
giampaolo Jan 19, 2026
e1da996
Raise on timeout < 0 and re-add test case
giampaolo Jan 19, 2026
5c78acc
Check if can really use can_use_pidfd() in unit tests
giampaolo Jan 19, 2026
df0538a
Check if can really use kqueue() in unit tests
giampaolo Jan 19, 2026
6d8e36c
Pre-emptively check whether to use the fast way methods
giampaolo Jan 19, 2026
6ba7465
Add test_fast_path_avoid_busy_loop
giampaolo Jan 19, 2026
e3c7977
Update comments
giampaolo Jan 19, 2026
97cc3be
Fix missing import on Windows
giampaolo Jan 19, 2026
c4342e3
Merge branch 'main' into subprocess-fast-wait
giampaolo Jan 19, 2026
5d78d24
Try to fix doc build error
giampaolo Jan 19, 2026
a916da3
Try to fix doc build error 2
giampaolo Jan 19, 2026
86200bd
Try to fix doc build error 3
giampaolo Jan 19, 2026
85c38bc
Try to fix doc build error 4
giampaolo Jan 19, 2026
3c92c1d
Try to fix doc build error 5
giampaolo Jan 19, 2026
33c8b1f
Minor rewordings
giampaolo Jan 19, 2026
525c047
Small refact
giampaolo Jan 21, 2026
2b931b8
Address review comments (bare assert, don't use ceil())
giampaolo Jan 21, 2026
d1c6e91
Address review comments (use test class VARIABLES)
giampaolo Jan 21, 2026
3e9d303
Address review comments (add periods in doc)
giampaolo Jan 21, 2026
a44b2d7
Merge branch 'main' into subprocess-fast-wait
giampaolo Jan 21, 2026
b4f6020
Remove weird unicode char from doc added by accident
giampaolo Jan 21, 2026
4e17856
Don't import select on Windows
giampaolo Jan 21, 2026
6f6600d
Make kq.control() invocation clearer by passing kwargs
giampaolo Jan 21, 2026
37288c6
Revert previous commit
giampaolo Jan 21, 2026
300692e
Remove bare assert statements
giampaolo Jan 22, 2026
7bee1e9
Merge branch 'main' into subprocess-fast-wait
giampaolo Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address review comments (add periods in doc)
  • Loading branch information
@giampaolo
giampaolo committed Jan 21, 2026
commit 3e9d303b06df69f22bac5ad63a250a8668a72c53
6 changes: 3 additions & 3 deletions Doc/whatsnew/3.15.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -743,9 +743,9 @@ subprocess
platform supports it, an efficient event-driven mechanism is used to wait for
process termination:

- Linux ≥= 5.3 uses :func:`os.pidfd_open` + :func:`select.poll`
- macOS and other BSD variants use :func:`select.kqueue` + ``KQ_FILTER_PROC`` + ``KQ_NOTE_EXIT``
- Windows keeps using ``WaitForSingleObject`` (unchanged)
- Linux ≥= 5.3 uses :func:`os.pidfd_open` + :func:`select.poll`.
- macOS and other BSD variants use :func:`select.kqueue` + ``KQ_FILTER_PROC`` + ``KQ_NOTE_EXIT``.
- Windows keeps using ``WaitForSingleObject`` (unchanged).

If none of these mechanisms are available, the function falls back to the
traditional busy loop (non-blocking call and short sleeps).
Expand Down