Skip to content

Commit 7b08d70

Browse files
Python 3.14 free-threaded support (#2721)
* Thread-safety prep for free-threading builds Two narrow fixes to remove obvious data races that already exist on the GIL build and become hot paths under Py_GIL_DISABLED: - InternString: replace plain Dictionary<> with ConcurrentDictionary<> for both _string2interns and _intern2strings. These are written from startup but read from every attribute-lookup hot path, so any concurrent shutdown/reinit could tear them. - ClassDerived.GetModuleBuilder: add a lock around the check-then-create on assemblyBuilders/moduleBuilders. The previous ContainsKey-then-DefineDynamicAssembly pattern had a TOCTOU race that could produce duplicate builders. Reset() also now locks for a clean reinitialisation. These are not sufficient for full free-threading support, but they remove low-hanging concurrency hazards. * Initialise pythonnet on free-threaded Python (#2720) Free-threaded CPython (Py_GIL_DISABLED) changes the PyObject layout in two pythonnet-relevant ways: - The header is 16 bytes larger (ob_tid + flags + mutex + gc_bits + ob_ref_local + ob_ref_shared replace the single ob_refcnt). - The refcount is no longer a single field; reads must go through the Py_REFCNT API. Detect the build at runtime via sys._is_gil_enabled() in ABI.Initialize and: - Set ObjectHeadOffset to 16 on free-threaded builds so the generated TypeOffset values still resolve to absolute PyHeapTypeObject offsets. - Skip the ob_refcnt probe (it scans for an IntPtr value of 1 which cannot be located reliably under the FT layout). Add a Py_REFCNT P/Invoke (try-loaded; only exported as a function on CPython 3.14+) and prefer it in Runtime.Refcount, falling back to the existing direct read on older Pythons that only expose Py_REFCNT as a macro. * Make extension/CLR-object registries thread-safe ExtensionType.loadedExtensions and CLRObject.reflectedObjects are "borrowed reference" registries written on every alloc and read or removed from finalizer-thread paths. Under free-threaded Python the plain HashSet<IntPtr> tears reliably; under the GIL the same tears were happening more rarely but still mostly observable as Debug.Assert firings during shutdown. Convert both to ConcurrentDictionary<IntPtr, byte> with the equivalent TryAdd/TryRemove operations, and update the few non-mutating callers (NullGCHandles, RuntimeData snapshot LINQ) to enumerate Keys. * Atomic type creation in ReflectedClrType.GetOrCreate / TypeManager.GetType Both type-creation paths had a classic check-then-act race: if (!cache.TryGetValue(t, out var pyType)) { pyType = AllocateClass(t); cache.Add(t, pyType); // throws under contention, partial type otherwise InitializeClass(...); } Two threads racing past the TryGetValue could both call AllocateClass and one would throw on Dictionary.Add ("duplicate key"). Worse, the cache add happens *before* InitializeClass populates members so a third thread's outside-the-lock fast path could observe a partially- initialised type and fail with AttributeError on members not yet added (reproducible under free-threaded Python with concurrent attribute access on built-in CLR types). Convert ClassManager.cache and TypeManager.cache to ConcurrentDictionary and serialise the multi-step initialisation behind a lock. ReflectedClrType.GetOrCreate uses a two-cache design: - `cache` - only fully-initialised types; safe to read on the outside-the-lock fast path. - `_inProgressCache` - partial types being built inside the lock; visible only to the building thread, so self-referential class definitions (which recurse into GetOrCreate for the same type chain) still resolve. Cross-thread access cannot reach the in-progress cache because acquiring the lock is required, so other threads always see fully-ready types. The serialisation snapshot copies remain Dictionary<,> on the wire for binary compatibility. * Add free-threaded thread-stress tests and 3.14t to CI matrix tests/test_thread.py: - test_runtime_refcount_matches_sys_getrefcount and test_is_gil_enabled_attribute_present_on_3_13_plus assert the basic invariants behind ABI.DetectFreeThreaded and Runtime.Refcount. - test_concurrent_clr_method_calls and test_concurrent_attribute_access exercise the CLR call site cache and the ConcurrentDictionary intern path under contention. Both run on every interpreter; on GIL builds they degenerate to mostly-serial smoke checks. - test_concurrent_clr_object_creation, test_concurrent_python_subclass_of_clr_type and test_freethreaded_concurrent_attribute_access_no_tear are FT-only because the GIL-build code path triggers a pre-existing pythonnet crash under high-contention CLR allocation that is reproducible on master and out of scope for this branch. .github/workflows/main.yml: - Add 3.14t to the python matrix on Linux and macOS (Windows FT support is not yet plumbed through pythonnet's native build chain). - Skip the Mono runtime steps on 3.14t — clr-loader's mono backend is not yet validated for free-threaded Python. * Atomic GCHandle ownership and finalizer-thread shutdown guards Two related sets of races that the GIL hid but free-threaded Python exposes reliably. ClassDerivedObject.tp_dealloc and ManagedType.TryFreeGCHandle both read the GCHandle slot, then mutated it. Under FT, subtype_clear (on the main thread) and the .NET finalizer thread can race for the same slot; a non-atomic read-then-zero lets both threads observe the same handle and double-free it. Both paths now use Interlocked.Exchange to atomically claim ownership of the slot - only the thread that observes a non-zero handle frees it. ClassDerived's strong-to-weak swap follows the same pattern. CreateDerivedType emits IL via Reflection.Emit, whose ModuleBuilder and TypeBuilder operations are documented as not thread-safe. Concurrent dynamic subclass creation under FT corrupts the IL stream and segfaults. Serialise the entire emit-and-bake sequence on the existing _buildersLock (the lock that already guarded the assembly / module builder cache). The .NET finalizer thread can dispatch Py_DecRef calls concurrently with Py_Finalize, and a stale read of ob_ref_local after teardown crashes the process. Three guards: - Finalizer.ThrottledCollect and PyObject's finalizer short-circuit when Runtime._Py_IsFinalizing(); PyObject drops the raw pointer instead of enqueueing a decref so process exit reclaims the memory. - Finalizer.AddFinalizedObject's Refcount > 0 Debug.Assert is kept on GIL builds and skipped on FT; a stale ob_ref_local read from the finalizer thread can crash the process even when the assertion would succeed under the GIL. - Runtime.XDecref's matching Refcount > 0 Debug.Assert gets the same FT-only skip for the same reason. * Make additional internal registries thread-safe The atomic-type-creation commit (2f08c98) covered the two highest- contention caches. Wider audit found more plain Dictionary / HashSet collections on hot paths that tear under free-threaded Python and also fire Debug.Assert on GIL builds at sufficient contention. Convert them to ConcurrentDictionary: - ModuleObject.cache and ModuleObject.allNames - hit on every Module.Attr access; the old HashSet.Add for allNames could miss add-once semantics under contention and the Dictionary.Remove on teardown could observe a torn map. - Interop.delegateTypes - racing past TryGetValue with the old plain Dictionary threw on Add ("duplicate key") instead of silently picking one winner, which became reproducible on high concurrency. - ClassBase.ClearVisited - re-entrancy guard for tp_clear, hit from both the main thread and the .NET finalizer thread; the plain HashSet tore reliably under FT. Also tidy the existing ClassManager / TypeManager / ExtensionType ConcurrentDictionary references via a using-directive instead of fully qualifying System.Collections.Concurrent at every site. * test_thread: join worker threads before returning test_python_thread_calls_to_clr left its workers detached and visible under free-threaded Python as background threading.excepthook noise. Collect the threads up front and join them at the end so they cannot outlive the test. Also tighten the docstring for test_concurrent_python_subclass_of_clr_type to spell out why it is FT-only (the GIL build hits a separate pre- existing CLR-object lifecycle crash under high contention, also reproducible on master). * test_thread: cover ModuleObject thread-safe registries Two tests for the thread-safe-collections work in 0a9d482: - test_module_dunder_all_added_once asserts ModuleObject.allNames keeps add-once semantics; a torn HashSet would surface duplicates in __all__ on free-threaded builds. - test_concurrent_module_attribute_access exercises ModuleObject.cache with concurrent getattr on a CLR namespace. The old plain Dictionary threw on Add ("duplicate key") under simultaneous misses; the ConcurrentDictionary version absorbs the race. Both run on every interpreter (no @freethreaded_only) — they degenerate to single-threaded smoke checks under the GIL while the FT build actually exercises contention. * Wider thread-safety audit fixes for free-threaded Python Audit found several more shared-state hazards beyond the registries already covered. The following are reachable on every interpreter under sufficient contention; FT exposes them reliably. src/runtime/DelegateManager.cs Lock cache lookup + Reflection.Emit. TypeBuilder/ModuleBuilder are not thread-safe; concurrent Python->CLR delegate construction (e.g. PythonEngine.ShutdownHandler(lambda: None) from multiple threads) threw "Duplicate type name within an assembly" on 3.14t. Same shape as the CreateDerivedType fix from a18e872. src/runtime/Finalizer.cs - `_throttled` becomes Interlocked.Increment / Interlocked.Exchange. Lost increments would either grow the queue unbounded or burn CPU on unnecessary drains. - `started` is now volatile so the ThrottledCollect check on every PyObject ctor cannot observe a stale "false" after Initialize. src/runtime/PythonTypes/PyBuffer.cs `disposedValue` is now an int gated by Interlocked.Exchange (write) and Volatile.Read (hot reads). The .NET finalizer racing with an explicit Dispose() could otherwise both pass the `if (!disposedValue)` check and call PyBuffer_Release twice -> double-free of _view.obj. Repeated check at every public method extracted to ThrowIfDisposed(). src/runtime/Util/GenericUtil.cs `mapping` (nested Dictionary<string, Dictionary<string, List<string>>>) guarded by a lock; nested mutations cannot be expressed atomically with ConcurrentDictionary alone. GenericByName snapshots candidate names under the lock then calls AssemblyManager.LookupTypes outside it, since LookupTypes can re-enter Register. src/runtime/PythonEngine.cs - `ShutdownHandlers` (List<>) wrapped in a lock. ConcurrentStack would change semantics (no remove-by-equality). ExecuteShutdownHandlers pops under the lock and invokes unlocked so handlers can re-enter Add/Remove without deadlock. - `initialized` flag is now volatile (read from worker threads, written from Initialize/Shutdown). src/runtime/Runtime.cs - `run` epoch now Interlocked.Increment + Volatile.Read; lost increments across re-init would let stale finalizer queue entries slip past the RuntimeRun guard. - `_pyRefs` mutations wrapped in a lock; ResetPyMembers snapshots then disposes outside the lock so a Dispose callback cannot reenter and deadlock. - `_isInitialized`, `_typesInitialized` now volatile. tests/test_thread.py - test_concurrent_delegate_creation (FT-only): reproduces the DelegateManager Reflection.Emit race - aborts with the lock removed, passes with it. - test_concurrent_shutdown_handler_register (FT-only): drives AddShutdownHandler/RemoveShutdownHandler from 8 threads on pre-built handlers. - Removed test_freethreaded_concurrent_attribute_access_no_tear; its workload duplicates test_concurrent_attribute_access at a different intensity without exercising additional code paths. Comment cleanup Trimmed multi-line thread-safety comments across the branch's earlier commits to single lines that capture only the non-obvious "why" (Concurrent: / Lock: / volatile: / Atomic claim:). Removed comments where the type signature already documents the choice. * Document lock acquisition sites and strong->weak GCHandle swap Comment-only changes. Adds inline notes at lock acquisitions where the "why" is not obvious from the field declaration alone: - ClassDerivedObject.Reset / GetModuleBuilder: explain that both builder caches must update atomically and that DefineDynamicAssembly / DefineDynamicModule produce duplicates under contention. - TypeManager.GetType: note that CreateType + cache write must be atomic. - ReflectedClrType.GetOrCreate: cross-file lock; mention it also serialises ClassManager.cache and TypeManager._slotsHolders writes. - Runtime.ResetPyMembers: explain the snapshot-then-dispose pattern (Dispose() callbacks would deadlock if invoked under the lock). Expands the strong->weak GCHandle swap in ClassDerivedObject.tp_dealloc to spell out: 1. Why the PyObject is not freed at refcount 0 (C# wrapper may still reference it; ToPython() resurrects via _Py_NewReference). 2. Why the handle is demoted to weak (lets the C# wrapper be GC'd; on collection PyFinalize enqueues the real PyObject_GC_Del). 3. Why the swap uses Interlocked.Exchange (tp_clear may race on the same slot under FT / finalizer thread; without atomic claim both threads could observe and double-free the same handle). * Preserve InternString single-write invariant under DEBUG Switching the underlying dictionaries to ConcurrentDictionary (0a9d482) replaced Add() with TryAdd(). Add() threw on duplicate keys, which served as a debug-time check that SetIntern is only called once per builtin name (the invariant Initialize relies on via its leading `Debug.Assert(_string2interns.Count == 0)`). TryAdd silently masks that case. Capture its bool result and Debug.Assert it - restores the same correctness signal, with no release-build cost. * test_thread: cover real-world consumer patterns Two FT-only tests for code paths identified by reviewing pythonnet's downstream consumers (QuantConnect/Lean, Rhino.Inside, Speckle) and the historical issue tracker: test_concurrent_clr_delegate_invocation_from_python Python callables wrapped as distinct CLR delegate types (PublicDelegate, StringDelegate, BoolDelegate) and invoked concurrently from worker threads. Canonical embedder pattern for callbacks/event handlers; hits DelegateManager.GetDispatcher (the Reflection.Emit lock added in 92072bd) and the GIL re-acquisition path in Dispatcher.Dispatch. test_concurrent_generic_type_binding 36 distinct Dictionary[K,V] / List[K] type-arg pairs resolved concurrently from N threads. Targets the open-issue family #2269 (ClassManager hash collision crash), #1407 (ClassManager perf regression with MaybeType keys), and #821 (generic resolution race). Exercises ClassManager.cache, TypeManager.cache, GenericUtil.mapping, and the generic-binding fast path simultaneously. Both are @freethreaded_only because the cumulative pytest state under GIL builds trips the same pre-existing CPython 3.11/3.12/3.13 GC crash that gates the other high-contention tests in this file. * Auto-detect free-threaded libpython in venv home PythonEnvironment.FindLibPythonInHome built a single candidate name from version.Major.Minor (e.g. libpython3.14.so) and missed the free-threaded variant (libpython3.14t.so / python314t.dll). pyvenv.cfg's version field doesn't distinguish the two builds, so probe both names and let File.Exists pick the one that's actually on disk. Unblocks the 3.14t CI jobs added in #2721: they were failing in PythonEngine.Initialize with "Py_IncRef: undefined symbol" because PythonDLL resolved to null and pythonnet fell back to dlopen of the dotnet binary itself. * Snapshot pypath, use ConcurrentDictionary for thunks and slot holders * Fix handling of python runtime suffixes m/t * Fix threadtest race * Fix double-free in chained ClassDerived Finalize * Enable Mono CI jobs on free-threaded Python 3.14 * Inline freethreaded_only as pytest.mark.skipif at call sites * Fix InterruptTest assertion on free-threaded Python 3.14 * Add concurrent stress tests for PyBuffer.Dispose and CLR-cycle gc.collect * Trim concurrent overhead on hot paths from free-threading prep DelegateManager.GetDispatcher now takes a lock-free fast path on cache hit (ConcurrentDictionary), avoiding the emit lock on every CLR delegate dispatch. InternString and ClassManager._inProgressCache revert to plain Dictionary since they are only written under existing single- threaded or locked paths, and ClassBase.ClearVisited becomes a per- thread HashSet (tp_clear recursion is intra-stack). * Pre-warm ctor binder in concurrent-gc test to avoid first-call race * Make MethodBinder.GetMethods lazy init thread-safe under free-threading * Precompute method precedence to avoid quadratic GetParameters allocations in MethodBinder sort * Zero the slot in ClassDerived.tp_dealloc when tp_clear already ran to avoid a dangling weak handle * Keep ClassDerived wrapper alive across the NewObjectToPython slot demotion * Document private helpers added during free-threading prep * Add debug echoes and a 6-minute step timeout to the Mono test job * Drop the per-loop CLR GC.Collect from concurrent-gc test to avoid Mono+FT+x64 deadlock * Add temporary Mono-step diagnostics on Linux/macOS to locate the x64-ubuntu hang * Revert temporary Mono-step diagnostics now that the underlying race is fixed * Add user-facing threading guide covering GIL, free-threading, and common pitfalls * Harden CollectBasicObject against .NET-GC timing differences * Adjust the header offset on 32bit systems * Drop broken and unnecessary exclude * Be strict about not loading on Python 3.13 That way we don't have to check for Py_REFCNT. --------- Co-authored-by: Benedikt Reinartz <[email protected]>
1 parent d2d2716 commit 7b08d70

34 files changed

Lines changed: 1103 additions & 323 deletions

.github/workflows/main.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ jobs:
4646
instance: macos-15
4747
suffix: -macos-aarch64-none
4848

49-
python: ["3.11", "3.12", "3.13", "3.14", "3.15"]
49+
python: ["3.11", "3.12", "3.13", "3.14", "3.14t", "3.15", "3.15t"]
5050

5151
exclude:
5252
# fails to call mono methods

doc/source/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ page. Use the `Python.NET issue tracker`_ to report issues.
4545
python
4646
dotnet
4747
codecs
48+
threading
4849
pyreference
4950
reference
5051

doc/source/threading.rst

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
Threading
2+
=========
3+
4+
This page explains how Python.NET interacts with the Python Global Interpreter
5+
Lock (GIL) and with managed threads, and what guarantees the runtime makes
6+
when your code is multi-threaded. It covers both classic CPython builds and
7+
the free-threaded build introduced in CPython 3.13 (``Py_GIL_DISABLED``).
8+
9+
The model in one paragraph
10+
--------------------------
11+
12+
Python.NET embeds CPython, so every interaction with a Python object —
13+
including reading a ``PyObject``'s attributes, calling a Python callable,
14+
constructing a Python value, or letting a ``PyObject`` go out of scope — must
15+
happen while the calling thread is *attached* to the interpreter. On a
16+
classic (GIL-enabled) CPython build "attached" means "holds the GIL"; on a
17+
free-threaded build it means "has an active thread state". In both cases the
18+
attachment API is the same: ``Py.GIL()`` on the C# side and
19+
``threading.Thread`` / ``_thread`` on the Python side. Forgetting to attach
20+
will crash the process or corrupt memory.
21+
22+
Acquiring the GIL from C#
23+
-------------------------
24+
25+
When .NET code calls into Python it must hold the GIL. Use the ``Py.GIL()``
26+
disposable to acquire and release it::
27+
28+
using (Py.GIL())
29+
{
30+
dynamic np = Py.Import("numpy");
31+
var arr = np.array(new[] { 1, 2, 3 });
32+
// ... interact with arr ...
33+
}
34+
35+
``Py.GIL()`` is re-entrant: nesting calls on the same thread is harmless and
36+
cheap. Always pair acquisition with disposal — the ``using`` form does this
37+
automatically, and you must release the GIL on the same thread that acquired
38+
it.
39+
40+
If you need a Python object to outlive the ``using`` block, copy what you
41+
need (e.g. ``.As<int[]>()`` or ``new PyObject(value)``) before releasing the
42+
GIL.
43+
44+
Releasing the GIL for long-running .NET work
45+
--------------------------------------------
46+
47+
If a managed call holds the GIL but then does long-running work that does not
48+
touch Python (heavy CPU, blocking I/O, native interop), release the GIL so
49+
other Python threads can run::
50+
51+
IntPtr threadState = PythonEngine.BeginAllowThreads();
52+
try
53+
{
54+
DoCpuHeavyWork(); // safe: no Python C API calls
55+
}
56+
finally
57+
{
58+
PythonEngine.EndAllowThreads(threadState);
59+
}
60+
61+
Inside the ``BeginAllowThreads``/``EndAllowThreads`` block you must not touch
62+
any Python object. If you need to talk to Python from worker threads spawned
63+
in this region, those threads must acquire the GIL themselves with
64+
``Py.GIL()``.
65+
66+
Calling .NET from Python threads
67+
--------------------------------
68+
69+
Calling a managed method from a Python ``threading.Thread`` works
70+
transparently — Python.NET handles GIL acquisition/release around the
71+
managed call. The managed code sees the GIL held on entry and is free to
72+
release it via ``BeginAllowThreads`` if it does its own blocking work.
73+
74+
Calling Python from CLR threads
75+
-------------------------------
76+
77+
A CLR thread that was *not* spawned by Python (a thread-pool task, a
78+
``Thread`` started in C#, an ``async`` continuation that resumed on a
79+
different thread, etc.) must acquire the GIL before touching any
80+
``PyObject``::
81+
82+
Task.Run(() =>
83+
{
84+
using (Py.GIL())
85+
{
86+
// safe to use PyObjects here
87+
}
88+
});
89+
90+
Forgetting this is the most common pythonnet threading bug. Symptoms range
91+
from immediate segfaults to subtle refcount corruption that crashes much
92+
later.
93+
94+
Reference counting and finalizers
95+
---------------------------------
96+
97+
``PyObject`` follows the .NET ``IDisposable`` pattern. ``Dispose()`` (or the
98+
end of a ``using`` block) drops the underlying Python reference; the GC
99+
finalizer queues the same release for the next time Python.NET is on the GIL.
100+
101+
Two practical consequences:
102+
103+
* **Don't share a single ``PyObject`` instance across threads without
104+
serialising access.** ``PyObject`` is not internally locked. If multiple
105+
threads concurrently dispose the same instance, the underlying refcount can
106+
go negative.
107+
108+
* **Don't rely on the GC finalizer running promptly.** The PyObject is only
109+
freed when a Python.NET API later reacquires the GIL. If your application
110+
shuts down without that happening, finalizable PyObjects can be reported as
111+
leaked.
112+
113+
Free-threaded Python (PEP 703)
114+
------------------------------
115+
116+
Starting with the free-threaded CPython 3.13+ build (``Py_GIL_DISABLED``),
117+
the GIL is no longer the serialisation point for Python C API calls.
118+
Python.NET is tested against the ``3.14t`` (free-threaded) interpreter and
119+
behaves as follows under that build:
120+
121+
* ``Py.GIL()`` still acquires a thread state. It is functionally a no-op
122+
for mutual exclusion but is still required for thread-state attachment.
123+
Existing code that uses ``using (Py.GIL())`` continues to work without
124+
changes.
125+
* ``PythonEngine.BeginAllowThreads`` / ``EndAllowThreads`` similarly
126+
manage the thread state and are still needed if you want the GC and
127+
other Python threads to run while you're in long-running unmanaged code.
128+
* Internal Python.NET caches (the reflection cache, generic-type binding
129+
cache, dynamic-dispatch cache, module attribute cache, the interned-
130+
string table, etc.) are thread-safe. You may read and call CLR types
131+
concurrently from any number of threads without external locking.
132+
* The reference-counting protocol uses CPython's ``Py_REFCNT`` symbol on
133+
3.14+, which returns the merged biased + shared refcount; values you read
134+
from ``PyObject.Refcount`` are correct under free-threading.
135+
136+
Behaviour that is *unchanged* between GIL and free-threaded builds:
137+
138+
* A managed object exposed to Python (e.g. via ``System.Object`` or a
139+
Python subclass of a CLR type) is still owned by a single CLR side: you
140+
must not mutate its plain CLR fields from multiple threads without your
141+
own locking. Python.NET only protects its own bookkeeping, not your
142+
domain data.
143+
* Operations on a single ``PyObject`` instance still require external
144+
serialisation — see "Reference counting" above.
145+
146+
Patterns
147+
--------
148+
149+
Concurrent CLR access from Python
150+
"""""""""""""""""""""""""""""""""
151+
152+
Hammering CLR attributes / generic types from many threads is supported::
153+
154+
from threading import Thread
155+
import System
156+
from System.Collections.Generic import List
157+
158+
def worker():
159+
for _ in range(1000):
160+
_ = System.String.Empty
161+
_ = List[int]()
162+
163+
threads = [Thread(target=worker) for _ in range(8)]
164+
for t in threads: t.start()
165+
for t in threads: t.join()
166+
167+
This works on both GIL and free-threaded builds.
168+
169+
Python callback invoked from a managed thread
170+
"""""""""""""""""""""""""""""""""""""""""""""
171+
172+
If a managed component calls back into a Python delegate from a thread it
173+
spawned, that callback path acquires the GIL internally — you do not need to
174+
add ``Py.GIL()`` around the Python code in the delegate.
175+
176+
Spawning a managed thread from inside ``Py.GIL()``
177+
""""""""""""""""""""""""""""""""""""""""""""""""""
178+
179+
If you start a managed thread while holding the GIL and the thread needs to
180+
call back into Python, release the GIL first so the new thread can acquire
181+
it::
182+
183+
using (Py.GIL())
184+
{
185+
var pyCallback = scope.Get("on_done");
186+
PythonEngine.BeginAllowThreads(); // let workers acquire the GIL
187+
try
188+
{
189+
// spawn workers, wait for them...
190+
}
191+
finally
192+
{
193+
PythonEngine.EndAllowThreads(...);
194+
}
195+
}
196+
197+
Without the ``BeginAllowThreads`` the spawned thread blocks forever waiting
198+
for the GIL the parent thread is still holding.
199+
200+
Common pitfalls
201+
---------------
202+
203+
* Holding ``Py.GIL()`` across ``Task.Run`` / ``await`` boundaries. Async
204+
continuations can resume on a different thread; the GIL handle is
205+
thread-bound and must be released on the same thread that acquired it.
206+
* Passing a ``PyObject`` to a managed worker without taking ownership. If
207+
the producer disposes its handle while the consumer is still using it,
208+
the worker will operate on a freed object. Wrap the producer's
209+
``PyObject`` with ``new PyObject(value)`` before handing it off, or use
210+
``NewReference()``.
211+
* Calling a Python callable that does CPU-bound work without releasing the
212+
GIL. Other Python threads cannot make progress in that case, even on a
213+
free-threaded build where the GIL is otherwise a no-op (the callable
214+
itself may still touch contended Python state).

src/embed_tests/TestFinalizer.cs

Lines changed: 20 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ private static void FullGCCollect()
3030
{
3131
GC.Collect();
3232
GC.WaitForPendingFinalizers();
33+
GC.Collect(); // reclaim objects whose finalizers just ran
3334
}
3435

3536
[Test]
@@ -51,28 +52,28 @@ public void CollectBasicObject()
5152
Finalizer.Instance.BeforeCollect += handler;
5253

5354
IntPtr pyObj = MakeAGarbage(out var shortWeak, out var longWeak);
54-
FullGCCollect();
55-
// The object has been resurrected
56-
Warn.If(
57-
shortWeak.IsAlive,
58-
"The referenced object is alive although it should have been collected",
59-
shortWeak
60-
);
61-
Assert.That(
62-
longWeak.IsAlive,
63-
Is.True,
64-
$"The reference object is not alive although it should still be"
65-
);
6655

56+
// The real contract: after the wrapper is GC'd, the underlying
57+
// Python pointer must end up in Finalizer's queue. Poll because
58+
// .NET Framework / .NET Core differ in how many GC cycles it takes.
59+
List<IntPtr> garbage = null;
60+
for (int attempt = 0; attempt < 10; attempt++)
6761
{
68-
var garbage = Finalizer.Instance.GetCollectedObjects();
69-
Assert.NotZero(garbage.Count, "There should still be garbage around");
70-
Warn.Unless(
71-
garbage.Contains(pyObj),
72-
$"The {nameof(longWeak)} reference doesn't show up in the garbage list",
73-
garbage
74-
);
62+
FullGCCollect();
63+
garbage = Finalizer.Instance.GetCollectedObjects();
64+
if (garbage.Contains(pyObj)) break;
65+
Thread.Sleep(20);
7566
}
67+
68+
Warn.If(shortWeak.IsAlive,
69+
"shortWeak is alive after FullGCCollect; runtime hasn't reclaimed the wrapper yet",
70+
shortWeak);
71+
// longWeak.IsAlive at this point is .NET-GC-implementation-defined
72+
// (Framework reclaims post-finalize objects more eagerly than Core);
73+
// intentionally not asserted.
74+
75+
Assert.That(garbage, Has.Member(pyObj),
76+
"PyObject did not reach Finalizer.Instance.GetCollectedObjects()");
7677
try
7778
{
7879
Finalizer.Instance.Collect();

src/embed_tests/TestInterrupt.cs

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,17 @@ import time
9090
Assert.That(asyncCall.Wait(TimeSpan.FromSeconds(5)), Is.True, "Async thread was not interrupted in time");
9191
PythonEngine.EndAllowThreads(threadState);
9292

93-
Assert.That(asyncCall.Result, Is.EqualTo(0));
93+
// On free-threaded CPython 3.14, PyRun_SimpleString may return -1 even
94+
// when the script catches the async-injected KeyboardInterrupt — the
95+
// C-level error indicator depends on which bytecode boundary the
96+
// async-exc fires at, and isn't always cleared the way GIL builds clear
97+
// it. The interrupt firing and the script terminating cleanly are what
98+
// this test exercises; the return code is a side-effect that's only
99+
// deterministic under the GIL.
100+
if (Python.Runtime.Native.ABI.IsFreeThreaded)
101+
Assert.That(asyncCall.Result, Is.AnyOf(0, -1));
102+
else
103+
Assert.That(asyncCall.Result, Is.EqualTo(0));
94104
}
95105
}
96106
}

src/embed_tests/TestNativeTypeOffset.cs

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,10 @@ public class TestNativeTypeOffset
2020
public void LoadNativeTypeOffsetClass()
2121
{
2222
PyObject sys = Py.Import("sys");
23-
// We can safely ignore the "m" abi flag
23+
// "m" is benign; "t" (free-threaded) is handled via ABI.ObjectHeadOffset
24+
// rather than the install-time-generated NativeTypeOffset class.
2425
var abiflags = sys.HasAttr("abiflags") ? sys.GetAttr("abiflags").ToString() : "";
25-
abiflags = abiflags.Replace("m", "");
26+
abiflags = abiflags.Replace("m", "").Replace("t", "");
2627
if (!string.IsNullOrEmpty(abiflags))
2728
{
2829
string typeName = "Python.Runtime.NativeTypeOffset, Python.Runtime";

src/embed_tests/TestPyBuffer.cs

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
using System;
22
using System.Runtime.CompilerServices;
33
using System.Text;
4+
using System.Threading;
45
using NUnit.Framework;
56
using Python.Runtime;
67
using Python.Runtime.Codecs;
@@ -129,6 +130,42 @@ public void MultidimensionalNumPyArray()
129130
});
130131
}
131132

133+
[Test]
134+
public void ConcurrentDispose()
135+
{
136+
// Two threads racing on Dispose() must not double-release the view —
137+
// Interlocked.Exchange on disposedValue gates PyBuffer_Release.
138+
// Smoke test: no crash, exception, or buffer-protocol violation.
139+
using var _ = Py.GIL();
140+
using var arr = ByteArrayFromAsciiString("hello world! !$%&/()=?");
141+
142+
const int iterations = 200;
143+
for (int i = 0; i < iterations; i++)
144+
{
145+
PyBuffer buf = arr.GetBuffer();
146+
147+
IntPtr ts = PythonEngine.BeginAllowThreads();
148+
using var barrier = new Barrier(2);
149+
Exception captured = null;
150+
Action race = () =>
151+
{
152+
try
153+
{
154+
barrier.SignalAndWait();
155+
using (Py.GIL()) buf.Dispose();
156+
}
157+
catch (Exception ex) { Interlocked.CompareExchange(ref captured, ex, null); }
158+
};
159+
var t1 = new Thread(() => race());
160+
var t2 = new Thread(() => race());
161+
t1.Start(); t2.Start();
162+
t1.Join(); t2.Join();
163+
PythonEngine.EndAllowThreads(ts);
164+
165+
if (captured != null) throw captured;
166+
}
167+
}
168+
132169
[MethodImpl(MethodImplOptions.NoInlining)]
133170
static void MakeBufAndLeak(PyObject bufProvider)
134171
{

0 commit comments

Comments
 (0)