[BUG] --report ndjson writes only the Initialize event since 4.27.3 — writer thread dies on _DeadlockError from the lazy import requests added in #4792 (silent data loss, exit 0) #4807

@@ -10,8 +10,6 @@
 from types import TracebackType
 from typing import IO, TYPE_CHECKING, Any

-import requests
-
 from schemathesis.core import NOT_SET
@@ -112,6 +110,9 @@ def serialize(obj: Any, *, sanitization: SanitizationConfig | None = None) -> Any:
         if obj.is_truncated:
             data["content_size"] = obj.content_size
         return data
+    # Imported late so the CLI does not load `requests` before a command needs it.
+    import requests
+
     if isinstance(obj, requests.PreparedRequest):

serialize() runs on the SchemathesisNdjsonWriter thread (cli/commands/run/handlers/ndjson.py::NdjsonHandler.start spawns it; _run() loops queue.get() → writer.write_event() → serialize()). That thread starts on Initialize, before the schema is loaded. The first event after Initialize makes it execute import requests while the main thread is loading the schema and, via the lazy from schemathesis.specs.openapi.schemas import OpenApiSchema in openapi/loaders.py::from_dict (line 241), executing specs/openapi/schemas.py:11 from requests.structures import CaseInsensitiveDict. The two threads take the requests and requests.structures import locks in opposite order and importlib's cycle detector raises _DeadlockError in the writer.

_run() has no exception handling, so the thread ends, NdjsonWriter.__exit__ closes the file, every later queue.put goes to a dead consumer, and shutdown()'s bounded worker.join(WRITER_WORKER_JOIN_TIMEOUT) lets the CLI finish normally. The traceback reaches stderr through threading.excepthook only; the exit code and summary are unchanged.

Before #4792 the module-level import meant requests was fully loaded on the main thread at CLI import time, before any thread existed, so the imports could never coincide. reporting/ndjson.py on master still has the in-function import at line 114 as of 2026-09-18, so 4.27.4 will ship it unless fixed.

To Reproduce

1. Start any HTTP server (below: a stdlib ThreadingHTTPServer on 127.0.0.1:PORT answering 200 {"ok": true} to every GET).
2. Save the minimal schema below as spec.json.
3. Run:

st run spec.json --url http://127.0.0.1:PORT --max-examples 2 --phases fuzzing \
   --checks not_a_server_error --report ndjson --report-ndjson-path out.ndjson --workers 1 --seed 1

4. wc -l out.ndjson → 1 on 4.27.3; 23 on 4.27.1 and 4.27.2. head -c 20 out.ndjson → {"Initialize": {…. The console shows Tested: 2, 4 generated, 4 passed, exit code 0, and lists out.ndjson under Reports.

Minimal schema:

{
  "openapi": "3.0.3",
  "info": {"title": "stub", "version": "1.0.0"},
  "paths": {
    "/health": {
      "get": {
        "operationId": "health",
        "responses": {"200": {"description": "ok", "content": {"application/json": {"schema": {"type": "object"}}}}}
      }
    },
    "/items/{id}": {
      "get": {
        "operationId": "getItem",
        "parameters": [{"name": "id", "in": "path", "required": true, "schema": {"type": "string"}}],
        "responses": {"200": {"description": "ok", "content": {"application/json": {"schema": {"type": "object"}}}}}
      }
    }
  }
}

Reproduction rate — 3 versions × 2 worker counts × 20 runs (120 runs). Each run is a fresh subprocess of the venv's interpreter invoking the CLI the way the console script does (python -c "import sys; from schemathesis.cli import schemathesis; sys.argv=['schemathesis']+sys.argv[1:]; schemathesis()" run …), same stub, same schema, same seed. "complete" = EngineFinished present AND ScenarioFinished count == console Tested.

Event mix of a complete file (4.27.1/4.27.2): Initialize 1, LoadingStarted 1, LoadingFinished 1, EngineStarted 1, PhaseStarted 6, PhaseFinished 6, SuiteStarted 1, ScenarioStarted 2, ScenarioFinished 2, SuiteFinished 1, EngineFinished 1. On 4.27.3, every run: Initialize 1. Deterministic, not intermittent: 40 of 40 runs on 4.27.3 lost the file; 80 of 80 on the two prior versions were complete.

Causation, two independent ways (20 runs each, workers 1 and 12, same stub/schema/seed):

Untouched 4.27.3: 0/40. (a) and (b): 40/40 each.

Expected behavior

out.ndjson contains every engine event of the run (23 lines for this schema), ending with EngineFinished, as on 4.27.2 and as documented. If the writer thread ever fails, the run should say so — in the summary and ideally in the exit code — rather than exit 0 with the path listed under Reports.

Environment

- OS: macOS (arm64)
- Python version: 3.14.6
- Schemathesis version: 4.27.3 (bug); 4.27.1 and 4.27.2 (not affected)
- Spec version: Open API 3.0.3
- hypothesis 6.168.0, jsonschema_rs 0.56.0, requests 2.34.2, click 8.5.0 (identical across the three venvs)

Additional context

Both parties of the lock cycle, captured in-process on 4.27.3 with importlib._bootstrap._ModuleLock.acquire wrapped to log every requests* module lock, and threading.excepthook set:

Writer-thread traceback (stderr, via threading.excepthook):

Exception in thread SchemathesisNdjsonWriter:
  threading.py:1082 _bootstrap_inner
  threading.py:1024 run
  schemathesis/cli/commands/run/handlers/ndjson.py:73 _run
  schemathesis/reporting/ndjson.py:201 write_event
  schemathesis/reporting/ndjson.py:114 serialize
  requests/__init__.py:158 <module>
  requests/utils.py:69 <module>
  <frozen importlib._bootstrap>:1368 _find_and_load
  <frozen importlib._bootstrap>:421 __enter__
  <frozen importlib._bootstrap>:346 acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('requests.structures') at 0x...

The same in-process script with import requests executed on the main thread before main() writes 23 lines.

Startup cost of the fix (st --help, 20 runs each, same -c bootstrap):

Restoring the import costs ~35 ms of the ~106 ms that #4792 saved.

Suggested fix. Either import requests at module level in reporting/ndjson.py again, or import it once on the main thread before NdjsonHandler.start() spawns the writer. Independently, _run() in cli/commands/run/handlers/ndjson.py should not let a writer-thread exception pass silently — surface it in the summary and/or exit code — otherwise any future failure inside serialize() reproduces this class of silent data loss.