-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Multi phase module init #6740
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multi phase module init #6740
Conversation
📝 WalkthroughWalkthroughIntroduce InterpreterBuilder and migrate stdlib/native module wiring from single‑step make_module to multi‑phase PyModuleDef APIs (module_def + module_exec). VM bootstrap now creates → inserts into sys.modules → exec (with rollback on failure). Callers updated to builder-based initialization and stdlib_module_defs(ctx). Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Builder as InterpreterBuilder (ctx)
participant DefList as stdlib_module_defs(&ctx)
participant BuilderActions as Builder.add_native_modules / build
participant Interpreter
participant VM as VirtualMachine
participant Sys as sys.modules
participant Def as PyModuleDef
User->>Builder: Interpreter::builder(settings)
Builder->>DefList: stdlib_module_defs(&builder.ctx)
DefList-->>Builder: Vec<&PyModuleDef>
User->>BuilderActions: add_native_modules(defs) / build()
BuilderActions->>Interpreter: initialize_main_vm(...)
Interpreter->>VM: construct VM with module_defs map
loop for each builtin module_def
VM->>Def: create_module(vm) %% Phase 1: create
Def-->>VM: PyModule instance
VM->>Sys: insert module into sys.modules (pre-exec)
VM->>Def: exec_module(vm, module) %% Phase 2: exec
Def-->>VM: module init completes or error
alt exec failed
VM->>Sys: remove partial module
end
end
Interpreter-->>User: Interpreter (with global_state)
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
660f2fb to
054b125
Compare
|
Code has been automatically formatted The code in this PR has been formatted using:
git pull origin multi-phase-module-init |
338374a to
5b2598b
Compare
crates/stdlib/src/openssl.rs
Outdated
| #[pymodule(with(cert::ssl_cert, ssl_error::ssl_error, ossl101, ossl111, windows))] | ||
| mod _ssl { | ||
| #[cfg(openssl_vendored)] | ||
| use super::PROBE; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does PROBE still need to defined on super?
85d072d to
ea3a345
Compare
ea3a345 to
5eed597
Compare
5eed597 to
c6dc3d2
Compare
c6dc3d2 to
7a38e56
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/vm/src/stdlib/os.rs (1)
1900-1923: Avoid panics inmodule_exec; propagate errors instead.
unwrap()during import can abort the interpreter. ReturningPyResult<()>and using?provides a proper Python exception instead. If you adopt this, update callers to handle the new result.🛠️ Proposed fix
-pub fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) { +pub fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { let support_funcs = _os::support_funcs(); let supports_fd = PySet::default().into_ref(&vm.ctx); let supports_dir_fd = PySet::default().into_ref(&vm.ctx); let supports_follow_symlinks = PySet::default().into_ref(&vm.ctx); for support in support_funcs { - let func_obj = module.get_attr(support.name, vm).unwrap(); + let func_obj = module.get_attr(support.name, vm)?; if support.fd.unwrap_or(false) { - supports_fd.clone().add(func_obj.clone(), vm).unwrap(); + supports_fd.clone().add(func_obj.clone(), vm)?; } if support.dir_fd.unwrap_or(false) { - supports_dir_fd.clone().add(func_obj.clone(), vm).unwrap(); + supports_dir_fd.clone().add(func_obj.clone(), vm)?; } if support.follow_symlinks.unwrap_or(false) { - supports_follow_symlinks.clone().add(func_obj, vm).unwrap(); + supports_follow_symlinks.clone().add(func_obj, vm)?; } } extend_module!(vm, module, { "supports_fd" => supports_fd, "supports_dir_fd" => supports_dir_fd, "supports_follow_symlinks" => supports_follow_symlinks, "error" => vm.ctx.exceptions.os_error.to_owned(), }); + Ok(()) }
🤖 Fix all issues with AI agents
In `@crates/vm/src/import.rs`:
- Around line 91-116: The module is inserted into sys_modules via
sys_modules.set_item(...) before def.exec_module(...) and if exec_module fails
the partially-initialized module remains in sys_modules; update the two-phase
init path (the block that uses vm.state.module_defs, def.create_module,
sys_modules.set_item, def.exec_module) to remove the module from sys_modules on
exec failure (call sys_modules.del_item(module_name, vm) or equivalent) before
returning the error so subsequent imports won't return the broken module,
ensuring the original error from def.exec_module is propagated by returning
Err(e).
In `@crates/vm/src/stdlib/errno.rs`:
- Around line 9-23: The module_exec function currently uses unwrap() when
calling errorcode.set_item(...) and module.set_attr(...), which will panic on
failure; change those unwrap() calls to propagate errors by using the ? operator
so the function returns a Python exception instead of aborting. Locate
module_exec (and the loop over super::ERROR_CODES) and replace the two .unwrap()
usages on set_item and set_attr with ? so the PyResult<()> return handles
failures; ensure signatures remain unchanged and error propagation compiles.
In `@crates/vm/src/stdlib/imp.rs`:
- Around line 122-149: When inserting the module into sys.modules before calling
the def.slots.exec slot (module variable, sys_modules.set_item and
def.slots.exec), ensure you remove that entry if exec returns an Err so a
partially-initialized module is not left cached; change the exec call to
match/if-let on its Result, and on Err call sys_modules.del_item(&*name, vm) (or
the equivalent removal API) before propagating the error so behavior matches
CPython’s cleanup on exec failure.
In `@crates/vm/src/stdlib/mod.rs`:
- Around line 96-104: The posix::module_def(ctx) is being registered twice for
WASI because both #[cfg(target_os = "wasi")] and the broader
#[cfg(all(any(not(target_arch = "wasm32"), target_os = "wasi"), not(any(unix,
windows))))] match; remove the redundant explicit #[cfg(target_os = "wasi")]
posix::module_def(ctx) entry so posix::module_def(ctx) is only included once
(leave the broader cfg that covers non-unix/windows and WASI targets or vice
versa based on project convention), ensuring only a single
posix::module_def(ctx) call remains in the builtin modules list.
In `@crates/vm/src/vm/interpreter.rs`:
- Around line 30-142: The code in initialize_main_vm currently calls Box::leak
for sysconfigdata_name on every VM creation, leaking memory each time; change
this to leak the platform-specific name only once per process by introducing a
static OnceLock/OnceCell (e.g., static SYSCONFIGDATA_LEAK: OnceLock<&'static
str>) and use get_or_init to perform Box::leak only the first time, then insert
the returned &'static str into all_module_defs; reference initialize_main_vm,
sysconfigdata_name, and the all_module_defs insertion where leaked_name is
created to locate the change.
🧹 Nitpick comments (5)
crates/wasm/src/js_module.rs (1)
623-623: LGTM - Re-export added for module_def pattern.The addition of
pub(crate) use _js::module_def;aligns with the crate-wide module initialization refactoring. Note that this re-export is placed at the end of the file, unlike most other modules where it appears at the top. While functionally equivalent, you may consider moving it to line 2 (after thePyJsValue, PyPromisere-export) for consistency with other stdlib modules.crates/vm/src/builtins/module.rs (1)
53-71: Remove redundant import.
PyPayloadis already imported at the file level (line 3), so theuse crate::PyPayload;on line 54 is unnecessary.Proposed fix
pub fn create_module(&'static self, vm: &VirtualMachine) -> PyResult<PyRef<PyModule>> { - use crate::PyPayload; - // Create module (use create slot if provided, else default creation) let module = if let Some(create) = self.slots.create {crates/stdlib/src/_sqlite3.rs (1)
847-867: Propagate setup_module_exceptions errors instead of panicking.Line 856 invokes
setup_module_exceptions, which still usesunwrap()internally. Sincemodule_execnow returnsPyResult<()>, consider returningPyResult<()>fromsetup_module_exceptionsand using?to avoid panics during module init.crates/vm/src/stdlib/sysconfigdata.rs (1)
14-25: Chain__module_execbefore custom init.Custom
module_execdoesn’t call the macro-generated__module_exec, so any#[pyattr]/#[pyfunction]exports added now or later won’t be registered. Consider calling it first, consistent with other stdlib modules.♻️ Suggested change
fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { + __module_exec(vm, module); // Set build_time_vars attribute let build_time_vars = build_time_vars(vm);crates/vm/src/stdlib/sys.rs (1)
168-173: Consider deduplicating builtin module names after augmentation.If any native module definition already uses
"sys"or"builtins", this could yield duplicates insys.builtin_module_names. A quickdedupafter sorting prevents that edge case.♻️ Proposed tweak
- module_names.sort(); + module_names.sort(); + module_names.dedup();
| // Check sys.modules first | ||
| if let Ok(module) = sys_modules.get_item(&*name, vm) { | ||
| return Ok(module); | ||
| } | ||
|
|
||
| // Try multi-phase init modules first (they need special handling) | ||
| if let Some(&def) = vm.state.module_defs.get(name.as_str()) { | ||
| // Phase 1: Create module (use create slot if provided, else default creation) | ||
| let module = if let Some(create) = def.slots.create { | ||
| // Custom module creation | ||
| create(vm, &spec, def)? | ||
| } else { | ||
| // Default module creation | ||
| PyModule::from_def(def).into_ref(&vm.ctx) | ||
| }; | ||
|
|
||
| // Initialize module dict and methods | ||
| // Corresponds to PyModule_FromDefAndSpec: md_def, _add_methods_to_object, PyModule_SetDocString | ||
| PyModule::__init_dict_from_def(vm, &module); | ||
| module.__init_methods(vm)?; | ||
|
|
||
| // Add to sys.modules BEFORE exec (critical for circular import handling) | ||
| sys_modules.set_item(&*name, module.clone().into(), vm)?; | ||
|
|
||
| // Phase 2: Call exec slot (can safely import other modules now) | ||
| if let Some(exec) = def.slots.exec { | ||
| exec(vm, &module)?; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clean up sys.modules when the exec slot fails.
Line 143 inserts the module into sys.modules before exec. If Line 147–149 errors, a partially initialized module remains cached and may be reused by later imports. Consider removing it on failure to match CPython’s behavior.
🛠️ Proposed fix
- if let Some(exec) = def.slots.exec {
- exec(vm, &module)?;
- }
+ if let Some(exec) = def.slots.exec {
+ if let Err(err) = exec(vm, &module) {
+ let _ = sys_modules.del_item(&*name, vm);
+ return Err(err);
+ }
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Check sys.modules first | |
| if let Ok(module) = sys_modules.get_item(&*name, vm) { | |
| return Ok(module); | |
| } | |
| // Try multi-phase init modules first (they need special handling) | |
| if let Some(&def) = vm.state.module_defs.get(name.as_str()) { | |
| // Phase 1: Create module (use create slot if provided, else default creation) | |
| let module = if let Some(create) = def.slots.create { | |
| // Custom module creation | |
| create(vm, &spec, def)? | |
| } else { | |
| // Default module creation | |
| PyModule::from_def(def).into_ref(&vm.ctx) | |
| }; | |
| // Initialize module dict and methods | |
| // Corresponds to PyModule_FromDefAndSpec: md_def, _add_methods_to_object, PyModule_SetDocString | |
| PyModule::__init_dict_from_def(vm, &module); | |
| module.__init_methods(vm)?; | |
| // Add to sys.modules BEFORE exec (critical for circular import handling) | |
| sys_modules.set_item(&*name, module.clone().into(), vm)?; | |
| // Phase 2: Call exec slot (can safely import other modules now) | |
| if let Some(exec) = def.slots.exec { | |
| exec(vm, &module)?; | |
| } | |
| // Check sys.modules first | |
| if let Ok(module) = sys_modules.get_item(&*name, vm) { | |
| return Ok(module); | |
| } | |
| // Try multi-phase init modules first (they need special handling) | |
| if let Some(&def) = vm.state.module_defs.get(name.as_str()) { | |
| // Phase 1: Create module (use create slot if provided, else default creation) | |
| let module = if let Some(create) = def.slots.create { | |
| // Custom module creation | |
| create(vm, &spec, def)? | |
| } else { | |
| // Default module creation | |
| PyModule::from_def(def).into_ref(&vm.ctx) | |
| }; | |
| // Initialize module dict and methods | |
| // Corresponds to PyModule_FromDefAndSpec: md_def, _add_methods_to_object, PyModule_SetDocString | |
| PyModule::__init_dict_from_def(vm, &module); | |
| module.__init_methods(vm)?; | |
| // Add to sys.modules BEFORE exec (critical for circular import handling) | |
| sys_modules.set_item(&*name, module.clone().into(), vm)?; | |
| // Phase 2: Call exec slot (can safely import other modules now) | |
| if let Some(exec) = def.slots.exec { | |
| if let Err(err) = exec(vm, &module) { | |
| let _ = sys_modules.del_item(&*name, vm); | |
| return Err(err); | |
| } | |
| } |
🤖 Prompt for AI Agents
In `@crates/vm/src/stdlib/imp.rs` around lines 122 - 149, When inserting the
module into sys.modules before calling the def.slots.exec slot (module variable,
sys_modules.set_item and def.slots.exec), ensure you remove that entry if exec
returns an Err so a partially-initialized module is not left cached; change the
exec call to match/if-let on its Result, and on Err call
sys_modules.del_item(&*name, vm) (or the equivalent removal API) before
propagating the error so behavior matches CPython’s cleanup on exec failure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/vm/src/stdlib/sys.rs (1)
1678-1685: Revert visibility change forsysconfigdata_name.The function is only called from within
crates/vm/src/(viacrate::stdlib::sys::sysconfigdata_name()ininterpreter.rsandsysconfigdata.rs), sopub(crate)visibility is sufficient. No external crates use this function. Change it back topub(crate)to maintain proper encapsulation.
♻️ Duplicate comments (1)
crates/vm/src/vm/interpreter.rs (1)
67-72: Avoid per-interpreterBox::leakfor_sysconfigdataaliasing.This repeats a previously flagged issue: leaking on each interpreter creation grows memory unboundedly. Prefer a process-wide
OnceLockto leak once.🩹 Suggested fix (same as prior review)
if let Some(&sysconfigdata_def) = all_module_defs.get("_sysconfigdata") { + use std::sync::OnceLock; let sysconfigdata_name = crate::stdlib::sys::sysconfigdata_name(); // Leak the string to get a 'static lifetime for the key - let leaked_name: &'static str = Box::leak(sysconfigdata_name.into_boxed_str()); - all_module_defs.insert(leaked_name, sysconfigdata_def); + static SYSCONFIGDATA_NAME: OnceLock<&'static str> = OnceLock::new(); + let leaked_name = + *SYSCONFIGDATA_NAME.get_or_init(|| Box::leak(sysconfigdata_name.into_boxed_str())); + all_module_defs.insert(leaked_name, sysconfigdata_def); }
7a38e56 to
203756d
Compare
b25c907 to
22ea6ad
Compare
22ea6ad to
bed12ed
Compare
📦 Library DependenciesThe following Lib/ modules were modified. Here are their dependencies: Click to expand dependency informationLegend:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/vm/src/vm/mod.rs (1)
1339-1339: Update test to use builder pattern for consistency.
test_nested_frozen(line 1314) usesInterpreter::builder(), butfrozen_origname_matches(line 1339) still usesInterpreter::with_init(). Update the latter to match the builder pattern for consistency with the rest of the test suite.
🤖 Fix all issues with AI agents
In `@crates/stdlib/src/_sqlite3.rs`:
- Around line 847-867: The call to __module_exec(vm, module) in module_exec
currently discards its Result; change the invocation to propagate errors (e.g.,
use __module_exec(vm, module)? or otherwise handle its PyResult) so that any
error returned by __module_exec is forwarded from module_exec; ensure
module_exec still returns PyResult<()> and that subsequent initialization
(ERROR_CODES loop, setup_module_exceptions,
CONVERTERS/ADAPTERS/USER_FUNCTION_EXCEPTION/ENABLE_TRACEBACK setup, and
module.set_attr calls) only runs after successful __module_exec.
In `@crates/vm/src/stdlib/ctypes.rs`:
- Around line 1281-1312: The call to __module_exec(vm, module) in module_exec
currently discards its PyResult return value; change the call to propagate
errors by using the ? operator (i.e., call __module_exec(vm, module)?;) so that
module_exec (which returns PyResult<()>) will return any error from
__module_exec instead of silently ignoring it.
In `@crates/vm/src/stdlib/typing.rs`:
- Around line 191-218: The call to __module_exec(vm, module) inside module_exec
currently discards its PyResult, so initialization errors are ignored; change
the call to propagate errors (e.g., use the ? operator) so that module_exec
returns early with the error when __module_exec fails—update the invocation in
the module_exec function to propagate the result rather than ignoring it.
In `@crates/wasm/src/vm_class.rs`:
- Around line 29-35: In module_exec, the result of __module_exec(vm, module) is
ignored so errors don't propagate; change the call to use the ? operator (i.e.,
__module_exec(vm, module)?;) so any PyErr returned by __module_exec is
propagated, then proceed to extend_module! and return Ok(()) from module_exec;
this ensures module initialization failures are handled consistently.
♻️ Duplicate comments (2)
crates/vm/src/stdlib/imp.rs (1)
143-149: Clean upsys.moduleswhen exec slot fails.This was flagged in a previous review: if
def.slots.execreturns an error on line 148, the module inserted at line 144 remains insys.modules, leaving a partially initialized module cached. This can cause issues with subsequent imports.🔧 Suggested fix
// Add to sys.modules BEFORE exec (critical for circular import handling) sys_modules.set_item(&*name, module.clone().into(), vm)?; // Phase 2: Call exec slot (can safely import other modules now) if let Some(exec) = def.slots.exec { - exec(vm, &module)?; + if let Err(err) = exec(vm, &module) { + let _ = sys_modules.del_item(&*name, vm); + return Err(err); + } }crates/vm/src/stdlib/mod.rs (1)
96-102: Potential duplicateposixmodule registration on WASI targets.The cfg conditions may still overlap for WASI:
- Line 96-97:
#[cfg(any(unix, target_os = "wasi"))]matches WASI- Lines 98-101:
#[cfg(all(any(not(target_arch = "wasm32"), target_os = "wasi"), not(any(unix, windows))))]also matches WASI sincetarget_os = "wasi"satisfies the first part and WASI is neither unix nor windowsThis could cause
posix::module_def(ctx)to be included twice in the vector for WASI builds.#!/bin/bash # Verify the cfg conditions for WASI target # Check if both conditions can be true simultaneously for wasi echo "Checking posix module cfg conditions in mod.rs..." rg -n "posix::module_def" crates/vm/src/stdlib/mod.rs -B2 -A1 echo "" echo "Checking how other files handle WASI posix module:" rg -n "target_os.*wasi" crates/vm/src/stdlib/mod.rs
🧹 Nitpick comments (4)
crates/vm/src/vm/interpreter.rs (1)
438-474: Consider tracking the FIXME about duplicate frozen core_modules.
If you’d like, I can help draft a follow-up to avoid double-generation while keepingwithout_stdlib()andinit_stdlib()consistent.crates/derive-impl/src/pymodule.rs (1)
56-90: Consider guardingdef.slots.execassignment or documenting thatmodule_execmust not be cfg-gated.The detection at lines 86-90 sets
has_module_exec = truebased on the function name alone, without checkingfunc.attrsfor#[cfg]guards. While no cfg-gatedmodule_execfunctions currently exist in the codebase (all use conditional code inside the function body instead), the implementation remains fragile: if a user adds a top-level#[cfg]-gatedmodule_exec, the fallback suppression (line 149) combined with the unconditional reference at line 142 would cause silent build failures on excluded configurations.Prevent this by either checking cfg attributes during detection, conditionally wrapping the
def.slots.execassignment, or documenting thatmodule_execcannot be cfg-gated at the function level.crates/vm/src/stdlib/os.rs (1)
1900-1924: Consider error propagation inmodule_exec.The function uses multiple
unwrap()calls (lines 1906, 1908, 1911, 1914) that could panic during module initialization ifget_attroraddfails. While the current callers inposix.rsandnt.rsdon't handle errors from this function, propagating errors would be more consistent with the pattern inerrno.rswhich uses?for error handling.Given the callers currently ignore the result anyway (they just call
os::module_exec(vm, module);), this is a lower-priority refactor that could be addressed in a follow-up.♻️ Suggested change for error propagation
-pub fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) { +pub fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { let support_funcs = _os::support_funcs(); let supports_fd = PySet::default().into_ref(&vm.ctx); let supports_dir_fd = PySet::default().into_ref(&vm.ctx); let supports_follow_symlinks = PySet::default().into_ref(&vm.ctx); for support in support_funcs { - let func_obj = module.get_attr(support.name, vm).unwrap(); + let func_obj = module.get_attr(support.name, vm)?; if support.fd.unwrap_or(false) { - supports_fd.clone().add(func_obj.clone(), vm).unwrap(); + supports_fd.clone().add(func_obj.clone(), vm)?; } if support.dir_fd.unwrap_or(false) { - supports_dir_fd.clone().add(func_obj.clone(), vm).unwrap(); + supports_dir_fd.clone().add(func_obj.clone(), vm)?; } if support.follow_symlinks.unwrap_or(false) { - supports_follow_symlinks.clone().add(func_obj, vm).unwrap(); + supports_follow_symlinks.clone().add(func_obj, vm)?; } } // ... extend_module! ... + Ok(()) }This would require updating callers in
posix.rs,nt.rs, andposix_compat.rsto propagate the error.crates/vm/src/vm/mod.rs (1)
231-232: Minor grammar issue in error message.Consider: "
rustpython_pylibmay not be set" instead of "maybe not set" for grammatical correctness.Suggested fix
let guide_message = if cfg!(feature = "freeze-stdlib") { - "`rustpython_pylib` maybe not set while using `freeze-stdlib` feature. Try using `rustpython::InterpreterBuilder::init_stdlib` or manually call `builder.add_frozen_modules(rustpython_pylib::FROZEN_STDLIB)` in `rustpython_vm::Interpreter::builder()`." + "`rustpython_pylib` may not be set while using `freeze-stdlib` feature. Try using `rustpython::InterpreterBuilder::init_stdlib` or manually call `builder.add_frozen_modules(rustpython_pylib::FROZEN_STDLIB)` in `rustpython_vm::Interpreter::builder()`."
| pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { | ||
| __module_exec(vm, module); | ||
|
|
||
| for (name, code) in ERROR_CODES { | ||
| let name = vm.ctx.intern_str(*name); | ||
| let code = vm.new_pyobj(*code); | ||
| module.set_attr(name, code, vm).unwrap(); | ||
| module.set_attr(name, code, vm)?; | ||
| } | ||
|
|
||
| setup_module_exceptions(module, vm); | ||
| setup_module_exceptions(module.as_object(), vm); | ||
|
|
||
| let _ = CONVERTERS.set(vm.ctx.new_dict()); | ||
| let _ = ADAPTERS.set(vm.ctx.new_dict()); | ||
| let _ = USER_FUNCTION_EXCEPTION.set(PyAtomicRef::from(None)); | ||
| let _ = ENABLE_TRACEBACK.set(Radium::new(false)); | ||
|
|
||
| module | ||
| .set_attr("converters", converters().to_owned(), vm) | ||
| .unwrap(); | ||
| module | ||
| .set_attr("adapters", adapters().to_owned(), vm) | ||
| .unwrap(); | ||
| module.set_attr("converters", converters().to_owned(), vm)?; | ||
| module.set_attr("adapters", adapters().to_owned(), vm)?; | ||
|
|
||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Return value of __module_exec is discarded; otherwise LGTM.
The module_exec function properly:
- Sets up error codes with
?propagation (line 853) - Configures exceptions via
setup_module_exceptions - Initializes converters/adapters
- Uses
?forset_attrcalls (lines 863-864)
However, line 848 discards the return value of __module_exec(vm, module). Consider propagating any potential errors.
🔧 Suggested fix
pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> {
- __module_exec(vm, module);
+ __module_exec(vm, module)?;
for (name, code) in ERROR_CODES {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { | |
| __module_exec(vm, module); | |
| for (name, code) in ERROR_CODES { | |
| let name = vm.ctx.intern_str(*name); | |
| let code = vm.new_pyobj(*code); | |
| module.set_attr(name, code, vm).unwrap(); | |
| module.set_attr(name, code, vm)?; | |
| } | |
| setup_module_exceptions(module, vm); | |
| setup_module_exceptions(module.as_object(), vm); | |
| let _ = CONVERTERS.set(vm.ctx.new_dict()); | |
| let _ = ADAPTERS.set(vm.ctx.new_dict()); | |
| let _ = USER_FUNCTION_EXCEPTION.set(PyAtomicRef::from(None)); | |
| let _ = ENABLE_TRACEBACK.set(Radium::new(false)); | |
| module | |
| .set_attr("converters", converters().to_owned(), vm) | |
| .unwrap(); | |
| module | |
| .set_attr("adapters", adapters().to_owned(), vm) | |
| .unwrap(); | |
| module.set_attr("converters", converters().to_owned(), vm)?; | |
| module.set_attr("adapters", adapters().to_owned(), vm)?; | |
| Ok(()) | |
| } | |
| pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { | |
| __module_exec(vm, module)?; | |
| for (name, code) in ERROR_CODES { | |
| let name = vm.ctx.intern_str(*name); | |
| let code = vm.new_pyobj(*code); | |
| module.set_attr(name, code, vm)?; | |
| } | |
| setup_module_exceptions(module.as_object(), vm); | |
| let _ = CONVERTERS.set(vm.ctx.new_dict()); | |
| let _ = ADAPTERS.set(vm.ctx.new_dict()); | |
| let _ = USER_FUNCTION_EXCEPTION.set(PyAtomicRef::from(None)); | |
| let _ = ENABLE_TRACEBACK.set(Radium::new(false)); | |
| module.set_attr("converters", converters().to_owned(), vm)?; | |
| module.set_attr("adapters", adapters().to_owned(), vm)?; | |
| Ok(()) | |
| } |
🤖 Prompt for AI Agents
In `@crates/stdlib/src/_sqlite3.rs` around lines 847 - 867, The call to
__module_exec(vm, module) in module_exec currently discards its Result; change
the invocation to propagate errors (e.g., use __module_exec(vm, module)? or
otherwise handle its PyResult) so that any error returned by __module_exec is
forwarded from module_exec; ensure module_exec still returns PyResult<()> and
that subsequent initialization (ERROR_CODES loop, setup_module_exceptions,
CONVERTERS/ADAPTERS/USER_FUNCTION_EXCEPTION/ENABLE_TRACEBACK setup, and
module.set_attr calls) only runs after successful __module_exec.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
__module_exec returns ()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@youknowone, thank you for the clarification! You're correct—if __module_exec returns () (unit type) rather than PyResult<()>, then the current code is correct as written. The call on line 848 doesn't need the ? operator since there's no Result to propagate.
The module_exec wrapper properly:
- Invokes the infallible
__module_exec(vm, module) - Performs additional fallible setup (error code registration, module attributes) with proper
?propagation - Returns
PyResult<()>to handle any errors from the subsequent operations
The implementation correctly separates the auto-generated initialization (__module_exec) from the module-specific setup that can fail.
| pub(crate) fn module_exec( | ||
| vm: &VirtualMachine, | ||
| module: &Py<crate::builtins::PyModule>, | ||
| ) -> PyResult<()> { | ||
| __module_exec(vm, module); | ||
|
|
||
| use super::{Generic, ParamSpec, ParamSpecArgs, ParamSpecKwargs, TypeVar, TypeVarTuple}; | ||
|
|
||
| TypeVar::make_class(&vm.ctx); | ||
| ParamSpec::make_class(&vm.ctx); | ||
| TypeVarTuple::make_class(&vm.ctx); | ||
| ParamSpecArgs::make_class(&vm.ctx); | ||
| ParamSpecKwargs::make_class(&vm.ctx); | ||
| Generic::make_class(&vm.ctx); | ||
|
|
||
| extend_module!(vm, module, { | ||
| "NoDefault" => vm.ctx.typing_no_default.clone(), | ||
| "TypeVar" => TypeVar::class(&vm.ctx).to_owned(), | ||
| "ParamSpec" => ParamSpec::class(&vm.ctx).to_owned(), | ||
| "TypeVarTuple" => TypeVarTuple::class(&vm.ctx).to_owned(), | ||
| "ParamSpecArgs" => ParamSpecArgs::class(&vm.ctx).to_owned(), | ||
| "ParamSpecKwargs" => ParamSpecKwargs::class(&vm.ctx).to_owned(), | ||
| "Generic" => Generic::class(&vm.ctx).to_owned(), | ||
| "Union" => vm.ctx.types.union_type.to_owned(), | ||
| }); | ||
|
|
||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Return value of __module_exec is discarded.
Similar to other modules, __module_exec(vm, module) on line 195 doesn't propagate its result. If the macro-generated initialization fails, the error will be silently ignored.
🔧 Suggested fix
pub(crate) fn module_exec(
vm: &VirtualMachine,
module: &Py<crate::builtins::PyModule>,
) -> PyResult<()> {
- __module_exec(vm, module);
+ __module_exec(vm, module)?;
use super::{Generic, ParamSpec, ParamSpecArgs, ParamSpecKwargs, TypeVar, TypeVarTuple};🤖 Prompt for AI Agents
In `@crates/vm/src/stdlib/typing.rs` around lines 191 - 218, The call to
__module_exec(vm, module) inside module_exec currently discards its PyResult, so
initialization errors are ignored; change the call to propagate errors (e.g.,
use the ? operator) so that module_exec returns early with the error when
__module_exec fails—update the invocation in the module_exec function to
propagate the result rather than ignoring it.
bed12ed to
04a9da3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/vm/src/stdlib/os.rs (1)
1900-1923: Replaceunwrap()calls inos::module_execwith?operator; change return type toPyResult<()>and update callers.The current
module_execfunction panics onget_attr()oradd()failures (OOM, missing attributes). Change the return type toPyResult<()>, replace the fourunwrap()calls with?operators, and update the three callers inposix.rs,nt.rs, andposix_compat.rsto use?when callingsuper::super::os::module_exec(vm, module).🔧 Proposed fix
-pub fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) { +pub fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { let support_funcs = _os::support_funcs(); let supports_fd = PySet::default().into_ref(&vm.ctx); let supports_dir_fd = PySet::default().into_ref(&vm.ctx); let supports_follow_symlinks = PySet::default().into_ref(&vm.ctx); for support in support_funcs { - let func_obj = module.get_attr(support.name, vm).unwrap(); + let func_obj = module.get_attr(support.name, vm)?; if support.fd.unwrap_or(false) { - supports_fd.clone().add(func_obj.clone(), vm).unwrap(); + supports_fd.clone().add(func_obj.clone(), vm)?; } if support.dir_fd.unwrap_or(false) { - supports_dir_fd.clone().add(func_obj.clone(), vm).unwrap(); + supports_dir_fd.clone().add(func_obj.clone(), vm)?; } if support.follow_symlinks.unwrap_or(false) { - supports_follow_symlinks.clone().add(func_obj, vm).unwrap(); + supports_follow_symlinks.clone().add(func_obj, vm)?; } } extend_module!(vm, module, { "supports_fd" => supports_fd, "supports_dir_fd" => supports_dir_fd, "supports_follow_symlinks" => supports_follow_symlinks, "error" => vm.ctx.exceptions.os_error.to_owned(), }); + Ok(()) }Update callers:
posix.rs:super::super::os::module_exec(vm, module)?;nt.rs:super::super::os::module_exec(vm, module)?;posix_compat.rs:super::super::os::module_exec(vm, module)?;This aligns with the pattern used in other stdlib modules (
sysconfigdata,errno,io,signal) that returnPyResult<()>and propagate errors rather than panicking during initialization.
🤖 Fix all issues with AI agents
In `@crates/derive-impl/src/pymodule.rs`:
- Around line 85-90: The derive currently marks context.has_module_exec = true
when it sees an Item::Fn named module_exec but doesn't handle cfg-gated
declarations, which leads to emitting def.slots.exec = Some(module_exec)
unconditionally; update the visitor in pymodule.rs to detect if the
function/item named module_exec has any #[cfg(...)] attributes (inspect
func.attrs for a Meta path "cfg") and, if so, emit a compile_error (or return a
syn::Error) rejecting cfg‑gated module_exec with a clear message that the cfg
must be moved to the containing module or a non-gated fallback provided, instead
of setting context.has_module_exec = true and generating def.slots.exec =
Some(module_exec).
In `@crates/stdlib/src/pyexpat.rs`:
- Around line 53-65: The submodules "model" and "errors" created in module_exec
(variables model and errors) are attached to the pyexpat module but not
registered in sys.modules with their qualified names; update module_exec to
insert entries for "pyexpat.model" and "pyexpat.errors" into the VM's
sys.modules mapping after creating the submodules and before returning (use
vm.sys_module or the VM API to access sys.modules), ensuring the Py<PyModule>
values for model and errors are stored under those qualified keys so imports and
introspection that expect sys.modules["pyexpat.model"] /
sys.modules["pyexpat.errors"] will work as in CPython.
In `@crates/vm/src/stdlib/ast/python.rs`:
- Around line 100-107: The call to __module_exec(...) currently ignores its
PyResult, letting initialization errors be dropped; change module_exec to
propagate the error from __module_exec (e.g. use the ? operator or return on
Err) before calling super::super::pyast::extend_module_nodes so that failures in
__module_exec are returned as Err(PyErr) instead of always returning Ok(()).
Ensure you reference the existing functions __module_exec and module_exec and
preserve the subsequent call to extend_module_nodes only after successful
__module_exec.
In `@crates/vm/src/stdlib/sys.rs`:
- Around line 168-174: The returned tuple from builtin_module_names may contain
duplicate "sys" or "builtins" entries; in the builtin_module_names function,
after collecting and pushing module names (module_names) and sorting, remove
duplicates (e.g., dedupe the Vec<String> in-place) before converting to a
PyTupleRef so the exported tuple contains unique module names only.
♻️ Duplicate comments (5)
crates/wasm/src/vm_class.rs (1)
24-35: Propagate__module_execerrors (already noted earlier).
Line 30 ignores thePyResultfrom__module_exec, so init errors won’t propagate.crates/vm/src/stdlib/typing.rs (1)
191-218: Propagate __module_exec errors.Same issue:
__module_exec(vm, module);discards errors; use?to avoid silent init failures.crates/vm/src/stdlib/ctypes.rs (1)
1281-1311: Propagate __module_exec errors.Same issue: ignoring the Result hides init failures.
crates/vm/src/stdlib/imp.rs (1)
122-149: Removesys.modulesentry if exec fails.If the exec slot errors, a partially initialized module remains cached. Clean it up before propagating the error to match CPython’s behavior.
🔧 Proposed fix
- if let Some(exec) = def.slots.exec { - exec(vm, &module)?; - } + if let Some(exec) = def.slots.exec { + if let Err(err) = exec(vm, &module) { + let _ = sys_modules.del_item(&*name, vm); + return Err(err); + } + }crates/vm/src/stdlib/mod.rs (1)
96-102: Avoid duplicateposixmodule registration on WASI.Both cfgs can match for
target_os = "wasi", causingposix::module_def(ctx)to be included twice. Consider narrowing the first entry to unix-only (or remove one entry).🛠 Suggested fix
- #[cfg(any(unix, target_os = "wasi"))] + #[cfg(unix)] posix::module_def(ctx),
| // Check if module_exec function is already defined | ||
| if let Item::Fn(func) = item | ||
| && func.sig.ident == "module_exec" | ||
| { | ||
| context.has_module_exec = true; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
# First, let's explore the file structure and locate the file
find . -name "pymodule.rs" -type fRepository: RustPython/RustPython
Length of output: 101
🏁 Script executed:
# Read the context around lines 85-90 in the file to understand the code
cat -n crates/derive-impl/src/pymodule.rs | sed -n '75,100p'Repository: RustPython/RustPython
Length of output: 1226
🏁 Script executed:
# Search for any cfg-gated module_exec definitions in the entire codebase
rg --type=rust -B5 'fn\s+module_exec' --multiline-dottedRepository: RustPython/RustPython
Length of output: 519
🏁 Script executed:
# Search for where def.slots.exec is set or assigned
rg --type=rust 'def\.slots\.exec|slots\.exec\s*=' -B2 -A2Repository: RustPython/RustPython
Length of output: 752
🏁 Script executed:
# Search for module_exec with surrounding context, checking for #[cfg(...)]
rg --type=rust -B10 'fn\s+module_exec' | head -50Repository: RustPython/RustPython
Length of output: 2866
🏁 Script executed:
# Search specifically for cfg-gated module_exec
rg --type=rust '#\[cfg' -A10 | rg -A10 'module_exec'Repository: RustPython/RustPython
Length of output: 2829
🏁 Script executed:
# Find where context.has_module_exec is used
rg --type=rust 'has_module_exec' -B2 -A2Repository: RustPython/RustPython
Length of output: 1176
🏁 Script executed:
# Find where def.slots.exec = Some(module_exec) is generated with more context
rg --type=rust 'def\.slots\.exec\s*=\s*Some' -B10 -A5 crates/derive-impl/src/pymodule.rsRepository: RustPython/RustPython
Length of output: 769
🏁 Script executed:
# Also search for where the module definition is generated to understand full flow
rg --type=rust 'gen_module_def|ModuleDef' crates/derive-impl/src/pymodule.rs | head -20Repository: RustPython/RustPython
Length of output: 293
🏁 Script executed:
# Search for any actual cfg-gated module_exec definitions that might have caused issues
rg --type=rust -B3 '#\[cfg' -A3 | rg -B3 -A3 'fn\s+module_exec'Repository: RustPython/RustPython
Length of output: 1309
🏁 Script executed:
# Read the full context around where has_module_exec is used to understand the code generation
cat -n crates/derive-impl/src/pymodule.rs | sed -n '110,150p'Repository: RustPython/RustPython
Length of output: 1913
🏁 Script executed:
# Check if there's any conditional generation based on has_module_exec
rg --type=rust 'has_module_exec' -B5 -A10 crates/derive-impl/src/pymodule.rsRepository: RustPython/RustPython
Length of output: 1806
Reject cfg‑gated module_exec to prevent build failures.
When a user defines module_exec with a #[cfg(...)] attribute, the code checks for its presence (line 87) and sets context.has_module_exec = true. However, the code generation at line 142 unconditionally generates def.slots.exec = Some(module_exec); regardless of whether module_exec is cfg-gated. In builds where the cfg condition is false, the function won't exist but the code still references it, causing a hard compile error.
The suggested diagnostic check is appropriate: detect cfg-gated module_exec and emit an error, directing users to either move the cfg to the containing module or provide an explicit fallback.
🤖 Prompt for AI Agents
In `@crates/derive-impl/src/pymodule.rs` around lines 85 - 90, The derive
currently marks context.has_module_exec = true when it sees an Item::Fn named
module_exec but doesn't handle cfg-gated declarations, which leads to emitting
def.slots.exec = Some(module_exec) unconditionally; update the visitor in
pymodule.rs to detect if the function/item named module_exec has any #[cfg(...)]
attributes (inspect func.attrs for a Meta path "cfg") and, if so, emit a
compile_error (or return a syn::Error) rejecting cfg‑gated module_exec with a
clear message that the cfg must be moved to the containing module or a non-gated
fallback provided, instead of setting context.has_module_exec = true and
generating def.slots.exec = Some(module_exec).
| pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { | ||
| __module_exec(vm, module); | ||
|
|
||
| // Add submodules | ||
| let model = super::_model::module_def(&vm.ctx).create_module(vm)?; | ||
| let errors = super::_errors::module_def(&vm.ctx).create_module(vm)?; | ||
|
|
||
| extend_module!(vm, module, { | ||
| "model" => model, | ||
| "errors" => errors, | ||
| }); | ||
|
|
||
| Ok(()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
# First, locate the file and understand the current implementation
find . -name "pyexpat.rs" -type fRepository: RustPython/RustPython
Length of output: 95
🏁 Script executed:
# Search for how other modules handle submodule registration in RustPython
fd -e rs -type f | xargs grep -l "sys_modules\|sys\.modules" | head -20Repository: RustPython/RustPython
Length of output: 235
🏁 Script executed:
# Look for any tests that use pyexpat.model or pyexpat.errors
rg -n "pyexpat\.(model|errors)" --type rsRepository: RustPython/RustPython
Length of output: 91
🏁 Script executed:
# Check for existing submodule patterns in the codebase
rg -n "create_module" --type rs -A 5 | head -60Repository: RustPython/RustPython
Length of output: 91
🏁 Script executed:
# Look at the _model and _errors module definitions
fd -e rs -type f -path "*pyexpat*" | xargs ls -laRepository: RustPython/RustPython
Length of output: 3391
🏁 Script executed:
# Read the pyexpat.rs file
cat -n crates/stdlib/src/pyexpat.rs | head -100Repository: RustPython/RustPython
Length of output: 4159
🏁 Script executed:
# Search for sys_modules usage patterns
rg -n "sys_modules\|sys\.modules" | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Search for create_module calls to see submodule patterns
rg -n "create_module" -A 3 | head -40Repository: RustPython/RustPython
Length of output: 2163
🏁 Script executed:
# Look for pyexpat tests
find . -name "*test*" -o -name "*tests*" | xargs grep -l "pyexpat" 2>/dev/null | head -5Repository: RustPython/RustPython
Length of output: 143
🏁 Script executed:
# Search for how other stdlib modules register submodules
rg -n "extend_module" -B 5 -A 5 | head -80Repository: RustPython/RustPython
Length of output: 5217
🏁 Script executed:
# Check the pyexpat test file to see how submodules are used
cat Lib/test/test_pyexpat.py | head -150Repository: RustPython/RustPython
Length of output: 4977
🏁 Script executed:
# Search for model/errors attribute access in test files
rg -n "pyexpat\.(model|errors)" Lib/Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at how create_module is implemented
rg -n "fn create_module" -A 10Repository: RustPython/RustPython
Length of output: 980
🏁 Script executed:
# Check other stdlib modules that have submodules for comparison
find crates/stdlib/src -name "*.rs" -type f | head -20Repository: RustPython/RustPython
Length of output: 699
🏁 Script executed:
# Look for other submodule patterns - search for modules that might have nested structure
rg -n "module_exec.*PyResult" -A 15 crates/stdlib/src/ | head -100Repository: RustPython/RustPython
Length of output: 6065
🏁 Script executed:
# Search for any import statements in tests that use qualified submodule syntax
rg -n "import.*pyexpat\." Lib/Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if there are any tests that try to import pyexpat.model or pyexpat.errors
rg -n "import pyexpat" Lib/test/ -A 5Repository: RustPython/RustPython
Length of output: 344
🏁 Script executed:
# Look at the _model and _errors submodule definitions in pyexpat
find crates/stdlib/src -name "*model*" -o -name "*errors*" | grep pyexpatRepository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Search for how other modules with submodules handle sys.modules registration
rg -n "__name__.*=.*\|sys_modules.*set" crates/stdlib/src/ | head -30Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at the extend_module macro implementation more closely
cat -n crates/vm/src/macros.rs | head -20Repository: RustPython/RustPython
Length of output: 1051
🏁 Script executed:
# Look at how __module_set_attr is implemented
rg -n "__module_set_attr" crates/vm/src/ -A 5 | head -40Repository: RustPython/RustPython
Length of output: 1128
🏁 Script executed:
# Search for how modules get registered in sys.modules
rg -n "sys_modules\|sys\.modules" crates/vm/src/builtins/module.rs -B 3 -A 3Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if there are any tests that specifically test pyexpat module attributes or imports
cat Lib/test/test_pyexpat.py | grep -n "expat\.\|pyexpat\." | head -20Repository: RustPython/RustPython
Length of output: 1229
🏁 Script executed:
# Look at the module.rs file to understand module creation and sys.modules registration
cat -n crates/vm/src/builtins/module.rs | head -80Repository: RustPython/RustPython
Length of output: 3343
🏁 Script executed:
# Search for how PyModule::from_def works and if it sets __name__ automatically
rg -n "from_def\|PyModule::new" crates/vm/src/builtins/module.rs -A 10 | head -60Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at how PyModule::from_def works and what __name__ is set to
rg -n "from_def\|__init_dict_from_def" crates/vm/src/builtins/module.rs -A 20 | head -80Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check how the pymodule macro initializes module __name__
rg -n "#\[pymodule" crates/stdlib/src/pyexpat.rs -B 2 -A 10Repository: RustPython/RustPython
Length of output: 638
🏁 Script executed:
# Look for how __name__ is set on modules during creation
rg -n "__name__\|\"__name__\"" crates/vm/src/builtins/module.rs -B 2 -A 2 | head -40Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Test if qualified imports actually work by looking at import system
rg -n "import_submodule\|find_submodule" crates/vm/src/ | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check how CPython's pyexpat.errors and pyexpat.model are used
rg -n "expat\.errors\|expat\.model" Lib/test/test_pyexpat.pyRepository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at how RustPython imports work, especially for submodules
find crates/vm/src -name "*import*" -type fRepository: RustPython/RustPython
Length of output: 88
🏁 Script executed:
# Check the import implementation
rg -n "def import_module\|import_from\|__import__" crates/vm/src/ | head -30Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look for how the import machinery handles parent.child imports
rg -n "parent.*module\|submodule" crates/vm/src/stdlib/ | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if there are any tests that test qualified submodule imports
find Lib/test -name "*.py" -exec grep -l "import.*\\..*" {} \; 2>/dev/null | head -5Repository: RustPython/RustPython
Length of output: 195
🏁 Script executed:
# Search the codebase for examples of modules with submodules that ARE registered in sys.modules
rg -n "set_item.*sys" crates/stdlib/src/ | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at how json module handles its submodules (as json has encoder/decoder)
grep -rn "json" crates/stdlib/src/ | grep module_exec | head -5Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at the import system implementation
cat -n crates/vm/src/import.rs | head -200Repository: RustPython/RustPython
Length of output: 8431
🏁 Script executed:
# Check if there are any package/namespace tests
rg -n "import.*\\..*as\|from.*\\..*import" Lib/test/ | grep -v "Binary\|\.pyc" | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look specifically for how the pymodule macro generates __name__
rg -n "pymodule\|__name__" crates/vm/src/py_serde.rs | head -40Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check the implementation of module initialization in the macro
find crates -name "*.rs" -exec grep -l "pymodule" {} \; | head -5Repository: RustPython/RustPython
Length of output: 3529
🏁 Script executed:
# Look at a stdlib module that imports submodules to see pattern
rg -n "module_exec\|__module_exec" crates/stdlib/src/json/ -A 10 | head -60Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check the pymodule macro implementation to understand how __name__ is set
cat -n crates/derive-impl/src/pymodule.rs | head -150Repository: RustPython/RustPython
Length of output: 6221
🏁 Script executed:
# Look for any tests that import qualified submodules
rg -n "from pyexpat import\|import pyexpat\." Lib/test/Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check Python standard library behavior for pyexpat submodules
python3 -c "import pyexpat; print('model' in dir(pyexpat)); import pyexpat.errors; print(pyexpat.errors.__name__)" 2>&1 || trueRepository: RustPython/RustPython
Length of output: 151
🏁 Script executed:
# Check if there's documentation or tests about how RustPython handles submodule imports
rg -n "submodule\|parent.*module" Lib/test/ | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at what __name__ is set to for the submodules created via create_module
rg -n "PyModule.*__name__\|set.*__name__" crates/vm/src/builtins/module.rs -B 2 -A 2Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Let me check if there are any tests specifically for pyexpat.model or pyexpat.errors
cat Lib/test/test_pyexpat.py | grep -n "\.model\|\.errors" | head -20Repository: RustPython/RustPython
Length of output: 127
🏁 Script executed:
# Check what happens with qualified module imports in RustPython
rg -n "import.*\\..*import" Lib/test/test_import*.py | head -20Repository: RustPython/RustPython
Length of output: 129
🏁 Script executed:
# Look at how modules set their __name__ attribute during creation
cat -n crates/vm/src/builtins/module.rs | sed -n '1,200p' | tail -100Repository: RustPython/RustPython
Length of output: 4025
🏁 Script executed:
# Check if there's anything about parent module setup in module creation
rg -n "__package__\|__path__" crates/vm/src/builtins/module.rsRepository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if there are any tests or code that use import pyexpat.model or pyexpat.errors
rg -n "import.*pyexpat\." Lib/test/test_pyexpat.pyRepository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at the test more carefully to see if qualified imports are tested
cat -n Lib/test/test_pyexpat.py | sed -n '1,50p'Repository: RustPython/RustPython
Length of output: 2047
🏁 Script executed:
# Check how create_module is implemented to understand what __name__ gets set
cat -n crates/vm/src/builtins/module.rs | sed -n '44,75p'Repository: RustPython/RustPython
Length of output: 1461
🏁 Script executed:
# Verify if vm.sys_module.get_attr is available and the set_item API works
rg -n "set_item\|get_attr" crates/stdlib/src/pyexpat.rsRepository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look for similar patterns in other modules to see if anyone else registers submodules
rg -n "sys_modules.*set_item" crates/stdlib/src/Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if test_pyexpat.py or other tests actually try qualified imports like "import pyexpat.errors"
grep -rn "import pyexpat" Lib/test/ | grep -v "from pyexpat"Repository: RustPython/RustPython
Length of output: 109
🏁 Script executed:
# Look at what the test actually expects - check line 275 more closely
sed -n '270,280p' Lib/test/test_pyexpat.pyRepository: RustPython/RustPython
Length of output: 600
🏁 Script executed:
# Check how the xml.parsers.expat module relates to pyexpat
find . -name "expat.py" -o -name "*expat*" -path "*/xml/parsers/*"Repository: RustPython/RustPython
Length of output: 91
🏁 Script executed:
# See if there are any imports of pyexpat.model or pyexpat.errors in the entire codebase
rg -rn "pyexpat\.model\|pyexpat\.errors"Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check the actual behavior - what __name__ does the model submodule have
rg -n "model\|errors" crates/stdlib/src/pyexpat.rs | grep -E "pymodule|module_def"Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at the xml.parsers.expat module to understand the relationship
cat -n Lib/xml/parsers/expat.py | head -50Repository: RustPython/RustPython
Length of output: 368
🏁 Script executed:
# Check if expat.py re-exports or wraps pyexpat
grep -n "pyexpat\|from.*import" Lib/xml/parsers/expat.py | head -30Repository: RustPython/RustPython
Length of output: 151
🏁 Script executed:
# Look for how sys.modules handles submodule registrations in import tests
find Lib/test -name "*.py" -exec grep -l "sys.modules" {} \; | head -5Repository: RustPython/RustPython
Length of output: 3453
🏁 Script executed:
# Check if there's any code that tests or uses qualified submodule imports
grep -rn "sys\.modules\[.*\.\|set_item.*\." Lib/test/ | head -20Repository: RustPython/RustPython
Length of output: 1996
🏁 Script executed:
# Check how the import system finds submodules - look for the import machinery
grep -n "find.*submodule\|parent.*module" crates/vm/src/import.rsRepository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# See the full expat.py file to understand how it uses the submodules
cat -n Lib/xml/parsers/expat.pyRepository: RustPython/RustPython
Length of output: 368
🏁 Script executed:
# Check if RustPython's import system is in importlib (frozen)
find . -name "*importlib*" -type f | grep -v ".pyc\|test"Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Test whether qualified imports actually work in RustPython by looking for test cases
rg -n "import.*\\..*\|sys.modules.*\\." Lib/test/test_importlib/ | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if there are any actual tests that try to import pyexpat.model or pyexpat.errors directly
grep -rn "import pyexpat\.model\|import pyexpat\.errors\|from pyexpat import model\|from pyexpat import errors" Lib/Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at what attributes are accessible on pyexpat module
cat -n crates/stdlib/src/pyexpat.rs | sed -n '53,75p'Repository: RustPython/RustPython
Length of output: 917
🏁 Script executed:
# Check if the import system in RustPython handles qualified imports
rg -n "resolve.*parent\|split.*module.*name" crates/vm/src/import.rs | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look for how other builtin modules with submodules handle this
find crates/stdlib/src -name "*.rs" -exec grep -l "super::_" {} \; | head -10Repository: RustPython/RustPython
Length of output: 150
🏁 Script executed:
# Final check - see if there's a test that verifies pyexpat.errors can be imported as a qualified name
rg -rn "pyexpat\.errors\|pyexpat\.model" Lib/test/ 2>/dev/nullRepository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look for documentation on how Python handles module.submodule imports
grep -rn "__path__" crates/vm/src/builtins/module.rsRepository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if pyexpat should be a package by looking at CPython source patterns
rg -n "is_package\|__path__" crates/stdlib/src/ | head -20Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Verify what __name__ is actually set to for the submodules by checking module creation
cat -n crates/vm/src/builtins/module.rs | sed -n '119,130p'Repository: RustPython/RustPython
Length of output: 513
🏁 Script executed:
# Check if there's any handling of qualified module imports in RustPython's import system
rg -n "\.split\(.*\.\)" crates/vm/src/import.rs | head -10Repository: RustPython/RustPython
Length of output: 47
Register pyexpat submodules with qualified names in sys.modules for CPython compatibility.
The current implementation attaches model and errors submodules as attributes but doesn't register them in sys.modules with qualified names (pyexpat.model, pyexpat.errors). This follows the pattern shown in Lib/xml/parsers/expat.py, which explicitly registers submodules in sys.modules. While RustPython's import system may not currently support qualified submodule imports like import pyexpat.model, registering them ensures compatibility with code that expects this behavior and with introspection tools that check sys.modules.
💡 Suggested fix
pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> {
__module_exec(vm, module);
// Add submodules
- let model = super::_model::module_def(&vm.ctx).create_module(vm)?;
- let errors = super::_errors::module_def(&vm.ctx).create_module(vm)?;
+ let sys_modules = vm.sys_module.get_attr("modules", vm)?;
+
+ let model = super::_model::module_def(&vm.ctx).create_module(vm)?;
+ model.set_attr("__name__", vm.ctx.new_str("pyexpat.model"), vm)?;
+ model.set_attr("__package__", vm.ctx.new_str("pyexpat"), vm)?;
+ sys_modules.set_item("pyexpat.model", model.clone().into(), vm)?;
+
+ let errors = super::_errors::module_def(&vm.ctx).create_module(vm)?;
+ errors.set_attr("__name__", vm.ctx.new_str("pyexpat.errors"), vm)?;
+ errors.set_attr("__package__", vm.ctx.new_str("pyexpat"), vm)?;
+ sys_modules.set_item("pyexpat.errors", errors.clone().into(), vm)?;
extend_module!(vm, module, {
"model" => model,
"errors" => errors,
});🤖 Prompt for AI Agents
In `@crates/stdlib/src/pyexpat.rs` around lines 53 - 65, The submodules "model"
and "errors" created in module_exec (variables model and errors) are attached to
the pyexpat module but not registered in sys.modules with their qualified names;
update module_exec to insert entries for "pyexpat.model" and "pyexpat.errors"
into the VM's sys.modules mapping after creating the submodules and before
returning (use vm.sys_module or the VM API to access sys.modules), ensuring the
Py<PyModule> values for model and errors are stored under those qualified keys
so imports and introspection that expect sys.modules["pyexpat.model"] /
sys.modules["pyexpat.errors"] will work as in CPython.
| fn builtin_module_names(vm: &VirtualMachine) -> PyTupleRef { | ||
| let mut module_names: Vec<_> = vm.state.module_inits.keys().cloned().collect(); | ||
| module_names.push("sys".into()); | ||
| module_names.push("builtins".into()); | ||
| let mut module_names: Vec<String> = | ||
| vm.state.module_defs.keys().map(|&s| s.to_owned()).collect(); | ||
| module_names.push("sys".to_owned()); | ||
| module_names.push("builtins".to_owned()); | ||
|
|
||
| module_names.sort(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deduplicate sys.builtin_module_names entries.
module_defs may already contain sys/builtins, so pushing them unconditionally can introduce duplicates in the exported tuple. Consider deduping after sort to keep the public surface clean.
🩹 Proposed fix
module_names.sort();
+ module_names.dedup();📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| fn builtin_module_names(vm: &VirtualMachine) -> PyTupleRef { | |
| let mut module_names: Vec<_> = vm.state.module_inits.keys().cloned().collect(); | |
| module_names.push("sys".into()); | |
| module_names.push("builtins".into()); | |
| let mut module_names: Vec<String> = | |
| vm.state.module_defs.keys().map(|&s| s.to_owned()).collect(); | |
| module_names.push("sys".to_owned()); | |
| module_names.push("builtins".to_owned()); | |
| module_names.sort(); | |
| fn builtin_module_names(vm: &VirtualMachine) -> PyTupleRef { | |
| let mut module_names: Vec<String> = | |
| vm.state.module_defs.keys().map(|&s| s.to_owned()).collect(); | |
| module_names.push("sys".to_owned()); | |
| module_names.push("builtins".to_owned()); | |
| module_names.sort(); | |
| module_names.dedup(); |
🤖 Prompt for AI Agents
In `@crates/vm/src/stdlib/sys.rs` around lines 168 - 174, The returned tuple from
builtin_module_names may contain duplicate "sys" or "builtins" entries; in the
builtin_module_names function, after collecting and pushing module names
(module_names) and sorting, remove duplicates (e.g., dedupe the Vec<String>
in-place) before converting to a PyTupleRef so the exported tuple contains
unique module names only.
04a9da3 to
1087d6c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 4
🤖 Fix all issues with AI agents
In `@crates/stdlib/src/openssl.rs`:
- Around line 100-108: module_exec currently ignores the PyResult returned by
__module_exec, masking initialization errors; change module_exec to propagate
that result instead of always returning Ok(()). After the existing
LazyLock::force call, call __module_exec(vm, module) and return its PyResult
(either via the ? operator or by returning the result directly) so any error
from __module_exec is forwarded to the caller.
In `@crates/vm/src/builtins/module.rs`:
- Around line 44-81: The create_module implementation must guard against create
slots returning modules with no def and pass a consistent spec object: when
self.slots.create is Some, build and pass a module spec object (not just a
string) compatible with imp.rs::create_builtin (i.e., an object with a name
attribute), then call the create slot; after creation, check the returned
module's def (module.def or module.def.is_some()) and only call
PyModule::__init_dict_from_def and module.__init_methods if the def is present;
otherwise skip those calls to avoid panics for modules created via
PyModule::new().
In `@crates/vm/src/stdlib/errno.rs`:
- Around line 9-22: The module_exec function currently calls __module_exec(vm,
module) and discards its PyResult, which can hide initialization errors; change
the call to propagate errors by using the ? operator (i.e., __module_exec(vm,
module)?;) so module_exec returns early with the underlying PyErr when
__module_exec fails, keeping the function's PyResult return type semantics
intact.
In `@crates/vm/src/stdlib/time.rs`:
- Around line 577-588: The module initialization currently calls
__module_exec(vm, module) but discards its PyResult, masking failures; update
module_exec to propagate errors by returning the result of __module_exec(vm,
module) (i.e., replace the ignored call + Ok(()) with returning __module_exec's
PyResult), preserving the existing unsafe c_tzset call and signature of
module_exec (working with VirtualMachine and Py<crate::builtins::PyModule>).
♻️ Duplicate comments (7)
crates/stdlib/src/pyexpat.rs (1)
42-66: Register pyexpat submodules in sys.modules for qualified importsThe module still attaches
model/errorsas attributes without registeringpyexpat.model/pyexpat.errorsinsys.modules, which can break qualified imports and introspection.crates/vm/src/stdlib/imp.rs (1)
118-149: Remove partially initialized modules on exec failure.
Ifexecerrors, the module stays insys.modulesand may be reused in a bad state.🛠️ Suggested fix
- if let Some(exec) = def.slots.exec { - exec(vm, &module)?; - } + if let Some(exec) = def.slots.exec { + if let Err(err) = exec(vm, &module) { + let _ = sys_modules.del_item(&*name, vm); + return Err(err); + } + }crates/vm/src/stdlib/typing.rs (1)
191-218: Propagate__module_execfailures.
The result is dropped; initialization errors can be silently ignored.🛠️ Suggested fix
- __module_exec(vm, module); + __module_exec(vm, module)?;crates/vm/src/stdlib/ctypes.rs (1)
1281-1312: Propagate__module_execfailures.
The result is dropped; initialization errors can be silently ignored.🛠️ Suggested fix
- __module_exec(vm, module); + __module_exec(vm, module)?;crates/vm/src/stdlib/sys.rs (1)
168-174: Duplicate: dedupebuiltin_module_namesafter sorting.
module_defscan already include"sys"/"builtins", so the tuple may contain duplicates.crates/derive-impl/src/pymodule.rs (1)
85-90: Duplicate: reject cfg‑gatedmodule_execto avoid missing symbol.crates/vm/src/stdlib/mod.rs (1)
96-102: Duplicateposixmodule registration on WASI targets.On WASI targets, both conditions match:
#[cfg(any(unix, target_os = "wasi"))](line 96) matches becausetarget_os = "wasi"#[cfg(all(any(not(target_arch = "wasm32"), target_os = "wasi"), not(any(unix, windows))))](lines 98-101) also matches because WASI iswasm32but hastarget_os = "wasi", and WASI is neitherunixnorwindowsThis results in
posix::module_def(ctx)being included twice in the vector for WASI builds, which could cause issues during module registration.🔧 Suggested fix
#[cfg(any(unix, target_os = "wasi"))] posix::module_def(ctx), #[cfg(all( any(not(target_arch = "wasm32"), target_os = "wasi"), - not(any(unix, windows)) + not(any(unix, windows, target_os = "wasi")) ))] posix::module_def(ctx),Alternatively, remove the WASI-specific entry since it's already covered:
- #[cfg(any(unix, target_os = "wasi"))] - posix::module_def(ctx), + #[cfg(unix)] + posix::module_def(ctx), + #[cfg(target_os = "wasi")] + posix::module_def(ctx), #[cfg(all( any(not(target_arch = "wasm32"), target_os = "wasi"), not(any(unix, windows)) ))] posix::module_def(ctx),
🧹 Nitpick comments (2)
src/interpreter.rs (2)
30-37: Potential panic if state has multiple references.The
.unwrap()onPyRc::get_mutwill panic if there are other references tovm.state. While this is likely safe during initialization (wheninit_hookruns), consider usingexpect()with a descriptive message to aid debugging if assumptions are violated.- let state = PyRc::get_mut(&mut vm.state).unwrap(); + let state = PyRc::get_mut(&mut vm.state) + .expect("VM state should be uniquely owned during init_hook");
58-76: Potential panic on non-UTF-8 paths.The
.unwrap()oninto_string()(line 64) will panic ifBUILDTIME_RUSTPYTHONPATHcontains non-UTF-8 characters. While unlikely in controlled build environments, consider usingto_string_lossy()or handling the error gracefully:- .map(|path| path.into_os_string().into_string().unwrap()) + .filter_map(|path| path.into_os_string().into_string().ok())Alternatively, if non-UTF-8 paths should cause a build/runtime failure, document this expectation.
| impl PyModuleDef { | ||
| /// Create a module from this definition (Phase 1 of multi-phase init). | ||
| /// | ||
| /// This performs: | ||
| /// 1. Create module object (using create slot if provided) | ||
| /// 2. Initialize module dict from def | ||
| /// 3. Add methods to module | ||
| /// | ||
| /// Does NOT add to sys.modules or call exec slot. | ||
| pub fn create_module(&'static self, vm: &VirtualMachine) -> PyResult<PyRef<PyModule>> { | ||
| use crate::PyPayload; | ||
|
|
||
| // Create module (use create slot if provided, else default creation) | ||
| let module = if let Some(create) = self.slots.create { | ||
| // Custom module creation | ||
| let spec = vm.ctx.new_str(self.name.as_str()); | ||
| create(vm, spec.as_object(), self)? | ||
| } else { | ||
| // Default module creation | ||
| PyModule::from_def(self).into_ref(&vm.ctx) | ||
| }; | ||
|
|
||
| // Initialize module dict and methods | ||
| PyModule::__init_dict_from_def(vm, &module); | ||
| module.__init_methods(vm)?; | ||
|
|
||
| Ok(module) | ||
| } | ||
|
|
||
| /// Execute the module's exec slot (Phase 2 of multi-phase init). | ||
| /// | ||
| /// Calls the exec slot if present. Returns Ok(()) if no exec slot. | ||
| pub fn exec_module(&'static self, vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { | ||
| if let Some(exec) = self.slots.exec { | ||
| exec(vm, module)?; | ||
| } | ||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
# Find ModuleCreate implementations and create slot definitions
rg -n "ModuleCreate" --type rs -A 5 -B 2Repository: RustPython/RustPython
Length of output: 91
🏁 Script executed:
# Find PyModuleSlots definitions with create field
rg -n "struct PyModuleSlots|create:" --type rs -B 2 -A 5Repository: RustPython/RustPython
Length of output: 91
🏁 Script executed:
# Find __init_dict_from_def implementation to check for unwrap
rg -n "__init_dict_from_def" --type rs -B 2 -A 10Repository: RustPython/RustPython
Length of output: 91
🏁 Script executed:
# Check what spec parameter is used for in create slots
rg -n "fn.*create.*spec" --type rs -B 2 -A 8Repository: RustPython/RustPython
Length of output: 91
🏁 Script executed:
# Find ModuleCreate and create slot definitions
rg "ModuleCreate|PyModuleSlots" -A 5 -B 2 -- '*.rs'Repository: RustPython/RustPython
Length of output: 109
🏁 Script executed:
# Find __init_dict_from_def implementation
rg "__init_dict_from_def" -A 10 -B 2 -- '*.rs'Repository: RustPython/RustPython
Length of output: 109
🏁 Script executed:
# Look at the module.rs file directly to understand the structure
cat crates/vm/src/builtins/module.rs | head -150Repository: RustPython/RustPython
Length of output: 4736
🏁 Script executed:
# Find where PyModule is defined and check def field
rg "pub struct PyModule|pub def:" -A 5 -- '*.rs'Repository: RustPython/RustPython
Length of output: 109
🏁 Script executed:
# Search for actual ModuleCreate implementations or assignments
rg "ModuleCreate" -B 2 -A 8Repository: RustPython/RustPython
Length of output: 1194
🏁 Script executed:
# Search for create: Some( patterns to find where ModuleCreate is used
rg "create:\s*Some" -B 2 -A 3Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if there are any tests or examples of custom create slots
rg "create.*=.*fn" -B 3 -A 5Repository: RustPython/RustPython
Length of output: 1841
🏁 Script executed:
# Look for where PyModuleSlots is instantiated with a create function
rg "PyModuleSlots\s*\{" -A 5Repository: RustPython/RustPython
Length of output: 810
🏁 Script executed:
# Search for any module creation patterns outside of the reviewed code
rg "PyModule::from_def|PyModule::new()" -B 3 -A 3Repository: RustPython/RustPython
Length of output: 1769
🏁 Script executed:
# Check the complete __init_dict_from_def and __init_methods implementations
rg "__init_dict_from_def|__init_methods" -B 2 -A 8Repository: RustPython/RustPython
Length of output: 5302
🏁 Script executed:
# Search for any actual custom module create implementations
rg "fn.*create.*PyModuleDef" -B 2 -A 10Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look for PyModule struct definition to see the def and name fields
rg "pub struct PyModule" -A 8Repository: RustPython/RustPython
Length of output: 2348
🏁 Script executed:
# Check what the type signature requires - look for how modules are created in practice
rg "PyRef::new_ref.*PyModule" -B 3 -A 3Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at imp.rs to see if the same pattern exists there
sed -n '1,200p' crates/vm/src/stdlib/imp.rs | grep -A 30 "create_module\|ModuleCreate"Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Search for any definitions or assignments of create slots
rg "create\s*:" --type-list | head -20Repository: RustPython/RustPython
Length of output: 454
🏁 Script executed:
# Look for files that might contain ModuleCreate implementations
fd -e rs | xargs grep -l "PyModuleSlots"Repository: RustPython/RustPython
Length of output: 127
🏁 Script executed:
# Check the entire module.rs file around the review area to understand context better
wc -l crates/vm/src/builtins/module.rsRepository: RustPython/RustPython
Length of output: 101
🏁 Script executed:
# Get the full module.rs file to understand all the context
cat crates/vm/src/builtins/module.rsRepository: RustPython/RustPython
Length of output: 12814
🏁 Script executed:
# Search for any actual module creation slot implementations or assignments
rg "create\s*=\s*Some" -B 3 -A 5Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if there are tests or examples that use custom ModuleCreate slots
rg "fn.*create.*vm.*spec.*def" -B 2 -A 8Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Look at imp.rs to see if the same pattern exists there
sed -n '1,300p' crates/vm/src/stdlib/imp.rs | catRepository: RustPython/RustPython
Length of output: 7377
🏁 Script executed:
# Look for any pymodule macro usage that might define custom create slots
rg "pymodule|create.*=\|slots.*=" -B 2 -A 5 | head -100Repository: RustPython/RustPython
Length of output: 5209
🏁 Script executed:
# Check if there are any integration tests or examples
fd -e rs -path "*/test*" -o -path "*/examples*" | xargs grep -l "create\|ModuleCreate" 2>/dev/null | head -10Repository: RustPython/RustPython
Length of output: 233
🏁 Script executed:
# Verify the type signature match - what spec parameter looks like in practice
rg "create\(vm\|spec" -B 3 -A 3Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Check if PyModule can be created with def=None
rg "def: None\|PyModule::new()" -B 2 -A 3Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Verify the type signature and what create slot implementations would need to return
rg "PyModule::from_def|from_def" -B 1 -A 1 | head -30Repository: RustPython/RustPython
Length of output: 1798
🏁 Script executed:
# Look for any tests or debug assertions that use custom create slots
rg "debug_assert.*def\|unwrap.*def" -B 2 -A 2Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Double-check the type signature and what ModuleCreate must return
rg "pub type ModuleCreate|ModuleCreate\s*=" -A 2Repository: RustPython/RustPython
Length of output: 348
🏁 Script executed:
# Verify __init_methods also has the same unwrap/assert pattern
rg "__init_methods" -A 8 | grep -A 8 "pub fn __init_methods"Repository: RustPython/RustPython
Length of output: 761
🏁 Script executed:
# Check if there are any other places calling __init_dict_from_def to see the pattern
rg "__init_dict_from_def" -B 3 -A 1Repository: RustPython/RustPython
Length of output: 1536
Guard create_module against create slots returning modules without def
create_module unconditionally calls __init_dict_from_def and __init_methods, both of which unwrap module.def (with __init_methods also asserting it's Some). If any custom create slot returns a module created via PyModule::new() (which sets def to None), this will panic at runtime. Additionally, the spec argument passed to create slots differs from imp.rs's create_builtin: this function passes a string, while create_builtin passes a spec object expected to have a name attribute. ModuleCreate implementations should be consistent with whichever signature they target.
🐛 Suggested guard to avoid panic
@@
// Initialize module dict and methods
+ if module.def.is_none() {
+ return Err(vm.new_runtime_error(
+ "module create slot must return a module created from this def".to_owned(),
+ ));
+ }
PyModule::__init_dict_from_def(vm, &module);
module.__init_methods(vm)?;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| impl PyModuleDef { | |
| /// Create a module from this definition (Phase 1 of multi-phase init). | |
| /// | |
| /// This performs: | |
| /// 1. Create module object (using create slot if provided) | |
| /// 2. Initialize module dict from def | |
| /// 3. Add methods to module | |
| /// | |
| /// Does NOT add to sys.modules or call exec slot. | |
| pub fn create_module(&'static self, vm: &VirtualMachine) -> PyResult<PyRef<PyModule>> { | |
| use crate::PyPayload; | |
| // Create module (use create slot if provided, else default creation) | |
| let module = if let Some(create) = self.slots.create { | |
| // Custom module creation | |
| let spec = vm.ctx.new_str(self.name.as_str()); | |
| create(vm, spec.as_object(), self)? | |
| } else { | |
| // Default module creation | |
| PyModule::from_def(self).into_ref(&vm.ctx) | |
| }; | |
| // Initialize module dict and methods | |
| PyModule::__init_dict_from_def(vm, &module); | |
| module.__init_methods(vm)?; | |
| Ok(module) | |
| } | |
| /// Execute the module's exec slot (Phase 2 of multi-phase init). | |
| /// | |
| /// Calls the exec slot if present. Returns Ok(()) if no exec slot. | |
| pub fn exec_module(&'static self, vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { | |
| if let Some(exec) = self.slots.exec { | |
| exec(vm, module)?; | |
| } | |
| Ok(()) | |
| } | |
| impl PyModuleDef { | |
| /// Create a module from this definition (Phase 1 of multi-phase init). | |
| /// | |
| /// This performs: | |
| /// 1. Create module object (using create slot if provided) | |
| /// 2. Initialize module dict from def | |
| /// 3. Add methods to module | |
| /// | |
| /// Does NOT add to sys.modules or call exec slot. | |
| pub fn create_module(&'static self, vm: &VirtualMachine) -> PyResult<PyRef<PyModule>> { | |
| use crate::PyPayload; | |
| // Create module (use create slot if provided, else default creation) | |
| let module = if let Some(create) = self.slots.create { | |
| // Custom module creation | |
| let spec = vm.ctx.new_str(self.name.as_str()); | |
| create(vm, spec.as_object(), self)? | |
| } else { | |
| // Default module creation | |
| PyModule::from_def(self).into_ref(&vm.ctx) | |
| }; | |
| // Initialize module dict and methods | |
| if module.def.is_none() { | |
| return Err(vm.new_runtime_error( | |
| "module create slot must return a module created from this def".to_owned(), | |
| )); | |
| } | |
| PyModule::__init_dict_from_def(vm, &module); | |
| module.__init_methods(vm)?; | |
| Ok(module) | |
| } | |
| /// Execute the module's exec slot (Phase 2 of multi-phase init). | |
| /// | |
| /// Calls the exec slot if present. Returns Ok(()) if no exec slot. | |
| pub fn exec_module(&'static self, vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { | |
| if let Some(exec) = self.slots.exec { | |
| exec(vm, module)?; | |
| } | |
| Ok(()) | |
| } |
🤖 Prompt for AI Agents
In `@crates/vm/src/builtins/module.rs` around lines 44 - 81, The create_module
implementation must guard against create slots returning modules with no def and
pass a consistent spec object: when self.slots.create is Some, build and pass a
module spec object (not just a string) compatible with imp.rs::create_builtin
(i.e., an object with a name attribute), then call the create slot; after
creation, check the returned module's def (module.def or module.def.is_some())
and only call PyModule::__init_dict_from_def and module.__init_methods if the
def is present; otherwise skip those calls to avoid panics for modules created
via PyModule::new().
1087d6c to
2037645
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@crates/vm/src/stdlib/io.rs`:
- Around line 4732-4744: The fileio module is missing a pub(crate) module_exec
function required by the call super::fileio::module_exec(vm, module)?; — add a
minimal wrapper with the signature pub(crate) fn module_exec(vm:
&VirtualMachine, module: &Py<PyModule>) -> PyResult<()> that calls the
autogenerated __module_exec(vm, module) and returns Ok(()), so the call site
compiles on non-WASM platforms; reference the __module_exec symbol inside the
new fileio::module_exec.
♻️ Duplicate comments (7)
crates/vm/src/stdlib/ctypes.rs (1)
1281-1288: Propagate errors from__module_exec.Line 1287 discards the
PyResultreturn; errors will be silently ignored.🔧 Suggested fix
- __module_exec(vm, module); + __module_exec(vm, module)?;crates/stdlib/src/pyexpat.rs (1)
53-63: Registerpyexpat.model/pyexpat.errorsinsys.modulesfor qualified imports.crates/vm/src/stdlib/imp.rs (1)
143-149: Rollback sys.modules if exec fails.Line 143 inserts the module before exec; if exec errors, the partially initialized module remains cached. Clean it up on failure to avoid stale imports.
🛠️ Proposed fix
- if let Some(exec) = def.slots.exec { - exec(vm, &module)?; - } + if let Some(exec) = def.slots.exec { + if let Err(err) = exec(vm, &module) { + let _ = sys_modules.del_item(&*name, vm); + return Err(err); + } + }crates/derive-impl/src/pymodule.rs (1)
85-90: Guard against cfg‑gatedmodule_execto avoid build breaks.
Line 85–90 marks anymodule_execas present, butdef.slots.exec = Some(module_exec)is always emitted. If a user#[cfg]‑gatesmodule_exec, builds with the cfg disabled will fail due to an unresolved symbol. This was previously flagged; consider rejecting cfg‑gatedmodule_execor requiring a non‑gated fallback.🛠️ Proposed fix
- if let Item::Fn(func) = item - && func.sig.ident == "module_exec" - { - context.has_module_exec = true; - } + if let Item::Fn(func) = item + && func.sig.ident == "module_exec" + { + let has_cfg = func.attrs.iter().any(|a| a.path().is_ident("cfg")); + if has_cfg { + context.errors.push(syn::Error::new_spanned( + &func.sig.ident, + "`module_exec` must not be cfg-gated; move the cfg to the module or provide a non-gated fallback", + )); + } else { + context.has_module_exec = true; + } + }#!/bin/bash # Find cfg-gated module_exec definitions to validate impact. rg -n --type=rust '#\[cfg[^\]]*\]\s*\n\s*fn\s+module_exec' -C2crates/vm/src/stdlib/mod.rs (1)
96-102: Avoid duplicateposix::module_defregistration on WASI.
Both cfg blocks match on WASI, soposix::module_def(ctx)is added twice. Keep a single entry to prevent double registration.🛠 Suggested fix
- #[cfg(any(unix, target_os = "wasi"))] - posix::module_def(ctx), + #[cfg(unix)] + posix::module_def(ctx),crates/wasm/src/vm_class.rs (1)
29-34: Propagate__module_execerrors.
The return value is ignored, so init failures won’t surface.🔧 Suggested fix
- __module_exec(vm, module); + __module_exec(vm, module)?;crates/vm/src/stdlib/sys.rs (1)
168-174: Deduplicatesys.builtin_module_namesentries.
module_defsmay already containsys/builtins, so pushing them unconditionally can introduce duplicates in the exported tuple. Adddedup()after sorting to ensure uniqueness.🩹 Proposed fix
module_names.sort(); + module_names.dedup();
🧹 Nitpick comments (2)
crates/vm/src/stdlib/sysconfigdata.rs (1)
14-23: Optional: skip redundant sys.modules overwrite.At Line 21–23, consider checking if
sys.modulesalready contains the platform-specific key before setting it to avoid unnecessary mutation during reloads.crates/vm/src/vm/interpreter.rs (1)
130-146: Consider returningglobal_statefrom the VM rather than cloning.The comment on line 131 mentions "Don't clone here" but line 145 does clone
vm.state. Sincevm.stateis already aPyRc<PyGlobalState>, the clone is cheap (just reference count bump). However, the function could simply returnvm.state.clone()directly without the intermediate variable for clarity.♻️ Optional simplification
vm.initialize(); - // Clone global_state for Interpreter after all initialization is done - let global_state = vm.state.clone(); - (vm, global_state) + (vm, vm.state.clone()) }
| pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>) -> PyResult<()> { | ||
| // Call auto-generated initialization first | ||
| __module_exec(vm, module); | ||
|
|
||
| // Initialize FileIO types on non-WASM platforms | ||
| #[cfg(any(not(target_arch = "wasm32"), target_os = "wasi"))] | ||
| super::fileio::module_exec(vm, module)?; | ||
|
|
||
| let unsupported_operation = unsupported_operation().to_owned(); | ||
| extend_module!(vm, module, { | ||
| "UnsupportedOperation" => unsupported_operation, | ||
| "BlockingIOError" => vm.ctx.exceptions.blocking_io_error.to_owned(), | ||
| }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
# Search for module_exec in the io.rs file and also look for the fileio module definition
rg -n "module_exec" crates/vm/src/stdlib/io.rs | head -20Repository: RustPython/RustPython
Length of output: 255
🏁 Script executed:
# Also look for the fileio module and its structure
rg -n "mod fileio" crates/vm/src/stdlib/io.rsRepository: RustPython/RustPython
Length of output: 82
🏁 Script executed:
# Search for fileio module_exec more broadly
rg -n "fileio.*module_exec|fn module_exec" crates/vm/src/stdlib/Repository: RustPython/RustPython
Length of output: 1161
🏁 Script executed:
# Read the fileio module starting from line 4751
sed -n '4751,4850p' crates/vm/src/stdlib/io.rsRepository: RustPython/RustPython
Length of output: 3361
🏁 Script executed:
# Continue reading the fileio module - let's search for module_exec within fileio specifically
sed -n '4751,5100p' crates/vm/src/stdlib/io.rs | grep -n "module_exec\|^}"Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Also count the lines to understand module extent
wc -l crates/vm/src/stdlib/io.rsRepository: RustPython/RustPython
Length of output: 96
🏁 Script executed:
# Search for both module_exec and __module_exec in the fileio range
rg -n "__module_exec|fn module_exec" crates/vm/src/stdlib/io.rs | grep -A2 -B2 "4[7-9][0-9][0-9]\|5[0-2][0-9][0-9]"Repository: RustPython/RustPython
Length of output: 201
🏁 Script executed:
# Find where fileio module ends by looking for the closing brace
tail -n 100 crates/vm/src/stdlib/io.rs | head -50Repository: RustPython/RustPython
Length of output: 1750
🏁 Script executed:
# Search for the pattern "pub fn module_exec\|pub(crate) fn module_exec" specifically in fileio range
sed -n '4751,5346p' crates/vm/src/stdlib/io.rs | grep -n "pub.*fn module_exec"Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
# Double-check by looking at the entire fileio module definition for any module_exec
sed -n '4751,5346p' crates/vm/src/stdlib/io.rs | grep -i "module_exec"Repository: RustPython/RustPython
Length of output: 47
Add missing module_exec to the fileio module.
The call to super::fileio::module_exec(vm, module)?; at line 4738 will not compile because module_exec does not exist in the fileio module. Add the following function to fix the compilation error on non-WASM platforms:
Minimal wrapper
+ pub(crate) fn module_exec(
+ vm: &VirtualMachine,
+ module: &Py<crate::builtins::PyModule>,
+ ) -> PyResult<()> {
+ __module_exec(vm, module);
+ Ok(())
+ }🤖 Prompt for AI Agents
In `@crates/vm/src/stdlib/io.rs` around lines 4732 - 4744, The fileio module is
missing a pub(crate) module_exec function required by the call
super::fileio::module_exec(vm, module)?; — add a minimal wrapper with the
signature pub(crate) fn module_exec(vm: &VirtualMachine, module: &Py<PyModule>)
-> PyResult<()> that calls the autogenerated __module_exec(vm, module) and
returns Ok(()), so the call site compiles on non-WASM platforms; reference the
__module_exec symbol inside the new fileio::module_exec.
Summary by CodeRabbit
New Features
Refactor
Behavior
✏️ Tip: You can customize this high-level summary in your review settings.