Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
288 changes: 231 additions & 57 deletions src/interpreter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,49 @@ use rustpython_vm::{Interpreter, PyRef, Settings, VirtualMachine, builtins::PyMo

pub type InitHook = Box<dyn FnOnce(&mut VirtualMachine)>;

/// The convenient way to create [rustpython_vm::Interpreter] with stdlib and other stuffs.
/// The convenient way to create [rustpython_vm::Interpreter] with stdlib and other components.
///
/// Basic usage:
/// ```
/// let interpreter = rustpython::InterpreterConfig::new()
/// # Basic Usage
/// ```no_run
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there benefits to add no_run here?
no_run means the code can be outdated easy without doctest

Copy link
Contributor Author

@77AXEL 77AXEL Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original code used compile_fail, which indicates the example is expected to fail compilation. However the code is valid and should compile successfully it just can't run in a doc test environment so I changed it to no_run for clarity.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one test had compile_fail, and the others were possible to be run on doc test environment.

/// use rustpython::InterpreterConfig;
///
/// let interpreter = InterpreterConfig::new()
/// .init_stdlib()
/// .interpreter();
/// ```
///
/// To override [rustpython_vm::Settings]:
/// ```
/// # Override Settings
/// ```no_run
/// use rustpython_vm::Settings;
/// // Override your settings here.
/// use rustpython::InterpreterConfig;
///
/// let mut settings = Settings::default();
/// settings.debug = 1;
/// // You may want to add paths to `rustpython_vm::Settings::path_list` to allow import python libraries.
/// settings.path_list.push("Lib".to_owned()); // add standard library directory
/// settings.path_list.push("".to_owned()); // add current working directory
/// let interpreter = rustpython::InterpreterConfig::new()
/// // Add paths to allow importing Python libraries
/// settings.path_list.push("Lib".to_owned()); // standard library directory
/// settings.path_list.push("".to_owned()); // current working directory
///
/// let interpreter = InterpreterConfig::new()
/// .settings(settings)
/// .interpreter();
/// ```
///
/// To add native modules:
/// ```compile_fail
/// let interpreter = rustpython::InterpreterConfig::new()
/// # Add Native Modules
/// ```no_run
/// use rustpython::InterpreterConfig;
/// use rustpython_vm::{VirtualMachine, PyRef, builtins::PyModule};
///
/// fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> {
/// // Your module implementation
/// # todo!()
/// }
Comment on lines +37 to +40
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still misleading about making a module.

Suggested change
/// fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> {
/// // Your module implementation
/// # todo!()
/// }
/// #[pymodule]
/// mod your_module {}

///
/// let interpreter = InterpreterConfig::new()
/// .init_stdlib()
/// .init_hook(Box::new(|vm| {
/// vm.add_native_module(
/// "your_module_name".to_owned(),
/// Box::new(your_module::make_module),
/// );
/// }))
/// .add_native_module(
/// "your_module_name".to_owned(),
/// make_module, // ← use make_module
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// make_module, // ← use make_module
/// your_module::make_module

/// )
/// .interpreter();
/// ```
#[derive(Default)]
Expand All @@ -44,9 +54,15 @@ pub struct InterpreterConfig {
}

impl InterpreterConfig {
/// Create a new interpreter configuration with default settings
pub fn new() -> Self {
Self::default()
}

/// Build the interpreter with the current configuration
///
/// # Panics
/// May panic if initialization hooks encounter fatal errors
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, this is a caveat of vm init, but probably helpful

pub fn interpreter(self) -> Interpreter {
let settings = self.settings.unwrap_or_default();
Interpreter::with_init(settings, |vm| {
Expand All @@ -56,14 +72,37 @@ impl InterpreterConfig {
})
}

/// Set custom settings for the interpreter
///
/// If called multiple times, only the last settings will be used
pub fn settings(mut self, settings: Settings) -> Self {
self.settings = Some(settings);
self
}

/// Add a custom initialization hook
///
/// Hooks are executed in the order they are added during interpreter creation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Hooks are executed in the order they are added during interpreter creation
/// This function is the only path in the entire interpreter through which a mutable reference to the VirtualMachine can be obtained.
/// Hooks are executed in the order they are added during interpreter creation

pub fn init_hook(mut self, hook: InitHook) -> Self {
self.init_hooks.push(hook);
self
}

/// Add a native module to the interpreter
///
/// # Arguments
/// * `name` - The module name that will be used for imports
/// * `make_module` - Function that creates the module when called
///
/// # Example
/// ```no_run
/// # use rustpython::InterpreterConfig;
/// # use rustpython_vm::{VirtualMachine, PyRef, builtins::PyModule};
/// # fn make_module(vm: &VirtualMachine) -> PyRef<PyModule> { todo!() }
/// let interpreter = InterpreterConfig::new()
/// .add_native_module("mymodule".to_owned(), make_module)
/// .interpreter();
/// ```
pub fn add_native_module(
self,
name: String,
Expand All @@ -73,56 +112,191 @@ impl InterpreterConfig {
vm.add_native_module(name, Box::new(make_module))
}))
}

/// Initialize the Python standard library
///
/// This adds all standard library modules to the interpreter.
/// Requires the `stdlib` feature to be enabled at compile time.
#[cfg(feature = "stdlib")]
pub fn init_stdlib(self) -> Self {
self.init_hook(Box::new(init_stdlib))
}

/// Initialize the Python standard library (no-op without stdlib feature)
///
/// When the `stdlib` feature is not enabled, this method does nothing
/// and prints a warning. Enable the `stdlib` feature to use the standard library.
#[cfg(not(feature = "stdlib"))]
pub fn init_stdlib(self) -> Self {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this useful? Users get compile error when no feature set. But after changes, it became runtime error.

eprintln!(
"Warning: stdlib feature is not enabled. Standard library will not be available."
);
self
}

/// Convenience method to set the debug level
///
/// # Example
/// ```no_run
/// # use rustpython::InterpreterConfig;
/// let interpreter = InterpreterConfig::new()
/// .with_debug(1)
/// .interpreter();
/// ```
pub fn with_debug(mut self, level: u8) -> Self {
self.settings.get_or_insert_with(Default::default).debug = level;
self
}

/// Convenience method to add a single path to the module search paths
///
/// # Example
/// ```no_run
/// # use rustpython::InterpreterConfig;
/// let interpreter = InterpreterConfig::new()
/// .add_path("Lib")
/// .add_path(".")
/// .interpreter();
/// ```
pub fn add_path(mut self, path: impl Into<String>) -> Self {
self.settings
.get_or_insert_with(Default::default)
.path_list
.push(path.into());
self
}

/// Add multiple paths to the module search paths at once
///
/// # Example
/// ```no_run
/// # use rustpython::InterpreterConfig;
/// let interpreter = InterpreterConfig::new()
/// .add_paths(vec!["Lib", ".", "custom_modules"])
/// .interpreter();
/// ```
pub fn add_paths<I, S>(mut self, paths: I) -> Self
where
I: IntoIterator<Item = S>,
S: Into<String>,
{
let settings = self.settings.get_or_insert_with(Default::default);
settings.path_list.extend(paths.into_iter().map(Into::into));
self
}
}

/// Initialize the standard library modules
///
/// This function sets up both native modules and handles frozen/dynamic stdlib loading
#[cfg(feature = "stdlib")]
pub fn init_stdlib(vm: &mut VirtualMachine) {
vm.add_native_modules(rustpython_stdlib::get_module_inits());

// if we're on freeze-stdlib, the core stdlib modules will be included anyway
#[cfg(feature = "freeze-stdlib")]
{
vm.add_frozen(rustpython_pylib::FROZEN_STDLIB);
setup_frozen_stdlib(vm);

// FIXME: Remove this hack once sys._stdlib_dir is properly implemented or _frozen_importlib doesn't depend on it anymore.
assert!(vm.sys_module.get_attr("_stdlib_dir", vm).is_err());
vm.sys_module
.set_attr(
"_stdlib_dir",
vm.new_pyobj(rustpython_pylib::LIB_PATH.to_owned()),
vm,
)
.unwrap();
#[cfg(not(feature = "freeze-stdlib"))]
setup_dynamic_stdlib(vm);
}

/// Setup frozen standard library
///
/// Used when the stdlib is compiled into the binary
#[cfg(all(feature = "stdlib", feature = "freeze-stdlib"))]
fn setup_frozen_stdlib(vm: &mut VirtualMachine) {
vm.add_frozen(rustpython_pylib::FROZEN_STDLIB);

// FIXME: Remove this hack once sys._stdlib_dir is properly implemented
// or _frozen_importlib doesn't depend on it anymore.
// The assert ensures _stdlib_dir doesn't already exist before we set it
assert!(vm.sys_module.get_attr("_stdlib_dir", vm).is_err());
vm.sys_module
.set_attr(
"_stdlib_dir",
vm.new_pyobj(rustpython_pylib::LIB_PATH.to_owned()),
vm,
)
.unwrap();
}
Comment on lines +207 to +221
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

# Search for all _stdlib_dir references
rg -n "_stdlib_dir" --type=rust -C3

Repository: RustPython/RustPython

Length of output: 897


🏁 Script executed:

# Look for where setup_frozen_stdlib is called to understand initialization order
rg -n "setup_frozen_stdlib" --type=rust -B2 -A2

Repository: RustPython/RustPython

Length of output: 624


🏁 Script executed:

# Check the VirtualMachine initialization and sys_module setup
rg -n "sys_module" src/interpreter.rs -B1 -A1 | head -50

Repository: RustPython/RustPython

Length of output: 259


🏁 Script executed:

# Look at the full context around line 197 to see initialization order
sed -n '180,210p' src/interpreter.rs

Repository: RustPython/RustPython

Length of output: 1046


🏁 Script executed:

# Check if sys_module is created before or after setup_frozen_stdlib
rg -n "sys_module.*=" src/interpreter.rs | head -20

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Look for any module initialization that might touch _stdlib_dir
rg -n "def.*_stdlib_dir|_stdlib_dir.*=" --type=python

Repository: RustPython/RustPython

Length of output: 95


🏁 Script executed:

# Search for where sys module is initially created/initialized
rg -n "sys_module" src/interpreter.rs | head -30

Repository: RustPython/RustPython

Length of output: 155


🏁 Script executed:

# Look for VirtualMachine::new to understand initialization order
ast-grep --pattern 'impl VirtualMachine {
  $$$
  pub fn new($_) {
    $$$
  }
  $$$
}'

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Search for any code path that could set _stdlib_dir before setup_frozen_stdlib
rg -n "set_attr.*_stdlib_dir|_stdlib_dir" src/

Repository: RustPython/RustPython

Length of output: 401


🏁 Script executed:

# Look for VirtualMachine creation and sys_module initialization
rg -n "sys_module\s*=" src/interpreter.rs -B2 -A2

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Find where init_stdlib is called to understand the full initialization sequence
rg -n "init_stdlib" --type=rust -B3 -A3

Repository: RustPython/RustPython

Length of output: 5257


🏁 Script executed:

# Check if there's any Python-level code or module that might set _stdlib_dir
fd --type f -e py | xargs rg "_stdlib_dir" 2>/dev/null || echo "No Python files found with _stdlib_dir"

Repository: RustPython/RustPython

Length of output: 103


🏁 Script executed:

# Look for VirtualMachine::new and sys module initialization
rg -n "fn new\(" src/interpreter.rs crates/vm/src/vm/mod.rs -A 50 | grep -A 50 "impl.*VirtualMachine" | head -100

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Search in vm crate for sys_module creation
rg -n "sys_module\s*:" crates/vm/src/ -B2 -A2

Repository: RustPython/RustPython

Length of output: 720


🏁 Script executed:

# Look at the actual flow: when is sys module initialized relative to init_stdlib
grep -n "pub fn interpreter\|init_stdlib\|init_hook" src/interpreter.rs | head -20

Repository: RustPython/RustPython

Length of output: 628


🏁 Script executed:

# Look at the Interpreter creation flow to see when sys_module is created
rg -n "pub fn interpreter" src/interpreter.rs -A 20

Repository: RustPython/RustPython

Length of output: 814


🏁 Script executed:

# Find where VirtualMachine is constructed
rg -n "VirtualMachine\s*{" crates/vm/src/vm/mod.rs -B5 -A20 | head -80

Repository: RustPython/RustPython

Length of output: 164


🏁 Script executed:

# Look for sys module initialization in new()
rg -n "sys_module.*PyModule\|fn new\(" crates/vm/src/vm/mod.rs -A 30 | head -100

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Find Interpreter::with_init to see when sys_module is created
rg -n "with_init\|pub fn with_init" crates/vm/src/ --type=rust -A 30 | head -100

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Look for sys module initialization/creation
rg -n "PyModule::new\(.*sys\|sys.*PyModule" crates/vm/src/ --type=rust -B2 -A2

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Check if sys module is set up before or after hooks are called
rg -n "pub fn with_init" crates/vm/src/ -A 50

Repository: RustPython/RustPython

Length of output: 3824


🏁 Script executed:

# Look at VirtualMachine::new to see when sys_module is created
rg -n "impl VirtualMachine" crates/vm/src/vm/mod.rs -A 100 | grep -A 50 "pub fn new"

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Find the VirtualMachine::new implementation
sed -n '500,700p' crates/vm/src/vm/mod.rs | head -150

Repository: RustPython/RustPython

Length of output: 5630


🏁 Script executed:

# Look for the initialize method to see what it does
rg -n "pub fn initialize\(" crates/vm/src/vm/mod.rs -A 30

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Find VirtualMachine::new
rg -n "fn new\(" crates/vm/src/vm/mod.rs -A 40

Repository: RustPython/RustPython

Length of output: 1734


🏁 Script executed:

# Find the initialize method
rg -n "fn initialize\(" crates/vm/src/vm/mod.rs -A 50

Repository: RustPython/RustPython

Length of output: 2578


🏁 Script executed:

# Look at stdlib::sys::init_module to see if it sets _stdlib_dir
rg -n "pub fn init_module\|_stdlib_dir" crates/stdlib/src/sys.rs -B2 -A2

Repository: RustPython/RustPython

Length of output: 129


🏁 Script executed:

# Check what attributes stdlib::sys::init_module actually sets
rg -n "def __module_def\|init_module" crates/stdlib/src/sys.rs -A 80 | head -150

Repository: RustPython/RustPython

Length of output: 129


🏁 Script executed:

# Find where sys module is defined
fd "sys.rs" --type f

Repository: RustPython/RustPython

Length of output: 92


🏁 Script executed:

# Search for sys module definition
rg -n "stdlib.*sys\|pub fn init_module.*sys" --type=rust | head -20

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Look at initialization order more carefully
sed -n '280,320p' crates/vm/src/vm/mod.rs

Repository: RustPython/RustPython

Length of output: 1877


🏁 Script executed:

# Check stdlib::sys::init_module to see what it does
rg -n "pub fn init_module" crates/vm/src/stdlib/sys.rs -A 100

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Search for _stdlib_dir in sys.rs
rg -n "_stdlib_dir" crates/vm/src/stdlib/sys.rs

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Look at the full sys.rs file to understand initialization
head -100 crates/vm/src/stdlib/sys.rs

Repository: RustPython/RustPython

Length of output: 3318


🏁 Script executed:

# Check if init_module sets _stdlib_dir or any similar attribute
cat crates/vm/src/stdlib/sys.rs | head -200

Repository: RustPython/RustPython

Length of output: 6292


🏁 Script executed:

# Look at the full sys module to see if _stdlib_dir is anywhere
wc -l crates/vm/src/stdlib/sys.rs

Repository: RustPython/RustPython

Length of output: 97


🏁 Script executed:

# Search the entire file for _stdlib_dir
grep -n "_stdlib_dir" crates/vm/src/stdlib/sys.rs

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

# Get a sense of the file structure
tail -100 crates/vm/src/stdlib/sys.rs

Repository: RustPython/RustPython

Length of output: 2953


🏁 Script executed:

# Check if any other code sets _stdlib_dir before setup_frozen_stdlib
rg -n "set_attr.*_stdlib_dir|_stdlib_dir.*set" --type=rust

Repository: RustPython/RustPython

Length of output: 164


🏁 Script executed:

# Check the initialization sequence in detail - get the full setup_frozen_stdlib and init_stdlib flow
sed -n '190,225p' src/interpreter.rs

Repository: RustPython/RustPython

Length of output: 1265


🏁 Script executed:

# Verify the timing: check if any init_hook before setup_frozen_stdlib could set _stdlib_dir
rg -n "init_hook\|setup_frozen_stdlib\|init_stdlib" src/interpreter.rs | head -30

Repository: RustPython/RustPython

Length of output: 47


The assert! on _stdlib_dir is safe and protected by initialization order.

The sys module is created as an empty struct in VirtualMachine::new(), and setup_frozen_stdlib() runs as an init hook before stdlib::sys::init_module() is called in vm.initialize(). Since no code path sets _stdlib_dir before this function, the attribute will not exist at the time of the assertion.

However, the current assert! panics without context. Consider replacing it with a more informative check:

if vm.sys_module.get_attr("_stdlib_dir", vm).is_ok() {
    panic!("sys._stdlib_dir already exists; stdlib initialization called in unexpected order");
}

This provides better debugging information if the initialization invariant is ever violated.

🤖 Prompt for AI Agents
In src/interpreter.rs around lines 207 to 221, replace the bare assert! checking
that sys._stdlib_dir does not already exist with an explicit conditional that
checks vm.sys_module.get_attr("_stdlib_dir", vm).is_ok() and, if true, panic
with a clear message like "sys._stdlib_dir already exists; stdlib initialization
called in unexpected order" so the code fails with informative context rather
than a plain panic; keep the subsequent set_attr call and unwrap as-is.


/// Setup dynamic standard library loading from filesystem
///
/// Used when the stdlib is loaded from disk at runtime
#[cfg(all(feature = "stdlib", not(feature = "freeze-stdlib")))]
fn setup_dynamic_stdlib(vm: &mut VirtualMachine) {
use rustpython_vm::common::rc::PyRc;

let state = PyRc::get_mut(&mut vm.state).unwrap();

let additional_paths = collect_stdlib_paths();

// Insert at the beginning so stdlib comes before user paths
for path in additional_paths.into_iter().rev() {
state.config.paths.module_search_paths.insert(0, path);
}
}

#[cfg(not(feature = "freeze-stdlib"))]
{
use rustpython_vm::common::rc::PyRc;

let state = PyRc::get_mut(&mut vm.state).unwrap();

// Collect additional paths to add
let mut additional_paths = Vec::new();

// BUILDTIME_RUSTPYTHONPATH should be set when distributing
if let Some(paths) = option_env!("BUILDTIME_RUSTPYTHONPATH") {
additional_paths.extend(
crate::settings::split_paths(paths)
.map(|path| path.into_os_string().into_string().unwrap()),
)
} else {
#[cfg(feature = "rustpython-pylib")]
additional_paths.push(rustpython_pylib::LIB_PATH.to_owned())
}

// Add to both path_list (for compatibility) and module_search_paths (for sys.path)
// Insert at the beginning so stdlib comes before user paths
for path in additional_paths.into_iter().rev() {
state.config.paths.module_search_paths.insert(0, path);
}
/// Collect standard library paths from build-time configuration
///
/// Checks BUILDTIME_RUSTPYTHONPATH environment variable or uses default pylib path
#[cfg(all(feature = "stdlib", not(feature = "freeze-stdlib")))]
fn collect_stdlib_paths() -> Vec<String> {
let mut additional_paths = Vec::new();

// BUILDTIME_RUSTPYTHONPATH should be set when distributing
if let Some(paths) = option_env!("BUILDTIME_RUSTPYTHONPATH") {
additional_paths.extend(crate::settings::split_paths(paths).map(|path| {
path.into_os_string()
.into_string()
.unwrap_or_else(|_| panic!("BUILDTIME_RUSTPYTHONPATH isn't valid unicode"))
}))
} else {
#[cfg(feature = "rustpython-pylib")]
additional_paths.push(rustpython_pylib::LIB_PATH.to_owned())
}

additional_paths
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn test_default_config() {
let config = InterpreterConfig::new();
assert!(config.settings.is_none());
assert!(config.init_hooks.is_empty());
}
Comment on lines +266 to +271
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really appreciate the effort to improve RustPython.
If you used AI assistance here, could you please double-check that the changes match your intent? I’m not sure some changes are actually reviewed by you.

I’m not saying that this patch is without value or that it’s doing the wrong thing. I generally agree with the direction of this patch, but I would appreciate a version that has been reviewed a bit more carefully.


#[test]
fn test_with_debug() {
let config = InterpreterConfig::new().with_debug(2);
let settings = config.settings.unwrap();
assert_eq!(settings.debug, 2);
}

#[test]
fn test_add_single_path() {
let config = InterpreterConfig::new().add_path("test/path");
let settings = config.settings.unwrap();
assert_eq!(settings.path_list.len(), 1);
assert_eq!(settings.path_list[0], "test/path");
}

#[test]
fn test_add_multiple_paths_sequential() {
let config = InterpreterConfig::new().add_path("path1").add_path("path2");
let settings = config.settings.unwrap();
assert_eq!(settings.path_list.len(), 2);
}

#[test]
fn test_add_paths_batch() {
let paths = vec!["path1", "path2", "path3"];
let config = InterpreterConfig::new().add_paths(paths);
let settings = config.settings.unwrap();
assert_eq!(settings.path_list.len(), 3);
}
}