-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Await fsm checkpoint #6765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
yusufyian
wants to merge
47
commits into
RustPython:main
Choose a base branch
from
yusufyian:await_fsm_checkpoint
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Await fsm checkpoint #6765
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
e9a70c7
Update dependencies in Cargo.lock and add Cargo.lock to .gitignore
shawhanken 68da611
Merge branch 'main' of https://github.com/yusufyian/RustPython
shawhanken 1bf2bdf
Update dependencies in Cargo.lock and add Cargo.lock to .gitignore
shawhanken 1726b36
Add additional reference files to .gitignore
shawhanken fd57302
Update .gitignore to exclude additional files and improve dependency …
shawhanken ecfabb8
Merge branch 'RustPython:main' into main
yusufyian bc9b80a
Merge branch 'main' of https://github.com/yusufyian/RustPython
shawhanken 1a1c97a
Add checkpoint functionality to VirtualMachine
shawhanken 3997507
Add checkpoint request handling and update checkpoint functionality
shawhanken 551d025
Enhance demo script with additional print statements for debugging
shawhanken 665790f
Update demo user and enhance README with testing instructions
shawhanken c2edc6b
Update .gitignore to include demo files
shawhanken 4dc6120
Update .gitignore and demo script for type checking
shawhanken 9ac5e54
Rename RustPython binary to "pvm" in Cargo.toml and update demo.py fo…
yusufyian 8ea2822
Rename RustPython binary to 'pvm' in Cargo.toml and update demo.py fo…
yusufyian 66dc584
Update README and test script to reflect binary name change from 'rus…
yusufyian 6c26391
Refactor demo.py for improved clarity and structure in checkpointing …
yusufyian a1c1891
Update demo.py and README for financial trading scenario simulation
yusufyian f41b080
Enhance demo.py with additional trading scenario features and update …
yusufyian 5f547a5
Implement PVM host and runtime modules with initial configurations
yusufyian 3ed0799
Update various files for improved functionality and clarity
yusufyian 9704677
Enhance VM functionality and code clarity
yusufyian 965b201
Enhance demo script and update .gitignore
yusufyian 9f354c2
Update .gitignore to include all reference files and remove specific …
yusufyian b50f1f3
Remove obsolete reference files for PVM integration and continuation …
yusufyian f8fc889
Remove obsolete demo files for checkpoint/resume functionality
yusufyian 26ac488
Remove obsolete binary snapshot file for demo functionality
yusufyian b2ba6ea
Remove obsolete test script for checkpoint/resume functionality
yusufyian 2dfed2d
Refactor comments in state_store.py for clarity and consistency
yusufyian d8aabdd
Enhance checkpoint functionality and update .gitignore
yusufyian 6d54427
Update source location handling in compiler-source
yusufyian a1a7ec6
Refactor checkpoint and snapshot handling for improved functionality
yusufyian 3620c13
Update version formatting in version.rs to reflect PVM 0.0.2 integration
yusufyian fe31a3d
Add PVM versioning support and update build script
yusufyian 8eaf89f
Implement multi-frame checkpoint support and enhance stack management
yusufyian 5544761
Enhance checkpoint functionality and update demo scripts
yusufyian 13117d6
Add support for enumerate, zip, map, and filter in snapshot handling
yusufyian 63c44b5
Enhance checkpoint and block stack management in VM
yusufyian c6806af
Add support for ListIterator and RangeIterator in snapshot handling
yusufyian 9844d0d
Add comprehensive demo snapshot to .gitignore
yusufyian a07fed6
Remove Cargo.lock file to prevent unnecessary tracking of dependencies
yusufyian 06e9e6a
Add error handling and execution options in PVM runtime
yusufyian 072c0cc
Add determinism support and enhance error handling in PVM runtime
yusufyian 5e01466
Enhance PVM runtime with import tracing and version updates
yusufyian b9dc39d
Add new business scenario demos to README
yusufyian d5afdde
Fix formatting issue in escrow_marketplace_demo.py by adding a newlin…
yusufyian 9f3c462
Enhance PVM runtime with continuation support and new features
yusufyian File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Implement PVM host and runtime modules with initial configurations
- Added `pvm-host` crate for Host API definitions and error types. - Introduced `pvm-runtime` crate for PVM runtime encapsulation, including new features and modules. - Updated Cargo.lock to reflect the addition of `pvm-host` and `pvm-runtime` dependencies.
- Loading branch information
commit 5f547a577b7eb21adebe2c56d1e7c1a4ef6587c0
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| [package] | ||
| name = "pvm-host" | ||
| version.workspace = true | ||
| authors.workspace = true | ||
| edition.workspace = true | ||
| rust-version.workspace = true | ||
| repository.workspace = true | ||
| license.workspace = true | ||
|
|
||
| [dependencies] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| use core::fmt; | ||
|
|
||
| pub type Bytes = Vec<u8>; | ||
|
|
||
| #[derive(Clone, Debug)] | ||
| pub struct HostContext { | ||
| pub block_height: u64, | ||
| pub block_hash: [u8; 32], | ||
| pub tx_hash: [u8; 32], | ||
| pub sender: Bytes, | ||
| pub timestamp_ms: u64, | ||
| } | ||
|
|
||
| #[derive(Clone, Debug, Eq, PartialEq)] | ||
| pub enum HostError { | ||
| OutOfGas, | ||
| InvalidInput, | ||
| NotFound, | ||
| StorageError, | ||
| Forbidden, | ||
| Internal, | ||
| } | ||
|
|
||
| impl fmt::Display for HostError { | ||
| fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { | ||
| let msg = match self { | ||
| HostError::OutOfGas => "out of gas", | ||
| HostError::InvalidInput => "invalid input", | ||
| HostError::NotFound => "not found", | ||
| HostError::StorageError => "storage error", | ||
| HostError::Forbidden => "forbidden", | ||
| HostError::Internal => "internal error", | ||
| }; | ||
| f.write_str(msg) | ||
| } | ||
| } | ||
|
|
||
| impl std::error::Error for HostError {} | ||
|
|
||
| pub type HostResult<T> = Result<T, HostError>; | ||
|
|
||
| pub trait HostApi { | ||
| fn state_get(&self, key: &[u8]) -> HostResult<Option<Bytes>>; | ||
| fn state_set(&mut self, key: &[u8], value: &[u8]) -> HostResult<()>; | ||
| fn state_delete(&mut self, key: &[u8]) -> HostResult<()>; | ||
|
|
||
| fn emit_event(&mut self, topic: &str, data: &[u8]) -> HostResult<()>; | ||
|
|
||
| fn charge_gas(&mut self, amount: u64) -> HostResult<()>; | ||
| fn gas_left(&self) -> u64; | ||
|
|
||
| fn context(&self) -> HostContext; | ||
| fn randomness(&self, domain: &[u8]) -> HostResult<[u8; 32]>; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| [package] | ||
| name = "pvm-runtime" | ||
| version.workspace = true | ||
| authors.workspace = true | ||
| edition.workspace = true | ||
| rust-version.workspace = true | ||
| repository.workspace = true | ||
| license.workspace = true | ||
|
|
||
| [features] | ||
| default = ["stdlib"] | ||
| stdlib = ["rustpython/stdlib"] | ||
|
|
||
| [dependencies] | ||
| pvm-host = { path = "../pvm-host" } | ||
| rustpython = { path = "../.." } | ||
| rustpython-vm = { workspace = true } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| use pvm_host::HostApi; | ||
| use std::cell::Cell; | ||
| use std::marker::PhantomData; | ||
| use std::mem; | ||
|
|
||
| type HostPtr = *mut (dyn HostApi + 'static); | ||
|
|
||
| thread_local! { | ||
| static HOST: Cell<Option<HostPtr>> = Cell::new(None); | ||
| } | ||
|
|
||
| pub struct HostGuard<'a> { | ||
| _marker: PhantomData<&'a mut dyn HostApi>, | ||
| } | ||
|
|
||
| impl<'a> HostGuard<'a> { | ||
| pub fn install(host: &'a mut dyn HostApi) -> Self { | ||
| let ptr = host as *mut dyn HostApi; | ||
| // Erase the lifetime; the guard ensures the pointer is only used in-scope. | ||
| let ptr = unsafe { mem::transmute::<*mut dyn HostApi, HostPtr>(ptr) }; | ||
| HOST.with(|cell| cell.set(Some(ptr))); | ||
| Self { | ||
| _marker: PhantomData, | ||
| } | ||
| } | ||
| } | ||
|
|
||
| impl Drop for HostGuard<'_> { | ||
| fn drop(&mut self) { | ||
| HOST.with(|cell| cell.set(None)); | ||
| } | ||
| } | ||
|
|
||
| pub(crate) fn with_host<R>(f: impl FnOnce(&mut dyn HostApi) -> R) -> Option<R> { | ||
| HOST.with(|cell| { | ||
| let ptr = cell.get()?; | ||
| // Safety: host pointer is installed for the duration of an execution. | ||
| Some(unsafe { f(&mut *ptr) }) | ||
| }) | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| mod host; | ||
| mod module; | ||
|
|
||
| use pvm_host::{Bytes, HostApi, HostError}; | ||
| use rustpython::InterpreterConfig; | ||
| use rustpython_vm::{ | ||
| PyResult, Settings, VirtualMachine, | ||
| builtins::PyNone, | ||
| compiler::Mode, | ||
| scope::Scope, | ||
| }; | ||
|
|
||
| pub fn execute_tx(host: &mut dyn HostApi, code: &[u8], input: &[u8]) -> Result<Bytes, HostError> { | ||
| let source = std::str::from_utf8(code).map_err(|_| HostError::InvalidInput)?; | ||
| let mut settings = Settings::default(); | ||
| settings.argv = vec!["<pvm>".to_owned()]; | ||
|
|
||
| let _host_guard = host::HostGuard::install(host); | ||
|
|
||
| let mut config = InterpreterConfig::new().settings(settings); | ||
| #[cfg(feature = "stdlib")] | ||
| { | ||
| config = config.init_stdlib(); | ||
| } | ||
| config = config.add_native_module("pvm_host".to_owned(), module::make_module); | ||
| let interpreter = config.interpreter(); | ||
|
|
||
| let result = interpreter.enter(|vm| { | ||
| let res = run_source(vm, source, input); | ||
| if let Err(err) = &res { | ||
| vm.print_exception(err.clone()); | ||
| } | ||
| res | ||
| }); | ||
|
|
||
| match result { | ||
| Ok(bytes) => Ok(bytes), | ||
| Err(_) => Err(HostError::Internal), | ||
| } | ||
| } | ||
|
|
||
| fn run_source(vm: &VirtualMachine, source: &str, input: &[u8]) -> PyResult<Bytes> { | ||
| let scope = setup_main_module(vm)?; | ||
| scope | ||
| .globals | ||
| .set_item("__pvm_input__", vm.ctx.new_bytes(input.to_vec()).into(), vm)?; | ||
|
|
||
| let code_obj = vm | ||
| .compile(source, Mode::Exec, "<pvm>".to_owned()) | ||
| .map_err(|err| vm.new_syntax_error(&err, Some(source)))?; | ||
| vm.run_code_obj(code_obj, scope.clone())?; | ||
|
|
||
| extract_output(vm, &scope) | ||
| } | ||
|
|
||
| fn setup_main_module(vm: &VirtualMachine) -> PyResult<Scope> { | ||
| let scope = vm.new_scope_with_builtins(); | ||
| let main_module = vm.new_module("__main__", scope.globals.clone(), None); | ||
| main_module | ||
| .dict() | ||
| .set_item("__annotations__", vm.ctx.new_dict().into(), vm) | ||
| .expect("Failed to initialize __main__.__annotations__"); | ||
|
|
||
| vm.sys_module | ||
| .get_attr("modules", vm)? | ||
| .set_item("__main__", main_module.into(), vm)?; | ||
|
|
||
| Ok(scope) | ||
| } | ||
|
|
||
| fn extract_output(vm: &VirtualMachine, scope: &Scope) -> PyResult<Bytes> { | ||
| let output = scope | ||
| .globals | ||
| .get_item_opt("__pvm_output__", vm)?; | ||
|
|
||
| let Some(output) = output else { | ||
| return Ok(Vec::new()); | ||
| }; | ||
|
|
||
| if output.downcast_ref::<PyNone>().is_some() { | ||
| return Ok(Vec::new()); | ||
| } | ||
|
|
||
| output.try_bytes_like(vm, |bytes| bytes.to_vec()) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| pub(crate) use pvm_host_module::make_module; | ||
|
|
||
| #[rustpython_vm::pymodule] | ||
| mod pvm_host_module { | ||
| use crate::host; | ||
| use ::pvm_host::{HostApi, HostContext, HostError}; | ||
| use rustpython_vm::{ | ||
| PyObjectRef, PyResult, VirtualMachine, | ||
| builtins::{PyBaseExceptionRef, PyStrRef}, | ||
| function::ArgBytesLike, | ||
| }; | ||
|
|
||
| fn host_error(vm: &VirtualMachine, err: HostError) -> PyBaseExceptionRef { | ||
| vm.new_runtime_error(format!("pvm host error: {err}")) | ||
| } | ||
|
|
||
| fn with_host<R>( | ||
| vm: &VirtualMachine, | ||
| f: impl FnOnce(&mut dyn HostApi) -> Result<R, HostError>, | ||
| ) -> PyResult<R> { | ||
| let result = host::with_host(f) | ||
| .ok_or_else(|| vm.new_runtime_error("pvm host is not initialized".to_owned()))?; | ||
| result.map_err(|err| host_error(vm, err)) | ||
| } | ||
|
|
||
| #[pyfunction] | ||
| fn get_state(key: ArgBytesLike, vm: &VirtualMachine) -> PyResult<PyObjectRef> { | ||
| let value = with_host(vm, |host| key.with_ref(|bytes| host.state_get(bytes)))?; | ||
| Ok(match value { | ||
| Some(data) => vm.ctx.new_bytes(data).into(), | ||
| None => vm.ctx.none(), | ||
| }) | ||
| } | ||
|
|
||
| #[pyfunction] | ||
| fn set_state(key: ArgBytesLike, value: ArgBytesLike, vm: &VirtualMachine) -> PyResult<()> { | ||
| with_host(vm, |host| { | ||
| key.with_ref(|k| value.with_ref(|v| host.state_set(k, v))) | ||
| })?; | ||
| Ok(()) | ||
| } | ||
|
|
||
| #[pyfunction] | ||
| fn delete_state(key: ArgBytesLike, vm: &VirtualMachine) -> PyResult<()> { | ||
| with_host(vm, |host| key.with_ref(|bytes| host.state_delete(bytes)))?; | ||
| Ok(()) | ||
| } | ||
|
|
||
| #[pyfunction] | ||
| fn emit_event(topic: PyStrRef, data: ArgBytesLike, vm: &VirtualMachine) -> PyResult<()> { | ||
| with_host(vm, |host| data.with_ref(|bytes| host.emit_event(topic.as_str(), bytes)))?; | ||
| Ok(()) | ||
| } | ||
|
|
||
| #[pyfunction] | ||
| fn charge_gas(amount: u64, vm: &VirtualMachine) -> PyResult<()> { | ||
| with_host(vm, |host| host.charge_gas(amount))?; | ||
| Ok(()) | ||
| } | ||
|
|
||
| #[pyfunction] | ||
| fn gas_left(vm: &VirtualMachine) -> PyResult<u64> { | ||
| with_host(vm, |host| Ok(host.gas_left())) | ||
| } | ||
|
|
||
| #[pyfunction] | ||
| fn context(vm: &VirtualMachine) -> PyResult<PyObjectRef> { | ||
| let ctx = with_host(vm, |host| Ok(host.context()))?; | ||
| Ok(host_context_to_dict(vm, ctx)?.into()) | ||
| } | ||
|
|
||
| #[pyfunction] | ||
| fn randomness(domain: ArgBytesLike, vm: &VirtualMachine) -> PyResult<PyObjectRef> { | ||
| let bytes = with_host(vm, |host| domain.with_ref(|d| host.randomness(d)))?; | ||
| Ok(vm.ctx.new_bytes(bytes.to_vec()).into()) | ||
| } | ||
|
|
||
| fn host_context_to_dict( | ||
| vm: &VirtualMachine, | ||
| ctx: HostContext, | ||
| ) -> PyResult<rustpython_vm::builtins::PyDictRef> { | ||
| let dict = vm.ctx.new_dict(); | ||
| dict.set_item("block_height", vm.new_pyobj(ctx.block_height), vm)?; | ||
| dict.set_item( | ||
| "block_hash", | ||
| vm.ctx.new_bytes(ctx.block_hash.to_vec()).into(), | ||
| vm, | ||
| )?; | ||
| dict.set_item( | ||
| "tx_hash", | ||
| vm.ctx.new_bytes(ctx.tx_hash.to_vec()).into(), | ||
| vm, | ||
| )?; | ||
| dict.set_item("sender", vm.ctx.new_bytes(ctx.sender).into(), vm)?; | ||
| dict.set_item("timestamp_ms", vm.new_pyobj(ctx.timestamp_ms), vm)?; | ||
| Ok(dict) | ||
| } | ||
|
|
||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: RustPython/RustPython
Length of output: 186
🏁 Script executed:
Repository: RustPython/RustPython
Length of output: 1828
🏁 Script executed:
Repository: RustPython/RustPython
Length of output: 1176
🏁 Script executed:
Repository: RustPython/RustPython
Length of output: 435
🏁 Script executed:
Repository: RustPython/RustPython
Length of output: 820
🏁 Script executed:
Repository: RustPython/RustPython
Length of output: 865
🏁 Script executed:
Repository: RustPython/RustPython
Length of output: 47
Restore previous TLS values on
HostGuarddrop to handle nested installs.The current implementation unconditionally resets TLS cells to
NoneinDrop, which corrupts outer guard state ifinstallis called on the same thread while another guard is in scope. Although nesting is unlikely in current usage, the guard pattern is fundamentally broken for this scenario. Store and restore previous values.Restore previous TLS values
pub struct HostGuard<'a> { + prev_host: Option<HostPtr>, + prev_runtime_config: Option<RuntimeConfig>, _marker: PhantomData<&'a mut dyn HostApi>, } impl<'a> HostGuard<'a> { pub fn install(host: &'a mut dyn HostApi, runtime_config: RuntimeConfig) -> Self { let ptr = host as *mut dyn HostApi; // Erase the lifetime; the guard ensures the pointer is only used in-scope. let ptr = unsafe { mem::transmute::<*mut dyn HostApi, HostPtr>(ptr) }; + let prev_host = HOST.with(|cell| cell.get()); + let prev_runtime_config = RUNTIME_CONFIG.with(|cell| cell.get()); HOST.with(|cell| cell.set(Some(ptr))); RUNTIME_CONFIG.with(|cell| cell.set(Some(runtime_config))); Self { + prev_host, + prev_runtime_config, _marker: PhantomData, } } } impl Drop for HostGuard<'_> { fn drop(&mut self) { - HOST.with(|cell| cell.set(None)); - RUNTIME_CONFIG.with(|cell| cell.set(None)); + HOST.with(|cell| cell.set(self.prev_host)); + RUNTIME_CONFIG.with(|cell| cell.set(self.prev_runtime_config)); } }🤖 Prompt for AI Agents