Remaster platform support with a common corelib ruby implementation and small platform drivers and more ... by janbiedermann · Pull Request #2746 · opal/opal

janbiedermann · 2025-04-02T08:03:20Z

Goals of this PR

Have a perfectly working corelib on all platforms*
Improve maintainability of platforms and across platforms
Improve situation for self hosted Opal

Goodies

Many bugs fixed along the way
Functionality improved
Overall 18084 specs working on node/linux with bundle exec rake mspec_ruby_node
Perfect #format*
Perfect #fnmatch*
Perfect #glob*
Perfect #expand_path and #absolute_path*
Perfect #dirname*
fileutils
tmpdir
complete Pathname
FileTest
Kernel#system, Kernel#backtick

Implementation

Before Opal had custom implementations of everything platform dependent for each platform, which required a lot of code and left most platforms badly and partially supported. This PR improves the situation by providing a common corelib implementation of methods, that depend on platform dependent code, that works across all platforms and requires therefore only small platform drivers to implement the final system calls or provide a emulation or raise a error, depending on platform capabilities.

Notes for Review

There are still a few todos and see platform footnotes below.

Platform Status

~~strike~~ means unable to run specs reliably or at all

Servers

driver	support	platform	version	os	mspec_ruby	mspec_opal	minitest	notes
x	x	node	22.16	lin	18084	673	2627
x	x	node	22.16	win	17416	673	1414	*1
x	x	node	22.16	mac	17696	673	3949	*1
-	-	deno	2.3.3	lin/win/mac	-	-	-	#2748
-	-	bun	1.2.13	lin/win/mac	-	-	-	#2706
x	-	graalnodejs	24.2.0	lin	-	-	-	*4

Browsers

driver	support	platform	version	os	mspec_ruby	mspec_opal	minitest
x	x	chrome	136	lin	14537	547	772, 1e
x	x	chrome	136	win	14536	547	772, 1e
x	x	chrome	136	mac	14536	547	772, 1e
x	x	edge	136	win	14536	547	772, 1e
x	x	firefox	138	lin	14534	547	769, 1e
x	x	firefox	138	win	14533	547	769, 1e
x	x	firefox	138	mac	14533	547	769, 1e
x	x	safari	18.5	mac	14531	547	769, 1e

Others

driver	support	platform	version	os	mspec_ruby	mspec_opal	notes
-	-	gjs	1.82.1	lin	0	0	*5
-	-	cjs	128.0	lin	0	0	*5
-	x	quickjs	2025-04-26	lin	14514	~~545, 2f~~	*8
-	-	quickjs-ng	0.9.0	lin	0	0
-	-	miniracer	0.18.1	lin	0	0	*6
-	-	osascript	15.4	mac	0	0	*7

Still to do

(no specific order)

Desired Extractions

String/Encoding
Regexp transpiler

*as much as possible
*1 numbers not updated
*4 graalnodejs is 10 times too slow, not running tests, but in terms of correctness its very, very close to nodejs results
*5 gjs and cjs suffer from a totally unstable API in terms of IO, not practical for any real use, specific runner and driver removed
*6 mini_racer just exits, best i can get is: in 'MiniRacer::Context#eval': unsupported type b (MiniRacer::InternalError)
*7 no specific driver for osascript anymore, works mostly but specs don't finish, mspec_opal and minitest work but statstics not visible, runner removed
*8 quickjs, as a platform with a runner but no driver, serves as example for a 'unknown' platform

spec/filters/platform/safari/literal_regexp.rb

opal/corelib/constants.rb

hmdne · 2025-04-04T08:41:37Z

opal/corelib/kernel.rb

  end

+  def __dir__
+    File.realpath(File.dirname(__FILE__))


__FILE__ here is opal/corelib/kernel.rb, so __dir__ is always opal/corelib

This dir FILE thing probably needs some further discussion/work. I add it to the TODO list above.

hmdne · 2025-04-04T08:42:15Z

opal/corelib/kernel.rb

    self
  end

+  def fork


opal/corelib/variables.rb

hmdne · 2025-04-04T09:05:26Z

We've read with @elia the patch until the platform code itself. In general, we agreed this kind of abstraction is a great idea, but some parts can be ideally split into smaller patches.

janbiedermann · 2025-04-08T23:47:34Z

This adds a tiny Virtual File System for browsers, which is required so that specs can run without exiting in between. The spec suite as run by node, deno, bun, etc. executes specs that during setup require IO. When these specs run without IO the whole suite exits with "example has not been reset". As we don't have a conditional spec file exclusion and as the drivers make it easily possible, i added a VFS, used by browsers, available via 'opal/full'. Its very simple, create, write, read, delete File or Dir works, even File.stat. Most IO/File/Dir specs still fail at this point, but specs overall run.

janbiedermann · 2025-04-18T05:55:44Z

@hmdne @elia Thanks for reviewing part 1, now part 2 is ready for review, which includes all the drivers. I updated the topmost comment above accordingly.

janbiedermann · 2025-04-18T09:50:03Z

rebase

janbiedermann · 2025-04-18T20:04:47Z

@hmdne @elia Part 3 is ready, fixes and adapts mspec_opal_*

janbiedermann · 2025-04-23T18:36:49Z

fileutils, tmpdir, pathname may seem out of scope of this PR, as it says "fully functional corelib", but corelib specs are not complete and implementing these helped fix some bugs and tests for some ambiguities in corelib. Also fileutils, tmpdir and pathname are very, very useful :-)

janbiedermann · 2025-04-25T07:46:35Z

@hmdne @elia Part 4 is ready

opal/corelib/env.rb

opal/corelib/dir.rb

janbiedermann · 2025-04-27T17:31:13Z

9f3ff8f changes the driver interface a bit, which only works with node and breaks all other platforms. Working on it ...

janbiedermann · 2025-04-27T17:33:03Z

Ooops, the process.rb part is still missing, working on it too ..., so specs may not run at all at this stage

janbiedermann · 2025-04-28T01:35:21Z

@hmdne @elia Part 5 is ready, it runs specs for node only, as i changed driver interface a bit and need to update other drivers, and specs for node do not even run clean, 4 issues in spec setups. Part 5 contains important changes:

new exit code
backtick default to false with a working Kernel#backtick
removal of MSpec platform rewriters

and fixes from @takaram

This contains kinda working process spawning methods, but they are still in progress, specifically the args checking needs to be common, coming next.

This should have been the "minispecs working part", but well, thats maybe for part 6 or 7.

hmdne · 2025-04-28T08:19:11Z

stdlib/stringio.rb

+    c
+  end
+
+  def gets(sep = $/, limit = nil, chomp: false)


Ruby StringIO has IO::generic_readable and IO::generic_writable as ancestors - which seem like to be an internal thing.

Still we may want to simulate it somewhat, so that certain functions are not duplicated.

hmdne · 2025-04-28T08:29:57Z

opal/corelib/string.rb

@@ -8,7 +8,18 @@
 class ::String < `String`


Can the changes on String/Encoding be separated to another PR?

I added another section to the top comment here "Desired Extractions", please add everything you want in a separate PR there or mention in a comment and i will add and i will then attend to them once everything is working.

janbiedermann · 2025-09-24T11:17:45Z

Some rspec actions where apperaring green although they should have been red, so i missed them. As they now appear red i removed the "ready to merge" label and handle the failing spec.

janbiedermann · 2025-09-24T12:38:34Z

Alright, DONE

Currently, some codes output warnings like the following. ``` warning: Backtick operator usage interpreted as intent to embed JavaScript; this code will break in Opal 2.0; add a magic comment: `# backtick_javascript: true` -- ./try_ruby.rb: ``` https://github.com/ruby/TryRuby/actions/runs/17408494030/job/49419277575#step:4:8 In Opal 2, `backtick_javascript` defaults to false, because `Kernel#backtick` is implemented. Files using inline JavaScript must add the `# backtick_javascript: true` magic comment. Ref: opal/opal#2746

janbiedermann · 2025-11-29T09:42:37Z

rebase to master

…nd small platform drivers; Make corelib work on all platforms as perfect as possible; Implement #format, #fnmatch, #glob, #expand_path, #absolute_path, #dirname; Implement platform driver for node; New #exit code based on SystemExit; Implement IO.copy_stream, Kernel.singleton_methods, FileUitls, Pathname, tmpdir, FileTest, Etc, ARGF, ARGV, ENV; Kernel and Process process spawning methods; Set backtick default to false; Always raise NotImplementedError for missing features; Document String and Encoding, Srting improvements

…ger supported.

janbiedermann · 2026-02-11T15:20:11Z

Rebased.

Somehow something in Node 25 creates a different byte stream for some encodings for #syswrite or #write, also affecting some other tests. I simply filtered the specs/tests. Those specs/tests work in Node 22/24 without problem. Don't know what is causing trouble in Node 25.

rspec-windows with ruby 4.0 as default exits with 1, even though specs pass.

janbiedermann requested review from elia, hmdne and takaram April 2, 2025 08:03

This was referenced Apr 2, 2025

Feature: Replace Glob by picomatch in stdlib/nodejs/dir #2666

Closed

resolves #2666 replace glob by picomatch #2667

Merged

hmdne reviewed Apr 4, 2025

View reviewed changes

spec/filters/platform/safari/literal_regexp.rb Outdated Show resolved Hide resolved

hmdne reviewed Apr 4, 2025

View reviewed changes

opal/corelib/constants.rb Show resolved Hide resolved

hmdne reviewed Apr 4, 2025

View reviewed changes

opal/corelib/kernel.rb Outdated

self

end

def fork

Copy link

Member

hmdne Apr 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice :D

hmdne reviewed Apr 4, 2025

View reviewed changes

opal/corelib/variables.rb Outdated Show resolved Hide resolved

janbiedermann mentioned this pull request Apr 5, 2025

Issues on Bun #2706

Open

janbiedermann force-pushed the raise_platform_foundation branch 2 times, most recently from c67f6aa to c5232b2 Compare April 18, 2025 05:49

janbiedermann force-pushed the raise_platform_foundation branch from c5232b2 to 3dcf042 Compare April 18, 2025 09:48

janbiedermann force-pushed the raise_platform_foundation branch from affc39c to a3faccc Compare April 18, 2025 20:03

janbiedermann force-pushed the raise_platform_foundation branch from 1a8254a to 7faa70e Compare April 25, 2025 07:45

takaram reviewed Apr 27, 2025

View reviewed changes

opal/corelib/env.rb Outdated Show resolved Hide resolved

opal/corelib/dir.rb Outdated Show resolved Hide resolved

janbiedermann force-pushed the raise_platform_foundation branch from 51cd7ff to d1ffb43 Compare April 28, 2025 01:26

hmdne reviewed Apr 28, 2025

View reviewed changes

janbiedermann removed the ready to merge label Sep 24, 2025

janbiedermann force-pushed the raise_platform_foundation branch 2 times, most recently from 1192fc1 to 3f0d06e Compare September 24, 2025 12:27

janbiedermann added the ready to merge label Sep 24, 2025

y-yagi mentioned this pull request Oct 2, 2025

Fix warnings related with Opal 2 ruby/TryRuby#215

Merged

janbiedermann mentioned this pull request Oct 8, 2025

Actions: Specify runner engine versions, least supported and latest, run on Edge #2705

Closed

janbiedermann force-pushed the raise_platform_foundation branch from d69e408 to c142f38 Compare October 19, 2025 19:35

janbiedermann force-pushed the raise_platform_foundation branch from a39153d to 5fb706e Compare November 29, 2025 09:38

janbiedermann added 8 commits February 11, 2026 12:05

Disable TruffleRuby cause 25.0 makes trouble again

d5b03fa

Fix gvars unavailable

0c93cf0

No need for default args

b6a3552

Remove require opal/base, its already required by opal/mini

58f22e3

Fix Node 25.0.0 Exception: The property 'options.recursive' is no lon…

31f01a9

…ger supported.

Exlude runtime/*.rb from forbidden prefixes (Missed that in rebase)

5024e59

Further String optimization, allow for "string ropes"

c8f0eac

janbiedermann force-pushed the raise_platform_foundation branch from 26d9b99 to c8f0eac Compare February 11, 2026 11:05

janbiedermann added 8 commits February 11, 2026 12:12

RuboCop fixes and use ruby 3.4 for actions

44cbab5

Adjust spec filters

bcc0dc4

Update spec filters, use ruby 4.0 for actions

7205f94

Node 25 somehow things are different, update spec filters

6885a64

Add ostruct as gem, does no longer ship with ruby 4

27e3046

More spec and test filters for node 25

ca05f21

More node 25 filters

adf6bcf

Adjust node 25 filters

f3edd54

Uh oh!

Conversation

janbiedermann commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goals of this PR

Goodies

Implementation

Notes for Review

Platform Status

Servers

Browsers

Others

Still to do

Desired Extractions

Uh oh!

Uh oh!

Uh oh!

hmdne Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janbiedermann Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

hmdne Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hmdne commented Apr 4, 2025

Uh oh!

janbiedermann commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janbiedermann commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janbiedermann commented Apr 18, 2025

Uh oh!

janbiedermann commented Apr 18, 2025

Uh oh!

janbiedermann commented Apr 23, 2025

Uh oh!

janbiedermann commented Apr 25, 2025

Uh oh!

Uh oh!

Uh oh!

janbiedermann commented Apr 27, 2025

Uh oh!

janbiedermann commented Apr 27, 2025

Uh oh!

janbiedermann commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hmdne Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

hmdne Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

janbiedermann May 2, 2025

Choose a reason for hiding this comment

Uh oh!

janbiedermann commented Sep 24, 2025

Uh oh!

janbiedermann commented Sep 24, 2025

Uh oh!

janbiedermann commented Nov 29, 2025

Uh oh!

janbiedermann commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

janbiedermann commented Apr 2, 2025 •

edited

Loading

hmdne Apr 4, 2025 •

edited

Loading

janbiedermann commented Apr 8, 2025 •

edited

Loading

janbiedermann commented Apr 18, 2025 •

edited

Loading

janbiedermann commented Apr 28, 2025 •

edited

Loading