Add untracked tests - (v3.13.11/v3.14.2) #6775

terryluan12 · 2026-01-18T09:21:17Z

So I know I was told to make all the downgrading changes in one big PR.
I have now also made all of the adding untracked tests in one big PR lmao.

(Please let me know if I should create individual PR's. I absolutely understand if I do, since this is massive)

This PR adds (most of) the rest of the untracked test libraries from version 3.13.11 that were left to be added.
I did not import the libraries themselves, as even I thought that was too much for one PR lol

I plan to update the libraries to v3.14.2 before merging, but I figure I will probably have to separate out at least some tests, so wanted to get out this PR before I do anything else

There are 6 commits, that I organized in a way where it should be easier to review. Specifically:

Updated all tests/test helper files... - just a pure "copy-paste" (I used the lib-updater tool, for the non-brand new tests) of the libraries.
Used a mixture of the bash script I coded up + manual annotation to add expectedFailures/skips
I used raise unittest.SkipTest or import_helper.import_module to skip test files which needed modules that were not implemented
For the doctests, I skipped the tests, and added a big chunk at the top clarifying the specific lines of codes and errors that arose (many are simple wording changes)
There were a couple trouble files where I commented out large chunks of code, because it wouldn't run otherwise; I just wanted to ask if there were any recommendations on better ways to fix the files in question (test_type_aliases.py + test_type_params.py.
By reformatting annotations, it should make using the lib_updater easier to use (so only major changes are noted)

A list of everything I added/updated (as well as some tests I had questions about)

New files:
* cjkencodings/
* crashers/
* leakers/
* regrtestdata/
* test_capi/
* test_gdb/
* test_interpreters/
* test_module/
* test_pydoc/
* test_sqlite3/
* test_tkinter/
* test_ttk/
* tkinterdata/
* tokenizedata/
* wheeldata/
* _test_embed_set_config.py
* _test_embed_structseq.py
* test_monitoring_shutdown.py
* .ruff.toml
* audit-tests.py
* bisect_cmd.py
* clinic.test.c
* cov.py
* curses_tests.py
* dis_module.py
* empty.vbs
* levenshtein_examples.json
* memory_watchdog.py
* multibytecodec_support.py
* profilee.py
* pstats.pck
* reperf.py
* ssltests.py
* test__interpchannels.py
* test__interpreters.py
* test__locale.py
* test_asdl_parser.py
* test_clinic.py
* test_coroutines.py
* test_cprofile.py
* test_curses.py
* test_dbm*.py
* test_embed.py
* test_extcall.py
* test_fileutils.py
* test_finalization.py
* test_flufl.py
* test_fork1.py
* test_frame.py
* test_frozen.py
* test_gc.py
* test_generator_stop.py
* test_genexps.py
* test_getpath.py
* test_idle.py
* test_imaplib.py
* test_launcher.py
* test_lltrace.py
* test_metaclass.py
* test_minidom.py
* test_modulefinder.py
* test_multibytecodec.py
* test_osx_env.py
* test_pdb.py
* test_peepholer.py
* test_pep646_syntax.py
* test_poplib.py
* test_profile.py
* test_pstats.py
* test_readline.py
* test_sax.py
* test_select.py
* test_source_encoding.py
* test_stable_abi_ctypes.py
* test_startfile.py
* test_strptime.py
* test_tcl.py
* test_threadsignals.py
* test_tracemalloc.py
* test_ttk_textonly.py
* test_turtle.py
* test_type_annotations.py
* test_type_cache.py
* test_unpack_ex.py
* test_unparse.py
* test_winconsoleio.py
* test_winsound.py
* test_zipimport_support.py

Modified files:
* test_dataclasses/
* test_importlib/
* test_zoneinfo/
* xmltestdata/
* test_except_star.py
* test_lzma.py
* test_struct.py
* test_winapi.py

Ignored files - I'm 90% sure I'm good to ignore it:
* test_cext/ - Test C extensions
* test_cppext/ - Test Cpp extensions
* test_peg_generator/ - Parser Generator tests
* test_xxlimited.py - CPython Internal tool
* test_xxtestfuzz.py - CPython Internal tool
* test_dict_version.py - Not importing per #5529

Ignored files - Bytecode/compilation specific files (please verify):
* test_compiler_assemble.py
* test_compiler_codegen.py

Ignored files - Ignored because I'm completely unsure:
* test_external_inspection.py
* test_generated_cases.py
* test_optimizer.py

These are tests which were added, but are not notated on #5529. Please verify if they should be added:
* test_free_threading/ - tests for multithreading
* test_apple.py - Apple tests
* test_codecencodings_.py - RustPython states that only utf-8 encoding is supported. Is this temporary?
* test_codecmaps_.py - RustPython states that only utf-8 encoding is supported. Is this temporary?
* test_monitoring.py
* test_msvcrt.py - Windows tests
* test_perfmaps.py
* test_perf_profiler.py
* test_termios.py - Termios is not mentioned in RustPython#5529
* test_type_aliases.py
* test_type_params.py
* test_wmi.py
* win_console_handler.py

coderabbitai · 2026-01-18T09:21:23Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

youknowone · 2026-01-18T09:38:56Z

Thanks. I will merge it as long as it pass CI. If not, splitting will be preferred. (and goes to 3.14, yeah)
I honestly don't believe this size patch can run without break.

We usually also don't bring test when we don't have feature. e.g. test_capi will test nothing right now.
That will become future burden to update them to new version without any benefit - because our library update is not an automated way yet.

youknowone · 2026-01-18T10:46:20Z

Lib/test/test_unparse.py

+# TODO: RUSTPYTHON
+try:
+    nonlocal_ex = """\
+def f():
+    x = 1
+    def g():
+        nonlocal x
+        x = 2
+        y = 7
+        def h():
+            nonlocal x, y
+"""
+    import ast
+    import unittest
+    ast1 = ast.parse(nonlocal_ex)
+    code2 = ast.unparse(ast1)
+except AttributeError:
+    raise unittest.SkipTest('TODO: RUSTPYTHON; type_comment attribute not implemented. FunctionDef, AsyncFunctionDef, For, AsyncFor, With and AsyncWith should have attribute')


Let's not add any tests with patched raise unittest.SkipTest
This is actually not being tested. And even when we implement it, this is not automatically detected,

We discussed a few times about skip, we don't like it. Please do not add skip without significant benefits.

Hey there, yep! Since being told, I've been very careful to add skips as little as possible (and also in cases like this where there _were_significant skips I separated them from the normal annotations for easier review).

In the cases in commit 3, I thought there would be a lot of benefit since out of 189 tests, 155 were failing for the same reason.

Sorry about the trouble, I will remove the annotations, and add expectedFailures.

Also, I may be mistaken, but I do think that it is being tested?

I moved out the logic from check_ast_roundtrip but the try statement is from the test_nonlocal test.

When the type_comment attribute is added, the exception should not be thrown anymore and the tests should be run as normal

youknowone · 2026-01-18T10:47:05Z

And we have to split the patch. json already got conflict. Other libraries will be too

New files: * cjkencodings/ * crashers/ * leakers/ * regrtestdata/ * test_gdb/ * test_interpreters/ * test_module/ * test_tkinter/ * test_ttk/ * tkinterdata/ * tokenizedata/ * wheeldata/ * audit-tests.py * bisect_cmd.py * clinic.test.c * cov.py * curses_tests.py * dis_module.py * empty.vbs * levenshtein_examples.json * memory_watchdog.py * multibytecodec_support.py * profilee.py * pstats.pck * reperf.py * .ruff.toml * ssltests.py * test_apple.py * test_asdl_parser.py * test_clinic.py * test_codecencodings_cn.py * test_codecencodings_hk.py * test_codecencodings_iso2022.py * test_codecencodings_jp.py * test_codecencodings_kr.py * test_codecencodings_tw.py * test_codecmaps_cn.py * test_codecmaps_hk.py * test_codecmaps_jp.py * test_codecmaps_kr.py * test_codecmaps_tw.py * test_coroutines.py * test_curses.py * test_dataclasses * test_dbm_gnu.py * test_dbm_ndbm.py * test_embed.py * _test_embed_set_config.py * _test_embed_structseq.py * test_except_star.py * test_fileutils.py * test_flufl.py * test_fork1.py * test_frame.py * test_frozen.py * test_generator_stop.py * test_idle.py * test__interpchannels.py * test_interpreters * test__interpreters.py * test_lltrace.py * test__locale.py * _test_monitoring_shutdown.py * test_osx_env.py * test_pep646_syntax.py * test_perfmaps.py * test_perf_profiler.py * test_readline.py * test_sax.py * test_select.py * test_source_encoding.py * test_stable_abi_ctypes.py * test_startfile.py * test_strptime.py * test_struct.py * test_tcl.py * test_termios.py * test_threadsignals.py * test_ttk_textonly.py * test_turtle.py * test_type_annotations.py * test_type_cache.py * test_unparse.py * test_winapi.py * test_winsound.py * test_zipimport_support.py * win_console_handler.py Modified files: * test_zoneinfo/ * test_except_star.py * test_struct.py * test_winapi.py

terryluan12 · 2026-01-19T02:20:24Z

We usually also don't bring test when we don't have feature. e.g. test_capi will test nothing right now. That will become future burden to update them to new version without any benefit - because our library update is not an automated way yet.

Re: this, should I deal with test_free_threading/test_monitoring.py which relies on the monitoring library? (On a side note, is the monitoring library being imported? It is not listed at #5529

youknowone · 2026-01-19T04:18:26Z

If you mean sys.monitoring, we don't have one yet. If that's missing in the issue, that's mistake. It is expected to be listed under tests without python libraries.
But yep, we gain nothing much by adding it before implementing sys.monitoring.

youknowone · 2026-01-19T04:46:37Z

I suggest to split the patches in different way.
Because this PR is too huge, i cannot reivew it easy.
Please split PRs entirely not including newly added skip. They will be comparably easier to be merged.
(Also grouping a few files to create multiple PRs will be a good idea)

For other patches with skips, please add the rationale for those skips for each library. From last contributions, I feel you definitely have lower bar to marking skip than us. We will discuss what kind of skip is acceptable while the others are not on these series of patches.

youknowone · 2026-01-19T07:42:06Z

* test_free_threading/ : we will eventually need it. RustPython is regarded as free threading. not sure actually comapatible right now.
* test_apple.py - Apple tests; sounds good
* test_codecencodings_*.py - RustPython states that only utf-8 encoding is supported. Is this temporary?; not sure. but no plan yet.
* test_codecmaps_*.py - RustPython states that only utf-8 encoding is supported. Is this temporary?; not sure. but no plan yet.
* test_monitoring.py; we don't have sys.monitoring
* test_msvcrt.py - Windows tests; we now have well-featured msvcrt! this test will be useful
* test_perfmaps.py; need to check result
* test_perf_profiler.py;  need to check result
* test_termios.py - Termios is not mentioned in RustPython#5529 ; useful
* test_type_aliases.py; useful
* test_type_params.py; useful
* test_wmi.py; not sure what it is. if windows require this feature, it need to be added
* win_console_handler.py; useful

terryluan12 · 2026-01-19T19:34:34Z

I suggest to split the patches in different way. Because this PR is too huge, i cannot reivew it easy. Please split PRs entirely not including newly added skip. They will be comparably easier to be merged. (Also grouping a few files to create multiple PRs will be a good idea)

For other patches with skips, please add the rationale for those skips for each library. From last contributions, I feel you definitely have lower bar to marking skip than us. We will discuss what kind of skip is acceptable while the others are not on these series of patches.

Yep, sounds good! And sure, but also just to be fair, I did understand that there were differences in the skips for files like test_unparse.py and a normal skip, which is also why I separated out the changes of the files into a separate commit, and pointed it out in the PR summary.

youknowone · 2026-01-20T01:34:50Z

About test_unparse, you made the last commit d3940a7
This is a good direction. After this commit, once we fix any bug from unparse, test will start fail by unexpected success. This is the way how we detect regression in both positive for negative way. Before the commit, it is just placed in our codebase, but doing nothing except for being skipped. That doesn't add any value to our testing system. In general, more code means more maintenance cost.

also tracking untracked files considered in #6775

terryluan12 marked this pull request as draft January 18, 2026 09:22

terryluan12 force-pushed the update_remaining_tests_semiautomated branch from 7e74525 to fcac1c4 Compare January 18, 2026 09:45

youknowone requested changes Jan 18, 2026

View reviewed changes

terryluan12 force-pushed the update_remaining_tests_semiautomated branch from fcac1c4 to 0e58f9e Compare January 19, 2026 02:13

terryluan12 added 3 commits January 18, 2026 21:16

Annotated/removing failing tests with expectedFailures/skips

492f031

Added skips for tests missing vital modules/attributes

4e9fb55

Fixed test_unparse

d3940a7

terryluan12 force-pushed the update_remaining_tests_semiautomated branch from 0e58f9e to d3940a7 Compare January 19, 2026 02:16

terryluan12 closed this Jan 19, 2026

youknowone reopened this Jan 20, 2026

youknowone mentioned this pull request Jan 24, 2026

update_lib todo also shows test todo #6859

Merged

youknowone added a commit that referenced this pull request Jan 24, 2026

update_lib todo also shows test todo (#6859)

691d281

also tracking untracked files considered in #6775

Add untracked tests - (v3.13.11/v3.14.2) #6775

Are you sure you want to change the base?

Add untracked tests - (v3.13.11/v3.14.2) #6775

Uh oh!

Conversation

terryluan12 commented Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai bot commented Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

youknowone commented Jan 18, 2026

Uh oh!

youknowone Jan 18, 2026

Choose a reason for hiding this comment

Uh oh!

terryluan12 Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terryluan12 Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

youknowone commented Jan 18, 2026

Uh oh!

terryluan12 commented Jan 19, 2026

Uh oh!

youknowone commented Jan 19, 2026

Uh oh!

youknowone commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

youknowone commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terryluan12 commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

youknowone commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

terryluan12 commented Jan 18, 2026 •

edited

Loading

coderabbitai bot commented Jan 18, 2026 •

edited

Loading

terryluan12 Jan 19, 2026 •

edited

Loading

youknowone commented Jan 19, 2026 •

edited

Loading

youknowone commented Jan 19, 2026 •

edited

Loading

terryluan12 commented Jan 19, 2026 •

edited

Loading

youknowone commented Jan 20, 2026 •

edited

Loading