Skip to content

Conversation

@youknowone
Copy link
Member

@youknowone youknowone commented Jan 19, 2026

cc @moreal

I am going to rewrite this tools more organized library update toolkit, and remain rooms for further automation.

scripts/update_lib will be main entrance.

lib_updater
-> scripts/update_lib patches (--from --to)
-> scripts/update_lib migrate (--quick-upgrade)

auto_mark_test
-> scripts/update_lib auto_mark

Also scripts/update_lib quick is added to upgrade a whole package at once, copy the library, remove unexected sucesss new add expectedFailure for newly added failing tests but not about regression.

The format also will be aligned to ruff format and PEP8 to fix #6701.

Keeping draft to test for a few more libraries. Please feedback.

Summary by CodeRabbit

  • Documentation

    • Updated library upgrade instructions to reflect new update_lib workflow with improved step-by-step guidance
    • Added troubleshooting section with issue reporting guidelines
  • Refactor

    • Deprecated legacy auto_mark_test.py tool
    • Introduced new update_lib utility offering enhanced test marker preservation and improved failure detection workflow

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jan 19, 2026

📝 Walkthrough

Walkthrough

The PR replaces the monolithic auto_mark_test.py script and legacy lib_updater workflow with a new modular update_lib package featuring specialized subcommands for copying libraries, migrating tests, auto-marking failures, and orchestrating updates. Documentation is updated to reflect the new tooling.

Changes

Cohort / File(s) Summary
Documentation Update
\.claude/commands/upgrade-pylib\.md
Updated instructions to reference update_lib instead of legacy lib_updater workflow; added "Report Tool Issues First" section; revised Steps 3–5 to describe new patching, auto-marking, and marker restoration capabilities.
Legacy Script Removal
scripts/auto_mark_test\.py
Deleted entire 411-line automated test-marking script that annotated failures with @unittest.expectedFailure; functionality merged into new scripts/update_lib/auto_mark\.py.
Core Library Modules
scripts/update_lib/__init__\.py, scripts/update_lib/patch_spec\.py
New package initializer re-exporting public API; refactored patch_spec from CLI tool to library with new functions extract_patches, patches_to_json, patches_from_json; renamed internal helpers to private (underscore-prefixed).
CLI Entry Points
scripts/update_lib/__main__\.py, scripts/update_lib/auto_mark\.py, scripts/update_lib/copy_lib\.py, scripts/update_lib/migrate\.py, scripts/update_lib/patches\.py, scripts/update_lib/quick\.py
New CLI interface dispatching to subcommands; auto_mark.py provides AST-driven test result parsing, inheritance-aware method resolution, and patch generation; copy_lib, migrate, and patches handle file operations and patching; quick.py orchestrates end-to-end workflows with optional git integration.
Path Utilities
scripts/update_lib/path\.py
New module providing path transformations (lib-to-test mapping, path parsing) and test file enumeration; supports flexible path handling for both library and test directories.
Comprehensive Test Coverage
scripts/update_lib/tests/test_auto_mark\.py, test_migrate\.py, test_patch_spec\.py, test_path\.py, test_quick\.py
New test suites covering all modules with 1,500+ lines; validates parsing, AST analysis, patch application, path resolution, and integration workflows; includes edge cases and regression-safety filters.

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant CLI as update_lib<br/>(quick)
    participant Migrate as migrate.py
    participant AutoMark as auto_mark.py
    participant TestRunner as run_test()
    participant FileSystem as File System

    User->>CLI: quick [path] --no-migrate=False
    CLI->>Migrate: patch_file/patch_directory
    Migrate->>FileSystem: Extract patches from Lib/test/<file>.py
    Migrate->>FileSystem: Read cpython/Lib/test/<file>.py
    Migrate->>FileSystem: Apply patches & write to Lib/test/<file>.py
    Migrate-->>CLI: Patching complete

    CLI->>AutoMark: auto_mark_file/auto_mark_directory
    AutoMark->>TestRunner: run_test(test_name)
    TestRunner->>FileSystem: Execute RustPython tests
    TestRunner-->>AutoMark: TestResult{failures, unexpected_successes}
    
    AutoMark->>AutoMark: Parse AST to extract test methods
    AutoMark->>AutoMark: Build inheritance info & resolve method defs
    AutoMark->>AutoMark: build_patches for failures
    AutoMark->>FileSystem: Remove `@expectedFailure` from passing tests
    AutoMark->>FileSystem: Apply patches to mark new failures
    AutoMark-->>CLI: Auto-marking complete (delta counts)

    CLI->>FileSystem: Optional: git commit with version tag
    CLI-->>User: Update workflow finished
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

Suggested reviewers

  • ShaharNaveh
  • arihant2math

Poem

🐰 Hop, hop, the scripts now shine so bright,
Modular magic, organized just right!
Patches and tests in harmony play,
Old scripts retired—update_lib saves the day!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Remake update_lib' clearly and concisely summarizes the main change: a comprehensive rewrite of the update_lib toolkit from a monolithic script to a modular package structure with subcommands.
Linked Issues check ✅ Passed The PR addresses issue #6701 by reformatting the codebase to align with ruff/PEP8 conventions (double quotes), as evidenced by the final commit 'Auto-format: ruff format' and the new modular structure.
Out of Scope Changes check ✅ Passed All changes are in-scope: the update_lib package restructuring, removal of legacy auto_mark_test.py, and documentation updates in the upgrade instructions align with the stated objectives of remaking update_lib and addressing formatting issues.
Docstring Coverage ✅ Passed Docstring coverage is 90.97% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 19, 2026

Code has been automatically formatted

The code in this PR has been formatted using:

  • ruff format
    Please pull the latest changes before pushing again:
git pull origin lib_updater

Comment on lines +10 to +20
If during the upgrade process you encounter any of the following issues with `scripts/update_lib`:
- A feature that should be automated but isn't supported
- A bug or unexpected behavior in the tool
- Missing functionality that would make the upgrade easier

**STOP the upgrade and report the issue first.** Describe:
1. What you were trying to do
- Library name
- The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
2. What went wrong or what's missing
3. Expected vs actual behavior
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than making machine work for workaround, putting the effort to enhance tooling can be better

@youknowone youknowone added the skip:ci Skip running the ci label Jan 19, 2026
# Add scripts directory to path for imports
scripts_dir = Path(__file__).parent.parent
if str(scripts_dir) not in sys.path:
sys.path.insert(0, str(scripts_dir))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we transform this scripts dir into a proper package? that way we could have custom entry-points and not have to mess with sys.path

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this but just found installing python package to run scripts is not a good idea to introduce it to new users. We will not be able to avoid explain to install it everywhere.


# Verify: parse converted and unparse should equal original
try:
converted_ast = ast.parse(converted, mode="eval")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we try to pass it to ruff if it's present on the machine?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have to format only the part we patched. running it for entire file break all the format

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true:/
we can do something hacky of placing the decorator onto a dummy function then passing it to ruff, then taking everything except the last two lines. but maybe that's too complicated

help="Source path (file or directory)",
)
parser.add_argument(
"--no-copy",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use https://docs.python.org/3/library/argparse.html#argparse.BooleanOptionalAction inseatd (and set the default value to False)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer pytest, but this works too

@moreal
Copy link
Contributor

moreal commented Jan 19, 2026

I knew I could push directly to this branch, but I was cautious so I sent a PR: youknowone#10

@youknowone youknowone marked this pull request as ready for review January 20, 2026 02:35
@youknowone
Copy link
Member Author

I ran this for dataclasses, inspect, test_types and working all good. Let's fix other bugs one by one later

@youknowone youknowone merged commit f1fd1e9 into RustPython:main Jan 20, 2026
8 of 9 checks passed
@youknowone youknowone deleted the lib_updater branch January 20, 2026 02:38
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
scripts/update_lib/patch_spec.py (1)

295-302: Handle files without imports to avoid assertion crashes.
If a test file has no imports, _find_import_insert_line asserts and apply_patches fails when adding import unittest. Add a safe fallback (e.g., after module docstring or at top).

🐛 Proposed fix
-    assert last_import_line is not None
-    return last_import_line
+    if last_import_line is not None:
+        return last_import_line
+    # Fallback: insert after module docstring if present, else at top
+    if (
+        tree.body
+        and isinstance(tree.body[0], ast.Expr)
+        and isinstance(tree.body[0].value, ast.Constant)
+        and isinstance(tree.body[0].value.value, str)
+    ):
+        return tree.body[0].end_lineno or tree.body[0].lineno
+    return 0
🤖 Fix all issues with AI agents
In @.claude/commands/upgrade-pylib.md:
- Around line 15-20: The markdown violates MD007 because the unordered list
items under "Describe:" are indented by two spaces; fix by removing the
two-space indentation so the "-" list markers start at column 0 (e.g., adjust
the lines containing "- Library name" and "- The full command executed..." to
have no leading spaces), and scan the rest of .claude/commands/upgrade-pylib.md
for any other unordered list items with extra leading spaces and normalize them
to 0 indentation to satisfy the markdownlint config.

In `@scripts/update_lib/auto_mark.py`:
- Around line 227-244: The helper _is_super_call_only currently only verifies
that the body returns a super() call but doesn't ensure the delegated attribute
matches the current function, which can prune methods like def foo(self): return
super().bar(); update _is_super_call_only to verify that the ast.Attribute on
the Call has attr equal to func_node.name (i.e., ensure call.func.attr ==
func_node.name) before returning True; keep the existing checks for the AST
shapes and optionally ensure arguments/keywords match if you want stronger
equivalence, but at minimum add the attribute-name equality guard in
_is_super_call_only.

In `@scripts/update_lib/copy_lib.py`:
- Around line 45-54: For directory copies, ensure the destination parent exists
before calling shutil.copytree: before calling shutil.copytree(src_path,
lib_path) create lib_path.parent (using mkdir with parents=True, exist_ok=True)
just like the file branch does; update the block that checks src_path.is_dir()
to call lib_path.parent.mkdir(parents=True, exist_ok=True) (and keep the verbose
print) so copytree won’t raise FileNotFoundError.

In `@scripts/update_lib/path.py`:
- Around line 42-78: The lib_to_test_path function mishandles package
__init__.py names (producing test___init__); update both branches (the "/Lib/"
and "Lib/" branches in lib_to_test_path) to detect when src_path.name ==
"__init__.py" (or lib_path.stem == "__init__") and in that case set lib_name to
the parent directory's name (e.g., lib_path.parent.name or src_path.parent.name)
instead of "__init__"; keep the existing directory-first then file-fallback
logic and continue returning the constructed pathlib.Path values as before.

In `@scripts/update_lib/tests/test_migrate.py`:
- Around line 1-196: The new tests in TestPatchSingleContent, TestPatchFile, and
TestPatchDirectory add general test logic and assertions which violates the
test-file modification policy; to fix, move these checks out of a "*test*.py"
file into a non-test helper/module (e.g., create a new module like
update_lib/tests_helpers.py) or replace the changes so that the only
modifications inside the test files are adding/removing
`@unittest.expectedFailure` with TODO comments (references: patch_single_content,
patch_file, patch_directory, and the test classes
TestPatchSingleContent/TestPatchFile/TestPatchDirectory), then update the
import/usage accordingly or request a policy exception.
🧹 Nitpick comments (6)
scripts/update_lib/patches.py (1)

24-30: Use explicit UTF‑8 encoding when writing output.
Locale defaults can vary; explicit UTF‑8 keeps patch output deterministic across environments.

♻️ Proposed tweak
-    with open(dest, "w") as fd:
+    with open(dest, "w", encoding="utf-8") as fd:
scripts/update_lib/path.py (1)

88-99: Consider adding type hint for consistency.

The function works correctly for the documented use cases. Consider adding a type hint for the parameter to match the style of other functions in this module:

Suggested improvement
-def test_name_from_path(test_path: pathlib.Path) -> str:
+def test_name_from_path(test_path: pathlib.Path | str) -> str:
scripts/update_lib/quick.py (4)

43-64: Consider specifying encoding for file reads.

Lines 59 and 63 use read_text() without explicit encoding. While Python 3 defaults to UTF-8 on most systems, explicitly specifying encoding="utf-8" improves cross-platform consistency and makes the intent clear.

Suggested improvement
     if lib_path.is_file():
-        return extract_test_methods(lib_path.read_text())
+        return extract_test_methods(lib_path.read_text(encoding="utf-8"))
     else:
         result = {}
         for lib_file in get_test_files(lib_path):
-            result[lib_file.resolve()] = extract_test_methods(lib_file.read_text())
+            result[lib_file.resolve()] = extract_test_methods(lib_file.read_text(encoding="utf-8"))
         return result

166-225: Consider using a placeholder email address.

The author string "CPython Developers <>" uses an empty email which may cause issues with certain git configurations or CI systems that validate commit metadata. Consider using a no-reply or placeholder email.

Suggested improvement
     subprocess.run(
         [
             "git",
             "commit",
             "--author",
-            "CPython Developers <>",
+            "CPython Developers <[email protected]>",
             "-m",
             message,
         ],
         check=True,
     )

228-260: Minor: Windows path separator not handled in shortcut check.

Line 237 only checks for "/" but not "\\", which could cause issues on Windows where paths might use backslashes. Consider normalizing first:

Suggested improvement
 def _expand_shortcut(path: pathlib.Path) -> pathlib.Path:
     # Only expand if it's a simple name (no path separators) and doesn't exist
-    if "/" in str(path) or path.exists():
+    path_str = str(path).replace("\\", "/")
+    if "/" in path_str or path.exists():
         return path

263-365: Consider handling subprocess.CalledProcessError for git operations.

The main() function catches ValueError and FileNotFoundError but not subprocess.CalledProcessError. If git commands fail (e.g., get_cpython_version or git_commit), the error will result in an uncaught exception with a full traceback instead of a clean error message.

Suggested improvement
+    except subprocess.CalledProcessError as e:
+        print(f"Git command failed: {e}", file=sys.stderr)
+        return 1
     except ValueError as e:
         print(f"Error: {e}", file=sys.stderr)
         return 1

Also add at the top of the file:

 import argparse
 import pathlib
+import subprocess
 import sys

Comment on lines +15 to +20
**STOP the upgrade and report the issue first.** Describe:
1. What you were trying to do
- Library name
- The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
2. What went wrong or what's missing
3. Expected vs actual behavior
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Fix MD007 list indentation for markdownlint.
The unordered list items are indented by 2 spaces but the config expects 0.

✍️ Suggested fix
-  - Library name
-  - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
+- Library name
+- The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
**STOP the upgrade and report the issue first.** Describe:
1. What you were trying to do
- Library name
- The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
2. What went wrong or what's missing
3. Expected vs actual behavior
**STOP the upgrade and report the issue first.** Describe:
1. What you were trying to do
- Library name
- The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
2. What went wrong or what's missing
3. Expected vs actual behavior
🧰 Tools
🪛 markdownlint-cli2 (0.18.1)

17-17: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)


18-18: Unordered list indentation
Expected: 0; Actual: 2

(MD007, ul-indent)

🤖 Prompt for AI Agents
In @.claude/commands/upgrade-pylib.md around lines 15 - 20, The markdown
violates MD007 because the unordered list items under "Describe:" are indented
by two spaces; fix by removing the two-space indentation so the "-" list markers
start at column 0 (e.g., adjust the lines containing "- Library name" and "- The
full command executed..." to have no leading spaces), and scan the rest of
.claude/commands/upgrade-pylib.md for any other unordered list items with extra
leading spaces and normalize them to 0 indentation to satisfy the markdownlint
config.

Comment on lines +227 to +244
def _is_super_call_only(func_node: ast.FunctionDef | ast.AsyncFunctionDef) -> bool:
"""Check if the method body is just 'return super().method_name()'."""
if len(func_node.body) != 1:
return False
stmt = func_node.body[0]
if not isinstance(stmt, ast.Return) or stmt.value is None:
return False
call = stmt.value
if not isinstance(call, ast.Call):
return False
if not isinstance(call.func, ast.Attribute):
return False
super_call = call.func.value
if not isinstance(super_call, ast.Call):
return False
if not isinstance(super_call.func, ast.Name) or super_call.func.id != "super":
return False
return True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Ensure super() delegation matches the current method before pruning overrides.
Without checking the attribute name, a method that returns super().other() could be removed, changing behavior when expectedFailure is stripped.

🐛 Proposed fix
-    if not isinstance(call.func, ast.Attribute):
-        return False
+    if not isinstance(call.func, ast.Attribute):
+        return False
+    if call.func.attr != func_node.name:
+        return False
🤖 Prompt for AI Agents
In `@scripts/update_lib/auto_mark.py` around lines 227 - 244, The helper
_is_super_call_only currently only verifies that the body returns a super() call
but doesn't ensure the delegated attribute matches the current function, which
can prune methods like def foo(self): return super().bar(); update
_is_super_call_only to verify that the ast.Attribute on the Call has attr equal
to func_node.name (i.e., ensure call.func.attr == func_node.name) before
returning True; keep the existing checks for the AST shapes and optionally
ensure arguments/keywords match if you want stronger equivalence, but at minimum
add the attribute-name equality guard in _is_super_call_only.

Comment on lines +45 to +54
# Copy
if src_path.is_dir():
if verbose:
print(f"Copying directory: {src_path} -> {lib_path}")
shutil.copytree(src_path, lib_path)
else:
if verbose:
print(f"Copying file: {src_path} -> {lib_path}")
lib_path.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(src_path, lib_path)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Ensure parent directories exist before copytree.

shutil.copytree requires the destination’s parent directory to exist; otherwise it raises FileNotFoundError when copying nested paths. Create lib_path.parent before copying directories.

🔧 Proposed fix
     if src_path.is_dir():
         if verbose:
             print(f"Copying directory: {src_path} -> {lib_path}")
+        lib_path.parent.mkdir(parents=True, exist_ok=True)
         shutil.copytree(src_path, lib_path)
🤖 Prompt for AI Agents
In `@scripts/update_lib/copy_lib.py` around lines 45 - 54, For directory copies,
ensure the destination parent exists before calling shutil.copytree: before
calling shutil.copytree(src_path, lib_path) create lib_path.parent (using mkdir
with parents=True, exist_ok=True) just like the file branch does; update the
block that checks src_path.is_dir() to call lib_path.parent.mkdir(parents=True,
exist_ok=True) (and keep the verbose print) so copytree won’t raise
FileNotFoundError.

Comment on lines +42 to +78
def lib_to_test_path(src_path: pathlib.Path) -> pathlib.Path:
"""
Convert library path to test path.
Examples:
cpython/Lib/dataclasses.py -> cpython/Lib/test/test_dataclasses/ (if dir exists)
cpython/Lib/typing.py -> cpython/Lib/test/test_typing.py (if file exists)
cpython/Lib/json/ -> cpython/Lib/test/test_json/
Lib/dataclasses.py -> Lib/test/test_dataclasses/
"""
path_str = str(src_path).replace("\\", "/")
lib_marker = "/Lib/"

if lib_marker in path_str:
lib_path = parse_lib_path(src_path)
lib_name = lib_path.stem if lib_path.suffix == ".py" else lib_path.name
prefix = path_str[: path_str.index(lib_marker)]
# Try directory first, then file
dir_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}/")
if dir_path.exists():
return dir_path
file_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}.py")
if file_path.exists():
return file_path
# Default to directory (caller will handle non-existence)
return dir_path
else:
# Path starts with Lib/ - extract name directly
lib_name = src_path.stem if src_path.suffix == ".py" else src_path.name
# Try directory first, then file
dir_path = pathlib.Path(f"Lib/test/test_{lib_name}/")
if dir_path.exists():
return dir_path
file_path = pathlib.Path(f"Lib/test/test_{lib_name}.py")
if file_path.exists():
return file_path
return dir_path
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find and examine parse_lib_path function and context
rg -n "def parse_lib_path" --type=py -A 15 scripts/

Repository: RustPython/RustPython

Length of output: 1046


🏁 Script executed:

#!/bin/bash
# Check usages of lib_to_test_path in the codebase
rg -n "lib_to_test_path" --type=py -B 2 -A 2

Repository: RustPython/RustPython

Length of output: 5519


🏁 Script executed:

#!/bin/bash
# Check the actual test directory structure for packages
# Looking for test_json, test_typing, etc.
find . -type d -name "test_*" -path "*/Lib/test/*" | head -20

Repository: RustPython/RustPython

Length of output: 598


🏁 Script executed:

#!/bin/bash
# Check what __init__.py files exist in Lib/
find cpython/Lib -name "__init__.py" -type f 2>/dev/null | head -10

Repository: RustPython/RustPython

Length of output: 47


🏁 Script executed:

#!/bin/bash
# Check the else branch of lib_to_test_path to see if it has the same issue
sed -n '69,78p' scripts/update_lib/path.py

Repository: RustPython/RustPython

Length of output: 510


🏁 Script executed:

#!/bin/bash
# View the complete test cases in test_path.py to understand all scenarios
sed -n '81,140p' scripts/update_lib/tests/test_path.py

Repository: RustPython/RustPython

Length of output: 2775


Fix __init__.py handling in both branches of lib_to_test_path.

The function incorrectly handles package __init__.py files in both the "/Lib/" and "Lib/" branches. For example, Lib/json/__init__.py would produce lib_name = "__init__", resulting in test___init__/ instead of the correct test_json/.

Check for __init__.py and use the parent directory name:

Suggested fix
     if lib_marker in path_str:
         lib_path = parse_lib_path(src_path)
-        lib_name = lib_path.stem if lib_path.suffix == ".py" else lib_path.name
+        if lib_path.stem == "__init__":
+            lib_name = lib_path.parent.name
+        elif lib_path.suffix == ".py":
+            lib_name = lib_path.stem
+        else:
+            lib_name = lib_path.name
         prefix = path_str[: path_str.index(lib_marker)]
     else:
         # Path starts with Lib/ - extract name directly
-        lib_name = src_path.stem if src_path.suffix == ".py" else src_path.name
+        if src_path.stem == "__init__":
+            lib_name = src_path.parent.name
+        elif src_path.suffix == ".py":
+            lib_name = src_path.stem
+        else:
+            lib_name = src_path.name
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def lib_to_test_path(src_path: pathlib.Path) -> pathlib.Path:
"""
Convert library path to test path.
Examples:
cpython/Lib/dataclasses.py -> cpython/Lib/test/test_dataclasses/ (if dir exists)
cpython/Lib/typing.py -> cpython/Lib/test/test_typing.py (if file exists)
cpython/Lib/json/ -> cpython/Lib/test/test_json/
Lib/dataclasses.py -> Lib/test/test_dataclasses/
"""
path_str = str(src_path).replace("\\", "/")
lib_marker = "/Lib/"
if lib_marker in path_str:
lib_path = parse_lib_path(src_path)
lib_name = lib_path.stem if lib_path.suffix == ".py" else lib_path.name
prefix = path_str[: path_str.index(lib_marker)]
# Try directory first, then file
dir_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}/")
if dir_path.exists():
return dir_path
file_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}.py")
if file_path.exists():
return file_path
# Default to directory (caller will handle non-existence)
return dir_path
else:
# Path starts with Lib/ - extract name directly
lib_name = src_path.stem if src_path.suffix == ".py" else src_path.name
# Try directory first, then file
dir_path = pathlib.Path(f"Lib/test/test_{lib_name}/")
if dir_path.exists():
return dir_path
file_path = pathlib.Path(f"Lib/test/test_{lib_name}.py")
if file_path.exists():
return file_path
return dir_path
def lib_to_test_path(src_path: pathlib.Path) -> pathlib.Path:
"""
Convert library path to test path.
Examples:
cpython/Lib/dataclasses.py -> cpython/Lib/test/test_dataclasses/ (if dir exists)
cpython/Lib/typing.py -> cpython/Lib/test/test_typing.py (if file exists)
cpython/Lib/json/ -> cpython/Lib/test/test_json/
Lib/dataclasses.py -> Lib/test/test_dataclasses/
"""
path_str = str(src_path).replace("\\", "/")
lib_marker = "/Lib/"
if lib_marker in path_str:
lib_path = parse_lib_path(src_path)
if lib_path.stem == "__init__":
lib_name = lib_path.parent.name
elif lib_path.suffix == ".py":
lib_name = lib_path.stem
else:
lib_name = lib_path.name
prefix = path_str[: path_str.index(lib_marker)]
# Try directory first, then file
dir_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}/")
if dir_path.exists():
return dir_path
file_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}.py")
if file_path.exists():
return file_path
# Default to directory (caller will handle non-existence)
return dir_path
else:
# Path starts with Lib/ - extract name directly
if src_path.stem == "__init__":
lib_name = src_path.parent.name
elif src_path.suffix == ".py":
lib_name = src_path.stem
else:
lib_name = src_path.name
# Try directory first, then file
dir_path = pathlib.Path(f"Lib/test/test_{lib_name}/")
if dir_path.exists():
return dir_path
file_path = pathlib.Path(f"Lib/test/test_{lib_name}.py")
if file_path.exists():
return file_path
return dir_path
🤖 Prompt for AI Agents
In `@scripts/update_lib/path.py` around lines 42 - 78, The lib_to_test_path
function mishandles package __init__.py names (producing test___init__); update
both branches (the "/Lib/" and "Lib/" branches in lib_to_test_path) to detect
when src_path.name == "__init__.py" (or lib_path.stem == "__init__") and in that
case set lib_name to the parent directory's name (e.g., lib_path.parent.name or
src_path.parent.name) instead of "__init__"; keep the existing directory-first
then file-fallback logic and continue returning the constructed pathlib.Path
values as before.

Comment on lines +1 to +196
"""Tests for migrate.py - file migration operations."""

import pathlib
import tempfile
import unittest

from update_lib.migrate import (
patch_directory,
patch_file,
patch_single_content,
)
from update_lib.patch_spec import COMMENT


class TestPatchSingleContent(unittest.TestCase):
"""Tests for patch_single_content function."""

def test_patch_with_no_existing_file(self):
"""Test patching when lib file doesn't exist."""
with tempfile.TemporaryDirectory() as tmpdir:
tmpdir = pathlib.Path(tmpdir)

# Create source file
src_path = tmpdir / "src.py"
src_path.write_text("""import unittest
class TestFoo(unittest.TestCase):
def test_one(self):
pass
""")

# Non-existent lib path
lib_path = tmpdir / "lib.py"

result = patch_single_content(src_path, lib_path)

# Should return source content unchanged
self.assertIn("def test_one(self):", result)
self.assertNotIn(COMMENT, result)

def test_patch_with_existing_patches(self):
"""Test patching preserves existing patches."""
with tempfile.TemporaryDirectory() as tmpdir:
tmpdir = pathlib.Path(tmpdir)

# Create source file (new version)
src_path = tmpdir / "src.py"
src_path.write_text("""import unittest
class TestFoo(unittest.TestCase):
def test_one(self):
pass
def test_two(self):
pass
""")

# Create lib file with existing patch
lib_path = tmpdir / "lib.py"
lib_path.write_text(f"""import unittest
class TestFoo(unittest.TestCase):
# {COMMENT}
@unittest.expectedFailure
def test_one(self):
pass
""")

result = patch_single_content(src_path, lib_path)

# Should have patch on test_one
self.assertIn("@unittest.expectedFailure", result)
self.assertIn(COMMENT, result)
# Should have test_two from source
self.assertIn("def test_two(self):", result)


class TestPatchFile(unittest.TestCase):
"""Tests for patch_file function."""

def test_patch_file_creates_output(self):
"""Test that patch_file writes output file."""
with tempfile.TemporaryDirectory() as tmpdir:
tmpdir = pathlib.Path(tmpdir)

# Create source file
src_path = tmpdir / "src.py"
src_path.write_text("""import unittest
class TestFoo(unittest.TestCase):
def test_one(self):
pass
""")

# Output path
lib_path = tmpdir / "Lib" / "test.py"

patch_file(src_path, lib_path, verbose=False)

# File should exist
self.assertTrue(lib_path.exists())
content = lib_path.read_text()
self.assertIn("def test_one(self):", content)

def test_patch_file_preserves_patches(self):
"""Test that patch_file preserves existing patches."""
with tempfile.TemporaryDirectory() as tmpdir:
tmpdir = pathlib.Path(tmpdir)

# Create source file
src_path = tmpdir / "src.py"
src_path.write_text("""import unittest
class TestFoo(unittest.TestCase):
def test_one(self):
pass
""")

# Create existing lib file with patch
lib_path = tmpdir / "lib.py"
lib_path.write_text(f"""import unittest
class TestFoo(unittest.TestCase):
# {COMMENT}
@unittest.expectedFailure
def test_one(self):
pass
""")

patch_file(src_path, lib_path, verbose=False)

content = lib_path.read_text()
self.assertIn("@unittest.expectedFailure", content)


class TestPatchDirectory(unittest.TestCase):
"""Tests for patch_directory function."""

def test_patch_directory_all_files(self):
"""Test that patch_directory processes all .py files."""
with tempfile.TemporaryDirectory() as tmpdir:
tmpdir = pathlib.Path(tmpdir)

# Create source directory with files
src_dir = tmpdir / "src"
src_dir.mkdir()
(src_dir / "test_a.py").write_text("# test_a")
(src_dir / "test_b.py").write_text("# test_b")
(src_dir / "subdir").mkdir()
(src_dir / "subdir" / "test_c.py").write_text("# test_c")

# Output directory
lib_dir = tmpdir / "lib"

patch_directory(src_dir, lib_dir, verbose=False)

# All files should exist
self.assertTrue((lib_dir / "test_a.py").exists())
self.assertTrue((lib_dir / "test_b.py").exists())
self.assertTrue((lib_dir / "subdir" / "test_c.py").exists())

def test_patch_directory_preserves_patches(self):
"""Test that patch_directory preserves patches in existing files."""
with tempfile.TemporaryDirectory() as tmpdir:
tmpdir = pathlib.Path(tmpdir)

# Create source directory
src_dir = tmpdir / "src"
src_dir.mkdir()
(src_dir / "test_a.py").write_text("""import unittest
class TestA(unittest.TestCase):
def test_one(self):
pass
""")

# Create lib directory with patched file
lib_dir = tmpdir / "lib"
lib_dir.mkdir()
(lib_dir / "test_a.py").write_text(f"""import unittest
class TestA(unittest.TestCase):
# {COMMENT}
@unittest.expectedFailure
def test_one(self):
pass
""")

patch_directory(src_dir, lib_dir, verbose=False)

content = (lib_dir / "test_a.py").read_text()
self.assertIn("@unittest.expectedFailure", content)


if __name__ == "__main__":
unittest.main()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Test additions violate the test-file modification policy.

This file adds new test logic and assertions, but the repo guideline for **/*test*.py restricts changes to only adding/removing @unittest.expectedFailure with TODO comments. Please move these checks to a non-test module or seek a policy exception. As per coding guidelines, ...

🤖 Prompt for AI Agents
In `@scripts/update_lib/tests/test_migrate.py` around lines 1 - 196, The new tests
in TestPatchSingleContent, TestPatchFile, and TestPatchDirectory add general
test logic and assertions which violates the test-file modification policy; to
fix, move these checks out of a "*test*.py" file into a non-test helper/module
(e.g., create a new module like update_lib/tests_helpers.py) or replace the
changes so that the only modifications inside the test files are adding/removing
`@unittest.expectedFailure` with TODO comments (references: patch_single_content,
patch_file, patch_directory, and the test classes
TestPatchSingleContent/TestPatchFile/TestPatchDirectory), then update the
import/usage accordingly or request a policy exception.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip:ci Skip running the ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lib_updater formatting is not aligned to ruff format

3 participants