-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Remake update_lib #6796
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remake update_lib #6796
Conversation
📝 WalkthroughWalkthroughThe PR replaces the monolithic Changes
Sequence Diagram(s)sequenceDiagram
actor User
participant CLI as update_lib<br/>(quick)
participant Migrate as migrate.py
participant AutoMark as auto_mark.py
participant TestRunner as run_test()
participant FileSystem as File System
User->>CLI: quick [path] --no-migrate=False
CLI->>Migrate: patch_file/patch_directory
Migrate->>FileSystem: Extract patches from Lib/test/<file>.py
Migrate->>FileSystem: Read cpython/Lib/test/<file>.py
Migrate->>FileSystem: Apply patches & write to Lib/test/<file>.py
Migrate-->>CLI: Patching complete
CLI->>AutoMark: auto_mark_file/auto_mark_directory
AutoMark->>TestRunner: run_test(test_name)
TestRunner->>FileSystem: Execute RustPython tests
TestRunner-->>AutoMark: TestResult{failures, unexpected_successes}
AutoMark->>AutoMark: Parse AST to extract test methods
AutoMark->>AutoMark: Build inheritance info & resolve method defs
AutoMark->>AutoMark: build_patches for failures
AutoMark->>FileSystem: Remove `@expectedFailure` from passing tests
AutoMark->>FileSystem: Apply patches to mark new failures
AutoMark-->>CLI: Auto-marking complete (delta counts)
CLI->>FileSystem: Optional: git commit with version tag
CLI-->>User: Update workflow finished
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Code has been automatically formatted The code in this PR has been formatted using:
git pull origin lib_updater |
| If during the upgrade process you encounter any of the following issues with `scripts/update_lib`: | ||
| - A feature that should be automated but isn't supported | ||
| - A bug or unexpected behavior in the tool | ||
| - Missing functionality that would make the upgrade easier | ||
|
|
||
| **STOP the upgrade and report the issue first.** Describe: | ||
| 1. What you were trying to do | ||
| - Library name | ||
| - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py) | ||
| 2. What went wrong or what's missing | ||
| 3. Expected vs actual behavior |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than making machine work for workaround, putting the effort to enhance tooling can be better
scripts/update_lib/__main__.py
Outdated
| # Add scripts directory to path for imports | ||
| scripts_dir = Path(__file__).parent.parent | ||
| if str(scripts_dir) not in sys.path: | ||
| sys.path.insert(0, str(scripts_dir)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we transform this scripts dir into a proper package? that way we could have custom entry-points and not have to mess with sys.path
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried this but just found installing python package to run scripts is not a good idea to introduce it to new users. We will not be able to avoid explain to install it everywhere.
|
|
||
| # Verify: parse converted and unparse should equal original | ||
| try: | ||
| converted_ast = ast.parse(converted, mode="eval") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we try to pass it to ruff if it's present on the machine?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we have to format only the part we patched. running it for entire file break all the format
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's true:/
we can do something hacky of placing the decorator onto a dummy function then passing it to ruff, then taking everything except the last two lines. but maybe that's too complicated
scripts/update_lib/quick.py
Outdated
| help="Source path (file or directory)", | ||
| ) | ||
| parser.add_argument( | ||
| "--no-copy", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can use https://docs.python.org/3/library/argparse.html#argparse.BooleanOptionalAction inseatd (and set the default value to False)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer pytest, but this works too
03ee58f to
1813c7d
Compare
|
I knew I could push directly to this branch, but I was cautious so I sent a PR: youknowone#10 |
212d54f to
fd0bc1b
Compare
fd0bc1b to
b295586
Compare
|
I ran this for dataclasses, inspect, test_types and working all good. Let's fix other bugs one by one later |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/update_lib/patch_spec.py (1)
295-302: Handle files without imports to avoid assertion crashes.
If a test file has no imports,_find_import_insert_lineasserts andapply_patchesfails when addingimport unittest. Add a safe fallback (e.g., after module docstring or at top).🐛 Proposed fix
- assert last_import_line is not None - return last_import_line + if last_import_line is not None: + return last_import_line + # Fallback: insert after module docstring if present, else at top + if ( + tree.body + and isinstance(tree.body[0], ast.Expr) + and isinstance(tree.body[0].value, ast.Constant) + and isinstance(tree.body[0].value.value, str) + ): + return tree.body[0].end_lineno or tree.body[0].lineno + return 0
🤖 Fix all issues with AI agents
In @.claude/commands/upgrade-pylib.md:
- Around line 15-20: The markdown violates MD007 because the unordered list
items under "Describe:" are indented by two spaces; fix by removing the
two-space indentation so the "-" list markers start at column 0 (e.g., adjust
the lines containing "- Library name" and "- The full command executed..." to
have no leading spaces), and scan the rest of .claude/commands/upgrade-pylib.md
for any other unordered list items with extra leading spaces and normalize them
to 0 indentation to satisfy the markdownlint config.
In `@scripts/update_lib/auto_mark.py`:
- Around line 227-244: The helper _is_super_call_only currently only verifies
that the body returns a super() call but doesn't ensure the delegated attribute
matches the current function, which can prune methods like def foo(self): return
super().bar(); update _is_super_call_only to verify that the ast.Attribute on
the Call has attr equal to func_node.name (i.e., ensure call.func.attr ==
func_node.name) before returning True; keep the existing checks for the AST
shapes and optionally ensure arguments/keywords match if you want stronger
equivalence, but at minimum add the attribute-name equality guard in
_is_super_call_only.
In `@scripts/update_lib/copy_lib.py`:
- Around line 45-54: For directory copies, ensure the destination parent exists
before calling shutil.copytree: before calling shutil.copytree(src_path,
lib_path) create lib_path.parent (using mkdir with parents=True, exist_ok=True)
just like the file branch does; update the block that checks src_path.is_dir()
to call lib_path.parent.mkdir(parents=True, exist_ok=True) (and keep the verbose
print) so copytree won’t raise FileNotFoundError.
In `@scripts/update_lib/path.py`:
- Around line 42-78: The lib_to_test_path function mishandles package
__init__.py names (producing test___init__); update both branches (the "/Lib/"
and "Lib/" branches in lib_to_test_path) to detect when src_path.name ==
"__init__.py" (or lib_path.stem == "__init__") and in that case set lib_name to
the parent directory's name (e.g., lib_path.parent.name or src_path.parent.name)
instead of "__init__"; keep the existing directory-first then file-fallback
logic and continue returning the constructed pathlib.Path values as before.
In `@scripts/update_lib/tests/test_migrate.py`:
- Around line 1-196: The new tests in TestPatchSingleContent, TestPatchFile, and
TestPatchDirectory add general test logic and assertions which violates the
test-file modification policy; to fix, move these checks out of a "*test*.py"
file into a non-test helper/module (e.g., create a new module like
update_lib/tests_helpers.py) or replace the changes so that the only
modifications inside the test files are adding/removing
`@unittest.expectedFailure` with TODO comments (references: patch_single_content,
patch_file, patch_directory, and the test classes
TestPatchSingleContent/TestPatchFile/TestPatchDirectory), then update the
import/usage accordingly or request a policy exception.
🧹 Nitpick comments (6)
scripts/update_lib/patches.py (1)
24-30: Use explicit UTF‑8 encoding when writing output.
Locale defaults can vary; explicit UTF‑8 keeps patch output deterministic across environments.♻️ Proposed tweak
- with open(dest, "w") as fd: + with open(dest, "w", encoding="utf-8") as fd:scripts/update_lib/path.py (1)
88-99: Consider adding type hint for consistency.The function works correctly for the documented use cases. Consider adding a type hint for the parameter to match the style of other functions in this module:
Suggested improvement
-def test_name_from_path(test_path: pathlib.Path) -> str: +def test_name_from_path(test_path: pathlib.Path | str) -> str:scripts/update_lib/quick.py (4)
43-64: Consider specifying encoding for file reads.Lines 59 and 63 use
read_text()without explicit encoding. While Python 3 defaults to UTF-8 on most systems, explicitly specifyingencoding="utf-8"improves cross-platform consistency and makes the intent clear.Suggested improvement
if lib_path.is_file(): - return extract_test_methods(lib_path.read_text()) + return extract_test_methods(lib_path.read_text(encoding="utf-8")) else: result = {} for lib_file in get_test_files(lib_path): - result[lib_file.resolve()] = extract_test_methods(lib_file.read_text()) + result[lib_file.resolve()] = extract_test_methods(lib_file.read_text(encoding="utf-8")) return result
166-225: Consider using a placeholder email address.The author string
"CPython Developers <>"uses an empty email which may cause issues with certain git configurations or CI systems that validate commit metadata. Consider using a no-reply or placeholder email.Suggested improvement
subprocess.run( [ "git", "commit", "--author", - "CPython Developers <>", + "CPython Developers <[email protected]>", "-m", message, ], check=True, )
228-260: Minor: Windows path separator not handled in shortcut check.Line 237 only checks for
"/"but not"\\", which could cause issues on Windows where paths might use backslashes. Consider normalizing first:Suggested improvement
def _expand_shortcut(path: pathlib.Path) -> pathlib.Path: # Only expand if it's a simple name (no path separators) and doesn't exist - if "/" in str(path) or path.exists(): + path_str = str(path).replace("\\", "/") + if "/" in path_str or path.exists(): return path
263-365: Consider handlingsubprocess.CalledProcessErrorfor git operations.The
main()function catchesValueErrorandFileNotFoundErrorbut notsubprocess.CalledProcessError. If git commands fail (e.g.,get_cpython_versionorgit_commit), the error will result in an uncaught exception with a full traceback instead of a clean error message.Suggested improvement
+ except subprocess.CalledProcessError as e: + print(f"Git command failed: {e}", file=sys.stderr) + return 1 except ValueError as e: print(f"Error: {e}", file=sys.stderr) return 1Also add at the top of the file:
import argparse import pathlib +import subprocess import sys
| **STOP the upgrade and report the issue first.** Describe: | ||
| 1. What you were trying to do | ||
| - Library name | ||
| - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py) | ||
| 2. What went wrong or what's missing | ||
| 3. Expected vs actual behavior |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix MD007 list indentation for markdownlint.
The unordered list items are indented by 2 spaces but the config expects 0.
✍️ Suggested fix
- - Library name
- - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
+- Library name
+- The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| **STOP the upgrade and report the issue first.** Describe: | |
| 1. What you were trying to do | |
| - Library name | |
| - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py) | |
| 2. What went wrong or what's missing | |
| 3. Expected vs actual behavior | |
| **STOP the upgrade and report the issue first.** Describe: | |
| 1. What you were trying to do | |
| - Library name | |
| - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py) | |
| 2. What went wrong or what's missing | |
| 3. Expected vs actual behavior |
🧰 Tools
🪛 markdownlint-cli2 (0.18.1)
17-17: Unordered list indentation
Expected: 0; Actual: 2
(MD007, ul-indent)
18-18: Unordered list indentation
Expected: 0; Actual: 2
(MD007, ul-indent)
🤖 Prompt for AI Agents
In @.claude/commands/upgrade-pylib.md around lines 15 - 20, The markdown
violates MD007 because the unordered list items under "Describe:" are indented
by two spaces; fix by removing the two-space indentation so the "-" list markers
start at column 0 (e.g., adjust the lines containing "- Library name" and "- The
full command executed..." to have no leading spaces), and scan the rest of
.claude/commands/upgrade-pylib.md for any other unordered list items with extra
leading spaces and normalize them to 0 indentation to satisfy the markdownlint
config.
| def _is_super_call_only(func_node: ast.FunctionDef | ast.AsyncFunctionDef) -> bool: | ||
| """Check if the method body is just 'return super().method_name()'.""" | ||
| if len(func_node.body) != 1: | ||
| return False | ||
| stmt = func_node.body[0] | ||
| if not isinstance(stmt, ast.Return) or stmt.value is None: | ||
| return False | ||
| call = stmt.value | ||
| if not isinstance(call, ast.Call): | ||
| return False | ||
| if not isinstance(call.func, ast.Attribute): | ||
| return False | ||
| super_call = call.func.value | ||
| if not isinstance(super_call, ast.Call): | ||
| return False | ||
| if not isinstance(super_call.func, ast.Name) or super_call.func.id != "super": | ||
| return False | ||
| return True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ensure super() delegation matches the current method before pruning overrides.
Without checking the attribute name, a method that returns super().other() could be removed, changing behavior when expectedFailure is stripped.
🐛 Proposed fix
- if not isinstance(call.func, ast.Attribute):
- return False
+ if not isinstance(call.func, ast.Attribute):
+ return False
+ if call.func.attr != func_node.name:
+ return False🤖 Prompt for AI Agents
In `@scripts/update_lib/auto_mark.py` around lines 227 - 244, The helper
_is_super_call_only currently only verifies that the body returns a super() call
but doesn't ensure the delegated attribute matches the current function, which
can prune methods like def foo(self): return super().bar(); update
_is_super_call_only to verify that the ast.Attribute on the Call has attr equal
to func_node.name (i.e., ensure call.func.attr == func_node.name) before
returning True; keep the existing checks for the AST shapes and optionally
ensure arguments/keywords match if you want stronger equivalence, but at minimum
add the attribute-name equality guard in _is_super_call_only.
| # Copy | ||
| if src_path.is_dir(): | ||
| if verbose: | ||
| print(f"Copying directory: {src_path} -> {lib_path}") | ||
| shutil.copytree(src_path, lib_path) | ||
| else: | ||
| if verbose: | ||
| print(f"Copying file: {src_path} -> {lib_path}") | ||
| lib_path.parent.mkdir(parents=True, exist_ok=True) | ||
| shutil.copy2(src_path, lib_path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ensure parent directories exist before copytree.
shutil.copytree requires the destination’s parent directory to exist; otherwise it raises FileNotFoundError when copying nested paths. Create lib_path.parent before copying directories.
🔧 Proposed fix
if src_path.is_dir():
if verbose:
print(f"Copying directory: {src_path} -> {lib_path}")
+ lib_path.parent.mkdir(parents=True, exist_ok=True)
shutil.copytree(src_path, lib_path)🤖 Prompt for AI Agents
In `@scripts/update_lib/copy_lib.py` around lines 45 - 54, For directory copies,
ensure the destination parent exists before calling shutil.copytree: before
calling shutil.copytree(src_path, lib_path) create lib_path.parent (using mkdir
with parents=True, exist_ok=True) just like the file branch does; update the
block that checks src_path.is_dir() to call lib_path.parent.mkdir(parents=True,
exist_ok=True) (and keep the verbose print) so copytree won’t raise
FileNotFoundError.
| def lib_to_test_path(src_path: pathlib.Path) -> pathlib.Path: | ||
| """ | ||
| Convert library path to test path. | ||
| Examples: | ||
| cpython/Lib/dataclasses.py -> cpython/Lib/test/test_dataclasses/ (if dir exists) | ||
| cpython/Lib/typing.py -> cpython/Lib/test/test_typing.py (if file exists) | ||
| cpython/Lib/json/ -> cpython/Lib/test/test_json/ | ||
| Lib/dataclasses.py -> Lib/test/test_dataclasses/ | ||
| """ | ||
| path_str = str(src_path).replace("\\", "/") | ||
| lib_marker = "/Lib/" | ||
|
|
||
| if lib_marker in path_str: | ||
| lib_path = parse_lib_path(src_path) | ||
| lib_name = lib_path.stem if lib_path.suffix == ".py" else lib_path.name | ||
| prefix = path_str[: path_str.index(lib_marker)] | ||
| # Try directory first, then file | ||
| dir_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}/") | ||
| if dir_path.exists(): | ||
| return dir_path | ||
| file_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}.py") | ||
| if file_path.exists(): | ||
| return file_path | ||
| # Default to directory (caller will handle non-existence) | ||
| return dir_path | ||
| else: | ||
| # Path starts with Lib/ - extract name directly | ||
| lib_name = src_path.stem if src_path.suffix == ".py" else src_path.name | ||
| # Try directory first, then file | ||
| dir_path = pathlib.Path(f"Lib/test/test_{lib_name}/") | ||
| if dir_path.exists(): | ||
| return dir_path | ||
| file_path = pathlib.Path(f"Lib/test/test_{lib_name}.py") | ||
| if file_path.exists(): | ||
| return file_path | ||
| return dir_path |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find and examine parse_lib_path function and context
rg -n "def parse_lib_path" --type=py -A 15 scripts/Repository: RustPython/RustPython
Length of output: 1046
🏁 Script executed:
#!/bin/bash
# Check usages of lib_to_test_path in the codebase
rg -n "lib_to_test_path" --type=py -B 2 -A 2Repository: RustPython/RustPython
Length of output: 5519
🏁 Script executed:
#!/bin/bash
# Check the actual test directory structure for packages
# Looking for test_json, test_typing, etc.
find . -type d -name "test_*" -path "*/Lib/test/*" | head -20Repository: RustPython/RustPython
Length of output: 598
🏁 Script executed:
#!/bin/bash
# Check what __init__.py files exist in Lib/
find cpython/Lib -name "__init__.py" -type f 2>/dev/null | head -10Repository: RustPython/RustPython
Length of output: 47
🏁 Script executed:
#!/bin/bash
# Check the else branch of lib_to_test_path to see if it has the same issue
sed -n '69,78p' scripts/update_lib/path.pyRepository: RustPython/RustPython
Length of output: 510
🏁 Script executed:
#!/bin/bash
# View the complete test cases in test_path.py to understand all scenarios
sed -n '81,140p' scripts/update_lib/tests/test_path.pyRepository: RustPython/RustPython
Length of output: 2775
Fix __init__.py handling in both branches of lib_to_test_path.
The function incorrectly handles package __init__.py files in both the "/Lib/" and "Lib/" branches. For example, Lib/json/__init__.py would produce lib_name = "__init__", resulting in test___init__/ instead of the correct test_json/.
Check for __init__.py and use the parent directory name:
Suggested fix
if lib_marker in path_str:
lib_path = parse_lib_path(src_path)
- lib_name = lib_path.stem if lib_path.suffix == ".py" else lib_path.name
+ if lib_path.stem == "__init__":
+ lib_name = lib_path.parent.name
+ elif lib_path.suffix == ".py":
+ lib_name = lib_path.stem
+ else:
+ lib_name = lib_path.name
prefix = path_str[: path_str.index(lib_marker)]
else:
# Path starts with Lib/ - extract name directly
- lib_name = src_path.stem if src_path.suffix == ".py" else src_path.name
+ if src_path.stem == "__init__":
+ lib_name = src_path.parent.name
+ elif src_path.suffix == ".py":
+ lib_name = src_path.stem
+ else:
+ lib_name = src_path.name📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def lib_to_test_path(src_path: pathlib.Path) -> pathlib.Path: | |
| """ | |
| Convert library path to test path. | |
| Examples: | |
| cpython/Lib/dataclasses.py -> cpython/Lib/test/test_dataclasses/ (if dir exists) | |
| cpython/Lib/typing.py -> cpython/Lib/test/test_typing.py (if file exists) | |
| cpython/Lib/json/ -> cpython/Lib/test/test_json/ | |
| Lib/dataclasses.py -> Lib/test/test_dataclasses/ | |
| """ | |
| path_str = str(src_path).replace("\\", "/") | |
| lib_marker = "/Lib/" | |
| if lib_marker in path_str: | |
| lib_path = parse_lib_path(src_path) | |
| lib_name = lib_path.stem if lib_path.suffix == ".py" else lib_path.name | |
| prefix = path_str[: path_str.index(lib_marker)] | |
| # Try directory first, then file | |
| dir_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}/") | |
| if dir_path.exists(): | |
| return dir_path | |
| file_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}.py") | |
| if file_path.exists(): | |
| return file_path | |
| # Default to directory (caller will handle non-existence) | |
| return dir_path | |
| else: | |
| # Path starts with Lib/ - extract name directly | |
| lib_name = src_path.stem if src_path.suffix == ".py" else src_path.name | |
| # Try directory first, then file | |
| dir_path = pathlib.Path(f"Lib/test/test_{lib_name}/") | |
| if dir_path.exists(): | |
| return dir_path | |
| file_path = pathlib.Path(f"Lib/test/test_{lib_name}.py") | |
| if file_path.exists(): | |
| return file_path | |
| return dir_path | |
| def lib_to_test_path(src_path: pathlib.Path) -> pathlib.Path: | |
| """ | |
| Convert library path to test path. | |
| Examples: | |
| cpython/Lib/dataclasses.py -> cpython/Lib/test/test_dataclasses/ (if dir exists) | |
| cpython/Lib/typing.py -> cpython/Lib/test/test_typing.py (if file exists) | |
| cpython/Lib/json/ -> cpython/Lib/test/test_json/ | |
| Lib/dataclasses.py -> Lib/test/test_dataclasses/ | |
| """ | |
| path_str = str(src_path).replace("\\", "/") | |
| lib_marker = "/Lib/" | |
| if lib_marker in path_str: | |
| lib_path = parse_lib_path(src_path) | |
| if lib_path.stem == "__init__": | |
| lib_name = lib_path.parent.name | |
| elif lib_path.suffix == ".py": | |
| lib_name = lib_path.stem | |
| else: | |
| lib_name = lib_path.name | |
| prefix = path_str[: path_str.index(lib_marker)] | |
| # Try directory first, then file | |
| dir_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}/") | |
| if dir_path.exists(): | |
| return dir_path | |
| file_path = pathlib.Path(f"{prefix}/Lib/test/test_{lib_name}.py") | |
| if file_path.exists(): | |
| return file_path | |
| # Default to directory (caller will handle non-existence) | |
| return dir_path | |
| else: | |
| # Path starts with Lib/ - extract name directly | |
| if src_path.stem == "__init__": | |
| lib_name = src_path.parent.name | |
| elif src_path.suffix == ".py": | |
| lib_name = src_path.stem | |
| else: | |
| lib_name = src_path.name | |
| # Try directory first, then file | |
| dir_path = pathlib.Path(f"Lib/test/test_{lib_name}/") | |
| if dir_path.exists(): | |
| return dir_path | |
| file_path = pathlib.Path(f"Lib/test/test_{lib_name}.py") | |
| if file_path.exists(): | |
| return file_path | |
| return dir_path |
🤖 Prompt for AI Agents
In `@scripts/update_lib/path.py` around lines 42 - 78, The lib_to_test_path
function mishandles package __init__.py names (producing test___init__); update
both branches (the "/Lib/" and "Lib/" branches in lib_to_test_path) to detect
when src_path.name == "__init__.py" (or lib_path.stem == "__init__") and in that
case set lib_name to the parent directory's name (e.g., lib_path.parent.name or
src_path.parent.name) instead of "__init__"; keep the existing directory-first
then file-fallback logic and continue returning the constructed pathlib.Path
values as before.
| """Tests for migrate.py - file migration operations.""" | ||
|
|
||
| import pathlib | ||
| import tempfile | ||
| import unittest | ||
|
|
||
| from update_lib.migrate import ( | ||
| patch_directory, | ||
| patch_file, | ||
| patch_single_content, | ||
| ) | ||
| from update_lib.patch_spec import COMMENT | ||
|
|
||
|
|
||
| class TestPatchSingleContent(unittest.TestCase): | ||
| """Tests for patch_single_content function.""" | ||
|
|
||
| def test_patch_with_no_existing_file(self): | ||
| """Test patching when lib file doesn't exist.""" | ||
| with tempfile.TemporaryDirectory() as tmpdir: | ||
| tmpdir = pathlib.Path(tmpdir) | ||
|
|
||
| # Create source file | ||
| src_path = tmpdir / "src.py" | ||
| src_path.write_text("""import unittest | ||
| class TestFoo(unittest.TestCase): | ||
| def test_one(self): | ||
| pass | ||
| """) | ||
|
|
||
| # Non-existent lib path | ||
| lib_path = tmpdir / "lib.py" | ||
|
|
||
| result = patch_single_content(src_path, lib_path) | ||
|
|
||
| # Should return source content unchanged | ||
| self.assertIn("def test_one(self):", result) | ||
| self.assertNotIn(COMMENT, result) | ||
|
|
||
| def test_patch_with_existing_patches(self): | ||
| """Test patching preserves existing patches.""" | ||
| with tempfile.TemporaryDirectory() as tmpdir: | ||
| tmpdir = pathlib.Path(tmpdir) | ||
|
|
||
| # Create source file (new version) | ||
| src_path = tmpdir / "src.py" | ||
| src_path.write_text("""import unittest | ||
| class TestFoo(unittest.TestCase): | ||
| def test_one(self): | ||
| pass | ||
| def test_two(self): | ||
| pass | ||
| """) | ||
|
|
||
| # Create lib file with existing patch | ||
| lib_path = tmpdir / "lib.py" | ||
| lib_path.write_text(f"""import unittest | ||
| class TestFoo(unittest.TestCase): | ||
| # {COMMENT} | ||
| @unittest.expectedFailure | ||
| def test_one(self): | ||
| pass | ||
| """) | ||
|
|
||
| result = patch_single_content(src_path, lib_path) | ||
|
|
||
| # Should have patch on test_one | ||
| self.assertIn("@unittest.expectedFailure", result) | ||
| self.assertIn(COMMENT, result) | ||
| # Should have test_two from source | ||
| self.assertIn("def test_two(self):", result) | ||
|
|
||
|
|
||
| class TestPatchFile(unittest.TestCase): | ||
| """Tests for patch_file function.""" | ||
|
|
||
| def test_patch_file_creates_output(self): | ||
| """Test that patch_file writes output file.""" | ||
| with tempfile.TemporaryDirectory() as tmpdir: | ||
| tmpdir = pathlib.Path(tmpdir) | ||
|
|
||
| # Create source file | ||
| src_path = tmpdir / "src.py" | ||
| src_path.write_text("""import unittest | ||
| class TestFoo(unittest.TestCase): | ||
| def test_one(self): | ||
| pass | ||
| """) | ||
|
|
||
| # Output path | ||
| lib_path = tmpdir / "Lib" / "test.py" | ||
|
|
||
| patch_file(src_path, lib_path, verbose=False) | ||
|
|
||
| # File should exist | ||
| self.assertTrue(lib_path.exists()) | ||
| content = lib_path.read_text() | ||
| self.assertIn("def test_one(self):", content) | ||
|
|
||
| def test_patch_file_preserves_patches(self): | ||
| """Test that patch_file preserves existing patches.""" | ||
| with tempfile.TemporaryDirectory() as tmpdir: | ||
| tmpdir = pathlib.Path(tmpdir) | ||
|
|
||
| # Create source file | ||
| src_path = tmpdir / "src.py" | ||
| src_path.write_text("""import unittest | ||
| class TestFoo(unittest.TestCase): | ||
| def test_one(self): | ||
| pass | ||
| """) | ||
|
|
||
| # Create existing lib file with patch | ||
| lib_path = tmpdir / "lib.py" | ||
| lib_path.write_text(f"""import unittest | ||
| class TestFoo(unittest.TestCase): | ||
| # {COMMENT} | ||
| @unittest.expectedFailure | ||
| def test_one(self): | ||
| pass | ||
| """) | ||
|
|
||
| patch_file(src_path, lib_path, verbose=False) | ||
|
|
||
| content = lib_path.read_text() | ||
| self.assertIn("@unittest.expectedFailure", content) | ||
|
|
||
|
|
||
| class TestPatchDirectory(unittest.TestCase): | ||
| """Tests for patch_directory function.""" | ||
|
|
||
| def test_patch_directory_all_files(self): | ||
| """Test that patch_directory processes all .py files.""" | ||
| with tempfile.TemporaryDirectory() as tmpdir: | ||
| tmpdir = pathlib.Path(tmpdir) | ||
|
|
||
| # Create source directory with files | ||
| src_dir = tmpdir / "src" | ||
| src_dir.mkdir() | ||
| (src_dir / "test_a.py").write_text("# test_a") | ||
| (src_dir / "test_b.py").write_text("# test_b") | ||
| (src_dir / "subdir").mkdir() | ||
| (src_dir / "subdir" / "test_c.py").write_text("# test_c") | ||
|
|
||
| # Output directory | ||
| lib_dir = tmpdir / "lib" | ||
|
|
||
| patch_directory(src_dir, lib_dir, verbose=False) | ||
|
|
||
| # All files should exist | ||
| self.assertTrue((lib_dir / "test_a.py").exists()) | ||
| self.assertTrue((lib_dir / "test_b.py").exists()) | ||
| self.assertTrue((lib_dir / "subdir" / "test_c.py").exists()) | ||
|
|
||
| def test_patch_directory_preserves_patches(self): | ||
| """Test that patch_directory preserves patches in existing files.""" | ||
| with tempfile.TemporaryDirectory() as tmpdir: | ||
| tmpdir = pathlib.Path(tmpdir) | ||
|
|
||
| # Create source directory | ||
| src_dir = tmpdir / "src" | ||
| src_dir.mkdir() | ||
| (src_dir / "test_a.py").write_text("""import unittest | ||
| class TestA(unittest.TestCase): | ||
| def test_one(self): | ||
| pass | ||
| """) | ||
|
|
||
| # Create lib directory with patched file | ||
| lib_dir = tmpdir / "lib" | ||
| lib_dir.mkdir() | ||
| (lib_dir / "test_a.py").write_text(f"""import unittest | ||
| class TestA(unittest.TestCase): | ||
| # {COMMENT} | ||
| @unittest.expectedFailure | ||
| def test_one(self): | ||
| pass | ||
| """) | ||
|
|
||
| patch_directory(src_dir, lib_dir, verbose=False) | ||
|
|
||
| content = (lib_dir / "test_a.py").read_text() | ||
| self.assertIn("@unittest.expectedFailure", content) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| unittest.main() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Test additions violate the test-file modification policy.
This file adds new test logic and assertions, but the repo guideline for **/*test*.py restricts changes to only adding/removing @unittest.expectedFailure with TODO comments. Please move these checks to a non-test module or seek a policy exception. As per coding guidelines, ...
🤖 Prompt for AI Agents
In `@scripts/update_lib/tests/test_migrate.py` around lines 1 - 196, The new tests
in TestPatchSingleContent, TestPatchFile, and TestPatchDirectory add general
test logic and assertions which violates the test-file modification policy; to
fix, move these checks out of a "*test*.py" file into a non-test helper/module
(e.g., create a new module like update_lib/tests_helpers.py) or replace the
changes so that the only modifications inside the test files are adding/removing
`@unittest.expectedFailure` with TODO comments (references: patch_single_content,
patch_file, patch_directory, and the test classes
TestPatchSingleContent/TestPatchFile/TestPatchDirectory), then update the
import/usage accordingly or request a policy exception.
cc @moreal
I am going to rewrite this tools more organized library update toolkit, and remain rooms for further automation.
scripts/update_libwill be main entrance.lib_updater
->
scripts/update_lib patches(--from --to)->
scripts/update_lib migrate(--quick-upgrade)auto_mark_test
->
scripts/update_lib auto_markAlso
scripts/update_lib quickis added to upgrade a whole package at once, copy the library, remove unexected sucesss new add expectedFailure for newly added failing tests but not about regression.The format also will be aligned to ruff format and PEP8 to fix #6701.
Keeping draft to test for a few more libraries. Please feedback.
Summary by CodeRabbit
Documentation
update_libworkflow with improved step-by-step guidanceRefactor
auto_mark_test.pytoolupdate_libutility offering enhanced test marker preservation and improved failure detection workflow✏️ Tip: You can customize this high-level summary in your review settings.