create: do not archive a root inside an earlier directory root twice, fixes #10393 - #10394
Open
ThomasWaldmann wants to merge 3 commits into
Open
ThomasWaldmann wants to merge 3 commits into
ThomasWaldmann wants to merge 3 commits into
Conversation
fixes borgbackup#10388 The duplicate root protection (borgbackup#5603) identified a finished root by (st_ino, st_dev) only. Hard links are different paths with the same inode, so they were taken for duplicates of the first root and silently not archived: - borg create arch input/file1 input/file2 (file2 being a hard link of file1) - borg create arch input/file1 input (hard links of file1 inside input) - borg create arch input/link input/target (followed symlink to a file) skip_key() now identifies directories by inode (as before) and all other fs objects by inode and path, so only the very same path is skipped. Co-Authored-By: Claude Fable 5.1 <[email protected]>
…ixes borgbackup#10393 The duplicate root protection (borgbackup#5603) only recorded finished recursion roots in skip_inodes, so a root that was already archived while recursing into a directory root given before it was archived again. _rec_walk now also records a path it is done with if that path is one of the recursion roots. This needs memory proportional to the count of roots only. A root that was not archived by the earlier walk (it did not recurse into a tagged directory or into another filesystem) is still archived. Co-Authored-By: Claude Fable 5.1 <[email protected]>
… refs borgbackup#10393 If an archive has the same hard link item twice (e.g. the same path was given twice via --paths-from-stdin) and that path is the remembered link target of its hard link group, extract removed the already extracted file and then failed to hard link the path to itself, so the file was missing afterwards. Such a duplicate item is skipped now, its path was extracted already. Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10394 +/- ##
==========================================
- Coverage 88.19% 88.16% -0.04%
==========================================
Files 103 103
Lines 18822 18831 +9
Branches 2919 2922 +3
==========================================
+ Hits 16600 16602 +2
- Misses 1548 1552 +4
- Partials 674 677 +3 ☔ View full report in Codecov by Harness. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #10393.
Stacked on #10389 (it needs
skip_key()from there): the first commit of this PR is the commit of #10389, only the last 2 commits are new. Will rebase after #10389 is merged.create: do not archive a root inside an earlier directory root twice
borg create arch input input/file3archivedinput/file3twice: the duplicate root protection (#5603) only recorded finished recursion roots inskip_inodes, not what was found while recursing, so only "child root first, then parent directory" was protected._rec_walknow also records a path it is done with, if that path is one of the (normalized) recursion roots. Memory usage is proportional to the count of roots, not to the count of files.A plain "is this root below an earlier directory root" check would be wrong: a root that the earlier walk did not reach (inside a directory tagged via
--exclude-if-present, on another filesystem with--one-file-system) must still be archived. That works here, because only paths that the walk really was done with get recorded. A directory that was not recursed into (other filesystem) is not recorded, so it still gets recursed into when it is given as a root.extract: do not remove the link target of a duplicate hard link item
If the duplicate item was the first extracted member of a hard link group,
borg extractremoved the already extracted file and then failed withlink: [Errno 2] No such file or directory: path -> path- the file was missing afterwards. There are other ways to get such an archive (e.g. the same path given twice via--paths-from-stdin, which the new test uses) and there are existing archives, so extract now skips such a duplicate item.Tests
test_create_dir_root_and_roots_inside_it: hard-linked file roots (all 3 names, so it does not depend on the platform's inode order) and a directory root inside an earlier directory root: every item once, extraction gives link count 3.test_create_dir_root_and_not_archived_roots_inside_it: roots inside a tagged directory of an earlier root still get archived.test_extract_duplicate_hardlink_item.All 3 fail without the respective fix. The #5603 tests and the tests of #10389 still pass; full test suite run locally on macOS (without the mount tests).
Not changed: a tagged directory (
--exclude-if-presentwith--keep-exclude-tags) that is given again as a root after its parent still gets its directory and tag file items duplicated, and--paths-from-stdin/--paths-from-commanddo not de-duplicate paths at all.🤖 Generated with Claude Code