Spark: RewriteTablePath: filter content files by snapshotId#12885
Conversation
dramaticlly
force-pushed
the
dr-filter-snapshot-id
branch
from
4f257c7 to
e61e572
Compare
dramaticlly
changed the title
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
There was a problem hiding this comment.
Looks mostly good to me.
Note: can improve the pr description a bit to give the context, something like, "RewriteTablePath incremental mode limits rewrite/copy plan to those between two snapshots. However, the returned copy plan ignores this limitation and includes all referenced files by these snapshots, even those not modified between the two snapshots. This leads to unnecessary copies".
…ing by snapshotId This help avoid repeated copy of data files when manifest rewrite happens
dramaticlly
force-pushed
the
dr-filter-snapshot-id
branch
from
e61e572 to
dda923c
Compare
|
Merged, thanks @dramaticlly ! |
|
Thanks @szehon-ho for the detailed review and merge! |
3 participants
RewriteTablePath incremental mode limits rewrite/copy plan to those between two snapshots. However, the returned copy plan ignores this limitation and includes all referenced files by these snapshots, even those not modified between the two snapshots. This leads to unnecessary copies. This patch allows of rewriting table path to use snapshots to filter both
added_snapshot_idin ManifestFilesnapshot_idin ManifestEntryIt help avoid repeated copy of same data files when manifest rewrite happens in between incremental copy. Manifest rewrite will only change added snapshot id for manifest, but data files remain the same and it shall not be copied over repeatedly.
If you can help take a look @flyrain @szehon-ho