Core: Fix partition field IDs in table replacement#2906
Merged
rdblue merged 5 commits intoAug 3, 2021
Conversation
rdblue
force-pushed
the
fix-partition-field-ids-in-replace-table
branch
from
52e4a9f to
15fa6a2
Compare
rdblue
force-pushed
the
fix-partition-field-ids-in-replace-table
branch
2 times, most recently
from
9f0b54b to
fdba13b
Compare
…alent fields when refreshing spec
Co-authored-by: Jun He <[email protected]>
rdblue
force-pushed
the
fix-partition-field-ids-in-replace-table
branch
from
fdba13b to
f996bff
Compare
jackye1995
approved these changes
Aug 2, 2021
Contributor
jackye1995
left a comment
jackye1995
left a comment
There was a problem hiding this comment.
Looks good to me. The failed Hive tests seem to be wrong, so we can just fix those tests.
rdblue
force-pushed
the
fix-partition-field-ids-in-replace-table
branch
from
f996bff to
15c233e
Compare
Contributor
Author
|
Thanks for reviewing, @jackye1995! I'm updating the tests as this catches more. It's just a few places that were changed by the behavior fix. |
jun-he
approved these changes
Aug 3, 2021
Contributor
|
Late +1 from me too. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes partition field ID assignment for
REPLACE TABLEoperations. Table replacement callsTableMetadata.buildReplacementthat is responsible for merging the existing table's metadata with table metadata passed to theCREATE OR REPLACE TABLEDDL. When the schema and partition spec are created in this path, the field IDs from the table are not known, so the spec and schema's IDs are consistent but reuse IDs that are already assigned in the table. Schema field IDs are already reassigned, but partition field IDs were conflicting up until now.This reassigns partition field IDs by first reusing existing IDs in the table and then by assigning new IDs.
Addresses this comment: #2089 (comment)
This is based on #2284. I've added Jun as a co-author in the commit. Thanks @jun-he!
Closes #2284.