Skip to content

Core: Include delete files in 'files' and add 'data_files' table#4336

Merged
szehon-ho merged 3 commits into
apache:masterfrom
szehon-ho:data_file_table
Mar 22, 2022
Merged

Core: Include delete files in 'files' and add 'data_files' table#4336
szehon-ho merged 3 commits into
apache:masterfrom
szehon-ho:data_file_table

Conversation

@szehon-ho
Copy link
Copy Markdown
Member

As discussed in #4139, this is the second step after adding 'delete_files' table, to synchronize the similar metadata tables to handle the delete file concept.

In this PR:

  • 'files' table now returns data_files and delete_files
  • 'data_files' table is newly added to return only data_files.

This might be hard to split into two prs, as it's doing a hot-swap migrating the 'files' table class (called DataFilesTable) to support new table 'data_files'.

@szehon-ho
Copy link
Copy Markdown
Member Author

@aokolnychyi @rdblue @RussellSpitzer fyi when anyone has a cycle, thanks

@szehon-ho szehon-ho changed the title Include delete files in 'files' and add 'data_files' table Core: Include delete files in 'files' and add 'data_files' table Mar 15, 2022
@szehon-ho szehon-ho force-pushed the data_file_table branch 2 times, most recently from 9c6b07b to fd93320 Compare March 16, 2022 17:38
@aokolnychyi
Copy link
Copy Markdown
Contributor

Let me take a look today.

Comment thread core/src/test/java/org/apache/iceberg/TestMetadataTableFilters.java
}
}

private int expectedCount(int partitions) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: expected count of what?

@szehon-ho szehon-ho merged commit 85e64cd into apache:master Mar 22, 2022
@szehon-ho
Copy link
Copy Markdown
Member Author

Merged, thanks @aokolnychyi for the review

sunchao pushed a commit to sunchao/iceberg that referenced this pull request May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants