Skip to content

REST catalog doesn't return old history if we execute CREATE OR REPLACE TABLE statement #11777

@ebyhr

Description

@ebyhr

Apache Iceberg version

1.7.1 (latest release)

Query engine

Trino

Please describe the bug 🐞

trinodb/trino#24312 is trying to use org.apache.iceberg.MetadataTableUtils#createMetadataTableInstance method for history metadata table. Only REST catalog doesn't return the past history if we execute CREATE OR REPLACE TABLE statement.

trino:tpch> TABLE "region$history";
          made_current_at           |     snapshot_id     | parent_id | is_current_ancestor
------------------------------------+---------------------+-----------+---------------------
 2024-12-13 16:26:02.387 Asia/Tokyo | 4753270524621689028 |      NULL | true
(1 row)

trino:tpch> CREATE OR REPLACE TABLE region AS SELECT * FROM orders;
CREATE TABLE: 15000 rows

trino:tpch> TABLE "region$history";
          made_current_at           |     snapshot_id     | parent_id | is_current_ancestor
------------------------------------+---------------------+-----------+---------------------
 2024-12-13 16:27:51.756 Asia/Tokyo | 7793069171159305791 |      NULL | true
(1 row)

trino:tpch> SHOW CREATE TABLE region;
                                                                          Create Table
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
 CREATE TABLE iceberg.tpch.region (
    orderkey bigint,
    custkey bigint,
    orderstatus varchar,
    totalprice double,
    orderdate date,
    orderpriority varchar,
    clerk varchar,
    shippriority integer,
    comment varchar
 )
 WITH (
    format = 'PARQUET',
    format_version = 2,
    location = '/var/folders/9s/_zwn4r_n2_9bp0krllp1pl3c0000gp/T/a6a26fac-b6be-4c9c-b05b-cbe7ec894b2f/iceberg_data/tpch/region-4f41a42ee7f84d3093d85649bc557e46'
 )
(1 row)

region-4f41a42ee7f84d3093d85649bc557e46.zip

The following request contains RemoveSnapshotRef that clears existing snapshot logs.

TableMetadata.Builder metadataBuilder = TableMetadata.buildFrom(base);
request.updates().forEach(update -> update.applyTo(metadataBuilder));
TableMetadata updated = metadataBuilder.build();

https://apache-iceberg.slack.com/archives/C03LG1D563F/p1733195549342209

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions