AWS: Retain Glue Catalog table description after updating Iceberg table#10199
Conversation
|
Awesome, thanks for contributing this @aajisaka! Manually did this change work? For adding some tests, if the unit test logic in this module is difficult. I'd suggest leveraging this modules integration test workflow to update an created table with the glue client directly. |
|
Thank you for your suggestion @geruh I'll test this patch manually and update the class. |
|
I ran the integration test using my AWS account and the |
|
Also, I built the jars and manually tested on AWS Glue as a Spark runtime. |
|
Hi @geruh would you review this PR? |
|
Hi @geruh @amogh-jahagirdar would you review the latest change? |
amogh-jahagirdar
left a comment
There was a problem hiding this comment.
Overall looks great @aajisaka, just some nits and then I think it's good to merge!
|
Awesome work @aajisaka, looks good to me! |
|
Thank you @geruh @amogh-jahagirdar |
Problem
In AWS Glue Catalog, user can set arbitrary description to the table and its columns via Web UI or even from Amazon Athena. However, the descriptions will be removed after upgrading the table.
What this PR do
This patch is to keep the original table description when calling
UpdateTableAPI. Keeping column description is hard and I think we can open a different PR to do this.Testing
Manually tested. Also, added an integration test and it passed on my environment.