[GLUTEN-12538][CORE] Preserve TIMESTAMP_NTZ literals - #13041
Draft
felipepessoto wants to merge 3 commits into
Draft
felipepessoto wants to merge 3 commits into
felipepessoto wants to merge 3 commits into
Conversation
|
Run Gluten Clickhouse CI on x86 |
Preserve NTZ logical types and microsecond values across JVM, Substrait and Velox literal conversion. Distinguish timestamp from timestamp_tz, retain complete types for null literals, and handle scalar vector-backed constants without a virtual-table container. Connect the existing JVM NTZ literal node to the literal factory and nested type handling. Include the NTZ type mapping required by independent literal and Values round trips, with dedicated native and Spark 4.1 regression coverage. Follow-up to apache#12967 (comment) Co-authored-by: Copilot <[email protected]>
felipepessoto
force-pushed
the
gluten-12538-timestamp-ntz-literals
branch
from
September 16, 2026 23:58
a518473 to
a18c8f2
Compare
|
Run Gluten Clickhouse CI on x86 |
Restore the original physical-kind encoding for non-timestamp nulls so DATE array operands remain compatible regardless of null position. Preserve full logical types for timestamp nulls and timestamp arrays, including nested arrays. Add DATE literal encoding and array round-trip regressions alongside timestamp-array null coverage. Co-authored-by: Copilot <[email protected]>
|
Run Gluten Clickhouse CI on x86 |
Reject expression-internal TIMESTAMP_NTZ on backends without NTZ support, including nested literals whose surrounding schemas hide the type. Preserve existing Velox behavior and cover backend capabilities, nested literal types, and native timezone independence. Co-authored-by: Copilot <[email protected]>
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Preserve
TIMESTAMP_NTZliteral types and values through the JVM/Substrait/Velox conversion paths. NTZ values use Substrait'stimestampliteral field, while regular timestamps continue to usetimestamp_tz.The patch carries logical types into reverse literal conversion and handles scalar vector-backed constants without treating them as virtual-table output. Logical-type-aware null encoding is limited to timestamps and timestamp arrays; other null types keep their previous encoding. It also connects the existing JVM NTZ literal node to scalar, collection, and struct literal construction.
This is a separate follow-up to the literal-conversion review comment on #12967. It is based on
mainand includes the NTZ type mapping needed by its own literal and Values round trips. It does not include the aggregation validator, shuffle, or Delta statistics changes from that PR.Related to #12538.
How was this patch tested?
Native conversion regressions check serialized literal fields, logical types, and exact microsecond values for scalar constants, vector-backed constants, typed nulls, Values plans, and array literals. Both regular timestamps and NTZ values are covered, including pre-epoch values. DATE scalar and array regressions protect the existing encoding, including comparisons of arrays with different leading-null positions.
The dedicated Spark 4.1
GlutenTimestampNtzLiteralSuitechecks JVM scalar/array/struct literal serialization and a non-foldabletimestampaddexpression with native-plan assertions. Using a row-dependent expression prevents Spark constant folding from hiding the native literal path.Existing related native conversion cases, formatting, and license-header checks were also run.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI 1.0.83