Skip to content

HIVE-29413: Avoid code duplication by updating getPartCols method for iceberg tables #6413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ramitg254 wants to merge 20 commits into
base: master
Choose a base branch
from
Open

HIVE-29413: Avoid code duplication by updating getPartCols method for iceberg tables #6413

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
2ce98d5
HIVE-29413: Avoid code duplication by updating getPartCols method for…
ramitg254 Apr 8, 2026
af646e7
commit-2
ramitg254 Apr 9, 2026
d5f101c
corrected bucket-map-join test
ramitg254 Apr 9, 2026
b248996
corrected update statements
ramitg254 Apr 10, 2026
70567f8
corrected load, partition evolution tests
ramitg254 Apr 11, 2026
44d22f8
refractored
ramitg254 Apr 11, 2026
24d37a8
addressed sonar issues
ramitg254 Apr 12, 2026
1dca652
updated table api and usage
ramitg254 Apr 26, 2026
1b9709f
introduced index optimization
ramitg254 May 24, 2026
aad2a39
corrected implementation
ramitg254 May 24, 2026
27c649e
updated describe implementation and outputs
ramitg254 May 24, 2026
11c279d
updated api and test
ramitg254 May 25, 2026
5b4e476
updated update implementation
ramitg254 May 25, 2026
63c2ac2
updated partition pruning and query rewriting
ramitg254 May 25, 2026
2742cf8
changes related to metatable
ramitg254 May 25, 2026
24542d1
corrected alter and semantic analyzer implementation
ramitg254 May 26, 2026
14467d8
updated merge implementation and test output
ramitg254 May 26, 2026
ebb9e01
updated ctas create and tests output
ramitg254 May 26, 2026
18243b1
updated stats autogather and test output
ramitg254 May 26, 2026
fef6436
updated getPartitionKeys
ramitg254 May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions ...g/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1536,10 +1536,10 @@ public List<FieldSchema> acidSelectColumns(org.apache.hadoop.hive.ql.metadata.Ta
case DELETE ->
// TODO: make it configurable whether we want to include the table columns in the select query.
// It might make delete writes faster if we don't have to write out the row object
ListUtils.union(ACID_VIRTUAL_COLS_AS_FIELD_SCHEMA, table.getCols());
ListUtils.union(ACID_VIRTUAL_COLS_AS_FIELD_SCHEMA, table.getStorageSchemaCols());
Copy link
Copy Markdown
Member

@deniskuzZ deniskuzZ May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drop this getStorageSchemaCols. union getCols and getPartitionCols

case UPDATE -> shouldOverwrite(table, operation) ?
ACID_VIRTUAL_COLS_AS_FIELD_SCHEMA :
ListUtils.union(ACID_VIRTUAL_COLS_AS_FIELD_SCHEMA, table.getCols());
ListUtils.union(ACID_VIRTUAL_COLS_AS_FIELD_SCHEMA, table.getStorageSchemaCols());
case MERGE -> ACID_VIRTUAL_COLS_AS_FIELD_SCHEMA;
default -> ImmutableList.of();
};
Expand Down Expand Up @@ -2130,6 +2130,9 @@ public List<Partition> getPartitions(org.apache.hadoop.hive.ql.metadata.Table hm
}

public boolean isPartitioned(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
if (hmsTable.getMetaTable() != null) {
return false;
}
if (!hmsTable.getTTable().isSetId()) {
return false;
}
Expand Down Expand Up @@ -2275,6 +2278,9 @@ public boolean canPerformMetadataDelete(org.apache.hadoop.hive.ql.metadata.Table

@Override
public List<FieldSchema> getPartitionKeys(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
if (hmsTable.getMetaTable() != null) {
return Collections.emptyList();
}
if (!hmsTable.getTTable().isSetId()) {
return Collections.emptyList();
}
Expand Down
14 changes: 10 additions & 4 deletions ...handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -946,7 +946,8 @@ public void testCreateTableWithoutColumnComments() {
@Test
public void testCreatePartitionedTableWithColumnComments() {
TableIdentifier identifier = TableIdentifier.of("default", "partitioned_with_comment_table");
String[] expectedDoc = new String[] {"int column", "string column", null, "partition column", null};
String[] expectedDoc = new String[] {"int column", "string column", null, "partition column",
"Transform: identity"};
shell.executeStatement("CREATE EXTERNAL TABLE partitioned_with_comment_table (" +
"t_int INT COMMENT 'int column', " +
"t_string STRING COMMENT 'string column', " +
Expand All @@ -959,13 +960,18 @@ public void testCreatePartitionedTableWithColumnComments() {

List<Object[]> rows = shell.executeStatement("DESCRIBE default.partitioned_with_comment_table");
List<Types.NestedField> columns = icebergTable.schema().columns();
List<String> partitionColumns = List.of("t_string_3", "t_string_4");
// The partition transform information and partition information is 6 extra lines, and 4 more line for the columns
Assert.assertEquals(columns.size() + 10, rows.size());
for (int i = 0; i < columns.size(); i++) {
Types.NestedField field = columns.get(i);
Assert.assertArrayEquals(new Object[] {field.name(), HiveSchemaUtil.convert(field.type()).getTypeName(),
field.doc() != null ? field.doc() : ""}, rows.get(i));
Assert.assertEquals(expectedDoc[i], field.doc());
String fieldDoc = field.doc();
if (fieldDoc == null && partitionColumns.contains(field.name())) {
fieldDoc = "Transform: identity";
}
Assert.assertArrayEquals(new Object[]{field.name(), HiveSchemaUtil.convert(field.type()).getTypeName(),
fieldDoc != null ? fieldDoc : ""}, rows.get(i));
Assert.assertEquals(expectedDoc[i], fieldDoc);
}
}

Expand Down
15 changes: 0 additions & 15 deletions iceberg/iceberg-handler/src/test/results/positive/alter_multi_part_table_to_iceberg.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,8 +177,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_orc
# col_name data_type comment
a int
b string
c string

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -453,8 +451,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_parquet
# col_name data_type comment
a int
b string
c string

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -729,8 +725,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_avro
# col_name data_type comment
a int
b string
c string

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -1066,9 +1060,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_orc_mixed
# col_name data_type comment
a int
b double
c int
d string

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -1513,9 +1504,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_parquet_mixed
# col_name data_type comment
a int
b double
c int
d string

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -1960,9 +1948,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_avro_mixed
# col_name data_type comment
a int
b double
c int
d string

# Partition Information
# col_name data_type comment
Expand Down
5 changes: 0 additions & 5 deletions iceberg/iceberg-handler/src/test/results/positive/alter_part_table_to_iceberg.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_orc
# col_name data_type comment
a int
b string

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -415,7 +414,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_parquet
# col_name data_type comment
a int
b string

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -770,7 +768,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_parquet_int
# col_name data_type comment
a int
b int

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -1125,7 +1122,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_parquet_double
# col_name data_type comment
a int
b double

# Partition Information
# col_name data_type comment
Expand Down Expand Up @@ -1426,7 +1422,6 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_avro
# col_name data_type comment
a int
b string

# Partition Information
# col_name data_type comment
Expand Down
16 changes: 8 additions & 8 deletions iceberg/iceberg-handler/src/test/results/positive/bucket_map_join_9.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,21 +34,21 @@ Stage-0
limit:-1
Stage-1
Map 1 vectorized
File Output Operator [FS_53]
Map Join Operator [MAPJOIN_52] (rows=2 width=530)
BucketMapJoin:true,Conds:SEL_51._col1, _col2=RS_49._col1, _col2(Inner),Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
File Output Operator [FS_23]
Map Join Operator [MAPJOIN_22] (rows=2 width=530)
BucketMapJoin:true,Conds:SEL_21._col1, _col2=RS_19._col1, _col2(Inner),Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
<-Map 2 [CUSTOM_EDGE] vectorized
MULTICAST [RS_49]
MULTICAST [RS_19]
PartitionCols:_col2, _col1
Select Operator [SEL_48] (rows=2 width=265)
Select Operator [SEL_18] (rows=2 width=265)
Output:["_col0","_col1","_col2"]
Filter Operator [FIL_47] (rows=2 width=265)
Filter Operator [FIL_17] (rows=2 width=265)
predicate:(id is not null and part is not null)
TableScan [TS_3] (rows=2 width=265)
default@tbl,tbl2,Tbl:COMPLETE,Col:COMPLETE,Output:["foid","part","id"]
<-Select Operator [SEL_51] (rows=2 width=265)
<-Select Operator [SEL_21] (rows=2 width=265)
Output:["_col0","_col1","_col2"]
Filter Operator [FIL_50] (rows=2 width=265)
Filter Operator [FIL_20] (rows=2 width=265)
predicate:(id is not null and part is not null)
TableScan [TS_0] (rows=2 width=265)
default@tbl,tbl,Tbl:COMPLETE,Col:COMPLETE,Grouping Num Buckets:100,Grouping Partition Columns:["id","part"],Output:["foid","part","id"]
Expand Down
4 changes: 2 additions & 2 deletions iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ PREHOOK: Input: default@tbl_ice_puffin
POSTHOOK: query: desc formatted tbl_ice_puffin C
POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_ice_puffin
col_name C
col_name c
data_type int
min 52
max 56
Expand All @@ -358,7 +358,7 @@ max_col_len
num_trues
num_falses
bit_vector HL
comment
comment Transform: identity
COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}}
PREHOOK: query: EXPLAIN select count(*) from src_ice t1 join tbl_ice_puffin t2 on (t1.a = t2.a)
PREHOOK: type: QUERY
Expand Down
2 changes: 0 additions & 2 deletions iceberg/iceberg-handler/src/test/results/positive/ctas_iceberg_partitioned_orc.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,8 +273,6 @@ POSTHOOK: query: describe formatted tbl_ice
POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_ice
# col_name data_type comment
a int
b string
c int

# Partition Information
Expand Down
Loading
Loading