HIVE-29578: Iceberg: add support for logical views by difin · Pull Request #6449 · apache/hive

difin · 2026-04-23T23:38:39Z

What changes were proposed in this pull request?

Added support for Iceberg logical views in Hive for both HMS and REST catalogs.

Why are the changes needed?

To support Iceberg logical views. This can be especially useful for REST Catalog clients.

Does this PR introduce any user-facing change?

Yes, new HQL syntax:

create view <view_name> as select * from <src_tbl> stored by iceberg;

How was this patch tested?

Created new and updated exiting unit and integration tests with Iceberg logical views test cases.

kasakrisz · 2026-04-28T12:42:28Z

+    result.setLastAccessTime(nowSec);
+    result.setRetention(Integer.MAX_VALUE);
+
+    boolean hiveEngineEnabled = false;


What is hiveEngineEnabled and why is it false?

hiveEngineEnabled switches how HiveOperationsBase.storageDescriptor fills the Storage Desacriptor: with HiveIcebergInputFormat / HiveIcebergOutputFormat / HiveIcebergSerDe when true, or the usual placeholder FileInputFormat / FileOutputFormat / LazySimpleSerDe when false.

Why it’s false in toHiveView:

This path materializes an HMS VIRTUAL_VIEW for REST catalog that expose Iceberg view metadata through the HMS API. That row isn’t meant to drive a Hive table scan the way a real Iceberg table commit does; execution still comes from the view definition / catalog, not from wiring Iceberg MR formats on the stub. HiveViewOperations does the same thing (hiveEngineEnabled = false).

So we keep a minimal SD consistent with normal virtual views and avoid implying this HMS object is an Iceberg-backed table for the Hive engine. For tables, HiveTableOperations still turns engine integration on/off via metadata + ConfigProperties.ENGINE_HIVE_ENABLED where that actually matters.

kasakrisz · 2026-04-28T13:11:40Z

+create view v_ice as select * from src_ice stored by iceberg;
+
+select * from v_ice;
+


Could you please add

logical view which does some transformation on it's base table and query from it?

create views when the schema is specified and not specified.

logical view which does some transformation on it's base table and query from it?

This is not supported by Hive itself:

update v_ice set last_name = last_name + 'a' fname=iceberg_native_view.q See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/ for specific test cases logs. org.apache.hadoop.hive.ql.parse.SemanticException: You cannot update or delete records in a view at org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.validateTargetTable(RewriteSemanticAnalyzer.java:265) at org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyze(RewriteSemanticAnalyzer.java:84) at org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:73) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:358) at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224) at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:109) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:499) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:451) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:415) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:409) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:234) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:203) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:129) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:430) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358) at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:790) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:760) at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:115) at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:139)

create views when the schema is specified and not specified.

Done

logical view which does some transformation on it's base table

Sorry I mean something like

select first_name || last_name from ... where <some filter condition>

because

select * from table;

as a view definition is a kind of edge case. It is ok for testing but not a typical use-case.

kasakrisz · 2026-04-28T13:28:44Z

+        break;
+      }
+    }
+    boolean icebergNativeView = validateOptionalViewStorageClause(storageClause);


Please do not hardcode anything like Iceberg into compiler code. The compiler is independent from the storage handler. I'm aware that we already hove lots of code which violates this principal and it already causes lots of troubles.

Fixed - moved all Iceberg-specific code into HiveIcebergStorageHandler and kept generic interfaces in the Compiler.

kasakrisz · 2026-04-28T13:29:39Z

  private static final long serialVersionUID = 1L;

+  /** HMS table property set when the view is declared with {@code STORED BY ICEBERG} (native Iceberg view). */
+  public static final String ICEBERG_NATIVE_VIEW_PROPERTY = "hive.iceberg.native.view";


Please remove this from here.

kasakrisz · 2026-04-28T13:29:44Z

  private final boolean ifNotExists;
  private final boolean replace;
  private final List<FieldSchema> partitionColumns;
+  private final boolean icebergNativeView;


Please remove this from here.

kasakrisz · 2026-04-28T13:29:58Z

+  @Explain(displayName = "iceberg native view", displayOnlyOnTrue = true)
+  public boolean isIcebergNativeView() {
+    return icebergNativeView;
+  }


Please remove this from here.

kasakrisz · 2026-05-11T12:35:55Z

+  VIEW_STORAGE_HANDLER_UNSUPPORTED(10448, "CREATE VIEW only supports STORED BY ICEBERG for native "
+      + "Iceberg views; unsupported storage clause: {0}", true),


Please rephrase this error message. Remove STORED BY and let Iceberg be a parameter.

kasakrisz · 2026-05-11T13:02:34Z

+        TableName.fromString(
+            view.name(), MetaStoreUtils.getDefaultCatalog(conf), Warehouse.DEFAULT_DATABASE_NAME);
+    result.setCatName(tableName.getCat());
+    result.setDbName(tableName.getDb());


What happens when a custom db is specified?

create view my_db.myview as...

this method is only called when reading from a view using a REST Catalog - it is called from HiveRESTCatalogClient.getTable(GetTableRequest tableRequest)

Sorry, my question was inspired by the fact that the database name is hardcoded, which makes it impossible to handle views in databases other than 'default'.

this method is only called when reading from a view using a REST Catalog - it is called from HiveRESTCatalogClient.getTable(GetTableRequest tableRequest)

Does this imply that I am unable to reference views from any database other than default ?

select * from my_db.my_view;

[difin] - there was no option to reply on this comment, see reply below.

kasakrisz · 2026-05-11T13:12:03Z

+    return conf;
+  }
+
+  private HiveCatalog verifyCatalog() {


Why is this called verify? Based on the implementation of this method it is more like loading a catalog.
Could you please share some background of the use case of this method.

Didn't notice, fixed.

kasakrisz · 2026-05-11T13:18:58Z

+                conf, DB, VIEW, cols, "select 2 as id", null, null, true, false))
+        .isTrue();
+
+    assertThat(verifyCatalog().viewExists(TableIdentifier.of(DB, VIEW))).isTrue();


I think it worth checking if the view definition is actually altered.

kasakrisz · 2026-05-11T13:22:21Z

+  ('fn7','ln7', 2);
+
+----------------------------------------------------------------
+-- Iceberg native view via TBLPROPERTIES before AS


Does before AS adds any value in this comment? AFAIK the grammar allows this way only.

Removed "before as" part.

kasakrisz · 2026-05-11T16:02:52Z

  iceberg_create_locally_zordered_table.q,\
  iceberg_merge_delete_files.q,\
  iceberg_merge_files.q,\
+  iceberg_native_view.q,\


Is there any LLAP specific in view DDLs ? If not; can this test run by the default Iceberg driver?

No, this test is not LLAP-specific. Changed to run it by TestIcebergCliDriver instead of TestIcebergLLAPLocalCliDriver.

kasakrisz · 2026-05-11T16:04:24Z

+  @Test
+  public void testParseCreateViewTblpropertiesViewFormatIceberg() throws Exception {
+    ASTNode tree = parseDriver.parse(
+        "create view v1 tblproperties ('view-format'='iceberg') as select * from t", null).getTree();
+    assertTrue(tree.dump(), tree.toStringTree().contains("tok_createview"));
+    assertTrue(tree.dump(), tree.toStringTree().contains("tok_tableproperties"));
+    assertTrue(tree.dump(), tree.toStringTree().contains("view-format"));
+    assertTrue(tree.dump(), tree.toStringTree().contains("iceberg"));
+  }
+


This test class is about the default keyword testing. Could you please move this test to the one which is about view creation. Please create a new one of not exists. Since this grammar is not Iceberg specific it can be a generic one.

Done, moved to a new class for testing view parsing.

kasakrisz · 2026-05-11T16:10:05Z

+    if (explicitViewFormat) {
+      throw new SemanticException(ErrorMsg.VIEW_STORAGE_HANDLER_UNSUPPORTED.getMsg(
+          "Native view metadata is not supported for storage handler: " + handlerClass));
+    }


We have the knowledge about explicit view format at the caller.
Could you please move this check to the caller method.

Done, moved to the caller.

kasakrisz · 2026-05-12T12:11:29Z

+    if (desc.usesNativeViewCatalog()) {
+      executeNativeCatalogView();
+      return 0;
+    }


I think this decision should be moved to BaseHiveIcebergMetaHook or HiveIcebergMetaHook.preCreateTable like we do in case of Iceberg tables.

Please set the StorageHandlerClass in CreateViewOperation.createViewObject to the newly created ql.metadata.Table object because later this object is passed to HMS and the meta hook.

if (desc.usesNativeViewCatalog()) { storageFormat.setStorageHandler(desc.getNativeViewStorageHandlerClass()); view.setProperty( org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE, desc.getNativeViewStorageHandlerClass()); }

I'm not a big fan of this meta hook solution but this is whet we have in case of Iceberg tables and IMHO it is better to be consistent.

Probably create or replace and if not exists doesn't have to be handled separately in case of Iceberg views of you make this change.

Please add some tests for create or replace and if not exists to the q test iceberg_native_view.q

Done - moved view creation to the hook and added test cases with create or replace and if not exists to iceberg_native_view.q

kasakrisz · 2026-05-12T12:15:06Z

  }

+  /**
+   * Resolves {@code STORED BY identifier} for CREATE VIEW (short names such as {@code ICEBERG} or an FQCN).


Was STORED BY left here intentionally?

No, it was a leftover. Fixed.

kasakrisz · 2026-05-12T12:24:25Z

+   * Keys should be removed when {@linkplain #clearNativeViewHmsTableProperties(Map)} is invoked for the same
+   * handler class recorded under {@link Constants.NATIVE_VIEW_STORAGE_HANDLER_CLASS_PARAM}.
+   */
+  default Map<String, String> getNativeViewHmsTableProperties() {


Can we call this simply getViewProperties? I would like to understand what native and hms mean in this context.

AFAIK, a native object is one that doesn't have a storage handler. To me, the word native in these method names is misleading. Could you please elaborate on this a bit?"

getNativeViewHmsTableProperties
clearNativeViewHmsTableProperties
createOrReplaceNativeView
supportsNativeViewCatalog

I agree that the word "native" was misleading outside of Iceberg handler.
Removed/rephrased the "iceberg native views" names that were used outside of the iceberg handler.

sonarqubecloud · 2026-05-19T23:11:42Z

Quality Gate passed

Issues
14 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

kasakrisz · 2026-05-21T09:31:56Z

@difin
Question about the jira summary (Open
HIVE-29578
Why are these referred to as 'native views' instead of 'logical views'?

Please don't squash the commits when addressing review comments. It makes review more difficult.

kasakrisz · 2026-05-13T09:48:46Z

+    assertThat(
+            NativeIcebergViewSupport.createOrReplaceNativeView(
+                conf, DB, VIEW, cols, "select 2 as id", null, null, false, true))
+        .isFalse();


Could you assert that the definition query hasn't changed?

kasakrisz · 2026-05-13T10:35:08Z

+--! Native Iceberg catalog view with default Iceberg storage handler (REST table with catalog in props)
+-----------------------------------------------------------------------------------------------
+
+create view ice_v4 as select * from ice_orc2;
+select * from ice_v4;
+desc formatted ice_v4;
+drop view ice_v4;


Thanks for the clarification. Is it mean that the view ice_v4 is created in the ice01 catalog too ?

Does creating views in one catalog referencing base tables from another catalog?

[difin] - there was no option to reply on this comment, see reply below.

kasakrisz · 2026-05-13T10:47:19Z

+        TableName.fromString(
+            view.name(), MetaStoreUtils.getDefaultCatalog(conf), Warehouse.DEFAULT_DATABASE_NAME);
+    result.setCatName(tableName.getCat());
+    result.setDbName(tableName.getDb());


Sorry, my question was inspired by the fact that the database name is hardcoded, which makes it impossible to handle views in databases other than 'default'.

this method is only called when reading from a view using a REST Catalog - it is called from HiveRESTCatalogClient.getTable(GetTableRequest tableRequest)

Does this imply that I am unable to reference views from any database other than default ?

select * from my_db.my_view;

[difin] - there was no option to reply on this comment, see reply below.

kasakrisz · 2026-05-20T10:32:06Z


  @Override
  public void commitCreateTable(org.apache.hadoop.hive.metastore.api.Table hmsTable) {
+    if (BaseHiveIcebergMetaHook.isNativeIcebergLogicalView(hmsTable)) {


nit.: this doesn't have to be qualified because isNativeIcebergLogicalView is defined in the super class.

Thanks, didn't notice.
Fixed.

kasakrisz · 2026-05-20T10:33:54Z

+        boolean replace =
+            Boolean.parseBoolean(
+                SessionStateUtil.getProperty(conf, Constants.EXTERNAL_LOGICAL_VIEW_DDL_REPLACE).orElse("false"));
+        boolean ifNotExists =
+            Boolean.parseBoolean(
+                SessionStateUtil.getProperty(conf, Constants.EXTERNAL_LOGICAL_VIEW_CREATE_IF_NOT_EXISTS)
+                    .orElse("false"));


nit.: .orElse("false") goes to new line or not. Both are good but please be consistent.

I put the second orElse it on a new line because of the 120 characters limit.

Moved the 1st one on a new line too for the consistency.

kasakrisz · 2026-05-21T08:44:26Z

+      if (!desc.usesStorageHandler()) {
+        // External logical view is replaced with a native Hive view
+        clearStorageHandlerProp(oldview);
+      }


Could you please share some repro steps for this use-case?

Added this test case to iceberg_native_logical_view.q:
v_ice before replacement is Iceberg-native logical view that is replaced with Hive-native logical view:

----------------------------------------------------------------------------------------- -- Replace Iceberg logical view with a Hive-native logical view ----------------------------------------------------------------------------------------- create or replace view v_ice as select first_name from src_ice where dept_id = 2; select * from v_ice; desc formatted v_ice; drop view v_ice;

kasakrisz · 2026-05-21T08:50:44Z

+    SessionStateUtil.addResource(context.getConf(), Constants.EXTERNAL_LOGICAL_VIEW_DDL_REPLACE,
+        Boolean.toString(replace));
+    SessionStateUtil.addResource(context.getConf(), Constants.EXTERNAL_LOGICAL_VIEW_CREATE_IF_NOT_EXISTS,
+        Boolean.toString(ifNotExists));
+  }


Why does these has to be added to session state?

IIUC replace and ifNotExists is handled here in CreateViewOperation

The values of replace and ifNotExists are added to the session state because they are needed when creating a view using Iceberg API, which happens in IcebergNativeLogicalViewSupport.createOrReplaceNativeView(). This method is called from HiveIcebergMetaHook (when using HMS) or from HiveRESTCatalogClient (when using REST Catalog).

difin · 2026-05-27T18:58:49Z

@difin Question about the jira summary (Open HIVE-29578 Why are these referred to as 'native views' instead of 'logical views'?

Please don't squash the commits when addressing review comments. It makes review more difficult.

Renamed Iceberg Native Views to Iceberg Logical Views in the PR description and JIRA.
I used to squash commits to see easier what was changed.
I won't squash commits going forward.

difin · 2026-05-27T22:42:26Z

#6449 (comment)

Thanks for the clarification. Is it mean that the view ice_v4 is created in the ice01 catalog too ?

Yes, ice_v4 is created in the ice01 catalog because the catalog that is used in this test is only ice01 and Iceberg logical views are created in the current catalog.

Does creating views in one catalog referencing base tables from another catalog?
I haven't implemented that functionality in this PR.
I didn't find any examples of federated cross-catalog queries in Hive, I don't think it's currently supported.

difin · 2026-05-27T22:50:11Z

#6449 (comment)

Does this imply that I am unable to reference views from any database other than default ?

No. That line does not restrict views to the default database.
TableName.fromString uses the third argument only as a fallback when the name has no ..
Iceberg views created through Hive are stored with a namespace/database, so view.name() is always database.viewname and the fallback is never used.

The same pattern exists in toHiveTable() for tables - the method above toHiveView().

Iceberg logical views are tested with non-default database, see for example iceberg_native_logical_view.q, it creates views in ice_native_view_db database.

sonarqubecloud · 2026-05-28T01:34:33Z

Quality Gate passed

Issues
14 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added tests pending tests unstable and removed tests pending labels Apr 23, 2026

difin force-pushed the iceberg_native_views branch from 4fdad42 to 252c608 Compare April 24, 2026 23:06

asf-ci-hive added tests pending and removed tests unstable labels Apr 24, 2026

difin force-pushed the iceberg_native_views branch from 252c608 to e10eba5 Compare April 24, 2026 23:31

difin marked this pull request as ready for review April 24, 2026 23:31

difin changed the title ~~HIVE-29578: Iceberg: support for Iceberg native views~~ HIVE-29578: Iceberg: support native views Apr 24, 2026

asf-ci-hive added tests failed and removed tests pending labels Apr 24, 2026

difin changed the title ~~HIVE-29578: Iceberg: support native views~~ HIVE-29578: Iceberg: add support for native views Apr 24, 2026

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending labels Apr 25, 2026

difin force-pushed the iceberg_native_views branch from e10eba5 to 96fa476 Compare April 25, 2026 20:46

asf-ci-hive added tests pending and removed tests unstable labels Apr 25, 2026

difin requested review from deniskuzZ and kasakrisz April 25, 2026 20:46

asf-ci-hive added tests unstable and removed tests pending labels Apr 25, 2026

difin force-pushed the iceberg_native_views branch from 96fa476 to 114412a Compare April 26, 2026 15:12

asf-ci-hive added tests pending tests passed and removed tests unstable tests pending labels Apr 26, 2026

kasakrisz requested changes Apr 27, 2026

View reviewed changes

Comment thread iceberg/iceberg-handler/src/test/queries/positive/iceberg_native_view.q Outdated

Comment thread ql/src/java/org/apache/hadoop/hive/ql/ddl/view/create/CreateViewAnalyzer.java Outdated

kasakrisz requested changes Apr 28, 2026

View reviewed changes

asf-ci-hive added tests pending tests passed and removed tests pending labels May 9, 2026

kasakrisz requested changes May 12, 2026

View reviewed changes

kasakrisz reviewed May 12, 2026

View reviewed changes

difin force-pushed the iceberg_native_views branch from 1886216 to 4743d79 Compare May 19, 2026 22:09

asf-ci-hive added tests pending and removed tests passed labels May 19, 2026

asf-ci-hive added tests passed and removed tests pending labels May 20, 2026

kasakrisz reviewed May 21, 2026

View reviewed changes

difin force-pushed the iceberg_native_views branch from 4743d79 to 2d5f0d0 Compare May 27, 2026 17:41

asf-ci-hive added tests pending and removed tests passed labels May 27, 2026

difin changed the title ~~HIVE-29578: Iceberg: add support for native views~~ HIVE-29578: Iceberg: add support for logical views May 27, 2026

HIVE-29578: Iceberg: support for Iceberg logical views.

db655f2

asf-ci-hive added tests passed and removed tests pending labels May 27, 2026

code review comments May 27

f3ddf97

difin force-pushed the iceberg_native_views branch from 2d5f0d0 to f3ddf97 Compare May 28, 2026 00:36

asf-ci-hive added tests pending and removed tests passed labels May 28, 2026

asf-ci-hive added tests passed and removed tests pending labels May 28, 2026

		create view v_ice as select * from src_ice stored by iceberg;

		select * from v_ice;

		VIEW_STORAGE_HANDLER_UNSUPPORTED(10448, "CREATE VIEW only supports STORED BY ICEBERG for native "
		+ "Iceberg views; unsupported storage clause: {0}", true),

Conversation

difin commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

difin May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

difin May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

difin May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kasakrisz May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kasakrisz May 13, 2026 • edited by difin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

difin commented Apr 23, 2026 •

edited

Loading

difin May 5, 2026 •

edited

Loading

difin May 5, 2026 •

edited

Loading

difin May 5, 2026 •

edited

Loading

kasakrisz May 11, 2026 •

edited

Loading

kasakrisz May 13, 2026 •

edited by difin

Loading

difin May 19, 2026 •

edited

Loading

difin May 19, 2026 •

edited

Loading