Repository navigation
Conversation
This patch adds the context parameter "sqlTimeCondition", which adds a time filter to all tables. The filter is ANDed with any other filter that the user may have provided. The goal of this feature is to enable UIs and APIs to accept user-written SQL along with having a separate control for time. This patch also fixes two pre-existing bugs that were discovered while writing tests for this feature: 1) In some cases, WHERE on time + ORDER BY non-time over a join could lose the time filter. This is fixed by passing scan.getIntervals() to WindowOperatorQuery in DruidQuery#toScanAndSortQuery. testLeftJoinWithTimeFilterAndNonTimeOrderBy is a regression test. 2) The SQL planner would produce "intervals" rather than a __time filter for certain time filters over a join of a left-hand subquery. This would cause execution to fail, because execution assumes that "intervals" are only used for regular tables. This is fixed by adjusting ExecutionVertexExplorer to avoid traversing through noncollapsible subqueries, making such a query no longer viewed by the SQL planner as table-based. The regression test is in testJoinWithSubqueryOnLeftAndOuterTimeFilter.
| .build() | ||
| ), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("dim2"), makeColumnExpression("j0.d0")), |
| .build() | ||
| ), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("dim2"), makeColumnExpression("j0.d0")), |
| ), | ||
| new LookupDataSource("lookyloo"), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("dim2"), makeColumnExpression("j0.k")), |
| ), | ||
| new LookupDataSource("lookyloo"), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("dim2"), makeColumnExpression("j0.k")), |
| new TableDataSource(CalciteTests.DATASOURCE1), | ||
| new LookupDataSource("lookyloo"), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("dim2"), makeColumnExpression("j0.k")), |
| .build() | ||
| ), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("dim1"), makeColumnExpression("j0.dim1")), |
| .build() | ||
| ), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("dim4"), makeColumnExpression("j0.dim4")), |
| .build() | ||
| ), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("dim4"), makeColumnExpression("j0.dim4")), |
| ), | ||
| new LookupDataSource("lookyloo"), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("d0"), makeColumnExpression("j0.k")), |
| ), | ||
| new LookupDataSource("lookyloo"), | ||
| "j0.", | ||
| equalsCondition(makeColumnExpression("d0"), makeColumnExpression("j0.k")), |
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
sqlTimeCondition is not honored for policy-protected tables in native SQL, so supported queries can silently read outside the requested intervals. Extend interval eligibility to policy-wrapped tables and add a regression test before merging.
Reviewed 24 of 24 changed files; findings in 1 file.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
| // Time filters within the datasource narrow the query intervals too, which lets the broker prune segments. | ||
| final Filtration queryFiltration = | ||
| Filtration.create(filter, getBaseIntervals(dataSource, rowSignature)).optimize(rowSignature); | ||
| Filtration.create(filter, getBaseIntervals(dataSource, timeCondition, rowSignature)).optimize(rowSignature); |
There was a problem hiding this comment.
[P1] Context is ignored for policy-wrapped tables
Finding: Native SQL applies policies before this method, representing protected tables as RestrictedDataSource. ExecutionVertex.isTableBased() returns false for that wrapper, so the early return bypasses the new timeCondition interval and the native engine reads outside the requested window.
Suggestion: Treat RestrictedDataSource as table-based for interval filtering, and add a restricted-table regression test.
| |`sqlReverseLookup`|Whether to consider the [reverse-lookup rewrite](lookups.md#reverse-lookup) of the `LOOKUP` function during SQL planning.<br /><br />Druid reverses calls to `LOOKUP` only when the number of matching keys is lower than both `inSubQueryThreshold` and `sqlReverseLookupThreshold`.|`true`| | ||
| |`sqlReverseLookupThreshold`|Maximum size of `IN` filter to create when applying a [reverse-lookup rewrite](lookups.md#reverse-lookup). If a `LOOKUP` call matches more keys than the specified threshold, it remains unchanged.<br /><br />If `inSubQueryThreshold` is lower than `sqlReverseLookupThreshold`, Druid uses `inSubQueryThreshold` threshold instead.|10000| | ||
| |`sqlPullUpLookup`|Whether to consider the [pull-up rewrite](lookups.md#pull-up) of the `LOOKUP` function during SQL planning.|`true`| | ||
| |`sqlTimeCondition`|Time condition to apply to all tables that the query reads. Provide either a comma-separated string of ISO8601 intervals, such as `"2000-01-01/2000-02-01,2001-01-01/2001-02-01"`, or an array of ISO8601 interval strings. An empty array matches nothing. An empty string is an error.<br /><br />Druid ANDs this condition with any other condition you provide on the `__time` column. Intervals without an explicit offset are in UTC; `sqlTimeZone` doesn't affect them.<br /><br />You can't use this parameter with `INSERT`, `REPLACE`, or [catalog DDL](../development/extensions-core/catalog.md#sql-ddl) statements.|none| |
There was a problem hiding this comment.
Could this also be defined as a SQL server configuration so it applies to all queries cluster-wide, similar to druid.sql.planner.requireTimeCondition? Or is that already available through druid.query.default.context.sqlTimeCondition?
| |`sqlReverseLookup`|Whether to consider the [reverse-lookup rewrite](lookups.md#reverse-lookup) of the `LOOKUP` function during SQL planning.<br /><br />Druid reverses calls to `LOOKUP` only when the number of matching keys is lower than both `inSubQueryThreshold` and `sqlReverseLookupThreshold`.|`true`| | ||
| |`sqlReverseLookupThreshold`|Maximum size of `IN` filter to create when applying a [reverse-lookup rewrite](lookups.md#reverse-lookup). If a `LOOKUP` call matches more keys than the specified threshold, it remains unchanged.<br /><br />If `inSubQueryThreshold` is lower than `sqlReverseLookupThreshold`, Druid uses `inSubQueryThreshold` threshold instead.|10000| | ||
| |`sqlPullUpLookup`|Whether to consider the [pull-up rewrite](lookups.md#pull-up) of the `LOOKUP` function during SQL planning.|`true`| | ||
| |`sqlTimeCondition`|Time condition to apply to all tables that the query reads. Provide either a comma-separated string of ISO8601 intervals, such as `"2000-01-01/2000-02-01,2001-01-01/2001-02-01"`, or an array of ISO8601 interval strings. An empty array matches nothing. An empty string is an error.<br /><br />Druid ANDs this condition with any other condition you provide on the `__time` column. Intervals without an explicit offset are in UTC; `sqlTimeZone` doesn't affect them.<br /><br />You can't use this parameter with `INSERT`, `REPLACE`, or [catalog DDL](../development/extensions-core/catalog.md#sql-ddl) statements.|none| |
There was a problem hiding this comment.
I also wonder how this feature would interplay with the existing requireTimeCondition feature - fixes being made in #20441.
@gianm @NishthaShah thoughts?
|
The conflict is from the #20578 which migrated the sql context parameterer to the new |
| |`sqlReverseLookup`|Whether to consider the [reverse-lookup rewrite](lookups.md#reverse-lookup) of the `LOOKUP` function during SQL planning.<br /><br />Druid reverses calls to `LOOKUP` only when the number of matching keys is lower than both `inSubQueryThreshold` and `sqlReverseLookupThreshold`.|`true`| | ||
| |`sqlReverseLookupThreshold`|Maximum size of `IN` filter to create when applying a [reverse-lookup rewrite](lookups.md#reverse-lookup). If a `LOOKUP` call matches more keys than the specified threshold, it remains unchanged.<br /><br />If `inSubQueryThreshold` is lower than `sqlReverseLookupThreshold`, Druid uses `inSubQueryThreshold` threshold instead.|10000| | ||
| |`sqlPullUpLookup`|Whether to consider the [pull-up rewrite](lookups.md#pull-up) of the `LOOKUP` function during SQL planning.|`true`| | ||
| |`sqlTimeCondition`|Time condition to apply to all tables that the query reads. Provide either a comma-separated string of ISO8601 intervals, such as `"2000-01-01/2000-02-01,2001-01-01/2001-02-01"`, or an array of ISO8601 interval strings. An empty array matches nothing. An empty string is an error.<br /><br />Druid ANDs this condition with any other condition you provide on the `__time` column. Intervals without an explicit offset are in UTC; `sqlTimeZone` doesn't affect them.<br /><br />You can't use this parameter with `INSERT`, `REPLACE`, or [catalog DDL](../development/extensions-core/catalog.md#sql-ddl) statements.|none| |
There was a problem hiding this comment.
I also want to know if the supporting of ARRAY is needed.
Because under current design, every query context parameter can be set via the SET statement, however, the SET statement requires a literal , we CAN'T do sth like this now:
SET sqlTimeCondition = ['a/b', 'c/d'];The only case that current simple String type can't express is the empty array case. But I'm not sure under which case we need to support this case,.
If this is a must case, I think maybe we can following existing SEGMENT_SORT_ORDER which accepts JSON array inside string, like:
SET sqlTimeCondition = ''; -- error
SET sqlTimeCondition = '[]'; -- match nothing
SET sqlTimeCondition = '["2000-01-01/2000-02-01","2001-01-01/2001-02-01"]';
SET sqlTimeCondition = '2000-01-01/2000-02-01,2001-01-01/2001-02-01';| static Pair<DataSource, Filtration> getFiltration( | ||
| DataSource dataSource, | ||
| DimFilter filter, | ||
| @Nullable List<Interval> timeCondition, |
There was a problem hiding this comment.
instead of a new argument to a lot of things; couldn't this be folded into the filter and then let the system recognize it like in other cases?
....or how about instead of passing it around transform all datasources of the queries into filtereddatasource?
I also wonder what would be the logical equiv of this:
- I think this could be viewed that all tables have logical column like:
__phantom_time; which applies to the subtree - this condition is applies to the
__phantom_timecolumn - the
__phantom_timeconditions should be pushed down to all leaf filters; regardless the intermediate nodes (rewrite shuttle) and turned into__time - this
__phantom_timecolumn is conceptual - if it would have no value exposing it could be skipped entirely; the shuttle would be enough
| * be rid of time filters. | ||
| * | ||
| * @param timeCondition intervals to AND with the filtration intervals if the datasource is table-based, from | ||
| * {@link PlannerContext#getTimeCondition()}. Must be sorted and non-overlapping. Null means |
There was a problem hiding this comment.
I think the source of a detail like this should not matter at this point
This patch adds the context parameter "sqlTimeCondition", which adds a time filter to all tables. The filter is ANDed with any other filter that the user may have provided. The goal of this feature is to enable UIs and APIs to accept user-written SQL along with having a separate control for time.
This patch also fixes two pre-existing bugs that were discovered while writing tests for this feature:
In some cases, WHERE on time + ORDER BY non-time over a join could
lose the time filter. This is fixed by passing scan.getIntervals()
to WindowOperatorQuery in DruidQuery#toScanAndSortQuery.
testLeftJoinWithTimeFilterAndNonTimeOrderBy is a regression test.
The SQL planner would produce "intervals" rather than a __time filter
for certain time filters over a join of a left-hand subquery. This
would cause execution to fail, because execution assumes that
"intervals" are only used for regular tables. This is fixed by
adjusting ExecutionVertexExplorer to avoid traversing through
noncollapsible subqueries, making such a query no longer viewed
by the SQL planner as table-based. The regression test is in
testJoinWithSubqueryOnLeftAndOuterTimeFilter.