branch-4.1: [fix](fe) Allocate a fresh StatementContext per EXECUTE to prevent FE OOM in long-lived prepared statements - #67256 - #67926
Open
starocean999 wants to merge 3 commits into
Open
Conversation
… OOM in long-lived prepared statements (apache#67256) Problem Summary: A prepared statement lives as long as its connection. The `PreparedStatementContext` kept in `ConnectContext.preparedStatementContextMap` retains a single `StatementContext` and reuses the same object across every `EXECUTE` for the whole connection lifetime. Because one object is reused across executions, its per-statement state keeps accumulating: bound tables (`tables`, `oneLevelTables`, `mtmvRelatedTables`, `insertTargetTables`, `viewInfos`), CTE maps, statistics (`relationIdToStatisticsMap`, `tableIdMapping`), MV/partition rewrite state (`mvCanRewritePartitionsMap`, `tmpPlanForMvRewrite`, `materializationRewrittenSuccessSet`), MVCC snapshots, connector write schemas, placeholder bindings (`idToPlaceholderRealExpr`), etc. On long-lived connections with a high number of `EXECUTE`s, these maps only grow and are never released until the connection closes, which can OOM the FE. **Root cause:** the `StatementContext` stored in `PreparedStatementContext` was treated as a permanent per-prepared-statement object and reused, so state that should be per-execution lived as long as the connection. **Fix:** instead of reusing (and clearing in place) the same `StatementContext`, allocate a brand-new context on every `EXECUTE` and carry over only the state that must survive between executions: - **ID generator positions** — so ids generated during this execution never collide with ids already present in the cached analyzed plan from `PREPARE`; - **placeholder real expressions** bound by the protocol layer for this `EXECUTE` (`idToPlaceholderRealExpr`) — this is the piece that prevents the apache#63920 parameter-mismatch regression; - the **placeholder → comparison-slot registry** (`idToComparisonSlot`) used by the short-circuit fast path; - the **placeholder list**; - the **short-circuit / nondeterministic flags** that gate the short-circuit fast path before any re-planning. After the swap, the previous context becomes unreachable and is promptly GC'd, so memory no longer grows with the number of executions. The cached analyzed plan and the point-query (short-circuit) cache live on `PrepareCommand` and `PreparedStatementContext` respectively, so they keep being reused across executions. **Changes:** - `IdGenerator`: add `getCurrentId()` so a fresh context can continue the id generators from the previous one. - `StatementContext`: add `createNextExecuteContext()` which allocates the fresh context and copies over the cross-execution state above. - `PreparedStatementContext`: add `nextStatementContext()` which swaps in the fresh context so the old one is released. - `ExecuteCommand`: `run()` now uses the fresh per-execution context (and the now-redundant in-place `resetConnectorStatementScope()`/`resetMvccSnapshots()` calls are removed since a fresh context starts empty by construction). - Unit tests updated to assert the fresh-context behavior (`ExecuteCommandTest`, `ConnectorStatementScopeTest`). None
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick #67256
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)