From a66652ba886a840fc9be8fbc2a181581f7adfdac Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Mon, 12 Oct 2026 10:39:19 +0300 Subject: [PATCH 1/3] docs(live-views): align live view docs with QuestDB 10.0.1 Audit the live view pages against the 10.0.1 release and fix the mismatches: - Statistical window functions, EMA and VWEMA are only accepted in anchored windows; bounded ROWS/RANGE frames fail with "incremental snapshot is not supported". ksum rejects RANGE frames, lag needs a bounded frame or an anchor, and VARCHAR, UUID, LONG256 and DECIMAL partition keys are rejected. Add a rolling standard deviation (Bollinger bands) recipe built from bounded avg(). - Freshness: interval-filtered reads, aggregates and ASOF JOIN see unflushed rows; LATEST ON, dedup-base views and replica views do not. - Live views have their own refresh worker pool (live.view.refresh.worker.*), not the mat view pool. Document the pool and cairo.live.view.checkpoint.purge.interval, and fix the growth budget, compaction threshold, retry budget and memory limit descriptions. - live_views(): add checkpoint_seal_failures, fix lag_seqtxn, last_processed_seqtxn, lag_micros and writer_stall_micros. - CREATE LIVE VIEW: fix the OWNED BY example (query must be parenthesized), document the designated timestamp projection rule and the remaining anchor rules, and rewrite the error tables with the verified messages. - Add an ALTER LIVE VIEW RESUME WAL page, add the UPDATE and REBASE WAL invalidation triggers, and correct the mat view and WAL apply worker count defaults. Co-Authored-By: Claude Opus 5.5 --- documentation/changelog.mdx | 18 ++ documentation/concepts/live-views.md | 285 +++++++++++++----- documentation/concepts/views.md | 3 +- documentation/configuration/cairo-engine.md | 7 +- documentation/configuration/live-views.md | 169 ++++++++--- .../configuration/materialized-views.md | 14 +- documentation/configuration/wal.md | 7 +- documentation/query/functions/meta.md | 24 +- .../query/sql/alter-live-view-resume-wal.md | 89 ++++++ documentation/query/sql/create-live-view.md | 161 +++++++--- documentation/query/sql/drop-live-view.md | 9 +- documentation/query/sql/show.md | 7 +- documentation/sidebars.js | 5 + 13 files changed, 617 insertions(+), 181 deletions(-) create mode 100644 documentation/query/sql/alter-live-view-resume-wal.md diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 833739bcce..29999e4d14 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -8,6 +8,24 @@ description: Recent updates and improvements to the QuestDB documentation. This page tracks significant updates to the QuestDB documentation. +## October 2026 + +### New + +- [ALTER LIVE VIEW RESUME WAL](/docs/query/sql/alter-live-view-resume-wal/) - New reference page for resuming a live view whose own WAL was suspended after an error, how to find suspended live views through `wal_tables()`, and why `SUSPEND WAL` does not pause a live view + +### Reference + +- Added the dedicated live view refresh worker pool settings [`live.view.refresh.worker.count`](/docs/configuration/live-views/#liveviewrefreshworkercount), `.affinity`, `.haltOnError`, `.nap.threshold`, `.sleep.threshold`, `.sleep.timeout`, and `.yield.threshold`, plus [`cairo.live.view.checkpoint.purge.interval`](/docs/configuration/live-views/#cairoliveviewcheckpointpurgeinterval). Live views no longer share the materialized view refresh pool +- Added the `checkpoint_seal_failures` column to [`live_views()`](/docs/query/functions/meta/#live_views) and corrected the `lag_seqtxn`, `last_processed_seqtxn`, `lag_micros`, and `writer_stall_micros` descriptions +- Corrected the defaults of [`mat.view.refresh.worker.count`](/docs/configuration/materialized-views/#matviewrefreshworkercount) and [`wal.apply.worker.count`](/docs/configuration/wal/#walapplyworkercount), and documented that `mat.view.refresh.worker.count=0` disables materialized view refresh + +### Updated + +- [Live views](/docs/concepts/live-views/) - Corrected the [supported window functions](/docs/concepts/live-views/#supported-window-functions) to match QuestDB 10.0.1: the statistical functions, EMA, and VWEMA run only in anchored windows, `ksum` does not support `RANGE` frames, `lag` needs a bounded frame or an anchor, and `VARCHAR`, `UUID`, `LONG256`, and `DECIMAL` partition keys are rejected. Added a [rolling standard deviation](/docs/concepts/live-views/#rolling-standard-deviation) recipe for Bollinger bands, rewrote [Freshness](/docs/concepts/live-views/#freshness) to reflect that interval-filtered reads, aggregates, and `ASOF JOIN` see unflushed rows while `LATEST ON` does not, and added the `UPDATE` and `REBASE WAL` invalidation triggers, base WAL retention, and replica freshness +- [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) - Fixed the `OWNED BY` example, which requires a parenthesized query, documented the designated-timestamp projection requirement and the remaining anchored-window rules, and expanded the error reference with verified messages +- [Live view configuration](/docs/configuration/live-views/) - Corrected the descriptions of the in-memory buffer growth budget, partition compaction threshold, flush retry budgets, and refresh memory limit + ## September 2026 ### QuestDB Enterprise Releases diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 143d7a66f9..82e4de5087 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -34,7 +34,7 @@ serve different query shapes: | Aspect | Live view | [Materialized view](/docs/concepts/materialized-views/) | | ------ | --------- | --------------------- | | Query shape | Window functions (`OVER`) | `SAMPLE BY` / time-based `GROUP BY` | -| Output cardinality | One row per base row | One row per time bucket | +| Output cardinality | One row per base row that passes `WHERE` | One row per time bucket | | Typical use | Running totals, moving averages, rankings | OHLC bars, downsampled summaries | | Base tables | A single WAL-backed table | One or more tables (JOINs allowed) | | Freshness / durability | `FLUSH EVERY`, `IN MEMORY` | `REFRESH` strategy | @@ -83,10 +83,8 @@ SELECT * FROM trades_ma; ``` The view updates incrementally as new rows arrive in `trades`. Each new trade -produces one output row carrying its moving average. A direct `SELECT` of the -full output rows sees data as soon as it is refreshed. Filtering a read to a -timestamp interval (for example `WHERE timestamp IN '$today'`) is served from the -disk tier and can trail by up to one `FLUSH EVERY` interval; see +produces one output row carrying its moving average. Queries see new rows as +soon as they are refreshed, without waiting for a flush; see [Freshness](#freshness). ## How live views work @@ -105,24 +103,33 @@ Reads combine both tiers. Recent rows are served from the in-memory tier and the older prefix from disk, so a query sees the freshest computed rows without waiting for a flush. +Views over [deduplicated](/docs/concepts/deduplication/) base tables, and every +live view on a QuestDB Enterprise read-only replica, work differently: they +refresh and flush together on the `FLUSH EVERY` cadence, with no in-memory rows +ahead of the disk tier. + ```questdb-sql title="Show the live view definition" SHOW CREATE LIVE VIEW trades_ma; ``` ### Freshness -Because refresh publishes to the in-memory tier ahead of flush, a direct -`SELECT` that reads the full output rows sees data as soon as it is refreshed. -This is independent of `FLUSH EVERY`, which is a durability and -write-amplification control, not a freshness control. +Because refresh publishes to the in-memory tier ahead of flush, queries see +refreshed rows without waiting for `FLUSH EVERY`, which is a durability and +write-amplification control, not a freshness control. This covers full-row +reads, projections and aggregates over the view, `WHERE` filters including +designated-timestamp intervals, and a live view used as the right-hand side of +an [`ASOF JOIN`](/docs/query/sql/asof-join/) or `LT JOIN`. -Some read shapes are served from the disk tier only and therefore trail by up to +The following reads are served from the disk tier only and can trail by up to one `FLUSH EVERY` interval: -- Reads that project or aggregate the view's columns rather than reading full - output rows -- Reads filtered to a timestamp interval -- A live view used as the right-hand side of an [`ASOF JOIN`](/docs/query/sql/asof-join/) +- [`LATEST ON`](/docs/query/sql/latest-on/) queries over the view +- Any read of a view over a deduplicated base table, or of a view on an + Enterprise read-only replica, because these views have no in-memory rows + ahead of disk +- Occasionally, a query that races a flush, which falls back to the disk tier + for that query only Keep `FLUSH EVERY` small (for example `1s`) so this lag stays negligible. @@ -137,31 +144,122 @@ interpret lag and detect a view that cannot keep up. ## Supported window functions Live views maintain window functions whose result can be computed incrementally -in a single forward pass and whose state can be checkpointed: - -- **Anchored ranking**: `row_number`, `rank`, `dense_rank` -- **Bounded or anchored aggregates**: `sum`, `avg`, `count`, `min`, `max`, - `ksum`, `first_value`, `last_value`, `nth_value` -- **Offset**: `lag` -- **Statistics**: `variance`, `stddev`, covariance, correlation, EMA, and VWEMA - -Stateful window functions must have a `PARTITION BY` clause. Both bounded `ROWS` -and bounded `RANGE` frames are supported. Ranking functions must use an anchored -window because an out-of-order row would otherwise change every later rank. +in a single forward pass and whose state can be checkpointed. Whether a function +is accepted depends on the window it runs over: a bounded frame such as +`ROWS 300 PRECEDING` or `RANGE '1' HOUR PRECEDING`, or an +[anchored window](#anchored-windows) that accumulates from a reset boundary. + +| Function | Bounded `ROWS` | Bounded `RANGE` | Anchored | +| -------- | -------------- | --------------- | -------- | +| `avg`, `count`, `first_value`, `max`, `min`, `nth_value`, `sum` | Yes | Yes | Yes | +| `ksum` | Yes | No | Yes | +| `last_value` | Yes | Yes | Yes, except with `IGNORE NULLS` | +| `lag` | Yes | Yes | Yes | +| `dense_rank`, `rank`, `row_number` | No | No | Yes | +| `corr`, `covar_pop`, `covar_samp` | No | No | Yes | +| `stddev`, `stddev_pop`, `stddev_samp`, `var_pop`, `var_samp`, `variance` | No | No | Yes | +| EMA and VWEMA, `avg(value, kind, param [, volume])` | No | No | Yes | + +The statistical functions, EMA, and VWEMA run only in an anchored window, so +they return a cumulative value since the last anchor boundary, such as a +session-to-date standard deviation. Over a bounded frame, the statistical +functions fail with +`incremental snapshot is not supported for this function yet`, and EMA and VWEMA +do not accept a frame clause at all. For a rolling standard deviation, see +[Rolling standard deviation](#rolling-standard-deviation). + +```questdb-sql title="Session-to-date volatility per symbol" +CREATE LIVE VIEW trades_session_stats +FLUSH EVERY 1s +START FROM NOW +AS +SELECT + timestamp, + symbol, + price, + avg(price) OVER w AS session_avg, + stddev_samp(price) OVER w AS session_stddev +FROM trades +WINDOW w AS ( + PARTITION BY symbol + ORDER BY timestamp + ANCHOR DAILY '00:00' +); +``` -String, `VARCHAR`, `BINARY`, `ARRAY`, and `SYMBOL` columns can appear as -pass-through output columns and as `count` arguments, but there are no string- -or array-valued window functions. +The following rules also apply: + +- **`PARTITION BY` is required.** Every window function needs a `PARTITION BY` + clause. The only exception is a `last_value` without `IGNORE NULLS` whose + frame ends at `CURRENT ROW`, which keeps no state. +- **Partition keys must be checkpointable.** `VARCHAR`, `UUID`, `LONG256`, + `DECIMAL`, `BINARY`, and array keys are rejected with + `incremental snapshot is not supported for this function yet`. In a bounded + window, cast a `VARCHAR` key to `SYMBOL` or `STRING`, for example + `PARTITION BY venue::symbol`. An anchored window must partition by base + columns directly. +- **Frames must not start at `UNBOUNDED PRECEDING`.** A window without a frame + clause, such as `OVER (PARTITION BY symbol ORDER BY timestamp)`, defaults to a + frame that starts at `UNBOUNDED PRECEDING`. Such windows are rejected unless + they are anchored. `last_value` without `IGNORE NULLS` is the only exception. +- **`RANGE` frames share one domain.** A `RANGE` frame must order by the + designated timestamp ascending, and all `RANGE` window functions in one view + must use the same `PARTITION BY` and `ORDER BY`. +- **`lag` needs a frame or an anchor.** `lag` ignores the frame, but a bare + `OVER (PARTITION BY ... ORDER BY ...)` counts as unbounded, so give it an + explicit bounded frame or an anchored window. In an anchored window, `lag` + returns `NULL` for the first row after each boundary. `lag(value, 0)` and + `lag` over a `DECIMAL` with precision above 18 are rejected. +- **No look-ahead or multi-pass functions.** `lead`, `percent_rank`, + `cume_dist`, and `ntile` are rejected. + +Columns of any type, including `VARCHAR`, `BINARY`, and arrays, can appear as +pass-through output columns. `count` also accepts `STRING`, `VARCHAR`, and +`SYMBOL` arguments, but not `BINARY` or arrays. Window function results can be +combined in projection expressions. + +### Rolling standard deviation + +Because the statistical functions run only in anchored windows, compute a +rolling standard deviation from bounded averages instead. The population +standard deviation is the square root of `avg(x * x) - avg(x) * avg(x)`, which a +live view can maintain over any bounded frame. This view keeps 20-trade +Bollinger bands per symbol: + +```questdb-sql title="Rolling 20-trade Bollinger bands" +CREATE LIVE VIEW trades_bollinger +FLUSH EVERY 1s +START FROM NOW +AS +SELECT + timestamp, + symbol, + price, + avg(price) OVER w AS sma20, + sqrt(greatest( + avg(price * price) OVER w - avg(price) OVER w * avg(price) OVER w, + 0 + )) AS stddev20 +FROM trades +WINDOW w AS (PARTITION BY symbol ORDER BY timestamp ROWS 19 PRECEDING); +``` -The following shapes cannot be maintained by an append-only incremental refresh -and are rejected at creation time: +```questdb-sql title="Read the bands" +SELECT + timestamp, + symbol, + price, + sma20 - 2 * stddev20 AS lower_band, + sma20 + 2 * stddev20 AS upper_band +FROM trades_bollinger +WHERE symbol = 'BTC-USDT' +LIMIT -10; +``` -- Multi-pass or look-ahead functions: `percent_rank`, `cume_dist`, `ntile`, - `lead` -- Stateful window functions without `PARTITION BY` -- Unanchored ranking functions -- Frames that start at `UNBOUNDED PRECEDING`, except stateless `last_value` - shapes +`greatest(..., 0)` guards against tiny negative values from floating-point +rounding when prices are flat. For the sample standard deviation that +`stddev_samp` returns, multiply the variance by `n / (n - 1)`, where `n` is +`count(price) OVER w`. ## Anchored windows @@ -192,9 +290,11 @@ time is interpreted in UTC; add an IANA time zone such as `'America/New_York'` when the boundary follows local civil time. An anchored window must be a named window, must partition by base-table columns, -must order by the designated timestamp ascending, and cannot use a bounded -frame. A live view supports at most one anchored window. An anchor expression -must be deterministic and return `TIMESTAMP`, `LONG`, or `INT`. +must order by the designated timestamp ascending, and must use the default +frame: an explicit `ROWS` frame or a bounded `RANGE` frame is rejected. A live +view supports at most one anchored window, and at least one window function must +reference it directly with `OVER w`. An anchor expression must be deterministic +and return `TIMESTAMP`, `LONG`, or `INT`. ## Start boundary and historical data @@ -206,8 +306,12 @@ lower bound for its rows: - `START FROM 'timestamp'` includes rows at or after the specified timestamp. The boundary is inclusive and is evaluated against the base table's designated -timestamp, not commit time. If qualifying rows already exist when the view is -created, QuestDB seeds them before switching to continuous refresh. +timestamp, not commit time. Rows below the boundary neither appear in the view +nor contribute to window state, and late out-of-order rows below it are dropped +and counted in the `o3_rejected_count` column of +[`live_views()`](/docs/query/functions/meta/#live_views). If qualifying rows +already exist when the view is created, QuestDB seeds them before switching to +continuous refresh. ```questdb-sql title="Include all existing history" CREATE LIVE VIEW trades_ma @@ -229,26 +333,39 @@ designated timestamps are at or above the resolved creation-time boundary. ## Base table lifecycle A live view is tied to a single WAL-backed base table and tracks the exact set of -base columns its query references: +base columns its query references, including columns used only in its `WHERE` +filter: - Changes to columns the view does not reference pass through transparently and the view keeps refreshing. - Dropping, renaming, or changing the type of a referenced column invalidates the view. - Renaming or dropping the base table invalidates the view. +- An [`UPDATE`](/docs/query/sql/update/) that changes base-table rows + invalidates the view. +- [`ALTER TABLE REBASE WAL`](/docs/query/sql/alter-table-rebase-wal/) on the + base table invalidates the view. - Exceeding [`cairo.live.view.refresh.memory.limit.bytes`](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) during a refresh invalidates the view. Before setting that limit, measure what a refresh needs as described in [Sizing a limit](/docs/configuration/cairo-engine/#sizing-a-limit). -- `DROP PARTITION`, `TRUNCATE`, and base TTL eviction freeze the already-emitted - rows and the view continues forward from where it was. +- Refresh failures that exhaust the + [`cairo.live.view.flush.retry.max`](/docs/configuration/live-views/#cairoliveviewflushretrymax) + retry budget invalidate the view. +- `DROP PARTITION`, `TRUNCATE`, and base TTL eviction keep the view valid. Rows + the view already emitted for the removed data stay in the view. Window state + is not reset, so later rows can still reflect values from removed rows, and a + later out-of-order commit can recompute emitted rows above the removed range + from the remaining base data. An invalidated view keeps serving its existing data and reports the reason in -[`live_views()`](/docs/query/functions/meta/#live_views). It stops refreshing. -Invalidation is permanent: reversing the schema change does not automatically -revalidate the view, and `ALTER LIVE VIEW ... RESUME WAL` only recovers a -suspended WAL writer. +[`live_views()`](/docs/query/functions/meta/#live_views), for example +`drop column operation [column=price]` or `update operation`. It stops +refreshing. Invalidation is permanent: reversing the schema change does not +automatically revalidate the view, and +[`ALTER LIVE VIEW RESUME WAL`](/docs/query/sql/alter-live-view-resume-wal/) only +recovers a suspended WAL writer. To recover, inspect `invalidation_reason`, repair the base-table schema or raise @@ -267,8 +384,8 @@ source history is no longer retained by the base table. Live views over [deduplicated](/docs/concepts/deduplication/) base tables are supported. A keep-last `UPSERT` replacement at an earlier timestamp is reflected -in the view. A view over a deduplicated base is one `FLUSH EVERY` cycle behind -rather than sub-cycle fresh, because its refresh is coupled to base apply. +in the view, and later rows are recomputed. Such a view has no in-memory rows +ahead of disk, so it trails the base table by up to one `FLUSH EVERY` cycle. ## Monitoring @@ -281,11 +398,14 @@ SELECT view_name, base_table_name, view_status, lag_seqtxn, lag_micros FROM live_views(); ``` -`lag_seqtxn` is the number of committed base-table WAL transactions beyond the +`lag_seqtxn` is the number of transactions applied to the base table beyond the view's durable processed watermark. It counts transactions, not rows: one transaction may contain one row or millions. A value of `0` means that the -durable tier is caught up. A temporary non-zero value is expected between -`FLUSH EVERY` cycles, especially when ingestion produces many small commits. +durable tier has caught up with the base table. A temporary non-zero value is +expected between `FLUSH EVERY` cycles, especially when ingestion produces many +small commits. Base commits that are not yet applied to the base table are not +counted, so when the base table itself lags or is suspended, check +[`wal_tables()`](/docs/query/functions/meta/#wal_tables) too. There is no universal acceptable non-zero value. Sample the metric over time and compare it with the view's normal flush-cycle baseline. A bounded sawtooth that @@ -297,17 +417,20 @@ up. flush-activity indicator, not the timestamp difference between base and view rows, and can continue growing while an idle view has `lag_seqtxn = 0`. -When lag grows persistently, check `view_status` and `writer_stall_micros` for a -failed or blocked flush. Also verify CPU and I/O capacity, reduce the number or -cost of maintained views, or increase the shared -[`mat.view.refresh.worker.count`](/docs/configuration/materialized-views/) -setting. +When lag grows persistently, check `view_status` and `invalidation_reason` for an +invalidated view, and `writer_stall_micros` for long-running queries that keep +the in-memory tier from publishing. Also verify CPU and I/O capacity, reduce the +number or cost of maintained views, or increase +[`live.view.refresh.worker.count`](/docs/configuration/live-views/#liveviewrefreshworkercount). +A single view is refreshed by one worker at a time, so more workers help only +when several views are busy. For out-of-order repair cost, compare `o3_replay_scan_rows`, `o3_resume_replay_rows`, and `o3_boundary_replay_rows`. The first counts base rows scanned; the other two split rows emitted by the resume-from-checkpoint and rebuild paths. `checkpoint_repair_plan` reports whether the view has a finite -`range`, `rows`, or `anchor` repair plan. The last disposition and denial +`range`, `rows`, or `anchor` repair plan, or `none` when every out-of-order row +below the head rebuilds the view's history. The last disposition and denial columns show which path actually ran and why a local rebuild was not selected. These counters reset on restart. @@ -315,7 +438,9 @@ The `checkpoint_timeline_*` columns describe the persistent timeline used for restart and out-of-order repair. In particular, `checkpoint_timeline_sharing_ratio` shows how effectively checkpoint state is shared, while `checkpoint_gc_lag_generations` and -`checkpoint_obsolete_segment_bytes` can expose delayed cleanup. See +`checkpoint_obsolete_segment_bytes` can expose delayed cleanup. A growing +`checkpoint_seal_failures` value means the view keeps serving correct results +while its restart-recovery state goes stale. See [`live_views()`](/docs/query/functions/meta/#live_views) for the complete catalog. @@ -333,7 +458,7 @@ outside it are rejected at creation time with a specific error: | Live view | Regular view | Yes | | Live view | Materialized view | No | | Live view | Live view | No | -| Regular view | Live view | No; a regular view is not a WAL table | +| Regular view | Live view | No; the view query must read a base table directly | | Materialized view | Live view | Yes | A full rebuild of a materialized view invalidates a live view that uses it as @@ -343,6 +468,8 @@ in [Base table lifecycle](#base-table-lifecycle). - **Single base table only.** No JOINs, subqueries, or CTEs in the view query. - **Explicit output columns only.** `SELECT *` and other wildcard projections are rejected because the output schema is fixed when the view is created. +- **Designated timestamp in the output.** The query must select the base + table's designated timestamp as a plain column under its own name. - **No pre-aggregation.** `SAMPLE BY` and `GROUP BY` are not allowed between the base table and the window functions. A view like "5-minute candles with a rolling VWAP" must pre-aggregate upstream. @@ -353,19 +480,26 @@ in [Base table lifecycle](#base-table-lifecycle). the `WHERE` filter, and window-function arguments. - **No TTL on the view.** Live-view disk growth is unbounded in this version. Size retention on the base table instead. +- **No direct writes.** `INSERT`, `UPDATE`, `TRUNCATE`, and structural + `ALTER` statements on a live view are rejected with `cannot modify live view`. ## Tradeoffs - **Storage grows with output.** The computed rows are stored on the live view's disk tier in addition to the base table's rows. For wide projections or long retention the view's footprint can exceed the base table. +- **Base WAL is retained for the view.** Base-table WAL segments are kept until + every valid live view on the table has consumed them and covered them with a + checkpoint, so a lagging view increases the base table's WAL disk usage. + Invalidated and dropped views release this hold. - **No admission control.** A view that cannot keep up with ingestion stays correct but stale, with no automatic throttle or drop. - **Per-partition state for partitioned windows grows with distinct partition cardinality.** A base table with high-cardinality partition keys (UUIDs, session ids) holds state per key. Use [`cairo.live.view.refresh.memory.limit.bytes`](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) - to bound the per-view refresh footprint. The `in_mem_bytes` column in + to cap the per-view refresh footprint; a view that exceeds the cap is + invalidated, not throttled. The `in_mem_bytes` column in [`live_views()`](/docs/query/functions/meta/#live_views) separately reports the peak-sticky capacity of the recent-row tier. @@ -379,11 +513,16 @@ views. Two dedicated permissions govern live-view DDL, modelled on the materialized-view permissions: -- `CREATE LIVE VIEW` is a database-level permission. +- `CREATE LIVE VIEW` is a database-level permission. Creating a view also + requires `SELECT` on the base-table columns the view reads; a table-level + `SELECT` grant covers them. - `DROP LIVE VIEW` is checked against the target view. -Querying a live view uses the standard table-level `SELECT` permission, since a -live view is a regular table token. See +The creator, or the `OWNED BY` principal when one is given, receives all +permissions on the new view with the `GRANT` option. Querying a live view uses +the standard table-level `SELECT` permission, since a live view is a regular +table token. [`ALTER LIVE VIEW RESUME WAL`](/docs/query/sql/alter-live-view-resume-wal/) +requires the `RESUME WAL` permission on the view. See [Role-based access control](/docs/security/rbac/) for the full permission model. ### Replication @@ -395,6 +534,13 @@ refreshes and flushes its own node-local live-view table: - A read-only replica refreshes from its locally applied copy of the base table. - Live-view WAL is not uploaded or transferred between nodes. +`CREATE LIVE VIEW` and `DROP LIVE VIEW` run on the primary, and a dropped view +is removed from replicas too. A replica keeps no in-memory rows ahead of disk, +so its live views trail by up to one `FLUSH EVERY` interval plus base-table +replication and apply lag. Refresh state and checkpoints are also node-local, +so a view invalidated on one node, for example by a memory limit breach, stays +valid on the others. + A role switch continues the local refresh state; it does not reconstruct or transfer the former primary's live-view rows. Replica freshness therefore also depends on base-table replication and apply lag. @@ -402,14 +548,17 @@ depends on base-table replication and apply lag. ### Backup and restore A live view is captured by the object-store backup like a materialized view: its -table data rides the standard table path and its definition sidecars are carried -in the backup manifest. After restore, unflushed rows are re-derived from the -base table. +table data and definition files are backed up with the table. The checkpoint +timeline is not backed up. After restore, window state is rebuilt from the +restored base table, and rows that were not yet flushed at backup time are +re-derived from it. ## Related documentation - **SQL commands** - [`CREATE LIVE VIEW`](/docs/query/sql/create-live-view/): Create a live view + - [`ALTER LIVE VIEW RESUME WAL`](/docs/query/sql/alter-live-view-resume-wal/): + Resume a suspended live view - [`DROP LIVE VIEW`](/docs/query/sql/drop-live-view/): Remove a live view - **Related concepts** diff --git a/documentation/concepts/views.md b/documentation/concepts/views.md index 411b38ff65..5ef4b9be97 100644 --- a/documentation/concepts/views.md +++ b/documentation/concepts/views.md @@ -352,7 +352,8 @@ Use a [live view](/docs/concepts/live-views/) instead when you need to incrementally maintain a row-per-input window computation, such as a moving average, running total, or ranking. -The output of a live view is one row per base row. +The output of a live view is one row per base row that passes the view's +`WHERE` filter and `START FROM` boundary. ## Security with views diff --git a/documentation/configuration/cairo-engine.md b/documentation/configuration/cairo-engine.md index d4b1b2262c..e83e510c4b 100644 --- a/documentation/configuration/cairo-engine.md +++ b/documentation/configuration/cairo-engine.md @@ -931,8 +931,7 @@ edit `server.conf` and call materialized view refreshes, and WAL apply batches use the updated limits; work already running keeps its original limit. A live view acquires its limit when it is first compiled and keeps it across refreshes, so a reloaded value reaches an -existing view only after a full teardown: invalidation, recreation, or a server -restart. +existing view only after the view is recreated or the server restarts. When a workload exceeds its limit, QuestDB raises an out-of-memory error at the allocation that crossed the line and aborts that workload, while unrelated @@ -1051,7 +1050,9 @@ you can reproduce as a plain query and measure the same way: the view retains between refreshes: the `IN MEMORY` tier, whose capacity [`live_views()`](/docs/query/functions/meta/#live_views) reports in its `in_mem_bytes` column, and the window state. Add these to the transient - figure, and keep the limit above the + figure. Window state grows with the number of distinct partition keys the + view has seen and does not shrink when keys go quiet, so leave headroom for + key growth, and keep the limit above the [allocation floor](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) described with the key. diff --git a/documentation/configuration/live-views.md b/documentation/configuration/live-views.md index e44ce2a952..b76e599e50 100644 --- a/documentation/configuration/live-views.md +++ b/documentation/configuration/live-views.md @@ -3,15 +3,13 @@ title: Live views description: Configuration settings for live views in QuestDB. --- -These settings control live view SQL support and the background refresh job that -maintains live views incrementally. For a conceptual overview, see -[Live views](/docs/concepts/live-views/). +These settings control live view SQL support, the refresh worker pool, and the +background refresh job that maintains live views incrementally. For a conceptual +overview, see [Live views](/docs/concepts/live-views/). -Live view refresh shares the materialized view refresh worker pool, so the -worker-pool settings under -[Materialized views](/docs/configuration/materialized-views/) -(`mat.view.refresh.worker.count`, `.affinity`, `.haltOnError`) also govern live -view refresh. There are no dedicated live-view worker-pool properties. +Live view refresh runs on its own worker pool, separate from the +[materialized view](/docs/configuration/materialized-views/) refresh pool. Size +it with [`live.view.refresh.worker.count`](#liveviewrefreshworkercount). ## cairo.live.view.checkpoint.compaction.interval @@ -20,7 +18,10 @@ view refresh. There are no dedicated live-view worker-pool properties. Number of checkpoint seals between physical compaction attempts. Compaction repacks live state pages from sparse data segments so obsolete segments can be -reclaimed. A value of `0` disables compaction. +reclaimed. A value of `0` disables compaction. Without compaction, a superseded +page is reclaimed only once its whole segment becomes unreferenced, so dead +bytes left behind by out-of-order repairs can accumulate. Enable it for views +that receive frequent out-of-order corrections. ## cairo.live.view.checkpoint.max.duration.micros @@ -31,6 +32,17 @@ Maximum wall-clock interval, in microseconds, between head-checkpoint writes. This caps restart and out-of-order replay work for low-rate views that do not reach the row-count trigger. +## cairo.live.view.checkpoint.purge.interval + +- **Default**: `1` +- **Reloadable**: no + +Number of checkpoint seals between sweeps that delete checkpoint segments no +longer referenced by any checkpoint generation. A value of `0` disables the +periodic sweep. Unreferenced segments are then reclaimed only by the sweep that +runs once per server start, so their disk space is not released until a +restart. + ## cairo.live.view.checkpoint.repair.replay.max.rows - **Default**: `1000000` @@ -65,9 +77,11 @@ limit is crossed, QuestDB uses the conservative unlocalized bound. A value of - **Default**: `1000000` - **Reloadable**: no -Number of newly flushed rows after which the refresh worker writes a head -checkpoint. Smaller values shorten restart replay at the cost of more checkpoint -writes. +Number of live view output rows applied since the previous head checkpoint after +which the refresh worker writes a new one. Smaller values shorten restart and +out-of-order replay at the cost of more checkpoint writes. The value is also the +per-turn row budget of the initial seed sweep, so lowering it also shortens each +seed turn. ## cairo.live.view.enabled @@ -75,43 +89,58 @@ writes. - **Reloadable**: no Enables or disables SQL support and the refresh job for live views. When -disabled, `CREATE LIVE VIEW` fails with `live views are disabled`. +disabled, `CREATE LIVE VIEW` fails with `live views are disabled`, and existing +live views are neither refreshed nor listed in +[`live_views()`](/docs/query/functions/meta/#live_views). They stay queryable as +plain tables without their in-memory tier, and the base table's WAL is no longer +retained on their behalf. Re-enabling live views rebuilds each existing view +from the applied base table without invalidating it. ## cairo.live.view.flush.retry.max - **Default**: `5` - **Reloadable**: no -Maximum number of consecutive flush attempts before a view is marked invalid. A -flush persists the in-memory rows to the view's disk tier. +Maximum number of consecutive failed refresh cycles, such as a failed flush to +the view's disk tier, before the view is marked invalid. The view is invalidated +as soon as either this count or +[`cairo.live.view.flush.retry.max.duration.micros`](#cairoliveviewflushretrymaxdurationmicros) +is exhausted. Failures caused by a transiently missing base table count only +against the duration budget. A breach of +[`cairo.live.view.refresh.memory.limit.bytes`](#cairoliveviewrefreshmemorylimitbytes) +invalidates the view without any retry. ## cairo.live.view.flush.retry.max.duration.micros - **Default**: `60000000` (60 seconds) - **Reloadable**: no -Maximum total time, in microseconds, spent retrying a stalled flush before the -view is marked invalid. +Maximum time, in microseconds, that a view may keep failing refresh cycles +before it is marked invalid. See +[`cairo.live.view.flush.retry.max`](#cairoliveviewflushretrymax) for how the two +budgets interact. ## cairo.live.view.in.memory.buffer.growth.bytes - **Default**: `16777216` (16 MiB) - **Reloadable**: no -Fast-path growth budget for the in-memory tier. Once the published buffer's -footprint reaches this size, refresh falls back to a swap that evicts expired -rows and may shrink the buffer instead of continuing to append in place. Raise -it for `IN MEMORY` windows expected to exceed the default. A value of `0` or -less forces this compaction path on every publish. Accepts a size suffix such as -`16M`. +Bytes of aged rows that the in-memory tier may hold before a refresh compacts +it. Aged rows are rows already flushed to disk and older than the view's +`IN MEMORY` window. Until this many bytes of aged rows accumulate, refresh keeps +appending in place; once they do, it swaps to a compacted buffer that evicts +them. The value therefore bounds how much RAM the tier holds beyond the +`IN MEMORY` window. A value of `0` or less compacts on every publish. Accepts a +size suffix such as `16M`. ## cairo.live.view.in.memory.buffer.initial.bytes - **Default**: `65536` (64 KiB) - **Reloadable**: no -Initial size of a live view's in-memory tier buffer. Accepts a size suffix such -as `64K`. +Initial size of each column buffer in the in-memory tier. The tier keeps two +buffer slots, and each slot holds one buffer per output column. Accepts a size +suffix such as `64K`. ## cairo.live.view.in.memory.max @@ -126,30 +155,39 @@ Upper bound on the `IN MEMORY` retention window. A `CREATE LIVE VIEW` whose - **Default**: `100000` - **Reloadable**: no -Anchor-map tombstone threshold that triggers compaction. Compaction also runs -when tombstones exceed half of the anchor map, regardless of this absolute -threshold. +Threshold for the sweep that drops anchored-window partition state left behind +the previous anchor boundary. The sweep runs only when all of the following +hold: the anchor map has more partitions than this value, at least this many of +them are reclaimable, and the reclaimable partitions make up at least half of +the map. A view whose anchor value is not monotonic, or is `NULL`, never +compacts. ## cairo.live.view.refresh.memory.limit.bytes - **Default**: `0` (unlimited) - **Reloadable**: yes -Per-live-view limit on the peak memory used during a refresh cycle. It includes -persistent window state, the `IN MEMORY` recent-row tier, staging memory, and -transient buffers such as Parquet row-group decode buffers. Exceeding the limit -invalidates the view immediately; recovery requires dropping and recreating it. +Per-live-view limit on tracked native memory. It covers persistent window state, +the `IN MEMORY` recent-row tier, staging memory, and transient buffers such as +Parquet row-group decode buffers. Exceeding the limit invalidates the view +immediately; recovery requires dropping and recreating it. Process-wide +out-of-memory errors are not limit breaches and are retried instead. This is one of the four workload [memory limits](/docs/configuration/cairo-engine/#memory-limits) and counts tracked native allocations only. A view acquires the limit when it is first -compiled and keeps it across refreshes, so a reloaded value reaches an existing -view only after invalidation, recreation, or a server restart. +compiled and keeps it until it is dropped or invalidated, so the limit applies +to the view's running footprint across refreshes rather than to a single cycle. +Persistent state grows with the number of distinct partition keys the view has +seen and does not shrink when keys go quiet, and the in-memory tier does not +shrink after a burst. A limit sized from an early measurement can therefore be +breached later without any change in workload. A reloaded value reaches an +existing view only after it is recreated or the server restarts. Size the limit above the refresh workload's allocation floor. In particular, a -bounded `ROWS` frame allocates at least one `cairo.sql.window.store.page.size` -page (1 MiB by default), and a Parquet-backed refresh may decode a complete row -group. +`RANGE` frame, or a `ROWS` frame or `lag()` with `PARTITION BY`, allocates at +least one `cairo.sql.window.store.page.size` page (1 MiB by default) on its +first partition, and a Parquet-backed refresh may decode a complete row group. ## cairo.live.view.refresh.turn.max.commits @@ -166,3 +204,60 @@ before yielding to other views. Maximum wall-clock time, in microseconds, a refresh worker spends on one view per turn before yielding. + +## live.view.refresh.worker.affinity + +- **Default**: none +- **Reloadable**: no + +Comma-separated list of CPU core indexes, one per refresh worker. + +## live.view.refresh.worker.count + +- **Default**: `2` on hosts with up to 8 CPU cores, `3` on up to 32 cores, `4` + on larger hosts +- **Reloadable**: no + +Number of worker threads that refresh live views. A single view is refreshed by +one worker at a time, so more workers help when several views are busy at once, +not when one view is slow. + +A value of `0` turns live view refresh off. `CREATE LIVE VIEW` then fails, and +existing views behave as described for +[`cairo.live.view.enabled`](#cairoliveviewenabled) set to `false`. + +## live.view.refresh.worker.haltOnError + +- **Default**: `false` +- **Reloadable**: no + +Flag that indicates if a refresh worker thread must stop when an unexpected +error occurs. + +## live.view.refresh.worker.nap.threshold + +- **Default**: `7000` +- **Reloadable**: no + +Number of idle worker-loop iterations before a refresh worker naps. + +## live.view.refresh.worker.sleep.threshold + +- **Default**: `10000` +- **Reloadable**: no + +Number of idle worker-loop iterations before a refresh worker sleeps. + +## live.view.refresh.worker.sleep.timeout + +- **Default**: `10ms` +- **Reloadable**: no + +Sleep duration when a refresh worker has no views to refresh. + +## live.view.refresh.worker.yield.threshold + +- **Default**: `1000` +- **Reloadable**: no + +Number of idle worker-loop iterations before a refresh worker yields. diff --git a/documentation/configuration/materialized-views.md b/documentation/configuration/materialized-views.md index 92182160b6..e8273329c4 100644 --- a/documentation/configuration/materialized-views.md +++ b/documentation/configuration/materialized-views.md @@ -72,18 +72,22 @@ has no effect; deferred out-of-memory retries use this backoff. ## mat.view.refresh.worker.affinity -- **Default**: equal to the CPU core count +- **Default**: none - **Reloadable**: no -Comma-separated list of numerical CPU core indexes. +Comma-separated list of numerical CPU core indexes, one per refresh worker. ## mat.view.refresh.worker.count -- **Default**: `0` +- **Default**: same as + [`wal.apply.worker.count`](/docs/configuration/wal/#walapplyworkercount) - **Reloadable**: no -Number of dedicated worker threads assigned to refresh materialized views. -When `0`, uses the shared worker pool. +Number of dedicated worker threads assigned to refresh materialized views. A +value of `0` disables materialized view refresh: `CREATE MATERIALIZED VIEW` +still succeeds, but nothing refreshes the view. Live views use their own pool, +sized by +[`live.view.refresh.worker.count`](/docs/configuration/live-views/#liveviewrefreshworkercount). ## mat.view.refresh.worker.haltOnError diff --git a/documentation/configuration/wal.md b/documentation/configuration/wal.md index e485910de4..723bac0c12 100644 --- a/documentation/configuration/wal.md +++ b/documentation/configuration/wal.md @@ -83,14 +83,15 @@ too high may cause excessive memory usage and increase latency. ## wal.apply.worker.affinity -- **Default**: equal to the CPU core count +- **Default**: none - **Reloadable**: no -Comma-separated list of CPU core indexes. +Comma-separated list of CPU core indexes, one per WAL apply worker. ## wal.apply.worker.count -- **Default**: equal to the CPU core count +- **Default**: `2` on hosts with up to 8 CPU cores, `3` on up to 32 cores, `4` + on larger hosts - **Reloadable**: no Number of dedicated worker threads assigned to handle WAL table data. diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index bd0f2e4c23..60ae8195e9 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -198,13 +198,13 @@ Returns a `table` with the following columns: | `in_mem_rows` | LONG | Live row count in the published in-memory tier | | `o3_rejected_count` | LONG | Late out-of-order rows rejected below the view lower bound; resets on restart | | `below_lower_bound_count` | LONG | In-order rows dropped below the view lower bound; resets on restart | -| `lag_seqtxn` | LONG | Committed base WAL transactions beyond `last_processed_seqtxn` | -| `lag_micros` | LONG | Microseconds since the last successful flush | -| `last_processed_seqtxn` | LONG | Last base transaction processed by the refresh worker | +| `lag_seqtxn` | LONG | Base transactions applied to the base table beyond `last_processed_seqtxn` | +| `lag_micros` | LONG | Microseconds since the last successful flush; `NULL` until the first flush after creation or restart | +| `last_processed_seqtxn` | LONG | Last base transaction whose output was flushed to the view; the in-memory tier can be ahead of it | | `applied_watermark` | LONG | Last base transaction durably applied to the view's disk tier | | `lv_consumed_seqtxn` | LONG | Base WAL purge floor held by this view | | `view_lower_bound_timestamp` | TIMESTAMP | Resolved `START FROM` boundary; `NULL` for `BEGINNING` | -| `writer_stall_micros` | LONG | Current uninterrupted flush-writer stall duration, or `0` | +| `writer_stall_micros` | LONG | How long the in-memory tier has been unable to publish because readers pin both buffer slots, or `0` | | `seed_target_seqtxn` | LONG | Target base transaction during initial seeding; otherwise `NULL` | | `o3_resume_replay_rows` | LONG | Rows emitted by repairs resumed from a checkpoint; resets on restart | | `o3_boundary_replay_rows` | LONG | Rows emitted by rebuild repairs; resets on restart | @@ -236,6 +236,7 @@ Returns a `table` with the following columns: | `checkpoint_repair_plan` | STRING | Available localized plan: `range`, `rows`, `anchor`, a `+` combination, `none`, or `NULL` before compilation | | `checkpoint_repair_last_disposition` | STRING | Last executor used: `localized rebuild`, `boundary rebuild`, or `resume from anchor` | | `checkpoint_repair_last_denial` | STRING | Reason the last repair did not use a localized rebuild; otherwise `NULL` | +| `checkpoint_seal_failures` | LONG | Failed checkpoint seals; growth means restart-recovery state is going stale; resets on restart | The `in_mem_bytes` and `in_mem_rows` columns are complementary. `in_mem_bytes` is the peak-sticky arena footprint that does not shrink after a burst, while @@ -243,12 +244,15 @@ is the peak-sticky arena footprint that does not shrink after a burst, while `IN MEMORY` window. Together they distinguish a view actively buffering rows from one holding capacity retained from a past burst. -`lag_seqtxn` counts transactions, not rows. A value of `0` means the durable -tier is caught up; a temporary non-zero value is expected between `FLUSH EVERY` -cycles. Alert on a value that remains above its normal flush-cycle baseline or -keeps increasing across samples rather than on a universal fixed threshold. -`lag_micros` measures flush activity and may grow while an idle view has -`lag_seqtxn = 0`. +`lag_seqtxn` counts transactions, not rows. A value of `0` means the view's +durable tier has caught up with every transaction applied to the base table; a +temporary non-zero value is expected between `FLUSH EVERY` cycles. Base commits +that are not yet applied to the base table are not counted, so when the base +table itself lags or is suspended, check +[`wal_tables()`](#wal_tables) as well. Alert on a value that remains above its +normal flush-cycle baseline or keeps increasing across samples rather than on a +universal fixed threshold. `lag_micros` measures flush activity and may grow +while an idle view has `lag_seqtxn = 0`. The checkpoint columns describe the current persistent timeline, its storage and lookup cost, and out-of-order repair activity. Timeline fields are `NULL` before diff --git a/documentation/query/sql/alter-live-view-resume-wal.md b/documentation/query/sql/alter-live-view-resume-wal.md new file mode 100644 index 0000000000..1abf532b0e --- /dev/null +++ b/documentation/query/sql/alter-live-view-resume-wal.md @@ -0,0 +1,89 @@ +--- +title: ALTER LIVE VIEW RESUME WAL +sidebar_label: RESUME WAL +description: + Documentation for the ALTER LIVE VIEW RESUME WAL SQL keyword in QuestDB. +--- + +Restarts [WAL](/docs/concepts/write-ahead-log/) apply on a +[live view](/docs/concepts/live-views/) whose own WAL was suspended after an +error, such as a full disk, so that the view's disk tier catches up again. + +## Syntax + +```questdb-sql +ALTER LIVE VIEW viewName RESUME WAL + [ FROM { TRANSACTION | TXN } sequencerTxn ]; +``` + +## Parameters + +| Parameter | Description | +| --------- | ----------- | +| `viewName` | Name of the live view to resume | +| `FROM TRANSACTION` | Optional starting transaction number (defaults to the failed transaction) | + +## When to use + +A live view writes its output to its own WAL-backed table. If applying one of +the view's WAL transactions fails, QuestDB suspends the view's WAL like that of +any other WAL table, and the view can serve stale rows until you resume it. +[`live_views()`](/docs/query/functions/meta/#live_views) still reports +`view_status = 'active'` for a suspended view, so check +[`wal_tables()`](/docs/query/functions/meta/#wal_tables) instead: + +```questdb-sql title="Find suspended live views" +SELECT w.name, w.writerTxn, w.sequencerTxn, w.errorTag, w.errorMessage +FROM wal_tables() w +JOIN live_views() v ON w.name = v.view_name +WHERE w.suspended; +``` + +`RESUME WAL` only recovers a suspended WAL writer. It does not revalidate a view +that has been [invalidated](/docs/concepts/live-views/#base-table-lifecycle); +an invalid view must be dropped and recreated. + +## Examples + +```questdb-sql title="Resume WAL processing" +ALTER LIVE VIEW trades_ma RESUME WAL; +``` + +```questdb-sql title="Resume from a specific transaction" +ALTER LIVE VIEW trades_ma RESUME WAL FROM TRANSACTION 5; +``` + +Resuming from a later transaction skips the earlier ones. Fix the underlying +issue and resume from the failed transaction where possible. + +:::note + +`ALTER LIVE VIEW viewName SUSPEND WAL` is also accepted, but it does not pause +a live view: the view's next refresh applies its pending WAL and clears the +suspension. No other `ALTER LIVE VIEW` operations are supported. + +::: + +## Permissions (Enterprise) + +Resuming WAL on a live view requires the `RESUME WAL` permission on the view: + +```questdb-sql title="Grant resume permission" +GRANT RESUME WAL ON trades_ma TO user1; +``` + +## Errors + +| Error | Cause | +| ----- | ----- | +| `live view name expected` | The name refers to a table or view that is not a live view | +| `table does not exist` | No table or view with this name exists | +| `'resume' or 'suspend' expected` | The statement uses an operation other than `RESUME WAL` or `SUSPEND WAL` | + +## See also + +- [Live views concept](/docs/concepts/live-views/) +- [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) +- [DROP LIVE VIEW](/docs/query/sql/drop-live-view/) +- [ALTER TABLE RESUME WAL](/docs/query/sql/alter-table-resume-wal/) +- [wal_tables() function](/docs/query/functions/meta/#wal_tables) diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index 6c6fd579dd..ee0a0f3397 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -32,22 +32,22 @@ AS [ ( ] query [ ) ] Where: -- `duration`: a single token with a unit of `ms`, `s`, `m`, `h`, or `d`, for - example `100ms`, `5s`, or `30m`. +- `duration`: a single token with a lowercase unit of `ms`, `s`, `m`, `h`, or + `d`, for example `100ms`, `5s`, or `30m`. - `query`: a `SELECT` over one WAL-backed base table whose projection contains [window functions](/docs/query/functions/window-functions/overview/). `FLUSH EVERY` is required and must come first. `START FROM` is also required and may appear in any order with the optional `IN MEMORY` and `PARTITION BY` clauses. These clauses all precede `AS`; the optional `OWNED BY` clause follows -the query. +the query. When `OWNED BY` is used, wrap the query in parentheses. ## Parameters | Parameter | Description | | --------- | ----------- | | `viewName` | Name for the live view | -| `IF NOT EXISTS` | Create only if a view with this name does not already exist | +| `IF NOT EXISTS` | Create only if a live view with this name does not already exist | | `FLUSH EVERY` | How often computed rows are persisted to disk. Required | | `IN MEMORY` | Window of recent rows kept in RAM for fresh reads. Defaults to `FLUSH EVERY` | | `PARTITION BY` | Partitioning unit for the view's disk tier. Defaults to the base table's scheme | @@ -108,7 +108,8 @@ FROM trades; ### PARTITION BY `PARTITION BY` sets the partitioning of the view's disk tier. If omitted, the -view inherits the base table's partitioning scheme. +view inherits the base table's partitioning scheme. `PARTITION BY NONE` is not +supported. ```questdb-sql CREATE LIVE VIEW trades_ma @@ -133,8 +134,9 @@ view. It is mandatory and accepts: to or later than that value. The boundary applies to the base table's designated timestamp, not to commit -time. QuestDB performs a resumable initial seed for qualifying rows already -present at creation, then continues refreshing from new base commits. +time. Rows below the boundary neither appear in the view nor contribute to +window state. QuestDB performs a resumable initial seed for qualifying rows +already present at creation, then continues refreshing from new base commits. ```questdb-sql CREATE LIVE VIEW trades_ma @@ -168,8 +170,9 @@ ANCHOR DAILY 'HH:MM' [ 'timezone' ] ANCHOR EXPRESSION expression ``` -`ANCHOR DAILY` requires a quoted 24-hour time. An optional IANA time zone makes -the reset follow local civil time; without one, the boundary is in UTC. +`ANCHOR DAILY` requires a quoted 24-hour time. An optional time zone, either an +IANA name or a UTC offset such as `'+02:00'`, makes the reset follow local time; +without one, the boundary is in UTC. ```questdb-sql title="Cumulative daily volume, reset each day" CREATE LIVE VIEW trades_daily_volume @@ -209,29 +212,43 @@ An anchored window must: - Be a named window; `ANCHOR` is not supported in an inline `OVER (...)`. - Use `PARTITION BY` with base-table columns directly. - `ORDER BY` the designated timestamp ascending. -- Use the default unbounded frame; `ANCHOR` cannot be combined with a bounded - `ROWS` or `RANGE` frame. - -A live view supports at most one anchored window. An `ANCHOR EXPRESSION` must be -deterministic, non-constant, and return `TIMESTAMP`, `LONG`, or `INT`. +- Use the default frame; `ANCHOR` cannot be combined with an explicit `ROWS` + frame or a bounded `RANGE` frame. +- Be referenced directly by at least one window function, as in `OVER w`. A + window that inherits from it, such as `WINDOW w2 AS (w ...)`, does not carry + the anchor. + +A live view supports at most one anchored window, which several window +functions can share alongside other, bounded windows. An `ANCHOR EXPRESSION` +must be deterministic, non-constant, and return `TIMESTAMP`, `LONG`, or `INT`. +It cannot contain aggregates, subqueries, or bind variables, and it can only +reference base columns that the query already selects or filters on. ## Query constraints The view query is validated at creation time and must: - Read a single WAL-backed base table that has a designated timestamp. No JOINs, - subqueries, or CTEs. -- Contain [window functions](/docs/query/functions/window-functions/overview/) - that can be maintained incrementally (see - [supported functions](/docs/concepts/live-views/#supported-window-functions)). -- Give every stateful window function a `PARTITION BY` clause. + subqueries, CTEs, or regular views in `FROM`. +- Select the base table's designated timestamp as a plain column under its own + name. Omitting it, computing it, or aliasing it is rejected. +- Contain at least one + [window function](/docs/query/functions/window-functions/overview/) that can + be maintained incrementally. The supported functions depend on the window + frame; see + [Supported window functions](/docs/concepts/live-views/#supported-window-functions). +- Give every stateful window function a `PARTITION BY` clause on a supported key + type. `VARCHAR`, `UUID`, `LONG256`, `DECIMAL`, `BINARY`, and array keys are + rejected. - Use a bounded `ROWS` or `RANGE` frame, or a named anchored window. Ranking - functions (`row_number`, `rank`, and `dense_rank`) must be anchored. Frames - starting at `UNBOUNDED PRECEDING` are rejected except for stateless - `last_value` shapes. -- Not use `SAMPLE BY`, `GROUP BY`, a top-level `ORDER BY`, or `LIMIT` in the - view query. The `ORDER BY` inside a window's `OVER (...)` is required and - allowed. + functions (`row_number`, `rank`, and `dense_rank`) must be anchored. A window + without a frame clause, and any frame starting at `UNBOUNDED PRECEDING`, is + rejected unless it is anchored; stateless `last_value` shapes are the only + exception. +- Order every window by the designated timestamp ascending, if it has an + `ORDER BY`. Named windows must declare it. +- Not use `SAMPLE BY`, `GROUP BY`, `LATEST ON`, `DISTINCT`, or `LIMIT`. A + top-level `ORDER BY` is accepted only on the designated timestamp ascending. - List output columns explicitly; wildcard projections such as `SELECT *` are not allowed. - Not filter on the base table's designated timestamp. Other deterministic @@ -291,7 +308,8 @@ FROM live_views(); ## Permissions (Enterprise) Creating a live view requires the database-level `CREATE LIVE VIEW` permission -and `SELECT` on the base table: +and `SELECT` on the base-table columns the view reads. A table-level `SELECT` +grant covers all of them: ```questdb-sql title="Grant permission to create live views" GRANT CREATE LIVE VIEW TO user1; @@ -302,45 +320,94 @@ GRANT SELECT ON trades TO user1; ``` When you create a live view you automatically receive all permissions on it, -including `DROP LIVE VIEW`, with the `GRANT` option. +including `DROP LIVE VIEW`, with the `GRANT` option. With `OWNED BY`, the named +owner receives these permissions instead. ### OWNED BY clause -Assign ownership to a user, group, or service account: +Assign ownership to a user, group, or service account. The query must be wrapped +in parentheses: ```questdb-sql CREATE GROUP analysts; CREATE LIVE VIEW trades_ma FLUSH EVERY 1s START FROM NOW -AS -SELECT timestamp, symbol, - avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) - AS moving_avg -FROM trades +AS ( + SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg + FROM trades +) OWNED BY analysts; ``` +External users must specify `OWNED BY`, and the creator must be a member of the +group named as the owner. + ## Errors -| Error | Cause | -| ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------- | -| `live views are disabled` | Live-view support is turned off (`cairo.live.view.enabled=false`) | -| `live view already exists` | A live view of this name exists and `IF NOT EXISTS` was not specified | -| `table or view with the requested name already exists` | The name is taken by a table, view, or materialized view | -| `live view FLUSH EVERY must be at least 100ms` | The `FLUSH EVERY` interval is below the minimum | -| `live view select must be a simple scan of a single WAL base table; joins, subqueries, GROUP BY, ORDER BY and LIMIT are not supported yet` | The view query is not a simple scan of one base table | -| `base table must be a WAL table` | The base object is a non-WAL table or a regular view | -| `live views are not allowed as base tables in V1` | The base object is another live view | -| `live view base table must have a designated timestamp` | The base table has no designated timestamp | -| `wildcard column select is not allowed in live view queries` | The top-level projection contains `*` | -| `live view unbounded window must have an ANCHOR clause` | A stateful partitioned window uses the default unbounded frame without an anchor | -| `non-deterministic function cannot be used in live view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | -| `permission denied` | Missing required permission (Enterprise) | +Errors that end in a bracketed detail, such as `[name=trades]`, are shown +without it. + +### Statement errors + +| Error | Cause | +| ----- | ----- | +| `live views are disabled` | Live-view support is turned off (`cairo.live.view.enabled=false`) | +| `live view refresh is disabled, set live.view.refresh.worker.count to a positive value` | [`live.view.refresh.worker.count`](/docs/configuration/live-views/#liveviewrefreshworkercount) is `0` | +| `live view already exists` | A live view of this name exists and `IF NOT EXISTS` was not specified | +| `table or view with the requested name already exists` | The name is taken by a table, view, or materialized view, even with `IF NOT EXISTS` | +| `live view FLUSH EVERY must be at least 100ms` | The `FLUSH EVERY` interval is below the minimum | +| `live view FLUSH EVERY must be at most cairo.live.view.in.memory.max` | `FLUSH EVERY`, and therefore the default `IN MEMORY`, exceeds the cap | +| `live view IN MEMORY must be at least FLUSH EVERY` | `IN MEMORY` is shorter than `FLUSH EVERY` | +| `live view IN MEMORY must be at most cairo.live.view.in.memory.max` | `IN MEMORY` exceeds the cap | +| `live view PARTITION BY NONE is not supported` | Live views must be partitioned | +| `live view requires a START FROM clause` | `START FROM` is missing | +| `Access denied for [CREATE LIVE VIEW]` | Missing `CREATE LIVE VIEW` permission (Enterprise) | +| `Access denied for [SELECT on ()]` | Missing `SELECT` on a base-table column the view reads (Enterprise) | +| `OWNED BY clause is required for external users` | An external user created a view without `OWNED BY` (Enterprise) | + +### Query shape errors + +| Error | Cause | +| ----- | ----- | +| `base table does not exist` | The base table does not exist | +| `base table must be a WAL table` | The base table is not a WAL table, including a table without a designated timestamp | +| `live views are not allowed as base tables in V1` | The base object is another live view | +| `live view requires a single base table in FROM clause` | `FROM` contains a subquery, a CTE, or a regular view | +| `live view select must be a simple scan of a single WAL base table; ...` | The query joins tables or uses `LATEST ON` | +| `live view select must be a plain windowed scan of the base table; ...` | The query uses `LIMIT` or `DISTINCT` | +| `live view select must contain at least one window function` | The projection has no window function | +| `live view select must project the base table's designated timestamp` | The designated timestamp is missing from the projection or computed | +| `live view select cannot override the designated timestamp` | The designated timestamp is aliased | +| `wildcard column select is not allowed in live view queries` | The top-level projection contains `*` | +| `live view select cannot filter on the designated timestamp yet` | The `WHERE` clause filters the designated timestamp | +| `non-deterministic function cannot be used in live view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | + +### Window errors + +| Error | Cause | +| ----- | ----- | +| `live view select cannot use window function (); incremental snapshot is not supported for this function yet` | The function does not support this frame, has no `PARTITION BY`, or partitions by an unsupported key type | +| `live view select may only use window functions that support incremental refresh` | The function needs look-ahead or multiple passes (`percent_rank`, `cume_dist`, `ntile`), or a window or top-level `ORDER BY` is not the designated timestamp ascending | +| `lead() is not supported in live views; use lag() for lookback` | The query uses `lead` | +| `live view unbounded window must have an ANCHOR clause` | A window without a frame clause is not anchored | +| `live view select cannot use () over a frame starting at UNBOUNDED PRECEDING` | An explicit frame starts at `UNBOUNDED PRECEDING` and is not anchored | +| `live view select cannot use () without an anchored WINDOW` | A ranking function is not anchored | +| `live view RANGE window functions must use the same PARTITION BY and ORDER BY domain` | Two `RANGE` windows partition or order differently | +| `live view named WINDOW must ORDER BY ASC` | A named window does not order by the designated timestamp ascending | +| `ANCHOR is only supported on named WINDOW clauses` | `ANCHOR` appears in an inline `OVER (...)` | +| `ANCHOR is incompatible with bounded frames` | An anchored window has an explicit frame | +| `live view anchored WINDOW requires PARTITION BY` | An anchored window has no `PARTITION BY` | +| `live view ANCHOR currently requires PARTITION BY to reference base columns directly` | An anchored window partitions by an expression | +| `live view supports at most one anchored WINDOW in V1` | The query declares more than one anchored window | +| `live view anchored WINDOW '' is not referenced by any window function` | No window function uses the anchored window directly | ## See also - [Live views concept](/docs/concepts/live-views/) +- [ALTER LIVE VIEW RESUME WAL](/docs/query/sql/alter-live-view-resume-wal/) - [DROP LIVE VIEW](/docs/query/sql/drop-live-view/) - [Window functions](/docs/query/functions/window-functions/overview/) - [live_views()](/docs/query/functions/meta/#live_views) diff --git a/documentation/query/sql/drop-live-view.md b/documentation/query/sql/drop-live-view.md index dd684e04b3..9686338d7a 100644 --- a/documentation/query/sql/drop-live-view.md +++ b/documentation/query/sql/drop-live-view.md @@ -19,7 +19,7 @@ DROP LIVE VIEW [ IF EXISTS ] viewName | Parameter | Description | | --------- | ----------- | | `viewName` | Name of the live view to drop | -| `IF EXISTS` | Suppress the error if the view does not exist | +| `IF EXISTS` | Suppress the error if no table or view with this name exists | ## Examples @@ -62,11 +62,12 @@ The view creator automatically receives this permission with the `GRANT` option. | Error | Cause | | ----- | ----- | -| `live view name expected` | The name refers to a table or view that is not a live view | -| `live view does not exist` | The view does not exist and `IF EXISTS` was not specified | -| `permission denied` | Missing `DROP LIVE VIEW` permission (Enterprise) | +| `live view name expected [name=]` | The name refers to a table or view that is not a live view, even with `IF EXISTS` | +| `live view does not exist [name=]` | The view does not exist and `IF EXISTS` was not specified | +| `Access denied for [DROP LIVE VIEW on ]` | Missing `DROP LIVE VIEW` permission (Enterprise) | ## See also - [Live views concept](/docs/concepts/live-views/) - [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) +- [ALTER LIVE VIEW RESUME WAL](/docs/query/sql/alter-live-view-resume-wal/) diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 29256373f2..06188b45cf 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -59,7 +59,8 @@ SHOW { COLUMNS FROM tableName - `SHOW SERVICE ACCOUNTS` lists all service accounts with their enabled flag and memory limit, or those a user or group can assume with the grant option (enterprise-only) -- `SHOW TABLES` returns all the tables. +- `SHOW TABLES` returns all tables, including views, materialized views, and + live views. - `SHOW USER` shows user secret (enterprise-only) - `SHOW USERS` lists all users with their enabled flag and memory limit (enterprise-only) @@ -101,7 +102,7 @@ SHOW CREATE DATABASE An optional `INCLUDE` or `EXCLUDE` clause selects which object categories to dump. Each `category` is one of: -- **Schema objects**: `TABLES`, `VIEWS`, `MATERIALIZED_VIEWS`. +- **Schema objects**: `TABLES`, `VIEWS`, `MATERIALIZED_VIEWS`, `LIVE_VIEWS`. - **Access control** (Enterprise only): `USERS`, `GROUPS`, `SERVICE_ACCOUNTS`, `PERMISSIONS`. - **Umbrellas**: `SCHEMA` (all schema objects), `ACL` (all access control @@ -221,7 +222,7 @@ SHOW CREATE LIVE VIEW trades_ma; This returns the `CREATE LIVE VIEW` statement that would recreate the [live view](/docs/concepts/live-views/), including its `FLUSH EVERY`, `IN MEMORY`, `PARTITION BY`, and `START FROM` clauses. On QuestDB Enterprise the -output also carries an `OWNED BY` clause identifying the view's owner. +output also carries an `OWNED BY` clause populated with the current user. ### SHOW CREATE MATERIALIZED VIEW diff --git a/documentation/sidebars.js b/documentation/sidebars.js index e824810980..8a76970a68 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -333,6 +333,11 @@ module.exports = { "query/sql/alter-table-change-symbol-capacity", // SYMBOL CAPACITY ], }, + { + type: "category", + label: "ALTER LIVE VIEW", + items: ["query/sql/alter-live-view-resume-wal"], + }, { type: "category", label: "ALTER MATERIALIZED VIEW", From 439017628d119f791ae88943f89c11f4cfc957e7 Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Mon, 12 Oct 2026 10:41:15 +0300 Subject: [PATCH 2/3] docs(live-views): drop mentions of ALTER LIVE VIEW SUSPEND WAL The statement is intentionally undocumented. Co-Authored-By: Claude Opus 5.5 --- documentation/changelog.mdx | 2 +- documentation/query/sql/alter-live-view-resume-wal.md | 10 +--------- 2 files changed, 2 insertions(+), 10 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 29999e4d14..469835b004 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -12,7 +12,7 @@ This page tracks significant updates to the QuestDB documentation. ### New -- [ALTER LIVE VIEW RESUME WAL](/docs/query/sql/alter-live-view-resume-wal/) - New reference page for resuming a live view whose own WAL was suspended after an error, how to find suspended live views through `wal_tables()`, and why `SUSPEND WAL` does not pause a live view +- [ALTER LIVE VIEW RESUME WAL](/docs/query/sql/alter-live-view-resume-wal/) - New reference page for resuming a live view whose own WAL was suspended after an error, and how to find suspended live views through `wal_tables()` ### Reference diff --git a/documentation/query/sql/alter-live-view-resume-wal.md b/documentation/query/sql/alter-live-view-resume-wal.md index 1abf532b0e..f49e3bc0b1 100644 --- a/documentation/query/sql/alter-live-view-resume-wal.md +++ b/documentation/query/sql/alter-live-view-resume-wal.md @@ -56,14 +56,6 @@ ALTER LIVE VIEW trades_ma RESUME WAL FROM TRANSACTION 5; Resuming from a later transaction skips the earlier ones. Fix the underlying issue and resume from the failed transaction where possible. -:::note - -`ALTER LIVE VIEW viewName SUSPEND WAL` is also accepted, but it does not pause -a live view: the view's next refresh applies its pending WAL and clears the -suspension. No other `ALTER LIVE VIEW` operations are supported. - -::: - ## Permissions (Enterprise) Resuming WAL on a live view requires the `RESUME WAL` permission on the view: @@ -78,7 +70,7 @@ GRANT RESUME WAL ON trades_ma TO user1; | ----- | ----- | | `live view name expected` | The name refers to a table or view that is not a live view | | `table does not exist` | No table or view with this name exists | -| `'resume' or 'suspend' expected` | The statement uses an operation other than `RESUME WAL` or `SUSPEND WAL` | +| `'resume' or 'suspend' expected` | The statement uses an unsupported `ALTER LIVE VIEW` operation, such as `SET TTL` or `ADD COLUMN` | ## See also From a5fa04356a7197f99df1b1caae45fbda13d05bd7 Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Mon, 12 Oct 2026 11:50:21 +0300 Subject: [PATCH 3/3] docs(live-views): address PR #580 review findings Concepts: - Say that out-of-order base rows rewrite view rows already emitted - Monitoring: lag is meaningful only for active views; a suspended view WAL still shows active, with a query to find such views - Give the sample stddev correction as SQL (integer division trap) - Retention: dropping and recreating is the only way to reclaim space - Scope the subquery/CTE rule to FROM; recovery recipe caveats for START FROM NOW, trailing semicolons, and Enterprise OWNED BY SQL reference: - CREATE LIVE VIEW: { } grammar, anchored WINDOW grammar, RANGE/WHERE and anchored session examples, corrected error causes, new rows, OWNED BY rules for non-admin creators - RESUME WAL: precise description and parameters, self-clearing suspension on the next flush after a base commit - Point ALTER TABLE RESUME WAL runbooks to the view-specific commands - EMA/VWEMA reject a frame clause rather than ignoring it - Mat view RESUME WAL needs the RESUME WAL permission Configuration: - Mat view and shared worker pool descriptions match 10.0.1 dedicated pools and CPU-based defaults - Anchor compaction applies only to TIMESTAMP anchors Co-Authored-By: Claude Opus 5.5 --- documentation/changelog.mdx | 10 +- documentation/concepts/live-views.md | 120 ++++++++++++----- documentation/concepts/write-ahead-log.md | 5 + documentation/configuration/live-views.md | 9 +- .../configuration/materialized-views.md | 11 +- documentation/configuration/shared-workers.md | 25 +++- .../operations/monitoring-alerting.md | 11 +- documentation/query/functions/meta.md | 8 +- .../functions/window-functions/reference.md | 4 +- .../query/sql/alter-live-view-resume-wal.md | 15 ++- .../query/sql/alter-mat-view-resume-wal.md | 8 +- .../query/sql/alter-table-rebase-wal.md | 3 + .../query/sql/alter-table-resume-wal.md | 10 ++ documentation/query/sql/create-live-view.md | 127 ++++++++++++++---- documentation/query/sql/drop-live-view.md | 7 +- documentation/query/sql/show.md | 5 +- 16 files changed, 285 insertions(+), 93 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 469835b004..3b82ff3147 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -16,15 +16,19 @@ This page tracks significant updates to the QuestDB documentation. ### Reference -- Added the dedicated live view refresh worker pool settings [`live.view.refresh.worker.count`](/docs/configuration/live-views/#liveviewrefreshworkercount), `.affinity`, `.haltOnError`, `.nap.threshold`, `.sleep.threshold`, `.sleep.timeout`, and `.yield.threshold`, plus [`cairo.live.view.checkpoint.purge.interval`](/docs/configuration/live-views/#cairoliveviewcheckpointpurgeinterval). Live views no longer share the materialized view refresh pool +- Added the dedicated live view refresh worker pool settings [`live.view.refresh.worker.count`](/docs/configuration/live-views/#liveviewrefreshworkercount), `.affinity`, `.haltOnError`, `.nap.threshold`, `.sleep.threshold`, `.sleep.timeout`, and `.yield.threshold`, plus [`cairo.live.view.checkpoint.purge.interval`](/docs/configuration/live-views/#cairoliveviewcheckpointpurgeinterval). Live views run on their own refresh pool, not the materialized view pool as previously documented - Added the `checkpoint_seal_failures` column to [`live_views()`](/docs/query/functions/meta/#live_views) and corrected the `lag_seqtxn`, `last_processed_seqtxn`, `lag_micros`, and `writer_stall_micros` descriptions - Corrected the defaults of [`mat.view.refresh.worker.count`](/docs/configuration/materialized-views/#matviewrefreshworkercount) and [`wal.apply.worker.count`](/docs/configuration/wal/#walapplyworkercount), and documented that `mat.view.refresh.worker.count=0` disables materialized view refresh ### Updated -- [Live views](/docs/concepts/live-views/) - Corrected the [supported window functions](/docs/concepts/live-views/#supported-window-functions) to match QuestDB 10.0.1: the statistical functions, EMA, and VWEMA run only in anchored windows, `ksum` does not support `RANGE` frames, `lag` needs a bounded frame or an anchor, and `VARCHAR`, `UUID`, `LONG256`, and `DECIMAL` partition keys are rejected. Added a [rolling standard deviation](/docs/concepts/live-views/#rolling-standard-deviation) recipe for Bollinger bands, rewrote [Freshness](/docs/concepts/live-views/#freshness) to reflect that interval-filtered reads, aggregates, and `ASOF JOIN` see unflushed rows while `LATEST ON` does not, and added the `UPDATE` and `REBASE WAL` invalidation triggers, base WAL retention, and replica freshness -- [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) - Fixed the `OWNED BY` example, which requires a parenthesized query, documented the designated-timestamp projection requirement and the remaining anchored-window rules, and expanded the error reference with verified messages +- [Live views](/docs/concepts/live-views/) - Corrected the [supported window functions](/docs/concepts/live-views/#supported-window-functions) to match QuestDB 10.0.1: the statistical functions, EMA, and VWEMA run only in anchored windows, `ksum` does not support `RANGE` frames, `lag` needs a bounded frame or an anchor, and `VARCHAR`, `UUID`, `LONG256`, `DECIMAL`, `BINARY`, and array partition keys are rejected. Added a [rolling standard deviation](/docs/concepts/live-views/#rolling-standard-deviation) recipe for Bollinger bands, rewrote [Freshness](/docs/concepts/live-views/#freshness) to reflect that interval-filtered reads, aggregates, and `ASOF JOIN` see unflushed rows while `LATEST ON` does not, and added the `UPDATE` and `REBASE WAL` invalidation triggers, base WAL retention, and replica freshness. Documented that an out-of-order base row rewrites view rows already emitted, that `view_status` stays `active` while the view's own WAL is suspended, and what to change in `SHOW CREATE LIVE VIEW` output before replaying it +- [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) - Fixed the `OWNED BY` example, which requires a parenthesized query, documented the designated-timestamp projection requirement and the remaining anchored-window rules, and expanded the error reference with verified messages. Added a grammar for anchored windows and [examples](/docs/query/sql/create-live-view/#examples) of a `RANGE` frame with a `WHERE` filter and an anchored session window - [Live view configuration](/docs/configuration/live-views/) - Corrected the descriptions of the in-memory buffer growth budget, partition compaction threshold, flush retry budgets, and refresh memory limit +- [Shared workers](/docs/configuration/shared-workers/) - The write pool no longer lists WAL apply and materialized view refresh, which run on [dedicated pools](/docs/configuration/shared-workers/#dedicated-pools) together with live view refresh +- [EMA and VWEMA](/docs/query/functions/window-functions/reference/#avg) - Corrected the frame behavior: both reject a frame clause instead of ignoring it +- [ALTER TABLE RESUME WAL](/docs/query/sql/alter-table-resume-wal/), [Monitoring and alerting](/docs/operations/monitoring-alerting/), and [Write-ahead log](/docs/concepts/write-ahead-log/) - Suspended materialized views and live views now point to their own `RESUME WAL` commands, since `ALTER TABLE` fails on them; [`wal_tables()`](/docs/query/functions/meta/#wal_tables) now lists live views +- [ALTER MATERIALIZED VIEW RESUME WAL](/docs/query/sql/alter-mat-view-resume-wal/) - Corrected the Enterprise permission to `RESUME WAL` ## September 2026 diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 82e4de5087..ce4b0c32b9 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -9,9 +9,9 @@ description: A live view is a QuestDB table that stores the incrementally maintained result of a window-function query over a single base table. As new rows arrive in the -base table, the window functions run once per new row and the output is appended -to the view. Querying the live view then scans precomputed rows instead of -reprocessing the base table on every read. +base table, the window functions run once per new row and the results are +written to the view. Querying the live view then scans precomputed rows instead +of reprocessing the base table on every read. Live views target workloads where the same window aggregate is read frequently against high-rate ingestion: rolling VWAP, cumulative volume, running ranks, or @@ -56,7 +56,8 @@ CREATE TABLE trades ( ) TIMESTAMP(timestamp) PARTITION BY DAY; ``` -Create a live view that keeps a 300-row moving average of price per symbol: +Create a live view that keeps a moving average of price per symbol over each +trade and the 300 trades before it: ```questdb-sql title="Live view with a moving average" CREATE LIVE VIEW trades_ma @@ -91,8 +92,8 @@ soon as they are refreshed, without waiting for a flush; see A live view is its own WAL-backed table maintained by a background refresh worker. The worker reads new committed rows from the base table and runs the -view's window functions over them, appending the output. Two independent -cadences govern how that output becomes visible and durable: +view's window functions over them, writing the output to the view. Two +independent cadences govern how that output becomes visible and durable: - **Refresh** runs continuously. As the worker computes output rows, it appends them to an in-memory tier. This is what keeps the view fresh. @@ -103,15 +104,18 @@ Reads combine both tiers. Recent rows are served from the in-memory tier and the older prefix from disk, so a query sees the freshest computed rows without waiting for a flush. +The view is not append-only. When a base row arrives out of order, with a +timestamp below rows the view has already emitted, QuestDB recomputes and +replaces the view's rows at or after that timestamp. A consumer that polls the +view only for rows newer than the last timestamp it has seen misses these +corrections. If the base table receives out-of-order data, re-read a trailing +time range that covers how late base rows can arrive. + Views over [deduplicated](/docs/concepts/deduplication/) base tables, and every live view on a QuestDB Enterprise read-only replica, work differently: they refresh and flush together on the `FLUSH EVERY` cadence, with no in-memory rows ahead of the disk tier. -```questdb-sql title="Show the live view definition" -SHOW CREATE LIVE VIEW trades_ma; -``` - ### Freshness Because refresh publishes to the in-memory tier ahead of flush, queries see @@ -196,8 +200,8 @@ The following rules also apply: `DECIMAL`, `BINARY`, and array keys are rejected with `incremental snapshot is not supported for this function yet`. In a bounded window, cast a `VARCHAR` key to `SYMBOL` or `STRING`, for example - `PARTITION BY venue::symbol`. An anchored window must partition by base - columns directly. + `PARTITION BY venue::symbol` for a `VARCHAR` column `venue`. An anchored + window must partition by base columns directly. - **Frames must not start at `UNBOUNDED PRECEDING`.** A window without a frame clause, such as `OVER (PARTITION BY symbol ORDER BY timestamp)`, defaults to a frame that starts at `UNBOUNDED PRECEDING`. Such windows are rejected unless @@ -257,9 +261,22 @@ LIMIT -10; ``` `greatest(..., 0)` guards against tiny negative values from floating-point -rounding when prices are flat. For the sample standard deviation that -`stddev_samp` returns, multiply the variance by `n / (n - 1)`, where `n` is -`count(price) OVER w`. +rounding when prices are flat. + +For the sample standard deviation that `stddev_samp` returns, multiply the +variance by `n / (n - 1)`, where `n` is `count(price) OVER w`. Write the +denominator as `n - 1.0`: QuestDB divides integers as integers, so `n / (n - 1)` +evaluates to `1` and leaves the population value unchanged. + +```questdb-sql title="Sample standard deviation over the same window" +sqrt(greatest( + (avg(price * price) OVER w - avg(price) OVER w * avg(price) OVER w) + * count(price) OVER w / (count(price) OVER w - 1.0), + 0 +)) AS stddev20_samp +``` + +The first row of each symbol reads `0` here, where `stddev_samp` returns `NULL`. ## Anchored windows @@ -336,13 +353,14 @@ A live view is tied to a single WAL-backed base table and tracks the exact set o base columns its query references, including columns used only in its `WHERE` filter: -- Changes to columns the view does not reference pass through transparently and - the view keeps refreshing. +- Schema changes to columns the view does not reference pass through + transparently and the view keeps refreshing. - Dropping, renaming, or changing the type of a referenced column invalidates the view. - Renaming or dropping the base table invalidates the view. - An [`UPDATE`](/docs/query/sql/update/) that changes base-table rows - invalidates the view. + invalidates the view, even when it changes only columns the view does not + reference. - [`ALTER TABLE REBASE WAL`](/docs/query/sql/alter-table-rebase-wal/) on the base table invalidates the view. - Exceeding @@ -377,8 +395,21 @@ dropping the view: SHOW CREATE LIVE VIEW trades_ma; ``` -Then drop and recreate the live view. Use `START FROM BEGINNING` or an explicit -timestamp if it must include history that is still present in the base table. +Then drop and recreate the live view. Review the saved statement before +replaying it: + +- `START FROM NOW` is saved as written, so replaying it starts the new view at + the time of the replay. To keep the original boundary, replace it with the + `view_lower_bound_timestamp` value from + [`live_views()`](/docs/query/functions/meta/#live_views), or use + `START FROM BEGINNING` to include all history still present in the base table. +- If the view was created with a trailing `;`, the saved query keeps it inside + `AS ( ... )` and replaying fails with `')' expected`. Remove that `;`. +- On QuestDB Enterprise, the saved `OWNED BY` names the user running + `SHOW CREATE`, not necessarily the view's owner, and dropping the view + removes its grants. Set `OWNED BY` to the intended owner and grant + permissions on the new view again. + Dropping the invalid view removes its materialized rows, including rows whose source history is no longer retained by the base table. @@ -390,22 +421,41 @@ ahead of disk, so it trails the base table by up to one `FLUSH EVERY` cycle. ## Monitoring The [`live_views()`](/docs/query/functions/meta/#live_views) function exposes -the state, refresh lag, in-memory footprint, and seed progress of every live -view: +the state, refresh lag, in-memory footprint, and out-of-order repair counters of +every live view: ```questdb-sql title="List all live views" SELECT view_name, base_table_name, view_status, lag_seqtxn, lag_micros FROM live_views(); ``` +Check `view_status` before reading lag. `seeding` means the view is still +processing the base rows that existed when it was created, and `invalid` means +it has stopped refreshing; see [Base table lifecycle](#base-table-lifecycle). +The lag columns are meaningful only for an `active` view: during the seed, +`lag_seqtxn` reads `0` while the view is still empty. + +`view_status` does not show a suspended WAL on the view itself: a view whose own +WAL apply failed still reports `active`. QuestDB retries the apply on the view's +next flush after a base commit, so a view whose base table is idle stays +suspended. Find suspended live views with +[`tables()`](/docs/query/functions/meta/#tables), or with +[`wal_tables()`](/docs/query/functions/meta/#wal_tables) for the error details, +and resume them with +[`ALTER LIVE VIEW RESUME WAL`](/docs/query/sql/alter-live-view-resume-wal/): + +```questdb-sql title="Find live views with a suspended WAL" +SELECT table_name FROM tables() WHERE table_type = 'L' AND table_suspended; +``` + `lag_seqtxn` is the number of transactions applied to the base table beyond the view's durable processed watermark. It counts transactions, not rows: one -transaction may contain one row or millions. A value of `0` means that the -durable tier has caught up with the base table. A temporary non-zero value is -expected between `FLUSH EVERY` cycles, especially when ingestion produces many -small commits. Base commits that are not yet applied to the base table are not -counted, so when the base table itself lags or is suspended, check -[`wal_tables()`](/docs/query/functions/meta/#wal_tables) too. +transaction may contain one row or millions. For an `active` view, a value of +`0` means that the durable tier has caught up with the base table. A temporary +non-zero value is expected between `FLUSH EVERY` cycles, especially when +ingestion produces many small commits. Base commits that are not yet applied to +the base table are not counted, so when the base table itself lags or is +suspended, check [`wal_tables()`](/docs/query/functions/meta/#wal_tables) too. There is no universal acceptable non-zero value. Sample the metric over time and compare it with the view's normal flush-cycle baseline. A bounded sawtooth that @@ -465,7 +515,8 @@ A full rebuild of a materialized view invalidates a live view that uses it as its base. Recreate the live view after the rebuild, following the recovery steps in [Base table lifecycle](#base-table-lifecycle). -- **Single base table only.** No JOINs, subqueries, or CTEs in the view query. +- **Single base table only.** No JOINs, subqueries, CTEs, or regular views in + `FROM`. - **Explicit output columns only.** `SELECT *` and other wildcard projections are rejected because the output schema is fixed when the view is created. - **Designated timestamp in the output.** The query must select the base @@ -479,7 +530,9 @@ in [Base table lifecycle](#base-table-lifecycle). `sysdate()`, `systimestamp()`, and `rnd_*()` are rejected in the projection, the `WHERE` filter, and window-function arguments. - **No TTL on the view.** Live-view disk growth is unbounded in this version. - Size retention on the base table instead. + Base-table retention does not shrink the view, because `DROP PARTITION`, + `TRUNCATE`, and TTL eviction on the base table leave the view's rows in place. + The only way to reclaim the space is to drop and recreate the view. - **No direct writes.** `INSERT`, `UPDATE`, `TRUNCATE`, and structural `ALTER` statements on a live view are rejected with `cannot modify live view`. @@ -495,8 +548,8 @@ in [Base table lifecycle](#base-table-lifecycle). - **No admission control.** A view that cannot keep up with ingestion stays correct but stale, with no automatic throttle or drop. - **Per-partition state for partitioned windows grows with distinct partition - cardinality.** A base table with high-cardinality partition keys (UUIDs, - session ids) holds state per key. Use + cardinality.** A view that partitions by a high-cardinality key, such as a + `SYMBOL` column of account or order ids, holds state per key. Use [`cairo.live.view.refresh.memory.limit.bytes`](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) to cap the per-view refresh footprint; a view that exceeds the cap is invalidated, not throttled. The `in_mem_bytes` column in @@ -542,8 +595,7 @@ so a view invalidated on one node, for example by a memory limit breach, stays valid on the others. A role switch continues the local refresh state; it does not reconstruct or -transfer the former primary's live-view rows. Replica freshness therefore also -depends on base-table replication and apply lag. +transfer the former primary's live-view rows. ### Backup and restore diff --git a/documentation/concepts/write-ahead-log.md b/documentation/concepts/write-ahead-log.md index 1114ca373b..10b0e13029 100644 --- a/documentation/concepts/write-ahead-log.md +++ b/documentation/concepts/write-ahead-log.md @@ -88,6 +88,11 @@ If WAL transactions are suspended (rare), resume them: ALTER TABLE prices RESUME WAL; ``` +For a suspended materialized view or live view, use +[`ALTER MATERIALIZED VIEW RESUME WAL`](/docs/query/sql/alter-mat-view-resume-wal/) +or [`ALTER LIVE VIEW RESUME WAL`](/docs/query/sql/alter-live-view-resume-wal/) +instead. + ## How WAL works When data is written to a WAL table: diff --git a/documentation/configuration/live-views.md b/documentation/configuration/live-views.md index b76e599e50..ff462a0912 100644 --- a/documentation/configuration/live-views.md +++ b/documentation/configuration/live-views.md @@ -1,6 +1,8 @@ --- title: Live views -description: Configuration settings for live views in QuestDB. +description: + "Live view server settings: refresh worker pool size, per-view memory limit, + in-memory tier buffers, checkpoints, and out-of-order repair budgets." --- These settings control live view SQL support, the refresh worker pool, and the @@ -159,8 +161,9 @@ Threshold for the sweep that drops anchored-window partition state left behind the previous anchor boundary. The sweep runs only when all of the following hold: the anchor map has more partitions than this value, at least this many of them are reclaimable, and the reclaimable partitions make up at least half of -the map. A view whose anchor value is not monotonic, or is `NULL`, never -compacts. +the map. Only anchors that return `TIMESTAMP` and advance with the designated +timestamp compact. An anchor value that is not monotonic or is `NULL`, and any +`ANCHOR EXPRESSION` that returns `LONG` or `INT`, never compacts. ## cairo.live.view.refresh.memory.limit.bytes diff --git a/documentation/configuration/materialized-views.md b/documentation/configuration/materialized-views.md index e8273329c4..71f7697e68 100644 --- a/documentation/configuration/materialized-views.md +++ b/documentation/configuration/materialized-views.md @@ -6,8 +6,9 @@ description: --- These settings control materialized view SQL support and the background refresh -job. Materialized views can use dedicated worker threads or share the server's -common pool. +job. Materialized views refresh on a dedicated worker pool sized by +[`mat.view.refresh.worker.count`](#matviewrefreshworkercount); `0` disables +refresh. To cap the native memory a single refresh may allocate, see [`cairo.mat.view.refresh.memory.limit.bytes`](/docs/configuration/cairo-engine/#memory-limits). @@ -79,8 +80,10 @@ Comma-separated list of numerical CPU core indexes, one per refresh worker. ## mat.view.refresh.worker.count -- **Default**: same as - [`wal.apply.worker.count`](/docs/configuration/wal/#walapplyworkercount) +- **Default**: `2` on hosts with up to 8 CPU cores, `3` on up to 32 cores, `4` + on larger hosts. This is the same CPU-based default as + [`wal.apply.worker.count`](/docs/configuration/wal/#walapplyworkercount); + setting that key does not change this one. - **Reloadable**: no Number of dedicated worker threads assigned to refresh materialized views. A diff --git a/documentation/configuration/shared-workers.md b/documentation/configuration/shared-workers.md index e585b57651..e577c7a02c 100644 --- a/documentation/configuration/shared-workers.md +++ b/documentation/configuration/shared-workers.md @@ -9,8 +9,8 @@ query execution, and write operations based on your workload profile. - **Network pool**: handles HTTP, PostgreSQL, and ILP server I/O - **Query pool**: executes parallel query operations (filters, group-by) -- **Write pool**: manages WAL apply jobs, table writes, materialized view - refresh, and housekeeping tasks +- **Write pool**: manages table writes and housekeeping tasks, and WAL apply + jobs when `wal.apply.worker.count` is `0` All three pools use the same default sizing formula, which scales with the number of available CPUs: @@ -26,6 +26,18 @@ server, each pool gets 62 threads. Adjust these values when you need to shift CPU capacity toward a specific workload, for example reducing the network pool and increasing the query pool on a read-heavy system. +## Dedicated pools + +WAL apply, materialized view refresh, and live view refresh run on their own +pools, not on the write pool. Each is sized by its own key: +[`wal.apply.worker.count`](/docs/configuration/wal/#walapplyworkercount), +[`mat.view.refresh.worker.count`](/docs/configuration/materialized-views/#matviewrefreshworkercount), +and +[`live.view.refresh.worker.count`](/docs/configuration/live-views/#liveviewrefreshworkercount). +Each defaults to `2` on hosts with up to 8 CPU cores, `3` on up to 32 cores, and +`4` on larger hosts. Setting `wal.apply.worker.count` to `0` moves WAL apply to +the write pool, while `0` for either refresh pool disables that refresh. + ## shared.network.worker.affinity - **Default**: none @@ -85,7 +97,8 @@ Comma-delimited list of CPU ids, one per thread specified in - **Default**: see table above - **Reloadable**: no -Number of worker threads for the write pool, which manages WAL apply jobs, -table writes, materialized view refresh, and housekeeping tasks. Increasing -this value raises write parallelism at the expense of CPU resources available -to network I/O and queries. +Number of worker threads for the write pool, which manages table writes and +housekeeping tasks, and WAL apply jobs when `wal.apply.worker.count` is `0`. +Increasing this value raises write parallelism at the expense of CPU resources +available to network I/O and queries. See [Dedicated pools](#dedicated-pools) +for the WAL apply and view refresh pools. diff --git a/documentation/operations/monitoring-alerting.md b/documentation/operations/monitoring-alerting.md index 5fdf1ce490..41e8970b22 100644 --- a/documentation/operations/monitoring-alerting.md +++ b/documentation/operations/monitoring-alerting.md @@ -91,9 +91,16 @@ the table. **Detection:** ```questdb-sql -SELECT table_name FROM tables() WHERE table_suspended; +SELECT table_name, table_type FROM tables() WHERE table_suspended; ``` +The result includes suspended materialized views (`table_type = 'M'`) and live +views (`table_type = 'L'`). Resume those with +[`ALTER MATERIALIZED VIEW RESUME WAL`](/docs/query/sql/alter-mat-view-resume-wal/) +or [`ALTER LIVE VIEW RESUME WAL`](/docs/query/sql/alter-live-view-resume-wal/); +`ALTER TABLE ... RESUME WAL` fails on views with `cannot modify materialized view` +or `cannot modify live view`. + To see why a table was suspended, query `wal_tables()`. Its `errorTag` column reads `OUT OF MEMORY` when the cause was a memory limit breach: @@ -103,7 +110,7 @@ SELECT name, errorTag, errorMessage FROM wal_tables() WHERE suspended; **Resolution:** -Resume from the failed transaction: +Resume a suspended table from the failed transaction: ```questdb-sql ALTER TABLE my_table RESUME WAL; diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 60ae8195e9..0415ac70b6 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -188,7 +188,7 @@ Returns a `table` with the following columns: | `view_table_dir_name` | STRING | View directory name on disk | | `base_table_name` | STRING | Base table name | | `view_sql` | STRING | Query used to maintain the view | -| `view_status` | STRING | Lifecycle status: `creating`, `active`, `seeding`, `invalid`, `dropping`, `version_unsupported`, or `state_unreadable` | +| `view_status` | STRING | Lifecycle status: `creating`, `active`, `seeding`, `invalid`, `dropping`, `version_unsupported`, or `state_unreadable`. A suspended WAL on the view still shows `active`; check [`wal_tables()`](#wal_tables) | | `invalidation_reason` | STRING | Message explaining why the view was marked invalid | | `flush_every_interval` | LONG | `FLUSH EVERY` interval value | | `flush_every_interval_unit` | STRING | `FLUSH EVERY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | @@ -1282,8 +1282,8 @@ and `errorMessage` of a suspended table. ::: `wal_tables()` returns the WAL status for all -[WAL tables](/docs/concepts/write-ahead-log/) or materialized views in the -database. +[WAL tables](/docs/concepts/write-ahead-log/), materialized views, and live +views in the database. **Arguments:** @@ -1293,7 +1293,7 @@ database. Returns a `table` including the following information: -- `name` - table or materialized view name +- `name` - table, materialized view, or live view name - `suspended` - suspended status flag - `writerTxn` - the last committed transaction in TableWriter (equivalent to `table_txn` in `tables()`) - `bufferedTxnSize` - the number of transactions that are kept invisible when diff --git a/documentation/query/functions/window-functions/reference.md b/documentation/query/functions/window-functions/reference.md index 4ef90af291..0007a889ad 100644 --- a/documentation/query/functions/window-functions/reference.md +++ b/documentation/query/functions/window-functions/reference.md @@ -121,7 +121,7 @@ WHERE timestamp IN '$today'; - NULL values are skipped; the previous EMA value is preserved - The first non-NULL value initializes the EMA - Works with both `TIMESTAMP` and `TIMESTAMP_NS` precision -- EMA ignores the frame clause and operates cumulatively from the first row +- EMA rejects a frame clause (`avg() does not support framing; remove ROWS/RANGE clause`) and accumulates from the first row of the partition ::: #### Volume-Weighted Exponential Moving Average (VWEMA) @@ -180,7 +180,7 @@ WHERE timestamp IN '$today'; :::note VWEMA behavior - Rows with NULL price, NULL volume, or zero volume are skipped - When timestamps are identical (Δt = 0), the row is skipped in time-weighted mode -- VWEMA ignores the frame clause and operates cumulatively from the first row +- VWEMA rejects a frame clause (`avg() does not support framing; remove ROWS/RANGE clause`) and accumulates from the first row of the partition ::: --- diff --git a/documentation/query/sql/alter-live-view-resume-wal.md b/documentation/query/sql/alter-live-view-resume-wal.md index f49e3bc0b1..e58b004412 100644 --- a/documentation/query/sql/alter-live-view-resume-wal.md +++ b/documentation/query/sql/alter-live-view-resume-wal.md @@ -2,7 +2,8 @@ title: ALTER LIVE VIEW RESUME WAL sidebar_label: RESUME WAL description: - Documentation for the ALTER LIVE VIEW RESUME WAL SQL keyword in QuestDB. + ALTER LIVE VIEW RESUME WAL restarts WAL apply on a live view suspended after an + error such as a full disk. Find suspended live views with wal_tables(). --- Restarts [WAL](/docs/concepts/write-ahead-log/) apply on a @@ -21,13 +22,16 @@ ALTER LIVE VIEW viewName RESUME WAL | Parameter | Description | | --------- | ----------- | | `viewName` | Name of the live view to resume | -| `FROM TRANSACTION` | Optional starting transaction number (defaults to the failed transaction) | +| `FROM TRANSACTION` or `FROM TXN` | Optional transaction to resume from. Defaults to the failed transaction | +| `sequencerTxn` | A transaction number of the view's own WAL, as reported by `wal_tables()`, not a base-table transaction | ## When to use A live view writes its output to its own WAL-backed table. If applying one of the view's WAL transactions fails, QuestDB suspends the view's WAL like that of -any other WAL table, and the view can serve stale rows until you resume it. +any other WAL table. The view's next flush after a base-table commit retries +the apply and clears the suspension if the cause is gone, but a view whose base +table is idle stays suspended until you resume it. [`live_views()`](/docs/query/functions/meta/#live_views) still reports `view_status = 'active'` for a suspended view, so check [`wal_tables()`](/docs/query/functions/meta/#wal_tables) instead: @@ -66,11 +70,14 @@ GRANT RESUME WAL ON trades_ma TO user1; ## Errors +Messages are shown by their stable prefix. The server may append details, such +as `[name=trades_ma]`. + | Error | Cause | | ----- | ----- | | `live view name expected` | The name refers to a table or view that is not a live view | | `table does not exist` | No table or view with this name exists | -| `'resume' or 'suspend' expected` | The statement uses an unsupported `ALTER LIVE VIEW` operation, such as `SET TTL` or `ADD COLUMN` | +| `'resume' or 'suspend' expected` | The statement uses an unsupported `ALTER LIVE VIEW` operation, such as `SET TTL` or `ADD COLUMN`. Only `RESUME WAL` is supported | ## See also diff --git a/documentation/query/sql/alter-mat-view-resume-wal.md b/documentation/query/sql/alter-mat-view-resume-wal.md index 1c8dd5aacf..5b3a9eb73a 100644 --- a/documentation/query/sql/alter-mat-view-resume-wal.md +++ b/documentation/query/sql/alter-mat-view-resume-wal.md @@ -86,11 +86,11 @@ ALTER MATERIALIZED VIEW trades_1h RESUME WAL FROM TRANSACTION 5; ## Permissions (Enterprise) -Resuming WAL on a materialized view requires the `ALTER MATERIALIZED VIEW` -permission on the specific view: +Resuming WAL on a materialized view requires the `RESUME WAL` permission on the +view: -```questdb-sql title="Grant alter permission" -GRANT ALTER MATERIALIZED VIEW ON trades_1h TO user1; +```questdb-sql title="Grant resume permission" +GRANT RESUME WAL ON trades_1h TO user1; ``` ## Errors diff --git a/documentation/query/sql/alter-table-rebase-wal.md b/documentation/query/sql/alter-table-rebase-wal.md index be4ba2dba8..0c4aeab699 100644 --- a/documentation/query/sql/alter-table-rebase-wal.md +++ b/documentation/query/sql/alter-table-rebase-wal.md @@ -45,6 +45,9 @@ table instead: - [Materialized views](/docs/concepts/materialized-views/) that depend on the table are fully refreshed, because their watermarks no longer map onto the reset sequencer. +- [Live views](/docs/concepts/live-views/) on the table are invalidated. Drop + and recreate them as described in + [Base table lifecycle](/docs/concepts/live-views/#base-table-lifecycle). ## Requirements diff --git a/documentation/query/sql/alter-table-resume-wal.md b/documentation/query/sql/alter-table-resume-wal.md index b2d453d635..393ac4c1a2 100644 --- a/documentation/query/sql/alter-table-resume-wal.md +++ b/documentation/query/sql/alter-table-resume-wal.md @@ -38,6 +38,16 @@ table; `OUT OF MEMORY` means a WAL apply batch ran out of memory, by breaching its own [memory limit](/docs/configuration/cairo-engine/#memory-limits) or the process-wide one. +:::note Materialized views and live views + +`ALTER TABLE ... RESUME WAL` does not work on views: it fails with +`cannot modify materialized view` or `cannot modify live view`. For a suspended +materialized view or live view, use +[`ALTER MATERIALIZED VIEW RESUME WAL`](/docs/query/sql/alter-mat-view-resume-wal/) +or [`ALTER LIVE VIEW RESUME WAL`](/docs/query/sql/alter-live-view-resume-wal/). + +::: + ## Examples Using the [`wal_tables()`](/docs/query/functions/meta/#wal_tables) function diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index ee0a0f3397..17e09e2ecd 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -2,7 +2,8 @@ title: CREATE LIVE VIEW sidebar_label: CREATE LIVE VIEW description: - Documentation for the CREATE LIVE VIEW SQL keyword in QuestDB. + "CREATE LIVE VIEW syntax for incrementally maintained window-function views: + FLUSH EVERY, IN MEMORY, START FROM, anchored windows, query rules, and errors." --- Creates a live view that incrementally maintains the result of a window-function @@ -24,18 +25,20 @@ rejected at creation time. CREATE LIVE VIEW [ IF NOT EXISTS ] viewName FLUSH EVERY duration [ IN MEMORY duration ] -[ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) ] -START FROM ( NOW | BEGINNING | 'timestamp' ) -AS [ ( ] query [ ) ] -[ OWNED BY ownerName ] +[ PARTITION BY { YEAR | MONTH | WEEK | DAY | HOUR } ] +START FROM { NOW | BEGINNING | 'startTimestamp' } +AS { query | ( query ) [ OWNED BY ownerName ] } ``` Where: - `duration`: a single token with a lowercase unit of `ms`, `s`, `m`, `h`, or `d`, for example `100ms`, `5s`, or `30m`. +- `startTimestamp`: a quoted timestamp literal, such as + `'2026-04-01T00:00:00.000000Z'` or the date-only `'2026-04-01'`. - `query`: a `SELECT` over one WAL-backed base table whose projection contains - [window functions](/docs/query/functions/window-functions/overview/). + [window functions](/docs/query/functions/window-functions/overview/). It may + declare an [anchored window](#anchored-windows). `FLUSH EVERY` is required and must come first. `START FROM` is also required and may appear in any order with the optional `IN MEMORY` and `PARTITION BY` @@ -163,11 +166,14 @@ FROM trades; ## Anchored windows An anchored window resets its functions on a boundary. Declare it in a named -`WINDOW` with one of these forms: +`WINDOW` clause of the view query, with `ANCHOR` as the last item: -```questdb-sql -ANCHOR DAILY 'HH:MM' [ 'timezone' ] -ANCHOR EXPRESSION expression +```questdb-sql title="Anchored named window" +WINDOW windowName AS ( + PARTITION BY columnName [, columnName ...] + ORDER BY timestampColumn [ ASC ] + ANCHOR { DAILY 'HH:MM' [ 'timeZone' ] | EXPRESSION anchorExpression } +) ``` `ANCHOR DAILY` requires a quoted 24-hour time. An optional time zone, either an @@ -222,7 +228,9 @@ A live view supports at most one anchored window, which several window functions can share alongside other, bounded windows. An `ANCHOR EXPRESSION` must be deterministic, non-constant, and return `TIMESTAMP`, `LONG`, or `INT`. It cannot contain aggregates, subqueries, or bind variables, and it can only -reference base columns that the query already selects or filters on. +reference base columns that the rest of the query already reads: in the +projection, the `WHERE` filter, a window-function argument, or a +`PARTITION BY`. Otherwise it fails with `Invalid column: `. ## Query constraints @@ -239,7 +247,9 @@ The view query is validated at creation time and must: [Supported window functions](/docs/concepts/live-views/#supported-window-functions). - Give every stateful window function a `PARTITION BY` clause on a supported key type. `VARCHAR`, `UUID`, `LONG256`, `DECIMAL`, `BINARY`, and array keys are - rejected. + rejected. In a bounded window, cast a `VARCHAR` key to `SYMBOL` or `STRING`, + for example `PARTITION BY venue::symbol`; an anchored window must partition + by base columns directly. - Use a bounded `ROWS` or `RANGE` frame, or a named anchored window. Ranking functions (`row_number`, `rank`, and `dense_rank`) must be anchored. A window without a frame clause, and any frame starting at `UNBOUNDED PRECEDING`, is @@ -257,7 +267,9 @@ The view query is validated at creation time and must: `systimestamp()`, or `rnd_*()`. - Not read another live view. -## Complete example +## Examples + +The examples use this base table: ```questdb-sql title="Base table" CREATE TABLE trades ( @@ -269,6 +281,8 @@ CREATE TABLE trades ( ) TIMESTAMP(timestamp) PARTITION BY DAY; ``` +### Fully specified live view + ```questdb-sql title="Fully specified live view" CREATE LIVE VIEW IF NOT EXISTS trades_ma FLUSH EVERY 1s @@ -294,7 +308,68 @@ This creates a view that: - Keeps 5 seconds of recent rows in RAM for fresh reads (`IN MEMORY 5s`) - Partitions its disk tier by hour (`PARTITION BY HOUR`) - Includes all existing history in `trades` (`START FROM BEGINNING`) -- Keeps a 300-row moving average of price per symbol +- Keeps a moving average of price per symbol over each trade and the 300 trades + before it + +### Rolling VWAP with a filter + +This view uses a time-based `RANGE` frame and a `WHERE` filter. It keeps a +one-hour volume-weighted average price per symbol over buy trades only; sell +trades neither appear in the view nor contribute to the window. + +```questdb-sql title="Rolling 1-hour VWAP of buy trades" +CREATE LIVE VIEW trades_buy_vwap_1h +FLUSH EVERY 1s +START FROM NOW +AS +SELECT + timestamp, + symbol, + price, + sum(price * amount) OVER w / sum(amount) OVER w AS vwap_1h +FROM trades +WHERE side = 'buy' +WINDOW w AS ( + PARTITION BY symbol + ORDER BY timestamp + RANGE '1' HOUR PRECEDING +); +``` + +### Session statistics from the market open + +This view resets an anchored window at 09:30 New York time, following +daylight-saving transitions. The trade sequence number, running volume, EMA, +VWEMA, and price change all restart at each open, and `lag` returns `NULL` for +the first trade of each session. The bounded `sma20` window runs alongside the +anchored one and does not reset. + +```questdb-sql title="Session stats from the New York open" +CREATE LIVE VIEW trades_ny_session +FLUSH EVERY 1s +START FROM NOW +AS +SELECT + timestamp, + symbol, + price, + row_number() OVER s AS trade_seq, + sum(amount) OVER s AS session_volume, + avg(price, 'period', 20) OVER s AS ema20, + avg(price, 'period', 20, amount) OVER s AS vwema20, + price - lag(price) OVER s AS price_change, + avg(price) OVER ( + PARTITION BY symbol + ORDER BY timestamp + ROWS 19 PRECEDING + ) AS sma20 +FROM trades +WINDOW s AS ( + PARTITION BY symbol + ORDER BY timestamp + ANCHOR DAILY '09:30' 'America/New_York' +); +``` ## Metadata @@ -342,13 +417,17 @@ AS ( OWNED BY analysts; ``` -External users must specify `OWNED BY`, and the creator must be a member of the -group named as the owner. +The example works as written for the built-in admin, which can name any +existing user, group, or service account as the owner. Any other creator can +name only themselves or a group they are a member of; naming another group fails +with ` is not part of group `. External users must specify +`OWNED BY` and can name only one of their groups. ## Errors -Errors that end in a bracketed detail, such as `[name=trades]`, are shown -without it. +Messages are shown by their stable prefix. The server may append details, such +as `[name=trades]`, the configured limit in parentheses, or a hint on how to fix +the query. ### Statement errors @@ -377,7 +456,7 @@ without it. | `live views are not allowed as base tables in V1` | The base object is another live view | | `live view requires a single base table in FROM clause` | `FROM` contains a subquery, a CTE, or a regular view | | `live view select must be a simple scan of a single WAL base table; ...` | The query joins tables or uses `LATEST ON` | -| `live view select must be a plain windowed scan of the base table; ...` | The query uses `LIMIT` or `DISTINCT` | +| `live view select must be a plain windowed scan of the base table; ...` | The query uses `LIMIT`, `DISTINCT`, or a top-level `ORDER BY` on a column other than the designated timestamp | | `live view select must contain at least one window function` | The projection has no window function | | `live view select must project the base table's designated timestamp` | The designated timestamp is missing from the projection or computed | | `live view select cannot override the designated timestamp` | The designated timestamp is aliased | @@ -390,19 +469,21 @@ without it. | Error | Cause | | ----- | ----- | | `live view select cannot use window function (); incremental snapshot is not supported for this function yet` | The function does not support this frame, has no `PARTITION BY`, or partitions by an unsupported key type | -| `live view select may only use window functions that support incremental refresh` | The function needs look-ahead or multiple passes (`percent_rank`, `cume_dist`, `ntile`), or a window or top-level `ORDER BY` is not the designated timestamp ascending | +| `live view select may only use window functions that support incremental refresh` | The function needs look-ahead or multiple passes (`percent_rank`, `cume_dist`, `ntile`), an inline `OVER (...)` window does not order by the designated timestamp ascending, or the top-level `ORDER BY` is on the designated timestamp descending | | `lead() is not supported in live views; use lag() for lookback` | The query uses `lead` | -| `live view unbounded window must have an ANCHOR clause` | A window without a frame clause is not anchored | -| `live view select cannot use () over a frame starting at UNBOUNDED PRECEDING` | An explicit frame starts at `UNBOUNDED PRECEDING` and is not anchored | +| `live view unbounded window must have an ANCHOR clause` | The window uses the default frame (no frame clause, or an explicit `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`) and is not anchored | +| `live view select cannot use () over a frame starting at UNBOUNDED PRECEDING` | Another frame starting at `UNBOUNDED PRECEDING`, such as `ROWS UNBOUNDED PRECEDING`, is not anchored | | `live view select cannot use () without an anchored WINDOW` | A ranking function is not anchored | | `live view RANGE window functions must use the same PARTITION BY and ORDER BY domain` | Two `RANGE` windows partition or order differently | -| `live view named WINDOW must ORDER BY ASC` | A named window does not order by the designated timestamp ascending | +| `live view named WINDOW must ORDER BY ` | A named window does not order by the designated timestamp ascending | +| `avg() does not support framing; remove ROWS/RANGE clause` | An EMA or VWEMA call, `avg(value, kind, param [, volume])`, has a frame clause | | `ANCHOR is only supported on named WINDOW clauses` | `ANCHOR` appears in an inline `OVER (...)` | | `ANCHOR is incompatible with bounded frames` | An anchored window has an explicit frame | | `live view anchored WINDOW requires PARTITION BY` | An anchored window has no `PARTITION BY` | | `live view ANCHOR currently requires PARTITION BY to reference base columns directly` | An anchored window partitions by an expression | | `live view supports at most one anchored WINDOW in V1` | The query declares more than one anchored window | | `live view anchored WINDOW '' is not referenced by any window function` | No window function uses the anchored window directly | +| `Invalid column: ` | An `ANCHOR EXPRESSION` references a base column that the rest of the query does not read | ## See also diff --git a/documentation/query/sql/drop-live-view.md b/documentation/query/sql/drop-live-view.md index 9686338d7a..f6ec34986a 100644 --- a/documentation/query/sql/drop-live-view.md +++ b/documentation/query/sql/drop-live-view.md @@ -60,10 +60,13 @@ The view creator automatically receives this permission with the `GRANT` option. ## Errors +Messages are shown by their stable prefix. The server may append details, such +as `[name=trades_ma]`. + | Error | Cause | | ----- | ----- | -| `live view name expected [name=]` | The name refers to a table or view that is not a live view, even with `IF EXISTS` | -| `live view does not exist [name=]` | The view does not exist and `IF EXISTS` was not specified | +| `live view name expected` | The name refers to a table or view that is not a live view, even with `IF EXISTS` | +| `live view does not exist` | The view does not exist and `IF EXISTS` was not specified | | `Access denied for [DROP LIVE VIEW on ]` | Missing `DROP LIVE VIEW` permission (Enterprise) | ## See also diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 06188b45cf..d25e03a2d5 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -222,7 +222,8 @@ SHOW CREATE LIVE VIEW trades_ma; This returns the `CREATE LIVE VIEW` statement that would recreate the [live view](/docs/concepts/live-views/), including its `FLUSH EVERY`, `IN MEMORY`, `PARTITION BY`, and `START FROM` clauses. On QuestDB Enterprise the -output also carries an `OWNED BY` clause populated with the current user. +output also carries an `OWNED BY` clause naming the user running `SHOW CREATE`, +not necessarily the view's owner. ### SHOW CREATE MATERIALIZED VIEW @@ -319,7 +320,7 @@ policy is not shown in `SHOW CREATE TABLE`. See #### Enterprise variant -[QuestDB Enterprise](/enterprise/) will include an additional `OWNED BY` clause populated with the current user. +[QuestDB Enterprise](/enterprise/) will include an additional `OWNED BY` clause naming the user running `SHOW CREATE`, not necessarily the table's owner. For example,