diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index a0bdd6bc94..1c99cc8d79 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -16,11 +16,18 @@ This page tracks significant updates to the QuestDB documentation. ### New +- [Memory limits](/docs/configuration/cairo-engine/#memory-limits) - New section covering the per-query, materialized view refresh, WAL apply, and live view refresh memory limits, what counts toward them, and what happens on a breach, plus the previously undocumented [`cairo.mat.view.max.refresh.retries`](/docs/configuration/materialized-views/#cairomatviewmaxrefreshretries), [`cairo.mat.view.refresh.busy.retry.limit`](/docs/configuration/materialized-views/#cairomatviewrefreshbusyretrylimit), [`cairo.mat.view.refresh.busy.retry.timeout`](/docs/configuration/materialized-views/#cairomatviewrefreshbusyretrytimeout), [`cairo.write.back.off.timeout.on.mem.pressure`](/docs/configuration/cairo-engine/#cairowritebackofftimeoutonmempressure), [`ram.usage.limit.bytes`](/docs/configuration/cairo-engine/#ramusagelimitbytes), and [`ram.usage.limit.percent`](/docs/configuration/cairo-engine/#ramusagelimitpercent) keys +- [RBAC memory limits](/docs/security/rbac/#memory-limits) - Per-user, per-group, and per-service-account query memory limits in QuestDB Enterprise: `SET MEMORY LIMIT` on `ALTER USER`, `ALTER GROUP`, and `ALTER SERVICE ACCOUNT`, how limits resolve, the `SET MEMORY LIMIT` permission, and the upgrade migration +- [ALTER GROUP](/docs/query/sql/acl/alter-group/) - New reference page covering `SET MEMORY LIMIT` and external alias mapping - [Migrate QuestDB onto the Kubernetes Operator](/docs/enterprise-kubernetes-operator/getting-started/migrate/) - Move an existing Enterprise deployment onto the Operator with a replica-first cutover: restore the source backup, consume replication WAL, then promote after a controlled source drain - [Copy a schema to another instance](/docs/cookbook/operations/copy-schema-between-instances/) - Recreate one instance's tables, views, and materialized views on another from `SHOW CREATE DATABASE`, either by replaying the statements over the REST API or by dumping them to a `.sql` file, with `INCLUDE (SCHEMA)` and `INCLUDE (ACL)` for separating structure from permissions on Enterprise, plus the Web Console schema explorer as a manual alternative and the ordering caveat that comes with it ### Updated +- [query_activity()](/docs/query/functions/meta/#query_activity) - Documented the `is_wal`, `memory_used`, and `memory_limit` columns +- [wal_tables()](/docs/query/functions/meta/#wal_tables) - Documented the `errorTag`, `errorMessage`, and `memoryPressure` columns; `errorTag` reads `OUT OF MEMORY` after a WAL apply memory limit breach +- [SHOW](/docs/query/sql/show/) - `SHOW USERS`, `SHOW GROUPS`, and `SHOW SERVICE ACCOUNTS`, including their filtered forms, gain a trailing `memory_limit` column. Clients that read these results by position need [updating](/docs/security/rbac/#memory-limit-upgrade) +- [CREATE GROUP](/docs/query/sql/acl/create-group/) - Documented the `WITH EXTERNAL ALIAS` form that creates a group and its OIDC or LDAP mapping in one statement - [Kafka connector](/docs/connect/message-brokers/kafka/) - Updated for QWP with a quick start, guidance on preventing duplicates and recovering from outages, and migration steps for existing HTTP pipelines - [Kubernetes Operator](/docs/enterprise-kubernetes-operator/) - Refreshed for Operator 0.2.1 across installation, configuration, high availability, backup and restore, known limitations, troubleshooting, and the generated [API reference](/docs/enterprise-kubernetes-operator/reference/api/) - [Rust](/docs/connect/clients/rust/) and [C and C++](/docs/connect/clients/c-and-cpp/) clients - Documented the UUID byte order, which was previously unstated. `Buffer::column_uuid` takes `(lo, hi)` where `hi` is the most significant half, matching `java.util.UUID`, and the chunk setter takes the 16 bytes in canonical RFC-4122 order diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 3271bd5161..143d7a66f9 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -236,6 +236,11 @@ base columns its query references: - Dropping, renaming, or changing the type of a referenced column invalidates the view. - Renaming or dropping the base table invalidates the view. +- Exceeding + [`cairo.live.view.refresh.memory.limit.bytes`](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) + during a refresh invalidates the view. Before setting that limit, measure + what a refresh needs as described in + [Sizing a limit](/docs/configuration/cairo-engine/#sizing-a-limit). - `DROP PARTITION`, `TRUNCATE`, and base TTL eviction freeze the already-emitted rows and the view continues forward from where it was. @@ -245,8 +250,11 @@ Invalidation is permanent: reversing the schema change does not automatically revalidate the view, and `ALTER LIVE VIEW ... RESUME WAL` only recovers a suspended WAL writer. -To recover, inspect `invalidation_reason`, repair the base-table schema, and -save the definition before dropping the view: +To recover, inspect `invalidation_reason`, repair the base-table schema or +raise +[`cairo.live.view.refresh.memory.limit.bytes`](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) +when the reason is a memory limit breach, and save the definition before +dropping the view: ```questdb-sql SHOW CREATE LIVE VIEW trades_ma; diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index cf7ed4ac26..d0bd267710 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -454,6 +454,13 @@ modified in incompatible ways: - Renaming the base table - `TRUNCATE` or `UPDATE` operations +A view is also invalidated when its refresh keeps failing with an out-of-memory +error, including a breach of the +[refresh memory limit](/docs/configuration/cairo-engine/#memory-limits), after +the deferred retries are exhausted. Before setting that limit, measure what a +refresh needs by running the view's query over one refresh worth of data, as +described in [Sizing a limit](/docs/configuration/cairo-engine/#sizing-a-limit). + Check for invalid views: ```questdb-sql title="Find invalid views" @@ -464,7 +471,10 @@ WHERE view_status = 'invalid'; ### Refreshing an invalid view -To restore an invalid view with a full refresh: +Restore an invalid view with a full refresh. If `invalidation_reason` reports +a memory limit breach, raise +[`cairo.mat.view.refresh.memory.limit.bytes`](/docs/configuration/cairo-engine/#memory-limits) +first, because the full refresh runs under the same limit: ```questdb-sql REFRESH MATERIALIZED VIEW view_name FULL; diff --git a/documentation/concepts/write-ahead-log.md b/documentation/concepts/write-ahead-log.md index a8454f044e..1114ca373b 100644 --- a/documentation/concepts/write-ahead-log.md +++ b/documentation/concepts/write-ahead-log.md @@ -122,6 +122,8 @@ WAL behavior can be tuned via server configuration: - `cairo.wal.enabled.default` — WAL enabled by default (default: `true`) - Parallel threads for WAL application — see [WAL configuration](/docs/configuration/wal/) +- `cairo.wal.apply.memory.limit.bytes` — cap on the native memory a WAL apply + batch may allocate; see [memory limits](/docs/configuration/cairo-engine/#memory-limits) To convert an existing table between WAL and non-WAL: diff --git a/documentation/configuration/cairo-engine.md b/documentation/configuration/cairo-engine.md index e809a8adc4..8ebbf7f74d 100644 --- a/documentation/configuration/cairo-engine.md +++ b/documentation/configuration/cairo-engine.md @@ -1,6 +1,8 @@ --- title: Cairo engine -description: Configuration settings for the Cairo SQL engine in QuestDB. +description: + Configuration settings for the Cairo SQL engine in QuestDB, including the + query, materialized view refresh, and WAL apply memory limits. --- The Cairo engine is the core storage and query engine in QuestDB. These settings @@ -8,6 +10,9 @@ control how data is written, read, indexed, and queried. Most defaults work well for typical workloads, but tuning may be needed for high-throughput ingestion, large analytical queries, or specific storage configurations. +To cap the native memory a single query, view refresh, or WAL apply batch may +allocate, see [Memory limits](#memory-limits). + ## General ### cairo.date.locale @@ -125,6 +130,16 @@ Whether WAL tables are the default when using `CREATE TABLE`. mmap sliding page size that the table writer uses to append data for each column, specifically for system tables. +### cairo.write.back.off.timeout.on.mem.pressure + +- **Default**: `4000` +- **Reloadable**: no + +Upper bound, in milliseconds, of the random delay a WAL apply job waits before +retrying a batch that failed with an out-of-memory error, once it has already +reduced its parallelism to one. Up to five such back-offs are attempted; if the +error persists, the table is suspended. See [memory limits](#memory-limits). + ### cairo.writer.alter.busy.wait.timeout - **Default**: `500` @@ -873,6 +888,240 @@ every window function execution. Prevents stack overflow errors when evaluating complex nested SQL. The value is the approximate number of nested SELECT clauses allowed. +## Memory limits + +These limits cap the native memory tracked for a single query, materialized view +refresh, live view refresh, or WAL apply batch. They help prevent runaway +workloads from exhausting server memory, and are available since QuestDB +10.0.0. Each workload has its own limit, in +addition to the process-wide native memory limit set by +[`ram.usage.limit.bytes`](#ramusagelimitbytes) and +[`ram.usage.limit.percent`](#ramusagelimitpercent), which is on by default at +90% of the memory visible to the JVM. Allocations still count toward the +process-wide limit, so concurrent workloads can reach it even when each stays +within its own budget. A limit bounds one query, refresh, or batch, not a +connection or a principal: concurrent queries each run under the full limit. + +Three of the four workload limits are documented in this section, together +with the two process-wide keys. The fourth workload limit, +[`cairo.live.view.refresh.memory.limit.bytes`](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes), +lives with the other live view settings. + +All four default to `0`, which means unlimited, so behavior matches a server +without limits until you opt in. Set each limit as a byte count or a size with a +`K`, `M`, or `G` suffix, for example `512M` or `2G`. Each suffix multiplies by +1024, so `512M` is 536870912 bytes. The limits are reloadable: +edit `server.conf` and call +[`reload_config()`](/docs/query/functions/meta/#reload_config). New queries, +materialized view refreshes, and WAL apply batches use the updated limits; work +already running keeps its original limit. A live view acquires its limit when it +is first compiled and keeps it across refreshes, so a reloaded value reaches an +existing view only after a full teardown: invalidation, recreation, or a server +restart. + +When a workload exceeds its limit, QuestDB raises an out-of-memory error at the +allocation that crossed the line and aborts that workload, while unrelated +workloads keep running. What happens next depends on the workload: + +- A user query fails with the error. The client connection stays open, and its + next statement runs under the same limit. +- A materialized view refresh first retries with smaller refresh intervals where + possible, up to + [`cairo.mat.view.max.refresh.retries`](/docs/configuration/materialized-views/#cairomatviewmaxrefreshretries) + times. If the error persists, + [incremental and scheduled period refreshes](/docs/concepts/materialized-views/#refresh-strategies) + are deferred for + [`cairo.mat.view.refresh.busy.retry.timeout`](/docs/configuration/materialized-views/#cairomatviewrefreshbusyretrytimeout), + with up to + [`cairo.mat.view.refresh.busy.retry.limit`](/docs/configuration/materialized-views/#cairomatviewrefreshbusyretrylimit) + retries before invalidation. `REFRESH ... FULL` and user-requested + `REFRESH ... RANGE FROM ... TO ...` invalidate without deferred retries. An + [invalid view](/docs/concepts/materialized-views/#refreshing-an-invalid-view) + is recovered with a full refresh, which runs under the same limit, so raise + the limit first. +- A live view refresh invalidates the view immediately. The live view limit + also counts the state a view retains between refreshes, so size it for the + view's retained state plus the transient buffers of one refresh. Only a + breach of this limit invalidates the view; a process-wide memory error during + a refresh is retried instead. +- A WAL apply first retries under the writer's memory-pressure control, which + shrinks the transaction block, reduces parallelism, and then backs off between + attempts for up to five random delays bounded by + [`cairo.write.back.off.timeout.on.mem.pressure`](#cairowritebackofftimeoutonmempressure). + If the breach persists after the back-off budget is exhausted, the + table is suspended and + [`wal_tables()`](/docs/query/functions/meta/#wal_tables) reports + `OUT OF MEMORY` in its `errorTag` column. Resume it with + [`ALTER TABLE RESUME WAL`](/docs/query/sql/alter-table-resume-wal/). + +The message names the workload so you can tell it apart from a process-wide +breach: + +``` +query memory limit exceeded [workload=QUERY, queryId=62179, limit=536870912, used=536346624, size=1048576, memoryTag=27] +``` + +`workload` is one of `QUERY`, `MAT_VIEW_REFRESH`, `LIVE_VIEW_REFRESH`, or +`WAL_APPLY`. The prefix reads `query memory limit exceeded` for every workload. +`limit` and `used` are bytes, `size` is the allocation that failed, and +`memoryTag` is the numeric id of the allocation category. `queryId` is the +`query_id` reported by +[`query_activity`](/docs/query/functions/meta/#query_activity) for a query, and +the `id` of the table or view itself, as reported by +[`tables()`](/docs/query/functions/meta/#tables), for a WAL apply batch, a +materialized view refresh, or a live view refresh. For a `COPY ... TO` export it +is the copy id, printed here in decimal while `COPY` reports it in hexadecimal. + +:::note + +Only tracked native allocations count toward a limit. Memory-mapped files, such +as table column files, are excluded, and some native allocations are not yet +covered. + +::: + +QuestDB Enterprise can additionally set a memory limit per user, group, or +service account, which overrides the query workload limit for a principal's +queries. See [role-based access control](/docs/security/rbac/#memory-limits). + +### Sizing a limit + +QuestDB does not estimate a workload's memory before running it: how much a +query allocates depends on the data it reads, the plan the engine picks, and +the number of worker threads it runs on. Measure it instead. Live usage and the +effective limit of each running query are exposed by +[`query_activity`](/docs/query/functions/meta/#query_activity) through its +`memory_used` and `memory_limit` columns. `memory_used` is a live gauge with no +peak value, so sample it repeatedly while the query runs and take the largest +value as the floor for the limit. Leave headroom above it: a later run over +more data allocates more. + +Run the query to size in one session with the limit at `0`, so it reports +`memory_used` without risk of a breach, and sample it from a second session: + +```questdb-sql title="Session 1: the query to size" +SELECT symbol, avg(price) AS avg_price +FROM trades +WHERE timestamp IN '2026-09-14' +SAMPLE BY 1m; +``` + +```questdb-sql title="Session 2: sample its usage while it runs" +SELECT query_id, memory_used, memory_limit, query +FROM query_activity() +WHERE query LIKE 'SELECT symbol, avg(price)%'; +``` + +| query_id | memory_used | memory_limit | query | +| -------- | ----------- | ------------ | -------------------------------------------------- | +| 57777 | 8388608 | null | SELECT symbol, avg(price) AS avg_price FROM trades ... | + +Background workloads do not appear in `query_activity`, but each runs SQL that +you can reproduce as a plain query and measure the same way: + +- A materialized view refresh runs the view's `SELECT`, taken from + [`SHOW CREATE MATERIALIZED VIEW`](/docs/query/sql/show/#show-create-materialized-view), + with the base table restricted to the time range being refreshed. Run that + `SELECT` with a `WHERE` clause on the base table's designated timestamp that + spans one refresh worth of data. An incremental refresh covers the rows + committed since the previous refresh, so use the busiest interval you expect + between refreshes. A full refresh, and a refresh after a large out-of-order + write, covers far more, so size for the whole base table if you need those to + succeed under the same limit. The measurement is a close proxy rather than an + exact figure, because the refresh may pick a different plan or degree of + parallelism. +- A live view refresh runs the view's window functions over each batch of new + base table rows, so the same proxy over a batch of rows measures the + transient buffers of one refresh. The live view limit also counts the state + the view retains between refreshes: the `IN MEMORY` tier, whose capacity + [`live_views()`](/docs/query/functions/meta/#live_views) reports in its + `in_mem_bytes` column, and the window state. Add these to the transient + figure, and keep the limit above the + [allocation floor](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) + described with the key. + +After a breach, the `used` and `size` values in the error message record the +footprint at the point of failure, and are the only record of it. A limit has +to be at least `used + size` to get past that allocation, and usually more, +because the workload was aborted before it finished. There is no dedicated +metric for breaches, so alert on the message in the server log. + +### cairo.mat.view.refresh.memory.limit.bytes + +- **Default**: `0` +- **Reloadable**: yes + +Maximum native memory a single materialized view refresh may allocate. `0` +disables the limit. + +### cairo.query.memory.limit.bytes + +- **Default**: `0` +- **Reloadable**: yes + +Maximum native memory a single user SQL query may allocate. `0` disables the +limit. It covers `SELECT`, `INSERT ... SELECT`, `CREATE TABLE AS SELECT`, +`UPDATE` on a non-WAL table, and every other statement that runs on the +caller's connection and appears in +[`query_activity`](/docs/query/functions/meta/#query_activity). It also covers +`COPY ... TO` exports, which run in the background under the issuing query's +budget but do not appear in `query_activity`. `CREATE MATERIALIZED VIEW` +charges only its DDL to this limit: the initial population runs as a refresh +under `cairo.mat.view.refresh.memory.limit.bytes`. Subqueries and other nested +work share the top-level query's budget rather than each acquiring their own. +On QuestDB +Enterprise the built-in admin cannot be given a per-principal override and runs +under this limit, so size it with the admin's diagnostic queries in mind. + +### cairo.wal.apply.memory.limit.bytes + +- **Default**: `0` +- **Reloadable**: yes + +Maximum native memory a single WAL apply batch may allocate. `0` disables the +limit. + +The limit covers only the SQL that WAL apply runs inside a batch: `UPDATE` +statements and non-structural `ALTER TABLE` changes. Memory used to commit data, +including out-of-order merges, is not tracked and is bounded only by the +process-wide native memory limit. In practice this limit rarely fires. Its main effect is +to keep WAL apply SQL on its own budget, separate from the query limit. + +### ram.usage.limit.bytes + +- **Default**: `0` +- **Reloadable**: no + +Process-wide limit on the native memory QuestDB may allocate, as a byte count or +a size with a `K`, `M`, or `G` suffix. `0` means no byte limit. When both this +key and `ram.usage.limit.percent` resolve to a limit, the smaller one applies. + +Despite the name, the limit counts tracked native allocations, not the process +RSS. The JVM heap, thread stacks, and memory-mapped files such as table column +files do not count, so a limit equal to a container's memory limit does not stop +the kernel from killing the process. Compare the `RSS` and `NATIVE_*` rows of +[`memory_metrics()`](/docs/query/functions/meta/#memory_metrics) to see the gap. + +An allocation that would cross the resolved limit fails with a +`global RSS memory limit exceeded [usage=..., RSS_MEM_LIMIT=..., size=..., memoryTag=...]` +error. The error lands on whichever workload +allocates last, so a query, a view refresh, or a WAL apply batch can fail +because of another workload's usage. A WAL apply that hits it goes through the +same retries and suspension as a breach of its own limit. The per-workload +[memory limits](#memory-limits) sit underneath this one and isolate workloads +from each other. + +### ram.usage.limit.percent + +- **Default**: `90` +- **Reloadable**: no + +Process-wide native memory limit as a percentage of the memory visible to the +JVM: the host's physical memory, or the container's cgroup memory limit when one +is set. It counts the same tracked allocations as `ram.usage.limit.bytes`. `0` +disables the percentage limit. When both this key and `ram.usage.limit.bytes` +resolve to a limit, the smaller one applies. + ## Batch operations ### cairo.create.as.select.retry.count diff --git a/documentation/configuration/live-views.md b/documentation/configuration/live-views.md index 204de67516..e44ce2a952 100644 --- a/documentation/configuration/live-views.md +++ b/documentation/configuration/live-views.md @@ -140,6 +140,12 @@ persistent window state, the `IN MEMORY` recent-row tier, staging memory, and transient buffers such as Parquet row-group decode buffers. Exceeding the limit invalidates the view immediately; recovery requires dropping and recreating it. +This is one of the four workload +[memory limits](/docs/configuration/cairo-engine/#memory-limits) and counts +tracked native allocations only. A view acquires the limit when it is first +compiled and keeps it across refreshes, so a reloaded value reaches an existing +view only after invalidation, recreation, or a server restart. + Size the limit above the refresh workload's allocation floor. In particular, a bounded `ROWS` frame allocates at least one `cairo.sql.window.store.page.size` page (1 MiB by default), and a Parquet-backed refresh may decode a complete row diff --git a/documentation/configuration/materialized-views.md b/documentation/configuration/materialized-views.md index 49f73b12ca..92182160b6 100644 --- a/documentation/configuration/materialized-views.md +++ b/documentation/configuration/materialized-views.md @@ -1,12 +1,17 @@ --- title: Materialized views -description: Configuration settings for materialized views in QuestDB. +description: + Materialized view configuration in QuestDB, covering refresh workers, parallel + SQL, and the retry limits that govern out-of-memory and busy refresh failures. --- These settings control materialized view SQL support and the background refresh job. Materialized views can use dedicated worker threads or share the server's common pool. +To cap the native memory a single refresh may allocate, see +[`cairo.mat.view.refresh.memory.limit.bytes`](/docs/configuration/cairo-engine/#memory-limits). + ## cairo.mat.view.enabled - **Default**: `true` @@ -14,6 +19,23 @@ common pool. Enables or disables SQL support and the refresh job for materialized views. +## cairo.mat.view.max.refresh.retries + +- **Default**: `10` +- **Reloadable**: yes + +Maximum number of immediate retries within a single refresh attempt. A retry +happens when the base table changes structurally during the refresh, when a +refresh step produces an oversized transaction, or when a step fails with an +out-of-memory error, including a breach of the +[refresh memory limit](/docs/configuration/cairo-engine/#memory-limits). +Retries after an oversized transaction or an out-of-memory error shrink the +refresh interval step; a retry after a structural change recompiles the view +with the same step, and if it keeps failing the refresh is queued again. Once +the out-of-memory retries are exhausted or the step cannot shrink further, the +error propagates and the deferred retries governed by +`cairo.mat.view.refresh.busy.retry.limit` take over. + ## cairo.mat.view.parallel.sql.enabled - **Default**: `true` @@ -22,6 +44,32 @@ Enables or disables SQL support and the refresh job for materialized views. When disabled, SQL executed by the materialized view refresh job always runs single-threaded. +## cairo.mat.view.refresh.busy.retry.limit + +- **Default**: `10` +- **Reloadable**: no + +Maximum number of deferred retries after an incremental or scheduled period +refresh fails with a transient error. If all retries fail, the view is +invalidated. A successful refresh resets the counter; `0` disables deferred +retries. + +Transient errors include a busy base table or view and out-of-memory errors, +including breaches of the +[refresh memory limit](/docs/configuration/cairo-engine/#memory-limits). Full +refreshes and user-requested `REFRESH ... RANGE FROM ... TO ...` do not use these +deferred retries. + +## cairo.mat.view.refresh.busy.retry.timeout + +- **Default**: `1000` +- **Reloadable**: no + +Delay in milliseconds before a deferred retry for an incremental or scheduled +period refresh. The retry is timer-driven and does not block a refresh worker. +The deprecated `cairo.mat.view.refresh.oom.retry.timeout` key is accepted but +has no effect; deferred out-of-memory retries use this backoff. + ## mat.view.refresh.worker.affinity - **Default**: equal to the CPU core count diff --git a/documentation/configuration/overview.md b/documentation/configuration/overview.md index 45dbc1c738..e0b9eb159e 100644 --- a/documentation/configuration/overview.md +++ b/documentation/configuration/overview.md @@ -194,7 +194,9 @@ If the value was reloaded successfully, the `reload_config` function returns Each key has a `reloadable` property that indicates whether the key can be reloaded. If yes, the `reload_config` function can be used to reload the -configuration. +configuration. The per-workload +[memory limits](/docs/configuration/cairo-engine/#memory-limits) for queries, +view refreshes, and WAL apply are reloadable, for example. All reloadable properties can be also queried from the server: diff --git a/documentation/configuration/wal.md b/documentation/configuration/wal.md index e855a9b640..e485910de4 100644 --- a/documentation/configuration/wal.md +++ b/documentation/configuration/wal.md @@ -7,6 +7,9 @@ These settings control the Write-Ahead Log (WAL) subsystem, including parallel apply threads, segment rollover, commit squashing, and cleanup of applied WAL files. +To cap the native memory a single WAL apply batch may allocate, see +[`cairo.wal.apply.memory.limit.bytes`](/docs/configuration/cairo-engine/#memory-limits). + ## cairo.wal.apply.parallel.sql.enabled - **Default**: `true` diff --git a/documentation/getting-started/capacity-planning.md b/documentation/getting-started/capacity-planning.md index 3548de46ba..e577dfafb0 100644 --- a/documentation/getting-started/capacity-planning.md +++ b/documentation/getting-started/capacity-planning.md @@ -175,6 +175,10 @@ For relatively small datasets i.e 4-40GB, and a read-heavy workload, performance can be improved by maximising use of the OS page cache. Users should consider increasing available RAM to improve the speed of read operations. +To help protect against runaway queries, materialized view refreshes, live view +refreshes, or WAL apply workloads, see +[memory limits](/docs/configuration/cairo-engine/#memory-limits). + ### Memory page size configuration With frequent out-of-order (O3) writes over a large number of columns/tables, diff --git a/documentation/getting-started/migrate-to-enterprise.md b/documentation/getting-started/migrate-to-enterprise.md index 409350911b..dfb97d8448 100644 --- a/documentation/getting-started/migrate-to-enterprise.md +++ b/documentation/getting-started/migrate-to-enterprise.md @@ -18,7 +18,8 @@ a newer release, see [Upgrade QuestDB](/docs/operations/upgrade/). ## What you get with QuestDB Enterprise - **TLS encryption** for all network interfaces -- **Role-based access control (RBAC)** with users, groups, and permissions +- **Role-based access control (RBAC)** with users, groups, permissions, and + per-principal query memory limits - **Single Sign-On (SSO)** via OpenID Connect - **Database replication** for high availability - **[Storage policies](/docs/concepts/storage-policy/)** for automated partition diff --git a/documentation/integrations/other/ignition.md b/documentation/integrations/other/ignition.md index 4bde119b40..84673da78a 100644 --- a/documentation/integrations/other/ignition.md +++ b/documentation/integrations/other/ignition.md @@ -69,8 +69,8 @@ Ignition can be configured to expose QuestDB's Postgres wire server, allowing fo Ignition also exposes settings to control the amount of memory allocated to the underlying database. - `historian.questdb.ramUsageLimitBytes` - - Corresponds to `ram.usage.limit.bytes` in QuestDB's `server.conf`. - - Controls the amount of RAM allocated to the database in bytes. + - Corresponds to [`ram.usage.limit.bytes`](/docs/configuration/cairo-engine/#ramusagelimitbytes) in QuestDB's `server.conf`. + - Caps the tracked native memory the database may allocate, in bytes. - `historian.questdb.ramUsageLimitPercent` - - Corresponds to `ram.usage.limit.percent` in QuestDB's `server.conf`. - - Control the amount of RAM allocated to the database as a percentage of system memory. + - Corresponds to [`ram.usage.limit.percent`](/docs/configuration/cairo-engine/#ramusagelimitpercent) in QuestDB's `server.conf`. + - Caps the tracked native memory the database may allocate as a percentage of system memory. diff --git a/documentation/operations/monitoring-alerting.md b/documentation/operations/monitoring-alerting.md index dd4c797d17..5fdf1ce490 100644 --- a/documentation/operations/monitoring-alerting.md +++ b/documentation/operations/monitoring-alerting.md @@ -83,8 +83,10 @@ ORDER BY ### Detect suspended tables A WAL table becomes suspended when an error occurs during WAL apply, such as -disk full, corrupted WAL segment, or kernel limits reached. While suspended, -new data continues to be written to WAL but is not applied to the table. +disk full, corrupted WAL segment, kernel limits reached, or a WAL apply batch +breaching its [memory limit](/docs/configuration/cairo-engine/#memory-limits). +While suspended, new data continues to be written to WAL but is not applied to +the table. **Detection:** @@ -92,6 +94,13 @@ new data continues to be written to WAL but is not applied to the table. SELECT table_name FROM tables() WHERE table_suspended; ``` +To see why a table was suspended, query `wal_tables()`. Its `errorTag` column +reads `OUT OF MEMORY` when the cause was a memory limit breach: + +```questdb-sql +SELECT name, errorTag, errorMessage FROM wal_tables() WHERE suspended; +``` + **Resolution:** Resume from the failed transaction: @@ -121,7 +130,10 @@ detailed recovery procedures including corrupted segment handling. Materialized views become invalid when their base table is modified in incompatible ways: dropping referenced columns, dropping partitions, renaming -the table, or running TRUNCATE/UPDATE operations. +the table, or running TRUNCATE/UPDATE operations. A view is also invalidated +when its refresh keeps failing with an out-of-memory error, including a breach +of the [refresh memory limit](/docs/configuration/cairo-engine/#memory-limits), +after the deferred retries are exhausted. **Detection:** @@ -133,7 +145,10 @@ WHERE view_status = 'invalid'; **Resolution:** -Perform a full refresh to rebuild the view: +Perform a full refresh to rebuild the view. If `invalidation_reason` reports a +memory limit breach, raise the +[refresh memory limit](/docs/configuration/cairo-engine/#memory-limits) first, +because the full refresh runs under the same limit: ```questdb-sql REFRESH MATERIALIZED VIEW my_view FULL; @@ -203,6 +218,9 @@ Other options: - Add more RAM to the server - Reduce concurrent ingestion load - Reduce the number of tables with active O3 writes +- Cap the memory a single query or view refresh may allocate with the + per-workload [memory limits](/docs/configuration/cairo-engine/#memory-limits), + which leaves more headroom for O3 merges See [Capacity planning](/docs/getting-started/capacity-planning/#memory-page-size-configuration) and [Optimize for many tables](/docs/cookbook/operations/optimize-many-tables/) diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index d2156a88fb..bd0f2e4c23 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -1,7 +1,9 @@ --- title: Meta functions sidebar_label: Meta -description: Database and table metadata function reference documentation. +description: + Meta functions for inspecting tables, WAL status, running queries with their + memory usage and limits, configuration, and server metadata. --- These functions provide instance-level information and table, column and @@ -323,7 +325,13 @@ materialized_views(); **Return value:** -Returns granular memory metrics. +Returns granular memory metrics. `RSS` is the process resident set size and +`TOTAL_USED` the sum of all tracked allocations. The `NATIVE_*` rows are the +tracked allocations that count toward +[`ram.usage.limit.bytes`](/docs/configuration/cairo-engine/#ramusagelimitbytes) +and the per-workload +[memory limits](/docs/configuration/cairo-engine/#memory-limits); the `MMAP_*` +rows do not. **Examples:** @@ -397,7 +405,7 @@ because inserted rows replicate as data. **Return value:** -Returns metadata on running SQL queries, including columns such as: +Returns metadata on running SQL queries, with the following columns: - query_id - identifier of the query that can be used with [cancel query](/docs/query/sql/cancel-query) command or @@ -409,18 +417,35 @@ Returns metadata on running SQL queries, including columns such as: - query_start - timestamp of when query started - state_change - timestamp of latest query state change, such as a cancellation - state - state of running query, can be `active` or `cancelled` +- is_wal - `true` when the SQL is being applied by the WAL apply job, such as + an `UPDATE` on a WAL table. Such queries cannot be cancelled - query - text of sql query +- memory_used - native memory currently allocated by the query, in bytes, as + tracked by the + [per-query memory limit](/docs/configuration/cairo-engine/#memory-limits) +- memory_limit - effective native memory limit for the query, in bytes, or + `null` when the query runs unlimited. On QuestDB Enterprise this is the + principal's [memory limit](/docs/security/rbac/#memory-limits) when one is + set, otherwise the workload limit. Unlike the `memory_limit` column of + `SHOW USERS`, it includes the workload limit + +`memory_used` is a live gauge with no peak value, and it is reported even when +`memory_limit` is `null`. Both memory columns are `null` for SQL that runs +under a background workload's tracker, such as the `SELECT` a materialized view +refresh runs or an `UPDATE` applied by the WAL apply job, because that SQL +charges the workload's budget instead of acquiring its own. **Examples:** ```questdb-sql -SELECT * FROM query_activity(); +SELECT query_id, username, state, memory_used, memory_limit, query +FROM query_activity(); ``` -| query_id | worker_id | worker_pool | username | query_start | state_change | state | query | -| -------- | --------- | ----------- | -------- | --------------------------- | --------------------------- | ------ | --------------------------------------------------------- | -| 62179 | 5 | shared | bob | 2024-01-09T10:03:05.557397Z | 2024-01-09T10:03:05.557397 | active | select \* from query_activity() | -| 57777 | 6 | shared | bob | 2024-01-09T08:58:55.988017Z | 2024-01-09T08:58:55.988017Z | active | SELECT symbol,approx_percentile(price, 50, 2) from trades | +| query_id | username | state | memory_used | memory_limit | query | +| -------- | -------- | ------ | ----------- | ------------ | ------------------------------------------------------------------------------------- | +| 62179 | john | active | 262144 | 536870912 | SELECT query_id, username, state, memory_used, memory_limit, query FROM query_activity() | +| 57777 | john | active | 8388608 | 536870912 | SELECT symbol, approx_percentile(price, 0.5, 2) FROM trades | ## reader_pool @@ -1244,10 +1269,11 @@ SELECT wait_wal_table('trades', 42); :::note For monitoring and observability, use [`tables()`](#tables) instead. -`tables()` provides all the same information plus additional metrics +`tables()` provides the same status information plus additional metrics (pending rows, memory pressure, deduplication stats, throughput histograms), and is fully in-memory. `wal_tables()` reads from disk and is less suitable -for frequent polling. +for frequent polling, but it is the only function that reports the `errorTag` +and `errorMessage` of a suspended table. ::: @@ -1266,10 +1292,19 @@ Returns a `table` including the following information: - `name` - table or materialized view name - `suspended` - suspended status flag - `writerTxn` - the last committed transaction in TableWriter (equivalent to `table_txn` in `tables()`) -- `writerLagTxnCount` - the number of transactions that are kept invisible when +- `bufferedTxnSize` - the number of transactions that are kept invisible when writing to the table; these transactions will be eventually moved to the table data and become visible for readers (equivalent to `wal_txn - table_txn`) - `sequencerTxn` - the last committed transaction in the sequencer (equivalent to `wal_txn` in `tables()`) +- `errorTag` - short classification of the error that suspended the table, such + as `OUT OF MEMORY` when a WAL apply batch breached its + [memory limit](/docs/configuration/cairo-engine/#memory-limits), or empty + when the table is not suspended +- `errorMessage` - full text of the error that suspended the table, or empty + when the table is not suspended +- `memoryPressure` - memory pressure level of the table writer: `0` for none, + `1` when parallelism is reduced, `2` when the writer backs off between + attempts (equivalent to `table_memory_pressure_level` in `tables()`) **Examples:** @@ -1277,11 +1312,11 @@ Returns a `table` including the following information: wal_tables(); ``` -| name | suspended | writerTxn | writerLagTxnCount | sequencerTxn | -| ----------- | --------- | --------- | ----------------- | ------------ | -| sensor_wal | false | 2 | 1 | 4 | -| weather_wal | false | 3 | 0 | 3 | -| test_wal | true | 7 | 1 | 9 | +| name | suspended | writerTxn | bufferedTxnSize | sequencerTxn | errorTag | errorMessage | memoryPressure | +| ----------- | --------- | --------- | --------------- | ------------ | ------------- | --------------------------------------------------------------------------------------------------------------------------- | -------------- | +| sensor_wal | false | 2 | 1 | 4 | | | 0 | +| weather_wal | false | 3 | 0 | 3 | | | 0 | +| test_wal | true | 7 | 1 | 9 | OUT OF MEMORY | query memory limit exceeded [workload=WAL_APPLY, queryId=12, limit=1073741824, used=1073217536, size=1048576, memoryTag=27] | 2 | ## writer_pool diff --git a/documentation/query/sql/acl/add-user.md b/documentation/query/sql/acl/add-user.md index 3b181417c3..35d0bde8a7 100644 --- a/documentation/query/sql/acl/add-user.md +++ b/documentation/query/sql/acl/add-user.md @@ -44,7 +44,7 @@ SHOW GROUPS john; that yields: -| name | -| ---------- | -| management | -| audit | +| name | external_alias | memory_limit | +| ---------- | -------------- | ------------ | +| management | | 2147483648 | +| audit | | null | diff --git a/documentation/query/sql/acl/alter-group.md b/documentation/query/sql/acl/alter-group.md new file mode 100644 index 0000000000..4ba8013397 --- /dev/null +++ b/documentation/query/sql/acl/alter-group.md @@ -0,0 +1,81 @@ +--- +title: ALTER GROUP reference +sidebar_label: ALTER GROUP +description: + "ALTER GROUP sets a per-group query memory limit or maps an external OIDC or + LDAP group alias. Applies to RBAC in QuestDB Enterprise." +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + RBAC provides fine-grained database permissions management. + + +`ALTER GROUP` modifies group settings. + +For full documentation of the Access Control List and Role-based Access Control, +see the [RBAC operations](/docs/security/rbac) page. + +--- + +## Syntax + +```questdb-sql title="Set or clear memory limit" +ALTER GROUP groupName SET MEMORY LIMIT { size | UNLIMITED }; +``` + +```questdb-sql title="Add or remove external alias" +ALTER GROUP groupName { WITH | DROP } EXTERNAL ALIAS externalAlias; +``` + +## Description + +- `ALTER GROUP groupName SET MEMORY LIMIT size` - caps the native memory that + each query run by a member of the group may allocate. `size` is a byte count + or a size with a `K`, `M`, or `G` suffix, such as `512M` or `2G`. +- `ALTER GROUP groupName SET MEMORY LIMIT UNLIMITED` - clears the group's limit. + Members without a limit of their own then fall back to the most restrictive + limit among their other groups, or to the workload limit + (`cairo.query.memory.limit.bytes`). `SET MEMORY LIMIT 0` does the same. +- `ALTER GROUP groupName WITH EXTERNAL ALIAS externalAlias` - maps an external + OIDC or LDAP group to this group. +- `ALTER GROUP groupName DROP EXTERNAL ALIAS externalAlias` - removes an external + group mapping. + +A group limit applies to a member only when that member has no limit of its own. +When several of a user's groups set a limit, the most restrictive one applies. +Setting a group limit requires the `SET MEMORY LIMIT` permission. See +[memory limits](/docs/security/rbac/#memory-limits) for how a group limit +interacts with the +[`cairo.query.memory.limit.bytes`](/docs/configuration/cairo-engine/#cairoquerymemorylimitbytes) +workload limit. + +Adding an alias requires the `ADD EXTERNAL ALIAS` permission and removing one +requires `REMOVE EXTERNAL ALIAS`. Quote the alias when it contains commas, +spaces, or `=`, as LDAP distinguished names do. For external group mapping with +OIDC or LDAP, see the +[OpenID Connect (OIDC) integration](/docs/security/oidc/#mapping-user-permissions) +guide. + +## Examples + +### Set memory limit + +```questdb-sql +-- cap queries of the group's members at 2 GiB of native memory +ALTER GROUP analysts SET MEMORY LIMIT 2G; +-- remove the limit +ALTER GROUP analysts SET MEMORY LIMIT UNLIMITED; +``` + +The configured value can be verified with +[`SHOW GROUPS`](/docs/query/sql/show/#show-groups), which reports it in the +`memory_limit` column. + +### Map an external group + +```questdb-sql +ALTER GROUP analysts WITH EXTERNAL ALIAS 'CN=Analysts,OU=Users,DC=example,DC=com'; +ALTER GROUP analysts DROP EXTERNAL ALIAS 'CN=Analysts,OU=Users,DC=example,DC=com'; +``` diff --git a/documentation/query/sql/acl/alter-service-account.md b/documentation/query/sql/acl/alter-service-account.md index a3de16f560..ca73247ac2 100644 --- a/documentation/query/sql/acl/alter-service-account.md +++ b/documentation/query/sql/acl/alter-service-account.md @@ -2,8 +2,9 @@ title: ALTER SERVICE ACCOUNT reference sidebar_label: ALTER SERVICE ACCOUNT description: - "ALTER SERVICE ACCOUNT SQL keywords reference documentation. Applies to RBAC - in QuestDB Enterprise." + "ALTER SERVICE ACCOUNT enables or disables a service account, manages + passwords and tokens, and sets its query memory limit. Applies to RBAC in + QuestDB Enterprise." --- import { EnterpriseNote } from "@site/src/components/EnterpriseNote" @@ -39,6 +40,10 @@ ALTER SERVICE ACCOUNT serviceAccountName DROP TOKEN TYPE { JWK | REST [token] }; ``` +```questdb-sql title="Set or clear memory limit" +ALTER SERVICE ACCOUNT serviceAccountName SET MEMORY LIMIT { size | UNLIMITED }; +``` + ## Description - `ALTER SERVICE ACCOUNT serviceAccountName ENABLE` - enables service account. @@ -56,6 +61,21 @@ ALTER SERVICE ACCOUNT serviceAccountName DROP TOKEN TYPE adds REST token to the service account. - `ALTER USER serviceAccountName DROP TOKEN TYPE REST token` - removes REST token from the service account. +- `ALTER SERVICE ACCOUNT serviceAccountName SET MEMORY LIMIT size` - caps the + native memory each of the service account's queries may allocate. `size` is a + byte count or a size with a `K`, `M`, or `G` suffix, such as `512M` or `2G`. +- `ALTER SERVICE ACCOUNT serviceAccountName SET MEMORY LIMIT UNLIMITED` - clears + the service account's limit. The workload limit + (`cairo.query.memory.limit.bytes`) then applies. `SET MEMORY LIMIT 0` does the + same. + +A user who assumes the service account runs under its memory limit. Group limits +are never merged into a service account. Setting it requires the +`SET MEMORY LIMIT` permission. See +[memory limits](/docs/security/rbac/#memory-limits) for how the limit interacts +with the +[`cairo.query.memory.limit.bytes`](/docs/configuration/cairo-engine/#cairoquerymemorylimitbytes) +workload limit. ## Examples @@ -165,3 +185,16 @@ SHOW SERVICE ACCOUNT client_app; | Password | true | | JWK Token | false | | REST Token | false | + +### Set memory limit + +```questdb-sql +-- cap the service account's queries at 1 GiB of native memory +ALTER SERVICE ACCOUNT client_app SET MEMORY LIMIT 1G; +-- remove the limit +ALTER SERVICE ACCOUNT client_app SET MEMORY LIMIT UNLIMITED; +``` + +The configured value can be verified with +[`SHOW SERVICE ACCOUNTS`](/docs/query/sql/show/#show-service-accounts), which +reports it in the `memory_limit` column. diff --git a/documentation/query/sql/acl/alter-user.md b/documentation/query/sql/acl/alter-user.md index b0823db42f..828980e378 100644 --- a/documentation/query/sql/acl/alter-user.md +++ b/documentation/query/sql/acl/alter-user.md @@ -2,8 +2,8 @@ title: ALTER USER reference sidebar_label: ALTER USER description: - "ALTER USER SQL keywords reference documentation. Applies to RBAC in QuestDB - Enterprise." + "ALTER USER enables or disables a user, manages passwords and tokens, and sets + a per-user query memory limit. Applies to RBAC in QuestDB Enterprise." --- import { EnterpriseNote } from "@site/src/components/EnterpriseNote" @@ -17,6 +17,8 @@ import { EnterpriseNote } from "@site/src/components/EnterpriseNote" For full documentation of the Access Control List and Role-based Access Control, see the [RBAC operations](/docs/security/rbac) page. +--- + ## Syntax ```questdb-sql title="Enable / disable" @@ -37,6 +39,10 @@ ALTER USER userName DROP TOKEN TYPE { JWK | REST [token] }; ``` +```questdb-sql title="Set or clear memory limit" +ALTER USER userName SET MEMORY LIMIT { size | UNLIMITED }; +``` + ## Description - `ALTER USER username ENABLE` - enables user account. @@ -54,6 +60,21 @@ ALTER USER userName DROP TOKEN TYPE REST token to user account. - `ALTER USER username DROP TOKEN TYPE REST token` - removes REST token from user account. +- `ALTER USER username SET MEMORY LIMIT size` - caps the native memory each of + the user's queries may allocate. `size` is a byte count or a size with a `K`, + `M`, or `G` suffix, such as `512M` or `2G`. +- `ALTER USER username SET MEMORY LIMIT UNLIMITED` - clears the user's own + limit. A group limit or the workload limit (`cairo.query.memory.limit.bytes`) + then applies. `SET MEMORY LIMIT 0` does the same. + +The limit applies to the user's queries on both the primary and replicas. +Setting it requires the `SET MEMORY LIMIT` permission. The built-in admin and +external (SSO/OIDC) users cannot be given a limit; the statement is rejected for +both. An external user inherits a limit from its groups instead. A set limit +takes priority over the user's groups and over the +[`cairo.query.memory.limit.bytes`](/docs/configuration/cairo-engine/#cairoquerymemorylimitbytes) +workload limit; see [memory limits](/docs/security/rbac/#memory-limits) for how +limits resolve. ## Examples @@ -160,3 +181,15 @@ SHOW USER john; | Password | true | | JWK Token | false | | REST Token | false | + +### Set memory limit + +```questdb-sql +-- cap the user's queries at 512 MiB of native memory +ALTER USER john SET MEMORY LIMIT 512M; +-- remove the limit +ALTER USER john SET MEMORY LIMIT UNLIMITED; +``` + +Use [`SHOW USERS`](/docs/query/sql/show/#show-users) to inspect the user's own +or inherited group limit in the `memory_limit` column. diff --git a/documentation/query/sql/acl/create-group.md b/documentation/query/sql/acl/create-group.md index a538cdc04e..838bdfcaf0 100644 --- a/documentation/query/sql/acl/create-group.md +++ b/documentation/query/sql/acl/create-group.md @@ -2,8 +2,8 @@ title: CREATE GROUP reference sidebar_label: CREATE GROUP description: - "CREATE GROUP SQL keywords reference documentation. Applies to RBAC in - QuestDB Enterprise." + "CREATE GROUP creates an RBAC group, optionally mapped to an external OIDC or + LDAP group with WITH EXTERNAL ALIAS. Applies to QuestDB Enterprise." --- import { EnterpriseNote } from "@site/src/components/EnterpriseNote" @@ -25,10 +25,30 @@ see the [RBAC operations](/docs/security/rbac) page. CREATE GROUP [IF NOT EXISTS] groupName; ``` +```questdb-sql title="Create a group mapped to an external group" +CREATE GROUP groupName WITH EXTERNAL ALIAS externalAlias; +``` + ## Description `CREATE GROUP` adds a new user group with no permissions. +`CREATE GROUP groupName WITH EXTERNAL ALIAS externalAlias` also maps an external +OIDC or LDAP group to the new group in one statement, so members of the external +group inherit its permissions on login. The group and the mapping are created +atomically. `WITH EXTERNAL ALIAS` cannot be combined with `IF NOT EXISTS`. To +map or unmap an existing group, use +[`ALTER GROUP`](/docs/query/sql/acl/alter-group/). For the external group +mapping flow, see the +[OpenID Connect (OIDC) integration](/docs/security/oidc/#mapping-user-permissions) +guide. + +`CREATE GROUP` cannot set a memory limit: a new group has none, and +`SHOW GROUPS` reports `null` in its `memory_limit` column. To cap the native +memory each query from the group's members may allocate, use +[`ALTER GROUP ... SET MEMORY LIMIT`](/docs/query/sql/acl/alter-group/#set-memory-limit) +after creating the group. + The chosen name must be unique across all users (including the built-in admin), groups and service accounts. If the name has already been reserved, the command fails and an error is raised, unless the `IF NOT EXISTS` clause is included in @@ -43,6 +63,8 @@ group only serves as a container for permissions which are shared between users. CREATE GROUP admins; CREATE GROUP IF NOT EXISTS admins; + +CREATE GROUP analysts WITH EXTERNAL ALIAS 'CN=Analysts,OU=Users,DC=example,DC=com'; ``` It can be verified with: @@ -53,6 +75,7 @@ SHOW GROUPS; that yields: -| name | -| ------ | -| admins | +| name | external_alias | memory_limit | +| -------- | --------------------------------------- | ------------ | +| admins | | null | +| analysts | CN=Analysts,OU=Users,DC=example,DC=com | null | diff --git a/documentation/query/sql/acl/create-service-account.md b/documentation/query/sql/acl/create-service-account.md index 8a6fa4fa06..ce9230ec9b 100644 --- a/documentation/query/sql/acl/create-service-account.md +++ b/documentation/query/sql/acl/create-service-account.md @@ -30,6 +30,11 @@ CREATE SERVICE ACCOUNT [IF NOT EXISTS] accountName [OWNED BY ownerName]; `CREATE SERVICE ACCOUNT` adds a new service account with no permissions. +`CREATE SERVICE ACCOUNT` cannot set a memory limit. To cap the native memory +each of the service account's queries may allocate, use +[`ALTER SERVICE ACCOUNT ... SET MEMORY LIMIT`](/docs/query/sql/acl/alter-service-account/#set-memory-limit) +after creating it. Service accounts do not inherit group limits. + The chosen name must be unique across all users (including the built-in admin), groups and service accounts. If the name has already been reserved, the command fails and an error is raised, unless the `IF NOT EXISTS` clause is included in diff --git a/documentation/query/sql/acl/create-user.md b/documentation/query/sql/acl/create-user.md index 8a8d428672..65457ccb06 100644 --- a/documentation/query/sql/acl/create-user.md +++ b/documentation/query/sql/acl/create-user.md @@ -31,6 +31,14 @@ CREATE USER [IF NOT EXISTS] userName `CREATE USER` adds a new user with no permissions, optionally a password can also be set for the user. +`CREATE USER` cannot set a memory limit. To cap the native memory each of the +user's queries may allocate, set a limit on the user with +[`ALTER USER ... SET MEMORY LIMIT`](/docs/query/sql/acl/alter-user/#set-memory-limit), +or on one of its groups with +[`ALTER GROUP ... SET MEMORY LIMIT`](/docs/query/sql/acl/alter-group/#set-memory-limit). +See [memory limits](/docs/security/rbac/#memory-limits) for how the two +interact. + The chosen name must be unique across all users (including the built-in admin), groups and service accounts. If the name has already been reserved, the command fails and an error is raised, unless the `IF NOT EXISTS` clause is included in diff --git a/documentation/query/sql/acl/grant-assume-service-account.md b/documentation/query/sql/acl/grant-assume-service-account.md index d78cc7c823..ce9cf72264 100644 --- a/documentation/query/sql/acl/grant-assume-service-account.md +++ b/documentation/query/sql/acl/grant-assume-service-account.md @@ -55,9 +55,9 @@ the effects of running SQL commands that follow are shown with GRANT ASSUME SERVICE ACCOUNT ingestion TO john; ``` -| name | grant_option | -| --------- | ------------ | -| ingestion | false | +| name | grant_option | memory_limit | +| --------- | ------------ | ------------ | +| ingestion | false | null | ### Assign a service account to a user with grant option @@ -65,9 +65,9 @@ GRANT ASSUME SERVICE ACCOUNT ingestion TO john; GRANT ASSUME SERVICE ACCOUNT ingestion TO john WITH GRANT OPTION; ``` -| name | grant_option | -| --------- | ------------ | -| ingestion | true | +| name | grant_option | memory_limit | +| --------- | ------------ | ------------ | +| ingestion | true | null | ### Removing grant option @@ -76,9 +76,9 @@ GRANT ASSUME SERVICE ACCOUNT ingestion TO john WITH GRANT OPTION; GRANT ASSUME SERVICE ACCOUNT ingestion TO john; ``` -| name | grant_option | -| --------- | ------------ | -| ingestion | false | +| name | grant_option | memory_limit | +| --------- | ------------ | ------------ | +| ingestion | false | null | ### Owner grants @@ -94,6 +94,6 @@ CREATE SERVICE ACCOUNT ingestion; SHOW SERVICE ACCOUNTS john; ``` -| name | grant_option | -| --------- | ------------ | -| ingestion | true | +| name | grant_option | memory_limit | +| --------- | ------------ | ------------ | +| ingestion | true | null | diff --git a/documentation/query/sql/acl/revoke-assume-service-account.md b/documentation/query/sql/acl/revoke-assume-service-account.md index 7508223469..f61a88d7aa 100644 --- a/documentation/query/sql/acl/revoke-assume-service-account.md +++ b/documentation/query/sql/acl/revoke-assume-service-account.md @@ -42,14 +42,14 @@ service account. GRANT ASSUME SERVICE ACCOUNT ingestion TO john WITH GRANT OPTION; ``` -| name | grant_option | -| --------- | ------------ | -| ingestion | t | +| name | grant_option | memory_limit | +| --------- | ------------ | ------------ | +| ingestion | true | null | ```questdb-sql REVOKE ASSUME SERVICE ACCOUNT ingestion FROM john; ``` -| name | grant_option | -| ---- | ------------ | -| | | +| name | grant_option | memory_limit | +| ---- | ------------ | ------------ | +| | | | diff --git a/documentation/query/sql/alter-mat-view-resume-wal.md b/documentation/query/sql/alter-mat-view-resume-wal.md index fc5668463c..1c8dd5aacf 100644 --- a/documentation/query/sql/alter-mat-view-resume-wal.md +++ b/documentation/query/sql/alter-mat-view-resume-wal.md @@ -35,17 +35,29 @@ due to an error. The view will be marked as `suspended = true` in the Use [`wal_tables()`](/docs/query/functions/meta/#wal_tables) to identify suspended views: -```questdb-sql title="List WAL status for all tables and views" -wal_tables(); +```questdb-sql title="List suspended tables and views" +SELECT name, suspended, writerTxn, sequencerTxn, errorTag +FROM wal_tables() +WHERE suspended; ``` -| name | suspended | writerTxn | sequencerTxn | -| --------- | --------- | --------- | ------------ | -| trades_1h | true | 3 | 5 | +| name | suspended | writerTxn | sequencerTxn | errorTag | +| --------- | --------- | --------- | ------------ | --------- | +| trades_1h | true | 3 | 5 | DISK FULL | The `trades_1h` view is suspended. The last successful commit was transaction `3`. +`wal_tables()` also reports the `errorTag` and `errorMessage` of a suspended +view. `OUT OF MEMORY` means a WAL apply batch on the view ran out of memory, by +breaching its own +[memory limit](/docs/configuration/cairo-engine/#memory-limits) or the +process-wide one. This is distinct from a refresh that breaches +[`cairo.mat.view.refresh.memory.limit.bytes`](/docs/configuration/cairo-engine/#cairomatviewrefreshmemorylimitbytes): +that does not suspend the view but, once its retries are exhausted, +[invalidates](/docs/concepts/materialized-views/#view-invalidation) it, which +`RESUME WAL` does not repair. + ### Resume from failed transaction Restart processing from the next transaction after the last successful one: diff --git a/documentation/query/sql/alter-table-resume-wal.md b/documentation/query/sql/alter-table-resume-wal.md index d368817e19..b2d453d635 100644 --- a/documentation/query/sql/alter-table-resume-wal.md +++ b/documentation/query/sql/alter-table-resume-wal.md @@ -33,18 +33,25 @@ the Sequencer. Once the error is resolved, `ALTER TABLE RESUME WAL` restarts the suspended WAL transactions from the failed transaction. Alternatively, an optional `sequencerTxn` value can be provided to skip the failed transaction. +`wal_tables()` also reports the `errorTag` and `errorMessage` of a suspended +table; `OUT OF MEMORY` means a WAL apply batch ran out of memory, by breaching +its own [memory limit](/docs/configuration/cairo-engine/#memory-limits) or the +process-wide one. + ## Examples Using the [`wal_tables()`](/docs/query/functions/meta/#wal_tables) function to investigate the table status: -```questdb-sql title="List all tables" -wal_tables(); +```questdb-sql title="List suspended tables" +SELECT name, suspended, writerTxn, sequencerTxn, errorTag +FROM wal_tables() +WHERE suspended; ``` -| name | suspended | writerTxn | sequencerTxn | -| ------ | --------- | --------- | ------------ | -| trades | true | 3 | 5 | +| name | suspended | writerTxn | sequencerTxn | errorTag | +| ------ | --------- | --------- | ------------ | --------- | +| trades | true | 3 | 5 | DISK FULL | The table `trades` is suspended. The last successful commit in the table is `3`. diff --git a/documentation/query/sql/cancel-query.md b/documentation/query/sql/cancel-query.md index 9afb843f42..f6f9922837 100644 --- a/documentation/query/sql/cancel-query.md +++ b/documentation/query/sql/cancel-query.md @@ -48,10 +48,10 @@ meta-function: SELECT * FROM query_activity(); ``` -| query_id | worker_id | worker_pool | username | query_start | state_change | state | query | -| -------- | --------- | ----------- | -------- | --------------------------- | --------------------------- | ------ | -------------------------------------------------------------------- | -| 29 | 1 | shared | joe | 2024-01-09T10:51:05.878627Z | 2024-01-09T10:51:05.878627Z | active | CREATE TABLE test_tab AS (SELECT x FROM long_sequence(10000000000)); | -| 30 | 21 | shared | joe | 2024-01-09T10:51:10.661032Z | 2024-01-09T10:51:10.661032Z | active | SELECT \* FROM query_activity(); | +| query_id | worker_id | worker_pool | username | query_start | state_change | state | is_wal | query | memory_used | memory_limit | +| -------- | --------- | ----------- | -------- | --------------------------- | --------------------------- | ------ | ------ | -------------------------------------------------------------------- | ----------- | ------------ | +| 29 | 1 | shared | joe | 2024-01-09T10:51:05.878627Z | 2024-01-09T10:51:05.878627Z | active | false | CREATE TABLE test_tab AS (SELECT x FROM long_sequence(10000000000)); | 913571840 | null | +| 30 | 21 | shared | joe | 2024-01-09T10:51:10.661032Z | 2024-01-09T10:51:10.661032Z | active | false | SELECT \* FROM query_activity(); | 262144 | null | We see that the two latest queries have `query_id`'s of 29 and 30, respectively. diff --git a/documentation/query/sql/copy.md b/documentation/query/sql/copy.md index 27751ad1f5..b1252b3ec6 100644 --- a/documentation/query/sql/copy.md +++ b/documentation/query/sql/copy.md @@ -254,6 +254,13 @@ COPY (selectQuery) TO 'destinationPath' - Supports partitioned exports matching table partitioning - Configurable size limits +An export runs under the query +[memory limit](/docs/configuration/cairo-engine/#memory-limits); on QuestDB +Enterprise it uses the issuing principal's +[per-principal limit](/docs/security/rbac/#memory-limits). A breach fails the +export with `query memory limit exceeded [workload=QUERY, ...]`, where `queryId` +is the copy id in decimal. Exports do not appear in `query_activity`. + ### Export root :::warning diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 533ba5a330..29256373f2 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -1,11 +1,14 @@ --- title: SHOW keyword sidebar_label: SHOW -description: SHOW SQL keyword reference documentation. +description: + SHOW statements for columns, partitions, parameters, and CREATE DDL, plus + Enterprise users, groups, and service accounts with their memory limits. --- -This keyword provides table, column, and partition information including -metadata. The `SHOW` keyword is useful for checking the +`SHOW` returns metadata about tables, columns, partitions, and configuration +parameters and, in QuestDB Enterprise, about users, groups, service accounts, +and permissions. It is useful for checking the [designated timestamp setting](/docs/concepts/designated-timestamp/) column, the [partition attachment settings](/docs/query/sql/alter-table-attach-partition/), and partition storage size on disk. @@ -26,7 +29,7 @@ SHOW { COLUMNS FROM tableName | PERMISSIONS [entityName] | SERVER_VERSION | SERVICE ACCOUNT [accountName] - | SERVICE ACCOUNTS [userName] + | SERVICE ACCOUNTS [{ userName | groupName }] | TABLES | USER [userName] | USERS }; @@ -44,8 +47,8 @@ SHOW { COLUMNS FROM tableName recreate a materialized view. - `SHOW CREATE TABLE` returns a DDL query that allows you to recreate the table. - `SHOW CREATE VIEW` returns a DDL query that allows you to recreate a view. -- `SHOW GROUPS` shows all groups the user belongs or all groups in the system - (enterprise-only) +- `SHOW GROUPS` lists all groups, or the groups a user belongs to, with each + group's external alias and memory limit (enterprise-only) - `SHOW PARAMETERS` shows configuration keys and their matching `env_var_name`, their values and the source of the value - `SHOW PARTITIONS` returns the partition information for the selected table. @@ -53,11 +56,13 @@ SHOW { COLUMNS FROM tableName (enterprise-only) - `SHOW SERVER_VERSION` displays PostgreSQL compatibility version - `SHOW SERVICE ACCOUNT` displays details of a service account (enterprise-only) -- `SHOW SERVICE ACCOUNTS` displays all service accounts or those assigned to the - user/group (enterprise-only) +- `SHOW SERVICE ACCOUNTS` lists all service accounts with their enabled flag and + memory limit, or those a user or group can assume with the grant option + (enterprise-only) - `SHOW TABLES` returns all the tables. - `SHOW USER` shows user secret (enterprise-only) -- `SHOW USERS` shows all users (enterprise-only) +- `SHOW USERS` lists all users with their enabled flag and memory limit + (enterprise-only) ## Examples @@ -349,21 +354,32 @@ including any `DECLARE` parameters if the view is parameterized. ### SHOW GROUPS -_Enterprise only._ +_Enterprise only._ Requires `LIST USERS`; filtering by another user requires +`USER DETAILS`. ```questdb-sql SHOW GROUPS; ``` -or +| name | external_alias | memory_limit | +| ---------- | -------------- | ------------ | +| management | | 2147483648 | + +Filtering by a user lists the groups that user belongs to, with the same +columns. Each row's `memory_limit` is that group's own limit: ```questdb-sql SHOW GROUPS john; ``` -| name | -| ---------- | -| management | +| name | external_alias | memory_limit | +| ---------- | -------------- | ------------ | +| management | | 2147483648 | + +The `memory_limit` column is reported in bytes (`2147483648` is 2 GiB) and is +`null` when the group has no limit of its own. `external_alias` is empty when +the group is not mapped to an external group. See +[memory limits](/docs/security/rbac/#memory-limits). ### SHOW PARAMETERS @@ -518,32 +534,38 @@ SHOW SERVICE ACCOUNT ilp_ingestion; ### SHOW SERVICE ACCOUNTS -_Enterprise only._ +_Enterprise only._ Requires `LIST USERS`; filtering by another user or group +requires `USER DETAILS`. ```questdb-sql SHOW SERVICE ACCOUNTS; ``` -| name | -| ---------- | -| management | -| svc1_admin | +| name | enabled | memory_limit | +| ---------- | ------- | ------------ | +| client_app | true | null | +| svc1_admin | true | 268435456 | + +Filtering by a user or group instead lists the service accounts that principal +can assume. The result has a `grant_option` column in place of `enabled`, +showing whether the user or group may grant the assumption to others, and +`memory_limit` reports each listed service account's own limit: ```questdb-sql SHOW SERVICE ACCOUNTS john; ``` -| name | -| ---------- | -| svc1_admin | +| name | grant_option | memory_limit | +| ---------- | ------------ | ------------ | +| svc1_admin | false | 268435456 | ```questdb-sql SHOW SERVICE ACCOUNTS admin_group; ``` -| name | -| ---------- | -| svc1_admin | +| name | grant_option | memory_limit | +| ---------- | ------------ | ------------ | +| svc1_admin | false | 268435456 | ### SHOW TABLES @@ -581,16 +603,35 @@ SHOW USER john; ### SHOW USERS -_Enterprise only._ +_Enterprise only._ Requires `LIST USERS`. ```questdb-sql SHOW USERS; ``` -| name | -| ----- | -| admin | -| john | +| name | enabled | memory_limit | +| ----- | ------- | ------------ | +| admin | true | null | +| john | true | 536870912 | + +The `memory_limit` column is reported in bytes (`536870912` is 512 MiB) and is +the user's own limit or, when it has none, the most restrictive of its groups'. +`null` means no principal override; the workload limit +(`cairo.query.memory.limit.bytes`) still applies, unlike the `memory_limit` +column of [`query_activity`](/docs/query/functions/meta/#query_activity), which +reports the effective limit and includes it. In `SHOW GROUPS` and +`SHOW SERVICE ACCOUNTS` above it is instead the +listed entity's own limit, since neither inherits one. See +[memory limits](/docs/security/rbac/#memory-limits). + +:::note + +`memory_limit` is appended as the last column of `SHOW USERS`, `SHOW GROUPS`, +and `SHOW SERVICE ACCOUNTS`, including their filtered forms. Tools that bind +these columns by position rather than by name must account for it. See +[upgrading](/docs/security/rbac/#memory-limit-upgrade). + +::: ## See also diff --git a/documentation/query/sql/update.md b/documentation/query/sql/update.md index c8130773e3..106039e254 100644 --- a/documentation/query/sql/update.md +++ b/documentation/query/sql/update.md @@ -25,6 +25,16 @@ SET columnName = expression [, columnName = expression ...] [attached by a symbolic link](/docs/query/sql/alter-table-attach-partition/#symbolic-links), the partition is read-only. `UPDATE` operation on a read-only partition will fail and generate an error. +- On a WAL table, `UPDATE` is applied by the WAL apply job and counts against + [`cairo.wal.apply.memory.limit.bytes`](/docs/configuration/cairo-engine/#cairowalapplymemorylimitbytes) + rather than the query memory limit. This holds for every form of `UPDATE` a + WAL table accepts, including one with a subquery in its `SET` or `WHERE` + clause: the whole statement is written to the WAL and executed by the apply + job. `UPDATE ... FROM`, which joins another table, is not supported on WAL + tables and is rejected with + `UPDATE statements with join are not supported yet for WAL tables`. On a + non-WAL table, `UPDATE` runs on the caller's connection under the caller's + [query memory limit](/docs/configuration/cairo-engine/#memory-limits). ::: diff --git a/documentation/security/oidc.mdx b/documentation/security/oidc.mdx index 4e32f35217..51e5d01ab5 100644 --- a/documentation/security/oidc.mdx +++ b/documentation/security/oidc.mdx @@ -654,6 +654,14 @@ ALTER GROUP groupName WITH EXTERNAL ALIAS 'CN=TestGroup1,OU=DC Users,DC=ad,DC=qu ALTER GROUP groupName DROP EXTERNAL ALIAS 'CN=TestGroup1,OU=DC Users,DC=ad,DC=quest,DC=dev'; ``` +See [`CREATE GROUP`](/docs/query/sql/acl/create-group/) and +[`ALTER GROUP`](/docs/query/sql/acl/alter-group/) for the full syntax. + +External users cannot be given a query memory limit directly; +`ALTER USER ... SET MEMORY LIMIT` is rejected for them. Set the limit on the +mapped QuestDB group instead; see +[memory limits](/docs/security/rbac/#memory-limits). + QuestDB works the list of external groups out from the User Info response message. diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 5b10f3c266..62dd034475 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -1,9 +1,8 @@ --- title: Role-based Access Control (RBAC) description: - Granular access control from database level down to individual columns and - rows. Learn how to secure your QuestDB instance with users, groups, and - fine-grained permissions. + Users, groups, service accounts, and permissions from database down to column + and row level, plus per-principal query memory limits, in QuestDB Enterprise. --- import Screenshot from "@theme/Screenshot" @@ -141,6 +140,17 @@ GRANT ILP TO ingest_app; -- InfluxDB Line Protocol access GRANT INSERT ON sensor_data TO ingest_app; -- Can only insert into sensor_data ``` +### Cap a noisy tenant's query memory + +Bound the native memory each of a user's queries may allocate, so one tenant's +runaway query fails instead of exhausting shared memory: + +```questdb-sql +ALTER USER tenant_a SET MEMORY LIMIT 1G; +``` + +See [Memory limits](#memory-limits) for how limits resolve. + ### Team-based access with groups Multiple users sharing the same permissions: @@ -560,6 +570,9 @@ SHOW USER username; -- Show auth methods for user SHOW PERMISSIONS username; -- Show permissions for user ``` +`SHOW USERS`, `SHOW GROUPS`, and `SHOW SERVICE ACCOUNTS` also report each +entity's [memory limit](#memory-limits). + Example output from `SHOW USER`: ``` @@ -578,6 +591,211 @@ information without these permissions. ::: +## Memory limits {#memory-limits} + +QuestDB Enterprise can limit the native memory tracked for a single query, +overriding the server-wide query memory limit for a specific user, group, or +service account. Use it to help prevent one tenant's runaway query from +exhausting shared memory, or to grant a trusted principal more headroom than the +default. Per-principal limits are available since QuestDB Enterprise 4.0.2; the +server-wide [workload limits](/docs/configuration/cairo-engine/#memory-limits) +they override are available since QuestDB 10.0.0. + +Set a limit with [`ALTER USER`](/docs/query/sql/acl/alter-user/), +[`ALTER GROUP`](/docs/query/sql/acl/alter-group/), or +[`ALTER SERVICE ACCOUNT`](/docs/query/sql/acl/alter-service-account/): + +```questdb-sql +ALTER USER john SET MEMORY LIMIT 512M; +ALTER GROUP analysts SET MEMORY LIMIT 2G; +ALTER SERVICE ACCOUNT ingest_app SET MEMORY LIMIT 1G; +ALTER USER john SET MEMORY LIMIT UNLIMITED; -- clear the user's own limit +``` + +The value is a byte count or a size with a `K`, `M`, or `G` suffix. Each suffix +multiplies by 1024, so `512M` is 536870912 bytes. Setting a limit requires the +`SET MEMORY LIMIT` permission, which is included in `GRANT ALL` and held +implicitly by database admins. + +:::warning + +Treat `SET MEMORY LIMIT` as an administrative permission, not a self-service +one. It takes no entity name, so its holder can set the limit of any principal, +including its own. Because a per-principal limit overrides the workload limit +rather than tightening it (see [How limits resolve](#how-limits-resolve)), a +non-admin who holds it can raise its own ceiling above +`cairo.query.memory.limit.bytes`. The built-in admin is exempt only as a +target: its limit cannot be set at all. + +`GRANT ALL` expands to individual permissions at the moment it is granted. A +principal granted `ALL` before upgrading to a version with this feature does +not acquire `SET MEMORY LIMIT`; only grants issued after the upgrade include it, +and no migration backfills it. Grant it explicitly to existing administrators: +`GRANT SET MEMORY LIMIT TO admins;`. + +::: + +### How limits resolve + +QuestDB resolves the limit for a query by strict precedence. It takes the first +level that is set, not the smallest across levels: + +1. **The principal's own limit.** For a user this is the user's own limit; for a + query that assumes a service account it is the service account's limit. +2. **A group limit** (users only). When a user has no limit of its own, it + inherits the + most restrictive (smallest positive) limit among the groups it belongs to. + Service accounts never inherit group limits. +3. **The workload limit.** Otherwise the server-wide + [`cairo.query.memory.limit.bytes`](/docs/configuration/cairo-engine/#cairoquerymemorylimitbytes) + applies. + +A value of `0` (or `UNLIMITED`) means "not set" at that level, so resolution +falls through to the next one. A more specific level, when set, fully overrides +the broader one and binds even when it is larger, so a per-user or per-group +override can raise a principal's ceiling above the workload limit, not only lower +it. A user who assumes a service account takes on the service account's limit. +The built-in admin cannot be given a limit, so `ALTER USER admin SET MEMORY +LIMIT` is rejected with `Cannot set memory limit for built-in admin`, and the +admin runs under the workload limit. Size that limit with the admin's +diagnostic queries in mind. + +The cap applies to the principal's queries on both the primary and replicas. The +statement itself runs on the primary only: a replica rejects it with +`replica cannot set memory limit` and receives the new value through the +replicated ACL tables. A changed limit applies to the +principal's next query, including on connections that are already open, while a +query already running keeps the limit it started with. + +A query that crosses its limit fails with the same +`query memory limit exceeded [workload=QUERY, ...]` error as a breach of the +workload limit. See +[memory limits](/docs/configuration/cairo-engine/#memory-limits) for the message +format. + +:::note + +A limit bounds a single query. Two concurrent queries by the same principal each +run under the full limit, so the principal's aggregate usage can exceed it. The +cap guards against one runaway workload, not total concurrent usage. + +::: + +### What a per-principal limit covers + +Per-principal limits have the same +[coverage](/docs/configuration/cairo-engine/#memory-limits) as workload limits. +They apply to tracked native allocations for: + +- The principal's queries. +- Its background [`COPY ... TO`](/docs/query/sql/copy/) exports, which use the + issuing principal's limit. Some memory used to produce the export file is not + yet covered, and exports do not appear in `query_activity`, so their usage + cannot be observed while they run. +- `UPDATE` on a non-WAL table, which is applied on the caller's own thread and + acquires its own query-workload tracker. + +It does not reach work that runs under an internal context rather than a +principal. That work stays bounded only by its own +[workload limit](/docs/configuration/cairo-engine/#memory-limits): + +- `UPDATE` on a WAL table, the default table type, because the statement + is applied by the WAL apply job and draws on that job's + `cairo.wal.apply.memory.limit.bytes` budget instead. Whether a large `UPDATE` + is capped by a `SET MEMORY LIMIT` override therefore depends on the table + type. A WAL-table `UPDATE` that breaches the WAL apply limit suspends the + table for every principal until + [`ALTER TABLE RESUME WAL`](/docs/query/sql/alter-table-resume-wal/), so the + failure is not isolated to the tenant that issued it. +- Materialized view refresh, live view refresh, and WAL apply itself. A WAL + apply batches many principals' transactions into one tracker, so it could not + attribute usage to a single principal in any case. +- `COPY ... FROM` imports, which acquire no memory tracker at all and are + unaffected by either kind of limit. + +:::note + +An external (SSO/OIDC) user can only receive a limit by inheriting one from a +group: `ALTER USER ... SET MEMORY LIMIT` on an external user is rejected with +`Cannot set memory limit for external user`. The user receives its inherited +group limit at login. On the primary, a later `ALTER GROUP ... SET MEMORY LIMIT` +also reaches the user's open sessions at their next query. On a replica, and on +the primary after a restart or promotion, an existing external session keeps +the limit it logged in with until it reconnects. + +::: + +### Inspecting limits + +- `SHOW USERS`, `SHOW GROUPS`, and `SHOW SERVICE ACCOUNTS` report per-principal + limits in a `memory_limit` column, in bytes. For users, it is the user's own + limit or, when it has none, the most restrictive of its groups'. For groups + and service accounts, it is the entity's own limit. The column excludes the + server-wide workload limit. `null` means the entity has no limit of its own + and, for a user, no inherited one either, so the workload limit applies to + its queries. +- The filtered forms `SHOW GROUPS userName` and + `SHOW SERVICE ACCOUNTS { userName | groupName }` carry the column too, reporting each + listed group's or service account's own limit. This shows which inherited + limit binds for a user with no limit of its own. +- [`query_activity`](/docs/query/functions/meta/#query_activity) exposes the + effective limit and live usage of each running query through its `memory_limit` + and `memory_used` columns. +- `SHOW USER` does not carry the column. The unfiltered `SHOW USERS`, + `SHOW GROUPS`, and `SHOW SERVICE ACCOUNTS` require `LIST USERS`; the filtered + forms require `USER DETAILS` unless the caller names itself or one of its own + groups. A user without `LIST USERS` can still read its own effective cap from the + `memory_limit` column of `query_activity`, which always lists the caller's + own queries. +- The stored value is persisted on the `sys.acl_entities` system table. That + table is protected: only the built-in admin can read it, and an ACL principal + holding `DATABASE ADMIN` is still denied. + +### Upgrading {#memory-limit-upgrade} + +:::warning Breaking change + +`memory_limit` is appended as the last column of `SHOW USERS`, `SHOW GROUPS`, +and `SHOW SERVICE ACCOUNTS`, including their filtered forms, whether or not any +limit is set. `SELECT *` on `sys.acl_entities` returns one more column as well. +Clients that read these results by position must be updated; clients that read +by column name are unaffected. + +::: + +The `SHOW` results carry the column from the first start of the upgraded +binary, reading `null` until limits are set. The `sys.acl_entities` column that +stores the value is added by an automatic migration when an upgraded node first +starts as a primary or is promoted from replica to primary. Persisted principal +limits are not enforced until the migration has been applied. The migration passes +through two windows, and each refuses a different set of statements: + +- Before the column exists, a `SET MEMORY LIMIT` with a non-zero size is refused + with: + + ``` + Cannot modify ACL entities: the memory_limit column has not been migrated in yet; retry shortly, or restart the node if it persists + ``` + + `SET MEMORY LIMIT UNLIMITED` and `ALTER ... ENABLE` or `DISABLE` still + succeed. + +- Once the column has been added but WAL apply has not yet reached it, + `SET MEMORY LIMIT UNLIMITED` and `ALTER ... ENABLE` or `DISABLE` are refused + as well, so that a stale in-memory value cannot overwrite a stored limit. A + non-zero `SET MEMORY LIMIT` keeps the first message; the others read: + + ``` + Cannot set memory limit while the ACL memory_limit column migration is still being applied, retry [name=john] + ``` + + or `Cannot enable while ...` and `Cannot disable while ...` for a status + change. + +The window normally closes on its own once WAL apply catches up, so retry the +statement first. If the error persists, restart the node: the migration runs +again at startup. + ## Permissions reference {#permissions} Use `all_permissions()` to see all available permissions: @@ -657,6 +875,7 @@ SELECT * FROM all_permissions(); | REMOVE EXTERNAL ALIAS | Remove external group mappings | | REMOVE PASSWORD | Remove passwords | | REMOVE USER | Remove users from groups | +| SET MEMORY LIMIT | Set memory limits on users, groups, and service accounts | | USER DETAILS | View user/group/service account details | ### Special permissions @@ -681,8 +900,9 @@ replication role, is an ordinary grantable permission. ## SQL commands reference - [ADD USER](/docs/query/sql/acl/add-user/) -- [ALTER USER](/docs/query/sql/acl/alter-user/) +- [ALTER GROUP](/docs/query/sql/acl/alter-group/) - [ALTER SERVICE ACCOUNT](/docs/query/sql/acl/alter-service-account/) +- [ALTER USER](/docs/query/sql/acl/alter-user/) - [ASSUME SERVICE ACCOUNT](/docs/query/sql/acl/assume-service-account/) - [CREATE GROUP](/docs/query/sql/acl/create-group/) - [CREATE SERVICE ACCOUNT](/docs/query/sql/acl/create-service-account/) diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 2c4c83c2a1..4c6bfe0cbf 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -287,6 +287,10 @@ module.exports = { type: "category", label: "ALTER", items: [ + { + id: "query/sql/acl/alter-group", + type: "doc", + }, { id: "query/sql/acl/alter-service-account", type: "doc",