diff --git a/src/main/asciidoc/core-concepts/chapter.adoc b/src/main/asciidoc/core-concepts/chapter.adoc index b7e90e15..44ca12df 100644 --- a/src/main/asciidoc/core-concepts/chapter.adoc +++ b/src/main/asciidoc/core-concepts/chapter.adoc @@ -13,6 +13,8 @@ include::indexes.adoc[] include::full-text-index.adoc[] +include::materialized-views.adoc[] + include::graphs.adoc[] include::databases.adoc[] diff --git a/src/main/asciidoc/core-concepts/materialized-views.adoc b/src/main/asciidoc/core-concepts/materialized-views.adoc new file mode 100644 index 00000000..7cb657e2 --- /dev/null +++ b/src/main/asciidoc/core-concepts/materialized-views.adoc @@ -0,0 +1,217 @@ +[[materialized-views]] +=== Materialized Views +image:../images/edit.png[link="https://github.com/ArcadeData/arcadedb-docs/blob/main/src/main/asciidoc/core-concepts/materialized-views.adoc" float=right] + +A materialized view is a schema-level object that stores the result of a SQL `SELECT` query as a backing document type. +Unlike regular views (which re-execute the query on every access), a materialized view holds a pre-computed snapshot of data that can be queried directly for fast reads. + +A materialized view: + +* Wraps a SQL `SELECT` query as its _defining query_ +* Stores results in a backing `DocumentType` with standard buckets +* Supports three _refresh modes_: manual, incremental (post-commit), and periodic (scheduled) +* Persists its definition and metadata in `schema.json` alongside other schema objects +* Can be created, dropped, refreshed, and altered via SQL DDL statements or the Java Schema API + +==== Refresh Modes + +[[refresh-mode-manual]] +===== MANUAL + +The view data is never automatically updated. +You must trigger a refresh explicitly via <> or the Java API. +Use this when you control refresh timing yourself or when source data changes infrequently. + +[[refresh-mode-incremental]] +===== INCREMENTAL + +After every committed transaction that modifies a source type, ArcadeDB automatically refreshes the view in a post-commit callback. +The refresh is: + +* *Transactionally safe*: runs after the source transaction commits successfully; rolled-back transactions do not trigger a refresh +* *Batched per transaction*: multiple record changes in a single transaction result in one refresh, not one per record +* For *simple queries* (single source type, no aggregates, no `GROUP BY`, no subqueries, no `JOIN`s): performs a full refresh +* For *complex queries* (aggregates, `GROUP BY`, etc.): also performs a full refresh (per-record incremental optimization is planned for a future release) + +If the refresh fails, the view is marked `STALE` and a warning is logged. +A manual refresh can recover it. + +[[refresh-mode-periodic]] +===== PERIODIC + +A background scheduler thread runs a full refresh at the specified interval after each successful refresh completes. +Intervals are specified in seconds, minutes, or hours: + +[source,sql] +---- +REFRESH EVERY 30 SECOND +REFRESH EVERY 5 MINUTE +REFRESH EVERY 1 HOUR +---- + +The scheduler uses a single daemon thread (`ArcadeDB-MV-Scheduler`) shared across all periodic views. +If the database is closed, all scheduled tasks are cancelled automatically. + +==== View Status + +Each view tracks a `status` field that reflects its current state: + +[%header,cols=2] +|=== +| Status | Meaning +| `VALID` | Data is up to date with the last refresh +| `STALE` | A refresh failed or was interrupted; data may be outdated +| `BUILDING` | A refresh is currently in progress +| `ERROR` | The last refresh encountered a fatal error +|=== + +If the database crashes while a view is `BUILDING`, the status is reset to `STALE` on the next startup to signal that the data may be incomplete. + +==== Querying a Materialized View + +Query a materialized view exactly like any other document type: + +[source,sql] +---- +SELECT * FROM ActiveUsers +SELECT name FROM ActiveUsers WHERE name LIKE 'A%' +SELECT count(*) FROM RecentOrders +---- + +==== Java API + +*Creating a view* + +[source,java] +---- +database.transaction(() -> { + database.getSchema().buildMaterializedView() + .withName("ActiveUsers") + .withQuery("SELECT name, email FROM User WHERE active = true") + .withRefreshMode(MaterializedViewRefreshMode.MANUAL) + .create(); +}); +---- + +Builder options: + +[%header,cols=2] +|=== +| Method | Description +| `withName(String)` | Name for the view (required) +| `withQuery(String)` | Defining SQL `SELECT` query (required) +| `withRefreshMode(MaterializedViewRefreshMode)` | `MANUAL`, `INCREMENTAL`, or `PERIODIC` (default: `MANUAL`) +| `withTotalBuckets(int)` | Number of buckets for the backing type +| `withPageSize(int)` | Page size for the backing type +| `withRefreshInterval(long)` | Interval in milliseconds for `PERIODIC` mode +| `withIgnoreIfExists(boolean)` | When `true`, returns existing view instead of throwing +|=== + +*Querying schema* + +[source,java] +---- +Schema schema = database.getSchema(); + +// Check existence +boolean exists = schema.existsMaterializedView("ActiveUsers"); + +// Get a specific view +MaterializedView view = schema.getMaterializedView("ActiveUsers"); + +// List all views +MaterializedView[] views = schema.getMaterializedViews(); +---- + +*Refreshing and dropping* + +[source,java] +---- +// Programmatic refresh +database.getSchema().getMaterializedView("ActiveUsers").refresh(); + +// Drop via schema +database.getSchema().dropMaterializedView("ActiveUsers"); + +// Drop via the view itself +database.getSchema().getMaterializedView("ActiveUsers").drop(); +---- + +*Inspecting a view* + +[source,java] +---- +MaterializedView view = database.getSchema().getMaterializedView("HourlySummary"); + +view.getName(); // "HourlySummary" +view.getQuery(); // the defining SQL query +view.getRefreshMode(); // MaterializedViewRefreshMode.PERIODIC +view.getStatus(); // "VALID", "STALE", "BUILDING", or "ERROR" +view.getLastRefreshTime(); // epoch millis of last successful refresh +view.isSimpleQuery(); // true if eligible for per-record optimization +view.getSourceTypeNames(); // list of source type names parsed from the query +view.getBackingType(); // the underlying DocumentType +---- + +==== Behavior and Constraints + +* *Backing type protection*: You cannot `DROP TYPE` on a type that backs a materialized view. Drop the materialized view first. +* *Name uniqueness*: The view name must not match any existing type or materialized view. +* *Source type validation*: All types referenced in the `FROM` clause must exist when the view is created. +* *Persistence*: View definitions are stored in `schema.json` under a `"materializedViews"` key and survive database restarts. Listener registration for `INCREMENTAL` views and scheduler tasks for `PERIODIC` views are re-established on startup. +* *Transaction safety*: The initial full refresh and all subsequent refreshes run inside their own transactions. +* *Query result columns*: Only non-internal properties (those not starting with `@`) are copied into the backing type during refresh. +* *No schema on backing type*: The backing document type is schema-less; property types are not enforced. + +==== Error Handling + +* If a post-commit refresh fails (`INCREMENTAL` mode), the view is marked `STALE` and a `WARNING` is logged. The source transaction is unaffected. +* If a periodic refresh fails, the view is marked `ERROR` and a `SEVERE` log entry is written. The scheduler continues running and will retry on the next interval. +* Callback errors in the transaction callback system are logged at `WARNING` level and do not affect the triggering transaction or other callbacks. + +==== Limitations + +* <> is not yet implemented. +* Per-record incremental refresh (tracking `_sourceRID` to update individual view rows) is a planned future optimization. Currently, all refresh operations perform a full truncate-and-reload. +* No support for cross-database queries in the defining query. +* Server replication: materialized view data lives in the local backing type and is replicated like any other document type in an HA cluster, but refresh triggering is local to the node that executes the write. + +==== Example: Sales Dashboard + +[source,sql] +---- +-- Source type +CREATE DOCUMENT TYPE Sale; + +-- A periodic summary refreshed every minute +CREATE MATERIALIZED VIEW SalesByProduct + AS SELECT product, sum(amount) AS total, count(*) AS count + FROM Sale + GROUP BY product + REFRESH EVERY 1 MINUTE; + +-- An incremental view of recent activity (simple query) +CREATE MATERIALIZED VIEW RecentSales + AS SELECT product, amount, date + FROM Sale + WHERE date >= '2026-01-01' + REFRESH INCREMENTAL; + +-- Query the views +SELECT * FROM SalesByProduct ORDER BY total DESC; +SELECT product, amount FROM RecentSales WHERE amount > 1000; + +-- Manual refresh after a bulk import +REFRESH MATERIALIZED VIEW SalesByProduct; + +-- Teardown +DROP MATERIALIZED VIEW SalesByProduct; +DROP MATERIALIZED VIEW RecentSales; +---- + +For more information, see: + +* <> +* <> +* <> +* <> diff --git a/src/main/asciidoc/query-languages/sql/chapter.adoc b/src/main/asciidoc/query-languages/sql/chapter.adoc index 6e870a17..5cd1585d 100644 --- a/src/main/asciidoc/query-languages/sql/chapter.adoc +++ b/src/main/asciidoc/query-languages/sql/chapter.adoc @@ -13,8 +13,10 @@ image:../images/edit.png[link="https://github.com/ArcadeData/arcadedb-docs/blob/ | <> | <> | <> | | <> | <> | <> | | <> | ALTER BUCKET (_not implemented_) | <> | -| <> | <> | | -| <> | <> | | +| <> | <> | <> | +| <> | <> | <> | +| | | <> | +| | | <> (_planned_) | | <> | <> | <> | | | <> | <> | | | <> | <> | @@ -126,6 +128,8 @@ include::sql-align-database.adoc[] include::sql-alter-database.adoc[] +include::sql-alter-materialized-view.adoc[] + include::sql-alter-property.adoc[] include::sql-alter-type.adoc[] @@ -140,6 +144,8 @@ include::sql-create-bucket.adoc[] include::sql-create-edge.adoc[] +include::sql-create-materialized-view.adoc[] + include::sql-create-index.adoc[] include::sql-create-property.adoc[] @@ -156,6 +162,8 @@ include::sql-drop-bucket.adoc[] include::sql-drop-index.adoc[] +include::sql-drop-materialized-view.adoc[] + include::sql-drop-property.adoc[] include::sql-drop-trigger.adoc[] @@ -178,6 +186,8 @@ include::sql-profile.adoc[] include::sql-rebuild-index.adoc[] +include::sql-refresh-materialized-view.adoc[] + include::sql-select.adoc[] include::sql-select-execution.adoc[] diff --git a/src/main/asciidoc/query-languages/sql/sql-alter-materialized-view.adoc b/src/main/asciidoc/query-languages/sql/sql-alter-materialized-view.adoc new file mode 100644 index 00000000..99691b76 --- /dev/null +++ b/src/main/asciidoc/query-languages/sql/sql-alter-materialized-view.adoc @@ -0,0 +1,27 @@ +[[sql-alter-materialized-view]] +[discrete] +==== SQL - `ALTER MATERIALIZED VIEW` +image:../images/edit.png[link="https://github.com/ArcadeData/arcadedb-docs/blob/main/src/main/asciidoc/query-languages/sql/sql-alter-materialized-view.adoc" float=right] + +Changes the refresh mode of an existing materialized view. + +NOTE: `ALTER MATERIALIZED VIEW` is planned for a future release and is not yet implemented. + +*Syntax* + +[source,sql] +---- +ALTER MATERIALIZED VIEW + REFRESH MANUAL | REFRESH INCREMENTAL | REFRESH EVERY SECOND | MINUTE | HOUR +---- + +* *``* The name of the materialized view to alter. +* *`REFRESH MANUAL`* Switches the view to manual refresh mode. +* *`REFRESH INCREMENTAL`* Switches the view to incremental refresh mode (automatic after each relevant commit). +* *`REFRESH EVERY SECOND | MINUTE | HOUR`* Switches the view to periodic refresh mode at the specified interval. + +For more information, see: + +* <> +* <> +* <> diff --git a/src/main/asciidoc/query-languages/sql/sql-create-materialized-view.adoc b/src/main/asciidoc/query-languages/sql/sql-create-materialized-view.adoc new file mode 100644 index 00000000..5d78fb44 --- /dev/null +++ b/src/main/asciidoc/query-languages/sql/sql-create-materialized-view.adoc @@ -0,0 +1,78 @@ +[[sql-create-materialized-view]] +[discrete] +==== SQL - `CREATE MATERIALIZED VIEW` +image:../images/edit.png[link="https://github.com/ArcadeData/arcadedb-docs/blob/main/src/main/asciidoc/query-languages/sql/sql-create-materialized-view.adoc" float=right] + +Creates a new materialized view in the schema. +A materialized view stores the result of a SQL `SELECT` query in a backing document type for fast reads. +The defining query is executed immediately on creation to perform the initial data load. + +*Syntax* + +[source,sql] +---- +CREATE MATERIALIZED VIEW [IF NOT EXISTS] + AS + [REFRESH MANUAL | REFRESH INCREMENTAL | REFRESH EVERY SECOND | MINUTE | HOUR] + [BUCKETS ] +---- + +* *``* Defines the name of the materialized view. It must not conflict with any existing type or materialized view. +* *`IF NOT EXISTS`* Silently skips creation if a view with the same name already exists, instead of raising an error. +* *``* A SQL `SELECT` statement that defines the data for the view. All source types referenced must exist at creation time. +* *`REFRESH MANUAL`* (default) The view is only refreshed when triggered explicitly. +* *`REFRESH INCREMENTAL`* The view is automatically refreshed after each committed transaction that modifies a source type. +* *`REFRESH EVERY SECOND | MINUTE | HOUR`* The view is refreshed on a schedule at the given interval. +* *`BUCKETS `* Sets the number of buckets for the backing document type. + +*Examples* + +* Create a simple manual-refresh view: + +[source,sql] +---- +CREATE MATERIALIZED VIEW ActiveUsers + AS SELECT name, email FROM User WHERE active = true +---- + +* Create a view that updates automatically after each relevant commit: + +[source,sql] +---- +CREATE MATERIALIZED VIEW RecentOrders + AS SELECT customer, amount FROM Order WHERE status = 'PENDING' + REFRESH INCREMENTAL +---- + +* Create a view that refreshes on a schedule: + +[source,sql] +---- +CREATE MATERIALIZED VIEW HourlySummary + AS SELECT product, sum(amount) AS total FROM Sale GROUP BY product + REFRESH EVERY 1 HOUR +---- + +* Skip creation if the view already exists: + +[source,sql] +---- +CREATE MATERIALIZED VIEW IF NOT EXISTS ActiveUsers + AS SELECT name FROM User WHERE active = true +---- + +* Create a view with a custom bucket count for large result sets: + +[source,sql] +---- +CREATE MATERIALIZED VIEW BigView + AS SELECT * FROM HugeType + BUCKETS 8 +---- + +For more information, see: + +* <> +* <> +* <> +* <> diff --git a/src/main/asciidoc/query-languages/sql/sql-drop-materialized-view.adoc b/src/main/asciidoc/query-languages/sql/sql-drop-materialized-view.adoc new file mode 100644 index 00000000..c1d475b2 --- /dev/null +++ b/src/main/asciidoc/query-languages/sql/sql-drop-materialized-view.adoc @@ -0,0 +1,39 @@ +[[sql-drop-materialized-view]] +[discrete] +==== SQL - `DROP MATERIALIZED VIEW` +image:../images/edit.png[link="https://github.com/ArcadeData/arcadedb-docs/blob/main/src/main/asciidoc/query-languages/sql/sql-drop-materialized-view.adoc" float=right] + +Removes a materialized view and its backing document type, including all stored data. + +*Syntax* + +[source,sql] +---- +DROP MATERIALIZED VIEW [IF EXISTS] +---- + +* *``* The name of the materialized view to remove. +* *`IF EXISTS`* Suppresses errors when the view does not exist. + +*Examples* + +* Drop a materialized view: + +[source,sql] +---- +DROP MATERIALIZED VIEW ActiveUsers +---- + +* Drop a view only if it exists: + +[source,sql] +---- +DROP MATERIALIZED VIEW IF EXISTS NonExistentView +---- + +NOTE: You cannot drop the backing document type of a materialized view directly using `DROP TYPE`. Drop the materialized view first. + +For more information, see: + +* <> +* <> diff --git a/src/main/asciidoc/query-languages/sql/sql-refresh-materialized-view.adoc b/src/main/asciidoc/query-languages/sql/sql-refresh-materialized-view.adoc new file mode 100644 index 00000000..4e8bdb82 --- /dev/null +++ b/src/main/asciidoc/query-languages/sql/sql-refresh-materialized-view.adoc @@ -0,0 +1,41 @@ +[[sql-refresh-materialized-view]] +[discrete] +==== SQL - `REFRESH MATERIALIZED VIEW` +image:../images/edit.png[link="https://github.com/ArcadeData/arcadedb-docs/blob/main/src/main/asciidoc/query-languages/sql/sql-refresh-materialized-view.adoc" float=right] + +Triggers an immediate full refresh of a materialized view's data. +The operation truncates all existing data in the backing type and reloads it by re-executing the defining query. +The refresh runs inside its own transaction. + +This command is useful for `MANUAL` refresh mode views and for recovering a `STALE` view after a failed automatic refresh. + +*Syntax* + +[source,sql] +---- +REFRESH MATERIALIZED VIEW +---- + +* *``* The name of the materialized view to refresh. + +*Examples* + +* Refresh a view immediately: + +[source,sql] +---- +REFRESH MATERIALIZED VIEW HourlySummary +---- + +* Refresh after a bulk import to bring a manual view up to date: + +[source,sql] +---- +REFRESH MATERIALIZED VIEW SalesByProduct +---- + +For more information, see: + +* <> +* <> +* <>