DuckDB: no help for missing incremental calculation, show full recalculation as such

CHANGELOG.md

@@ -4,6 +4,11 @@ Changes to the model, the data, the tests and the method, newest first. A change
 
 ## 0.9-draft, 2026-11-04
 
+- DuckDB gets no help with incremental calculation any more. `duckdb-tables` keeps its calculated tables as ordinary durable tables in the database file instead of in-memory `TEMP` tables, and every write, load of new data or model change recalculates all of them, with `sql/calculate.sql`, instead of only the tables the change reaches. R8 opens the file and builds nothing. DuckDB has no incremental calculation, and the benchmark shows that gap instead of compensating for it. The `rebuild/` SQL is gone from the entry. The ClickHouse and PostgreSQL entries are not changed. The DuckDB results published so far are from the earlier setup and are being measured again.
+- The `duckdb-tables` write, increment load (L2), master data change (L3) and model changes (M1 to M4) commit their input change together with `calculated = false` in a one-row table, `calc_state`. They then calculate every table outside any transaction, one commit per table, and commit `calculated = true` last. An open that finds `calculated = false`, because the process died in a recalculation, calculates every table again before the first read, so after a crash the first open pays one full recalculation. R8 normally builds nothing. One transaction around the input change and the recalculation was tried first: on an n2d-standard-4 one recalculation took about 271 s inside the transaction against about 4 s outside, so a 10m run would have taken about 20 hours.
+- New `duckdb-tables` results at 10m and 100m, on an n2d-standard-4, from FPABench with #53. Every test passed. The whole run took 11.4 min at 10m and 56.7 min at 100m. A write, a load of new data, a master data change and a model change each take about one full recalculation, 3.6 s at 10m and 17.5 s at 100m, and R8 takes 0.07 s and 0.08 s. The DuckDB results at 1b are from the earlier setup and are not measured again.
+- `duckdb-tables` gets a Caveats section (its README, the results README and the website notice): recalculating inside one transaction costs far more than outside it, and DuckDB commits each statement by itself and has no materialized views or triggers, so the entry takes the fast but not atomic choice with a calculated flag.
+- A test result can carry `"recalc": "full"`: the system has no incremental calculation, so it recalculated everything. The runner copies it from the test file's `recalc` export. The website shows such tests grey with diagonal stripes and the text "not supported - full recalc results", in the legend and in a tooltip, in both themes. The time stays in the score. Only the DuckDB entry sets it, on L2, L3, W1 to W8 and M1 to M4. `tests/method.md` defines it.
 - A new entry, `postgres-tables`: PostgreSQL 18 with one `UNLOGGED` table per calculated variable, built again after each change that reaches it (#42). The fixes from its review followed (#44, #45, #46, #49). Its calculated tables survive a clean stop, so its R8 starts without building them again, as `tests/method.md` allows for stored results.
 - First results for PostgreSQL 18.6, at 10m on an n2d-standard-4, from FPABench with #49. Every test passed, and the whole run took 17.3 minutes. Its loads, writes and model changes take 4 to 6 times as long as DuckDB's and ClickHouse's, and its R8 takes 0.06 s. There are no PostgreSQL results at 100m and 1b yet.
 

model/lib/sql/generate.mjs

@@ -6,9 +6,10 @@
 //
 // schema.sql holds the empty tables and calculate.sql every calculated table. For each entry, members.sql,
 // dimensions.sql and import/<Variable>.sql hold the load. PostgreSQL's L1 loads with load/<Variable>.sql, the
-// same without ON CONFLICT. rebuild/<Variable>.sql holds what a write to an input
-// variable must run. m1.sql and m4.sql are the migrations of tests M1
-// and M4: only what a database built from calculate.sql must run to reach the changed model. M2 and M3 change members only, and the
+// same without ON CONFLICT. For an entry that rebuilds only what a write reaches, rebuild/<Variable>.sql holds what a
+// write to an input variable must run. An entry with `fullRecalc` has none: every write runs calculate.sql.
+// m1.sql and m4.sql are the migrations of tests M1 and M4: only what a database built from calculate.sql must
+// run to reach the changed model, or with `fullRecalc` the new tables and then all of calculate.sql. M2 and M3 change members only, and the
 // SQL never names a member that only the data brings, so they need no SQL of their own. The SQL does not depend
 // on the dataset: members the model names have fixed ids (see dict.mjs), and the Versions and Years come
 // from the data generator's own settings in data/params.mjs.
@@ -35,8 +36,10 @@ const system = {
     Months: MONTHS,
 };
 
-// Each entry and the translator options it uses.
-const ENTRIES = { 'duckdb-tables': {}, 'clickhouse-tables': { dialect: 'clickhouse' }, 'postgres-tables': { dialect: 'postgres' } };
+// Each entry and the translator options it uses. `fullRecalc` is for an entry whose system has no incremental
+// calculation: it runs all of calculate.sql after every change, so it gets no rebuild/ files and its
+// migrations build every table.
+const ENTRIES = { 'duckdb-tables': { fullRecalc: true }, 'clickhouse-tables': { dialect: 'clickhouse' }, 'postgres-tables': { dialect: 'postgres' } };
 const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'fpabench-sql-'));
 const MODELS = {
     base: path.join(root, 'model'),
@@ -225,7 +228,7 @@ const WHOLE = [
 
 const files = new Map();
 const stmts = (text) => new Set(text.split(/;\s*\n/).map((x) => x.trim()).filter(Boolean));
-for (const [entry, options] of Object.entries(ENTRIES)) {
+for (const [entry, { fullRecalc = false, ...options }] of Object.entries(ENTRIES)) {
     const dialect = options.dialect ?? 'duckdb';
     const make = (dir) => {
         const model = loadModel(dir);
@@ -250,7 +253,7 @@ for (const [entry, options] of Object.entries(ENTRIES)) {
         for (const [rel, text] of Object.entries(load(model, newDict(model, system)))) files.set(`${entry}/sql/${rel}`, text);
         // What a write to each input variable must run: every calculated table that reads it, directly or
         // through other tables, in dependency order.
-        for (const v of model.variables.filter((v) => !v.formula && !v.measure)) {
+        for (const v of fullRecalc ? [] : model.variables.filter((v) => !v.formula && !v.measure)) {
             const steps = base.rebuildSql(new Set([v.name]));
             const names = base.calculated.filter((n, i) => steps.includes(base.tables[i]));
             files.set(
@@ -282,17 +285,19 @@ for (const [entry, options] of Object.entries(ENTRIES)) {
         if (next.lookups.trim() !== base.lookups.trim()) throw new Error(`${name} changes the dimension lookups, which a migration does not do yet`);
         const newSchema = [...stmts(next.schema)].filter((x) => !baseSchema.has(x));
         const changed = next.calculated.filter((v, i) => baseTable.get(v) !== next.tables[i].trim());
-        const steps = next.rebuildSql(new Set(changed));
-        const stepNames = next.calculated.filter((v, i) => steps.includes(next.tables[i]));
+        // With fullRecalc every table is built again, after the dimension lookups, as calculate.sql does.
+        const steps = fullRecalc ? next.tables : next.rebuildSql(new Set(changed));
+        const stepNames = fullRecalc ? next.calculated : next.calculated.filter((v, i) => steps.includes(next.tables[i]));
         files.set(
             `${entry}/sql/${name}.sql`,
             [
                 `-- Generated by model/lib/sql/generate.mjs: the migration of test ${name.toUpperCase()}, ${dialect} dialect. Do not edit.`,
-                `-- It takes a database built from calculate.sql to the changed model. New or changed tables: ${changed.length}. Statements: ${steps.length}.\n`,
+                `-- It takes a database built from calculate.sql to the changed model. New or changed tables: ${changed.length}. ${fullRecalc ? `Every one of the ${steps.length} calculated tables is built again.` : `Statements: ${steps.length}.`}\n`,
                 '-- @schema: new input tables',
                 newSchema.map((x) => `${x};`).join('\n'),
                 '',
-                ...stepNames.map((v, i) => `-- @table ${JSON.stringify(v)}${changed.includes(v) ? '' : ' (reads a changed table)'}\n${steps[i].trim()}\n`),
+                ...(fullRecalc ? ['-- @recalculate: every calculated table, built again', '-- The dimension lookups: one flattened table per dimension.', next.lookups.trim(), ''] : []),
+                ...stepNames.map((v, i) => `-- @table ${JSON.stringify(v)}${fullRecalc || changed.includes(v) ? '' : ' (reads a changed table)'}\n${steps[i].trim()}\n`),
             ].join('\n')
         );
         files.set(`${entry}/sql/${name}.json`, JSON.stringify({ model: name, dialect, system, changed, tables: stepNames }, null, 1) + '\n');

model/lib/sql/translate.mjs

@@ -1,6 +1,7 @@
 // Translates the FPABench model into SQL for DuckDB, ClickHouse or PostgreSQL: one table per input variable, and one
 // table per calculated variable, built in dependency order. Running the model SQL again after a write
-// rebuilds them all.
+// rebuilds them all. `rebuildSql` also gives the dependency-ordered subset a change reaches, for the entries
+// that rebuild only that.
 //
 // Neither system can keep a derived table current by itself. DuckDB has no materialized views and no
 // triggers. ClickHouse's materialized views react to inserts into one table only. Plain views give the
@@ -50,11 +51,12 @@ export function translate(model, { versions, years, dialect = 'duckdb', dict })
     const list = (d, xs) => (CH ? `arrayJoin([${xs.map((x) => mid(d, x)).join(', ')}])` : `unnest(${PG ? 'ARRAY' : ''}[${xs.map((x) => mid(d, x)).join(', ')}])`);
     // ClickHouse forces every new part to disk, so writes are durable, as DuckDB's are by default.
     const DURABLE = "SETTINGS fsync_after_insert = 1, fsync_part_directory = 1";
-    // Calculated tables live in memory: DuckDB TEMP tables, ClickHouse Memory tables. PostgreSQL's are
-    // UNLOGGED tables, which skip the write-ahead log, since its TEMP tables belong to one session. Only the inputs need
-    // to survive a crash, since every calculated table can be built again from them. After a restart DuckDB and
-    // ClickHouse build them again on start, and that time counts in R8. PostgreSQL keeps UNLOGGED tables over a
-    // clean restart, and empties them only in crash recovery.
+    // DuckDB's calculated tables are ordinary tables in the database file, so they are as durable as the inputs and
+    // nothing is built again on start. ClickHouse's are Memory tables. PostgreSQL's are UNLOGGED tables, which skip the
+    // write-ahead log, since its TEMP tables belong to one session. Only the inputs need to survive a crash there,
+    // since every calculated table can be built again from them. After a restart ClickHouse builds them again on
+    // start, and that time counts in R8. PostgreSQL keeps UNLOGGED tables over a clean restart, and empties them only
+    // in crash recovery.
     // In ClickHouse the table is created first with plain column types, then filled. A table created from the
     // SELECT would get Nullable columns from join_use_nulls, and Nullable columns are slower to read.
     // insert_null_as_default = 0 makes a NULL fail the insert instead of turning into 0.
@@ -67,7 +69,7 @@ export function translate(model, { versions, years, dialect = 'duckdb', dict })
             : PG
               ? `DROP TABLE IF EXISTS ${name};
 CREATE UNLOGGED TABLE ${name} WITH (autovacuum_enabled = false) AS`
-              : `CREATE OR REPLACE TEMP TABLE ${name} AS`;
+              : `CREATE OR REPLACE TABLE ${name} AS`;
     // PostgreSQL plans each statement from the statistics of the tables it reads, and a new table has none until
     // autovacuum gets to it. So each calculated table is analyzed as soon as it is built. At 10m a full rebuild
     // took 76 s without this, and 15 s with it.
@@ -624,7 +626,7 @@ CREATE UNLOGGED TABLE ${name} WITH (autovacuum_enabled = false) AS`
 
     const calc = model.order.filter((v) => v.formula && !v.measure);
     const tables = calc.map((v) => `-- ${v.name} = ${v.formula.replace(/\s+/g, ' ')}\n${translateVariable(v)}${analyze(rel(v))}`);
-    // What each calculated table reads, so a write can rebuild only the tables it reaches.
+    // What each calculated table reads, so an entry can rebuild only the tables a write reaches.
     const reads = new Map(calc.map((v) => [v.name, new Set([...nodesOf(parseFormula(v.formula))].filter((n) => n.type === 'name').map((n) => model.variable(n.name).name))]));
     const flats = flatSql();
     /**