[psiha]

DuckDB has no incremental calculation. The benchmark was working around that: it kept DuckDB's calculated tables as in-memory TEMP tables and rebuilt only the tables a write reaches. That is help the system does not have, so this PR removes it and shows the gap instead.

What changes

• The DuckDB entry keeps calculated tables as ordinary, durable tables in the database file. A write, a load of new data or a model change recalculates every calculated table (sql/calculate.sql). The partial-rebuild SQL (sql/rebuild/) is no longer generated for this entry, and the M1/M4 migrations build every table. Loads, reads, integer member ids and the other settings are unchanged.
• A calculated flag. A one-row table, calc_state, says whether the calculated tables are current. A write commits its input change together with calculated = false, calculates every table outside any transaction (DuckDB commits each statement by itself), then commits calculated = true. An open that finds false, because the process died in a recalculation, calculates every table again before the first read. After a crash the first open pays one full recalculation; R8 normally builds nothing. A reader opens the file only after the write has returned, so it never reads a half-built state as valid.
• Results get an optional per-test field, "recalc": "full", set by a test file's recalc export and copied by the runner. Only the DuckDB entry sets it, on L2, L3, W1 to W8 and M1 to M4. tests/method.md defines it.
• The website draws those bars and cells grey with diagonal stripes, with the text "not supported - full recalc results". The numbers stay in the scores as measured.
• New DuckDB results at 10m and 100m, and the docs, including a Caveats section (below).

The mechanism is generic. The ClickHouse and PostgreSQL entries are not changed.

Caveats of DuckDB (also in the entry README, results/README.md and the website notice)

• Recalculating the derived tables inside one transaction costs far more than outside it. Measured on an n2d-standard-4 (4 vCPU), on a copy of the 10m snapshot: about 271 s per recalculation inside BEGIN ... COMMIT (463 s with threads = 1) against about 4 s with each statement committing by itself. On a 24-core machine it took 17 to 28 s. CPU-bound, no I/O wait. The cause is not established.
• DuckDB commits each statement by itself and has no materialized views or triggers, so there is no way to refresh derived tables atomically with the input write without paying that. The choice is atomic and slow, or fast and not atomic (a crash between the input commit and the end of the recalculation leaves stale derived tables). The entry takes the second, with the flag and the recalculation on open.

Results (n2d-standard-4, local NVMe SSD, one run each, all 29 tests passed)

Run times: 10m 11.4 min (before 4.3 min), 100m 56.7 min (before 21.0 min). Peak resident memory of the run: 0.69 GB at 10m, 2.2 GB at 100m. The "before" numbers are the earlier setup (calculated tables in memory, only affected tables rebuilt).

Test	10m before (s)	10m after (s)	100m before (s)	100m after (s)
L1	4.4	5.72	21	27.4
L2 (full recalc)	2.76	4.04	12.1	19
L3 (full recalc)	0.333	3.66	1.81	17.3
W1 (full recalc)	1.27	3.65	7.03	17.4
W2 (full recalc)	1.33	3.69	7.27	17.5
W3 (full recalc)	0.409	3.67	2.17	17.5
W4 (full recalc)	1.01	3.63	5.3	17.3
W5 (full recalc)	0.341	3.63	1.84	17.5
W6a (full recalc)	0.263	3.66	1.14	17.6
W6b (full recalc)	0.264	3.66	1.15	17.5
W7 (full recalc)	1.39	3.65	7.03	17.4
W8 (full recalc)	0.394	3.65	2.04	17.4
R1	0.00228	0.00283	0.00324	0.00481
R2a	0.00259	0.00309	0.00378	0.00515
R2b	0.00244	0.00252	0.00311	0.00453
R2c	0.0023	0.00311	0.00314	0.00465
R3	0.00496	0.00762	0.00754	0.0112
R4	0.0357	0.0334	0.0533	0.0448
R5	0.0272	0.03	0.0441	0.0546
R6	0.00361	0.0051	0.0114	0.0158
R7	0.00222	0.00261	0.00547	0.00858
R8	2.32	0.0706	10.1	0.0805
M1 (full recalc)	0.272	3.59	1.46	17.5
M2 (full recalc)	2.27	3.62	10.1	17.5
M3 (full recalc)	2.32	3.62	10.3	17.5
M4 (full recalc)	0.224	3.75	0.916	18.2

Checks

• generate.mjs --check (all generators) and website/build.mjs --check pass.
• Crash test: a write was killed with SIGKILL 2.5 s into its recalculation. A raw open then showed calculated = false. The next start printed that the last recalculation did not finish, calculated every table again (7.3 s at 10m) and set true. The proving read, the total EBIT, matched the same write run without a kill to the last digit of a double (relative difference 1.7e-16, summation order).
• Rendered in light and dark; the DuckDB bars and cells are grey and striped with real data.

Draft, open points

• 1b is not re-measured. The DuckDB results at 1b are from the earlier setup, carry no mark, and the site and results/README.md say so. A 1b run is still to do.
• ClickHouse and PostgreSQL keep their partial rebuild. Whether to mark them or run them with a full recalculation is left to the owner.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request due to rate limiting. Please try again later. See Usage limits for GitHub Copilot for details.