Gate the shape layer on the graph, and let Node's heap be raised

[ecrum19]

Two memory guards, each fixed after it failed on the same 170,935,101-triple graph during a scale experiment.

1. The shape-layer gate measured the wrong thing

pyshacl is in-memory, so the wrapper size-gates it — but on the packaged artifact's bytes, while what pyshacl pays for is the graph. One graph therefore landed on both sides of one 512 MiB gate purely by packaging:

artifact	bytes	vs 512 MiB	outcome
cottas	390,728,158	under	shapes attempted → OOM
nt.gz	756,594,166	over	skipped
hdt	1,182,206,289	over	skipped

The COTTAS run was SIGKILLed at 32.2 GB RSS on a 31 GB machine — after COTTAS decoding and rapper had both succeeded on all 170,935,101 triples. Nothing was wrong with the artifact or the engine.

The perverse part: the better a format compresses, the likelier it was to exhaust memory. COTTAS's headline advantage (0.52× the gzipped N-Triples) is precisely what defeated the guard. The existing comment says the gate exists because "silently trying would turn a safety net into an OOM" — which is what happened.

Fix: the authoritative gate moves into the runner, the only place that knows what pyshacl will actually load, and counts triples — which no packaging can change. --shacl-max-triples defaults to 50M: above the largest graph the published campaign validated with shapes (17.1M, the 100,000-record HG005 slice) and below the one that exhausted 31 GB (170.9M). A skip is recorded with its reason in shacl.json rather than being silent.

The wrapper's byte gate stays as a cheap pre-filter, now documented as non-authoritative.

2. Node chose a ceiling the machine did not

On the same graph the HDT endpoint died on the sample-level query:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Worker 53 died with SIGABRT. Starting new worker.

q01–q04 had passed (~36 min each); q05 returned IncompleteRead, and q06–q13 then failed instantly with Connection refused. The kernel OOM killer was never involved — ~25 GB was free. Node does not size its old-space from the host, and nothing in the image, runner or wrapper set NODE_OPTIONS.

Fix: --node-heap-mb sets --max-old-space-size for the three Comunica-backed engines (comunica, hdt, cottas — all via ComunicaHttpEndpointMixin). It appends to any existing NODE_OPTIONS rather than replacing it.

Compatibility

--node-heap-mb is unset by default, so every published measurement keeps the conditions it was produced under. The SHACL gate is the one behaviour change: a graph above 50M triples that previously attempted shapes now records a skip instead — which is the intended outcome, since attempting it is what caused the OOM.

A crash the tests found

shacl_exceeds_limit treats an unknown triple count as too large, which is the right call. But the skip reason then formatted that count with f"{None:,}", which raises TypeError. With shapes requested, two cases became EXECUTION_FAILED:

• rapper succeeds but prints no parsable count line.
• rapper is missing. That should have been BLOCKED_BY_PREFLIGHT, but the gate runs before the preflight check.

The skip is now recorded with a reason stating the count is unknown. The decision itself is unchanged.

Testing

test/test_shacl_gate_and_node_heap_unit.py, 53 tests. Every line and branch this PR adds is covered by this file alone. Each of 12 deliberate mutations, such as dropping env= from Popen, the wrapper dropping a falsy 0, or an exclusive boundary, fails at least one test.

The two worth reading:

• test_the_packaging_no_longer_decides: the property the bug violated. All three packagings of one graph must reach one verdict.
• test_the_old_byte_gate_did_disagree_across_packagings: not a test of current behaviour but a record of the defect, in the numbers that produced it, so nobody reinstates the byte gate.

What else is covered:

• The gate helper: the campaign's largest validated graph still passes. An unknown count is refused (skipping is recoverable, OOM is not). 0 disables the gate, the boundary is inclusive, and the default sits between the graph that worked and the graph that died.
• The gate inside run_validation: a graph over the limit never reaches pyshacl. The skip and its reason are written to shacl.json, announced on stderr, and match summary.json. A skip is not treated as a shape failure. A graph at or under the limit still gets its shapes checked. An unknown count is recorded as a skip, not a crash, and a missing rapper still reaches BLOCKED_BY_PREFLIGHT.
• Node heap: comunica, hdt and cottas all read the ceiling, and it reaches the spawned process's NODE_OPTIONS, appended to the caller's own options. Unset leaves the environment unchanged.
• Runner options: the defaults, integer parsing, and a non-integer value exits 2.
• Wrapper main(): --shacl-max-triples accepts 0 and rejects lots, 1.5 and -1 with exit 2. --node-heap-mb rejects 0, negatives and non-integers.
• Wrapper to runner: both flags cross the container boundary, including a limit of 0, which a truthiness check would drop. The runner accepts the flag names and value types the wrapper sends.
• Minimal namespace: run_validation accepts a hand-built namespace without either option, and falls back to the documented defaults.

Full unit suite: 994 tests, OK. test_validation_mutation_unit (116 tests) also passes.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 99.75309% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
test/test_shacl_gate_and_node_heap_unit.py	99.72%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!