Skip to content

Fix multi-DDict hash-table lookup and load factor - #4927

Open
terrelln wants to merge 2 commits into
meta:devfrom
terrelln:2026-11-11-ddict-hash-fixes
Open

terrelln wants to merge 2 commits into
meta:devfrom
terrelln:2026-11-11-ddict-hash-fixes

Conversation

@terrelln

@terrelln terrelln commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Fixes two issues in the experimental, opt-in ZSTD_d_refMultipleDDicts mode:

  • Distinguish dictionary ID 0 from an empty hash-table slot. Raw-content dictionaries legitimately have ID 0. Lookup now stops at a NULL entry and returns only an exact ID match, so a raw dictionary cannot incorrectly match—or hide—a different dictionary with a colliding hash.
  • Correct the load-factor calculation. The table now grows at 75% occupancy rather than approximately 25%, avoiding premature allocations and rehashing.

Additionally, one temporary fix so that dictionaries are only selected for a valid zstd frame header, so that the added regression tests pass. A more comprehensive fix will be stacked on these changes.

Adds regression tests in tests/fuzzer.c using dictionaries with distinct contents and checking decoded bytes. The tests cover missing and registered IDs that collide with ID 0, successful raw-dictionary lookup, the allocation threshold, and lookup after rehashing.

The test case infrastructure will be reused for a stacked PR that changes multi-DDict behavior more.

Terminate multi-DDict lookup at NULL entries, not dictionary ID zero. Cover missing and registered IDs that collide with a raw-content dictionary, using distinct dictionary contents.
Compare scaled occupancy directly so the table grows at 75% instead of 25%. Count allocations and verify buffered-streaming lookup after rehashing, independently of the one-shot decoder changes.
@terrelln
terrelln force-pushed the 2026-11-11-ddict-hash-fixes branch from b2560a0 to eac75fd Compare November 12, 2026 20:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant