Repository navigation
Conversation
Terminate multi-DDict lookup at NULL entries, not dictionary ID zero. Cover missing and registered IDs that collide with a raw-content dictionary, using distinct dictionary contents.
Compare scaled occupancy directly so the table grows at 75% instead of 25%. Count allocations and verify buffered-streaming lookup after rehashing, independently of the one-shot decoder changes.
Drop the borrowed-reference table on empty dictionary loads and when disabling multi-DDict mode. Disabling a populated set also clears its selected dictionary, so callers can release every DDict. Reuse cleanup for reset and refDDict(NULL), document the lifetime rule, and test immediate release.
Loading any dictionary or prefix discards the borrowed DDict set without changing the opt-in parameter. refDDict already clears the local dictionary or prefix before registering a borrowed DDict. Document this invariant and test replacement in both directions, including immediate release of borrowed storage.
Select dictionaries only after a complete ordinary frame header is available, before installing dictionary-backed decoding state. Return dictionary_wrong instead of falling back to the current dictionary when the set has no exact nonzero ID match. Frames with ID 0 use a registered ID-0 DDict if present, or no dictionary otherwise. Document the compatibility change and test missing IDs, dictionary-free concatenated frames, raw-content ID-0 dictionaries, and both streaming paths.
Keep the single-DDict state empty while a DDict set exists. Resolve each ordinary frame into a local DDict pointer before installing decoder state, using no dictionary when ID 0 has no registered entry, and leave explicit-dictionary APIs independent of the set. Preserve trace metadata without retaining a borrowed DDict. Reject implicit set-based legacy decoding, retain explicit legacy decoding, and cover immediate release after disabling a set, raw ID 0, dictionary-free frames, and unchanged local-prefix behavior.
terrelln
force-pushed
the
2026-11-11-dict-set-v2
branch
from
November 11, 2026 22:14
82e0df9 to
37961cd
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes dictionary lifetime and selection behavior in the experimental ZSTD_d_refMultipleDDicts API. A populated DDict set now acts as an explicit mapping from frame dictionary IDs to borrowed DDicts, with no implicit default or remembered selection. The separate hash-table lookup and growth fixes are part of PR #4927.
The four commits are:
8a1bf5f — Clear remaining multi-DDict references.
Centralizes set cleanup and releases the set when loading an empty dictionary or disabling multi-DDict mode. This makes the documented “return to no-dictionary mode” behavior reliable and lets callers immediately free every formerly registered DDict.
201aae6 — Make local dictionaries and DDict sets mutually exclusive.
Loading a dictionary or referencing a prefix discards the set; registering a DDict discards any local dictionary or prefix. This gives dictionary replacement an unambiguous lifetime and prevents two competing sources of dictionary state from coexisting.
845b46b — Require exact matches for nonzero frame IDs.
Selects dictionaries only after a complete, valid ordinary-frame header is available, before installing decoding state. Missing nonzero IDs return ZSTD_error_dictionary_wrong instead of falling back to the previous dictionary. ID-less frames use a registered ID-0 DDict when present, or no dictionary otherwise.
82e0df9 — Resolve set entries independently for each frame.
Keeps the single-dictionary state empty while a set exists and resolves each frame into a local DDict pointer. Explicit-dictionary APIs remain independent of the set, tracing no longer needs a retained DDict pointer, and implicit set-based legacy decoding is rejected.
The default DDict was an extremely confusing concept, as it ended up being either the last DDict loaded, or the last DDict used regardless of load order. Also explicitly disables legacy frame decoding when there is a non-empty DDict set. The prior behavior was 100% not correct, so this isn't a big loss.
Design rationale
Testing
Adds regression coverage for immediate dictionary release, replacement in both directions, missing nonzero IDs, registered and absent ID-0 entries, dictionary-free frames mixed with dictionary-backed frames, concatenated and magicless input, fragmented streaming and its one-shot shortcut, explicit-dictionary APIs, prefix compatibility, and the legacy-format boundary. An ID-less frame that requires dictionary bytes also verifies that the fallback does not reuse a previous dictionary.
Passed: