fix(index): keep node line index sub-linear for single-line JSON documents - #622
Open
s04 wants to merge 1 commit into
Open
fix(index): keep node line index sub-linear for single-line JSON documents#622s04 wants to merge 1 commit into
s04 wants to merge 1 commit into
Conversation
…ments Since v0.38.0 the per-line node index (map_index_nodes.go) stores entries in a small slice that is scanned linearly on every insert and lookup. That is fast for YAML, where a line holds a handful of nodes, but a minified JSON spec puts the entire document on line 1. Building the index then costs O(n^2): an 8 MB single-line spec with ~700k nodes (e.g. Cloudflare's published OpenAPI JSON) pins a CPU core inside addNodeLineEntry for minutes. v0.37.3 builds the same document in ~1.5s. Fix: make inserts append-only, then sort each line by column exactly once when MapNodes finishes and collapse duplicate columns keeping the last write, which preserves the existing "parents win collisions" semantics. Lookups become a binary search, so both build and GetNode stay sub-linear per line. The [][]nodeLineEntry type and its call sites are unchanged; the first entry of a line is now the leftmost node, which is what the KeyNode lookup in extract_refs_lookup.go intends. Adds a single-line document test, a sort/dedupe test, and a benchmark with ~200k nodes on one line that guards against the regression: BenchmarkSpecIndex_MapNodes_SingleLine before: 101.8 s/op after: 0.30 s/op The 8 MB single-line spec now builds its model in ~0.7s.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #622 +/- ##
=======================================
Coverage 99.78% 99.78%
=======================================
Files 283 283
Lines 34456 34468 +12
=======================================
+ Hits 34382 34394 +12
Misses 46 46
Partials 28 28
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #621
Problem
Since v0.38.0 (4a22282) the per-line node index stores entries in a slice that is scanned linearly on every insert and lookup. For YAML that is cheap, but a minified JSON document puts every node on line 1, so building the index is O(n²) in node count. An 8 MB single-line spec (~700k nodes) pins a core inside
addNodeLineEntryfor minutes; v0.37.3 builds it in ~1.5 s.Change
index/map_index_nodes.go:addNodeLineEntryis now append-only. No scan per insert.sortNodeLinesruns once at the end ofMapNodes: stable-sorts each line by column and collapses duplicate columns keeping the last write.mapNodesRecursivewrites parents after children, so parents still win collisions, matching the previous behaviour and the original map semantics.lookupNodeLinesis a binary search over the sorted line.The
[][]nodeLineEntrytype and all call sites (GetNode,GetNodeMap,FindNodeOrigin, theKeyNodelookup inextract_refs_lookup.go) are unchanged. One behavioural nuance:entries[0]on a line is now the leftmost node rather than the first one visited. For theKeyNodelookup that is the intended node, and in practice they coincide because DFS visits the leftmost child first.Tests
TestSpecIndex_MapNodes_SingleLineDocument: minified document with ~16k nodes on one line; every node is retrievable by position and the legacy map agrees with the line index.TestSpecIndex_SortNodeLines_OrdersAndDedupes: out-of-order inserts and duplicate columns.BenchmarkSpecIndex_MapNodes_SingleLine: ~200k nodes on one line.Full
go test ./...passes. The 8 MB real-world single-line spec builds its V3 model in ~0.7 s with this change.