Skip to content

fix(index): keep node line index sub-linear for single-line JSON documents - #622

Open
s04 wants to merge 1 commit into
pb33f:mainfrom
s04:fix/node-line-index-single-line-json
Open

fix(index): keep node line index sub-linear for single-line JSON documents#622
s04 wants to merge 1 commit into
pb33f:mainfrom
s04:fix/node-line-index-single-line-json

Conversation

@s04

@s04 s04 commented Sep 1, 2026

Copy link
Copy Markdown

Fixes #621

Problem

Since v0.38.0 (4a22282) the per-line node index stores entries in a slice that is scanned linearly on every insert and lookup. For YAML that is cheap, but a minified JSON document puts every node on line 1, so building the index is O(n²) in node count. An 8 MB single-line spec (~700k nodes) pins a core inside addNodeLineEntry for minutes; v0.37.3 builds it in ~1.5 s.

Change

index/map_index_nodes.go:

  • addNodeLineEntry is now append-only. No scan per insert.
  • New sortNodeLines runs once at the end of MapNodes: stable-sorts each line by column and collapses duplicate columns keeping the last write. mapNodesRecursive writes parents after children, so parents still win collisions, matching the previous behaviour and the original map semantics.
  • lookupNodeLines is a binary search over the sorted line.

The [][]nodeLineEntry type and all call sites (GetNode, GetNodeMap, FindNodeOrigin, the KeyNode lookup in extract_refs_lookup.go) are unchanged. One behavioural nuance: entries[0] on a line is now the leftmost node rather than the first one visited. For the KeyNode lookup that is the intended node, and in practice they coincide because DFS visits the leftmost child first.

Tests

  • TestSpecIndex_MapNodes_SingleLineDocument: minified document with ~16k nodes on one line; every node is retrievable by position and the legacy map agrees with the line index.
  • TestSpecIndex_SortNodeLines_OrdersAndDedupes: out-of-order inserts and duplicate columns.
  • BenchmarkSpecIndex_MapNodes_SingleLine: ~200k nodes on one line.
BenchmarkSpecIndex_MapNodes_SingleLine   before (origin/main): 101,835,604,666 ns/op
                                         after:                    301,491,542 ns/op
BenchmarkSpecIndex_MapNodes              unchanged (~378 µs/op)

Full go test ./... passes. The 8 MB real-world single-line spec builds its V3 model in ~0.7 s with this change.

…ments

Since v0.38.0 the per-line node index (map_index_nodes.go) stores entries in a
small slice that is scanned linearly on every insert and lookup. That is fast
for YAML, where a line holds a handful of nodes, but a minified JSON spec puts
the entire document on line 1. Building the index then costs O(n^2): an 8 MB
single-line spec with ~700k nodes (e.g. Cloudflare's published OpenAPI JSON)
pins a CPU core inside addNodeLineEntry for minutes. v0.37.3 builds the same
document in ~1.5s.

Fix: make inserts append-only, then sort each line by column exactly once when
MapNodes finishes and collapse duplicate columns keeping the last write, which
preserves the existing "parents win collisions" semantics. Lookups become a
binary search, so both build and GetNode stay sub-linear per line. The
[][]nodeLineEntry type and its call sites are unchanged; the first entry of a
line is now the leftmost node, which is what the KeyNode lookup in
extract_refs_lookup.go intends.

Adds a single-line document test, a sort/dedupe test, and a benchmark with
~200k nodes on one line that guards against the regression:

  BenchmarkSpecIndex_MapNodes_SingleLine  before: 101.8 s/op  after: 0.30 s/op

The 8 MB single-line spec now builds its model in ~0.7s.
@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.78%. Comparing base (ca6ca73) to head (d0be530).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #622   +/-   ##
=======================================
  Coverage   99.78%   99.78%           
=======================================
  Files         283      283           
  Lines       34456    34468   +12     
=======================================
+ Hits        34382    34394   +12     
  Misses         46       46           
  Partials       28       28           
Flag Coverage Δ
unittests 99.78% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v0.38: building a single-line (minified JSON) document is O(n²) in MapNodes

1 participant