Skip to content

fix(python): suppress bare calls shadowed by an enclosing parameter - #1912

Merged
DeusData merged 5 commits into
mainfrom
feat/python-bare-local-binding
Sep 3, 2026
Merged

fix(python): suppress bare calls shadowed by an enclosing parameter#1912
DeusData merged 5 commits into
mainfrom
feat/python-bare-local-binding

Conversation

@DeusData

@DeusData DeusData commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Distilled from #1386 with Co-authored-by: credit to @Joseph-MingEn, who identified this defect class. The mechanism differs from theirs — see below.

The gap

After #1903 landed Python member-call suppression, this still resolves wrongly:

def _run_with_heavy_slot(run):
    return run()          # suffix_matches onto SatoriLive.run

#1903 structurally cannot catch it — there is no receiver, so its guard never fires. #1647's cross-language suffix guard only fires when caller and target languages differ, and both are Python here. So the class survives everything currently on main.

Keyed on local binding, not on a name list

#1386 keyed this on {get, run, execute}. That is a claim about spellings, not about whether the resolver knew anything — and it would age invisibly: nothing fails when the generic-name distribution shifts, the graph just quietly loses different edges. It is also open-ended (why not start, send, handle, process?) and corpus-dependent.

This keys on the fact that decides the question: the callee identifier is bound as a parameter of an enclosing function, so it shadows any project function and short-name resolution is fabricated by construction. No list, decidable outright, and it covers handler(), callback(), fn() and every Callable-parameter shape.

Parameters only, deliberately. A parameter is in scope for the entire body regardless of position, and Python forbids global on a parameter (SyntaxError: name is parameter and global), so no flow analysis is needed. Local assignments are flow- and binding-form-sensitive across for, with…as, except…as, :=, unpacking, nested def/class, import, plus global/nonlocal — a partial version would be incomplete invisibly, which is the failure mode the name list was rejected for. That wants a real scope analyser and its own evidence.

Enclosing scopes are covered to any depth — def outer(run): def inner(): return run(), a common decorator/callback shape, is caught — and depth costs nothing, for the reason set out below. The one knowingly over-flagged shape (global run in a nested function whose outer scope has a run parameter) is documented in the helper rather than left silent.

A new field rather than overloading is_method

callee_is_locally_bound is a distinct field on CBMCall. is_method means "member call with unresolved receiver" and is read by the pxc synthetic-carrier dedup key (pass_lsp_cross.c:801/:815); overloading it for bare calls would corrupt that key and make the field's documented contract untrue.

One drop-list, shared

The weak-strategy list (suffix_match / unique_name / field_type_hint / fuzzy) is now a single weak_short_name_strategy() used by both the member guard and this one. Two copies would have let the guards silently disagree about what "weak" means. weak_call_guards_share_one_drop_list pins that agreement across 15 strategies — and it fired correctly during revert-check.

Python-gated, wired at both pass_calls.c and pass_parallel.c with textually identical gates. ArkTS preserved in both member gates.

Evidence — three revert-check rounds, each isolating one claim

round break result
A disable the pass_parallel.c gate only exactly 1 failure — the ≥50-file test; the sequential test still passed. Proves the parallel test genuinely exercises pass_parallel.c rather than falling back to the sequential path
B under-suppress (flag + guard) exactly 5 failures, all this change's — extraction flag, both registry drops, both pipeline negatives
C over-suppress (guard ignores the binding) exactly 3 failures — the registry keep-assertion and both pipeline positive controls

Round C is the one that matters. It proves the positive controls are not vacuous: uses_free_function → compute_widget_total is a cross-file bare call with no import, so it resolves by a weak strategy this guard can drop — and an over-suppressing guard does drop it. #1386's positive asserted a same_module edge that no guard touches for any input, so it would have passed even if the change dropped every other Python edge. Both pipeline tests also carry ASSERT_GTE(CALLS, 1) anti-vacuity.

The extraction test pins all 7 parameter binding forms (bare, typed, default, keyword-only, *args, **kwargs, lambda, closure) plus 3 negatives (unbound, imported, nested def), each asserted to appear exactly once.

Verification

pipeline registry parallel extraction complexity lsp_resolution_probe805 passed, 0 failed. Built in a fresh worktree with no prior build/, so no stale-binary exposure. All 6 new tests confirmed by name in output, and RUN_TESTs verified inside SUITE(pipeline) rather than the pipeline_semantic_manifest_repro decoy that has swallowed tests before.

Cost is O(1) per bare call, plus O(params) once when a def or lambda scope opens — never corpus-coupled, so no complexity-guard interaction. CBMCall is not serialized, so no cache or index-format bump.

How the lookup got to O(1) — two corrections worth reading

The first cut answered "is this callee a parameter of an enclosing scope?" by ascending the tree per call, which is the obvious implementation and the wrong one.

Correction 1 — the ascent was a hang, not a slowdown. ts_node_parent() is not O(1): it restarts at the tree root and descends to find the parent, so each hop costs O(depth), a parent chain is O(depth²) per call, and since every level of f(f(f(...))) is itself a bare call, the file-wide cost is O(depth³). stack_overflow_b's 30,000-deep fixture went to rc=124 at 900s with zero tests completed, on all seven CI legs. A hop-count cap alone was measured and rejected: with a 64-hop cap the suite still timed out, because 64 × O(30,000) × 30,000 is ~5.7e10. When the cost is per hop, bounding the number of hops cannot fix it.

Correction 2 — the cheap ascent still needed a cap, and the cap failed open. Ascending with the unified walk's own cursor makes each hop genuinely O(1), which fixed the hang (900s → 46s). But the walk is still O(depth) per call, so it needed a 64-ancestor bound — and past that bound the guard stopped suppressing. Deep-but-ordinary code silently kept the fabricated edges this PR exists to remove, and no test failed when it did.

The fix is to stop recomputing. The unified walk binds a def's or lambda's parameters when it opens that scope and unwinds them when it closes, so the guard is a map lookup. This is the lesson CBMWalkScope already records one struct away: carry walk state, never recompute it per node.

  • CBMParamSlot is a name → active-count map, not a set: def outer(run): def inner(run): binds one name twice, and leaving the inner scope must not unbind the outer.
  • Bindings are pushed at the end of push_boundary_scopes, after every scope push for that node, so they land in the frame just pushed — covering function_definition (via push_function_scope) and lambda (via push_lexical_boundary) alike — and pop_expired_scopes unwinds to the frame's entry height.
  • Allocation failure sets py_param_tracking_failed and the lookup answers false thereafter: a lost suppression, never a lost edge.

The repo's other binding table cannot serve this, and it is worth recording why: CBMLexicalBinding is qsort-ed in cbm_finalize_lexical_usages after the walk and its active_end stays 0 until then, so its binary search is invalid from handle_calls, which runs mid-walk and before handle_usages.

With no cap there is nothing left to fail open, so the test that pinned the cap's contract is replaced by extract_python_bare_call_flag_is_depth_independent. It keeps the shallow and 200-ancestor cases — the deep one now asserts the call is flagged, so reintroducing a cap fails it — and adds an unwind case: handler as both a parameter and a module-level def, asserting the two shadowed calls are flagged and the sibling call after that scope closed is not. A binding outliving its frame would destroy that true edge, the one direction this guard must never fail in.

Re-verified on the final tree: extraction 327 passed, pipeline registry parallel complexity 407 passed, stack_overflow_b 6 passed in 38s (46s with the cursor ascent, 900s rc=124 before it) with lsp_python_deep_nesting_no_crash green, and make lint-ci (cppcheck + clang-format) clean.

DeusData and others added 3 commits August 29, 2026 17:59
A Python `foo()` whose callee identifier is bound as a parameter of an
enclosing scope cannot be the module-level `foo` -- the parameter shadows it
for the whole body -- so resolving the call to a project Function/Method by a
weak short-name strategy fabricates the edge by construction:

    def _run_with_heavy_slot(run):
        return run()          # bound an unrelated SatoriLive.run

The receiver-aware weak-member guard (#1276) cannot see this class at all: a
bare call has no receiver, so is_method is false and the guard never fires.
This is the bare-call counterpart of python_receiver_is_exempt.

Keyed on the SCOPE FACT, not on the callee's spelling. A list of
generic-looking names (get / run / execute) asserts that certain spellings are
usually noise, which is a claim about corpus fashion rather than about what the
resolver knew, and it ages invisibly: nothing fails when the distribution
shifts, the graph just quietly loses different edges. A parameter binding is
decidable from this file's AST outright.

Parameters only, deliberately. A parameter is in scope for the entire body
regardless of position and Python forbids `global` on one, so no flow analysis
is needed. Local assignments are flow- and binding-form-sensitive (`for`,
`with as`, `except as`, `:=`, unpacking, plus global/nonlocal overrides); a
partial body scan would suppress the wrong edges invisibly -- the same failure
mode that rules out the name list. Enclosing scopes are walked to the file root
so a closure over an outer parameter counts.

Wired at both pass_calls.c and pass_parallel.c with an identical language gate:
a guard on one resolver only diverges the sequential and parallel paths. The
weak-strategy drop-list is now a single shared predicate used by both the member
guard and this one, so they cannot disagree about what "weak" means; a unit test
pins that agreement.

Tests pin both directions. The pipeline positive control is a cross-file bare
call with NO import, so it resolves by a weak strategy this guard could have
killed -- asserting a same_module edge would prove nothing, since no guard
touches same_module for any input. Verified by breaking the guard in both
directions: under-suppressing fails the negatives and the extraction flag;
over-suppressing fails the positives, so they are not vacuous. Disabling only
the pass_parallel.c gate fails only the >=50-file test and nothing else.

805 passed / 0 failed across pipeline registry parallel extraction complexity
lsp_resolution_probe.

The defect class was identified by Joseph-MingEn in #1386, which proposed a
name-keyed shape; the diagnosis is theirs.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Joseph-MingEn <125283161+Joseph-MingEn@users.noreply.github.com>
Changed-range clang-format (Homebrew LLVM 22.1.8, the build CI's lint-ci
uses) flagged two alignment violations in the tests added by the previous
commit. Whitespace only; 805 passed / 0 failed re-run after the reformat.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#1912 hung stack_overflow_b on all seven CI legs: rc=124 at 900s with zero
tests completed, on lsp_python_deep_nesting_no_crash.

The bare-call scope walk used ts_node_parent(), which is NOT O(1) -- it
restarts at the tree root and descends to find the parent (vendored
ts_runtime/src/node.c). Each hop therefore costs O(depth), a parent chain is
O(depth^2) per call, and since every level of f(f(f(...))) is itself a bare
call, O(depth^3) across the file. The fixture nests 30,000 deep, so this is a
hang rather than a slowdown.

A hop-count cap alone is NOT sufficient, and this was measured rather than
assumed: with a 64-hop cap the suite still timed out, because 64 x O(30,000)
x 30,000 calls is still ~5.7e10. When the cost is per hop, bounding the
number of hops cannot fix it -- the walk itself has to be cheap.

Use the unified walk's own cursor. WalkState.current_cursor is already parked
on the current node and ts_tree_cursor_goto_parent() IS O(1), because the
cursor carries its path stack, so ascending costs nothing per hop. Same
current_cursor idiom, including the ts_node_eq identity guard before trusting
the shared cursor, as usage_current_field_name in extract_usages.c. This is
the same lesson CBMWalkScope already records: carry walk state, never
recompute it per node.

The 64-hop cap is retained and now does real work, bounding the remaining
O(depth) per call. Precedent for the shape: CBM_LSP_PERL_MAX_WALK_DEPTH and
LEAN_MAX_PARENT_DEPTH in this file. Both the cap and a missing or mismatched
cursor FAIL OPEN -- they can only ever cost a suppression, never a true edge,
which is the safe direction for a guard whose justification is precision.

lsp_python_deep_nesting_no_crash: 900s timeout, 0 completions -> PASS.
stack_overflow_b overall: 6 passed in 46s.

The regression test pins the cap's contract deterministically rather than by
wall clock: within the cap a shadowed callee is flagged; past 64 ancestors the
guard fails open and leaves it unflagged. Raising the cap flips the deep case
and fails the test, verified.

826 passed / 0 failed across stack_overflow_a stack_overflow_b stack_overflow_c
pipeline registry parallel extraction complexity lsp_resolution_probe.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: Joseph-MingEn <125283161+Joseph-MingEn@users.noreply.github.com>
@DeusData DeusData added bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Sep 1, 2026
Replaces the cursor ascent from 0b0d143. That commit fixed a real hang --
ts_node_parent() is O(depth), so a parent chain is O(depth^2) per call and
O(depth^3) across a file where every level of f(f(f(...))) is itself a bare
call, which hung stack_overflow_b's 30,000-deep fixture at 900s. But it fixed
it by making each hop cheap rather than by removing the per-call walk, so the
cost stayed O(depth) per call and needed a 64-ancestor cap to stay bounded.

That cap FAILS OPEN. Past 64 ancestors the guard stopped suppressing, so
deep-but-ordinary code silently kept the fabricated edges the guard exists to
remove -- and nothing failed when it did.

Carry the answer instead. The unified walk binds a def's or lambda's parameters
when it opens that scope and unwinds them when it closes, so the guard is an
O(1) map lookup. This is the lesson CBMWalkScope already records one struct
away: carry walk state, never recompute it per node.

  - CBMParamSlot is a name -> active-count map, not a set. `def outer(run):
    def inner(run):` binds one name twice and leaving the inner scope must not
    unbind the outer.
  - Bindings are pushed at the end of push_boundary_scopes, after every scope
    push for that node, so they land in the frame just pushed -- covering
    function_definition (push_function_scope) and lambda (push_lexical_boundary)
    alike -- and pop_expired_scopes unwinds to the frame's entry height.
  - Allocation failure sets py_param_tracking_failed and the lookup answers
    false forever after: a lost suppression, never a lost edge.

The repo's other binding table cannot serve this. CBMLexicalBinding is qsort-ed
in cbm_finalize_lexical_usages AFTER the walk and its active_end stays 0 until
then, so its binary search is invalid from handle_calls, which runs mid-walk
and before handle_usages.

extract_python_bare_call_scope_walk_is_bounded pinned the cap's fail-open
contract, which no longer exists, and is replaced by
extract_python_bare_call_flag_is_depth_independent. It keeps the shallow and
200-ancestor cases -- the deep one now asserts the call IS flagged, so
reintroducing a cap fails it -- and adds an unwind case: `handler` as both a
parameter and a module-level def, asserting the two shadowed calls are flagged
and the sibling call after that scope closed is NOT. A binding outliving its
frame would destroy that true edge, the one direction this guard must never
fail in. It passed on the first run.

extraction 327 passed; pipeline registry parallel complexity 407 passed;
stack_overflow_b 6 passed in 38s (46s with the cursor ascent, 900s rc=124
before it), lsp_python_deep_nesting_no_crash PASS.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit cc3e263 into main Sep 3, 2026
64 of 66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working parsing/quality Graph extraction bugs, false positives, missing edges priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant