Skip to content

feat(search): default to ripgrep scanning and parallel lexical/semantic query routing - #140

Merged
spelech merged 4 commits into
mainfrom
feat/ripgrep-default-and-parallel-query-router
Nov 13, 2026
Merged

spelech merged 4 commits into
mainfrom
feat/ripgrep-default-and-parallel-query-router

Conversation

@spelech

@spelech spelech commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • Default ripgrep (rg) engine across grep and indexing (ripgrep_runner.py, code_grep_scanner.py, grep_handlers.py, local_syncer.py, git_delta_scanner.py): Replaces Python os.walk + line-by-line re.search scans with streaming rg --json and rg --files, including PCRE2 fallback for lookarounds, broken symlink safety, and pure-Python fallback when rg is unavailable. Removes the duplicated 100-line extended os.walk loop from grep_handlers.py.
  • Word-token query classification & t=0 parallel Track A / Track B search (query_router.py, lexical_retriever.py, search.py):
    • Distinguishes exact identifier/path/quoted queries (lexical_exact), agent keyword-series queries such as "order processor retry payment" (lexical_keywords), and conceptual natural-language questions (semantic_prose) using word-token morphology and prose stopword/interrogative density.
    • Runs Track A (tokenized AST symbol lookup + rg multi-token cluster scan + file summaries) concurrently at t=0 alongside Track B (embedding + vector search).
    • Applies a word-token coverage confidence gate to return Track A immediately without waiting for slow embeddings on exact and high-confidence keyword-series queries, while awaiting and fusing Track A + Track B on conceptual/ambiguous queries.
  • Whole-function AST chunk preservation (state.py, processor.py, symbol_extractor.py): Introduces CODE_CHUNK_SIZE=4000 (up from 1500) so functions and methods stay intact in single AST chunks.
  • Windows portable rg.exe bundling (build_portable_windows.ps1, build-portable-windows.yml): Downloads and packages rg.exe alongside contextcortex.exe with CI smoke verification.

Verification

  • pytest -q: 678 passed, 16 skipped (including new tests in tests/test_ripgrep_and_query_router.py and style guardrails <= 500 LOC / semantic naming checks).

@spelech
spelech force-pushed the feat/ripgrep-default-and-parallel-query-router branch from 4b9dcda to 8591bb7 Compare November 13, 2026 17:36
Comment thread benchmarks/run_benchmark.py Fixed
@spelech
spelech merged commit 22bd3a5 into main Nov 13, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants