Skip to content

Prototype: parse SQL with the ANTLR C++ runtime - #2605

Open
shangyian wants to merge 7 commits into
DataJunction:mainfrom
shangyian:native-antlr-parser-prototype
Open

shangyian wants to merge 7 commits into
DataJunction:mainfrom
shangyian:native-antlr-parser-prototype

Conversation

@shangyian

@shangyian shangyian commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Builds the C++ lexer and parser from the grammar, returns the parse tree as flat arrays, and rebuilds the Python context objects so the existing visitor runs unchanged.

The parse plus visit is about 1.8x faster with this change.

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify

netlify Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

👷 Deploy request for thriving-cassata-78ae72 pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 863c946

@shangyian
shangyian force-pushed the native-antlr-parser-prototype branch from 453d242 to 9ab2d8e Compare October 21, 2026 05:23
@shangyian
shangyian force-pushed the native-antlr-parser-prototype branch from 9ab2d8e to 87fc087 Compare November 14, 2026 08:36
@shangyian
shangyian marked this pull request as ready for review November 15, 2026 04:43
@greptile-apps

greptile-apps Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The latest change appears safe to merge and fixes the previous test finding.

Summary

Adds an opt-in ANTLR C++ parser and rebuilds Python parse trees for the existing visitor.

  • SQL parsing can use the C++ parser and keep the existing visitor.
  • Build tools generate and compile the native parser from DJ's SQL grammar.

Reviews (2) · Last reviewed commit: "Compare the native parser against the Py..." · Reviewed by Greptile

@shangyian
shangyian force-pushed the native-antlr-parser-prototype branch 2 times, most recently from 858051c to 6c14f39 Compare November 15, 2026 23:44
Builds the C++ lexer and parser from the grammar, returns the parse tree as
flat arrays, and rebuilds the Python context objects so the existing visitor
runs unchanged. The DJ AST matches the Python parser's on every query in the
test suite that parses, and parse plus visit is about 1.8x faster. Not wired
into DJ; build and test instructions are in native_parser/README.md.
parse_sql_with_sll_fallback tries the C++ parser first and falls back to the
Python parser if anything goes wrong, so errors stay the Python ones. Off by
default. Adds a workflow that builds the extension and runs the server suite
with it switched on.
The Python target appends an underscore to labels that are Python keywords.
The bridge set `from`, so the visitor saw no `ctx.from_` for INTERVAL ... TO ...
and the one server test that casts to an interval failed.
Annotate the token cache for mypy and apply the trailing-comma formatting.
So the two server runs can be compared on the same runners.
Tests inject a fake dj_native_parse that returns flat arrays built from a Python parse, so native.py and the antlr4.py hook run in the normal test shards.
python_ast now turns the native parser off, so the enabled CI job compares native output with Python output instead of with itself.
@shangyian
shangyian force-pushed the native-antlr-parser-prototype branch from 6c14f39 to 863c946 Compare November 16, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant