Milestone 6: hardening — error fidelity, difftest, benchmarks, fuzzing

.github/workflows/difftest.yml

@@ -0,0 +1,47 @@
+name: difftest
+
+# The milestone-6 cron lane: differential fuzzing of darkwing against the
+# pinned DuckDB CLI, plus a native fuzz run of the public Parse API. The
+# nightly artifact only matches the pin until upstream moves it, so the
+# lane verifies the version first and skips (green) once it drifts —
+# advancing the pin is its own workflow (cmd/regenerate), not this one.
+
+on:
+  schedule:
+    - cron: "17 5 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  difftest:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+      - name: Download the nightly DuckDB CLI
+        run: |
+          curl -sSL -o duckdb.zip https://artifacts.duckdb.org/latest/duckdb-binaries-linux-amd64.zip
+          unzip -o -q duckdb.zip duckdb_cli-linux-amd64.zip
+          unzip -o -q duckdb_cli-linux-amd64.zip
+          ./duckdb --version
+      - name: Check the nightly against the pin
+        id: pin
+        run: |
+          pin=$(sed -n 's/.*PinnedVersion = "\(.*\)"/\1/p' internal/duckdbsrc/duckdbsrc.go)
+          version=$(./duckdb --version)
+          if echo "$version" | grep -qF "$pin"; then
+            echo "matches=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "matches=false" >> "$GITHUB_OUTPUT"
+            echo "nightly is $version, pin is $pin - skipping until the pin advances"
+          fi
+      - name: Differential fuzzing vs the pinned CLI
+        if: steps.pin.outputs.matches == 'true'
+        run: DARKWING_DUCKDB=$PWD/duckdb go run ./cmd/difftest -seed $(date +%s) -duration 20m
+      - name: Native fuzzing of Parse
+        run: go test ./parser -run '^$' -fuzz FuzzParse -fuzztime 10m

CLAUDE.md

@@ -77,6 +77,10 @@ Two corpus gates run under `go test ./parser`:
   pinned DuckDB binary parses it. Since milestone 3 the classification
   runs the full Parse pipeline, so transformer-raised Parser Errors count
   alongside the matcher's syntax errors.
+- **Error fidelity** (`TestCorpusErrorMessages`, since milestone 6):
+  every rejection renders the oracle's recorded first line verbatim.
+  Positions and messages the corpus can't pin are covered by unit tests
+  in `parser/errors_test.go`.
 - **Tree shape** (`TestSerializeGoldens`): `internal/serialize` output
   must match vendored `json_serialize_sql` goldens for a corpus sample
   (see `parser/testdata/serialize/README.md`).

README.md

@@ -20,28 +20,34 @@ loader, and matcher (with packrat memoization) — to Go.
 
 ## Status
 
-Milestone 2 (corpus + accept/reject conformance) is in place: the corpus
-under `parser/testdata/` is extracted from DuckDB's own test suite and
-classified by the pinned DuckDB CLI (`cmd/regenerate`); `go test ./parser`
-enforces *darkwing accepts iff pinned DuckDB parses*, with remaining
-disagreements tracked in todo metadata (`cmd/next-test`).
-
-Milestone 1 (engine) is complete:
-
-- `token/` — token kinds and DuckDB's keyword categories
-- `lexer/` — port of the parsing tokenizer
-- `internal/grammar/` — vendored grammar (pinned upstream commit recorded in
-  `internal/grammar/README.md`) and the grammar loader
-- `internal/matcher/` — matcher tree, packrat memoization, rule overrides,
-  furthest-failure error reporting
-- `cmd/debug-parse` — dump tokens or the raw parse tree for SQL on the
-  command line
-
-Next: the typed AST and transformer core (milestone 3).
+Milestones 1–6 are complete: the engine (tokenizer, grammar loader,
+matcher with packrat memoization), the corpus conformance gates, the full
+typed AST with a transformer for every statement, and the hardening pass.
+`go test ./parser` enforces, corpus-wide against the pinned DuckDB CLI:
+
+- **Accept/reject** — darkwing accepts a statement iff the pinned binary
+  parses it (`TestCorpus`), including transformer-raised parser errors.
+- **Error fidelity** — every rejection renders the oracle's error message
+  verbatim (`TestCorpusErrorMessages`), with positions pinned by unit
+  tests.
+- **Tree shape** — `internal/serialize` output matches vendored
+  `json_serialize_sql` goldens for the SELECT subset
+  (`TestSerializeGoldens`), with AST snapshots covering the rest.
+
+Hardening: `FuzzParse` holds the public API to its invariants on
+arbitrary input, benchmarks pin parse throughput and upstream's
+pathological-backtracking case (19 unmatched parens: milliseconds, thanks
+to packrat), and `cmd/difftest` mutates corpus seeds and diffs darkwing's
+verdicts against the pinned CLI (run nightly in CI while the nightly
+artifact matches the pin).
+
+Next: advancing the pin to the v2.0.0 tag when it lands (milestone 7),
+then sqlc integration.
 
 ```
 $ go run ./cmd/debug-parse 'SELECT 1'
 $ go run ./cmd/debug-parse -tokens 'SELECT * FROM t'
+$ go run ./cmd/debug-parse -ast 'FROM t SELECT x'
 ```
 
 ## License