Skip to content

pampa: record byte-for-byte provenance of Math.text (Math.text_source) - #705

Draft
cscheid wants to merge 1 commit into
mainfrom
feature/bd-ieldbghj-pampa-record-text-sourceinfo
Draft

cscheid wants to merge 1 commit into
mainfrom
feature/bd-ieldbghj-pampa-record-text-sourceinfo

Conversation

@cscheid

@cscheid cscheid commented Sep 21, 2026

Copy link
Copy Markdown
Member

Prerequisite for the quarto-math crate (bd-entbg6x3, plan claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md, Phase 0). Strand: bd-ieldbghj.

Opened as a draft on purpose: the branch is complete and verified, but we are holding the merge for a while (it will follow #704). Expected rebase conflicts against main once #704 lands are three regenerable artifacts only: Cargo.lock, crates/pampa/snapshots/json/math-with-attr.snap, ts-packages/annotated-qmd/examples/academic-paper.json.

Why

Math.source_info spans the whole $…$ / $$…$$ node, but Math.text is not a plain substring of it once a math node spans lines: the reader folds each soft break of inline math to a literal \n and strips the block-continuation gutter (> , list indentation) from the interior lines of display math. A consumer that parses text further (the coming quarto-math crate) needs to map any byte of it back to the .qmd.

What

  • quarto-pandoc-types: Math.text_source: Option<SourceInfo>, length equal to text.len(), map_offset(i) lands on the byte text[i] came from. None means no mapping finer than source_info is known.
  • pampa reader: location::provenance_builder_with_context (a ProvenanceBuilder rooted the way node_source_info_with_context roots node spans). Inline math: text segments verbatim, a bare \n soft break verbatim, any other soft break (gutter, CRLF) a replacement so the folded newline maps to the source line ending. Display math: first line and every \n verbatim, each stripped gutter a zero-length replacement. The strip behavior itself is unchanged (bd-q6ed / bd-qpa2 still own it).
  • JSON wire: new textS sidecar (bare pool ref) after t, emitted by both the Value and the streaming writer, read back by the reader; absent/null reads as None. Documented next to a/targetS in claude-notes/designs/raw-json-format.md.
  • Filters and transforms: Lua assignment to .text clears the mapping; pandoc.Math() builds with None; crossref_render keeps the mapping when appending \tag{N} by concatenating a zero-width synthesized piece; every other constructor passes None.
  • Strip lists: textS added to Rust remove_location_fields and the TS stripSourceInfoFields (+ unit test), pampaOracle, tiptap spike canonical/pampa; textS?: number on the TS Annotated_Inline_Math type.

Tests

  • crates/pampa/tests/integration/math_text_source.rs (15): single- and multi-line inline math, blockquote and list gutters, display math plain / quoted / nested-quoted / listed / labeled / inside strong, several nodes per paragraph, JSON round trip of textS, JSON without the sidecar reads as None, and a sweep over every in-tree .qmd with math (21 files, 36 nodes) asserting each text byte maps inside the node span to the identical source byte unless it is a folded newline.
  • quarto-core crossref_render: equation_tag_extends_text_source_instead_of_dropping_it.
  • test_location_health validates text_source alongside source_info.

Snapshot changes (reviewed)

  • crates/pampa/snapshots/json/math-with-attr.snap (1 file): each of the three Math nodes gains a textS ref and the pool gains the matching body ranges (e.g. 29..37 for E = mc^2 inside $E = mc^2$ at 28..38). Nothing else changed.
  • ts-packages/annotated-qmd/examples/academic-paper.json regenerated with the live pampa -t json binary (guard test); only textS refs and pool rows added.

Verification

cargo nextest run --workspace (14,032 passed) and full cargo xtask verify (all steps passed) on this branch.

🤖 Generated with Claude Code

…) (bd-ieldbghj)

`Math.source_info` spans the whole `$…$` / `$$…$$` node, but `text` is
not a plain substring of it once a math node spans lines: the reader
folds each soft break of inline math to a literal `\n` and strips the
block-continuation gutter (`> `, list indentation) from the interior
lines of display math. Consumers that parse `text` further (the coming
quarto-math crate, bd-entbg6x3) need to map any byte of it back to the
`.qmd`. This adds that mapping.

quarto-pandoc-types
- `Math.text_source: Option<SourceInfo>`: length == `text.len()`,
  `map_offset(i)` lands on the byte `text[i]` came from. `None` means no
  mapping finer than `source_info` is known.

pampa reader
- `location::provenance_builder_with_context`: a ProvenanceBuilder
  rooted like `node_source_info_with_context` (in_parent under an
  embedded re-parse, in_file otherwise).
- inline math: text segments verbatim; a soft break whose bytes are
  exactly `\n` verbatim; any other soft break (gutter, CRLF) a
  `replacement(range, 1)` so the folded newline maps to the source line
  ending.
- display math: first line and every `\n` verbatim; each stripped
  continuation gutter a zero-length replacement (deletion). Behavior of
  the strip itself is unchanged (bd-q6ed / bd-qpa2 still own it).

JSON wire
- new `textS` sidecar (bare pool ref) after `t`, emitted by both the
  Value and the streaming writer, read back by the reader; absent or
  null reads as `None`. Documented next to `a`/`targetS` in
  claude-notes/designs/raw-json-format.md.

Filters and transforms
- Lua: assigning `.text` clears the mapping; `pandoc.Math()` builds
  with `None`.
- crossref_render: appending `\tag{N}` keeps the mapping by
  concatenating a zero-width synthesized piece at the node end.
- every other constructor (tests, generators, equation label, math_js,
  JSON reader) passes `None`.

Tests
- crates/pampa/tests/integration/math_text_source.rs (15): single- and
  multi-line inline math, blockquote and list gutters, display math
  plain / quoted / nested-quoted / listed / labeled / inside strong,
  several nodes per paragraph, JSON round trip of `textS`, JSON without
  the sidecar reads as `None`, and a sweep over every in-tree `.qmd`
  with math (21 files, 36 nodes) asserting each text byte maps inside
  the node span to the identical source byte unless it is a folded
  newline.
- quarto-core crossref_render: `equation_tag_extends_text_source_
  instead_of_dropping_it`.
- test_location_health now validates `text_source` alongside
  `source_info`.
- `textS` added to every source-key strip list: Rust
  `remove_location_fields`; TS `stripSourceInfoFields` (+ unit test),
  `pampaOracle`, tiptap spike `canonical`/`pampa`; `textS?: number` on
  the TS `Annotated_Inline_Math` type.

Snapshots (1 file updated, reviewed)
- crates/pampa/snapshots/json/math-with-attr.snap: each of the three
  Math nodes gains a `textS` ref and the pool gains the matching body
  ranges (e.g. 29..37 for `E = mc^2` inside `$E = mc^2$` at 28..38).
  No other change.
- ts-packages/annotated-qmd/examples/academic-paper.json regenerated
  with the live writer (guard test); only `textS` refs + pool rows
  added.

Verified: cargo nextest run --workspace (14,032 passed) and full
cargo xtask verify (all steps passed).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@posit-snyk-bot

posit-snyk-bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants