Conversation
…q9k90z2) Research summary and phased plan for a pandoc-free docx writer: python-docx (MIT) as the OOXML reference, redoc-style .qmd/source-map sidecars, and a quarto-math crate using mitex's Apache-2.0 LaTeX parser as the front end with OMML and Typst writers and quarto-source-map annotations. Also preserves the two throwaway probes (python-docx element-model extractor, mitex parse-tree/span probe) under claude-notes/research/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…thML decisions (bd-entbg6x3) Keeps pandoc's style names for reference-doc compatibility, recommends vendoring mitex-lexer/parser (upstream velocity near zero), files the MathML writer as bd-9z83tcv0, and explains why reference docs don't widen the pullback's blast radius. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…mitex crates, math subsystem 20, API boundary, python-docx schemas) (bd-entbg6x3) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Phase 0 of claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md. - crates/quarto-math: workspace member with no dependencies yet; lib.rs carries the crate contract (mitex front end, q2-owned spec, MathAst, OMML + Typst writers, SourceInfo on every node, no dependency on quarto-pandoc-types). - tests/fixtures/<group>/<name>[.display].tex: one construct per file, 14 groups (basic, fractions, radicals, scripts, operators, delimiters, text-and-styles, accents, environments, macros, spacing, symbols, corpus, errors). corpus/ is 26 expressions copied verbatim from docs/, the Q1 test corpus and quarto-web; errors/ are inputs that must diagnose, never panic. - tests/integration/fixture_corpus.rs: shared loader (load_all) that every later snapshot test iterates, plus a guard test for the conventions (kebab-case, unique ids, no CRLF, one trailing newline, all groups present). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…entbg6x3-quarto-math Math.text_source prerequisite for quarto-math (Phase 0 of claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md). The strand branch is based on origin/main and will go up as its own PR; merging it here lets Phase 1 build on it in the meantime. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…merged) (bd-entbg6x3) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…entbg6x3)
Phase 0 of claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md
(decisions 2 and 6).
vendor/ — mitex-glob, mitex-spec, mitex-lexer, mitex-parser copied from
mitex-rs/mitex 985d8e7 (Apache-2.0, LICENSE next to each crate). Local
patches, all listed in vendor/VENDORED.md:
1. manifests rewritten (explicit version/edition 2021/publish=false,
concrete dependency versions, no [lints], no benches);
2. rkyv removed from mitex-spec (features, derives, to/from_bytes,
stream.rs); serde stays;
3. tests consolidated into tests/integration/ per crate;
4. DEFAULT_SPEC for tests loaded from the JSON dump instead of
mitex-spec-gen.
No source change in src/ beyond (2). Builds clean under clippy
-D warnings; the 52 upstream lexer/parser tests pass.
spec/upstream/mitex-default-spec.json — mitex's prebuilt command spec
(default.rkyv, artifacts 9eb762a) dumped to sorted JSON in
CommandSpecRepr shape: 995 entries (968 commands, 27 environments). It
is the generation input for the q2-owned spec and the test spec of the
vendored crates; README.md explains provenance and regeneration. The
dump tool lives with the other probes
(claude-notes/research/2026-09-21-quarto-math-probes/spec-dump/) and
reproduces the committed file byte for byte.
New third-party dependencies: rowan 0.15, logos 0.14, ena 0.14, ecow
0.2 (plus their small transitive set).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…est helper (bd-entbg6x3) Closes Phase 0 of claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md. tests/schemas/ooxml/: the transitive import closure of shared-math.xsd (OMML) from python-docx ref/xsd at v1.2.0 (e454546, MIT; ECMA-376 transitional namespaces), 12 files + W3C xml.xsd, ~500 KB. The plan assumed shared-math.xsd stood alone; it imports wml.xsd and shared-commonSimpleTypes.xsd, so the whole WordprocessingML closure is vendored once and Phase 2 validates document.xml from the same place. One local patch, in the two files that import the xml: namespace: the import gained schemaLocation="xml.xsd" (libxml2 cannot compile the set without it; pandoc's validator carries the same patch). README.md records provenance, the patch and the verification. tests/integration/omml_schema.rs: validate_omml / validate_against / assert_valid_omml run `xmllint --noout --schema` over stdin and return Valid / Invalid(stderr) / Skipped(reason); Skipped when xmllint is not on PATH (Windows CI) so the suite stays green there while Unix legs keep the gate. Guard tests: a correct fragment (frac, sSup, nary with limits) validates; den-before-num is rejected naming the expected elements; an unknown element is rejected; a minimal w:document with m:oMath validates against wml.xsd; Skipped carries a reason. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…sion budget (bd-entbg6x3)
Phase 1, reader item, of
claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md.
quarto-math
- spec::Spec wraps mitex's CommandSpec; Spec::builtin() embeds
spec/upstream/mitex-default-spec.json (until commands.json exists)
and parses it once per process; Spec::from_json for custom specs.
- reader::parse(text, &Spec) -> Parsed { root, spans, leaves }: mitex's
lossless CST plus a side table giving every leaf token's byte range in
the original input. Macro bodies map to their definition site, macro
arguments to the use site.
- tests/integration/reader.rs (10 tests): explicit spans on a simple
expression, the probe's two macro cases plus a macro used twice, the
single-character argument split (`\frac12`), multi-byte input, empty
input, and two corpus-wide properties over all 263 fixtures — every
leaf span selects the leaf's own bytes, and for macro-free input the
spans are monotone with gaps only where mitex drops bytes.
Vendored mitex patches (vendor/VENDORED.md #5, #6)
- #5 mitex-parser: Parser records a span at each of its five
builder.token sites by slice arithmetic against the input; the two
sites that emitted fresh one-character Strings now emit subslices.
New Parser::parse_with_spans and mitex_parser::parse_with_spans;
parse() unchanged.
- #6 mitex-lexer: MacroEngine charges an expansion budget (200 000
tokens) and, once exhausted, emits (Token::Error, "macro expansion
limit exceeded") instead of expanding. Upstream loops forever on
`\newcommand{\loop}{\loop} \loop`; the errors/recursive-macro fixture
now parses in milliseconds with an error leaf.
Findings recorded in the plan: mitex's tree is not byte-lossless (it
drops whitespace before bare arguments and the `\begin{…}`/`\end{…}`
syntax around an environment name), and the arity-mismatch error token
is the one leaf whose text is synthesized (no span). The tests pin both.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n, and OMML writer conventions (bd-entbg6x3) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… trial-merge findings against the pandoc-hybrid leg (bd-entbg6x3) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ask gen-math-spec (bd-entbg6x3)
Phase 1, spec item, of
claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md.
spec/commands.json (1013 rows) is generated, never hand-edited:
`cargo xtask gen-math-spec` runs quarto_math::spec::generator over the
mitex spec dump (spec/upstream/) plus the hand-written
spec/overrides.json (485 rows). Each row is
{ parse: mitex CommandSpecItem, sem: Semantics, typst: alias }, so the
vendored parser sees mitex's argument shapes unchanged while the
writers get q2's semantics. Zero-argument commands resolve their Typst
alias to Unicode through codex (`lt.eq` -> `≤`); literals without ASCII
letters are used as-is; overrides supply every structural kind (frac,
sqrt, nary, func, accent, bar, group-chr, lim-pos, x-arrow, style,
text, space, phantom, cancel, not, color, scripts, env, ignore) and the
symbols codex 0.3 has renamed since mitex aliased them (sect, the
*.circle family, planck.reduce, …). Whatever is left is `unsupported`
with a reason, visible in the JSON.
- spec::semantics — the Semantics enum (serde, kebab-case tags) with
LimLoc / FracStyle / Pos / Variant / ScriptsOp / EnvLayout.
- spec::Spec now loads commands.json: command_spec() for the parser,
semantics(name) / typst_alias(name) / row(name) for the writers.
Spec::builtin() embeds the committed file.
- spec::generator (feature `spec-gen`, pulls codex): generate(upstream,
overrides) -> commands.json text; sorted and pretty-printed so a
no-op regeneration changes no bytes.
- xtask gen-math-spec [--check]: writes the file or fails when stale.
xtask depends on quarto-math with spec-gen; the crate's own
integration tests enable the same feature via a self dev-dependency
so one implementation serves both.
Tests (tests/integration/spec.rs): committed file equals a
regeneration; every upstream name has a row; symbol texts are
non-empty and short; the builtin spec is the committed file; an insta
inventory of the 87 unsupported rows by reason (34 document-level, 51
macro/document declarations, 2 without semantics); and the acceptance
bar for the overrides — every command the 263-fixture corpus uses,
outside errors/ and fixture-local macros, has writer semantics.
Also: `\R \N \Z \Q \C` moved from symbols/ to errors/
(undefined-blackboard-shortcuts) — they are not standard commands.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Phase 1, normalization item, of
claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md.
ast.rs — Node { kind, span } with the writer-facing NodeKind vocabulary
from the plan (Row, Run, Sym, Space, Frac, Sqrt, Scripts, Nary, Func,
Delimited, Accent, Bar, GroupChr, LimPos, XArrow, Style, Text, Phantom,
Cancel, Color, Matrix, Break, Error), children()/walk()/has_errors(),
and a compact Display used by the snapshots.
normalize.rs — normalize(text, Mode, &Spec) -> Normalized { mode, root,
problems }. Rules, each pinned by a test:
- attachments fold into one Scripts; a repeated ^/_ is an Error that
keeps the first; \limits/\nolimits set LimLoc on the operator they
bind to (Auto otherwise, resolved by the writers);
- big operators carry no body (TeX gives a summand no scope);
- \left…\right -> Delimited, `.` -> None, \middle splits parts;
a missing side is an Error;
- environments split at `\\` (a *token* of kind ItemNewLine in mitex)
and `&`; trailing `\\` adds no row; ragged rows are padded and
reported; array's column spec is an argument, not a cell; \hline is
dropped with a note; an environment closed by the wrong \end or not
closed at all is kept verbatim as an Error;
- commands resolve through the spec's semantics; unknown commands and
environments, missing arguments, unclosed braces and the macro
engine's synthesized errors become Error nodes plus problems;
- every node span is the union of its leaves' original spans, extended
over the \begin{…}/\end{…} syntax mitex drops; the root spans the
whole text; Runs select their own bytes.
Tests (tests/integration/normalize.rs): 263 per-fixture snapshots
(tree + problems) — new files under tests/integration/snapshots/,
reviewed by spot checks and by listing every non-error fixture that
reports a problem (only the by-design ones remain: \hline dropped,
ragged rows padded, and the two engine gaps below) — plus 22 structural
tests covering the rules above.
Spec/overrides: `\ ` is the empty-name command in mitex's spec (was
keyed " "), \substack is a command whose one argument holds the rows,
so a command with env semantics now builds a Matrix; commands.json
regenerated (86 unsupported rows; inventory snapshot updated: the
empty-name row gained semantics).
Vendored patch #7 (mitex-lexer): \renewcommand of a built-in command
(`\renewcommand{\vec}[1]{…}`) no longer emits an error token; TeX allows
redefining any existing command.
Known gaps, filed as bd-f047ynng and allowlisted in the no-errors
assertion so the snapshots record them: the vendored macro engine does
not expand `\def` with parameters nor optional macro arguments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the problems -> diagnostics mapping (bd-entbg6x3) Phase 1, diagnostics item, of claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md. Subsystem 22 `math` (20 and 21 are taken by PR #704's pandoc and typst). - crates/quarto-error-catalog/error_catalog.json: Q-22-1 .. Q-22-13, one per normalize::ProblemKind — unknown command, unknown environment, mismatched environment, missing argument, unbalanced brace, unbalanced \left/\right, double script, macro expansion limit, unsupported command, ragged rows, ignored \limits, dropped construct, syntax error. - docs/errors/math/Q-22-*.qmd: a page per code (what / why / fix / where it is reported), status stub; docs/_quarto.yml gains the `math` sidebar section after `image`. `cargo xtask lint` enforces both. - quarto_math::diagnostics: code_for / title_for / severity_for / diagnostic_for / diagnostics. Errors stop the conversion (the seam falls back to verbatim TeX); Unsupported, RaggedRows, IgnoredLimits and Dropped are warnings. A diagnostic's location is the problem's span as a SourceInfo::substring of the math text's own SourceInfo (pampa's Math.text_source), so it resolves to the exact .qmd bytes; a span-less problem (the macro engine's synthesized arity error) points at the whole expression. - Tests (tests/integration/diagnostics.rs, 6): every kind has a distinct Q-22 code present in the live catalog with the same title, subsystem and docs_url; severities; code/title/message/hint on a built diagnostic; location arithmetic through a file-rooted SourceInfo; span-less fallback; warning path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Phase 1, OMML writer item, of
claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md.
omml.rs — to_omml(&Normalized) -> String (m:oMath inline, m:oMathPara
> m:oMath display; embeddable, no xmlns) and to_omml_document (same
with the m:/w: namespaces declared, for validation). The writer is
total: an Error node becomes a literal run (m:lit + m:nor), so pampa's
docx writer decides from Node::has_errors whether to emit the OMML or
fall back to verbatim TeX. Element choices, per the plan's conventions:
- runs m:r > m:t (xml:space="preserve" at edge spaces); Style folds
into m:sty/m:scr on contained runs; Text is m:nor, with weight/slant
as w:b/w:i because CT_RPR makes nor a choice against scr/sty; Color
is w:color; spaces are Unicode spaces by width, negative ones dropped;
- m:f (m:type noBar for \atop and \binom, the latter fenced in m:d),
m:rad (m:degHide + empty m:deg when no degree), m:sSub/sSup/sSubSup;
- m:nary with m:chr, m:limLoc (undOvr in display except integral-like
operators, subSup inline, explicit \limits wins), m:subHide/m:supHide
for absent limits (the schema requires both m:sub and m:sup) and an
empty m:e — TeX gives a summand no scope;
- m:func with the name upright (m:sty p) and an empty body; \lim-style
limits as m:limLow/m:limUpp on the name, side limits as scripts;
- m:d with explicit m:begChr/m:endChr ("" for `.`), m:sepChr "|" and
one m:e per \middle part;
- m:acc, m:bar, m:groupChr (+m:vertJc bot on top braces), \underbrace_x
and \overbrace^x as m:limLow/m:limUpp around the group, m:limUpp/
m:limLow for \overset/\underset and the \xrightarrow family;
- m:m with m:mcs column count inside m:d when fenced; cases/rcases as
m:d with one empty fence around an m:eqArr; aligned and top-level `\\`
as m:eqArr with literal & alignment marks (§22.1.2.34); m:phant,
m:borderBox strikeBLTR for \cancel.
Tests (tests/integration/omml.rs): 263 per-fixture snapshots (new files
under tests/integration/snapshots/, spot-checked); every fixture's
document form validates against tests/schemas/ooxml/shared-math.xsd
via xmllint (skips without it); a tag-balance check that runs
everywhere; 14 structural tests over wrappers, escaping, spaces and
each construct above.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… greedy color clauses (bd-entbg6x3)
Phase 1, Typst writer + word-splitting items, of
claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md.
typst.rs — to_typst(&Normalized) -> String: the inside of a `$ … $`
block in plain Typst math, no mitex prelude, so a Quarto Typst document
imports nothing. Every syntax choice was compiled with Typst 0.15
before adoption and the corpus test compiles all 263 fixtures in one
document:
- runs split letter by letter (`4ac` -> `4 a c`, TeX semantics; Typst
reads adjacent letters as one identifier), numbers kept, Typst
specials escaped (`\/ \_ \^ \{ \} \& \# \$ \\ \" \@ \*`);
- symbols as Unicode; spaces as thin/med/thick/quad/wide or #h(…em);
- (a)/(b), display(…)/inline(…), binom, mat(delim: #none, a; b) for
\atop; sqrt / root; x_(i)^(2) with compound bases parenthesized and an
empty base written as ""; sum/product/integral… with limits()/
scripts() for \limits/\nolimits, literal characters for the four
operators Typst 0.15 has no name for; known operators bare,
op("name"[, limits: #true]) otherwise;
- lr(\( … \)) with escaped fences (Typst needs balanced brackets even
inside lr; `\(` still sizes) and mid(|); hat/tilde/macron/arrow/…,
overline/underline, overbrace(x, n)/underbrace, limits(base)^(ann),
stretch(→)^(f); bold/italic/upright/bb/cal/frak/sans/mono, strings
for text; std.hide, cancel, text(fill: #rgb("…"), x); mat(delim: …),
cases(…)/cases(reverse: #true, …), `a & = b \ c & = d`; errors as
string literals.
split.rs — split_run(text, span): identifiers (one letter each),
numbers (digits with one interior point), operators (one char), spans
partitioning the run. The Typst writer uses it now; MathML
(bd-9z83tcv0) will apply it to the AST.
normalize.rs — \operatorname{name} (and the starred / withlimits forms)
is a Func named by its argument, not upright text, which also gives
OMML an m:func; a greedy command's single clause (`\color{blue} x`)
holds the group and the rest, so a clause is unwrapped only when the
group is its sole child and \color splits it into color and body.
Tests: tests/integration/typst.rs (11: per-fixture snapshots, corpus
compile, splitting, escaping, spaces, each construct); split unit tests
(4). Snapshots: 263 new typst__ files; ast/omml snapshots for
operators/operatorname, operators/operatorname-star and
text-and-styles/color-switch updated for the two normalization changes
(reviewed).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…-entbg6x3)
Closes Phase 1 of claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md.
quarto_math::convert(text, Mode, Target, &text_source, &Spec) ->
Conversion { output, diagnostics, tree } is the entry point pampa's
docx and Typst writers will call (decision 8: no dependency on pampa's
AST types; the caller passes Math.text and Math.text_source). `output`
is None whenever the normalized tree has an Error node — half-converted
math is worse than the verbatim TeX the writer emits instead — and
warnings ride along with output. render() writes a tree regardless.
Tests: tests/integration/convert.rs (4): both targets on clean input,
errors withhold output and locate, warnings accompany output,
locations follow the text source's offset.
pampa: quarto-math becomes a native dev-dependency for
tests/integration/quarto_math_seam.rs (2): a block-quoted multi-line
display equation with an unknown command is parsed by
readers::qmd::read, converted with Math.text_source, and the Q-22-1
diagnostic resolves through map_offset to exactly the bytes `\bogus` on
row 2 of the .qmd (inside a block quote the text source is a Concat, so
preimage_in declines by design and map_offset is the right resolver);
and clean inline + display math converts for both targets without
diagnostics.
Verified: full `cargo xtask verify` passed (lint, clippy, workspace
build and tests, ts-packages, hub-client build and tests).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
cscheid
added this pull request to stack #707
September 21, 2026 21:48
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This was referenced Sep 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of
claude-notes/plans/2026-09-21-quarto-math-and-native-docx.md(strand bd-entbg6x3, epic bd-pq9k90z2: native docx writer). Stacked on #705 (pampaMath.text_source), which it builds on; draft while both are held.What this adds
A new crate
quarto-math, a micro-pandoc for math expressions: one reader, a spec we own, one writer per target.reader.rs): the mitex LaTeX parser, vendored undercrates/quarto-math/vendor/(4 crates, Apache-2.0, 7 documented local patches inVENDORED.md), plus a leaf → original-span side table. Macro bodies map to their definition, macro arguments to the use site.spec/commands.json, 1013 rows): generated bycargo xtask gen-math-specfrom mitex's spec dump plusspec/overrides.json, withcodexresolving Typst symbol names to Unicode. Each row is{parse: mitex item, sem: writer semantics, typst: alias}. Drift, coverage and corpus-acceptance tests.MathAst+ normalization (ast.rs,normalize.rs): folded scripts, bound\limits, resolved\left…\right, environments split into rows/cells, symbols resolved, exact nested spans, errors as nodes plus problems; never panics.diagnostics.rs):Q-22-1…Q-22-13under a newmathsubsystem (catalog,docs/errors/math/pages, sidebar). Locations are substrings of the math text'sSourceInfo(pampa'sMath.text_sourcefrom pampa: record byte-for-byte provenance of Math.text (Math.text_source) #705).omml.rs): every fixture validates against the vendored ECMA-376shared-math.xsd(12-file transitional closure undertests/schemas/ooxml/, from python-docx'sref/).typst.rs): plain Typst math, no mitex prelude; every syntax choice probed against Typst 0.15; the whole corpus compiles in onetypst compile(skipped without the binary).convert()seam (convert.rs): the entry point the docx/Typst writers will call. Output is withheld when the tree has an error so writers fall back to verbatim TeX. End-to-end test through pampa: a block-quoted multi-line equation with an unknown command yields a diagnostic that resolves to exactly the.qmdbytes.Fixture corpus: 263 TeX snippets in 14 groups (
tests/fixtures/), including 26 copied verbatim fromdocs/, the Q1 test corpus and quarto-web; each is snapshotted for the AST, OMML and Typst.Not in this PR
Phase 2 (OPC package layer, validation harness), Phase 3 (docx writer,
--to docxwiring), Phase 4 (sidecars, comment round trip). MathML is bd-9z83tcv0. Two mitex macro-engine gaps (\defwith parameters, optional macro arguments) are filed as bd-f047ynng and allowlisted.Verification
cargo xtask verify(full: lint, clippy, workspace tests, ts-packages, hub-client build and tests) passed at the tip. 138 tests in quarto-math and the vendored crates plus the pampa seam tests.🤖 Generated with Claude Code