From 569e3b690488f0b26cd6c04ca1478264d2a6253b Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 17 Sep 2026 03:02:47 +0800 Subject: [PATCH 1/6] One reading of a compile-flag element; the databases list what the compiler receives (#655) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An element of `cflags`, `cxxflags` and `asmflags` now has one meaning on every host: the POSIX shell's word syntax without expansions, where a backslash escapes only a blank, a quote or a backslash (mcpp.manifest.flag_words, SPEC-004 §8). The ninja writer quotes each word for its host, and compile_commands.json and `mcpp emit build-database` list the words, so the databases equal the build by construction instead of re-reading command lines. Before, an element meant whatever the host's command-line reader made of it (measured on Linux and Windows runners, #656: five of fourteen spellings gave different macros), and the databases listed the text with a third reading, which is how libarchive's `-DPLATFORM_CONFIG_H=\"...\"` reached consumers with its backslashes. - A `defines` entry is one value: `N="x"` reaches the compiler as `-DN="x"` (it arrived as `-DN=x`). Features, globs, targets and `mcpp:cfg=` alike. - Link units keep their rendered text; GAS units' database entries list their edge's list; the `-std` guard, dialect promotion and include normalisation read words. - Windows `shell_quote_arg` doubles backslashes before a quote and the closing quote. - The first plan after an upgrade names an element whose words changed (`build/flag-words`); a repeated plan says nothing. No flag-list element of the published index changes. Tests: FlagWords.* (syntax table and round trip), CompileCommandsArgs.* (edge equals database, host quoting round trip, /bin/sh as the reference), e2e 736 (same assertions on every host; fails on 2026.9.16.2). Records: .agents/docs/2026-09-17-655-*. Version 2026.9.17.1. --- ...ents-are-not-the-command-the-build-runs.md | 186 ++++++++++++++ .../2026-09-17-655-implementation-plan.md | 240 ++++++++++++++++++ .agents/docs/2026-09-17-655-index-readings.py | 70 +++++ .agents/docs/2026-09-17-655-probe.sh | 33 +++ .agents/docs/README.md | 6 +- CHANGELOG.md | 25 ++ docs/04-mcpp-toml.md | 41 +++ docs/30-build-mcpp.md | 6 +- docs/specs/build-database.md | 13 +- docs/specs/manifest-semantics.md | 40 ++- docs/zh/04-mcpp-toml.md | 35 +++ docs/zh/30-build-mcpp.md | 6 +- mcpp.toml | 2 +- modules/buildmcpp/src/directives.cppm | 4 +- modules/manifest/src/flag_words.cppm | 211 +++++++++++++++ modules/manifest/src/manifest.cppm | 2 + modules/manifest/src/types.cppm | 11 +- modules/manifest/tests/test_flag_words.cpp | 100 ++++++++ modules/versioning/src/version.cppm | 2 +- src/build/build_database.cppm | 64 +---- src/build/compile_commands.cppm | 128 ++++------ src/build/flags.cppm | 57 ++++- src/build/ninja_backend.cppm | 56 ++-- src/build/plan.cppm | 5 +- src/build/prepare.cppm | 131 +++++++++- src/modgraph/scanner.cppm | 24 +- ...ds_reach_the_compiler_and_the_databases.sh | 156 ++++++++++++ tests/unit/test_compile_commands.cpp | 136 +++++++++- 28 files changed, 1572 insertions(+), 218 deletions(-) create mode 100644 .agents/docs/2026-09-17-655-arguments-are-not-the-command-the-build-runs.md create mode 100644 .agents/docs/2026-09-17-655-implementation-plan.md create mode 100644 .agents/docs/2026-09-17-655-index-readings.py create mode 100644 .agents/docs/2026-09-17-655-probe.sh create mode 100644 modules/manifest/src/flag_words.cppm create mode 100644 modules/manifest/tests/test_flag_words.cpp create mode 100644 tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh diff --git a/.agents/docs/2026-09-17-655-arguments-are-not-the-command-the-build-runs.md b/.agents/docs/2026-09-17-655-arguments-are-not-the-command-the-build-runs.md new file mode 100644 index 000000000..6c631e7dd --- /dev/null +++ b/.agents/docs/2026-09-17-655-arguments-are-not-the-command-the-build-runs.md @@ -0,0 +1,186 @@ +--- +subject: triage +status: active +--- + +# #655: the `arguments` array is not the command the build runs + +**Revision 2026-09-17 (before implementation).** §3 proposed that the +database adopt each host's own reading (C1) and that defines be tagged at +ingestion (C2). The maintainer asked for the option with the best cross-platform +compatibility, so the adopted design replaces C1: an element of `cflags`, +`cxxflags` and `asmflags` has one host-independent syntax, the build quotes each +word for its host, and the databases list the words without reading any command +line back. C2 is kept in a simpler form (a define enters the list already +spelled as one word). §3 below is the proposal as written; the adopted design, +its criteria and the findings made while implementing are +`2026-09-17-655-implementation-plan.md`. + +**Issue:** mcpp-community/mcpp#655, "emit build-database: shell-escaped quotes +inside the `arguments` array break consumers". + +Engine code was read at `51aeee0e` (origin/main, mcpp 2026.9.16.2). A statement +marked *measured* was run on Linux x86_64 with the released mcpp 2026.9.16.2 and +`gcc@16.1.0`; the probe is `2026-09-17-655-probe.sh`. A statement marked *read* +names a file and line and was not executed. Nothing here ran on macOS or Windows. + +## 1. Verdict + +The report is a real engine defect, and it is not a documentation question. +SPEC-005 R3.6 and R3.7 state that a unit's `arguments` is its compile command +and that it is the same record `compile_commands.json` lists; the JSON +Compilation Database and S1 both define `arguments` as an argument vector that +is executed without a shell. Consumers should not unescape defensively: after +the fix below, a defensive unescape would corrupt a define whose value really +contains a backslash. + +The issue describes one case. The defect is wider. The probe compares, for five +macros, the value the compiler receives when ninja runs the build with the value +it receives when the database's `arguments` is executed directly (*measured*): + +| Declared in `mcpp.toml` | Build (ninja, `/bin/sh -c`) | `arguments` executed | Agree | +|---|---|---|---| +| `cflags = ["-DESC=\\\"esc.h\\\""]` (the libarchive form) | `ESC "esc.h"` | `ESC \"esc.h\"` | no, #655 | +| `cflags = ["-DMID=\"mid\""]` | `MID mid` | `MID "mid"` | no | +| `cflags = ["-DSQ='sq'"]` | `SQ sq` | `SQ 'sq'` | no | +| `defines = ["DEF=\"def\""]` | `DEF def` | `DEF "def"` | no, and the build is the wrong one | +| `defines = ["SPACE=\"a b\""]` | `SPACE "a b"` | `SPACE "a` plus a stray argument `b"`; the compiler exits 1 | no | + +Only the database's reading of `SPACE` stops the compiler. The other four +disagreements compile silently with a different macro value, which is the worse +form for a language server. + +## 2. Cause + +### 2.1 Two word splitters answer one question, and the per-unit path uses the wrong one + +The ninja rule is `command = $cc $cflags $unit_cflags ...` +(`src/build/ninja_backend.cppm:1246`), and ninja runs it with `/bin/sh -c` on +POSIX hosts and with `CreateProcess` on Windows. Every flag string in the engine +is therefore text in the host's command-line syntax. The database has to turn +that text back into words by the same rules the host applies. + +The engine contains two splitters for that purpose (*read*): + +- `mcpp::build::split_flags` (`src/build/compile_commands.cppm:133`) is used for + every translation unit in both databases. It undoes ninja's `$` escapes and + removes a quote only when the quote opens a token. It does not implement + backslash escapes, it treats a quote in the middle of a token as data, and it + has one rule for both hosts. The unit test + `CompileCommandsArgs.InnerQuotesAreNotStripped` + (`tests/unit/test_compile_commands.cpp:307`) asserts the second of these + choices, so the divergence in rows MID and SQ is pinned by a test rather than + merely untested. +- `split_command_words` (`src/build/build_database.cppm:309`), added in #636 for + the standard library units (R3.11), implements POSIX `sh` word splitting + (backslash outside quotes, the four escapable characters inside double quotes, + quotes anywhere in a word) and, separately, the MSVCRT rules that + `CreateProcess` programs use. + +The second splitter is the correct one, and it already exists. The per-unit path +was never moved onto it. + +### 2.2 The database joins package flags differently from the build + +`package_flag_args` (`src/build/compile_commands.cppm:211`) joins +`packageCflags`/`packageCxxflags` with bare spaces. The build joins the same +vector with `join_flags` (`src/build/ninja_backend.cppm:198`), which +shell-quotes a `-D` element that contains a space (mcpp#234). The database +therefore parses a string the build never wrote, which is why `SPACE` is cut in +two even though its quotes open the token. + +### 2.3 The silent twin: `defines` values lose their quotes in the build + +`join_flags` quotes a define only when it contains a space. The `defines` key is +documented as `name` or `name=value` (`docs/04-mcpp-toml.md:333`), that is a +value and not shell text, but `DEF="def"` reaches `sh` unquoted and the compiler +receives `-DDEF=def` (*measured*). This is a build defect that the database +defect was hiding: once the database agrees with the build, it will faithfully +report the wrong value. + +The quoting cannot simply be widened to every `-D` element, because +`scanner.cppm:1209` pushes `defines` into the same vector as raw `cflags`, and a +raw `cflags` element such as libarchive's `-DPLATFORM_CONFIG_H=\"...\"` is shell +text that must not be quoted again. The origin of the element is lost at +ingestion. + +## 3. Proposed changes + +### C1. One splitter, one rendering (fixes #655 and rows MID, SQ, SPACE) + +1. Move `split_command_words` out of `build_database.cppm` into + `compile_commands.cppm` (or `flags.cppm`) as the single host-command-line + splitter, and make `split_flags` "undo ninja `$` escapes, then + `split_command_words(s, is_windows)`". The ninja unescape must come first, as + it does today, because `escape_ninja_chars` runs before `shell_quote_arg`. +2. Make `package_flag_args` call `join_flags` (export it from `ninja_backend` or + move it beside `shell_quote_arg` in `flags.cppm`), so the database parses the + byte string the build writes. +3. Replace `InnerQuotesAreNotStripped` with a test stating the opposite: a + mid-token quote is quoting, because `sh` treats it so. + +No manifest or index change is needed. The libarchive recipe +(`mcpp-index/pkgs/c/compat.libarchive.lua:48`) is correct shell text for the +contract the build already implements. + +### C2. `defines` are values, rendered by the engine (fixes row DEF) + +Keep the origin of a define through ingestion: either a separate +`packageDefines` vector on the unit, or the element pushed already rendered with +`shell_quote_arg`. `join_flags` then quotes every define that contains any +character in `shell_quote_arg`'s trigger set, not only a space, and never +touches a raw `cflags` element. + +This changes the build's output for a define whose value contains a quote, +backslash, `$` or similar. No package in `mcpp-community/mcpp-index` declares +such a `defines` entry (*read*, grep over `pkgs/`), so the change is visible +only to a project that relied on the value being shell text. It should be named +in the release notes. + +### C3. State the flag contract in the documentation + +`docs/04-mcpp-toml.md` (and its `zh` copy) should state that a `cflags`, +`cxxflags` or `ldflags` element is command-line text in the host's syntax and may +hold several words (`-include foo.h`), whereas a `defines` element is one value +that mcpp quotes. SPEC-005 R3.6 should add one sentence: `arguments` holds the +words the host passes to the compiler, after the host's quoting is removed. + +A further change, holding flags as argument vectors from ingestion to the ninja +writer so that no string is ever parsed back, would remove the splitter +altogether. It is not proposed here: the index publishes flag elements that pack +several words, so it needs a manifest contract change and a migration, and C1 +already makes the database agree with the build by construction. + +## 4. Criteria + +- **K1 (differential, the one that decides C1).** For a table of flag strings, + including every row of §1, a path with a space, `$ORIGIN`, `-include x.h`, a + backslash that precedes no quote, and an empty quoted argument, the words + `split_flags` returns equal the argv that `/bin/sh -c 'printf "%s\0" '` + prints. On Windows the reference is a helper that prints its own `argv`. + The oracle is the host, not an expectation written by hand. +- **K2 (end to end).** `2026-09-17-655-probe.sh` prints identical `READING + build` and `READING database` lines for all five macros, and the database's + exec returns 0. +- **K3 (row DEF).** `READING build: #define DEF "def"`. Before C2 it reads + `DEF def`. +- **K4 (revert).** Revert C1 alone and K1 and K2 turn red; revert C2 alone and + K3 turns red. A criterion that stays green without its change measures nothing. +- **K5 (consumer).** The reporter's case: `openxlings/xlings` with the released + build produces `-DPLATFORM_CONFIG_H="mcpp_libarchive_config.h"` in both + databases, and mcpp-language-server indexes libarchive without the + `expected "FILENAME"` error. + +## 5. Open points for review + +1. Windows was not measured. `split_command_words(…, windows=true)` follows the + MSVCRT rules; whether every Windows compile edge is a plain `CreateProcess` + (no `cmd /c` wrapper, which would add cmd.exe's own parsing) should be read + from `ninja_backend.cppm` before K1's Windows leg is written. +2. A raw `cflags` element with single quotes (`-DSQ='sq'`) already yields a + different macro on Linux (`sq`) and on Windows (`'sq'`), because the flag is + host command-line text. C1 makes the database report each host faithfully; + it does not make the manifest portable. C3 documents that; whether to + normalise raw flags to one syntax is a separate decision. +3. The reply to the reporter: `arguments` is exec-ready, the escaped form is an + engine defect, and consumers should not unescape. diff --git a/.agents/docs/2026-09-17-655-implementation-plan.md b/.agents/docs/2026-09-17-655-implementation-plan.md new file mode 100644 index 000000000..08003961e --- /dev/null +++ b/.agents/docs/2026-09-17-655-implementation-plan.md @@ -0,0 +1,240 @@ +--- +subject: triage +status: active +--- + +# #655 implemented: one reading of a compile-flag element, the plan, and the ledger + +This record turns `2026-09-17-655-arguments-are-not-the-command-the-build-runs.md` +("the triage record") into work. The triage record measured that the compile +databases list arguments the compiler never receives, in five spellings, and +that one of the five is a defect of the build itself. Its §3 proposed that the +databases adopt each host's reading. The maintainer asked for the design with +the best cross-platform compatibility; §1 states the design adopted instead and +why it is that design, §2 reviews it from the angles the change touches, §3 is +the ledger, and §5 records what implementing it found. + +Base commit: mcpp `51aeee0e` (2026.9.16.2). The engine release is 2026.9.17.1. + +## 1. The design + +### 1.1 What was measured + +The temporary pull request mcpp-community/mcpp#656 ran the released engine +(2026.9.16.1, whose flag handling equals 2026.9.16.2) over fourteen spellings, +one project each, on Linux, macOS and Windows runners (run 35137218111). The +value each macro received (`<08>` is the byte a stringized `\b` becomes): + +| element | Linux | Windows | +|---|---|---| +| `-DV=\"esc.h\"` | `"esc.h"` | `"esc.h"` | +| `-DV="mid"` | `mid` | `mid` | +| `-DV='sq'` | `sq` | `'sq'` | +| `-DV='a b'` | `'a b'` | `'a b'` | +| `-DV=a\b` | `ab` | `a<08>` (backslash kept) | +| `-DV=a\\b` | `a<08>` | `a\b` (both kept) | +| `-DV=a$b` | `a` | `a` | +| `defines = ["V='c'"]` | `c` | `'c'` | +| `defines = ["V=\"def\""]` | `def` | `def` | + +Every database listed the element text unread (`-DV=\"esc.h\"`, `-DV='a`, +`b'`). The macOS job failed at the link of every probe project under the store +binary, which is a defect of the probe's environment, not a reading; macOS runs +POSIX `sh` under ninja as Linux does, and e2e 736 on the macOS runner of the +engine pull request is its measurement. + +Two facts follow. The meaning of an element depended on the host: five of the +fourteen spellings gave different macros on Linux and Windows. And no reading +of the text back can make the databases agree with the build on both hosts +while the element itself means two things. + +### 1.2 The adopted design + +An element of `cflags`, `cxxflags` and `asmflags` has one meaning on every host +(SPEC-004 §8, `mcpp.manifest.flag_words`): + +- the POSIX shell's word syntax without expansions: blanks separate words, + `'...'` is literal, `"..."` honours `\"` and `\\`, quoted and unquoted pieces + concatenate; +- outside quotes a backslash escapes only a blank, a quote or a backslash, so a + Windows path keeps its backslashes and pkg-config's `\ ` is a space; +- `$` and the shell operators are ordinary characters; +- a `defines` entry is one value, spelled into the list with `flag_element` so + that it reads back as exactly one word. + +The ninja writer quotes each word for its host (`ninja_command_word`: +`shell_quote_arg`, then `$` doubled), and the databases list the words +(`package_flag_args`). Nothing in the unit path reads a command line back. The +host readers remain only for text the engine renders itself (the global flag +strings, the std module command), where `split_flags` undoes ninja's escapes and +then applies `host_command_words`. + +### 1.3 Why this design and not the triage record's C1 + +C1 makes the databases faithful to a build whose meaning differs by host. It +fixes the reported symptom and keeps `-DV='sq'` meaning two macros. The adopted +design removes the host dependence at the manifest, so the same manifest builds +the same program on Linux, macOS and Windows, and the databases are equal to the +build by construction rather than by a second parser agreeing with the first. + +The compatibility cost is bounded by measurement rather than by argument: + +- every string literal of the published index that starts like a flag was + read under the new syntax and under both hosts' previous readings + (mcpplibs/mcpp-index `c176883`, 285 literals, + `2026-09-17-655-index-readings.py`): 9 differ, and all 9 are pieces of the CMake command line in + `compat.mysql-connector-cpp.lua`'s install hook, not elements of a flag list. + libarchive's `\"` and compat.lua's packed `-include` read the same under all + three (§4 K6); +- an element whose reading does change is named once, with both readings, on the + first plan after the upgrade (§2.5). + +## 2. Review from the angles the change touches + +### 2.1 Architecture + +One module owns the meaning (`modules/manifest/src/flag_words.cppm`), below the +build layer, so the manifest parser, the scanner, the build-program directives, +the planner, the ninja writer and both databases read the same function. The two +duplicate splitters that existed before (`compile_commands::split_flags` and +`build_database::split_command_words`) are reduced to one host reader in the same +module. The link path is deliberately not moved: its elements are rendered text, +not manifest elements (§5 F1). + +### 2.2 Stability + +A plain word (no blank, quote or backslash) is written byte-identically, so the +build.ninja of a project without such elements does not change beyond the +version-driven fingerprint. The round-trip properties are the invariants the +engine relies on, and each is a test: `flag_words(flag_element(w)) == {w}` +(2000 generated words), `split_flags(ninja_command_word(w)) == {w}` (1000 words, +on each CI host), and `/bin/sh` receiving `shell_quote_arg(w)` as `w`. + +### 2.3 Simplicity + +The syntax is the shell's, minus expansions, with one narrowing (backslash). +`join_flags` for compile lists is one line; the defines channel needs no tag, +only a spelling. The diagnostic's model of the previous reading is 25 lines and +disappears when the model is no longer needed. + +### 2.4 Cross-platform consistency + +The e2e criterion (§4 K2) asserts the same macro values and the same database +words on Linux, macOS and Windows, which the old build could not satisfy +(§1.1). The Windows quoting was corrected at the same time: `shell_quote_arg` +doubles the backslashes before a quote and before the closing quote, so a word +ending in `\` no longer escapes the quote that ends it. + +### 2.5 Upgrade + +The fingerprint names the mcpp version and every flag, so the first plan after +an upgrade writes a new output directory. The notes collected while manifests +load are released only when that directory has no `build.ninja` yet: once after +an upgrade, after an edit of a flag, and in a fresh checkout. A build that +repeats the plan says nothing, so a manifest already spelled for the new reading +is not warned about on every run (measured: three repeated builds, zero notes; +an edit of a define, one note). + +### 2.6 Test coverage + +| layer | test | what fails without the change | +|---|---|---| +| syntax | `modules/manifest/tests/test_flag_words.cpp` (6) | the module does not exist | +| edge == database | `CompileCommandsArgs.TheEdgeAndTheDatabaseListTheSameWords` | words listed as text | +| host quoting | `CompileCommandsArgs.AWordOnAnEdgeReadsBackAsTheWord` | the empty word vanished; a trailing `\` on Windows | +| host itself | `CompileCommandsArgs.ShellReceivesTheQuotedWordsAsWritten` | model and `/bin/sh` disagree | +| unit record | `CompileCommandsEmit.AUnitListsTheWordsOfItsFlagList` | text listed | +| end to end | e2e 736 | fails on 2026.9.16.2: `V_DOL=[a]`, `V_A=[1 -DV_B=2]` | + +## 3. The ledger + +Status is `todo`, `branch` (implemented on the pull request's branch, with its +reading), `done` (merged or published, with its reading) or `dropped` (with the +reason). + +### 3.1 Engine: mcpp-community/mcpp, one pull request (`fix/655-flag-words`) + +| id | task | depends on | status | +|---|---|---|---| +| E1 | `mcpp.manifest.flag_words`: `flag_words`, `flag_element`, `host_command_words` | - | branch: `FlagWords.*` 6/6 | +| E2 | every define push site spells one word (`[build]`, globs, targets, features, `mcpp:cfg=`) | E1 | branch: e2e 736 `V_DEF=["def"]` | +| E3 | compile edges write words quoted for the host; link edges keep the rendered text | E1 | branch: e2e 307, 615, 736 | +| E4 | databases list words; `split_flags` = ninja unescape + host reader; GAS units list their edge's list | E1 | branch: `CompileCommandsArgs.*`, `BuildDatabase.*` | +| E5 | include normalisation, the `-std` guard, dialect promotion and the std module's target-side flags read words | E1 | branch: build and e2e subset | +| E6 | Windows `shell_quote_arg` follows the MSVCRT backslash rule | - | branch: round trip on the Windows runner | +| E7 | `build/flag-words` note on the first plan | E1 | branch: e2e 736 D and E | +| E8 | SPEC-004 §8, SPEC-005 R3.7, docs/04 and docs/30 with their Chinese mirrors, CHANGELOG | E1-E7 | branch | +| E9 | version 2026.9.17.1 | - | branch | +| E10 | CI green on every workflow of the pull request | E1-E9 | todo | + +### 3.2 Release and ecosystem, in order + +| id | task | depends on | status | +|---|---|---|---| +| R1 | merge the engine pull request; verify the `origin/main` run | E10 | todo | +| R2 | dispatch `release.yml`; upload each archive to GitCode with the local `gtc` as it appears | R1 | todo | +| R3 | merge the xim-pkgindex bump; read the index artifact, not git | R2 | todo | +| R4 | bootstrap pin 2026.9.17.1 (pull request) | R3 | todo | +| R5 | sandbox verification with CN mirrors (§6) | R3 | todo | +| R6 | reply on #655; close #656 | R5 | todo | + +No change is needed in mcpplibs/mcpp-index (§4 K6), in mcpp-plugins (no flag +list) or in openxlings/xlings (its manifest pins an older engine and carries no +flag element; its build database is verified in R5). + +## 4. Criteria + +- **K1** Syntax table and round trip (`FlagWords.*`). +- **K2** e2e 736 green on Linux, macOS and Windows with the same assertions. +- **K3** The same e2e fails on 2026.9.16.2 (measured locally: `V_DOL=[a]`, + `V_A=[1 -DV_B=2]`, `V_B=[V_B]`). +- **K4** The issue's probe reads identical build and database values for all + five macros, and the database's arguments execute with status 0 (measured + locally with the branch binary). +- **K5** The e2e subset that touches flags or databases (80 scripts) passes. +- **K6** No flag-list element of the published index reads differently (static + reading above), and building libarchive and lua consumers in the sandbox + produces no `build/flag-words` note (§6). +- **K7** In the sandbox, `openxlings/xlings`'s build database lists + `-DPLATFORM_CONFIG_H="mcpp_libarchive_config.h"` and one libarchive unit's + arguments execute with `-fsyntax-only`. + +## 5. Findings while implementing + +- **F1 The link path carries rendered text.** The first implementation routed + every `join_flags` caller through the word reading. Two callers pass link + units' flags, which the engine writes already quoted and escaped + (`-Wl,-rpath,'$$ORIGIN'`); the edge became `'-Wl,-rpath,$$$$ORIGIN'` and e2e + 307 and 615 failed with `libwrap.so not found` at the closure check, while + 2026.9.16.2 passed both. The compile and link joins are now two functions. +- **F2 The database listed the wrong list for a GAS unit.** Its edge carries the + `-D/-U/-I` subset and asmflags, the database listed all C flags. Both now read + `unit_asm_flags`. +- **F3 An empty word vanished on the edge.** `shell_quote_arg("")` is empty by + design for its other callers; `ninja_command_word` writes `''` or `""`. +- **F4 An unquoted `$ ` splits.** A unit test asserted that `-Idir$ with$ space` + is one argument. ninja turns `$ ` into a space before the host reads the line, + so it is three; the engine quotes every word it writes, and the test now + states the host's reading. +- **F5 A note that repeats on every build is noise.** The first version warned on + each plan; §2.5 is the rule adopted. + +## 6. Sandbox verification + +Recorded after the release in §7. + +## 7. Readings and closure + +Recorded as each row closes. + +## 8. Residuals + +- `ldflags`, `dialect_cxxflags` and `std-module-flags` keep their current + meaning (documented in docs/04). Moving `ldflags` needs the link path's + rendered text separated from manifest elements first (F1). +- `-B` in the global compile flags is ninja-escaped but not quoted, so + a registry path with a space would split; the databases now report that split + faithfully. +- The detach-codegen edge on Windows runs `cmd.exe /c`, which interprets + `<>|&^` outside its own quote counting; a word containing them is not covered + by the round-trip criterion there. diff --git a/.agents/docs/2026-09-17-655-index-readings.py b/.agents/docs/2026-09-17-655-index-readings.py new file mode 100644 index 000000000..47d4c58ca --- /dev/null +++ b/.agents/docs/2026-09-17-655-index-readings.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +# #655 K6: every flag-like string literal of an index checkout, read under the +# new flag syntax and under the POSIX and MSVCRT readings the previous build +# applied. Run from the index checkout; prints the literals whose readings differ. +import re, shlex, subprocess, sys +def flag_words(s): + out=[];w=[];st=False;i=0 + while i'] + posix=[p.replace('\x00','$') for p in posix] + win=crt(ws) + if new!=posix or new!=win: + bad+=1; print('DIFF', repr(e), 'new', new, 'posix', posix, 'win', win) +print(len(lits),'literals;',bad,'differ') diff --git a/.agents/docs/2026-09-17-655-probe.sh b/.agents/docs/2026-09-17-655-probe.sh new file mode 100644 index 000000000..04e3985c4 --- /dev/null +++ b/.agents/docs/2026-09-17-655-probe.sh @@ -0,0 +1,33 @@ +#!/bin/sh +# #655 probe: the command the build runs against the `arguments` both databases list. +# Usage: MCPP= sh 2026-09-17-655-probe.sh +set -eu +M=${MCPP:-mcpp}; S=${1:?scratch dir} +rm -rf "$S"; mkdir -p "$S/src"; cd "$S" +cat > mcpp.toml <<'TOML' +[package] +name = "i655" +version = "0.1.0" + +[build] +sources = ["src/*.c"] +cflags = ["-DESC=\\\"esc.h\\\"", "-DMID=\"mid\"", "-DSQ='sq'"] +defines = ["DEF=\"def\"", "SPACE=\"a b\""] +TOML +printf 'int main(void){return 0;}\n' > src/main.c +"$M" build >/dev/null 2>&1 +"$M" emit build-database --format json > db.json 2>/dev/null +D=$(ls -d target/*/*/ | head -1) +CMD=$(cd "$D" && ninja -t commands obj/main.o | tail -1 | sed 's/ -MMD .*$/ -E -dM -x c \/dev\/null/') +(cd "$D" && /bin/sh -c "$CMD") | grep -E 'define (ESC|MID|SQ|DEF|SPACE) ' | sed 's/^/READING build: /' +python3 - <<'PY' +import json, subprocess +a = json.load(open('compile_commands.json'))[0]['arguments'] +i = a.index('-c') +r = subprocess.run(a[:i] + ['-E', '-dM', '-x', 'c', '/dev/null'], capture_output=True, text=True) +print('READING database exec rc=%d' % r.returncode) +for l in r.stdout.splitlines(): + w = l.split() + if len(w) > 1 and w[1] in ('ESC', 'MID', 'SQ', 'DEF', 'SPACE'): + print('READING database:', l) +PY diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 86a59763a..9a4d56b44 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -291 records. +293 records. ## By subject @@ -62,6 +62,8 @@ Records that declare one. Everything else is listed by date below. ### triage +- [#655 implemented: one reading of a compile-flag element, the plan, and the ledger](2026-09-17-655-implementation-plan.md) — active +- [#655: the `arguments` array is not the command the build runs](2026-09-17-655-arguments-are-not-the-command-the-build-runs.md) — active - [#646 to #649 implemented: the plan, its review from eight angles, and the ledger](2026-09-16-646-649-implementation-plan.md) — landed - [Four issues from a framework and an editor: #646, #647, #648 and #649, read against the engine and routed by home](2026-09-16-646-649-four-issues-by-home.md) — landed - [Link forms, standard levels and a path limit: the asks of #641 and #642, read against the code](2026-09-15-641-642-link-forms-standards-and-paths.md) — landed @@ -76,6 +78,8 @@ Records that declare one. Everything else is listed by date below. ### 2026-09 +- [#655 implemented: one reading of a compile-flag element, the plan, and the ledger](2026-09-17-655-implementation-plan.md) — active +- [#655: the `arguments` array is not the command the build runs](2026-09-17-655-arguments-are-not-the-command-the-build-runs.md) — active - [#646 to #649 implemented: the plan, its review from eight angles, and the ledger](2026-09-16-646-649-implementation-plan.md) — landed - [Four issues from a framework and an editor: #646, #647, #648 and #649, read against the engine and routed by home](2026-09-16-646-649-four-issues-by-home.md) — landed - [Link forms, standard levels and a path limit: the asks of #641 and #642, read against the code](2026-09-15-641-642-link-forms-standards-and-paths.md) — landed diff --git a/CHANGELOG.md b/CHANGELOG.md index a5c56827a..1fd75f808 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,31 @@ ## [Unreleased] +### 编译 flag 列表元素的一种读法,以及与构建一致的编译数据库:#655(2026.9.17.1) + +`mcpp emit build-database` 与 `compile_commands.json` 的 `arguments` 此前把 flag 文本经一种 +手写的读法切分,既不是 POSIX `sh` 也不是 MSVCRT 的规则:libarchive 描述符的 +`-DPLATFORM_CONFIG_H=\"mcpp_libarchive_config.h\"` 被列成带反斜杠的参数,消费者按原样执行时 +libarchive、bzip2、lz4、xz、zlib 与 zstd 的每个 C 文件都无法解析。实测同一类不一致有五种拼写, +其中一种是构建本身的缺陷:`defines = ["N=\"x\""]` 到达编译器时丢掉了引号。设计、测量与计划: +`.agents/docs/2026-09-17-655-*.md`。 + +- **一个元素代表的词在每个宿主上相同。** `cflags`、`cxxflags`、`asmflags` 的元素按 POSIX `sh` + 的词法读取,不做展开,且反斜杠只在空格、制表符、引号与反斜杠之前转义(Windows 路径保持原样); + 构建为宿主给每个词加引号,编译器收到的参数不再取决于 ninja 把命令交给 `sh` 还是 + CreateProcess。规则见 SPEC-004 §8 与 docs/04「Compile-flag syntax」。 + (单测 `FlagWords.*`、`CompileCommandsArgs.*`,e2e 736) +- **`defines` 的条目是一个值。** `N="x"` 以 `-DN="x"` 到达编译器;构建程序的 `mcpp:cfg=` + 与 feature 的 `defines` 相同。 +- **两个编译数据库列出编译器收到的词。** 单元自己的 flag 直接列出词,引擎渲染的全局 flag + 按宿主的读取规则还原;GAS 单元列出与其边相同的 `-D/-U/-I` 子集与 asmflags。 + SPEC-005 R3.7 陈述这一点。 +- **升级提示。** 一个元素的词与 2026.9.17.1 之前在同一宿主上传入的参数不同时,首次规划以 + `build/flag-words` 警告并给出两者;重复同一规划的构建不再提示。索引中已发布的描述符没有 + 这样的元素。 +- Windows 上 `shell_quote_arg` 按 MSVCRT 规则加倍引号前与结尾处的反斜杠,以 `\` 结尾的 + 词不再吞掉闭合引号。 + ### 内置 xlings 升至 2026.9.16.1(2026.9.16.2) `kXlingsVersion` 与 `.github/` 中的全部 xlings pin 前移到 2026.9.16.1(openxlings/xlings#601)。 diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index ef2f95b39..dc65ac4a7 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -418,6 +418,47 @@ jobs = "auto" # Concurrent compiles: a positive number, or " bmi_schedule = "auto" # Module-edge scheduling: auto (= off) | on | off (§ below) ``` +#### Compile-flag syntax *(mcpp 2026.9.17.1+)* + +An element of `cflags`, `cxxflags` or `asmflags` stands for one or more compiler +arguments ("words"). The syntax is the same on every host, wherever the list is +written: `[build]`, `[targets.]`, a `flags` glob entry, a feature, a +`[target..build]` section, an xpkg descriptor, and the `mcpp:cflag=` / +`mcpp:cxxflag=` directives of a build program. + +| Written | Words the compiler receives | +|---|---| +| `"-O2 -g"` | `-O2`, `-g` | +| `"-include config.h"` | `-include`, `config.h` | +| `"'-DNAME=a b'"` or `"-DNAME=\"a b\""` | `-DNAME=a b` | +| `"-DNAME=\\\"text\\\""` | `-DNAME="text"` (a string literal) | +| `"-I/opt/my\\ dir/include"` | `-I/opt/my dir/include` | +| `"-IC:\\sdk\\include"` | `-IC:\sdk\include` | +| `"-DNAME=a$b"` | `-DNAME=a$b` | + +The rules, stated on the element's text (after TOML or Lua has removed its own +escapes): + +- unquoted spaces and tabs separate words; +- `'...'` is literal up to the next `'`; +- `"..."` is literal except that `\"` and `\\` stand for `"` and `\`; +- outside quotes, a backslash before a space, a tab, `"`, `'` or `\` stands for + that character; any other backslash is literal; +- quoted and unquoted pieces that touch form one word; +- `$`, `*`, `;`, `|` and the other shell operators have no meaning. + +A `defines` entry is one value and is not read by this syntax: `defines = +["NAME=\"text\""]` passes the single word `-DNAME="text"`. `ldflags`, +`dialect_cxxflags` and `std-module-flags` are not covered by this section. + +`compile_commands.json` and `mcpp emit build-database` list the same words in +`arguments`, ready to execute without a shell. + +On the first plan of a project, mcpp warns under `build/flag-words` about an +element whose words differ from the arguments a release before 2026.9.17.1 +passed on the same host, and names both. A build that repeats the plan does not +repeat the warning. + #### `dependency_linkage` — static or shared is the consumer's decision ```toml diff --git a/docs/30-build-mcpp.md b/docs/30-build-mcpp.md index d06cce58a..a76771373 100644 --- a/docs/30-build-mcpp.md +++ b/docs/30-build-mcpp.md @@ -53,11 +53,11 @@ is ignored, so diagnostics may be logged freely. | Directive | Effect | |---|---| -| `mcpp:cxxflag=` | add `` to the C++ compile flags | -| `mcpp:cflag=` | add `` to the C compile flags | +| `mcpp:cxxflag=` | add `` to the C++ compile flags; `` is read as [compile-flag syntax](04-mcpp-toml.md#compile-flag-syntax-mcpp-20269171) (2026.9.17.1+) | +| `mcpp:cflag=` | add `` to the C compile flags, read the same way | | `mcpp:link-lib=` | link `-l` | | `mcpp:link-search=` | add a library search dir (`-L`; relative dirs resolve against the project root) | -| `mcpp:cfg=` | define `-D` for both C and C++ | +| `mcpp:cfg=` | define `-D` for both C and C++; `` is one value, as a `defines` entry | | `mcpp:generated=` | add a generated source to the build. **A relative path resolves against the project root for the root package, but against `MCPP_OUT_DIR` for a dependency's build.mcpp** — emit an absolute path if the package is both (see below) | | `mcpp:source=` *(0.0.100+)* | select a **pre-existing** source file into the build (absolute, or relative to the package root). Same downstream effect as `generated=`; use it for files the program *chose* (payload/vendored tree) rather than wrote — e.g. a per-target source selection over a large tarball | | `mcpp:include-dir=` *(0.0.100+)* | add a **private** include directory (`-I`) for this package's own TUs (absolute, or relative to the package root; normalized). Replaces the `cxxflag=-I` + `cflag=-I` double emission | diff --git a/docs/specs/build-database.md b/docs/specs/build-database.md index 36648a178..60b42f3ed 100644 --- a/docs/specs/build-database.md +++ b/docs/specs/build-database.md @@ -4,12 +4,12 @@ |---|---| | 规范编号 | SPEC-005 | | 标题 | mcpp 输出的构建数据库:内容、取值规则与不写工程目录的保证 | -| 状态 | 评审中 v1.1 | -| 版本 | 1.1 | -| 最后修改 | 2026-09-16 | +| 状态 | 评审中 v1.2 | +| 版本 | 1.2 | +| 最后修改 | 2026-09-17 | | 对应实现 | mcpp >= 2026.9.15.1 | | 相关设计文档 | `.agents/docs/2026-09-14-636-build-database-and-the-latest-xlings.md` | -| 相关 issue | #636, #648 | +| 相关 issue | #636, #648, #655 | | 依据的外部规范 | S1「C++ Build Database: IDE Profile」profile 0.2.0 与 S2 0.2.0 §3.4,取自 https://github.com/Sunrisepeak/lsp-mcpp-private 提交 `b82859d`(schema 自提交 `28ecd6e` 起未变);JSON Compilation Database | ## 0. 适用范围 @@ -98,7 +98,9 @@ mcpp 输出的 S1 文档满足 S1 等级 2,不输出 `ide.options`。等级 3 - **R3.7** 除 NASM 单元外,构建计划中的每个编译单元是一个翻译单元。`source`、 `work-directory`、`arguments`、`object` 与 `compile_commands.json` 中对应条目的 `file`、`directory`、`arguments`、`output` 取自同一条记录,因而逐字相同。 - **已实现** + `arguments` 中的每一项是编译器收到的一个参数,不带任何宿主的引号或转义,不经 shell + 即可执行:单元自己的 flag 列表按 SPEC-004 §8 读成的词列出,引擎为宿主渲染的文本 + 按该宿主的读取规则(POSIX `sh` 或 MSVCRT)还原。**已实现** - **R3.8** `provides` 把单元提供的模块名映射到空字符串,命令不执行构建(S1-8-6); `requires` 为单元导入的模块名,分区写全名 `M:P`。`private` 为 `false`,理由同 R3.4。 **已实现** @@ -175,3 +177,4 @@ mcpp 输出的 S1 文档满足 S1 等级 2,不输出 `ide.options`。等级 3 |---|---|---| | 1.0 | 2026-09-14 | 首版(#636)。 | | 1.1 | 2026-09-16 | R5.2 增加离线诊断码 `MCPP_OFFLINE_DOWNLOAD_REQUIRED`;R5.3 的 `network` 按观测列出;新增 R5.4(子进程不继承调用方描述符,xlings 子进程有期限并随 mcpp 结束)(#648)。 | +| 1.2 | 2026-09-17 | R3.7 陈述 `arguments` 的每一项是编译器收到的参数,单元 flag 按 SPEC-004 §8 的词列出(#655)。 | diff --git a/docs/specs/manifest-semantics.md b/docs/specs/manifest-semantics.md index cb1ae3595..326225155 100644 --- a/docs/specs/manifest-semantics.md +++ b/docs/specs/manifest-semantics.md @@ -5,8 +5,8 @@ | **规范编号** | SPEC-004 | | **标题** | `mcpp.toml` 的平面划分、条件化形状、解析轴与命名规约 | | **状态** | **草案(Draft)** | -| **版本** | 1.4 | -| **最后修改** | 2026-09-15 | +| **版本** | 1.5 | +| **最后修改** | 2026-09-17 | | **最低实现版本** | 条件化形状:mcpp **2026.8.29.1**(`[target..build-dependencies]` 起齐备);目标轴:mcpp **2026.9.6.4** | | **作者/维护** | mcpp-community | | **相关设计文档** | `.agents/docs/2026-09-07-mcpp-toml-unified-semantics-design.md`
`.agents/docs/2026-06-04-manifest-schema-ownership.md`
`.agents/docs/2026-09-03-xlings-workspace-as-the-one-table.md` | @@ -33,6 +33,7 @@ 一个条目按什么解析、键怎么命名。它不列举字段——字段在 docs/05。 它回答的是一个新字段或新 section 该长什么样,以及一份 manifest 为什么这样组织。 +§8 另陈述编译 flag 列表中一个元素代表哪些参数。 **边界。** 本规范不覆盖字段的准入条件,那由 docs/05 附录 A(Schema Ownership Principle)规定,本规范不重复它,只在 §6 引用并补充一条。 @@ -327,6 +328,40 @@ feature-deps feature-xlings ← 限定词是门 `package-default`;边上的 `linkage = "static"` 与写下的 `dependency_linkage = "static"` 都得到静态形态与原因 `requested`,且 `--strict` 下构建通过;按行 `linkage` 替换无条件的 `kind = "shared"`(`tests/e2e/692_a_package_states_its_default_link_form.sh`)。 +10. §8 的判据取自三处并要求一致:程序打印每个宏收到的值,`compile_commands.json` 与 + `mcpp emit build-database` 列出的词,三者都等于按 §8 读出的词;同一份断言在 Linux、 + macOS 与 Windows 上不变(`tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh`)。 + 规则本身由读回性质陈述:任意词经实现的拼写读回为它自身,经宿主引号读回也为它自身, + 且 POSIX 宿主上由 `/bin/sh` 实测(`modules/manifest/tests/test_flag_words.cpp`、 + `tests/unit/test_compile_commands.cpp`)。 + +## 8. 编译 flag 列表的元素 + +`cflags`、`cxxflags` 与 `asmflags` 的一个元素是一段文本,代表零个或多个词;编译器收到的 +参数就是这些词,按列表顺序排列。本节在元素的文本上陈述(TOML 或 Lua 先去掉自己的转义)。 +该读法对这三个键的每一个出现位置相同:`[build]`、`[targets.]`、`flags` 的 glob 条目、 +feature、`[profile.]`、`[target..build]`、xpkg 描述符,以及构建程序的 +`mcpp:cflag=` 与 `mcpp:cxxflag=` 指令。 + +1. 未加引号的空格与制表符分隔词,连续的分隔符等同于一个。 +2. `'` 开启一段单引号区域,区域内的字符按字面取到下一个 `'` 为止。 +3. `"` 开启一段双引号区域,区域内 `\"` 代表 `"`、`\\` 代表 `\`,其余字符按字面取到下一个 + 未转义的 `"` 为止。 +4. 引号区域之外,`\` 后跟空格、制表符、`"`、`'` 或 `\` 时代表该字符;其余 `\` 按字面。 +5. 相邻的区域与字面片段组成一个词;一个只含 `''` 或 `""` 的词是空词。 +6. 未闭合的引号区域延伸到元素末尾。 +7. 除上述字符外,任何字符没有特殊含义;实现**禁止**对 `$`、`*`、`?`、`;`、`|`、`&`、`<`、 + `>`、`` ` ``、`~` 做展开或解释。 + +`defines` 的一个条目 `X` 是一个值:它代表一个词 `-D` 与 `X` 的拼接,**禁止**按上述规则 +读取。实现向这三个列表插入一个词 `w` 时,**必须**使用一个按上述规则读回恰为 `w` 的拼写。 + +实现**必须**把每个词原样交给编译器,与宿主的命令行读取规则(POSIX `sh`、MSVCRT)无关; +`compile_commands.json` 与构建数据库(SPEC-005 R3.7)列出的参数**必须**是这些词。 + +`ldflags`、`dialect_cxxflags` 与 `std-module-flags` 不在本节范围内。 + +**状态:已实现(mcpp 2026.9.17.1)。** ## 变更记录 @@ -337,3 +372,4 @@ feature-deps feature-xlings ← 限定词是门 | 1.2 | 2026-09-07 | 一个包一个版本(mcpp 2026.9.6.6):新增 §4.5(身份=`(namespace, name)`,版本是约束;裁决与校验两步;范围必须双向可解且可被 `xpkg_dir` 回答);§4.3.1 改为「禁止命名**被解析的**层」,`accelerator` 明确被接受(2026.9.6.5);§7 补第 5 条的反向腿与第 6、7 条判据。 | | 1.3 | 2026-09-14 | 条件依赖声明替换同一身份的无条件声明,`targets.` 成为可条件化的 section,不读取的 section 必须报出(mcpp 2026.9.14.2):新增 §3.1.1 与 §7 第 8 条判据。 | | 1.4 | 2026-09-15 | 库目标的默认链接形态 `linkage`(mcpp 2026.9.15.2):§3.1.1 补默认值的语义、优先顺序与拒绝条件;§7 补第 9 条判据。 | +| 1.5 | 2026-09-17 | 编译 flag 列表元素的读法(mcpp 2026.9.17.1,#655):新增 §8 与 §7 第 10 条判据。 | diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index 07b98fbc1..6e1e03cb3 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -371,6 +371,41 @@ jobs = "auto" # 并发编译数:正整数,或 "auto"(见下 bmi_schedule = "auto" # 模块边调度:auto(= 关)| on | off(见下节) ``` +#### 编译 flag 的写法 *(mcpp 2026.9.17.1+)* + +`cflags`、`cxxflags` 与 `asmflags` 的一个元素代表一个或多个编译器参数(下称「词」)。 +写法在每个宿主上相同,也不论列表写在哪里:`[build]`、`[targets.]`、`flags` 的 +glob 条目、feature、`[target..build]`、xpkg 描述符,以及构建程序的 +`mcpp:cflag=` / `mcpp:cxxflag=` 指令。 + +| 写法 | 编译器收到的词 | +|---|---| +| `"-O2 -g"` | `-O2`、`-g` | +| `"-include config.h"` | `-include`、`config.h` | +| `"'-DNAME=a b'"` 或 `"-DNAME=\"a b\""` | `-DNAME=a b` | +| `"-DNAME=\\\"text\\\""` | `-DNAME="text"`(字符串字面量) | +| `"-I/opt/my\\ dir/include"` | `-I/opt/my dir/include` | +| `"-IC:\\sdk\\include"` | `-IC:\sdk\include` | +| `"-DNAME=a$b"` | `-DNAME=a$b` | + +规则陈述在元素的文本上(TOML 或 Lua 先去掉它们自己的转义): + +- 未加引号的空格与制表符分隔词; +- `'...'` 按字面取到下一个 `'`; +- `"..."` 按字面取,只有 `\"` 与 `\\` 分别代表 `"` 与 `\`; +- 引号之外,反斜杠后跟空格、制表符、`"`、`'` 或 `\` 时代表该字符,其余反斜杠按字面; +- 相邻的带引号与不带引号的片段组成一个词; +- `$`、`*`、`;`、`|` 等 shell 运算符没有特殊含义。 + +`defines` 的一个条目是一个值,不按此写法读取:`defines = ["NAME=\"text\""]` 传入的是 +一个词 `-DNAME="text"`。`ldflags`、`dialect_cxxflags` 与 `std-module-flags` 不在本节范围内。 + +`compile_commands.json` 与 `mcpp emit build-database` 在 `arguments` 中列出同样的词, +不经 shell 即可执行。 + +一个工程的首次规划中,若某个元素的词与 2026.9.17.1 之前的版本在同一宿主上传入的参数不同, +mcpp 以 `build/flag-words` 警告并给出两者。重复同一规划的构建不再重复该警告。 + #### `dependency_linkage` —— 静态还是动态由消费者决定 ```toml diff --git a/docs/zh/30-build-mcpp.md b/docs/zh/30-build-mcpp.md index c042d4d9e..014aa6759 100644 --- a/docs/zh/30-build-mcpp.md +++ b/docs/zh/30-build-mcpp.md @@ -50,11 +50,11 @@ mcpp build # 编译 + 运行 build.mcpp,然后构建工程 | 指令 | 作用 | |---|---| -| `mcpp:cxxflag=` | 给 C++ 编译追加 `` | -| `mcpp:cflag=` | 给 C 编译追加 `` | +| `mcpp:cxxflag=` | 给 C++ 编译追加 ``;`` 按[编译 flag 的写法](04-mcpp-toml.md#编译-flag-的写法-mcpp-20269171)读取(2026.9.17.1+) | +| `mcpp:cflag=` | 给 C 编译追加 ``,读法相同 | | `mcpp:link-lib=` | 链接 `-l` | | `mcpp:link-search=` | 增加库搜索目录(`-L`;相对路径按工程根目录解析) | -| `mcpp:cfg=` | 为 C 与 C++ 同时定义 `-D` | +| `mcpp:cfg=` | 为 C 与 C++ 同时定义 `-D`;`` 是一个值,与 `defines` 条目相同 | | `mcpp:generated=` | 把生成的源码加入构建。**相对路径在根工程按工程根解析,在依赖的 build.mcpp 里按 `MCPP_OUT_DIR` 解析** —— 两种角色都可能出现的包应发绝对路径(见下文) | | `mcpp:source=` *(0.0.100+)* | 把一份**既有**源文件选入构建(绝对路径,或相对包根)。下游效果与 `generated=` 相同;语义区别在于文件是程序*选中*的(tarball payload / vendored 源树)而非程序写出的——例如对大型源码包做 per-target 源选择 | | `mcpp:include-dir=` *(0.0.100+)* | 为本包自身 TU 增加一个**私有** include 目录(`-I`;绝对路径或相对包根,自动规范化)。取代过去 `cxxflag=-I` + `cflag=-I` 的双重裸发 | diff --git a/mcpp.toml b/mcpp.toml index 45889e15f..d6b414bb1 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.9.16.2" +version = "2026.9.17.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/buildmcpp/src/directives.cppm b/modules/buildmcpp/src/directives.cppm index a63e74d1e..44d97293a 100644 --- a/modules/buildmcpp/src/directives.cppm +++ b/modules/buildmcpp/src/directives.cppm @@ -674,7 +674,9 @@ std::string transformed(const Def& def, std::string_view raw, case Transform::LibFlag: return mcpp::toolchain::lib_flag_for(dial, raw); case Transform::LibSearchPath: return std::string(dial.libSearchPrefix) + abs_against(root, raw); - case Transform::DefinePrefix: return std::string(dial.definePrefix) + std::string(raw); + // A define is one word of a compile-flag list, whatever it contains. + case Transform::DefinePrefix: + return mcpp::manifest::flag_element(std::string(dial.definePrefix) + std::string(raw)); case Transform::AbsPath: return abs_against(root, raw); // Absolute on purpose: the link runs in the build directory, so a // relative script path resolves against the wrong root and lld diff --git a/modules/manifest/src/flag_words.cppm b/modules/manifest/src/flag_words.cppm new file mode 100644 index 000000000..50da8af58 --- /dev/null +++ b/modules/manifest/src/flag_words.cppm @@ -0,0 +1,211 @@ +// mcpp.manifest.flag_words — what an element of a compile-flag list means. +// +// A manifest states compile flags as lists of strings: `cflags`, `cxxflags` +// and `asmflags`, in `[build]`, in a target, in a glob or feature entry, in a +// `cfg` section, in an xpkg descriptor, and through a build program's +// `mcpp:cflag=` directive. Until 2026.9.17.1 the engine gave an element no +// meaning of its own. It pasted the element into a ninja rule, so the element +// meant whatever the host's command-line reader made of it: POSIX `sh` on +// Linux and macOS, the MSVCRT argument rules on Windows. The compile +// databases then re-read the same text with a third, hand-written reading +// (#655), and a `defines` value lost its quotes on the way to the shell. +// +// This module gives the element one meaning on every host. `flag_words` reads +// an element into the words the compiler receives, and every consumer of a +// flag list reads through it: the ninja writer quotes each word for its host, +// the compile databases list the words, and the assembler filter and the +// include-path normalisation look at words rather than at text. A value the +// engine itself inserts into such a list (a `defines` entry, a feature macro) +// is spelled with `flag_element`, so that it reads back as exactly one word. +// +// THE SYNTAX. It is the POSIX shell's word syntax without expansions, with one +// narrowing that keeps Windows paths intact: +// +// - unquoted spaces and tabs separate words; +// - `'...'` is literal up to the next `'`; +// - `"..."` is literal except that `\"` and `\\` stand for `"` and `\`; +// - outside quotes, a backslash before a space, a tab, `"`, `'` or `\` +// stands for that character, and any other backslash is literal; +// - quoted and unquoted pieces that touch form one word, so `""` is an +// empty word and `-DV="a b"` is the single word `-DV=a b`. +// +// `$`, `*`, `;` and the other shell operators have no meaning. The one +// spelling the published index relies on, libarchive's +// `-DPLATFORM_CONFIG_H=\"mcpp_libarchive_config.h\"`, reads as the word +// `-DPLATFORM_CONFIG_H="mcpp_libarchive_config.h"`, which is what both hosts +// passed before. pkg-config's `\ ` for a space in a path reads as a space. +// `C:\Users\x\include` keeps its backslashes, which POSIX `sh` would drop. +// +// `host_command_words` is the other direction's reader: the words a host +// makes of a command line the engine rendered for it. The build database +// needs it for command lines that are rendered text rather than lists (the +// global flag strings, the standard library module's command), and the +// upgrade diagnostic needs it to state what a previous release passed. + +export module mcpp.manifest.flag_words; + +import std; + +export namespace mcpp::manifest { + +// The words one element of a compile-flag list stands for. An element of +// spaces only stands for no word. An unterminated quote extends to the end of +// the element. +std::vector flag_words(std::string_view element); + +// The words of a whole list, in order. +std::vector flag_words(const std::vector& elements); + +// The spelling of one word as a list element: `flag_words(flag_element(w))` +// is `{w}` for every string `w`, the empty string included. A word that needs +// no quoting is returned unchanged, so a plain flag keeps its spelling. +std::string flag_element(std::string_view word); + +// The words a host makes of a command line: POSIX `sh` word splitting (quote +// removal and backslash escapes, no expansions) when `windows` is false, the +// MSVCRT argument rules when it is true. +std::vector host_command_words(std::string_view command, bool windows); + +} // namespace mcpp::manifest + +namespace mcpp::manifest { + +namespace { + +bool is_blank(char c) { return c == ' ' || c == '\t'; } + +bool escapable_outside_quotes(char c) { + return c == ' ' || c == '\t' || c == '"' || c == '\'' || c == '\\'; +} + +} // namespace + +std::vector flag_words(std::string_view s) { + std::vector out; + std::string word; + bool started = false; + auto flush = [&] { + if (started) out.push_back(std::move(word)); + word.clear(); + started = false; + }; + for (std::size_t i = 0; i < s.size(); ++i) { + const char c = s[i]; + if (is_blank(c)) { flush(); continue; } + started = true; + if (c == '\'') { + for (++i; i < s.size() && s[i] != '\''; ++i) word.push_back(s[i]); + continue; + } + if (c == '"') { + for (++i; i < s.size() && s[i] != '"'; ++i) { + if (s[i] == '\\' && i + 1 < s.size() + && (s[i + 1] == '"' || s[i + 1] == '\\')) { + ++i; + } + word.push_back(s[i]); + } + continue; + } + if (c == '\\' && i + 1 < s.size() && escapable_outside_quotes(s[i + 1])) { + word.push_back(s[++i]); + continue; + } + word.push_back(c); + } + flush(); + return out; +} + +std::vector flag_words(const std::vector& elements) { + std::vector out; + for (auto const& e : elements) + for (auto& w : flag_words(e)) out.push_back(std::move(w)); + return out; +} + +std::string flag_element(std::string_view word) { + const bool plain = !word.empty() + && word.find_first_of(" \t\"'\\") == std::string_view::npos; + if (plain) return std::string(word); + // Single quotes hold everything except `'`, which closes the region, is + // written as an escaped quote outside it, and reopens it. + std::string out = "'"; + for (char c : word) { + if (c == '\'') out += "'\\''"; + else out.push_back(c); + } + out += "'"; + return out; +} + +std::vector host_command_words(std::string_view s, bool windows) { + std::vector out; + std::string cur; + bool started = false; + auto flush = [&] { + if (started) out.push_back(std::move(cur)); + cur.clear(); + started = false; + }; + if (windows) { + bool quoted = false; + for (std::size_t i = 0; i < s.size(); ++i) { + const char c = s[i]; + if (c == '\\') { + std::size_t j = i; + while (j < s.size() && s[j] == '\\') ++j; + const std::size_t count = j - i; + started = true; + if (j < s.size() && s[j] == '"') { + cur.append(count / 2, '\\'); + if (count % 2 == 1) { cur.push_back('"'); i = j; } + else { i = j - 1; } + continue; + } + cur.append(count, '\\'); + i = j - 1; + continue; + } + if (c == '"') { quoted = !quoted; started = true; continue; } + if (!quoted && is_blank(c)) { flush(); continue; } + cur.push_back(c); + started = true; + } + flush(); + return out; + } + char quote = 0; + for (std::size_t i = 0; i < s.size(); ++i) { + const char c = s[i]; + if (quote == '\'') { + if (c == '\'') quote = 0; else cur.push_back(c); + continue; + } + if (quote == '"') { + if (c == '"') { quote = 0; continue; } + if (c == '\\' && i + 1 < s.size() + && (s[i + 1] == '"' || s[i + 1] == '\\' || s[i + 1] == '$' + || s[i + 1] == '`' || s[i + 1] == '\n')) { + if (s[i + 1] != '\n') cur.push_back(s[i + 1]); + ++i; + continue; + } + cur.push_back(c); + continue; + } + if (c == '\'' || c == '"') { quote = c; started = true; continue; } + if (c == '\\' && i + 1 < s.size()) { + if (s[i + 1] != '\n') { cur.push_back(s[i + 1]); started = true; } + ++i; + continue; + } + if (is_blank(c) || c == '\n') { flush(); continue; } + cur.push_back(c); + started = true; + } + flush(); + return out; +} + +} // namespace mcpp::manifest diff --git a/modules/manifest/src/manifest.cppm b/modules/manifest/src/manifest.cppm index 2960f3ef5..0a5aeb742 100644 --- a/modules/manifest/src/manifest.cppm +++ b/modules/manifest/src/manifest.cppm @@ -10,9 +10,11 @@ // mcpp.manifest.types shared data model (Manifest, Target, errors) // mcpp.manifest.toml mcpp.toml parsing (projects / packages on disk) // mcpp.manifest.xpkg xpkg .lua `mcpp = {}` segment (index descriptors) +// mcpp.manifest.flag_words the words an element of a compile-flag list stands for export module mcpp.manifest; export import mcpp.manifest.types; export import mcpp.manifest.toml; export import mcpp.manifest.xpkg; +export import mcpp.manifest.flag_words; diff --git a/modules/manifest/src/types.cppm b/modules/manifest/src/types.cppm index 68bec4f5a..c5e23f244 100644 --- a/modules/manifest/src/types.cppm +++ b/modules/manifest/src/types.cppm @@ -10,6 +10,7 @@ import mcpp.pm.dep_spec; // M5.x pm/ subsystem refactor: DependencySpec live import mcpp.pm.compat; // Legacy dependency-key compatibility helpers import mcpp.pm.index_spec; // IndexSpec for [indices] section import mcpp.platform; +import mcpp.manifest.flag_words; export namespace mcpp::manifest { @@ -2101,8 +2102,12 @@ std::optional cxx_layer_implementation_standard( // modules need them reachable; they live in the manifest namespace) ── export namespace mcpp::manifest { +// True when a compile-flag element holds a `-std` word. The element is read as +// words (mcpp.manifest.flag_words), so `"-O2 -std=c++20"` is caught as well. bool starts_with_std_flag(std::string_view flag) { - return flag == "-std" || flag.starts_with("-std="); + for (auto const& w : flag_words(flag)) + if (w == "-std" || w.starts_with("-std=")) return true; + return false; } bool is_basename(std::string_view value) { @@ -2225,8 +2230,8 @@ std::vector dialect_flags(const BuildConfig& bc) { out.push_back(f); }; for (auto& f : bc.dialectCxxflags) add(f); - for (auto& f : bc.cxxflags) - if (is_dialect_flag(f)) add(f); + for (auto& w : flag_words(bc.cxxflags)) + if (is_dialect_flag(w)) add(w); return out; } diff --git a/modules/manifest/tests/test_flag_words.cpp b/modules/manifest/tests/test_flag_words.cpp new file mode 100644 index 000000000..476bc7725 --- /dev/null +++ b/modules/manifest/tests/test_flag_words.cpp @@ -0,0 +1,100 @@ +#include + +import std; +import mcpp.manifest.flag_words; + +// SUBSYSTEM-LEVEL. The words an element of a compile-flag list stands for are +// part of what a manifest MEANS, so the syntax is stated here, with nothing +// else present. That the build and the compile databases both use it is +// stated in tests/unit (test_compile_commands.cpp). + +using mcpp::manifest::flag_element; +using mcpp::manifest::flag_words; +using mcpp::manifest::host_command_words; +using Words = std::vector; + +// Each row is a spelling found in a manifest or a descriptor and the words it +// stands for on every host. The first rows are the ones #655 measured. +TEST(FlagWords, TheSyntaxTable) { + const std::vector> rows{ + // libarchive's descriptor, the one quoted spelling the index publishes. + {R"(-DPLATFORM_CONFIG_H=\"mcpp_libarchive_config.h\")", + {R"(-DPLATFORM_CONFIG_H="mcpp_libarchive_config.h")"}}, + // Quotes inside a word are quoting, as in `sh`. + {R"(-DMID="mid")", {"-DMID=mid"}}, + {"-DSQ='sq'", {"-DSQ=sq"}}, + {R"(-DV="a b")", {"-DV=a b"}}, + {"'-DV=a b'", {"-DV=a b"}}, + // compat.lua's descriptor: one element, two words. + {"-include mcpp_lua_platform_config.h", {"-include", "mcpp_lua_platform_config.h"}}, + // pkg-config writes a space in a path as `\ `. + {R"(-I/opt/my\ dir/include)", {"-I/opt/my dir/include"}}, + // A backslash before an ordinary character is literal: Windows paths. + {R"(-IC:\Users\x\include)", {R"(-IC:\Users\x\include)"}}, + {R"("-IC:\Program Files\x")", {R"(-IC:\Program Files\x)"}}, + // Escapes a backslash can make outside and inside double quotes. + {R"(-DV=a\\b)", {R"(-DV=a\b)"}}, + {R"("a\"b\\c\d")", {R"(a"b\c\d)"}}, + {R"(-DV=\'c\')", {"-DV='c'"}}, + // Single quotes hold backslashes and double quotes literally. + {R"('a\"b')", {R"(a\"b)"}}, + // No expansions: `$` and the shell operators are ordinary characters. + {"-DV=$HOME;x|y", {"-DV=$HOME;x|y"}}, + // Blanks separate words; tabs count; runs collapse. + {" -O2\t -g ", {"-O2", "-g"}}, + // Empty and blank elements. + {"", {}}, + {" ", {}}, + {R"("")", {""}}, + {"''", {""}}, + {R"(-DV="")", {"-DV="}}, + // An unterminated quote extends to the end of the element. + {R"(-DV="a b)", {"-DV=a b"}}, + // A trailing backslash has nothing to escape. + {R"(-DV=a\)", {R"(-DV=a\)"}}, + }; + for (auto const& [element, words] : rows) + EXPECT_EQ(flag_words(element), words) << "element: " << element; +} + +TEST(FlagWords, AListIsTheConcatenationOfItsElements) { + const std::vector list{"-include x.h", R"(-DV="a b")", "", "-O2"}; + EXPECT_EQ(flag_words(list), (Words{"-include", "x.h", "-DV=a b", "-O2"})); +} + +TEST(FlagWords, APlainWordIsItsOwnElement) { + for (std::string_view w : {"-O2", "-DV=1", "-IC:/x", "/std:c++latest", "-Wl,-z,defs", "$x"}) + EXPECT_EQ(flag_element(w), w); +} + +// The property the engine relies on when it inserts a value into a flag list: +// the element it writes reads back as exactly that one word. +TEST(FlagWords, AnElementReadsBackAsItsWord) { + Words words{"", " ", "a b", "'", "\"", "\\", "a\\", "\\\\srv\\share", "it's", + "-DV=\"def\"", "-DV='c'", "x\ty", "a\"b'c\\d e"}; + // A deterministic walk over the characters the syntax gives meaning to. + const std::string alphabet = "ab \t\"'\\$"; + std::uint32_t state = 655; + for (int n = 0; n < 2000; ++n) { + std::string w; + const int length = static_cast(state % 9); + for (int i = 0; i < length; ++i) { + state = state * 1664525u + 1013904223u; + w.push_back(alphabet[(state >> 16) % alphabet.size()]); + } + state = state * 1664525u + 1013904223u; + words.push_back(std::move(w)); + } + for (auto const& w : words) + EXPECT_EQ(flag_words(flag_element(w)), Words{w}) << "word: [" << w << "]"; +} + +TEST(HostCommandWords, PosixUndoesShellQuoting) { + EXPECT_EQ(host_command_words(R"(a 'b c' "d\"e" f\ g "h\x" -DV="mid")", false), + (Words{"a", "b c", "d\"e", "f g", "h\\x", "-DV=mid"})); +} + +TEST(HostCommandWords, WindowsFollowsTheRuntimeRules) { + EXPECT_EQ(host_command_words(R"(a "b c" "d\"e" C:\x\ "y\\" z\\\"w 'q')", true), + (Words{"a", "b c", "d\"e", "C:\\x\\", "y\\", "z\\\"w", "'q'"})); +} diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index d2e414c7f..f0532680f 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.9.16.2"; +inline constexpr std::string_view MCPP_VERSION = "2026.9.17.1"; } // namespace mcpp diff --git a/src/build/build_database.cppm b/src/build/build_database.cppm index 5b7a9a50b..e905909e1 100644 --- a/src/build/build_database.cppm +++ b/src/build/build_database.cppm @@ -94,6 +94,8 @@ std::string toolchain_id(const mcpp::toolchain::Toolchain& tc, // Splits a command string mcpp rendered for the host shell into words, undoing // its quoting: POSIX `sh` rules, or the Microsoft C runtime's rules on Windows. +// The reader is mcpp::manifest::host_command_words; this name is kept for the +// standard library units' recovery below. std::vector split_command_words(std::string_view command, bool windows); // The working directory and argument vector of the command in `commands` whose @@ -307,67 +309,7 @@ std::string toolchain_id(const mcpp::toolchain::Toolchain& tc, } std::vector split_command_words(std::string_view s, bool windows) { - std::vector out; - std::string cur; - bool started = false; - auto flush = [&] { - if (started) out.push_back(std::move(cur)); - cur.clear(); - started = false; - }; - if (windows) { - bool quoted = false; - for (std::size_t i = 0; i < s.size(); ++i) { - const char c = s[i]; - if (c == '\\') { - std::size_t j = i; - while (j < s.size() && s[j] == '\\') ++j; - const std::size_t count = j - i; - started = true; - if (j < s.size() && s[j] == '"') { - cur.append(count / 2, '\\'); - if (count % 2 == 1) { cur.push_back('"'); i = j; } - else { i = j - 1; } - continue; - } - cur.append(count, '\\'); - i = j - 1; - continue; - } - if (c == '"') { quoted = !quoted; started = true; continue; } - if (!quoted && (c == ' ' || c == '\t')) { flush(); continue; } - cur.push_back(c); - started = true; - } - flush(); - return out; - } - char quote = 0; - for (std::size_t i = 0; i < s.size(); ++i) { - const char c = s[i]; - if (quote == '\'') { - if (c == '\'') quote = 0; else cur.push_back(c); - continue; - } - if (quote == '"') { - if (c == '"') { quote = 0; continue; } - if (c == '\\' && i + 1 < s.size() - && (s[i + 1] == '"' || s[i + 1] == '\\' || s[i + 1] == '$' - || s[i + 1] == '`')) { - cur.push_back(s[++i]); - continue; - } - cur.push_back(c); - continue; - } - if (c == '\'' || c == '"') { quote = c; started = true; continue; } - if (c == '\\' && i + 1 < s.size()) { cur.push_back(s[++i]); started = true; continue; } - if (c == ' ' || c == '\t' || c == '\n') { flush(); continue; } - cur.push_back(c); - started = true; - } - flush(); - return out; + return mcpp::manifest::host_command_words(s, windows); } std::optional recover_invocation(const std::vector& commands, diff --git a/src/build/compile_commands.cppm b/src/build/compile_commands.cppm index e6ae9be7e..8bc73878d 100644 --- a/src/build/compile_commands.cppm +++ b/src/build/compile_commands.cppm @@ -20,16 +20,32 @@ import mcpp.build.plan; import mcpp.build.flags; import mcpp.libs.json; import mcpp.platform.fs; +import mcpp.platform; +import mcpp.manifest.flag_words; export namespace mcpp::build { -// Split one flag string into CDB `arguments` tokens. +// The words the compiler receives from one flag string the engine rendered +// for a ninja `command =` line: ninja's `$` escapes are undone, then the +// host's command-line reader splits the text (POSIX `sh`, or the MSVCRT rules +// on Windows; mcpp::manifest::host_command_words). +// +// It serves the rendered strings only, the global `$cflags`/`$cxxflags`/ +// `$asmflags`. A unit's own flag lists are never rendered and re-read: the +// databases list their words directly (mcpp::manifest::flag_words). // // Exported because it IS the contract: what a consumer receives in // `arguments` is decided here, and that contract needs pinning by test // rather than by inspection of a whole generated document. std::vector split_flags(std::string_view s); +// The flag list an assembly unit's edge carries: the -D/-U/-I words of the +// unit's C flags (feature defines land there), then its per-glob asmflags. +// NASM shares the GNU -D/-U/-I spelling (and 2.14 and later insert a missing +// -I path separator itself), so one filter serves both assembler rules, and +// the compile database lists the same list for a GAS unit. +std::vector unit_asm_flags(const CompileUnit& cu); + // ONE TRANSLATION UNIT AS THE COMPILER IS INVOKED FOR IT. // // The compile database and the build database (mcpp.build.build_database) both @@ -97,81 +113,29 @@ namespace { } // namespace -// Split one flag string into CDB `arguments` tokens. -// -// Three things happen here, and the ORDER between them is the whole point. -// -// 1. Ninja escapes are undone (`$ ` -> space, `$:` -> `:`, `$$` -> `$`). -// `flags.cppm::escape_path` adds them so a path survives embedding in a -// ninja rule string. A CDB consumer execs `arguments` literally -- no -// ninja -- so `C$:\Users\...` would be a path that does not exist. -// -// 2. Shell quoting is removed. `flags.cppm::shell_quote_arg` wraps a token -// whose characters would split or alter a word in `sh -c` / cmd.exe -- -// and its trigger set contains the BACKSLASH, so on Windows every -// path-bearing flag is quoted. That quoting is correct for ninja and -// wrong for a consumer that never invokes a shell: the quotes arrive as -// part of the filename. -// -// 3. Tokens split on spaces -- but not on a space that came from `$ `, and -// not on one inside quotes. -// -// Getting (3) wrong is what shipped: quoting was ignored entirely, so a -// quoted path containing a space was cut in half at that space. Measured on -// a real build under `/tmp/.../my project`: +// The order is the one ninja and the host apply: ninja replaces `$ `, `$:` and +// `$$` while it builds the command line, and only then does the host read the +// line into words. flags.cppm escapes for ninja before it quotes for the host, +// so a quoted path's space arrives here as `$ ` inside the quotes, and it is a +// space again before the reader sees the quotes. // -// '-fmodule-file=std=/tmp/.../my project/.../std.pcm <- closing quote gone -// '-fprebuilt-module-path=/tmp/.../my project/.../pcm.cache' -// -// One argument became two, one of them opening a quote that never closes. -// The comment that used to live here stated the right principle -- consumers -// exec literally, so escapes must be undone -- and implemented half of it. -// -// A quote in the MIDDLE of a token is data: `-DGREETING="hi"` keeps its -// inner quotes, or the define changes meaning. Only a quote that OPENS a -// region is quoting. +// The reader is the host's own. The version before #655 split on spaces and +// removed a quote only when it opened a token, which is neither host's rule: +// `\"` stayed escaped and a quote inside a word stayed in the word, so the +// database listed arguments the compiler never received. std::vector split_flags(std::string_view s) { - std::vector out; - std::string token; - bool started = false; // token has begun (so "" is a real empty arg) - char quote = 0; // active quote char, 0 when outside - - auto flush = [&] { - if (started) { out.push_back(std::move(token)); token.clear(); } - started = false; - }; - + std::string unescaped; + unescaped.reserve(s.size()); for (std::size_t i = 0; i < s.size(); ++i) { - char c = s[i]; - - // Ninja escapes first, and INSIDE quotes too -- a quoted path's space - // arrives as `$ ` because flags.cppm escapes before it quotes. - if (c == '$' && i + 1 < s.size()) { - char nc = s[i + 1]; - if (nc == ' ' || nc == ':' || nc == '$') { - token.push_back(nc); - started = true; - ++i; - continue; - } - } - - if (quote) { - if (c == quote) { quote = 0; continue; } // closes the region - token.push_back(c); + if (s[i] == '$' && i + 1 < s.size() + && (s[i + 1] == ' ' || s[i + 1] == ':' || s[i + 1] == '$')) { + unescaped.push_back(s[++i]); continue; } - - // Opens a region only at a token boundary; elsewhere it is data. - if ((c == '"' || c == '\'') && !started) { quote = c; started = true; continue; } - - if (c == ' ') { flush(); continue; } - - token.push_back(c); - started = true; + unescaped.push_back(s[i]); } - flush(); - return out; + return mcpp::manifest::host_command_words(unescaped, + mcpp::platform::is_windows); } namespace { @@ -208,14 +172,11 @@ std::vector local_include_args(const CompileUnit& cu) { return args; } +// The unit's own flag lists as words. The edge writes the same words, each +// quoted for the host (ninja_backend.cppm::join_flags), so the database lists +// what the compiler receives without reading a command line back. std::vector package_flag_args(const CompileUnit& cu, bool isCSource) { - std::string joined; - auto const& flags = isCSource ? cu.packageCflags : cu.packageCxxflags; - for (auto const& flag : flags) { - joined += ' '; - joined += flag; - } - return split_flags(joined); + return mcpp::manifest::flag_words(isCSource ? cu.packageCflags : cu.packageCxxflags); } void sort_entries_by_file(nlohmann::json& entries) { @@ -235,6 +196,16 @@ CompileCommandsWriteError write_error(std::string message) { } // namespace +std::vector unit_asm_flags(const CompileUnit& cu) { + std::vector out; + for (auto& w : mcpp::manifest::flag_words(cu.packageCflags)) { + if (w.starts_with("-D") || w.starts_with("-U") || w.starts_with("-I")) + out.push_back(mcpp::manifest::flag_element(w)); + } + out.insert(out.end(), cu.packageAsmflags.begin(), cu.packageAsmflags.end()); + return out; +} + std::vector unit_invocations(const BuildPlan& plan, const CompileFlags& flags) { std::vector out; @@ -266,7 +237,8 @@ std::vector unit_invocations(const BuildPlan& plan, inv.arguments.push_back(std::move(f)); for (auto& f : split_flags(flagStr)) inv.arguments.push_back(std::move(f)); - for (auto& f : package_flag_args(cu, isCSource)) + for (auto& f : isGasSource ? mcpp::manifest::flag_words(unit_asm_flags(cu)) + : package_flag_args(cu, isCSource)) inv.arguments.push_back(std::move(f)); inv.arguments.push_back("-c"); inv.arguments.push_back(inv.file); diff --git a/src/build/flags.cppm b/src/build/flags.cppm index 7e656f686..18f6bff55 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -19,6 +19,7 @@ import mcpp.diag; import mcpp.freestanding.target; import mcpp.freestanding.linkline; import mcpp.manifest.types; +import mcpp.manifest.flag_words; import mcpp.modgraph.scanner; import mcpp.platform; import mcpp.platform.runtime_search; @@ -219,10 +220,22 @@ std::string atomic_link_flag(const std::vector& linkDirs, // tokens that actually contain whitespace or a shell-significant character // are quoted — plain framework flags (`-std=c++23`, `-O2`, `-I/abs/path`) // come back unchanged, byte-for-byte. POSIX: wrap in single quotes (embedded -// `'` escaped as `'\''`). Windows: wrap in double quotes (embedded `"` -// escaped as `\"`) — cmd.exe/CreateProcess argv convention. +// `'` escaped as `'\''`). Windows: wrap in double quotes under the MSVCRT +// argument rules (an embedded `"` becomes `\"`, and the backslashes before it +// or before the closing quote are doubled). The inverse is +// `mcpp::manifest::host_command_words`: reading the result back yields `arg`. std::string shell_quote_arg(std::string_view arg); +// One word of a compile-flag list as it is written on a ninja `command =` +// line: quoted for the host's command-line reader, then `$` doubled so that +// ninja hands the reader a literal `$`. A plain word comes back unchanged. +std::string ninja_command_word(std::string_view word); + +// The ninja text of a compile-flag list: every element read into words +// (`mcpp::manifest::flag_words`), every word written by `ninja_command_word`, +// each preceded by a space. The compile databases list the same words. +std::string ninja_flag_list(const std::vector& elements); + // Ninja's own escaping for a value that will sit on a `command = ` line: // ` `, `$` and `:` get a leading `$`. Exported because it is needed WITH // shell_quote_arg, not instead of it — quoting stops the SHELL from splitting @@ -381,13 +394,24 @@ std::string shell_quote_arg(std::string_view arg) { return std::string(arg); if constexpr (mcpp::platform::is_windows) { - // cmd.exe / CreateProcess argv convention: wrap in double quotes, - // escape embedded `"` as `\"`. + // MSVCRT argument rules: inside double quotes a run of backslashes is + // literal unless a `"` follows it, in which case every backslash of + // the run is doubled and the quote is escaped. The closing quote is + // such a `"`, so a trailing run is doubled as well; `C:\dir\` would + // otherwise escape the quote that ends the word. std::string out = "\""; + std::size_t backslashes = 0; for (char c : arg) { - if (c == '"') out += "\\\""; - else out.push_back(c); + if (c == '\\') { ++backslashes; continue; } + if (c == '"') { + out.append(backslashes * 2 + 1, '\\'); + } else { + out.append(backslashes, '\\'); + } + out.push_back(c); + backslashes = 0; } + out.append(backslashes * 2, '\\'); out += "\""; return out; } else { @@ -404,6 +428,27 @@ std::string shell_quote_arg(std::string_view arg) { } } +std::string ninja_command_word(std::string_view word) { + // An empty word is still a word. shell_quote_arg leaves it empty, which is + // right for its callers that append optional pieces, and wrong here. + if (word.empty()) return mcpp::platform::is_windows ? "\"\"" : "''"; + std::string out; + for (char c : shell_quote_arg(word)) { + if (c == '$') out.push_back('$'); + out.push_back(c); + } + return out; +} + +std::string ninja_flag_list(const std::vector& elements) { + std::string out; + for (auto const& word : mcpp::manifest::flag_words(elements)) { + out += ' '; + out += ninja_command_word(word); + } + return out; +} + std::string render_link_intent_flags( const mcpp::manifest::LinkIntent& intent, LinkIntentFlavor flavor) { diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 9445d4932..ff7c25f17 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -195,7 +195,25 @@ std::string local_include_flags(const CompileUnit& cu, return flags; } -std::string join_flags(const std::vector& flags) { +// A unit's compile-flag list on its edge. Each element is read into words and +// each word is quoted for the host (flags.cppm::ninja_flag_list), so what the +// compiler receives does not depend on whether ninja hands the line to `sh` or +// to CreateProcess, and the compile databases list the same words without +// re-reading any command line (#655). The two shapes that needed the verbatim +// paste below keep their meaning under the word reading: a packed element +// (compat.lua's `-include mcpp_lua_platform_config.h`) is two words, and a +// define with a space arrives as one quoted word from `flag_element`. +std::string join_compile_flags(const std::vector& flags) { + return ninja_flag_list(flags); +} + +// A link unit's flag list. Its elements are rendered text, not manifest +// elements: the engine writes them already quoted and escaped for ninja and +// the host (`-Wl,-rpath,'$$ORIGIN'`), so they are pasted as they are. Reading +// them as words would quote the quotes and double the `$$` (measured on the +// first implementation: the edge carried `'-Wl,-rpath,$$$$ORIGIN'` and the +// closure check refused the program with `libwrap.so not found`). +std::string join_link_flags(const std::vector& flags) { // mcpp#234: a manifest `defines = ["T=long long"]` arrives as the single // element `-DT=long long` (pushed whole by apply_glob_flags) — a space // that is genuinely PART of one argv token. Joining with a bare space let @@ -428,20 +446,6 @@ bool is_scan_exempt(const mcpp::build::CompileUnit& cu) { return mcpp::is_scan_exempt(cu.kind); } -// Per-unit flags an assembler can take: the -D/-U/-I subset of the unit's C -// flags (feature defines land there). NASM shares the GNU -D/-U/-I spelling -// (and ≥2.14 inserts a missing -I path separator itself), so one filter -// serves both asm rules. Explicit per-glob asmflags (G4) append after the -// filtered subset — author-directed flags win. -std::vector asm_unit_flags(const CompileUnit& cu) { - std::vector out; - for (auto& f : cu.packageCflags) { - if (f.starts_with("-D") || f.starts_with("-U") || f.starts_with("-I")) - out.push_back(f); - } - out.insert(out.end(), cu.packageAsmflags.begin(), cu.packageAsmflags.end()); - return out; -} std::string ltrim_copy(std::string_view s) { while (!s.empty() && std::isspace(static_cast(s.front()))) @@ -1857,7 +1861,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { : escape_ninja_path(cu.object))); if (auto includes = local_include_flags(cu, dial); !includes.empty()) append(std::format(" local_includes ={}\n", includes)); - if (auto flags = join_flags(cu.packageCxxflags); !flags.empty()) + if (auto flags = join_compile_flags(cu.packageCxxflags); !flags.empty()) append(std::format(" unit_cxxflags ={}\n", flags)); // The scan has the same suffix-recognition problem as the compile: // a driver that does not know `.ixx` hands it to the linker and @@ -1974,7 +1978,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { e += std::format(" sched_cap = {}\n", plan.scheduleCompilerCap); if (auto inc = local_include_flags(cu, dial); !inc.empty()) e += " local_includes =" + inc + "\n"; - if (auto fl = join_flags(cu.packageCxxflags); !fl.empty()) + if (auto fl = join_compile_flags(cu.packageCxxflags); !fl.empty()) e += " unit_cxxflags =" + fl + "\n"; append(std::move(e)); // The join. THE SOURCE IS AN INPUT, and the BMI is only an @@ -2039,7 +2043,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { e += "\n dyndep = " + it->second + "\n"; if (auto inc = local_include_flags(cu, dial); !inc.empty()) e += " local_includes =" + inc + "\n"; - if (auto fl = join_flags(cu.packageCxxflags); !fl.empty()) + if (auto fl = join_compile_flags(cu.packageCxxflags); !fl.empty()) e += " unit_cxxflags =" + fl + "\n"; append(std::move(e)); }; @@ -2076,13 +2080,13 @@ std::string emit_ninja_string(const BuildPlan& plan) { if (auto includes = local_include_flags(cu, dial); !includes.empty()) out_line += " local_includes =" + includes + "\n"; if (is_gas_source(cu) || is_nasm_source(cu)) { - if (auto flags = join_flags(asm_unit_flags(cu)); !flags.empty()) + if (auto flags = join_compile_flags(unit_asm_flags(cu)); !flags.empty()) out_line += " unit_asmflags =" + flags + "\n"; } else if (is_c_source(cu)) { - if (auto flags = join_flags(cu.packageCflags); !flags.empty()) + if (auto flags = join_compile_flags(cu.packageCflags); !flags.empty()) out_line += " unit_cflags =" + flags + "\n"; } else { - if (auto flags = join_flags(cu.packageCxxflags); !flags.empty()) + if (auto flags = join_compile_flags(cu.packageCxxflags); !flags.empty()) out_line += " unit_cxxflags =" + flags + "\n"; } append(std::move(out_line)); @@ -2128,13 +2132,13 @@ std::string emit_ninja_string(const BuildPlan& plan) { if (auto includes = local_include_flags(cu, dial); !includes.empty()) out_line += " local_includes =" + includes + "\n"; if (is_gas_source(cu) || is_nasm_source(cu)) { - if (auto flags = join_flags(asm_unit_flags(cu)); !flags.empty()) + if (auto flags = join_compile_flags(unit_asm_flags(cu)); !flags.empty()) out_line += " unit_asmflags =" + flags + "\n"; } else if (is_c_source(cu)) { - if (auto flags = join_flags(cu.packageCflags); !flags.empty()) + if (auto flags = join_compile_flags(cu.packageCflags); !flags.empty()) out_line += " unit_cflags =" + flags + "\n"; } else { - if (auto flags = join_flags(cu.packageCxxflags); !flags.empty()) + if (auto flags = join_compile_flags(cu.packageCxxflags); !flags.empty()) out_line += " unit_cxxflags =" + flags + "\n"; } // Clang needs $bmi_out to emit -fmodule-output=$bmi_out @@ -2359,12 +2363,12 @@ std::string emit_ninja_string(const BuildPlan& plan) { // `$ORIGIN` in DT_RPATH and an artifact loaded a different build // of libX11 than it was linked against. mcpp::build::link_line::UnitTail tail; - tail.dependencies = join_flags(lu.linkFlags); + tail.dependencies = join_link_flags(lu.linkFlags); // #618: this executable's own subsystem and entry. Rendered here // rather than carried in `linkFlags`, because the spelling depends // on `sepLinker`, which only this emitter knows. if (lu.kind == LinkUnit::Binary) - tail.dependencies += join_flags(windows_executable_link_flags( + tail.dependencies += join_link_flags(windows_executable_link_flags( plan, sepLinker, lu.windowsSubsystem, lu.windowsEntry)); // mcpp#426: a link unit with no C++ in it takes only the part of // the contract that is not a statement about the C++ runtime. diff --git a/src/build/plan.cppm b/src/build/plan.cppm index 432630726..b11552033 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -2284,8 +2284,9 @@ make_plan(const mcpp::manifest::Manifest& manifest, for (auto& cu : plan.compileUnits) { if (cu.source != main_cu.source) continue; for (auto const& d : t.defines) { - cu.packageCflags.push_back("-D" + d); - cu.packageCxxflags.push_back("-D" + d); + const auto element = mcpp::manifest::flag_element("-D" + d); + cu.packageCflags.push_back(element); + cu.packageCxxflags.push_back(element); } for (auto const& f : t.cflags) cu.packageCflags.push_back(f); for (auto const& f : t.cxxflags) cu.packageCxxflags.push_back(f); diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 1678eabe8..6439566aa 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -546,12 +546,116 @@ export void merge_conditional_config(mcpp::manifest::Manifest& m, // actually read. // // Idempotent: clearing the vector after folding makes repeated calls harmless. +// ── An element whose words changed in 2026.9.17.1 (#655) ───────────────────── +// +// A compile-flag element used to reach the compiler as its host's command-line +// reader made it (POSIX `sh`, or the MSVCRT rules), after ninja had replaced +// `$` sequences, with a `-D` element containing a space quoted whole (#234). +// It now reaches the compiler as `flag_words` reads it, and a `defines` value +// is one word. Most spellings mean the same under both readings; the ones that +// do not are told what the compiler receives now and what it received before. +// The previous reading is modelled on quote removal only: `sh` expansions +// (`$VAR`, globs) are not reproduced. +// +// WHEN IT IS SAID. The notes are collected while manifests load and released +// where the output directory is decided, only if that directory has no +// build.ninja yet. The fingerprint names the mcpp version and every flag, so +// that is the first plan after an upgrade, after a flag was edited, or in a +// fresh checkout. A build that repeats a plan says nothing, so a manifest that +// is already spelled for the new reading is not warned about on every run. +std::vector>& pending_flag_words_notes() { + static std::vector> notes; + return notes; +} +std::vector previous_release_words(std::string element, bool define) { + if (define) element = "-D" + element; + if ((element.starts_with("-D") || element.starts_with("/D")) + && element.find(' ') != std::string::npos) + element = mcpp::build::shell_quote_arg(element); + std::string line; + for (std::size_t i = 0; i < element.size(); ++i) { + if (element[i] != '$' || i + 1 == element.size()) { line.push_back(element[i]); continue; } + const char n = element[i + 1]; + if (n == '$' || n == ' ' || n == ':') { line.push_back(n); ++i; continue; } + // A ninja variable reference: `${name}` or `$name`, empty on a compile edge. + std::size_t j = i + 1; + if (n == '{') { + while (j < element.size() && element[j] != '}') ++j; + } else { + while (j + 1 < element.size() + && (std::isalnum(static_cast(element[j + 1])) + || element[j + 1] == '_' || element[j + 1] == '-')) + ++j; + } + i = j; + } + return mcpp::manifest::host_command_words(line, mcpp::platform::is_windows); +} + +void report_flag_words_changes(const mcpp::manifest::Manifest& m) { + auto check = [&](std::string_view where, const std::vector& list, + bool define) { + for (auto const& e : list) { + auto now = define ? std::vector{"-D" + e} + : mcpp::manifest::flag_words(e); + auto before = previous_release_words(e, define); + if (now == before) continue; + auto show = [](const std::vector& words) { + std::string out = "["; + for (auto const& w : words) + out += std::format("{}'{}'", out.size() > 1 ? ", " : "", w); + return out + "]"; + }; + auto note = std::pair{std::format( + "{}: {} element '{}' reaches the compiler as {}; mcpp before " + "2026.9.17.1 passed {} on this host", + m.package.name.empty() ? std::string("(root)") : m.package.name, + where, e, show(now), show(before)), + std::string( + "a compile-flag element is read by one syntax on every host, and a " + "`defines` entry is one value (docs/04-mcpp-toml.md, " + "\"Compile-flag syntax\"); spell the element so that it reads as the " + "words meant")}; + auto& notes = pending_flag_words_notes(); + if (std::ranges::find(notes, note) == notes.end()) notes.push_back(std::move(note)); + } + }; + auto const& bc = m.buildConfig; + check("[build] cflags", bc.cflags, false); + check("[build] cxxflags", bc.cxxflags, false); + check("[build] defines", bc.defines, true); + for (auto const& gf : bc.globFlags) { + check("flags cflags", gf.cflags, false); + check("flags cxxflags", gf.cxxflags, false); + check("flags asmflags", gf.asmflags, false); + check("flags defines", gf.defines, true); + } + for (auto const& [feature, defines] : bc.featureDefines) + check(std::format("features.{} defines", feature), defines, true); + for (auto const& [feature, globs] : bc.featureFlags) { + for (auto const& gf : globs) { + check(std::format("features.{} cflags", feature), gf.cflags, false); + check(std::format("features.{} cxxflags", feature), gf.cxxflags, false); + check(std::format("features.{} asmflags", feature), gf.asmflags, false); + check(std::format("features.{} defines", feature), gf.defines, true); + } + } + for (auto const& t : m.targets) { + check(std::format("targets.{} cflags", t.name), t.cflags, false); + check(std::format("targets.{} cxxflags", t.name), t.cxxflags, false); + check(std::format("targets.{} defines", t.name), t.defines, true); + } +} + // Both `cflags` and `cxxflags` get the macro; assembly units pick it up for // free via the -D/-U/-I subset the ninja backend filters out of packageCflags. +// A define is a value, so it enters the flag list as one word +// (`flag_element`): `N="x"` reaches the compiler as `-DN="x"` on every host. void fold_build_defines_into_flags(mcpp::manifest::BuildConfig& bc) { for (auto const& d : bc.defines) { - bc.cflags.push_back("-D" + d); - bc.cxxflags.push_back("-D" + d); + const auto element = mcpp::manifest::flag_element("-D" + d); + bc.cflags.push_back(element); + bc.cxxflags.push_back(element); } bc.defines.clear(); } @@ -1846,6 +1950,7 @@ prepare_build(bool print_fingerprint, // run tier and a `mcpp run` that did not would be the same defect twice. const ToolPurpose toolPurpose = overrides.will_run ? ToolPurpose::Run : ToolPurpose::Build; + pending_flag_words_notes().clear(); // A refusal decided early and released late. `host_can_serve` answers // "does a payload on this machine produce this target", which is knowable @@ -3227,6 +3332,7 @@ prepare_build(bool print_fingerprint, // `[build].defines` must reach the scanner (P1689) and the compile edge, // and must participate in the fingerprint. Fold before dependency // resolution / fingerprinting. + report_flag_words_changes(*m); fold_build_defines_into_flags(m->buildConfig); // ORIGIN, RESOLVED ONCE. @@ -5861,6 +5967,7 @@ prepare_build(bool print_fingerprint, merge_conditional_config(*manifest, cfgCtx()); } + report_flag_words_changes(*manifest); fold_build_defines_into_flags(manifest->buildConfig); // The root's `abi.threads` reaches a dependency's C translation units // here; its C++ units already receive it through the dialect flag set. @@ -7882,6 +7989,7 @@ prepare_build(bool print_fingerprint, merge_conditional_config(*dep_manifest, cfgCtx()); } + report_flag_words_changes(*dep_manifest); fold_build_defines_into_flags(dep_manifest->buildConfig); // The root's `abi.threads` reaches this dependency's C translation // units here, as it does for a version dependency. @@ -8785,7 +8893,7 @@ prepare_build(bool print_fingerprint, if (auto it = pkg.manifest.buildConfig.featureDefines.find(f); it != pkg.manifest.buildConfig.featureDefines.end()) for (auto& d : it->second) { - auto fdef = "-D" + d; + auto fdef = mcpp::manifest::flag_element("-D" + d); pkg.manifest.buildConfig.cflags.push_back(fdef); pkg.manifest.buildConfig.cxxflags.push_back(fdef); pkg.privateBuild.cflags.push_back(fdef); @@ -11792,8 +11900,8 @@ prepare_build(bool print_fingerprint, // std", which the scan graph holds and does not expose in that shape. // Recorded here so the next person meets the reason and not the gap. for (auto const& pkg : std::span{packages}.first(1)) { - auto missing = mcpp::manifest::dialect_flags_missing_from_prebuild( - pkg.manifest.buildConfig.cxxflags, prebuilt); + const auto words = mcpp::manifest::flag_words(pkg.manifest.buildConfig.cxxflags); + auto missing = mcpp::manifest::dialect_flags_missing_from_prebuild(words, prebuilt); if (missing.empty()) continue; std::string list; for (auto const& f : missing) { @@ -12007,8 +12115,10 @@ prepare_build(bool print_fingerprint, if (flags.find(q) == std::string::npos) flags += q; } } - for (auto& f : targetSideUsage.cxxflags) - flags += " " + mcpp::xlings::shq(f); + // The same words the package's own units receive from this list + // (mcpp.manifest.flag_words), one quoted word each. + for (auto& w : mcpp::manifest::flag_words(targetSideUsage.cxxflags)) + flags += " " + mcpp::xlings::shq(w); tc->stdModuleFlags = flags; break; } @@ -12284,6 +12394,13 @@ prepare_build(bool print_fingerprint, ctx.cacheMode = cacheMode; ctx.projectRoot= *root; ctx.outputDir = target_dir(*tc, fp, workRoot); + { + std::error_code ec; + const bool firstPlan = !std::filesystem::exists(ctx.outputDir / "build.ninja", ec); + for (auto const& [what, hint] : pending_flag_words_notes()) + if (firstPlan) mcpp::diag::warning("build/flag-words", what, hint); + pending_flag_words_notes().clear(); + } ctx.stdBmi = stdBmiPath; ctx.stdObject = stdObjectPath; ctx.stdModule = std::move(describedStdModule); diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index bafecfa69..16a9be18f 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -101,6 +101,10 @@ ScanResult scan_package(const std::filesystem::path& root, // Called at every point where per-unit flag vectors are attached (the // scanner here; plan.cppm for a target's entry unit; flags.cppm for the // manifest-global [build] include_dirs). +// +// It reads the list as words (mcpp.manifest.flag_words) and writes it back one +// word per element, so `-isystem hdr` in one element is the separated spelling +// and a root that contains a space stays inside its word. void normalize_include_flags(const std::filesystem::path& root, std::vector& flags); @@ -723,21 +727,24 @@ void normalize_include_flags(const std::filesystem::path& root, static constexpr std::string_view kIncPrefixes[] = {"-I", "-iquote", "-isystem", "-idirafter", "-iprefix", "-L"}; - for (std::size_t i = 0; i < flags.size(); ++i) { + auto words = mcpp::manifest::flag_words(flags); + for (std::size_t i = 0; i < words.size(); ++i) { for (auto pre : kIncPrefixes) { - if (flags[i] == pre && i + 1 < flags.size()) { // separated - rewrite_rel(flags[i + 1], root); + if (words[i] == pre && i + 1 < words.size()) { // separated + rewrite_rel(words[i + 1], root); ++i; break; } - if (flags[i].size() > pre.size() && flags[i].starts_with(pre)) { // joined - std::string tail = flags[i].substr(pre.size()); + if (words[i].size() > pre.size() && words[i].starts_with(pre)) { // joined + std::string tail = words[i].substr(pre.size()); std::string abs = rewrite_rel_copy(tail, root); - if (abs != tail) flags[i] = std::string(pre) + abs; + if (abs != tail) words[i] = std::string(pre) + abs; break; } } } + flags.clear(); + for (auto const& w : words) flags.push_back(mcpp::manifest::flag_element(w)); } std::expected scan_file(const std::filesystem::path& file, @@ -1206,8 +1213,9 @@ void scan_one_into(ScanResult& result, // defines reach asm units too — via the -D subset the backend // filters out of packageCflags (no third copy needed here). for (auto const& d : gf.defines) { - u.packageCflags.push_back("-D" + d); - u.packageCxxflags.push_back("-D" + d); + const auto element = mcpp::manifest::flag_element("-D" + d); + u.packageCflags.push_back(element); + u.packageCxxflags.push_back(element); } for (auto const& f : gf.cflags) u.packageCflags.push_back(f); for (auto const& f : gf.cxxflags) u.packageCxxflags.push_back(f); diff --git a/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh b/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh new file mode 100644 index 000000000..8f5ea5e1a --- /dev/null +++ b/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh @@ -0,0 +1,156 @@ +#!/usr/bin/env bash +# requires: python3 +# 736 -- an element of a compile-flag list stands for the same words on every +# host, the compiler receives those words, and both compile databases list +# them (#655). +# +# Every expectation below is the same on Linux, macOS and Windows: the words +# are defined by the manifest's flag syntax, not by the host's command-line +# reader. Criteria: +# A. the program prints the value each macro received; +# B. compile_commands.json lists the words for the unit; +# C. `mcpp emit build-database` lists the same words for the unit; +# D. the first plan names the elements whose words changed in 2026.9.17.1, +# and does not name the ones whose words did not; +# E. a build that repeats the plan names nothing. +set -e + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; [ -n "${2:-}" ] && cat "$2"; exit 1; } + +mkdir -p "$TMP/app/src" +cd "$TMP/app" +# The TOML below is written with a quoted heredoc, so every backslash is the +# manifest's own. Each element is followed by the words it stands for. +cat > mcpp.toml <<'EOF' +[package] +name = "flagwords" +version = "0.1.0" + +[build] +cxxflags = [ + "-DV_ESC=\\\"esc.h\\\"", # -DV_ESC="esc.h" (the libarchive spelling) + "-DV_MID=\"mid\"", # -DV_MID=mid + "-DV_SQ='sq'", # -DV_SQ=sq + "'-DV_SP=a b'", # -DV_SP=a b + "-DV_BS=a\\ b", # -DV_BS=a b (pkg-config's escaped space) + "-DV_DOL=a$b", # -DV_DOL=a$b + "-DV_A=1 -DV_B=2", # -DV_A=1 -DV_B=2 +] +defines = [ + "V_DEF=\"def\"", # -DV_DEF="def" + "V_DSP=a b", # -DV_DSP=a b +] +EOF +cat > src/main.cpp <<'EOF' +#include +#define STR2(...) #__VA_ARGS__ +#define STR(...) STR2(__VA_ARGS__) +static void show(const char* name, const char* value) { + std::printf("%s=[%s]\n", name, value); +} +int main() { + show("V_ESC", STR(V_ESC)); + show("V_MID", STR(V_MID)); + show("V_SQ", STR(V_SQ)); + show("V_SP", STR(V_SP)); + show("V_BS", STR(V_BS)); + show("V_DOL", STR(V_DOL)); + show("V_A", STR(V_A)); + show("V_B", STR(V_B)); + show("V_DEF", STR(V_DEF)); + show("V_DSP", STR(V_DSP)); + return 0; +} +EOF + +"$MCPP" build > build1.log 2>&1 || fail "the first build failed" build1.log + +# A. What the compiler received. +"$MCPP" run > run.log 2>&1 || fail "the program did not run" run.log +tr -d '\r' < run.log > run.txt +cat > want.txt <<'EOF' +V_ESC=["esc.h"] +V_MID=[mid] +V_SQ=[sq] +V_SP=[a b] +V_BS=[a b] +V_DOL=[a$b] +V_A=[1] +V_B=[2] +V_DEF=["def"] +V_DSP=[a b] +EOF +for line in $(seq 1 10); do + want=$(sed -n "${line}p" want.txt) + grep -qxF -- "$want" run.txt || fail "A: the program did not print $want" run.txt +done + +# B and C. What the databases list. The expected words are the ones above. +cat > words.json <<'EOF' +["-DV_ESC=\"esc.h\"", "-DV_MID=mid", "-DV_SQ=sq", "-DV_SP=a b", "-DV_BS=a b", + "-DV_DOL=a$b", "-DV_A=1", "-DV_B=2", "-DV_DEF=\"def\"", "-DV_DSP=a b"] +EOF +cdb=$(find target -name compile_commands.json | head -1) +[ -n "$cdb" ] || cdb=compile_commands.json +[ -f "$cdb" ] || fail "B: no compile_commands.json" +"$MCPP" emit build-database --format json > db.json 2> db.err || fail "C: emit failed" db.err +python3 - "$cdb" db.json words.json <<'PY' || exit 1 +import json, sys +words = json.load(open(sys.argv[3])) + +def contains(args): + return all(w in args for w in words) and not any( + "\\\"" in a or a.endswith("'") for a in args if a.startswith("-DV_")) + +entries = json.load(open(sys.argv[1])) +main = [e for e in entries if e["file"].replace("\\", "/").endswith("src/main.cpp")] +if not main or not contains(main[0]["arguments"]): + print("FAIL: B: compile_commands.json does not list the words") + print(json.dumps(main[0]["arguments"] if main else entries, indent=1)) + sys.exit(1) + +def lists(node): + if isinstance(node, dict): + for v in node.values(): + yield from lists(v) + elif isinstance(node, list): + if node and all(isinstance(x, str) for x in node): + yield node + for v in node: + yield from lists(v) + +doc = json.load(open(sys.argv[2])) +found = [] +for args in lists(doc): + if any(a.startswith("-DV_") for a in args): + found.append(args) +# S1 may split a unit's arguments into a set's baseline and the unit's own +# local arguments; the words must be present across the lists that carry them. +flat = [a for args in found for a in args] +if not contains(flat): + print("FAIL: C: the build database does not list the words") + print(json.dumps(found, indent=1)) + sys.exit(1) +PY + +# D. The first plan names the elements whose words changed, and only those. +for element in "V_DEF=\"def\"" "-DV_BS=a\\ b" "-DV_DOL=a\$b" "-DV_A=1 -DV_B=2"; do + grep -qF -- "element '$element' reaches the compiler as" build1.log \ + || fail "D: the first plan does not name $element" build1.log +done +for element in "-DV_MID=\"mid\"" "V_DSP=a b" "-DV_ESC="; do + if grep -F -- "reaches the compiler as" build1.log | grep -qF -- "element '$element"; then + fail "D: the first plan names $element, whose words did not change" build1.log + fi +done + +# E. A repeated plan names nothing. +touch src/main.cpp +"$MCPP" build > build2.log 2>&1 || fail "the second build failed" build2.log +if grep -q "reaches the compiler as" build2.log; then + fail "E: a repeated plan repeated the notes" build2.log +fi + +echo "OK" diff --git a/tests/unit/test_compile_commands.cpp b/tests/unit/test_compile_commands.cpp index e21c81c8f..db7964983 100644 --- a/tests/unit/test_compile_commands.cpp +++ b/tests/unit/test_compile_commands.cpp @@ -6,6 +6,8 @@ import mcpp.build.compile_commands; import mcpp.build.flags; import mcpp.build.plan; import mcpp.libs.json; +import mcpp.manifest.flag_words; +import mcpp.platform; using namespace mcpp::build; @@ -292,23 +294,137 @@ TEST(CompileCommandsArgs, AQuotedPathWithASpaceStaysOneToken) { for (auto const& t : out) EXPECT_FALSE(has_edge_quote(t)) << t; } -// Unquoted input must keep behaving exactly as before: ninja escapes undone, -// split on spaces. -TEST(CompileCommandsArgs, UnquotedInputIsUnchanged) { +// Unquoted input: ninja's escapes are undone first, and then the host splits +// the line. `$ ` is a plain space by the time the host reads it, so an +// unquoted `-Idir$ with$ space` is three words to the compiler, and the +// database says so. The engine quotes every word it writes (include_token, +// ninja_command_word); the quoted spelling is the test above. +TEST(CompileCommandsArgs, UnquotedInputSplitsWhereTheHostSplits) { auto out = mcpp::build::split_flags("-IC$:/x -DA=1 -Idir$ with$ space"); - ASSERT_EQ(out.size(), 3u); + ASSERT_EQ(out.size(), 5u); EXPECT_EQ(out[0], "-IC:/x"); EXPECT_EQ(out[1], "-DA=1"); - EXPECT_EQ(out[2], "-Idir with space"); + EXPECT_EQ(out[2], "-Idir"); + EXPECT_EQ(out[3], "with"); + EXPECT_EQ(out[4], "space"); } -// A quote in the MIDDLE of a token is data, not quoting: `-DA="x"` must keep -// its inner quotes or the define changes meaning. -TEST(CompileCommandsArgs, InnerQuotesAreNotStripped) { +// A quote inside a word is quoting on both hosts' readers: `sh` and the MSVCRT +// rules both hand the compiler `-DGREETING=hi` for this text. The version of +// this test before #655 asserted the opposite, which pinned the compile +// database to a reading no host applies. +TEST(CompileCommandsArgs, QuotesInsideAWordAreQuoting) { auto out = mcpp::build::split_flags(R"(-DGREETING="hi")"); ASSERT_EQ(out.size(), 1u); - EXPECT_EQ(out[0], R"(-DGREETING="hi")"); - EXPECT_FALSE(is_quoted(out[0])); + EXPECT_EQ(out[0], "-DGREETING=hi"); +} + +// ── A unit's flag list: the edge and the databases agree (#655) ───────────── +// +// The edge writes each word of a unit's flag list quoted for the host +// (ninja_flag_list); the databases list the words. Reading the edge's text +// the way ninja and the host read it must give back the listed words. The +// test runs the reader of the host it runs on, so each CI host checks its own. + +namespace { + +const std::vector kIssue655Elements{ + R"(-DPLATFORM_CONFIG_H=\"mcpp_libarchive_config.h\")", + R"(-DMID="mid")", + "-DSQ='sq'", + "-include mcpp_lua_platform_config.h", + R"(-I/opt/my\ dir/include)", + R"(-IC:\Users\x\include)", + "-DDOLLAR=a$b", + // What fold_build_defines_into_flags inserts for `defines = ["DEF=\"def\""]` + // and `defines = ["SPACE=a b"]`. + mcpp::manifest::flag_element(R"(-DDEF="def")"), + mcpp::manifest::flag_element("-DSPACE=a b"), +}; + +std::vector random_words() { + std::vector words{"", "a b", "\\", "a\\", "C:\\dir\\", "\"", "'", + "it's", "-DV=\"x\\\"y\"", "$x", "a$$b"}; + const std::string alphabet = "ab \t\"'\\$:;&|<>"; + std::uint32_t state = 655; + for (int n = 0; n < 1000; ++n) { + std::string w; + const int length = static_cast(state % 10); + for (int i = 0; i < length; ++i) { + state = state * 1664525u + 1013904223u; + w.push_back(alphabet[(state >> 16) % alphabet.size()]); + } + state = state * 1664525u + 1013904223u; + words.push_back(std::move(w)); + } + return words; +} + +} // namespace + +TEST(CompileCommandsArgs, TheEdgeAndTheDatabaseListTheSameWords) { + const auto words = mcpp::manifest::flag_words(kIssue655Elements); + EXPECT_EQ(mcpp::build::split_flags(mcpp::build::ninja_flag_list(kIssue655Elements)), + words); + const std::vector want{ + R"(-DPLATFORM_CONFIG_H="mcpp_libarchive_config.h")", "-DMID=mid", "-DSQ=sq", + "-include", "mcpp_lua_platform_config.h", "-I/opt/my dir/include", + R"(-IC:\Users\x\include)", "-DDOLLAR=a$b", R"(-DDEF="def")", "-DSPACE=a b"}; + EXPECT_EQ(words, want); +} + +TEST(CompileCommandsArgs, AWordOnAnEdgeReadsBackAsTheWord) { + for (auto const& w : random_words()) { + EXPECT_EQ(mcpp::build::split_flags(mcpp::build::ninja_command_word(w)), + std::vector{w}) + << "word: [" << w << "]"; + } +} + +// The reader above is mcpp's model of the host. On a POSIX host the host itself +// is available, so the model is checked against it: `/bin/sh` receives the +// quoted words and prints each one back. +TEST(CompileCommandsArgs, ShellReceivesTheQuotedWordsAsWritten) { + if constexpr (mcpp::platform::is_windows) { + GTEST_SKIP() << "the POSIX shell is the reader measured here"; + } + TempDir temp; + const auto out = temp.path / "words.txt"; + auto words = random_words(); + std::erase_if(words, [](const std::string& w) { return w.empty(); }); + words.resize(200); + std::string command = "printf '%s\\n'"; + for (auto const& w : words) { + command += ' '; + command += mcpp::build::shell_quote_arg(w); + } + command += " > '" + out.string() + "'"; + ASSERT_EQ(std::system(command.c_str()), 0); + std::ifstream in(out); + std::vector printed; + for (std::string line; std::getline(in, line);) printed.push_back(line); + EXPECT_EQ(printed, words); +} + +TEST(CompileCommandsEmit, AUnitListsTheWordsOfItsFlagList) { + BuildPlan plan; + plan.projectRoot = "/p"; + plan.outputDir = "/p/target"; + plan.compileUnits.push_back({ + .source = std::filesystem::path("/p/src/archive.c"), + .kind = mcpp::SourceKind::C, + .object = std::filesystem::path("obj") / "archive.o", + .packageName = "demo", + .packageCflags = kIssue655Elements, + }); + CompileFlags flags; + flags.ccBinary = std::filesystem::path("/usr/bin/gcc"); + auto j = nlohmann::json::parse(emit_compile_commands(plan, flags)); + std::vector args; + for (auto const& a : j[0]["arguments"]) args.push_back(a.get()); + const auto words = mcpp::manifest::flag_words(kIssue655Elements); + auto at = std::search(args.begin(), args.end(), words.begin(), words.end()); + EXPECT_NE(at, args.end()) << "the unit's words are not listed in order"; } // ── Emitted paths use native separators ───────────────────────────────────── From 15e7dff6e9b4554afbb7e0c2e4471adc1a61ed52 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 17 Sep 2026 03:08:38 +0800 Subject: [PATCH 2/6] 655: the ecosystem verification script --- .agents/docs/2026-09-17-655-verify.sh | 241 ++++++++++++++++++++++++++ 1 file changed, 241 insertions(+) create mode 100644 .agents/docs/2026-09-17-655-verify.sh diff --git a/.agents/docs/2026-09-17-655-verify.sh b/.agents/docs/2026-09-17-655-verify.sh new file mode 100644 index 000000000..34d7637c1 --- /dev/null +++ b/.agents/docs/2026-09-17-655-verify.sh @@ -0,0 +1,241 @@ +#!/usr/bin/env bash +# Ecosystem verification for #655 against the PUBLISHED mcpp and index, run +# inside a SubOS sandbox with CN mirrors for xlings and mcpp. +# +# B64=$(base64 -w0 .agents/docs/2026-09-17-655-verify.sh) +# E2E=$(base64 -w0 tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh) +# xlings subos new v655 +# xlings subos use v655 --sandbox --cmd \ +# "echo $B64 | base64 -d > /tmp/v.sh && echo $E2E | base64 -d > /tmp/e2e736.sh && MCPP_VERIFY_VERSION=2026.9.17.1 bash /tmp/v.sh" +# +# Run it against the previous release first (MCPP_VERIFY_VERSION=2026.9.16.2): +# the sections marked CHANGE must fail there and pass on the new release, and +# the sections marked GUARD must pass on both. A CHANGE section that passes on +# both releases measured nothing. +# +# The sandbox shares the xlings data directory, so the published mcpp is +# addressed by its store path and exact version. Its $HOME content persists +# between runs of one SubOS, so every section clears its own probe directory. +set -u + +VER="${MCPP_VERIFY_VERSION:?set MCPP_VERIFY_VERSION}" +STORE="${MCPP_VERIFY_BIN:-$HOME/.xlings/data/xpkgs/xim-x-mcpp/$VER/bin/mcpp}" + +fails=0 +skipped="" +fail() { printf 'ASSERT-FAIL: %s\n' "$1"; fails=$((fails + 1)); } +ok() { printf 'ok: %s\n' "$1"; } +section() { printf '\n== %s ==\n' "$1"; } +skip() { printf 'NOT RUN: %s\n' "$1"; skipped="$skipped + - $1"; } +reading() { printf 'READING %s\n' "$*"; } +unset XLINGS_ACTIVE_SUBOS + +root="$HOME/verify-655" +rm -rf "$root"; mkdir -p "$root" + +# The first entry of compile_commands.json whose file matches $2 and whose +# arguments contain $3, executed with `-fsyntax-only` in place of `-c -o +# `. Prints the exit status and the first error line. +exec_unit() { # + python3 - "$@" <<'PY' +import json, subprocess, sys +db, fsub, asub = sys.argv[1], sys.argv[2], sys.argv[3] +for e in json.load(open(db)): + args = e["arguments"] + if fsub in e["file"] and any(asub in a for a in args): + i = args.index("-c") + argv = args[:i] + ["-fsyntax-only", e["file"]] + r = subprocess.run(argv, cwd=e["directory"], capture_output=True, text=True) + err = next((l for l in r.stderr.splitlines() if "error" in l), "") + print(f"rc={r.returncode} file={e['file'].split('/')[-1]} arg={[a for a in args if asub in a][0]!r} {err}") + sys.exit(0) +print("rc=none no unit matched") +PY +} + +section "A. GUARD: the published mcpp answers for itself, with CN mirrors for xlings and mcpp" +if [ ! -x "$STORE" ]; then + fail "no mcpp at $STORE"; printf '\nfails=%d (nothing else can run)\n' "$fails"; exit 1 +fi +got=$("$STORE" --version 2>&1 | head -1) +case "$got" in *"$VER"*) ok "mcpp --version says $got" ;; *) fail "mcpp --version says '$got', expected $VER" ;; esac +if command -v xlings >/dev/null 2>&1; then + xlings config --mirror CN >/dev/null 2>&1 && ok "xlings config --mirror CN" || fail "xlings config --mirror CN" +fi +"$STORE" self config --mirror CN >/dev/null 2>&1 && ok "mcpp self config --mirror CN" || fail "mcpp self config --mirror CN" +grep -q '"mirror": *"CN"' "$HOME/.mcpp/registry/.xlings.json" 2>/dev/null \ + && ok "mcpp's xlings reads mirror CN" || fail "mcpp's xlings does not read mirror CN" + +section "B. CHANGE: the issue's five macros read the same in the build and in the database" +d=$root/b; rm -rf "$d"; mkdir -p "$d/src" +cat > "$d/mcpp.toml" <<'EOF' +[package] +name = "i655" +version = "0.1.0" + +[build] +sources = ["src/*.c"] +cflags = ["-DESC=\\\"esc.h\\\"", "-DMID=\"mid\"", "-DSQ='sq'"] +defines = ["DEF=\"def\"", "SPACE=\"a b\""] +EOF +printf 'int main(void){return 0;}\n' > "$d/src/main.c" +( cd "$d" && timeout 1800 "$STORE" build > build.log 2>&1 ) || fail "B: the build failed" +build_cmd=$(cd "$d"/target/*/*/ 2>/dev/null && ninja -t commands obj/main.o 2>/dev/null | tail -1) +if [ -z "$build_cmd" ]; then + skip "B: ninja is not on PATH in the sandbox; the database leg runs alone" +fi +python3 - "$d/compile_commands.json" > "$d/db-macros.txt" <<'PY' +import json, subprocess, sys +e = json.load(open(sys.argv[1]))[0] +a = e["arguments"]; i = a.index("-c") +r = subprocess.run(a[:i] + ["-E", "-dM", "-x", "c", "/dev/null"], cwd=e["directory"], capture_output=True, text=True) +print(f"rc {r.returncode}") +for l in r.stdout.splitlines(): + w = l.split() + if len(w) > 1 and w[1] in ("ESC", "MID", "SQ", "DEF", "SPACE"): + print(l) +PY +reading "B database macros: $(tr '\n' ';' < "$d/db-macros.txt")" +grep -qx 'rc 0' "$d/db-macros.txt" || fail "B: the database's arguments do not execute" +for want in '#define ESC "esc.h"' '#define MID mid' '#define SQ sq' '#define DEF "def"' '#define SPACE "a b"'; do + grep -qxF "$want" "$d/db-macros.txt" || fail "B: the database gives no '$want'" +done +if [ -n "$build_cmd" ]; then + (cd "$d"/target/*/*/ && /bin/sh -c "$(printf '%s' "$build_cmd" | sed 's/ -MMD .*$/ -E -dM -x c \/dev\/null/')") \ + | grep -E 'define (ESC|MID|SQ|DEF|SPACE) ' | sort > "$d/build-macros.txt" + grep -v '^rc' "$d/db-macros.txt" | sort | diff -q - "$d/build-macros.txt" >/dev/null \ + && ok "B: the build and the database define the same five macros" \ + || fail "B: the build and the database differ: $(tr '\n' ';' < "$d/build-macros.txt")" +fi + +section "C. CHANGE: e2e 736 against the published binary" +if [ -f /tmp/e2e736.sh ]; then + if MCPP="$STORE" timeout 1800 bash /tmp/e2e736.sh > "$root/e2e736.log" 2>&1; then + ok "C: e2e 736 passes" + else + fail "C: e2e 736: $(grep -m1 FAIL "$root/e2e736.log")" + fi +else + skip "C: /tmp/e2e736.sh was not passed in" +fi + +section "D. CHANGE and GUARD: an index consumer of compat.libarchive" +d=$root/d; rm -rf "$d"; mkdir -p "$d/src" +cat > "$d/mcpp.toml" <<'EOF' +[package] +name = "arc" +version = "0.1.0" + +[dependencies.compat] +libarchive = "3.8.7" +EOF +cat > "$d/src/main.cpp" <<'EOF' +#include +#include +int main() { std::printf("%s\n", archive_version_string()); return 0; } +EOF +( cd "$d" && timeout 3600 "$STORE" run > run.log 2>&1 ); rc=$? +reading "D run exit=$rc: $(grep -m1 -E 'libarchive [0-9]' "$d/run.log")" +[ "$rc" -eq 0 ] && grep -q 'libarchive 3.8.7' "$d/run.log" && ok "D: the consumer builds and runs" \ + || fail "D: the consumer did not run: $(tail -3 "$d/run.log" | tr '\n' ' ')" +grep -q 'reaches the compiler as' "$d/run.log" \ + && fail "D GUARD: a published descriptor produced a build/flag-words note" \ + || ok "D GUARD: no build/flag-words note for the published descriptors" +if [ -f "$d/compile_commands.json" ]; then + grep -qF -- '"-DPLATFORM_CONFIG_H=\"mcpp_libarchive_config.h\""' "$d/compile_commands.json" \ + && ok "D: compile_commands.json lists -DPLATFORM_CONFIG_H=\"mcpp_libarchive_config.h\"" \ + || fail "D: compile_commands.json lists $(grep -o '"-DPLATFORM_CONFIG_H[^,]*' "$d/compile_commands.json" | head -1)" + unit=$(exec_unit "$d/compile_commands.json" "libarchive/archive_entry.c" "PLATFORM_CONFIG_H") + reading "D libarchive unit: $unit" + case "$unit" in rc=0*) ok "D: a libarchive unit's arguments execute" ;; *) fail "D: a libarchive unit's arguments do not execute" ;; esac +else + fail "D: no compile_commands.json" +fi + +section "E. GUARD: an index consumer of compat.lua (a packed -include element)" +d=$root/e; rm -rf "$d"; mkdir -p "$d/src" +cat > "$d/mcpp.toml" <<'EOF' +[package] +name = "luac" +version = "0.1.0" + +[dependencies.compat] +lua = "5.4.7" +EOF +cat > "$d/src/main.cpp" <<'EOF' +extern "C" { +#include +#include +#include +} +#include +int main() { + lua_State* L = luaL_newstate(); + luaL_openlibs(L); + luaL_dostring(L, "return 6 * 7"); + std::printf("lua=%d\n", static_cast(lua_tointeger(L, -1))); + lua_close(L); + return 0; +} +EOF +( cd "$d" && timeout 3600 "$STORE" run > run.log 2>&1 ); rc=$? +reading "E run exit=$rc: $(grep -m1 'lua=' "$d/run.log")" +grep -q 'lua=42' "$d/run.log" && ok "E: the lua consumer builds and runs" \ + || fail "E: the lua consumer did not run: $(tail -3 "$d/run.log" | tr '\n' ' ')" +grep -q 'reaches the compiler as' "$d/run.log" \ + && fail "E GUARD: a published descriptor produced a build/flag-words note" \ + || ok "E GUARD: no build/flag-words note" + +section "F. CHANGE: the reporter's project, openxlings/xlings" +d=$root/f; rm -rf "$d" +if timeout 900 git clone -q --depth 1 https://gitcode.com/openxlings/xlings.git "$d" 2>/dev/null \ + || timeout 900 git clone -q --depth 1 https://github.com/openxlings/xlings.git "$d" 2>/dev/null; then + reading "F xlings commit: $(git -C "$d" log -1 --format=%h)" + ( cd "$d" && timeout 3600 "$STORE" emit build-database --format json > db.json 2> db.err ); rc=$? + reading "F emit exit=$rc" + python3 - "$d/db.json" > "$d/db-reading.txt" <<'PY' +import json, sys +doc = json.load(open(sys.argv[1])) +hits = set() +def walk(o): + if isinstance(o, dict): + for v in o.values(): walk(v) + elif isinstance(o, list): + for v in o: walk(v) + elif isinstance(o, str) and "PLATFORM_CONFIG_H" in o: + hits.add(o) +walk(doc) +for h in sorted(hits): print(repr(h)) +PY + reading "F build database PLATFORM_CONFIG_H: $(tr '\n' ' ' < "$d/db-reading.txt")" + grep -qxF "'-DPLATFORM_CONFIG_H=\"mcpp_libarchive_config.h\"'" "$d/db-reading.txt" \ + && ok "F: the build database lists the define without shell escapes" \ + || fail "F: the build database lists $(cat "$d/db-reading.txt")" + # The unit is taken from the build database itself: its `arguments` and + # `work-directory`, with `-fsyntax-only` in place of `-c -o `. + unit=$(python3 - "$d/db.json" <<'PY2' +import json, subprocess, sys +db = json.load(open(sys.argv[1]))["data"]["database"] +for s in db["sets"]: + for u in s.get("translation-units", []): + args = u.get("arguments", []) + if u.get("source", "").endswith("libarchive/archive_read.c") and any("PLATFORM_CONFIG_H" in a for a in args): + i = args.index("-c") + r = subprocess.run(args[:i] + ["-fsyntax-only", u["source"]], cwd=u.get("work-directory", "."), + capture_output=True, text=True) + err = next((l for l in r.stderr.splitlines() if "error" in l), "") + print(f"rc={r.returncode} set={s.get('name')} {err}") + sys.exit(0) +print("rc=none no unit matched") +PY2 +) + reading "F libarchive unit from the build database: $unit" + case "$unit" in rc=0*) ok "F: a libarchive unit of xlings executes from the build database" ;; *) fail "F: a libarchive unit of xlings does not execute from the build database" ;; esac +else + skip "F: could not clone openxlings/xlings" +fi + +printf '\n== summary ==\nversion=%s fails=%d\n' "$VER" "$fails" +[ -n "$skipped" ] && printf 'not run:%s\n' "$skipped" +exit 0 From 9460bacf5c4dff83385376fa92c74bd8e538d657 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 17 Sep 2026 03:17:46 +0800 Subject: [PATCH 3/6] CI on macOS 27: ci-macos, ci-macos-e2e and both fresh-install channels GitHub serves macOS 27 under the preview label xcode-27 (macOS 27.0 with Xcode 27, actions/runner-images#14404); there is no macos-27 label. Measured on #658: the label delivers macOS 27.0 (26A5406e), arm64, ImageOS macos27. - ci-macos and ci-macos-e2e run on macos-15 and xcode-27. - ci-fresh-install runs the xlings and Homebrew channels on macos-14 (the floor) and xcode-27 (the newest release). - Every macOS 27 leg asserts sw_vers major 27, because the label names an Xcode and its base OS has changed once. - setup-macos-llvm takes the image as an input and puts it in the xlings cache key, which named macos15 for every image. --- .../2026-09-17-655-implementation-plan.md | 18 ++++++++- .github/actions/setup-macos-llvm/action.yml | 11 ++++- .github/workflows/ci-fresh-install.yml | 40 ++++++++++++++++--- .github/workflows/ci-macos-e2e.yml | 11 ++++- .github/workflows/ci-macos.yml | 27 +++++++++++-- CHANGELOG.md | 4 ++ 6 files changed, 97 insertions(+), 14 deletions(-) diff --git a/.agents/docs/2026-09-17-655-implementation-plan.md b/.agents/docs/2026-09-17-655-implementation-plan.md index 08003961e..2a816b4cb 100644 --- a/.agents/docs/2026-09-17-655-implementation-plan.md +++ b/.agents/docs/2026-09-17-655-implementation-plan.md @@ -165,7 +165,8 @@ reason). | E7 | `build/flag-words` note on the first plan | E1 | branch: e2e 736 D and E | | E8 | SPEC-004 §8, SPEC-005 R3.7, docs/04 and docs/30 with their Chinese mirrors, CHANGELOG | E1-E7 | branch | | E9 | version 2026.9.17.1 | - | branch | -| E10 | CI green on every workflow of the pull request | E1-E9 | todo | +| E10 | CI green on every workflow of the pull request | E1-E9, E11 | todo | +| E11 | macOS 27 legs (maintainer request, 2026-09-17; label measured in §5.1): `ci-macos` and `ci-macos-e2e` run on `macos-15` and macOS 27; `ci-fresh-install` runs its xlings and Homebrew channels on `macos-14` and macOS 27; each macOS 27 leg asserts `sw_vers` major 27 | - | branch | ### 3.2 Release and ecosystem, in order @@ -227,6 +228,21 @@ Recorded after the release in §7. Recorded as each row closes. +## 5.1 macOS 27 on GitHub-hosted runners + +GitHub publishes no `macos-27` label. The runner-images README (read on +2026-09-17) lists macOS 26 as the newest GA image (`macos-26`, `macos-latest`) +and macOS 14 as deprecated (unsupported from 2026-11-02). macOS 27 is served as +the preview label `xcode-27`, whose base OS moved from macOS 26 to macOS 27.0 on +2026-09-16 (actions/runner-images#14404), with a warning that capacity may +queue. A temporary pull request (#658) measured which labels are served; its +first run, naming `macos-27`, `macos-27-arm64`, `macos-26` and `macos-latest`, +was cancelled while every job was still queued behind this repository's runner +limit, and the second names `xcode-27` and reads +`label=xcode-27 27.0 build=26A5406e arch=arm64 xcode=Xcode 27.0 clang=Apple clang +version 21.0.0` with `ImageOS=macos27` (run on 2026-09-16T19:16Z). Because the label names an Xcode and its +base OS has changed once, every macOS 27 leg asserts that `sw_vers` reports 27. + ## 8. Residuals - `ldflags`, `dialect_cxxflags` and `std-module-flags` keep their current diff --git a/.github/actions/setup-macos-llvm/action.yml b/.github/actions/setup-macos-llvm/action.yml index 458fbad3e..5ff93c043 100644 --- a/.github/actions/setup-macos-llvm/action.yml +++ b/.github/actions/setup-macos-llvm/action.yml @@ -16,6 +16,13 @@ inputs: # .github/actions/bootstrap-mcpp/action.yml for why 0.4.69 is required # (two packages named `lua` in one repo need openxlings/xlings#381). default: '2026.9.16.1' + image: + description: > + The runner label the job runs on (macos-15, xcode-27). It is part of the + xlings cache key: the cache holds installed binaries, and a key that does + not name the OS release lets one image restore what another installed. + required: false + default: 'macos-15' runs: using: composite @@ -24,9 +31,9 @@ runs: uses: actions/cache@v4 with: path: ~/.xlings - key: xlings-macos15-arm64-v3-xl${{ inputs.xlings-version }}-${{ hashFiles('.xlings.json') }} + key: xlings-${{ inputs.image }}-arm64-v3-xl${{ inputs.xlings-version }}-${{ hashFiles('.xlings.json') }} restore-keys: | - xlings-macos15-arm64-v3-xl${{ inputs.xlings-version }}- + xlings-${{ inputs.image }}-arm64-v3-xl${{ inputs.xlings-version }}- - name: Bootstrap xlings shell: bash diff --git a/.github/workflows/ci-fresh-install.yml b/.github/workflows/ci-fresh-install.yml index ea90dcd9d..2f3d5c41d 100644 --- a/.github/workflows/ci-fresh-install.yml +++ b/.github/workflows/ci-fresh-install.yml @@ -357,13 +357,25 @@ jobs: # ────────────────────────────────────────────────────────────────── macos-fresh: needs: [wait-index] - name: macOS fresh install + name: macOS fresh install (${{ matrix.image }}) if: ${{ github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' }} + # Two images, the two ends of the supported range. + # # macos-14: the support floor (mcpp ≥0.0.50 / xlings ≥0.4.50 ship # minos=14.0 static-libc++ binaries). A fresh install passing here # is the continuous proof of the macOS 14 floor — and of host-tool # independence (this image has no sha256sum; macos-15 does). - runs-on: macos-14 + # + # xcode-27: macOS 27, the newest release, served under the preview label + # `xcode-27` (actions/runner-images#14404; there is no `macos-27` label). + # The same released binaries, the same steps; a failure only here is a + # change in the newest system (SDK, loader, system libc++ headers) that the + # floor cannot show. + strategy: + fail-fast: false + matrix: + image: [macos-14, xcode-27] + runs-on: ${{ matrix.image }} timeout-minutes: 30 env: # The one derived value (see the header comment): every install job names @@ -372,6 +384,14 @@ jobs: steps: - uses: actions/checkout@v4 + - name: The image is the macOS it names + run: | + echo "macOS $(sw_vers -productVersion) | $(uname -m)" + # See ci-macos.yml: `xcode-27` names an Xcode, not an OS. + if [ "${{ matrix.image }}" = xcode-27 ]; then + [ "$(sw_vers -productVersion | cut -d. -f1)" = 27 ] || { echo "::error::xcode-27 delivered macOS $(sw_vers -productVersion), not 27"; exit 1; } + fi + - name: Install xlings env: XLINGS_NON_INTERACTIVE: '1' @@ -452,16 +472,24 @@ jobs: # channel healthy the entire time. So the trust gate itself is asserted # from BOTH sides: refused before `brew trust`, accepted after. macos-brew-fresh: - name: macOS fresh install (Homebrew) + name: macOS fresh install (Homebrew, ${{ matrix.image }}) if: ${{ github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' }} - # Same floor as macos-fresh: the formula declares `depends_on macos: - # :sonoma` + arm64, and macos-14 is the oldest image satisfying it. - runs-on: macos-14 + # Same two images as macos-fresh: the formula declares `depends_on macos: + # :sonoma` + arm64, macos-14 is the oldest image satisfying it, and + # xcode-27 is macOS 27, the newest release. + strategy: + fail-fast: false + matrix: + image: [macos-14, xcode-27] + runs-on: ${{ matrix.image }} timeout-minutes: 30 steps: - name: Environment run: | echo "macOS $(sw_vers -productVersion) | $(uname -m) | $(brew --version | head -1)" + if [ "${{ matrix.image }}" = xcode-27 ]; then + [ "$(sw_vers -productVersion | cut -d. -f1)" = 27 ] || { echo "::error::xcode-27 delivered macOS $(sw_vers -productVersion), not 27"; exit 1; } + fi # ① The command README.md documents, on a machine that has never tapped. # Fully qualified, so Homebrew treats it as explicit intent. diff --git a/.github/workflows/ci-macos-e2e.yml b/.github/workflows/ci-macos-e2e.yml index 80a0e0a4d..ac6f72e4b 100644 --- a/.github/workflows/ci-macos-e2e.yml +++ b/.github/workflows/ci-macos-e2e.yml @@ -22,8 +22,13 @@ concurrency: jobs: e2e: - name: e2e suite (macOS ARM64, self-host) - runs-on: macos-15 + name: e2e suite (macOS ARM64, self-host, ${{ matrix.image }}) + # The same two images as ci-macos.yml; `xcode-27` is macOS 27 (see there). + strategy: + fail-fast: false + matrix: + image: [macos-15, xcode-27] + runs-on: ${{ matrix.image }} timeout-minutes: 60 # NOTE: no MCPP_VERBOSE — the e2e suite asserts mcpp's default quiet # output (tests 48/53). @@ -37,6 +42,8 @@ jobs: with: submodules: recursive - uses: ./.github/actions/setup-macos-llvm + with: + image: ${{ matrix.image }} - name: Build mcpp from source (self-host) run: | diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index 4f3f9a7ad..aa670d4c0 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -19,9 +19,22 @@ concurrency: jobs: macos-xlings-llvm: - name: macOS ARM64 — xlings LLVM end-to-end - runs-on: macos-15 - timeout-minutes: 30 + name: macOS ARM64 — xlings LLVM end-to-end (${{ matrix.image }}) + # Two images: macos-15, the image every other macOS job uses, and macOS 27, + # the newest macOS release. The newest release is where a change of the SDK, + # the system libc++ headers or the loader first shows; the packaged binaries + # carry minos=14.0, and ci-fresh-install covers that floor. + # + # GitHub serves macOS 27 under the preview label `xcode-27` (macOS 27.0 with + # Xcode 27, actions/runner-images#14404); there is no `macos-27` label, and + # a job naming one waits for a runner that never comes. The "System info" + # step prints the OS the label delivered. + strategy: + fail-fast: false + matrix: + image: [macos-15, xcode-27] + runs-on: ${{ matrix.image }} + timeout-minutes: 45 # NOTE: no MCPP_VERBOSE here — keep this job's output shape identical to # ci-macos-e2e.yml, which asserts mcpp's default quiet output (48/53). steps: @@ -31,10 +44,18 @@ jobs: run: | uname -a sw_vers + # `xcode-27` names an Xcode, not an OS, and the image changed its base + # OS once already; the leg exists for macOS 27, so it says so or fails. + if [ "${{ matrix.image }}" = xcode-27 ]; then + major=$(sw_vers -productVersion | cut -d. -f1) + [ "$major" = 27 ] || { echo "::error::xcode-27 delivered macOS $(sw_vers -productVersion), not 27"; exit 1; } + fi xcrun --show-sdk-path echo "SDK: $(xcrun --show-sdk-version)" - uses: ./.github/actions/setup-macos-llvm + with: + image: ${{ matrix.image }} - name: Inspect LLVM package structure run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fd75f808..f1bb65716 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,10 @@ libarchive、bzip2、lz4、xz、zlib 与 zstd 的每个 C 文件都无法解析 这样的元素。 - Windows 上 `shell_quote_arg` 按 MSVCRT 规则加倍引号前与结尾处的反斜杠,以 `\` 结尾的 词不再吞掉闭合引号。 +- **CI 覆盖 macOS 27。** `ci-macos` 与 `ci-macos-e2e` 在 `macos-15` 与 macOS 27 上各跑一遍, + `ci-fresh-install` 的 xlings 与 Homebrew 两条安装渠道在 `macos-14` 与 macOS 27 上各跑一遍。 + GitHub 以预览标签 `xcode-27` 提供 macOS 27(无 `macos-27` 标签),每条 macOS 27 的腿断言 + `sw_vers` 主版本为 27。 ### 内置 xlings 升至 2026.9.16.1(2026.9.16.2) From 10224fb84769807afe8b840e88bff4a8c8f2e756 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 17 Sep 2026 03:45:46 +0800 Subject: [PATCH 4/6] macOS 27: the std module builds; the -D compatibility rule; host-shaped test fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit macOS 27 (measured on the xcode-27 image, #659): the 27.0 SDK's math.h leaves INFINITY and NAN to once modules are on, and clang 22's header does not supply them under -std=c++23, so libc++'s std module stopped building. apple_float_macro_words (hostflags.cppm) states both with the SDK's own GNU-mode spellings for clang on an Apple target; its readers quote the words their own way (ninja text, the std module command, graph std flags, build program argv). gnu++23, -U__STRICT_ANSI__ and -D__need_infinity_nan were rejected for their effect beyond the defect. The raw clang steps of ci-macos state the same words. Compatibility: an element beginning with -D or /D that contains a space is one word, verbatim, as every release since #234 passed it (SPEC-004 §8 rule 8). Windows: flag_element no longer quotes a backslash the syntax does not read as an escape, normalize_include_flags keeps the spelling of an unchanged element, and the split_flags quoting tests follow the host's reader. --- .../2026-09-17-655-implementation-plan.md | 51 +++++++++++++++++++ .github/workflows/ci-macos.yml | 37 +++++++++----- CHANGELOG.md | 6 +++ docs/04-mcpp-toml.md | 7 ++- docs/specs/manifest-semantics.md | 2 + docs/zh/04-mcpp-toml.md | 6 ++- modules/manifest/src/flag_words.cppm | 28 ++++++++-- modules/manifest/tests/test_flag_words.cpp | 23 ++++++--- src/build/build_program.cppm | 1 + src/build/flags.cppm | 2 + src/build/prepare.cppm | 4 ++ src/modgraph/scanner.cppm | 45 +++++++++++----- src/toolchain/hostflags.cppm | 33 ++++++++++++ src/toolchain/stdmod.cppm | 4 ++ ...ds_reach_the_compiler_and_the_databases.sh | 19 ++++--- tests/unit/test_compile_commands.cpp | 13 ++++- tests/unit/test_hostflags.cpp | 21 ++++++++ 17 files changed, 246 insertions(+), 56 deletions(-) diff --git a/.agents/docs/2026-09-17-655-implementation-plan.md b/.agents/docs/2026-09-17-655-implementation-plan.md index 2a816b4cb..6eb8bea32 100644 --- a/.agents/docs/2026-09-17-655-implementation-plan.md +++ b/.agents/docs/2026-09-17-655-implementation-plan.md @@ -243,6 +243,57 @@ limit, and the second names `xcode-27` and reads version 21.0.0` with `ImageOS=macos27` (run on 2026-09-16T19:16Z). Because the label names an Xcode and its base OS has changed once, every macOS 27 leg asserts that `sw_vers` reports 27. +## 5.2 The first macOS 27 run: the std module does not build + +Both macOS 27 legs failed on their first run (runs 35139712291 and +35139712295): the std module precompile with llvm 22.1.8 stops at +`:1012: use of undeclared identifier 'INFINITY'` (14 errors), in the +raw clang step and in mcpp's own std module build. The maintainer asked for the +fix in the same pull request, with the host kept to the minimum. Three probe +rounds on the `xcode-27` image (#659) read: + +- The 27.0 SDK's `` defines `INFINITY` and `NAN` itself only when + `__has_feature(modules)` is false; with modules on it includes `` + with `__need_infinity_nan` and expects the compiler's header to supply them. +- A module interface unit has `__has_feature(modules)` true; a plain unit does + not. +- `-std=c++23` and `-std=c++26` fail; `-std=gnu++23`, `-U__STRICT_ANSI__`, the + 26.5 SDK, `-D__need_infinity_nan`, and `-DINFINITY=HUGE_VALF + -DNAN=__builtin_nanf("0x7fc00000")` build. `-fbuiltin-headers-in-system-modules` + and `-fno-implicit-module-maps` do not. +- The CLT SDK and the Xcode SDK on the image are the same 27.0 SDK, so the + sysroot choice is not the cause. + +Adopted: `apple_float_macro_words` (hostflags.cppm) states the two macros with +the SDK's own GNU-mode spellings for clang on an Apple target. The alternatives +were rejected on their effect beyond the defect: `gnu++23` changes the language +dialect of every unit, `-U__STRICT_ANSI__` exposes non-standard declarations of +the C library, and `-D__need_infinity_nan` makes the first inclusion of +`` supply only the two macros, so a unit that includes it once loses +`FLT_MAX` (clang's `float.h` lines 15-46). The identical spellings make the +redefinition in the SDK's non-module path silent; plain C and C++ units +including ``, ``, `` and `` build with +`-Wall -Werror` on the 27.0 SDK. The value holds quotes and parentheses, so the +decision is a list of words and each reader quotes it: the ninja compile flags +(`ninja_command_word`), the std module command (single quotes), the +graph-supplied std module flags (`shq`) and a build program's argv (none). The +decision reads the target triple only; it reads nothing from the host. + +The same round found two host-shaped test defects on the Windows runner of the +first push, both in this pull request's own code: `normalize_include_flags` +wrote every word back through `flag_element`, which quoted any backslash, so +Windows paths came back single-quoted; and two `split_flags` tests assumed +POSIX quoting on every host. `flag_element` now quotes a backslash only where +the syntax would read it as an escape, an element whose words did not change +keeps its spelling, and the POSIX-quoting tests are POSIX-only with a Windows +counterpart. + +It also found a compatibility defect of the syntax itself: `NinjaBackend. +QuotesFlagValueWithSpace` (mcpp#234) failed, because `-DT=long long` read as +two words. Every release since #234 passed such an element as one argument, so +the syntax gained one exception (SPEC-004 §8 rule 8): an element that begins +with `-D` or `/D` and contains a space is one word, verbatim. + ## 8. Residuals - `ldflags`, `dialect_cxxflags` and `std-module-flags` keep their current diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index aa670d4c0..643d32d5f 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -72,6 +72,15 @@ jobs: echo "=== Module manifest ===" "$CXX" -print-library-module-manifest-path 2>/dev/null || echo "(not available)" + # THE TWO FLOATING MACROS mcpp STATES FOR AN APPLE TARGET + # (src/toolchain/hostflags.cppm, apple_float_macro_words). The macOS 27 + # SDK leaves INFINITY and NAN to once modules are on, and clang + # 22's header does not supply them in a -std=c++23 compile, so these raw + # compiles state them the way mcpp does. Unquoted words with no spaces, so + # plain word splitting of the variable is exact. + - name: The floating macros mcpp states for an Apple target + run: echo 'APPLE_FLOAT_MACROS=-DINFINITY=HUGE_VALF -DNAN=__builtin_nanf("0x7fc00000")' >> "$GITHUB_ENV" + - name: Test — non-module C++23 compilation run: | WORK=$(mktemp -d) @@ -84,7 +93,7 @@ jobs: return 0; } EOF - "$CXX" -std=c++23 -o hello main.cpp + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -o hello main.cpp ./hello - name: Test — import std (two-stage module compilation) @@ -103,11 +112,11 @@ jobs: echo "=== Step 1: Precompile std module ===" mkdir -p pcm.cache - "$CXX" -std=c++23 -Wno-reserved-module-identifier \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -Wno-reserved-module-identifier \ --precompile "$STD_CPPM" -o pcm.cache/std.pcm echo "=== Step 2: Compile std.pcm → std.o ===" - "$CXX" -std=c++23 -Wno-reserved-module-identifier \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -Wno-reserved-module-identifier \ pcm.cache/std.pcm -c -o std.o echo "=== Step 3: Compile main.cpp with import std ===" @@ -118,7 +127,7 @@ jobs: return 0; } EOF - "$CXX" -std=c++23 -fmodule-file=std=pcm.cache/std.pcm -c main.cpp -o main.o + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -fmodule-file=std=pcm.cache/std.pcm -c main.cpp -o main.o echo "=== Step 4: Link ===" "$CXX" main.o std.o -o hello_modules @@ -141,16 +150,16 @@ jobs: mkdir -p pcm.cache # Build std first - "$CXX" -std=c++23 -Wno-reserved-module-identifier \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -Wno-reserved-module-identifier \ --precompile "$STD_CPPM" -o pcm.cache/std.pcm - "$CXX" -std=c++23 -Wno-reserved-module-identifier \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -Wno-reserved-module-identifier \ pcm.cache/std.pcm -c -o std.o # Build std.compat (depends on std) - "$CXX" -std=c++23 -Wno-reserved-module-identifier \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -Wno-reserved-module-identifier \ -fmodule-file=std=pcm.cache/std.pcm \ --precompile "$STD_COMPAT_CPPM" -o pcm.cache/std.compat.pcm - "$CXX" -std=c++23 -Wno-reserved-module-identifier \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -Wno-reserved-module-identifier \ -fmodule-file=std=pcm.cache/std.pcm \ pcm.cache/std.compat.pcm -c -o std.compat.o @@ -162,7 +171,7 @@ jobs: return 0; } EOF - "$CXX" -std=c++23 \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS \ -fmodule-file=std=pcm.cache/std.pcm \ -fmodule-file=std.compat=pcm.cache/std.compat.pcm \ -c main.cpp -o main.o @@ -178,9 +187,9 @@ jobs: mkdir -p pcm.cache # Build std - "$CXX" -std=c++23 -Wno-reserved-module-identifier \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -Wno-reserved-module-identifier \ --precompile "$STD_CPPM" -o pcm.cache/std.pcm - "$CXX" -std=c++23 -Wno-reserved-module-identifier \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -Wno-reserved-module-identifier \ pcm.cache/std.pcm -c -o std.o # User module: greeter @@ -193,9 +202,9 @@ jobs: } } EOF - "$CXX" -std=c++23 -fmodule-file=std=pcm.cache/std.pcm \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -fmodule-file=std=pcm.cache/std.pcm \ --precompile greeter.cppm -o pcm.cache/greeter.pcm - "$CXX" -std=c++23 -fmodule-file=std=pcm.cache/std.pcm \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS -fmodule-file=std=pcm.cache/std.pcm \ pcm.cache/greeter.pcm -c -o greeter.o # Main @@ -207,7 +216,7 @@ jobs: return 0; } EOF - "$CXX" -std=c++23 \ + "$CXX" -std=c++23 $APPLE_FLOAT_MACROS \ -fmodule-file=std=pcm.cache/std.pcm \ -fmodule-file=greeter=pcm.cache/greeter.pcm \ -c main.cpp -o main.o diff --git a/CHANGELOG.md b/CHANGELOG.md index f1bb65716..c2ea86d6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,12 @@ libarchive、bzip2、lz4、xz、zlib 与 zstd 的每个 C 文件都无法解析 这样的元素。 - Windows 上 `shell_quote_arg` 按 MSVCRT 规则加倍引号前与结尾处的反斜杠,以 `\` 结尾的 词不再吞掉闭合引号。 +- **macOS 27 上 `import std` 可以构建。** 27.0 SDK 的 `` 在模块开启时把 `INFINITY` 与 + `NAN` 交给 ``,而 clang 22 的头在 `-std=c++23` 下不提供它们,std 模块预编译因此失败 + (`:1012: use of undeclared identifier 'INFINITY'`)。对 Apple 目标上的 clang,mcpp 以 + SDK 自身的拼写陈述这两个宏(`-DINFINITY=HUGE_VALF -DNAN=__builtin_nanf("0x7fc00000")`), + 普通单元中的重定义逐词相同,不产生警告。判据只读目标三元组。 +- **以 `-D` 或 `/D` 开头且含空格的元素仍是一个参数**,按原样传入,与 mcpp#234 以来各版本相同。 - **CI 覆盖 macOS 27。** `ci-macos` 与 `ci-macos-e2e` 在 `macos-15` 与 macOS 27 上各跑一遍, `ci-fresh-install` 的 xlings 与 Homebrew 两条安装渠道在 `macos-14` 与 macOS 27 上各跑一遍。 GitHub 以预览标签 `xcode-27` 提供 macOS 27(无 `macos-27` 标签),每条 macOS 27 的腿断言 diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index dc65ac4a7..c48709fc2 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -430,7 +430,8 @@ written: `[build]`, `[targets.]`, a `flags` glob entry, a feature, a |---|---| | `"-O2 -g"` | `-O2`, `-g` | | `"-include config.h"` | `-include`, `config.h` | -| `"'-DNAME=a b'"` or `"-DNAME=\"a b\""` | `-DNAME=a b` | +| `"'-DNAME=a b'"` or `"-I\"my dir\""` | `-DNAME=a b`, `-Imy dir` | +| `"-DNAME=long long"` | `-DNAME=long long` (see below) | | `"-DNAME=\\\"text\\\""` | `-DNAME="text"` (a string literal) | | `"-I/opt/my\\ dir/include"` | `-I/opt/my dir/include` | | `"-IC:\\sdk\\include"` | `-IC:\sdk\include` | @@ -445,7 +446,9 @@ escapes): - outside quotes, a backslash before a space, a tab, `"`, `'` or `\` stands for that character; any other backslash is literal; - quoted and unquoted pieces that touch form one word; -- `$`, `*`, `;`, `|` and the other shell operators have no meaning. +- `$`, `*`, `;`, `|` and the other shell operators have no meaning; +- an element that begins with `-D` or `/D` and contains a space is one word, + taken verbatim, as in every earlier release. A `defines` entry is one value and is not read by this syntax: `defines = ["NAME=\"text\""]` passes the single word `-DNAME="text"`. `ldflags`, diff --git a/docs/specs/manifest-semantics.md b/docs/specs/manifest-semantics.md index 326225155..88ae37705 100644 --- a/docs/specs/manifest-semantics.md +++ b/docs/specs/manifest-semantics.md @@ -352,6 +352,8 @@ feature、`[profile.]`、`[target..build]`、xpkg 描述符,以及 6. 未闭合的引号区域延伸到元素末尾。 7. 除上述字符外,任何字符没有特殊含义;实现**禁止**对 `$`、`*`、`?`、`;`、`|`、`&`、`<`、 `>`、`` ` ``、`~` 做展开或解释。 +8. 例外:以 `-D` 或 `/D` 开头且含空格(U+0020)的元素代表一个词,即元素文本本身,规则 1 至 7 + 不适用。自 mcpp#234 起,每个版本在宿主读取之前都把这样的元素整体加引号,该例外保持其含义不变。 `defines` 的一个条目 `X` 是一个值:它代表一个词 `-D` 与 `X` 的拼接,**禁止**按上述规则 读取。实现向这三个列表插入一个词 `w` 时,**必须**使用一个按上述规则读回恰为 `w` 的拼写。 diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index 6e1e03cb3..37d9c4b93 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -382,7 +382,8 @@ glob 条目、feature、`[target..build]`、xpkg 描述符,以及构 |---|---| | `"-O2 -g"` | `-O2`、`-g` | | `"-include config.h"` | `-include`、`config.h` | -| `"'-DNAME=a b'"` 或 `"-DNAME=\"a b\""` | `-DNAME=a b` | +| `"'-DNAME=a b'"` 或 `"-I\"my dir\""` | `-DNAME=a b`、`-Imy dir` | +| `"-DNAME=long long"` | `-DNAME=long long`(见下) | | `"-DNAME=\\\"text\\\""` | `-DNAME="text"`(字符串字面量) | | `"-I/opt/my\\ dir/include"` | `-I/opt/my dir/include` | | `"-IC:\\sdk\\include"` | `-IC:\sdk\include` | @@ -395,7 +396,8 @@ glob 条目、feature、`[target..build]`、xpkg 描述符,以及构 - `"..."` 按字面取,只有 `\"` 与 `\\` 分别代表 `"` 与 `\`; - 引号之外,反斜杠后跟空格、制表符、`"`、`'` 或 `\` 时代表该字符,其余反斜杠按字面; - 相邻的带引号与不带引号的片段组成一个词; -- `$`、`*`、`;`、`|` 等 shell 运算符没有特殊含义。 +- `$`、`*`、`;`、`|` 等 shell 运算符没有特殊含义; +- 以 `-D` 或 `/D` 开头且含空格的元素是一个词,按原样取,与此前各版本相同。 `defines` 的一个条目是一个值,不按此写法读取:`defines = ["NAME=\"text\""]` 传入的是 一个词 `-DNAME="text"`。`ldflags`、`dialect_cxxflags` 与 `std-module-flags` 不在本节范围内。 diff --git a/modules/manifest/src/flag_words.cppm b/modules/manifest/src/flag_words.cppm index 50da8af58..28172dfde 100644 --- a/modules/manifest/src/flag_words.cppm +++ b/modules/manifest/src/flag_words.cppm @@ -27,10 +27,17 @@ // - outside quotes, a backslash before a space, a tab, `"`, `'` or `\` // stands for that character, and any other backslash is literal; // - quoted and unquoted pieces that touch form one word, so `""` is an -// empty word and `-DV="a b"` is the single word `-DV=a b`. +// empty word and `-I"my dir"` is the single word `-Imy dir`. // -// `$`, `*`, `;` and the other shell operators have no meaning. The one -// spelling the published index relies on, libarchive's +// `$`, `*`, `;` and the other shell operators have no meaning. +// +// ONE EXCEPTION, KEPT FOR COMPATIBILITY. An element that begins with `-D` or +// `/D` and contains a space is one word, taken verbatim. Every release since +// mcpp#234 quoted such an element whole before the host read it, so +// `-DT=long long` has always been the single argument `-DT=long long` on every +// host; reading it as two words would change working manifests. +// +// The one spelling the published index relies on, libarchive's // `-DPLATFORM_CONFIG_H=\"mcpp_libarchive_config.h\"`, reads as the word // `-DPLATFORM_CONFIG_H="mcpp_libarchive_config.h"`, which is what both hosts // passed before. pkg-config's `\ ` for a space in a path reads as a space. @@ -81,6 +88,8 @@ bool escapable_outside_quotes(char c) { } // namespace std::vector flag_words(std::string_view s) { + if ((s.starts_with("-D") || s.starts_with("/D")) && s.find(' ') != std::string_view::npos) + return {std::string(s)}; std::vector out; std::string word; bool started = false; @@ -125,8 +134,17 @@ std::vector flag_words(const std::vector& elements) { } std::string flag_element(std::string_view word) { - const bool plain = !word.empty() - && word.find_first_of(" \t\"'\\") == std::string_view::npos; + // A word with a space that begins with -D or /D is its own element (the + // exception above); quoting it as well would be equally correct, and + // leaving it keeps the spelling every release wrote. + if ((word.starts_with("-D") || word.starts_with("/D")) && word.find(' ') != std::string_view::npos) + return std::string(word); + // Plain means the syntax reads the word back unchanged: no blank, no + // quote, and no backslash in front of a character it would escape. A + // Windows path (`C:\sdk\include`) is plain and keeps its spelling. + bool plain = !word.empty() && word.find_first_of(" \t\"'") == std::string_view::npos; + for (std::size_t i = 0; plain && i + 1 < word.size(); ++i) + if (word[i] == '\\' && escapable_outside_quotes(word[i + 1])) plain = false; if (plain) return std::string(word); // Single quotes hold everything except `'`, which closes the region, is // written as an escaped quote outside it, and reopens it. diff --git a/modules/manifest/tests/test_flag_words.cpp b/modules/manifest/tests/test_flag_words.cpp index 476bc7725..e997fc238 100644 --- a/modules/manifest/tests/test_flag_words.cpp +++ b/modules/manifest/tests/test_flag_words.cpp @@ -23,8 +23,13 @@ TEST(FlagWords, TheSyntaxTable) { // Quotes inside a word are quoting, as in `sh`. {R"(-DMID="mid")", {"-DMID=mid"}}, {"-DSQ='sq'", {"-DSQ=sq"}}, - {R"(-DV="a b")", {"-DV=a b"}}, {"'-DV=a b'", {"-DV=a b"}}, + // The compatibility exception: a -D or /D element with a space is one + // word, verbatim (mcpp#234), quotes and backslashes included. + {"-DT=long long", {"-DT=long long"}}, + {R"(-DV="a b")", {R"(-DV="a b")"}}, + {"/DV=a b", {"/DV=a b"}}, + {"-DA=1 -DB=2", {"-DA=1 -DB=2"}}, // compat.lua's descriptor: one element, two words. {"-include mcpp_lua_platform_config.h", {"-include", "mcpp_lua_platform_config.h"}}, // pkg-config writes a space in a path as `\ `. @@ -32,6 +37,8 @@ TEST(FlagWords, TheSyntaxTable) { // A backslash before an ordinary character is literal: Windows paths. {R"(-IC:\Users\x\include)", {R"(-IC:\Users\x\include)"}}, {R"("-IC:\Program Files\x")", {R"(-IC:\Program Files\x)"}}, + // Without a space the syntax applies to a -D element as to any other. + {R"(-DV="mid")", {"-DV=mid"}}, // Escapes a backslash can make outside and inside double quotes. {R"(-DV=a\\b)", {R"(-DV=a\b)"}}, {R"("a\"b\\c\d")", {R"(a"b\c\d)"}}, @@ -49,7 +56,7 @@ TEST(FlagWords, TheSyntaxTable) { {"''", {""}}, {R"(-DV="")", {"-DV="}}, // An unterminated quote extends to the end of the element. - {R"(-DV="a b)", {"-DV=a b"}}, + {R"(-I"a b)", {"-Ia b"}}, // A trailing backslash has nothing to escape. {R"(-DV=a\)", {R"(-DV=a\)"}}, }; @@ -58,12 +65,13 @@ TEST(FlagWords, TheSyntaxTable) { } TEST(FlagWords, AListIsTheConcatenationOfItsElements) { - const std::vector list{"-include x.h", R"(-DV="a b")", "", "-O2"}; - EXPECT_EQ(flag_words(list), (Words{"-include", "x.h", "-DV=a b", "-O2"})); + const std::vector list{"-include x.h", R"(-I"my dir")", "", "-O2"}; + EXPECT_EQ(flag_words(list), (Words{"-include", "x.h", "-Imy dir", "-O2"})); } TEST(FlagWords, APlainWordIsItsOwnElement) { - for (std::string_view w : {"-O2", "-DV=1", "-IC:/x", "/std:c++latest", "-Wl,-z,defs", "$x"}) + for (std::string_view w : {"-O2", "-DV=1", "-IC:/x", "/std:c++latest", "-Wl,-z,defs", "$x", + R"(-IC:\sdk\include)", R"(C:\dir\)"}) EXPECT_EQ(flag_element(w), w); } @@ -71,9 +79,10 @@ TEST(FlagWords, APlainWordIsItsOwnElement) { // the element it writes reads back as exactly that one word. TEST(FlagWords, AnElementReadsBackAsItsWord) { Words words{"", " ", "a b", "'", "\"", "\\", "a\\", "\\\\srv\\share", "it's", - "-DV=\"def\"", "-DV='c'", "x\ty", "a\"b'c\\d e"}; + "-DV=\"def\"", "-DV='c'", "x\ty", "a\"b'c\\d e", "-DT=long long", + "-DV=\"a b\"", "/DV=x y"}; // A deterministic walk over the characters the syntax gives meaning to. - const std::string alphabet = "ab \t\"'\\$"; + const std::string alphabet = "ab \t\"'\\$-D"; std::uint32_t state = 655; for (int n = 0; n < 2000; ++n) { std::string w; diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index a064020cb..8b807fb06 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -509,6 +509,7 @@ std::vector host_base_flags(const mcpp::toolchain::Toolchain& tc, const mcpp::toolchain::PathEscape plain = mcpp::toolchain::no_escape; auto f = mcpp::toolchain::host_compile_tokens(tc, opt, plain); + for (auto& w : mcpp::toolchain::apple_float_macro_words(tc)) f.push_back(std::move(w)); for (auto& t : mcpp::toolchain::host_link_tokens(tc, opt, plain)) f.push_back(t); return f; diff --git a/src/build/flags.cppm b/src/build/flags.cppm index 18f6bff55..e0c07bedf 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -741,6 +741,8 @@ CompileFlags compute_flags(const BuildPlan& plan) { hopt.appleSdkCxxHeaders = plan.toolchain.appleSdkCxxHeaders; compile_toolchain_flags = mcpp::toolchain::render_tokens( mcpp::toolchain::host_compile_tokens(plan.toolchain, hopt, ninjaEsc)); + for (auto const& w : mcpp::toolchain::apple_float_macro_words(plan.toolchain)) + compile_toolchain_flags += " " + ninja_command_word(w); } else { // Skipped entirely, not filtered. What this block emits is the // HOST's world reconstructed by hand — libc++'s headers, glibc's diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 6439566aa..05e38d622 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -12114,6 +12114,10 @@ prepare_build(bool print_fingerprint, const auto q = " " + mcpp::xlings::shq(t); if (flags.find(q) == std::string::npos) flags += q; } + for (auto& t : mcpp::toolchain::apple_float_macro_words(*tc)) { + const auto q = " " + mcpp::xlings::shq(t); + if (flags.find(q) == std::string::npos) flags += q; + } } // The same words the package's own units receive from this list // (mcpp.manifest.flag_words), one quoted word each. diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index 16a9be18f..2308bde3f 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -727,24 +727,41 @@ void normalize_include_flags(const std::filesystem::path& root, static constexpr std::string_view kIncPrefixes[] = {"-I", "-iquote", "-isystem", "-idirafter", "-iprefix", "-L"}; - auto words = mcpp::manifest::flag_words(flags); - for (std::size_t i = 0; i < words.size(); ++i) { - for (auto pre : kIncPrefixes) { - if (words[i] == pre && i + 1 < words.size()) { // separated - rewrite_rel(words[i + 1], root); - ++i; - break; + // Words are rewritten per element, and an element none of whose words + // changed keeps its spelling. A separated prefix whose path is the first + // word of the next element is rewritten there. + bool pendingPath = false; + for (auto& element : flags) { + auto words = mcpp::manifest::flag_words(element); + bool changed = false; + for (std::size_t i = 0; i < words.size(); ++i) { + if (pendingPath) { + auto abs = rewrite_rel_copy(words[i], root); + changed = changed || abs != words[i]; + words[i] = std::move(abs); + pendingPath = false; + continue; } - if (words[i].size() > pre.size() && words[i].starts_with(pre)) { // joined - std::string tail = words[i].substr(pre.size()); - std::string abs = rewrite_rel_copy(tail, root); - if (abs != tail) words[i] = std::string(pre) + abs; - break; + for (auto pre : kIncPrefixes) { + if (words[i] == pre) { // separated + pendingPath = true; + break; + } + if (words[i].size() > pre.size() && words[i].starts_with(pre)) { // joined + std::string tail = words[i].substr(pre.size()); + std::string abs = rewrite_rel_copy(tail, root); + if (abs != tail) { words[i] = std::string(pre) + abs; changed = true; } + break; + } } } + if (!changed) continue; + element.clear(); + for (auto const& w : words) { + if (!element.empty()) element += ' '; + element += mcpp::manifest::flag_element(w); + } } - flags.clear(); - for (auto const& w : words) flags.push_back(mcpp::manifest::flag_element(w)); } std::expected scan_file(const std::filesystem::path& file, diff --git a/src/toolchain/hostflags.cppm b/src/toolchain/hostflags.cppm index af1145311..d7d8f45f7 100644 --- a/src/toolchain/hostflags.cppm +++ b/src/toolchain/hostflags.cppm @@ -155,6 +155,32 @@ struct HostFlagOptions { bool appleSdkCxxHeaders = false; }; +// THE TWO C FLOATING MACROS AN APPLE SDK LEAVES TO , as argv words: +// `-DINFINITY=HUGE_VALF` and `-DNAN=__builtin_nanf("0x7fc00000")` for a clang +// compiling for an Apple target, and nothing otherwise. +// +// The macOS 27.0 SDK's defines INFINITY and NAN itself only when +// `__has_feature(modules)` is false. With modules on it includes +// with `__need_infinity_nan` set and expects the compiler's header to supply +// them, and clang 22's does not in a strict (`-std=c++23`) compile. A module +// interface unit has modules on, so libc++'s std module stopped building +// (measured on the `xcode-27` image, macOS 27.0 26A5406e, llvm 22.1.8: +// `:1012: use of undeclared identifier 'INFINITY'`; the 26.5 SDK and +// `-std=gnu++23` both build). +// +// The values are the SDK's own GNU-mode spellings, token for token, so where +// does define them the redefinition is identical and silent, and +// clang's <__float_infinity_nan.h> undefines before it defines. They are +// stated for every Apple compile rather than only for module units because +// the condition is the SDK's, and one rule is easier to hold than a +// per-unit one. Measured: plain C and C++ units including , , +// and build with `-Wall -Werror` on the 27.0 SDK. +// +// Words, not rendered text: the NAN value holds quotes and parentheses, and +// each reader (ninja text, the std module's shell command, a build program's +// argv) quotes a word its own way. +std::vector apple_float_macro_words(const Toolchain& tc); + // Host-compile flags as argv tokens, in the order the string channels have // always emitted them (clang cfg → deployment target → C library), so // rendering reproduces today's command lines byte for byte. @@ -211,6 +237,13 @@ std::optional orphaned_reference( namespace mcpp::toolchain { +std::vector apple_float_macro_words(const Toolchain& tc) { + if (tc.compiler != CompilerId::Clang) return {}; + auto tt = triple::parse(tc.targetTriple); + if (!tt || !tt->is_apple()) return {}; + return {"-DINFINITY=HUGE_VALF", "-DNAN=__builtin_nanf(\"0x7fc00000\")"}; +} + std::vector host_compile_tokens(const Toolchain& tc, const HostFlagOptions& opt, const PathEscape& esc) { diff --git a/src/toolchain/stdmod.cppm b/src/toolchain/stdmod.cppm index 3927e3f7c..3fbf95136 100644 --- a/src/toolchain/stdmod.cppm +++ b/src/toolchain/stdmod.cppm @@ -289,6 +289,10 @@ std::expected derive_std_module( hopt.cAbiPrebuilt = tc.cAbiPrebuilt; std::string sysroot_flag = render_tokens(host_compile_tokens(tc, hopt, shellEsc)); + // Single-quoted for the shell that runs this command; the value holds `"` + // and parentheses. Apple targets only (hostflags.cppm). + for (auto const& w : apple_float_macro_words(tc)) + sysroot_flag += std::format(" '{}'", w); // Deployment target appended here rather than passed to the producer // ONLY to keep this command string byte-identical to what earlier diff --git a/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh b/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh index 8f5ea5e1a..30d8bb168 100644 --- a/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh +++ b/tests/e2e/736_compile_flag_words_reach_the_compiler_and_the_databases.sh @@ -34,9 +34,10 @@ cxxflags = [ "-DV_MID=\"mid\"", # -DV_MID=mid "-DV_SQ='sq'", # -DV_SQ=sq "'-DV_SP=a b'", # -DV_SP=a b - "-DV_BS=a\\ b", # -DV_BS=a b (pkg-config's escaped space) "-DV_DOL=a$b", # -DV_DOL=a$b - "-DV_A=1 -DV_B=2", # -DV_A=1 -DV_B=2 + "-O0 -DV_A=1", # -O0 -DV_A=1 + "-DV_B=long long", # -DV_B=long long (a -D element with a space + # is one word, verbatim: mcpp#234) ] defines = [ "V_DEF=\"def\"", # -DV_DEF="def" @@ -55,7 +56,6 @@ int main() { show("V_MID", STR(V_MID)); show("V_SQ", STR(V_SQ)); show("V_SP", STR(V_SP)); - show("V_BS", STR(V_BS)); show("V_DOL", STR(V_DOL)); show("V_A", STR(V_A)); show("V_B", STR(V_B)); @@ -75,22 +75,21 @@ V_ESC=["esc.h"] V_MID=[mid] V_SQ=[sq] V_SP=[a b] -V_BS=[a b] V_DOL=[a$b] V_A=[1] -V_B=[2] +V_B=[long long] V_DEF=["def"] V_DSP=[a b] EOF -for line in $(seq 1 10); do +for line in $(seq 1 9); do want=$(sed -n "${line}p" want.txt) grep -qxF -- "$want" run.txt || fail "A: the program did not print $want" run.txt done # B and C. What the databases list. The expected words are the ones above. cat > words.json <<'EOF' -["-DV_ESC=\"esc.h\"", "-DV_MID=mid", "-DV_SQ=sq", "-DV_SP=a b", "-DV_BS=a b", - "-DV_DOL=a$b", "-DV_A=1", "-DV_B=2", "-DV_DEF=\"def\"", "-DV_DSP=a b"] +["-DV_ESC=\"esc.h\"", "-DV_MID=mid", "-DV_SQ=sq", "-DV_SP=a b", + "-DV_DOL=a$b", "-O0", "-DV_A=1", "-DV_B=long long", "-DV_DEF=\"def\"", "-DV_DSP=a b"] EOF cdb=$(find target -name compile_commands.json | head -1) [ -n "$cdb" ] || cdb=compile_commands.json @@ -136,11 +135,11 @@ if not contains(flat): PY # D. The first plan names the elements whose words changed, and only those. -for element in "V_DEF=\"def\"" "-DV_BS=a\\ b" "-DV_DOL=a\$b" "-DV_A=1 -DV_B=2"; do +for element in "V_DEF=\"def\"" "-DV_DOL=a\$b"; do grep -qF -- "element '$element' reaches the compiler as" build1.log \ || fail "D: the first plan does not name $element" build1.log done -for element in "-DV_MID=\"mid\"" "V_DSP=a b" "-DV_ESC="; do +for element in "-DV_MID=\"mid\"" "V_DSP=a b" "-DV_ESC=" "-DV_B=long long" "-O0 -DV_A=1"; do if grep -F -- "reaches the compiler as" build1.log | grep -qF -- "element '$element"; then fail "D: the first plan names $element, whose words did not change" build1.log fi diff --git a/tests/unit/test_compile_commands.cpp b/tests/unit/test_compile_commands.cpp index db7964983..8f2a8bc3a 100644 --- a/tests/unit/test_compile_commands.cpp +++ b/tests/unit/test_compile_commands.cpp @@ -268,7 +268,12 @@ bool has_edge_quote(std::string_view s) { } // namespace +// The host's own reader: single quotes are quoting for `sh` and literal +// characters for the MSVCRT rules, so this row holds on POSIX hosts only. TEST(CompileCommandsArgs, StripsPosixQuotingFromAToken) { + if constexpr (mcpp::platform::is_windows) { + GTEST_SKIP() << "single quotes are literal under the MSVCRT rules"; + } auto out = mcpp::build::split_flags("-O2 '-fprebuilt-module-path=/a/b' -g"); ASSERT_EQ(out.size(), 3u); EXPECT_EQ(out[1], "-fprebuilt-module-path=/a/b"); @@ -287,8 +292,12 @@ TEST(CompileCommandsArgs, StripsWindowsQuotingFromAToken) { // quotes, so the un-escape has to happen INSIDE the quotes — do it in the // wrong order and the token splits exactly here. TEST(CompileCommandsArgs, AQuotedPathWithASpaceStaysOneToken) { - auto out = mcpp::build::split_flags( - "-O2 '-fmodule-file=std=/tmp/my$ project/std.pcm' -g"); + // The quotes are the host's: flags.cppm quotes with `'` on POSIX and `"` + // under the MSVCRT rules. + const std::string_view line = mcpp::platform::is_windows + ? "-O2 \"-fmodule-file=std=/tmp/my$ project/std.pcm\" -g" + : "-O2 '-fmodule-file=std=/tmp/my$ project/std.pcm' -g"; + auto out = mcpp::build::split_flags(line); ASSERT_EQ(out.size(), 3u) << "the quoted path was split"; EXPECT_EQ(out[1], "-fmodule-file=std=/tmp/my project/std.pcm"); for (auto const& t : out) EXPECT_FALSE(has_edge_quote(t)) << t; diff --git a/tests/unit/test_hostflags.cpp b/tests/unit/test_hostflags.cpp index ceb56a1f9..f08d0284c 100644 --- a/tests/unit/test_hostflags.cpp +++ b/tests/unit/test_hostflags.cpp @@ -723,3 +723,24 @@ TEST(HostFlags, TheCxxLayerDecidesWhoseLibcxxHeadersAreEmitted) { EXPECT_TRUE(has(d, "-nostdinc++")); } + +// The floating macros an Apple SDK leaves to once modules are on +// (apple_float_macro_words). Stated for clang on an Apple target and for +// nothing else, as the SDK's own spellings, and quotable by every reader. +TEST(HostFlags, AppleFloatMacrosAreStatedForClangOnAppleTargetsOnly) { + const std::vector words{ + "-DINFINITY=HUGE_VALF", "-DNAN=__builtin_nanf(\"0x7fc00000\")"}; + auto tc = [](CompilerId id, std::string triple) { + mcpp::toolchain::Toolchain t; + t.compiler = id; + t.targetTriple = std::move(triple); + return t; + }; + EXPECT_EQ(mcpp::toolchain::apple_float_macro_words(tc(CompilerId::Clang, "arm64-apple-darwin27.0.0")), words); + EXPECT_EQ(mcpp::toolchain::apple_float_macro_words(tc(CompilerId::Clang, "aarch64-macos")), words); + EXPECT_EQ(mcpp::toolchain::apple_float_macro_words(tc(CompilerId::Clang, "arm64-apple-ios18.0")), words); + EXPECT_TRUE(mcpp::toolchain::apple_float_macro_words(tc(CompilerId::Clang, "x86_64-linux-gnu")).empty()); + EXPECT_TRUE(mcpp::toolchain::apple_float_macro_words(tc(CompilerId::Clang, "wasm32-emscripten")).empty()); + EXPECT_TRUE(mcpp::toolchain::apple_float_macro_words(tc(CompilerId::GCC, "x86_64-linux-gnu")).empty()); + EXPECT_TRUE(mcpp::toolchain::apple_float_macro_words(tc(CompilerId::MSVC, "x86_64-pc-windows-msvc")).empty()); +} From e90674fb420e0927cc1e7087bbbc7ed18043745b Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 17 Sep 2026 04:01:12 +0800 Subject: [PATCH 5/6] CI macOS 27: wrap a bootstrap mcpp that predates the float macro fix The raw clang steps now build import std on macOS 27 (run 35142520288). The self-host step failed because the bootstrap, 2026.9.16.1, predates the fix. setup-macos-llvm wraps a bootstrap older than 2026.9.17.1 on macOS 27 or later and passes the same two words through CCC_OVERRIDE_OPTIONS; the freshly built binary runs unwrapped, and the wrapper retires with the bootstrap pin. --- .../2026-09-17-655-implementation-plan.md | 12 +++++++++ .github/actions/setup-macos-llvm/action.yml | 27 +++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/.agents/docs/2026-09-17-655-implementation-plan.md b/.agents/docs/2026-09-17-655-implementation-plan.md index 6eb8bea32..f6d4ced04 100644 --- a/.agents/docs/2026-09-17-655-implementation-plan.md +++ b/.agents/docs/2026-09-17-655-implementation-plan.md @@ -294,6 +294,18 @@ two words. Every release since #234 passed such an element as one argument, so the syntax gained one exception (SPEC-004 §8 rule 8): an element that begins with `-D` or `/D` and contains a space is one word, verbatim. +The second macOS 27 run (`10224fb8`) confirmed the fix at the compiler: the raw +clang steps print `C++23 import std works on macOS via xlings LLVM!`. It then +failed where the job builds mcpp with the bootstrap binary, 2026.9.16.1, which +predates the fix and so cannot build its std module on macOS 27. The +`setup-macos-llvm` action wraps a bootstrap older than 2026.9.17.1 on macOS 27 +or later: the wrapper passes the same two words to clang through +`CCC_OVERRIDE_OPTIONS` (quiet form, measured on clang 22.1.8: both definitions +arrive exactly and nothing is written to stderr). The binary the job builds +runs unwrapped, so the unit tests, the second self-host build and the e2e suite +measure the engine's own fix; e2e 252 keeps its real old-client leg through the +wrapper. The wrapper retires when the bootstrap pin reaches 2026.9.17.1. + ## 8. Residuals - `ldflags`, `dialect_cxxflags` and `std-module-flags` keep their current diff --git a/.github/actions/setup-macos-llvm/action.yml b/.github/actions/setup-macos-llvm/action.yml index 5ff93c043..caa13212c 100644 --- a/.github/actions/setup-macos-llvm/action.yml +++ b/.github/actions/setup-macos-llvm/action.yml @@ -75,6 +75,33 @@ runs: # `xlings install mcpp` resolved 0.0.105 here on a cold cache — below # the index floor, which makes every descriptor read return nothing. MCPP=$(bash "$GITHUB_WORKSPACE/.github/tools/install_pinned_mcpp.sh" "$GITHUB_WORKSPACE") + + # A BOOTSTRAP OLDER THAN THE macOS 27 FIX, ON macOS 27, IS WRAPPED. + # + # The 27.0 SDK leaves INFINITY and NAN to once modules are on, + # and mcpp states them itself from 2026.9.17.1 (hostflags.cppm, + # apple_float_macro_words). A released mcpp older than that cannot + # build its std module here, so it could not build the source that + # carries the fix. The wrapper hands clang the same two words through + # CCC_OVERRIDE_OPTIONS (a leading `#` keeps clang quiet) for the + # bootstrap binary only; the binary this job builds runs unwrapped, so + # the fix is what every later step measures. It retires itself: once the + # bootstrap pin is 2026.9.17.1 or later, nothing is wrapped. + major=$(sw_vers -productVersion | cut -d. -f1) + boot=$("$MCPP" --version 2>/dev/null | awk '{print $2}') + if [ "$major" -ge 27 ] && [ -n "$boot" ] \ + && [ "$(printf '%s\n' "$boot" 2026.9.17.1 | sort -V | head -1)" = "$boot" ] \ + && [ "$boot" != 2026.9.17.1 ]; then + wrapper="$RUNNER_TEMP/mcpp-bootstrap-macos27" + { + echo '#!/bin/sh' + echo "CCC_OVERRIDE_OPTIONS='# +-DINFINITY=HUGE_VALF +-DNAN=__builtin_nanf(\"0x7fc00000\")' exec \"$MCPP\" \"\$@\"" + } > "$wrapper" + chmod +x "$wrapper" + echo "bootstrap mcpp $boot on macOS $major: wrapped ($wrapper)" + cat "$wrapper" + MCPP="$wrapper" + fi echo "MCPP=$MCPP" >> "$GITHUB_ENV" echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" From 5bab34389626c9287448e11f2c7ad2ad3eeccc5d Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 17 Sep 2026 04:34:49 +0800 Subject: [PATCH 6/6] Review: the diagnostic block sits above the defines comment; correct the 252 claim; index reading with the -D exception - report_flag_words_changes was inserted between fold_build_defines_into_flags and its comment; it now precedes the comment. - The record claimed e2e 252 keeps its real old-client leg on macOS through the wrapper. It does not, on any macOS leg including main: MCPP_BOOT is the xlings shim. Corrected and listed as a residual, with dialect promotion reading words. - The index reading script applies the -D exception: 285 literals, 0 differ. --- .../2026-09-17-655-implementation-plan.md | 28 +++++++++++++------ .agents/docs/2026-09-17-655-index-readings.py | 3 ++ src/build/prepare.cppm | 24 ++++++++-------- 3 files changed, 35 insertions(+), 20 deletions(-) diff --git a/.agents/docs/2026-09-17-655-implementation-plan.md b/.agents/docs/2026-09-17-655-implementation-plan.md index f6d4ced04..788ba9236 100644 --- a/.agents/docs/2026-09-17-655-implementation-plan.md +++ b/.agents/docs/2026-09-17-655-implementation-plan.md @@ -82,10 +82,11 @@ The compatibility cost is bounded by measurement rather than by argument: - every string literal of the published index that starts like a flag was read under the new syntax and under both hosts' previous readings (mcpplibs/mcpp-index `c176883`, 285 literals, - `2026-09-17-655-index-readings.py`): 9 differ, and all 9 are pieces of the CMake command line in - `compat.mysql-connector-cpp.lua`'s install hook, not elements of a flag list. - libarchive's `\"` and compat.lua's packed `-include` read the same under all - three (§4 K6); + `2026-09-17-655-index-readings.py`): before the `-D` exception (§5.2) 9 + differed, all pieces of the CMake command line in + `compat.mysql-connector-cpp.lua`'s install hook rather than elements of a flag + list; with the exception, 0 differ. libarchive's `\"` and compat.lua's packed + `-include` read the same under all three (§4 K6); - an element whose reading does change is named once, with both readings, on the first plan after the upgrade (§2.5). @@ -165,8 +166,8 @@ reason). | E7 | `build/flag-words` note on the first plan | E1 | branch: e2e 736 D and E | | E8 | SPEC-004 §8, SPEC-005 R3.7, docs/04 and docs/30 with their Chinese mirrors, CHANGELOG | E1-E7 | branch | | E9 | version 2026.9.17.1 | - | branch | -| E10 | CI green on every workflow of the pull request | E1-E9, E11 | todo | -| E11 | macOS 27 legs (maintainer request, 2026-09-17; label measured in §5.1): `ci-macos` and `ci-macos-e2e` run on `macos-15` and macOS 27; `ci-fresh-install` runs its xlings and Homebrew channels on `macos-14` and macOS 27; each macOS 27 leg asserts `sw_vers` major 27 | - | branch | +| E10 | CI green on every workflow of the pull request | E1-E9, E11 | branch: `e90674fb` 12/12 runs green (macOS 27 unit 121/121, macOS 27 e2e 182 passed 0 failed, Windows unit and e2e) | +| E11 | macOS 27 legs and the macOS 27 std module fix (maintainer request, 2026-09-17; §5.1, §5.2): `ci-macos` and `ci-macos-e2e` run on `macos-15` and macOS 27; `ci-fresh-install` runs its xlings and Homebrew channels on `macos-14` and macOS 27; each macOS 27 leg asserts `sw_vers` major 27 | - | branch | ### 3.2 Release and ecosystem, in order @@ -303,11 +304,22 @@ or later: the wrapper passes the same two words to clang through `CCC_OVERRIDE_OPTIONS` (quiet form, measured on clang 22.1.8: both definitions arrive exactly and nothing is written to stderr). The binary the job builds runs unwrapped, so the unit tests, the second self-host build and the e2e suite -measure the engine's own fix; e2e 252 keeps its real old-client leg through the -wrapper. The wrapper retires when the bootstrap pin reaches 2026.9.17.1. +measure the engine's own fix. The wrapper retires when the bootstrap pin +reaches 2026.9.17.1. (Corrected in review: an earlier sentence here said e2e 252 +keeps its real old-client leg through the wrapper. It does not, and did not +before this pull request on any macOS leg: `MCPP_BOOT` is the xlings shim, +which answers no `--version` inside the test's temporary directory, so the test +prints its NOTE and runs only its static half. Recorded in §8.) ## 8. Residuals +- e2e 252's real old-client leg does not run on the macOS legs (main included): + `MCPP_BOOT` is the xlings shim, not a store binary. Pointing it at + `~/.xlings/data/xpkgs/xim-x-mcpp//bin/mcpp` would make it run. +- Dialect promotion now reads words, so a packed element such as + `"-fno-exceptions -fno-rtti"` is promoted into the std module's dialect set; + the compiler already received both flags, so the std BMI now matches its + importers where it did not. - `ldflags`, `dialect_cxxflags` and `std-module-flags` keep their current meaning (documented in docs/04). Moving `ldflags` needs the link path's rendered text separated from manifest elements first (F1). diff --git a/.agents/docs/2026-09-17-655-index-readings.py b/.agents/docs/2026-09-17-655-index-readings.py index 47d4c58ca..93ade0580 100644 --- a/.agents/docs/2026-09-17-655-index-readings.py +++ b/.agents/docs/2026-09-17-655-index-readings.py @@ -4,6 +4,9 @@ # applied. Run from the index checkout; prints the literals whose readings differ. import re, shlex, subprocess, sys def flag_words(s): + # SPEC-004 section 8 rule 8: a -D or /D element with a space is one word. + if (s.startswith('-D') or s.startswith('/D')) and ' ' in s: + return [s] out=[];w=[];st=False;i=0 while i` on both C and C++ flag channels. -// -// ORDER (both halves are load-bearing): this must run AFTER -// merge_conditional_config — `defines` is a BuildInputs member, so a -// matching `[target.'cfg(...)'.build] defines` has been appended by then and -// folds in the same pass, landing after the unconditional entries so GNU -// last-wins gives the conditional rule precedence — and BEFORE the manifest is -// snapshotted into packages[] / fingerprinted, because that snapshot (not the -// manifest) is what the P1689 scan, the compile edges and compute_fingerprint -// actually read. -// -// Idempotent: clearing the vector after folding makes repeated calls harmless. // ── An element whose words changed in 2026.9.17.1 (#655) ───────────────────── // // A compile-flag element used to reach the compiler as its host's command-line @@ -647,6 +635,18 @@ void report_flag_words_changes(const mcpp::manifest::Manifest& m) { } } +// Desugar `[build].defines` into `-D` on both C and C++ flag channels. +// +// ORDER (both halves are load-bearing): this must run AFTER +// merge_conditional_config — `defines` is a BuildInputs member, so a +// matching `[target.'cfg(...)'.build] defines` has been appended by then and +// folds in the same pass, landing after the unconditional entries so GNU +// last-wins gives the conditional rule precedence — and BEFORE the manifest is +// snapshotted into packages[] / fingerprinted, because that snapshot (not the +// manifest) is what the P1689 scan, the compile edges and compute_fingerprint +// actually read. +// +// Idempotent: clearing the vector after folding makes repeated calls harmless. // Both `cflags` and `cxxflags` get the macro; assembly units pick it up for // free via the -D/-U/-I subset the ninja backend filters out of packageCflags. // A define is a value, so it enters the flag list as one word