diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 20dbcf3..6baa4bf 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -5,6 +5,28 @@ were written by hand instead: [docs/BAZEL-RULES.md](docs/BAZEL-RULES.md). ## CMake Experiments +### Ladybird (cmake + vcpkg + cargo -> Bazel) + +- Source: https://github.com/LadybirdBrowser/ladybird +- Generated BUILD files; the workspace overlay is in + [`examples/ladybird/`](examples/ladybird/README.md), the 34 findings in + [`docs/CASE-ladybird-migration.md`](docs/CASE-ladybird-migration.md) +- `git clone && bazel build //:ladybird` builds the browser: **Bazel owns the + whole dependency closure**, with no CMake build and no network in any action — + the 77 vcpkg ports (fetched from a captured asset pin) and the 10 Rust crates + + `flapc` (154 crates.io archives resolved from `Cargo.lock`) are Bazel targets +- All 51 code generators run under Bazel with output byte-identical to CMake's + (1,408/1,408 files); the 6 binaries render `--headless=text` and + `--headless=layout-tree` byte-identically to the CMake reference +- C++23, and the largest subject here: 34 libraries, ~3.7k TUs, LibWeb alone + ~1,961 compile inputs — which is what surfaced the extractor's depSet OOM +- Generated custom Bazel rules, all because the recipe was worth keeping and the + ecosystem ruleset would have replaced it: `vcpkg_tree`/`vcpkg_lib` (vcpkg as an + ordinary action under `x-block-origin`, not `rules_foreign_cc`), and + `rust_sysroot`/`cargo_crate`/`cargo_lib`/`cargo_binary` (offline cargo, not + `rules_rust`/`crate_universe`) +- Qt via `kklochkov/rules_qt` — the one place a ruleset *was* adopted + ### BoringSSL (cmake <-> Bazel) - Source: https://github.com/google/boringssl diff --git a/README.md b/README.md index e0d156d..e6b0dbe 100644 --- a/README.md +++ b/README.md @@ -318,12 +318,18 @@ scripts/ triage.py groups diff.json into a systematic-cause worklist serialize.py model ↔ JSON (the contract between stages) tests/ + run_all.py discovers and runs every test_*.py; fails on a silent file + test_run_all.py the runner's own guards, triggered rather than asserted test_engine.py diff/canonicalize/roles/config/TU-set behavior test_extractors.py full pipeline on synthetic File-API + aquery fixtures test_maven.py Maven frontend + Java branch of reconstruct test_extract_npm.py npm frontend (NDJSON → action IR) + role classification test_triage.py triage grouping/histogram/cap behavior test_configure.py configure_file trace extraction + test_diff_ts.py the standalone TS source→emit diff + test_emit_cargo.py the Ladybird Rust ring emitter (crates/index/ring/binaries) + test_emit_vcpkg.py the Ladybird vcpkg pin emitter (versions db → http_file) + test_vcpkg_plumbing.py what the generated vcpkg BUILD/bzl files must not say docs/ DESIGN-action-based-ir.md the action-based IR reframe (grounded in CMake/Bazel/Maven) proposal-doc.md the broader Build IR spec (frontends → IR → backends) @@ -390,14 +396,34 @@ Run as a skill, Claude drives the generation and triage/fix loop automatically. ## Tests ```bash -python3 tests/test_engine.py -python3 tests/test_extractors.py -python3 tests/test_maven.py -python3 tests/test_extract_npm.py -python3 tests/test_triage.py -python3 tests/test_configure.py +python3 tests/run_all.py # the whole suite, one exit code (~0.3s) +python3 tests/run_all.py -v cargo # verbose, filtered by file or test name ``` +`run_all.py` **discovers** `tests/test_*.py` rather than reading a list, and it +fails the run if a test *file* contributed nothing — no tests defined, or a module +that would not import. Both of those are the reason it exists: this README used to +list six commands for ten files, and three of those files had no +`if __name__ == "__main__"` block at all, so running them imported the module, +defined 46 test functions, called none of them, and exited 0. There is no pytest +here, so nothing else called them either. + +That is the same bug as `glob(..., allow_empty = True)` over a directory that is +not there (case study finding 35): **a check that cannot fail is indistinguishable +from one that is not needed.** A per-file runner is what was missing, but a +per-file runner is also exactly what nobody notices the absence of — so the fix is +one entry point that knows how many files there are and how many tests each +contributed. Its three guards are themselves tested, by triggering them +(`test_run_all.py`). + +Run one file or one test with a substring filter (`run_all.py test_triage`, +`run_all.py -v cargo`). The per-file `if __name__` runners are gone: in one file +that block had ended up **mid-file**, so four tests appended after it were defined, +never called, and the file still reported `6/6 passed`. A test's position in the +file should not decide whether it runs. As the suite grows the next step is a `py_test` per file under +`bazel test //...`, so the caching and parallelism come for free and a commit gate +runs it without anyone remembering to — this repo has no `MODULE.bazel` yet. + The extractor tests run against fixtures under `tests/` that mirror the documented File API, aquery, Maven argfile, and npm-NDJSON schemas. Real projects exercise schema details the fixtures may not (fragment quoting, `external/` repo diff --git a/SKILL.md b/SKILL.md index 9e6b7db..b67d07e 100644 --- a/SKILL.md +++ b/SKILL.md @@ -392,11 +392,17 @@ python3 scripts/diff_ts.py model.npm.json model.bazel.json # standalone check ## Tests ```bash -python3 tests/test_engine.py && python3 tests/test_extractors.py \ - && python3 tests/test_maven.py && python3 tests/test_extract_npm.py \ - && python3 tests/test_triage.py && python3 tests/test_configure.py +python3 tests/run_all.py # discovers every tests/test_*.py, one exit code ``` +Do not hand-list the files. This block used to name six of the eleven and chain +them with `&&`, so it stopped at the first failure and never reached the rest — +and three of the files it omitted had no `if __name__ == "__main__"` block, so +running them defined 46 tests, called none, and exited 0. `run_all.py` discovers +the files and **fails if any file contributed no tests**, which is the +`allow_empty = False` of test discovery (case study finding 35). Run one file with +a substring filter: `python3 tests/run_all.py test_triage`. + Extractor tests run against fixtures that mirror the documented File API, aquery, Maven argfile, and npm-NDJSON schemas. When a real project surfaces a schema detail the extractors mishandle (fragment quoting, `external/` repo paths diff --git a/docs/BAZEL-RULES.md b/docs/BAZEL-RULES.md index ff613df..89cbc0b 100644 --- a/docs/BAZEL-RULES.md +++ b/docs/BAZEL-RULES.md @@ -59,15 +59,30 @@ Ranges, not recommendations. "Declared" is what a `MODULE.bazel` asked for; | ruleset | declared range | resolved | migrations | how far it was exercised | |---|---|---|---|---| -| `rules_cc` | 0.0.16 – 0.2.22 | 0.2.14 – 0.2.22 | Dolphin; fmt, spdlog, TinyXML2, zlib; BoringSSL, Abseil, RE2 | Dolphin: `//...` green in 3 configurations, 1350 tests pass, action-graph parity 0 errors. Build re-verified green at declared 0.0.16 / 0.1.1 / 0.2.22; `compatibility_level = 1` and both `//cc:defs.bzl` and the per-rule `.bzl` files exist across that whole span | +| `rules_cc` | 0.0.16 – 0.2.22 | 0.2.14 – 0.2.22 | Dolphin; **Ladybird**; fmt, spdlog, TinyXML2, zlib; BoringSSL, Abseil, RE2 | Dolphin: `//...` green in 3 configurations, 1350 tests pass, action-graph parity 0 errors. Build re-verified green at declared 0.0.16 / 0.1.1 / 0.2.22; `compatibility_level = 1` and both `//cc:defs.bzl` and the per-rule `.bzl` files exist across that whole span. Ladybird: 34 libraries + 6 executables, ~3.7k C++23 TUs, renders pages byte-identically to the CMake reference | | `platforms` | 0.0.10 – 1.1.0 | 1.0.0 – 1.1.0 | same | as above; only ever a transitive/constraint dep | +| `rules_shell` | 0.6.1 | 0.6.1 | Ladybird | one point. Only because Bazel 9 removed the native `sh_binary`, which the vcpkg/cargo build wrappers are | +| `rules_qt` (**kklochkov**, not the BCR module) | 2.0.1 | 2.0.1 (`archive_override`) | Ladybird | `qt.local_repo` (qmake-discovered host Qt 6.10.2) + `qt_cc_moc` over 11 `Q_OBJECT` headers + `qt_qrc`/`qt_cc_rcc`; all 11 moc bodies byte-identical to CMake's, GUI runs. See [the section below](#dolphin-qt-handled-by-hand-and-a-ruleset-that-was-missed) | | `rules_jvm_external` | 6.7 | — | Guava (Maven frontend) | one point, coordinate deps only; the Maven frontend is argv-floor, so this is not a parity claim | | `rules_license`, `googletest`, `google_benchmark`, `rules_python` | see note | — | BoringSSL, Abseil, RE2 | **not our choices** — these are what those projects' *own* Bazel builds declare (`rules_license` 1.0.0, `googletest` 1.17.0.bcr.2, `google_benchmark` 1.9.4/1.9.5, `rules_python` 1.7.0). We diffed against them; we did not select them | Bazel itself: **7.5.0** (VSCode, pinned in `.bazelversion`) and **9.2.0** -(Dolphin). No migration here has needed a Bazel-version-specific workaround. +(Dolphin, Ladybird). One Bazel-version-specific consequence, not a workaround: +on 9.2.0 the native `sh_binary` is gone, so Ladybird has to declare +`rules_shell` for wrappers that needed no dep on 7.x. Every "resolved" column above is a fact about 9.2.0, not about the declared file. +Ladybird declares **4** modules and resolves **27** — the same transitive blow-up +as Dolphin (`rules_cc → protobuf → …` brings in `rules_kotlin`, `rules_android`, +`rules_swift`, `rules_jvm_external`), and worth restating because Ladybird's Qt +and Rust rings deliberately avoid rulesets: the transitive cost arrives through +`rules_cc` regardless of how little else you adopt. + +Ladybird has `common --check_direct_dependencies=error` in its `.bazelrc`, per the +rule above — and turning it on immediately failed: it declared `rules_cc 0.2.17` +while 0.2.19 resolved. The declared version was inert, exactly as this section +predicts. Corrected to what resolves, with the build re-verified green after. + ## Written by hand instead of adopting a ruleset The interesting cases. Both are C++/TS migrations where an obvious ecosystem @@ -160,8 +175,31 @@ is in the ruleset that is **not** in the BCR under `rules_qt`. Querying the registry for `rules_qt` returns the prebuilt-single-version one, which would have pinned Qt 6.8.3 and turned the migration into a different build. -Neither has been used here, so this is a pointer for whoever hits Qt next, not a -recommendation. The transferable parts: +**Since written, `kklochkov/rules_qt` 2.0.1 has been used** — Ladybird's Qt UI +runs on it (`qt.local_repo` + `qt_cc_moc`/`qt_qrc`), so the paragraph above is +now a result rather than a prediction, and it held: `qmake -query` found the same +6.10.2 SDK `find_package` does. Two things only the use showed, both in +[CASE-ladybird-migration](CASE-ladybird-migration.md) §Qt (findings 19–21): + +* **The moc rule must own the include paths, and this one does.** moc emits + metatype includes only for types whose definition it has *seen*, so a + hand-rolled genrule needs a `moc_input_headers` filegroup plus matching `-I` + flags kept in sync by hand — and getting it wrong yields *wrong output that + still compiles* (four outputs silently lost an include), caught only by + byte-diffing against CMake. `qt_cc_moc` reads the dirs off the Qt toolchain's + `CcInfo` compilation context and stages the headers itself: byte-identical moc + bodies with **zero** include flags in the BUILD file. This is the concrete form + of "the rule knows what moc needs better than its callers do", and it is the + strongest argument for the ruleset over a genrule. +* **A good ruleset withholds the knob you would have misused.** Splitting CMake's + unity `mocs_compilation.cpp` into per-header TUs exposed a latent Ladybird bug + (an inline `dynamic_cast` on a forward-declared type, compiling only because of + include *order* inside the unity file). moc's `-b` flag would have papered over + it in the build system; `qt_cc_moc` offers no such knob, which forced the + one-line upstreamable header fix instead. Unity builds hide incomplete-type + bugs, so a migration off one should expect to find them. + +The transferable parts of the comparison: * **BCR presence is not fitness, and BCR absence is not nonexistence.** The registry is a distribution channel, not a curated index. Search wider, and @@ -186,6 +224,32 @@ prebuilt artifact you never claimed parity on is a different thing — VSCode's `nodegyp_module` shells out to `node-gyp rebuild` for native `.node` modules, and nothing is lost, because those binaries were never part of the parity claim. +Ladybird walks that line at scale, and how it does so is the reusable part. +Its 77 vcpkg ports and 10 Rust crates are third-party leaves, so running their +native builders (`vcpkg install`, `cargo build`) costs nothing a parity claim +needed. But rather than reach for `rules_foreign_cc`, both are **ordinary Bazel +actions** — `vcpkg_tree` and `cargo_crate` — and the reason is what the wrapper +would have taken away: + +* **Fetching is the part Bazel must own; building is the part it need not.** The + dependency's *identity* (URL + hash) is exactly what a build has to pin, and + what `rules_foreign_cc` leaves to the foreign tool's own downloader. So the + fetch is hoisted out into `http_file`/`http_archive` repos generated from a + captured pin, and the recipe runs offline against them — `vcpkg` under + `x-block-origin`, `cargo` under a network-blocked action with a vendored + registry. The invariant is verifiable and was verified by removal: **zero + network access in any action**. +* **A `repository_rule` was the tempting wrong answer.** `vcpkg_tree` is + deliberately a rule, not a repo rule: repository fetches escape the action + graph, the sandbox and remote execution, so a 45-minute dependency build would + have been invisible to `aquery` and uncacheable in the normal way. As a rule its + inputs and outputs are declared like anything else. + +Which sharpens the section title: the question is not "is this code I am +migrating?" but "**which properties do I need to remain checkable?**" For +Ladybird's deps that is hermeticity and pinning, not compile parity — and +`rules_foreign_cc` happens to give up precisely those. + ## Checklist for the Bazel side of a migration 1. Resolve every version at migration time; do not write one from memory. @@ -197,6 +261,10 @@ nothing is lost, because those binaries were never part of the parity claim. 5. Decide what equivalence you need — byte-identical or content-equivalent — before choosing. Ecosystem rules target the latter. 6. Weigh the transitive graph (`bazel mod graph`), not just the direct dep. -7. Keep `rules_foreign_cc` away from the code under migration. +7. Keep `rules_foreign_cc` away from the code under migration. For third-party + leaves, run the foreign builder as a plain Bazel *action* (not a + `repository_rule`) over separately-pinned fetched inputs, so hermeticity stays + checkable even where compile parity is not claimed. 8. When the migration converges, run `bazel mod graph` and record the **resolved** - versions. That is the only version list worth keeping. + versions, and put `common --check_direct_dependencies=error` in `.bazelrc` so + the two cannot drift apart again. That is the only version list worth keeping. diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index e050520..269e295 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -8,6 +8,21 @@ generated BUILD files — is in [`examples/ladybird/`](../examples/ladybird/), along with an honest inventory of what still stops it from being a clone-and-build. +Bazel now builds the browser with **nothing** read out of CMake's build tree +(`Build/full`) — 0 targets in the closure of all six binaries, down from 741. That +was claimed prematurely and was false for most of this migration; +[finding 35](#finding-35-the-claim-was-false-and-a-glob-is-why-nobody-noticed) is +the autopsy, and it is the finding to read first if you are migrating a project of +your own — the bug was in the *shape of the verification*, not in the Bazel rules. + +**`git clone && bazel build` on a fresh clone still fails, though, and finding 36 +is the second autopsy.** Removing `Build/full` was the goal, so `Build/full` is +what I verified; the clone needs four *other* things nobody had asked for — a +vcpkg checkout (globbed with `allow_empty = True`, the same pattern one tree over), +its `.git`, an unpinned HSTS table, and `pip install ply` inside a vcpkg port that +`x-block-origin` never sees. **The only check that finds all of them is doing the +clone**, which is finding 36's one-line summary and took two goes to learn. + ## Why Ladybird A real, recognizable, from-scratch browser engine — a strong headline for the @@ -47,7 +62,12 @@ AK is the foundation library (38 sources, external deps fmt/simdutf/mimalloc/ cpptrace, two configure-generated headers). Standing it up end-to-end proved the whole loop works on real Ladybird and surfaced the reusable scaffolding: -- **`MODULE.bazel`**: bzlmod, `rules_cc` 0.2.17, `platforms` 1.0.0. +- **`MODULE.bazel`**: bzlmod, `rules_cc` 0.2.19, `platforms` 1.0.0. (Declared + 0.2.17 for most of this migration, which was inert: MVS resolved 0.2.19 via + `bazel_tools` on Bazel 9.2.0. Caught only on adding + `common --check_direct_dependencies=error`, which is now in the `.bazelrc` so + the declared and resolved versions cannot drift apart again — see + [BAZEL-RULES](BAZEL-RULES.md).) - **vcpkg shim**: a `BUILD.bazel` dropped into `Build/full/vcpkg_installed/x64-linux-dynamic/` — one `cc_library(:headers)` over the whole `include/` tree + `cc_import` per prebuilt `.so`. Consumes @@ -1423,10 +1443,1196 @@ Both remaining debts are over-declaration, and they are honest: - Each `vcpkg_lib` still declares the whole vcpkg tree as its header input (finding 33's leftover). Same shape: paths are per-port, the input *set* is not. -`git clone && bazel build //:ladybird` is now true for the browser. `Build/full` -is still needed to *regenerate* the BUILD files and to run the parity harness — -that is a converter-development dependency, not a build dependency, and it is a -different claim from the one this ring closed. +Nothing in the emitted build reads `Build/full` any more. It is still needed to +*regenerate* the BUILD files and to run the parity harness — a converter-development +dependency, not a build dependency, and a different claim from the one this ring +closed. (It is also a different claim from "a fresh clone builds", which is still +false for unrelated reasons — finding 36.) + +**That claim was still false when I wrote it, and finding 35 is the autopsy.** I +had closed the two *binary* dependencies (vcpkg, Rust) and concluded the tree was +free; it was not. Every binary still depended on **741 targets under `Build/full`** +for *generated headers*, and I had not checked, because the thing that would have +told me — a build failure — could not happen: the shims globbed a foreign tree +with `allow_empty = True`. Read finding 35 before believing any "verified by +removal" in this document, including the ones above: removal is only a test of +what you actually remove. + +## Finding 35: the claim was false, and a glob is why nobody noticed + +Ulf asked where things stood on checking out Ladybird and building it with Bazel. +I said it worked — findings 33 and 34 had closed vcpkg and Rust, the two *binary* +dependencies on CMake's tree, and the README said so in bold. Then I checked, +which I should have done before saying it. + +It did not work. Every one of the six binaries depended on **741 targets under +`Build/full`**, CMake's build tree, and the overlay in `examples/ladybird/` +shipped four `BUILD.bazel` files and **zero headers**. A fresh clone got no error +from that — it built for roughly 1,600 actions and then died on `fatal error: +LibXML/Export.h: No such file or directory`, a message that names neither the shim +nor the missing tree. + +**The mechanism is the finding.** The shims were + +```python +cc_library( + name = "generated_lib_headers", + hdrs = glob(["**/*.h"], allow_empty = True), + includes = ["."], +) +``` + +over a directory that only exists after a CMake build. `allow_empty = True` turns +"the tree you depend on is absent" into an empty list and no diagnostic. A glob +over a foreign tree **cannot fail**, so a shim that is broken and a shim that is +unnecessary are indistinguishable — and I had been reading a green build as +evidence for the second. + +That generalizes past this migration, and it is the transferable lesson: **if the +emptiness of a glob means "a thing I depend on is missing", `allow_empty = True` +converts a build error into a mystery.** Either let it fail, or don't pretend the +input is optional. + +### 741 → 31: most of them were not gaps, they were shadows + +Counting the true gaps needed care, and my first count was nonsense — "741 +targets, 76 covered", which is impossible. The bug was **matching headers by +basename**: every `Export.h` matched every other `Export.h`, so coverage looked +enormous. Comparing exact logical paths gave the real picture: + +| | count | what it was | +|---|---|---| +| LibWeb bindings headers | 666 | **Bazel already generates all 692.** The shim was *shadowing* Bazel's own outputs, silently winning or losing on include order | +| `generate_export_header` `Export.h` | 15 | real gap | +| Rust FFI headers (`RustFFI.h`, `CraneliftFFI.h`) | 15 | real gap | +| Qt `moc_predefs.h` | 1 | not a gap at all — Bazel runs moc itself via `rules_qt`; nothing referenced it | +| AK `configure_file` headers | 2 | already closed earlier in the session | + +So 666 of the 741 were pure duplication, which I proved by deleting them: the +build stayed green and recompiled 2,635 actions from Bazel's own headers. The +honest gap was **31 files**, and the reason a 90%-redundant shim survived is the +same `allow_empty` — nothing ever compared the two sets. + +### The 15 `Export.h`: derive the template, don't copy the output + +`Meta/emit_export_headers_bazel.py` emits them, and two decisions in it matter. + +All 15 normalize to **one** template; the per-library tokens are derived the way +`Meta/CMake/targets.cmake` derives them (api = `upper(lib - "Lib") + "_API"`, +prefix = `upper(lib)`, exports = `lib + "_EXPORTS"`), so adding a library needs no +edit. All 17 emitted artifacts (15 + AK's two) are **byte-identical to CMake's**, +checked by `--check Build/full`, and that check earned its keep twice: once on a +`#cmakedefine` regex (the templates put the `#` at column 0 with the indent +*after* it — `# cmakedefine01 FOO`, which two regex attempts got wrong), and +once on a **1-byte** difference a `render()`-level check could never see — a +heredoc adds a trailing newline, so the emitted shell needed one stripped from the +body. Only byte-comparing the *built artifact* catches that. + +`AK/Backtrace.h` is deliberately **not** in the parity check, and that is the +interesting one. It is not a template — CMake writes it from +`find_package(Backtrace)`, i.e. from a *question about the host*. So the genrule +**compiles a probe** rather than baking my machine's answer. Comparing the result +against my own tree would only re-confirm my own tree, which is why it is +excluded: test the variable, not the value. + +`Libraries/LibWeb` needed its own emitter mode, for a reason worth stating because +Bazel is right and I was wrong: include dirs cannot escape a package, so LibWeb's +`Export.h` must be an output *of the LibWeb package*. `includes = ["../.."]` is +rejected ("resolves to the workspace root"). It lands at `genroot/LibWeb/Export.h` +with `includes = ["genroot"]`. + +### Bug 1: eight crates ship a `RustFFI.h`, and LibRegex had the wrong one + +Removing the shim exposed a **real, pre-existing correctness bug**. Eight of the +ten Rust crates emit a header named literally `RustFFI.h`, and four TUs include it +with no directory (`#include `). CMake is unambiguous because +`FFI_OUTPUT_DIR` defaults to the consuming library's *own* binary dir. Bazel puts +every dep's include dirs on one command line, and my rules published every +crate's unprefixed `ffi/` dir to every consumer — so **LibRegex was compiling +against LibUnicode's header**, and the only thing hiding it was a leftover +`-IBuild/full/Libraries/LibRegex` that shadowed both. Delete the tree and it +becomes `'RustRegexFlags' has not been declared`. + +The fix needed **two** steps, and the first one alone looked sufficient — which is +the part I would otherwise have shipped: + +1. **Publish the unprefixed dir on its own target** (`cargo_bare_include`), not on + `cargo_lib`. Necessary because `CcInfo`'s `system_includes` propagate + transitively: folded into `cargo_lib`, LibGfx inherited LibRegex's *and* + LibTextCodec's bare dirs and failed with `'FFI' does not name a type`. + (`cc_common.create_compilation_context` has no settable "local includes"; + I tried.) +2. **Depend on it through `implementation_deps`.** Step 1 stops the dir leaking out + of `cargo_lib`; it does *not* stop it leaking out of **LibTextCodec**. Include + dirs propagate along the C++ dep graph too, so `LibGfx → LibTextCodec` handed + LibGfx someone else's `RustFFI.h` anyway, and `YUVData.cpp` failed identically. + +`implementation_deps` is Bazel's name for exactly the scope CMake's +`target_include_directories(... PRIVATE)` has — which is *why* a bare include is +unambiguous in CMake, and what had to be reproduced rather than approximated. The +three crates that need it are **derived by scanning the source** for a +directory-less include, not hardcoded, and the build emitter now **imports** that +derived set instead of keeping its own copy beside it (it had one; a hand-kept copy +of a derived set is finding 23 in miniature). + +### Bug 2: an entire Rust target was missing, behind two host escapes + +`//:WebContent` and four others then failed with one error: +`CraneliftBridge.cpp:13:10: fatal error: CraneliftFFI.h: No such file or directory`. + +Cranelift — Ladybird's AOT WebAssembly compiler, `ENABLE_CRANELIFT_JIT=ON` — was +absent from the Bazel graph **entirely**. `Libraries/LibWasm/CMakeLists.txt` was +not in the emitter's `CMAKELISTS`, and it declares its crate with +`build_rust_binary()`, which the parser did not handle at all. Two host escapes had +been covering for it: + +- the FFI header sat in `Build/full/Libraries/LibWasm`, which a global `-I` reached; +- the compiler binary was named by **an absolute path to my machine**, baked into + `-DWASM_CRANELIFT_COMPILER_PATH="/home/ubuntu/ladybird-work/Build/full/bin/cranelift-compiler"`. + +The second one alone would have broken every other checkout on earth, and it was +in a checked-in generated file. Nothing was going to tell me: it is a *define*, so +it compiles fine and the lookup only fails at run time, on a code path that needs a +WebAssembly page big enough to trigger AOT compilation. + +A `build_rust_binary` crate is a genuinely different shape, not a variant spelling, +and the rules now say so: there is **no archive to link** (`cargo rustc --bin` is +the whole build, and what C++ consumes is the *executable*, spawned at run time), +but it **still emits a cbindgen header** its caller includes. So `cargo_binary` +declares FFI headers like `cargo_crate` does, `cargo_lib` yields a headers-only +`CcInfo` when the archive is absent, and the crate is consumed through the same two +labels a staticlib crate is. + +For the binary itself, the fix is *not* to point the define at `bazel-bin` — that +is the same escape with a nicer prefix. Ladybird already resolves the compiler +through a chain (`resolve_cranelift_compiler_path`: `$LADYBIRD_CRANELIFT_COMPILER` +→ compile-time path → **sibling-of-self**), and Bazel puts every root-package +output in one bin directory, so link 3 finds it with no path baked in at all. The +define becomes the bare filename and the binary is attached as `data`, so it is +genuinely *there* in the runfiles of everything that links LibWasm. **The +dependency is declared; the path is not asserted.** + +### What actually verified it + +Not "the build is green" — that was the state I started from. The header is +byte-identical to CMake's; then `Build/full/{Libraries,Services,UI,bin,cargo}` was +moved off the machine and the three shim packages **deleted**, leaving zero +Ladybird-generated headers under `Build/full`; all six binaries rebuilt from +scratch (2,704 actions); and both `--headless=text` and `--headless=layout-tree` +came out byte-identical to the CMake reference on three pages, two of which execute +WebAssembly. `cquery` over the closure of all six binaries now returns **0** targets +under `Build/full`. + +One control worth recording, because it is the kind of check that keeps an +enthusiastic conclusion honest: I instrumented `cranelift-compiler` with a wrapper +to prove the browser was invoking it, and **it never was** — even on a +300-function module. Before concluding the wiring was broken I ran the same probe +against the *CMake reference build*, which also never invoked it. Compilation is +submitted to a thread pool and the page finishes first; the Bazel build matches the +reference exactly. Without the reference-side control I would have "fixed" a +non-bug. + +Two smaller repairs fell out of the same pass, both drift the emitter rule exists +to prevent: `emit_libweb_bazel.py` did not emit LibWeb's export-header block, so +regenerating would have silently truncated it; and six of the seven per-target +`-IBuild/full` copts existed only because CMake spells a target's own binary dir +relative to itself (`Build/full/Services/WebContent/../..`) and the emitter +compared paths as **strings** against its global-roots set. `os.path.normpath` +before the comparison removed all six. + +### The same bug, a third time, in this repo's own tests + +Having learned that a check which cannot fail is indistinguishable from one that +is not needed, I applied it to the test suite itself — by counting, not by reading +the exit code. `for f in tests/test_*.py; do python3 "$f"; done` exited 0 for all +ten files, but three of them (`test_emit_cargo.py`, `test_emit_vcpkg.py`, +`test_vcpkg_plumbing.py`) had **no `if __name__ == "__main__"` runner at all**: +Python imported the module, defined 46 `def test_` functions, called none of them, +and exited 0. There is no pytest in this sandbox, so nothing else was calling them +either. Exactly the shape of `allow_empty = True` — a green result carrying no +information. + +With runners added, 6 of those 46 failed, and every one was a *true* report about +the finding-35 changes rather than a stale assertion: the ring emitter had grown a +third parameter (the binary crates), `cargo_lib` had gained an +`archive == None` branch for a `--bin` crate that has no archive, the shared +FFI-header lookup had moved into `cargo_vendor.sh` so both drivers use one copy, +and one test read `Build/full/Libraries/BUILD.bazel` — a file whose *deletion* was +the fix. That last one is the pleasing part: the test failed because the thing it +asserted about no longer exists, so it became the stronger assertion that the path +**must not** exist. + +The repair is not "add a runner to each file", though that is what was missing: a +per-file runner is precisely the thing nobody notices the absence of, so doing only +that leaves the next silent file undetected. What was missing was **one entry point +that knows how many test files exist and how many tests each contributed**, so a +file going quiet is a failure and not just a smaller number nobody was counting. +`tests/run_all.py` discovers `tests/test_*.py`, and fails the run if any file +defines no tests or will not import — the `allow_empty = False` of test discovery. +Its guards are tested by triggering them, because a guard never seen to fire is +back where this started. The suite is 135 tests over 11 files, 0.3s, one exit code. + +The general lesson, which outlives this repo's test layout: **the unit that has to +be accounted for is the CONTAINER, not the item.** Counting passing tests cannot +detect a missing file; counting matched files cannot detect an empty glob; +counting green actions cannot detect a header supplied by a shim. Each time, the +fix was to make the enclosing thing declare how many children it should have. + +## Finding 36: I removed the dependency I was looking for, so that is the one I found + +Ulf asked, again: *can I clone and build with Bazel?* Finding 35 had just closed the +741 `Build/full` header dependencies, `cquery` returned 0, and the README said yes. +This time I did not answer from the README. I ran `git clone` into an empty +directory, dropped the overlay in, and typed the command. + +**It failed. Six times, for six different reasons, and five of them have nothing to +do with `Build/full`.** + +| What was missing | The error a cloner gets | Why my machine hid it | +|---|---|---| +| `Build/vcpkg` — a microsoft/vcpkg checkout at `vcpkg.json`'s `builtin-baseline` | `/tmp/.../root/vcpkg: No such file or directory` | `Meta/ladybird.py vcpkg` had been run months earlier | +| its `.git` (120 MB), which vcpkg needs for `git read-tree` to resolve versioned ports — and which the filegroup **excludes** | `fatal: not a git repository: '.git'` … `while checking out port sqlite3` | the action is `no-sandbox`, so it read the real checkout regardless of what was declared | +| `Build/caches/HSTSPreload/transport_security_state_static.json`, an **unversioned** download CMake does at configure time from Chromium's `main` | `missing input file '//:Build/caches/...'` | CMake's configure had already fetched it. **Now fixed**: pinned downstream to a commit + sha256 (`hsts_preload.bzl`) and fetched by Bazel | +| two path bugs in `Meta/vcpkg_build.sh`: the distfile index *and its entries* were execroot-relative, and vcpkg runs the asset script from its own cwd | `awk: cannot open ...`, then `cp: cannot stat ...`, surfacing as `no asset cache hits` and `x-block-origin blocks trying the authoritative source` | the checkout already had `downloads/tools/cmake-4.4.0-linux`, so vcpkg never *asked* the script for a tool | +| `ply`, which the `angle` port installs with **`pip install ply`** | `No matching distribution found for ply` | this sandbox exports `HTTP_PROXY`, and the action inherits it | +| the four `vcpkg_from_git` archives, staged from a directory I had made **by hand** — `vcpkg_git_archives.bzl` is generated, committed, documented, and **loaded by nothing** | 20 minutes in: `git fetch https://android.googlesource.com/.../piex.git … Error code: 128` | the directory existed on my disk from when I built the pin | + +### Three of them are the same bug as finding 35, in three different syntaxes + +`//Build/vcpkg:tree` is `glob(["**"], allow_empty = True)` over a directory a fresh +clone does not have. It matches exactly one file — *its own `BUILD.bazel`* — reports +nothing, and the build dies later somewhere else. That is the finding-35 pattern +verbatim; I had deleted the three `Build/full` shims and left a fourth shim, over a +different tree, in place. **I had been looking for `Build/full`, so `Build/full` is +what I removed.** + +The git-archive staging is the same idea written in bash, and it is worth quoting +because it packs three independent ways to succeed while doing nothing into four +lines: + +```bash +if [ -d "$SRC/Meta/CMake/vcpkg/git-archives" ]; then # skips + cp "$SRC/Meta/CMake/vcpkg/git-archives/"*.tar.gz "$ROOT/downloads/" \ + 2>/dev/null || true # hides, forgives +fi +``` + +A directory test that skips silently, a redirect that swallows the error, and a +`|| true` that forgives the exit code. The four tarballs it was supposed to place +only ever existed because *I* had made that directory while capturing the pin. +`vcpkg_git_archives.bzl` — generated, checked in, listed in the README's file table +— is loaded by **nothing**; it is documentation wearing a `.bzl` extension. Without +those files skia gets 20 minutes in and dies fetching `piex.git` from +android.googlesource.com, naming neither the directory nor the tarball nor the port +that pinned it. It now fails in four seconds saying exactly what is missing and +where it comes from. + +The `.git` case is worse than a missing input, because it is a *lie in the +declaration*: the filegroup explicitly excludes `.git/**`, the port resolution +genuinely requires it, and the build works anyway because `no-sandbox: "1"` lets the +action read the real path instead of the declared inputs. An excluded input that the +action reads is strictly worse than an undeclared one — the exclusion looks like a +decision. + +### The fifth is a hole in a claim I had "verified" + +"The 77 vcpkg ports are built with **zero network access**" was measured with +`x-block-origin`, and that measurement is sound as far as it goes: remove a distfile +from the index and the build hard-fails instead of fetching. But `x-block-origin` +governs **vcpkg's own downloader** and nothing else. The `angle` overlay-port calls +`x_vcpkg_get_python_packages`, which runs `pip install ply` — not a distfile, not an +asset, never seen by the pin. + +And nothing stopped it, because I had written the enforcement as a *label*: + +```python +execution_requirements = {"local": "1", "no-sandbox": "1", "requires-network": "0"} +``` + +`requires-network: "0"` is a scheduling hint. It does not build a network namespace, +and `no-sandbox: "1"` guarantees there is none to build. With +`use_default_shell_env = True` the action inherits this sandbox's `HTTP_PROXY`, so +pip quietly succeeded for months. **A control that is not enforced is +indistinguishable from a control that is not there** — the same sentence as finding +35's glob, applied to an `execution_requirements` key instead of a `glob()` +argument. + +The fix needs no patch to the portfile, because pip has supported offline switches: +the wheel is pinned by URL and sha256 (`vcpkg_python_packages.bzl` → `http_file`), +declared as an input of the vcpkg action, staged into a find-links directory, and pip +runs with `PIP_NO_INDEX=1` and the proxy variables unset. An unpinned package is then +`No matching distribution found` — an error, not a download, which is precisely the +property `x-block-origin` gives the other 76. Two details are worth keeping: the URL +is `files.pythonhosted.org`'s content-addressed path (immutable for a version, unlike +`pip install ply`, which resolves against whatever PyPI serves today), and the wheel +had to be added to `use_repo` *and* to the emitter's `--use-repo` output — it is the +one vcpkg input no instrument can capture, so it is also the one a regeneration can +silently drop. + +The other half of the fix is not code: **verify in an environment with no route to +the network.** A flag asserting there is no network is exactly the kind of evidence +this migration keeps getting wrong. + +### What generalizes + +Three times now the same shape: a `glob` that cannot fail, test files that ran no +tests, and an `execution_requirements` key that enforces nothing. Each one was +*green*, and green was the problem. What is worth taking from finding 36 specifically +is narrower and more uncomfortable: + +**Verification finds what it is aimed at.** "Verified by removal" is the strongest +check in this document, and it is still only a test of *what you remove*. I removed +`Build/full` because `Build/full` was the thing I had been arguing about; a clone +needs `Build/vcpkg`, a `.git`, an HSTS table and a Python package, and no amount of +rigor about `Build/full` was ever going to mention them. The check that finds all +five is not a better `cquery` or a wider removal — it is **the actual user's first +command, run the way the user runs it, in a directory that has never seen this +project.** That is one line of shell, it costs ten minutes, and I should have run it +before the first time I said yes. + +### Then the seventh: the recipe for running it pointed at CMake's build tree + +With all six fixed, the clone builds — `//:vcpkg_installed` offline (76 ports, +`pip is offline; 1 pinned wheel(s)`), then all six binaries, 2,842 actions, RC=0 — +and then the *documented run command* fails with `Runtime error: mkdir: Permission +denied (errno=13)`, which is what Ladybird says when it cannot find its resource +root. The README's staging block ended with: + +```sh +ln -sfn "$PWD/Build/full/share/Lagom" "$ER/bazel-out/k8-fastbuild/share" +``` + +**`Build/full`.** Six findings about a fresh clone not having CMake's build tree, +and the last line of the recipe symlinks CMake's build tree. It had never been read +as an instruction, only as something that already worked on my machine — the same +mechanism as all six, one layer further out, in prose rather than in a `glob`. The +resource tree needs no CMake at all: it is `Base/res` (in the clone) plus pdf.js from +`//:vcpkg_installed` (`share/pdfjs/{build,web}`, with +`pdfjs-ladybird-transport.mjs` moved into `web/`, which is where +`UI/cmake/ResourceFiles.cmake` puts it). Assembled that way it is `diff -rq`-identical +to `Build/full/share/Lagom`, and the fresh clone's binaries then render +`--headless=text` and `--headless=layout-tree` **byte-identically to the CMake +reference on all three test pages**. Two smaller notes worth keeping: Bazel's outputs +are read-only, so `cp -r` propagates that and the second staging run fails with +`Permission denied` (`cp --no-preserve=mode`); and the six render RCs were 1 for a +*single* reason — a missing resource root — which is a reminder that a nonzero exit +from a browser is one bit and says nothing about which of six pages failed. + +So the honest state is: **a fresh clone builds and renders, with three inputs staged +by hand** (`Build/vcpkg` + its `.git`, the unpinned HSTS table, the four +`git archive` tarballs). That is a different sentence from "clone and build", and the +difference is exactly what rows 1–3 of the README table say. + +### What closing the last three actually takes — and why my first answer was wrong + +Ulf's follow-up was the right one: *so what's needed to make it work?* My first +answer was "a custom `repository_rule` that clones vcpkg at the baseline — it closes +two of the three blockers." I had done real work to support it: I confirmed +`git_repository`/`new_git_repository` **strip `.git`** (so the built-in rule cannot +deliver the one property this dependency needs), that a custom rule shelling out to +`git clone` keeps it, and that `glob(["**"])` then carries the `.git` files as +declared inputs. All true, and all beside the point. + +**Ladybird already ships the thing I was proposing to write.** `Meta/ladybird.py +vcpkg` — 45 lines in `Meta/Utils/build_vcpkg.py` — clones microsoft/vcpkg, checks out +`vcpkg.json`'s `builtin-baseline`, and bootstraps the tool at a tag+SHA512 the +checkout itself pins. I ran it from an empty directory: **75 seconds, RC=0, `.git` +present at the right commit, `vcpkg` binary built.** The README's step 1a has been +telling cloners to run it the whole time. So blockers 1 and 2 are not missing +machinery; they are a **prefetch step that the recipe must run in the right order**, +and the honest fix is a `MODULE.bazel`-adjacent note plus a check that fails clearly +when it has not been run — not a repo rule reimplementing a script the project +maintains. + +Two lessons, and the second is the uncomfortable one. First: **a repo rule that +re-implements the foreign project's own bootstrap script is a fork of it.** It would +drift the moment Ladybird bumps its baseline or its tool metadata, and the drift +would look like a Bazel bug. Second: **I answered a "what's needed" question by +designing, not by reading the project.** Nothing in the environment stopped me from +opening `Meta/Utils/build_vcpkg.py` before proposing to duplicate it — the same +failure mode as answering from the README instead of doing the clone, one level up: +the fix I imagine is more available to me than the fix that exists. + +**The four `git archive` tarballs (row 6) collapse the same way.** No repo rule +needed: clone the pinned URL, `git archive `. I reproduced `libyuv`'s tarball +that way and its SHA512 matched `vcpkg_git_archives.bzl`'s committed value **exactly** +— which is the useful part, because it means the committed hashes are *checkable* and +the reproduction is verifiable in one line rather than trusted. Eight lines of shell, +against a `repository_rule` I would have had to design, test and maintain. + +**Then Ulf asked the question that dissolved even the eight lines: "how does a +Ladybird developer get the correct stuff on disk?"** The answer is that they do +nothing, because **one ordinary `./Meta/ladybird.py build` produces all three +inputs.** `build` calls `build_vcpkg()` itself (not just the `vcpkg` subcommand), so +the checkout and its `.git` appear; the CMake *configure* downloads the HSTS table via +`hsts_preload.cmake`, gated on `ENABLE_NETWORK_DOWNLOADS`, **default ON**; and vcpkg +writes the four `git archive` tarballs into `Build/vcpkg/downloads/` while building +skia and angle. I checked that last one against my own tree: the SHA512s of +`Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz` **equal the committed values** in +`vcpkg_git_archives.bzl`. So "the directory I made by hand months ago" was a copy of +vcpkg's own download cache — I had not built a pin, I had copied one, and then +forgotten which. + +That reframes the finding-36 table — but it also let me answer the wrong question. +"Run the normal build first" is fine for a Ladybird developer and **useless for the +case the whole exercise is about**: Bazel without CMake. Ulf had to ask a third time +before I built it. + +**Without CMake, two of the three are now closed, and the third is a one-line +upstream fix.** + +`Meta/ladybird.py vcpkg` is a standalone subcommand — no configure, no CMake — so the +checkout and its `.git` cost ~70 s. The four `vcpkg_from_git` tarballs are what +needed building, and the shape of the solution is the interesting part, because I got +it wrong twice on the way: + +1. **A static parse of the portfiles is unsound, and wrong in both directions at + once.** My first script scanned skia's and angle's portfiles for + `declare_external_from_git` / `checkout_in_path` and produced **8** archives for + skia where 4 are real, while **missing libyuv entirely**. Both errors have one + cause: `declare_external_from_git` only *declares*, and + `get_externals(${required_externals})` picks from that under feature and platform + `if()`s — the set is decided by CMake evaluation, not by the text — while libyuv's + archive comes from the libyuv *port* calling `vcpkg_from_git` directly, which a + scan of skia+angle cannot see. This is finding 30's lesson recurring: **portfiles + are programs, so do not re-derive what they compute.** +2. **So take the list from the pin and use vcpkg as the instrument for regenerating + it.** `vcpkg install --only-downloads` runs the portfiles' fetch phase and stops: + ~6 minutes, no compilation, no CMake, and vcpkg_from_git produces its tarballs at + the refs the real resolution picks. Same tactic as the 76-distfile asset capture + — instrument the foreign build system rather than predicting it. +3. **`Meta/fetch_vcpkg_git_archives.py` then reproduces each pinned tarball with + `git clone` + `git -c core.autocrlf=false archive `** (byte-for-byte what + `vcpkg_from_git.cmake` runs internally) **and verifies it against the committed + SHA512.** Result: **4/4 reproduced from scratch, byte-identical to the pin.** The + hashes came from vcpkg; git reproducing them is the proof the two agree, so the + pin is checked rather than trusted. Static resolution survives only where it is + sound: mapping an already-known archive *name* to a clone URL. + +One asymmetry is recorded rather than smoothed over: `--only-downloads` yields **3 of +the 4**, because angle's zlib is fetched from angle's *build* phase via +`checkout_in_path`, not its fetch phase. The script says so; "--only-downloads gets +them all" would have been the comfortable, false version. + +That left **exactly one** genuine hermeticity defect: the HSTS table, fetched from +Chromium's unversioned `main`. It is now closed too — pinned *downstream*, to the +commit `main` is serving rather than to a release tag, which is the part I got wrong +first; see below. A useful detail found while checking how invasive the upstream fix +would be, and which also makes the downstream pin shareable: CMake's `download_file` +is a no-op when the file already exists (verified with `ENABLE_NETWORK_DOWNLOADS=OFF`), +so the Bazel-fetched pinned file, copied into `Build/caches/HSTSPreload/` before +configuring, is consumed by CMake unchanged. + +My four successive answers to "what is needed" were: a `repository_rule`, a prefetch +script, a `cp`, and finally a script plus a capture instrument. The middle two were +smaller because I kept reading further into what the project already does — but the +last one is *bigger* than the `cp`, and that is the actual lesson. **"It falls out of +the existing build" was a true sentence that dissolved the question instead of +answering it.** For the audience that has CMake it is the right answer; for the +audience this migration exists to serve it is a non-answer, and I gave it because it +let me stop working. + +**The HSTS table (row 3) is the one whose unpinned fetch is not ours to fix — which +turns out not to mean we cannot pin it.** `Meta/CMake/hsts_preload.cmake` fetches +`raw.githubusercontent.com/chromium/chromium/**main**/net/http/transport_security_state_static.json` +— an unversioned ref, at CMake *configure* time. My first answer was "pin it +upstream, until then stage it", and it was wrong in the way that matters: **it made +someone else's repo a prerequisite for our hermeticity.** A converter usually cannot +change the project it converts, so an answer that requires an upstream patch is an +answer that never ships. + +The correction, and it is a general shape worth stating: **a converter cannot pin an +input on the foreign build system's behalf, but it can pin it for itself — provided +it pins the revision the foreign system is currently *serving*, and proves that with +a byte comparison rather than a hash it invented.** What made me think otherwise was +picking the wrong revision. I tested a Chromium *release tag* (`139.0.7258.5`): 18.7 +MB against `main`'s 10.5 MB, **168,593** generated entries against **94,626** — so +pinning *that* really would have traded a hermeticity gap for a parity gap. But the +commit `main` pointed at when this machine configured serves bytes identical to what +CMake downloaded (`cmp`, 10,521,748 bytes), and pinning **that** costs no parity at +all. A tag is a pin to a *different table*; a commit is a pin to *this* one. The +distinction is the whole finding, and I had generalized "pinning breaks parity" from a +single badly chosen pin. + +So the overlay now pins downstream: `hsts_preload.bzl` (an `http_file` at that commit ++ sha256, consumed by `gen_HSTSPreloadData` as `@hsts_preload_json//file`) and +`Meta/pin_hsts_preload.py`, which re-pins by *measuring* — it downloads the file and +writes the hash it computed, and `--expect-same-as` refuses to write a pin whose bytes +differ from the file the other build system already has. Verified on the fresh clone +with the CMake-downloaded file **deleted** — so the pin is the only possible source: +`HSTSPreloadData.h`/`.cpp` byte-identical to the CMake reference, and `//:LibHTTP` +compiles and links them (RC=0). The residual cost is stated rather than hidden: CMake still +tracks `main`, so a configure newer than the pin disagrees with Bazel — one pinned +input against one unpinned one, a dated disagreement with a sha to look at, instead of +two unpinned fetches that happened to agree. The upstream one-liner is filed as a bug, +not depended upon. + +### Can we just `http_file` the HSTS table? Measured, all four combinations + +Asked directly, so I ran it rather than reasoned about it. `http_file` has two knobs +that matter — pinned ref or `main`, `sha256` or none — and Bazel behaves differently +in all four: + +| URL ref | `sha256` | what Bazel does | what you get | +|---|---|---|---| +| `main` | none | **fetches, builds, and prints** `DEBUG: … a canonical reproducible form can be obtained by modifying arguments integrity = "sha256-ObT9…"` | works; unpinned | +| `main` | given | fails the moment upstream moves: `Checksum was 5d5df26… but wanted 000…` | a build that breaks on Chromium's commit rate | +| pinned tag | given | fetches; 18.7 MB | hermetic, **and not what CMake built** | +| any | none + plain `http://` | refuses outright: `No URLs left after removing plain http URLs due to missing checksum` | — | + +So the answer to "can we?" is **yes, mechanically** — row 1 builds today. Two measured +facts decide whether we should. + +**First: unpinned means Bazel caches whatever it saw first, forever.** With a local +HTTP server as the origin I fetched `VERSION-ONE`, changed the file upstream to +`VERSION-TWO`, rebuilt: `-> VERSION-ONE`, in 0.3 s, no refetch, no warning. Same with +a `file://` origin. That is the right behaviour for a *pinned* input and the worst +possible behaviour for an unpinned one: **two developers who first built on different +days build different browsers and neither can tell.** The output is a +94,000-entry `constexpr Array` of domains that get forced to HTTPS — a silent +difference in security behaviour, not in a log line. + +**Second, and this is the number that settles it:** the table moved *while I was +working on this*. The file this machine's CMake configure fetched from `main` and the +one `http_file` fetched from `main` today differ by one entry: + +``` +< static constexpr Array s_hsts_preload_entries { { +> static constexpr Array s_hsts_preload_entries { { +- HSTSPreloadEntry { "service.gov.scot"sv, true }, +``` + +One domain left Chromium's preload list, so `HSTSPreloadData.cpp` is 53 bytes shorter, +and the byte-parity claim this whole document rests on would have failed for a reason +that has nothing to do with the migration. And the *pinned* tag is not a way out +either: `139.0.7258.5` yields **168,593** entries against today's **94,626** — the +list was pruned hard in between, so pinning unilaterally on the Bazel side doesn't +drift, it just diverges by 74,000 entries. + +So row 1 is out (an unpinned `http_file` is the only input in the overlay whose +staleness would be invisible), row 3's *tag* is out (it diverges by 74,000 entries), +and what is left is row 3 with a **commit**: `3d75766` — the newest commit touching +the path, i.e. what `main` serves — whose bytes are identical to CMake's download. +That is the pin that shipped. The rule I was reaching for and got backwards on the +first pass: **pin what the other build system is serving today, and prove it with +`cmp`; do not pin what looks canonical.** A release tag looks like the responsible +choice and is the one that breaks parity. + +Two shortcuts I tested and rejected, both of which look like simplifications and one +of which I would have shipped: + +- **`--depth 1` on the vcpkg clone.** 8.7 MB instead of 121 MB, and `read-tree` even + succeeds for some ports — then resolution fails on ffmpeg and harfbuzz with + `failed to unpack tree object` and vcpkg's own advice, `Try again with a full vcpkg + clone`. The pinned versions' port trees live in **history**, not at the baseline + commit; that is what a version database is. +- **Dropping `builtin-baseline`** so `.git` is not needed at all. vcpkg then resolves + against the checked-out `ports/` and needs no `read-tree` — it "works". It also + **silently moves 10 dependencies**: ffmpeg 7.1.1#5 → 8.1.2#3, harfbuzz 10.2.0 → + 14.2.1#2, mimalloc 2.2.7 → 3.4.3, plus zlib, freetype, dbus, fontconfig, libedit, + libwebp and cpptrace. A "fix" for a hermeticity blocker that changes ten dependency + versions is the same class of error as everything else in this document, dressed as + simplification. + +The positive control for all of it: a **fresh** full clone at the baseline, driven by +the same manifest, resolves all **78 ports to exactly the versions this dev checkout +resolves** (`diff`, 0 differences). The checkout carries no local state beyond the +ref — which is precisely why a prefetch step is sufficient and a rule is not needed. + +### Why not a git submodule? + +The obvious question, since a submodule is git's own answer to "vendor another repo at +a pinned commit" and it would give the cloner `Build/vcpkg` with a working `.git` from +`git clone --recurse-submodules`. It **does** work mechanically — I checked, because +the `.git` here is load-bearing and a submodule's `.git` is not a directory but a +*gitfile* (`gitdir: ../../.git/modules/Build/vcpkg`). vcpkg's +`git --git-dir .git read-tree ` follows that indirection fine: `READ-TREE OK`. +So "submodules break vcpkg" is not the reason. + +The reason is that **a submodule pins the wrong thing.** A submodule pins one commit +and gives you its *checkout*; vcpkg's manifest pins a baseline commit and then reads +**history behind it**. Ladybird's `vcpkg.json` carries 45 `overrides`, and **14 of +them name a version that is not what `ports/` contains at the baseline**: + +| pinned in `vcpkg.json` | what `ports/` holds at the baseline | +|---|---| +| ffmpeg 7.1.1#5 | 8.1.2#3 | +| harfbuzz 10.2.0 | 14.2.1#2 | +| mimalloc 2.2.7 | 3.4.3 | +| qtbase 6.10.0#1 | 6.11.1#1 | +| freetype 2.13.3 | 2.14.3 | +| simdutf 9.0.0 | 8.2.0 *(older than the pin)* | +| …plus zlib, dbus, fontconfig, libedit, libwebp, libtommath, cpptrace, angle | | + +Concretely: ffmpeg 7.1.1#5's port is git-tree `0988005f…`, while +`HEAD:ports/ffmpeg` at the baseline is `c40aaa40…`. The bytes vcpkg builds are +**not in the working tree at any single commit** — they are extracted from the object +database by `read-tree`, per port, per pinned version. That is what the version +database *is*, and it is why `--depth 1` fails with vcpkg's own `Try again with a full +vcpkg clone`: shallow gives you the tree, and the tree is not the pin. + +So a submodule would deliver exactly the state that is *insufficient* — the baseline +checkout — while still requiring the full history behind it to be present, and it +would add costs of its own: the same 119 MB in `.git/modules` (no saving), plus +`Build/vcpkg` is inside a `Build*/`-ignored path that vcpkg fills with ~3 GB of +`downloads/`, `installed/`, `buildtrees/` scratch, so every cloner's `git status` +would show the submodule dirty forever (needing `ignore = dirty` in `.gitmodules` to +paper over it), and `git submodule update` would fight vcpkg for who owns the +directory. And a submodule still would not produce the `vcpkg` **binary** — that is +not tracked in the repo; it is bootstrapped from a tag+SHA512 pinned in +`scripts/vcpkg-tool-metadata.txt`. `Meta/ladybird.py vcpkg` does that too. + +The generalizable point, and the reason this is worth a section rather than a +footnote: **`Build/vcpkg` is not a vendored dependency, it is a package manager's +cache directory that happens to be a git checkout.** Every instinct that treats it as +"a pinned copy of another repo" — submodule, `git_repository`, `http_archive` of a +tarball, `--depth 1` — pins the checkout and loses the history, and the history is the +dependency. Getting this right is what `builtin-baseline` + `overrides` means, and it +is why the answer to "how do we get it" keeps coming back to *run the project's own +bootstrap*. + +## Finding 37: the overlay was not reproducible, and the thing that proved it was a `cp` + +Asked to get the tree onto another machine, I reached for the obvious answer — publish +the branch, `cp -r workspace/. ladybird/` — and then ran it on an empty directory +instead of describing it. Four things were wrong, and only the first was one I could +have found by reading. + +**Nothing recorded which Ladybird commit the overlay describes.** The generated BUILD +files name ~1,961 LibWeb compile inputs and 665 IDL bindings *by path*; they were +generated from exactly one upstream tree. That tree's sha appeared nowhere — not in +the README, not in `cmake2bazel.json`, not in a comment. Every parity claim in this +document is relative to a commit the document never named. This is the same class as +the `--depth 1` and release-tag mistakes: **a pin that is not written down is not a +pin**, and the reason it survived so long is that my working copy *was* the pin. + +**The interesting one: the overlay and upstream's bootstrap fight over `Build/vcpkg`.** +`Build/vcpkg/BUILD.bazel` is an overlay file, so a `cp -r` creates the *directory* +`Build/vcpkg`. Upstream's `Meta/Utils/build_vcpkg.py` then does: + +```python +if not vcpkg_checkout.is_dir(): + git clone … +else: + bootstrapped = git -C Build/vcpkg rev-parse HEAD +``` + +The directory exists, so it takes the `else`, and `git -C Build/vcpkg rev-parse HEAD` +— with no `.git` inside — **walks up to Ladybird's own repository** and cheerfully +returns *Ladybird's* HEAD. It then tries to check vcpkg's baseline out of the Ladybird +repo: `fatal: unable to read tree (40f3c709…)`. Two correct programs, one wrong +composition: upstream infers "cloned" from `is_dir()`, and the overlay's job is to put +a file in that directory. **`git`'s upward search for `.git` is what turns a missing +directory into a wrong answer instead of an error** — the same property that makes +`git -C` convenient makes it unsafe as an existence check. The fix is ordering +(prefetch, *then* stage that one file), and `apply_overlay.sh` defers it and explains +why at the point of deferral. + +The other two were mundane and would have cost someone an afternoon: `bazelrc.txt` +must be renamed to `.bazelrc` (stored under a different name precisely so a `cp -r` +cannot be mistaken for a working build — and then the recipe relies on a human +remembering the rename), and the two upstream patches must be *applied*, not merely +shipped. + +So the deliverable is a script, and the part worth keeping is `--verify`, which checks +what a file copy cannot: HEAD is the pinned commit, all 44 files are byte-identical, +the patches are applied (`git apply --check -R` succeeding is the proof — a patch that +reverse-applies cleanly is already in the tree), and the `.sh` files still have their +executable bit. That last check exists because of an earlier bug of exactly this shape: +scripts committed `100644` while my dev tree had them `+x` by hand, so only a fresh +clone failed, and only at action time. **The general rule this migration keeps +rediscovering: my working tree carries state git does not, and the only way to find it +is to reconstruct the tree somewhere else and diff.** `apply_overlay.sh /tmp/lbfresh2` +now produces a tree byte-identical to the one that renders, which is the first time +that sentence has been checked rather than assumed. + +## Finding 38: the pin recorded what my machine lacked, not what the build needs + +Ulf's clone failed where mine never could: + +``` +vcpkg_build: distfile MISSING FROM INDEX: + .../ninja-build/ninja/releases/download/v1.13.2/ninja-linux.zip +error: there were no asset cache hits, and x-block-origin blocks trying the + authoritative source +``` + +The 76-distfile pin came from *instrumenting vcpkg's own downloader* (finding 28) — +the strongest evidence available, and the thing I have leaned on hardest in this +document, because the capture cannot invent a URL and cannot miss one vcpkg asked +for. It has one blind spot, and it is not in the instrument, it is in the +**subject**: `vcpkg_find_acquire_program` probes the host *before* downloading. My +machine has `/usr/bin/ninja` at exactly 1.13.2 — the version vcpkg wants — so vcpkg +never asked for ninja, so the capture never saw it, so the pin never had it. Nothing +was broken. The observation was faithful; it was an observation *of my machine*. + +The reason this went unnoticed is the reason it is worth a finding: **cmake is in the +pin, and only by luck.** The host cmake is 4.2.3 against the required 4.4.0, so vcpkg +*did* download that one — and its presence made the whole class look covered. One +member of a category being present by accident is what a partial pin looks like from +the inside. + +Two general shapes, both of which I had already written down in weaker forms: + +- **An instrument that records what a program *did* cannot pin what the program + *would do elsewhere*, when the program's behaviour depends on the machine.** The + capture is exact about vcpkg's requests and silent about vcpkg's *decisions*. + Finding 36 said "a green check has to be compared against something it did not + produce"; the comparand here is vcpkg's own tool metadata, + `scripts/vcpkg-tools.json`, versioned inside the checkout at the baseline, carrying + url + sha512 + archive name for every tool on every platform. That is a **pin** + rather than an observation, so it is complete regardless of what is installed + anywhere. +- **Host-tool discovery is a hermeticity boundary that looks like a convenience.** + Every `find_program`/`find_package` is a place where the build's inputs depend on + the machine, and a capture-based pin will silently record the *complement* of + whatever is installed. The pin's contents should not be a function of one + machine's `/usr/bin`. + +The fix that first suggested itself — derive the tools from `vcpkg-tools.json` at emit +time — was wrong, and **two existing tests caught it before Ulf could**: the emitter's +central promise is that *the committed pin alone regenerates every Bazel file with no +vcpkg checkout, no CMake and no network*, and reading vcpkg's metadata during emit +quietly made a vcpkg checkout a requirement again. So the derivation is a separate, +deliberate step (`--capture-tools`) whose output is committed as +`Meta/vcpkg_tool_assets.tsv`, exactly like the asset capture; emitting unions it in +and, *when* a checkout happens to be present, cross-checks it and warns if the +committed pin has gone stale. That is the same division as everywhere else here: +regenerating a pin may use the world, consuming one may not. + +Scoping, stated because a reviewer should not have to infer it: +`vcpkg-tools.json` also pins dotnet, node, powershell-core, azcopy, gsutil, coscli and +nuget — ~400 MB for tools no port in this closure invokes (only the unrelated +`vbs-enclave-tooling-codegen` does). `BUILD_TOOLS` is `("cmake", "ninja")`, the two +`scripts/detect_compiler` needs before any port builds at all, and the emitter +*reports* the ones it skipped rather than hiding the decision. + +Verified: the ninja `http_file` fetches and integrity-checks (sha512 confirmed against +upstream independently of Ulf's error text), `bazel query 'deps(//:vcpkg_installed, 1)'` +now lists **78** distfiles including ninja, and the index the asset script reads +resolves that hash to the Bazel-fetched file — the exact lookup that failed on his +machine. Suite 182/182, six new tests, one of which is the regression test for the +bug report itself. + +## Finding 39: the pin fixed one class; the next two failures were a different class + +Finding 38's fix was correct and did not survive contact. Ulf pulled it, rebuilt, and +got — twenty minutes in, from inside `libvpx`: + +``` +CMake Error at scripts/cmake/vcpkg_find_acquire_program.cmake:201 (message): + Could not find nasm. Please install it via your package manager: +``` + +Installed nasm, rebuilt, twenty more minutes, from `gperf`: + +``` +CMake Error at .../share/vcpkg-make/vcpkg_make.cmake:108 (message): + gperf currently requires the following programs from the system package + manager: + + autoconf autoconf-archive automake libtoolize +``` + +Same blind spot as finding 38 — *the capturing machine had the tool* — but a +different **class**, and I had assumed the class was closed. Finding 38's fix reads +vcpkg's `scripts/vcpkg-tools.json` and pins url+sha512 for the tools vcpkg fetches +for itself. That works only for tools vcpkg *can* fetch. On Linux: + +```cmake +set(program_name nasm) +set(apt_package_name "nasm") +if(CMAKE_HOST_WIN32) + set(download_urls "https://www.nasm.us/.../nasm-3.01-win64.zip" ...) +endif() +``` + +Three URLs and a sha512 — **all inside the Windows branch**. On Linux there is +nothing to pin. vcpkg probes the host, does not find it, and stops. Six ports in +this closure need it (`dav1d`, `ffmpeg`, `libjpeg-turbo`, `libvpx`, `openh264`, +`openssl`). So the honest statement is not "the pin is incomplete" but **"this is +the boundary of the port"** — and the thing that was actually broken was not the +hermeticity, it was *how you find out*. + +**Three classes of input, not two.** Ring 2 had a two-box model — distfiles Bazel +fetches, and vcpkg's own tools (finding 38's pin). The third box is tools that can +only come from the host, and it needs different treatment because there is no URL +to put in it. Naming a gap is not fixing it; but an unnamed gap costs 20 minutes +per member to discover, one at a time, in an error that points at the wrong place. + +**And it has two mechanisms, which is why my first attempt missed half of it.** +I generalised from `nasm`, shipped a scan of `vcpkg_find_acquire_program` call +sites, and that would not have caught the autotools failure at all: `vcpkg-make` +never calls `vcpkg_find_acquire_program`. It calls bare +`find_program(AUTORECONF NAMES autoreconf)` and raises `FATAL_ERROR` with an apt +line. Worse, it does so from a **helper port**, so the error names `gperf` while +the requirement lives in a file `gperf` does not mention. A scan built from one +example is a scan calibrated to one example. + +So the derivation covers both, and the list is *derived* from vcpkg's own scripts +(`emit_vcpkg_bazel.py --host-tools` -> committed `Meta/vcpkg_host_tools.tsv`), +never hand-written: a baseline bump that adds a requirement is a +regenerate-and-review, not a rediscovery. + +**Most of the work was suppressing false positives, and that is the finding.** A +preflight that demands packages you do not need is one the third person deletes. +Four separate ways the naive scan cried wolf, each needing a real distinction: + +- **`CLANG`.** Both call sites are behind `if(... STREQUAL "MSVC")`. Reading call + sites without evaluating their guards demands a 2 GB toolchain on every Linux + machine. +- **`openssl`'s `NASM` and `CLANG`.** In `ports/openssl/windows/portfile.cmake` — + guarded by nothing in the file, only by the `include()` in its parent. The + platform split is at the **path** level, so the path has to be read. +- **`else()` after a negated test.** `dav1d` is `if(NOT VCPKG_TARGET_IS_WINDOWS) + ... else()` — that `else` *is* the Windows branch. Treating every `else()` as + reachable imports the Windows-only `GASPREPROCESSOR`. +- **`angle`'s `mesa-common-dev`.** A `message(WARNING)`. The portfile *also* has an + unrelated `FATAL_ERROR` about architectures, so a file-level "does it contain + both a FATAL_ERROR and an apt line" check staples them together. **Advice is not + a requirement**; the check anchors on the text of the `FATAL_ERROR` itself. + +Two entries can only be *named*, not verified: `autoconf-archive` ships m4 macros +and `libltdl-dev` ships headers, so there is no binary to probe. They are reported +as unverifiable rather than assumed satisfied — finding 35's rule again, in a third +place: a check that cannot fail must not look like a check that passed. + +Verified negatively, which finding 38 could not be (`sudo` hangs in this sandbox, +so I could not hide `/usr/bin/ninja`). Here the probe is `command -v`, so a +restricted `$PATH` *is* a machine without the tools: + +``` +vcpkg_build: MISSING host tool: libtoolize|glibtoolize (apt: libtool) +vcpkg_build: needed by: vcpkg-make +vcpkg_build: MISSING host tool: nasm (apt: nasm) +vcpkg_build: needed by: dav1d,ffmpeg,libjpeg-turbo,libvpx,openh264 +vcpkg_build: sudo apt install libtool nasm autoconf-archive libltdl-dev +``` + +Both, from one run, in one second, with the ports that need each and one pasteable +line — against the two failures that cost Ulf 40 minutes to learn two package +names. Confirmed in a real `bazel build //:vcpkg_installed`: the TSV is a declared +input (`aquery` shows `Meta/vcpkg_host_tools.tsv`), and the action ran the +preflight and proceeded into the build. Suite 193/193, 11 new tests — one per false +positive above, because each was a real bug in my own derivation. + +**The retrospective bit.** The environment notes at the bottom of this document +have said "plus autoconf/nasm/glslang/mesa GL dev libs" since the beginning. The +requirement was *documented and never checked* — so it was invisible to everyone +who did not read the bottom of a 2,300-line file, which is everyone. Prose in a +case study is not a preflight. `glslangValidator` is the same shape and is still +open: two genrules in `codegen_root.bzl` name `/usr/bin/glslangValidator`, and +`vcpkg_installed` does not ship it. + +## Finding 40: nine failures, one bug — a host requirement upstream checks and my overlay inherited silently + +Ulf's Ubuntu 24.04 machine ran the Bazel-built Ladybird headless and got a SIGSEGV +from the GUI: + +``` +ladybird(...) +QApplicationPrivate::init() +QXcbConnection::initializeScreens(bool) +QXcbConnection::handleScreenAdded(...) +--> SIGSEGV in libQt6Core +``` + +and offered a deal: *"if you can figure out how to make it work, then I won't +upgrade."* That machine is the most valuable thing in this migration. It has found +**eight** real defects (findings 37, 38, 39 and the gaps between them) that this +sandbox is structurally incapable of finding: gcc 15.2, Qt 6.10.2, ICU 78, +`nasm`/`perl`/`glslang`/`libdrm` all present, every requirement silently satisfied. +A machine that *disagrees* with mine is a test oracle, not an inconvenience, and the +only way to keep it is to stop breaking it. + +### The bug + +The binary links Qt from the Bazel repo and loads Qt's **plugins** from wherever +`libQt6Core`'s baked-in prefix points — on his box, the *distro* Qt's plugin +directory. Two different Qt builds in one process. + +Qt does not link its QPA platform plugin; `QApplication`'s constructor `dlopen`s it. +Where it looks is decided inside `libQt6Core`: `qt.conf` next to the executable, then +`QT_PLUGIN_PATH`, then the prefix compiled into the library. Qt 6.9.2's `qt_prfxpath` +is **empty** (`strings lib/libQt6Core.so.6.9.2`), so the prefix falls back to *the +directory of the executable* — and a Bazel binary's directory has no `platforms/`, so +the search falls through to the compiled-in system path and Qt loads +`/usr/lib/x86_64-linux-gnu/qt6/plugins/platforms/libqxcb.so` into a process whose +`libQt6Core` came from `bazel-bin/_solib_k8/...rules_qt++qt+qt...`. + +rules_qt is not at fault: it wires up Qt's *link* half faithfully (one SDK, discovered +by `qmake -query`; headers, libs and moc all from it). Nothing wired up the *runtime* +half, because on the machine where the overlay was written there was nothing to +notice. + +**Which way the skew points decides which failure you get.** Both halves reproduced +here, with a real X server: + +| plugin vs. linked libs | Qt's version gate | outcome | +|---|---|---| +| plugin **older** | rejects it: `factoryloader: Ignoring QPA plugin due to mismatching Qt versions 395520 394240` | `no Qt platform plugin could be initialized`, clean abort | +| plugin **newer or equal-minor** | **passes** | plugin calls into an ABI it was not built against → SIGSEGV in `initializeScreens` → `handleScreenAdded` | + +(Those integers are Qt versions: `(v>>16, (v>>8)&255, v&255)`, so 395520 = 6.9.0 and +394240 = 6.4.0.) The gate is the cruel part: it catches the harmless direction and +waves through the one that corrupts memory. + +And on **my** box it passes. `QT_DEBUG_PLUGINS=1` on the Bazel-built binary here +scanned `/usr/lib/x86_64-linux-gnu/qt6/plugins/platforms` and loaded the **distro** +`libqxcb.so` — the exact same wrong lookup — while `libQt6Core.so.6.10.2` came from +Bazel's solib dir. Both are 6.10.2, so the ABI happens to match. **The bug was +present in every green GUI run this project has ever reported.** + +Then the strongest evidence available: pointing `qt.local_repo` at the aqt **6.9.2** +SDK on this machine and rebuilding reproduced **his backtrace, frame for frame** — +`Ladybird::Application::create_platform_event_loop` → `QApplicationPrivate::init` → +`QXcbConnection::initializeScreens` → `handleScreenAdded` → a fault in libQt6Core at +`mov 0x8(%rdi),%rbx` with `rdi = 0`. Not a machine I cannot see any more. + +### The exoneration that mattered + +My first hypothesis was two ICUs: the binary needs `libicu*.so.78` (vcpkg) and aqt's +`libQt6Core` needs `libicu*.so.73`, both in one process, and `ld` even warns *"may +conflict"*. Wrong — worth recording because it is the kind of theory that sounds too +good to check. I linked ICU 78 **and** aqt's ICU 73 into one process with +`-Wl,--no-as-needed` and constructed a `QApplication`: `screens=1 qt=6.9.2`. ICU +coexists fine; two sonames are two libraries. (The fixed build now does exactly this +on purpose: `objdump -p` shows `libicu*.so.73` *and* `libicu*.so.78` in one binary, +and it runs.) Any earlier note here blaming "two ICUs" is superseded: it was never an +ICU problem, it was this same plugin/library provenance skew seen through a different +symptom. + +### The fix: the Qt edge becomes self-contained, like `vcpkg_lib` + +`qt_runtime.bzl`, and the shape is deliberately the one Ring 2 arrived at for vcpkg — +*the dependency arrives with the dep edge*: + +1. **`qt_plugins`** (repository rule) reads **@qt's own generated `qtconf.bzl`** for + `QT_INSTALL_PLUGINS` and symlinks every plugin the SDK ships into a repo, one + `filegroup` per plugin type. Reading @qt's file rather than a path of my own is the + whole point: the plugins cannot come from a different Qt than the libraries, + because both names come from one `qmake -query`. Every type, not the four Ladybird + needs today — a hand-picked list drifts, and a missing input method or file dialog + is a defect nobody notices for a month. +2. **`qt_plugin_tree`** re-declares them as outputs of the package that holds the + binary, so they land at `bazel-bin/plugins//*.so` — and, as `data` of + `//:ladybird`, in the runfiles tree too. +3. **`qt_conf`** writes the file that redirects the search: + + ``` + [Paths] + Prefix = . + Plugins = plugins + ``` + + Setting `Prefix` **replaces** the compiled-in prefix, so `/usr` is not outranked, + it is *never scanned* — `QT_DEBUG_PLUGINS=1` on the fixed build shows zero scans + of any `/usr` directory. `Prefix = .` is what makes one file correct in both + layouts, because Qt resolves it against the directory of the executable via + `/proc/self/exe`, so `bazel-bin/ladybird` and the runfiles tree's symlink to it + both land on the staged tree. +4. **`runtime_libs`** carries the private libraries an SDK bundles beside Qt. + +That fourth piece was the hard one, and it is where the loader stopped being +intuition and became a measurement. aqt's `libQt6Core` needs `libicui18n.so.73`, +which exists only inside the SDK, so the binary died in `ld.so` before `main()` and +`LD_LIBRARY_PATH=/lib` was the workaround — a workaround a human has to remember +is a bug that has been rounded down to a habit. **No rpath on the binary can fix +it**, for two reasons I only believed after reducing them to three generated `.so` +files: + +- `libQt6Core` finds its own ICU through `RUNPATH $ORIGIN`, and `$ORIGIN` is the + directory the loader **opened the object by** — Bazel's solib dir, not the SDK. (`ldd` + on that very symlink resolves ICU happily, because `ldd`'s `$ORIGIN` is the + realpath's directory. That near-miss is what made this look like a path problem.) +- Adding the SDK dir to *our* rpath does not help either: `DT_RUNPATH` is consulted + only for an object's own direct dependencies, and while `DT_RPATH` **is** inherited + by transitive loads, an intermediate object that has a `DT_RUNPATH` of its own + blocks the inherited `DT_RPATH` entirely. `libQt6Core` has one. All four + combinations measured before believing it. + +So the fix is not a search path at all: make the SDK's private libraries real link +inputs, so **Bazel** stages them and the binary's own runpath — the one glibc will +consult, because they are now direct dependencies — resolves them. The list is +derived, not written down: DT_NEEDED of the SDK's Qt modules ∩ the non-Qt `.so` files +beside them. For aqt 6.9.2 that is exactly `libicui18n/libicuuc/libicudata.so.73`; +for a distro Qt it is empty and the target degenerates to nothing. + +Five things I got wrong on the way, each caught by a probe rather than by reasoning: + +- **The staged plugins must be SYMLINKS, not copies.** A plugin needs Qt libraries + the binary does not link (`libqxcb.so` → `libQt6XcbQpa.so.6`), and aqt's plugins + carry `RUNPATH $ORIGIN/../../lib`, resolved from the object's real path. Copying + breaks exactly that; the distro's plugins have no `RUNPATH` at all, so a *copied* + distro plugin resolves `libQt6XcbQpa.so.6` from `/usr` — reintroducing the bug + through the fix. +- **`plugins/` is a substring of `qt_plugins/`.** My path-stripping matched inside the + *repository name* and staged everything one directory too deep + (`bazel-bin/plugins/plugins/...`), which built cleanly and pointed `qt.conf` at an + empty tree. Found by looking at the output, not by the build failing. +- **"Non-Qt libraries beside libQt6Core" is the whole system on a distro Qt.** Its lib + dir *is* `/usr/lib/x86_64-linux-gnu`, so the first version of the derivation + proposed linking 56 `cc_import`s including `ld-linux` into the binary. It is a + system directory, so it "worked" — which is precisely the kind of accident this + finding is about. The discriminator is whether the Qt lib dir is itself a default + loader directory. +- **An embedded `:/qt/etc/qt.conf` Qt resource does not work.** Tempting, since we + already run `rcc`, and Qt 6.9.2 does look for that path — but the resource is + registered by a static initialiser in the binary and every variant I built ignored + it, while a file on disk worked first try. +- **`--no-as-needed` is load-bearing.** The binary references no ICU 73 symbol (it has + its own ICU 78), so the linker drops the `DT_NEEDED` as unused and the staging + silently stops working. + +### The class, which is the finding + +Every one of the nine failures Ulf's machine has produced is the same sentence: + +| # | symptom on his box | the host requirement | who declares it | +|---|---|---|---| +| 1 | `vcpkg: No such file or directory` | the `Build/vcpkg` clone | `Meta/ladybird.py vcpkg` | +| 2 | `fatal: unable to read tree` | that clone's `.git`, at a baseline | vcpkg's baseline resolution | +| 3 | HSTS table differs | Chromium's JSON, *unpinned* | `Meta/CMake/hsts_preload.cmake` | +| 4 | `glslangValidator: No such file` | `glslang-tools` | nothing — my genrule hardcodes `/usr/bin` | +| 5 | `ninja-linux.zip MISSING FROM INDEX` | `ninja` ≥ 1.13.2 | `vcpkg_find_acquire_program` (probes, then downloads) | +| 6 | `Could not find nasm` | `nasm` | `vcpkg_find_acquire_program` (Linux: no URL to pin) | +| 7 | `gperf requires autoconf autoconf-archive automake libtoolize` | autotools | `vcpkg-make`'s bare `find_program` | +| 8 | libdrm headers not found | `libdrm-dev` | CMake `pkg_check_modules(... REQUIRED)` | +| 9 | **SIGSEGV in `initializeScreens`** | **Qt ≥ 6.9, and its plugins** | **`find_package(Qt6 6.9 REQUIRED)`** — upstream checks it; my overlay did not | + +Not nine bugs. One bug, nine times: **a host requirement that upstream declares and +checks, which the Bazel overlay inherited without inheriting the check.** CMake's +`find_package`, `pkg_check_modules(REQUIRED)` and `vcpkg_find_acquire_program` are not +ceremony — they are the *preflight*, and translating a build system means translating +its preflight, not only its compile lines. Nine times I translated the commands and +dropped the assertion; nine times the machine that disagreed with mine was the one +that told me. + +So `qt_plugins` also carries the floor. `UI/Qt/CMakeLists.txt` says +`find_package(Qt6 6.9 REQUIRED COMPONENTS Core Widgets)`; the repo rule now fails at +fetch time with the version it found, the prefix it found it in, the package to +install, and a warning not to mix SDKs — the finding-39 mechanism (derive the +requirement from the source of truth, check it where it is needed, name the fix in the +error) extended from vcpkg's driver to the overlay's own build. `glslangValidator` +(#4) is the last member of the table with no check at all. + +### One more wrong path, found by walking the recipe + +Verifying the GUI meant following the README's own staging recipe, which promptly +failed with `UNEXPECTED ERROR: stat: No such file or directory at +UI/Qt/WebContentView.cpp:1047` — the theme `.ini`. Two of its paths were wrong, both +in the same way as this finding: written down instead of derived. The vcpkg tree it +copies from is built in the **exec** configuration, so `bazel-bin/vcpkg_installed` +does not exist at all (`k8-fastbuild-exec` is the directory); and the resource root +is not `/share/Lagom` but `/../share/Lagom`, because +`LibWebView/Utilities.cpp`'s `find_prefix()` takes the **parent** of the binary's +directory. Both are now `bazel info` / `bazel cquery --output=files` invocations, +which answer for whatever configuration you are on. Prose describing an output path +is a pin with no checker (gap 5, closed). + +**Verified by removal**, which is the only kind of verification this document +accepts. With `/usr/lib/x86_64-linux-gnu/qt6/plugins` hidden behind an empty tmpfs and +**no `LD_LIBRARY_PATH`**: the aqt build loads `libqxcb.so` from the aqt SDK (previously: +the distro's), zero `/usr` directories are scanned, and against the host Qt the GUI +opens its window on Xvfb and stays up. Headless still renders the reference page. +Passing because the host happened to agree is how all nine of these got here. + +## Finding 41: the plugin fix was right and the crash stayed — `-fPIE` gave `qApp` a copy relocation + +Finding 40 fixed the plugin path, Ulf updated his tree, and the GUI segfaulted in +`QXcbConnection::initializeScreens` **again**, with the same backtrace. That is the +most useful shape a bug can have: it proves the previous fix was necessary, not +sufficient, and it means the earlier reasoning had a passenger. + +What the new evidence ruled out, before any theory: + +- `QT_DEBUG_PLUGINS=1` showed the scan hitting `bazel-out/.../bin/plugins/platforms` + and every plugin resolving to *his own* SDK's realpath. `/usr/lib/x86_64-linux-gnu/qt6` + was never touched. Finding 40's fix was working exactly as documented. +- `info sharedlibrary` showed **one** `libQt6Core` in the process, and the `Qt 6.9.2 + (x86_64-little_endian-lp64 ... GCC 10.3.1)` build strings of the Bazel-staged copy and + the SDK copy were byte-identical. Not a version mix, not two Qts. +- The **offscreen** QPA plugin crashed identically. Whatever this was, it was not the + platform plugin, not xcb, not the X server, not a screen enumeration quirk. + +### The bug + +At the fault, `rdi = 0` and `si_addr = 0x8`, on `mov 0x8(%rdi),%rbx` — inside +`doActivate` (`libQt6Core` + 0x1e89ce, just past +`QObjectPrivate::ConnectionData::deleteOrphaned`), reached from +`QWindowSystemInterface::handleScreenAdded` → `QGuiApplication::screenAdded`. Qt was +emitting a signal from a **null `qApp`**, in the middle of `QApplication`'s own +constructor, which had just set it. + +`readelf -rW` says why, in three lines: + +| object | relocation against `_ZN16QCoreApplication4selfE` | +|---|---| +| aqt `libQt6Core.so.6.9.2` | **none** — accessed PC-relative, its own BSS | +| aqt `libQt6Gui.so.6.9.2` | `R_X86_64_GLOB_DAT` — read through the GOT | +| `bazel-bin/ladybird` | **`R_X86_64_COPY`** — definition materialised in the exe | + +A copy relocation moves the *definition* of an extern data symbol into the +executable's BSS and repoints every GOT slot at it. So `QApplication`'s constructor +made Core write `self` in **Core's** BSS, while Gui read the **executable's** BSS, +which was still zero. The first emit through it dies. His distro Qt 6.4.2 *does* +carry a `GLOB_DAT` for `self` in Core, so both halves agree there and the bug cannot +appear — which is why this only ever showed up against an official SDK. + +The cause is one flag. CMake puts `-fPIE` on every executable target +(`CMAKE_POSITION_INDEPENDENT_CODE` + an exe), the capture recorded it faithfully, and +`emit_build_bazel.py` copied it into `copts` on all seven generated `cc_binary` +targets. Bazel appends per-target copts **after** the `.bazelrc`'s `--copt=-fPIC`, +and for GCC the last of the pair wins: the UI/Qt objects compiled `-fPIE` while every +library around them compiled `-fPIC`. Under `-fPIE` GCC may reference extern data +directly instead of through the GOT, and the linker turns that into the copy +relocation. Qt is built with `reduce_relocations` (in `mkspecs/qconfig.pri`; +Debian's is not), which is what makes Core's side of the disagreement PC-relative. + +**Qt diagnoses this and the diagnosis could not fire.** +`qcompilerdetection.h` has `#error "-fPIE is not sufficient if Qt was configured with +-DFEATURE_reduce_relocations=ON ... Compile your code with -fPIC and without -fPIE"` +— guarded by `#if ... || defined(__PIC__)`. Bazel passed **both** flags, so `__PIC__` +was defined at preprocess time and the guard never triggered. The build was clean and +the binary was broken: a compiler-authored checker, defeated by flag order. + +### Reduced, then fixed + +Eight lines, no Ladybird, no Bazel — `QGuiApplication` plus one `printf`, against the +same SDK: + +``` +g++ -fPIE -pie ... -> R_X86_64_COPY for self -> SIGSEGV (identical stack) +g++ -fPIC -pie ... -> GOT, no COPY reloc -> "instance=0x… screens=1" +``` + +`-Wl,-z,nocopyreloc` is **not** an alternative: it converts the same defect into +`Symbol _ZN16QCoreApplication4selfE causes overflow in R_X86_64_PC32`. The fix is to +stop asking for `-fPIE`, which is also not a divergence from CMake's semantics — +Bazel compiles a `cc_binary`'s objects PIC and links `-pie`, which is what +`POSITION_INDEPENDENT_CODE` was asking for. So the flag is dropped in the generator +(`DROPPED_TARGET_FLAGS`), not in the generated file, because `BUILD.bazel` is output +and a hand-edit survives exactly one regeneration. + +**Verified by removal**, on both Qts and all six executables: `R_X86_64_COPY` count +39 → **0** (including `qApp`, and incidentally `stdout`, `QString::_empty` and 20-odd +`staticMetaObject`s, every one of them the same latent hazard). Against the aqt 6.9.2 +SDK the GUI now starts where it previously segfaulted under *both* the xcb and the +offscreen plugin; against the distro Qt, unchanged. Guarded by +`tests/test_pie_copy_relocation.py` (5 tests), which asserts the generated file is +`-fPIE`-free, that the drop lives in the emitter and is consulted, that the reason is +written at the drop site, and that global `--copt=-fPIC` — the thing that makes +dropping `-fPIE` correct — is still in `bazelrc.txt`. + +**What this cost, and the lesson.** Two rounds on Ulf's machine for one crash, because +after finding 40 I read "SIGSEGV in `initializeScreens`" as "the plugin bug, again" +instead of as an unexplained fault. The plugin evidence was *checkable in one command* +(`QT_DEBUG_PLUGINS=1`) and I asked for it only on the second pass. A backtrace that +survives a fix is not the same bug; the frames say where a process died, never why. +It also makes finding 40's own "verified by removal" honest about its scope: hiding +the host plugin directory proved the plugins were right, and proved nothing at all +about the objects that linked them. Two Qts in one process was a real bug. One Qt with +two copies of one pointer was the next one down. ## Plan for the rest @@ -1493,24 +2699,118 @@ every process (UI, WebContent, Compositor, RequestServer, ImageDecoder) Bazel-built, proven by removing the reference services and watching the CMake binary fail while the Bazel one renders. -And with findings 33 and 34 it is a **clone-and-build**: Bazel fetches and builds -the 77 vcpkg ports *and* the 10 Rust crates + `flapc` itself, with zero network -access in the build actions, verified by removing each reference tree from the -machine in turn. `Build/full` is now only the *converter's* input — the model the -emitters read and the baseline the parity harness diffs — not the build's. +And with findings 33, 34 **and 35**, Bazel fetches and builds the 77 vcpkg ports, +the 10 Rust crates, `flapc`, `cranelift-compiler` and every generated header itself, +with vcpkg's own downloader reaching the network zero times. It is **not** yet a +clone-and-build: finding 36 lists the four inputs a fresh clone still lacks and the +one vcpkg port that bypasses the pin with `pip`; findings 38 and 39 add the two +classes of *host tool* that a capture on one machine structurally cannot see. +`Build/full` is now only the *converter's* input — the model the emitters read and +the baseline the parity harness diffs — not the build's, and this time that is +measured: **0 targets under `Build/full` in the `cquery` closure of all six +binaries** (down from 741), the three shim packages deleted outright, and all six +binaries rebuilt from scratch with CMake's generated tree moved off the machine, +rendering `--headless=text` *and* `--headless=layout-tree` byte-identically on +three pages including two that execute WebAssembly. + +Findings 33 and 34 alone were **not** enough, and the gap between "I closed the +binary dependencies" and "the build does not read the tree" is the most +instructive part of this migration — see finding 35. **Scoreboard:** 43 `cc_library` + 6 `cc_binary` targets; ~4,400 Bazel actions from scratch (2,700 C++ plus the vcpkg and Rust rings); LibWeb alone 1,961 TUs (1,273 checked-in + 688 generated) with **zero** define/flag/include discrepancies vs CMake; 1,379/1,379 generated files -byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; 34 findings, 3 of them real any2bazel engine fixes with +byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; **0 targets under `Build/full` in the closure of all six binaries (down from 741) and 0 absolute host paths in the emitted BUILD files** (finding 35); a fresh clone still needs 4 inputs the overlay does not carry and one port reaches PyPI via pip (finding 36 — found by actually cloning), and the host tools vcpkg cannot download are +now named and preflighted rather than discovered one 20-minute build at a time +(findings 38, 39); the Qt runtime half -- the plugins Qt `dlopen`s -- wired to the +same SDK as the libraries (finding 40) and the `-fPIE` copy relocation that broke +`qApp` against any `reduce_relocations` Qt removed (finding 41, 39 -> 0 +`R_X86_64_COPY`); 41 findings, 3 of them real any2bazel engine fixes with regression tests. +**One number in this scoreboard was wrong for most of the migration, and it is +worth ending on.** "Clone and build" was claimed after the two *binary* +dependencies were closed, and the 741 remaining *header* dependencies went uncounted +because the shims that supplied them could not fail. Every "verified by removal" in +this document is only as strong as what was actually removed — which is the argument +for removal as a method, not against it: it is the only check here that ever found +this class of bug, and each time I widened what I removed, it found another. + +## Finding 42: the browser ran, then ran out of file descriptors — and my first four diagnoses were theories, not counts (and my fifth, the patch, was only half of it) + +The Bazel-built browser worked and then died overnight with `dup: Too many open files +(errno=24)`. This finding is not about the migration at all: it is an upstream Ladybird +bug that a long-running browser finds and a short test never does. It earns a place here +because of *how badly I diagnosed it*, four times, before doing the obvious thing. + +The wrong answers, in order: lost fd acknowledgements in `TransportSocket` (Ulf correctly +objected that a local `SOCK_STREAM` socketpair does not lose acks); a missed +`AnonymousBuffer` / image-frame cache eviction (there were no memfds at all); leaked +`TransportSocket`s (an artifact of **my own broken one-liner** — `sed 's/[0-9]*$//'` does +not strip `pipe:[123]`, and `| head` hid the real distribution); and then +`MessagePort::entangle_with`, which is a real leak that I read out of the code with high +confidence and which **is not the bug that killed his browser**. + +What settled it was a count, not an argument. Ulf's two `/proc//fd` censuses showed +17,423 → 17,497 `socket:` with a *flat* 18 `pipe:`. The MessagePort leak allocates one +socketpair **and two `pipe2` pairs** per port, so it leaks pipes and sockets at 4:1 — I +reproduced it locally at 1,628 pipes : 409 sockets. **It cannot produce a sockets-only +census.** The signature falsified my own best theory. + +Following the signature instead led to a leak of exactly **one socket per completed HTTP +request** (103 requests → 107 sockets; 1,164 → 1,157; pipes never move), which matches +Ulf's shape exactly and needs no exotic page — any browsing session leaks monotonically. +`internals.dumpGCGraph()` names the holder directly, because GC roots carry their source +location: after 202 requests, `808 Root +nonstandard_resource_loader_file_or_http_network_fetch Fetching.cpp:2338` — 4 roots × +202 requests. The four fetch callbacks are `GC::Root`s (strong, uncollectable) moved into +lambdas on the refcounted `Requests::Request`, which the `Response` then holds back by +`RefPtr` — a cycle spanning the GC heap and the refcount heap, which neither collector +can break, keeping the response fd open forever. `Request::defer_teardown()` is the only +thing that clears the callbacks, and normal completion never calls it (only `stop()` and +`did_transfer()` do). The A/B: aborted fetches, which *do* reach `stop()`, leak zero. + +Then the last lesson, which is the one I keep having to relearn: I wrote the "obvious" +one-line fix (`defer_teardown()` at the end of `did_finish()`), rebuilt, and the leak went +to **zero** — because loading was broken and pages rendered blank. A fix that makes the +symptom disappear by removing the behaviour is indistinguishable from a fix, unless you +check that the feature still works. I reverted it, re-verified that the clean build both +renders *and* leaks, and took the diagnosis upstream without pretending I had the patch. + +Write-up for upstream: `docs/UPSTREAM-ladybird-fd-leaks.md` (both bugs, with the +reproductions, the census method, and the cascade that turns one `EMFILE` into three dead +processes via `MUST`/`VERIFY`). + +**Postscript, and the actual lesson.** An upstream patch equivalent to that fix landed; +Ulf applied it; his browser still leaked. My patch was necessary and not sufficient, and I +had written the doc as though it were the fix. What resolved it was making the census +*self-classifying* instead of arguing about which page shape was to blame: an in-process +`poll()`/`MSG_PEEK` probe on each retained response fd, which reports whether the **peer** +is dead or alive. That one column splits the leak into two bugs with different fixes — +peer **dead** means the request completed and WebContent is retaining a corpse (the +teardown patch fixes exactly this, 143 → 0 locally); peer **alive** means `on_finish` +never ran, so no teardown at that point can fire, and the fd is held by the GC-root ↔ +`RefPtr` cycle itself (40 → 40, unchanged by the patch). The same probe also separates +*in-flight* from *retained* by age, which is what stops a freshly restarted process from +looking like a fix — a mistake I made once already in this investigation. + +Two habits earned that: measuring the resource rather than reasoning about the code (the +`pipe:`-to-`socket:` ratio is what falsified my confident MessagePort diagnosis), and +making the *instrument* answer the classification question, so the next report is a count +instead of another theory. The diagnostic build is kept, deliberately outside the overlay's +`patches/*.patch` glob and pinned there by a test, as +`examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt`. + ## Environment notes (this sandbox) - Toolchain via apt (needs passwordless sudo): cmake, ninja, ccache, build-essential, Qt6 (`qt6-base-dev` etc., 6.10.2), plus autoconf/nasm/ - glslang/mesa GL dev libs. Rust via rustup (`~/.cargo`). Node/vcpkg bootstrap + glslang/mesa GL dev libs. **The vcpkg half of that list is no longer prose:** + `Meta/vcpkg_host_tools.tsv` is derived from vcpkg's own scripts and checked + before the build starts (finding 39). This line said "autoconf/nasm" for + months while both failures below were waiting to happen — documenting a + requirement is not checking it. Rust via rustup (`~/.cargo`). Node/vcpkg bootstrap per `Meta/Utils/build_vcpkg.py`. - `~/lb-env.sh` sets `LADYBIRD_SOURCE_DIR`, `VCPKG_ROOT`, CA certs. - Disk ~126G, RAM 16G, 16 cores. vcpkg from-source (no cache hit) for diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md new file mode 100644 index 0000000..096aa6f --- /dev/null +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -0,0 +1,711 @@ +# Ladybird: fd leaks in WebContent (per HTTP request, and per un-`close()`d MessagePort) + +Status: reproduced locally on a **CMake** build of `f9e34731` (no Bazel involved); +present in upstream `master` (`50eef049`) by inspection of the same code paths. + +**Read this first — the per-request leak is two distinct bugs, not one.** An upstream +patch equivalent to the teardown fix below now exists, Ulf applied it, and *his browser +still leaks*. That is not a contradiction: the fix closes the class of retained request +whose peer is already **dead**, and there is a second class whose peer is still **alive** +that no teardown at that point can reach. The discriminator is one column of `ss -np`: + +| class | `ss -np` peer inode | what happened | teardown patch | +|---|---|---|---| +| A: completed | `* 0` (**dead**) | request finished, RequestServer closed its half, WebContent retains a corpse | **fixes it** (143 → 0 locally) | +| B: stalled | a real inode (**alive**) | `on_finish` never ran, so no teardown fires at all | **unaffected** (40 → 40 locally) | + +Count them before choosing a fix: + +``` +ss -np | grep "pid=$P," | grep -c ' \* 0 ' # class A (dead peer) +ss -np | grep -c "pid=$P," # all of this process's unix sockets +``` + +**Two instruments, both usable on any tree, no patch required:** + +- `examples/ladybird/fd_census.py` — *how many, which class, and is it still growing.* +- `examples/ladybird/fdtrace.c` + `fdtrace_report.py` — *which call site.* Build once + (`cc -shared -fPIC -O2 -g -o fdtrace.so fdtrace.c -ldl`), `LD_PRELOAD` it into an + unmodified browser, and the report says **who sent** each fd that was never + closed. It hooks `recvmsg`/SCM_RIGHTS because the leaked fd is **received, never + opened** — a tracer wrapping only `open()`/`socket()` sees nothing — and it reads + `SO_PEERCRED` on the receiving socket, because the acquisition *stack* of an + attachment is always the IPC read thread and therefore identical for every peer. + Validated on the known leak: `143 from RequestServer, 5 from Ladybird`, matching + the census's 143 dead-peer sockets exactly. + +`examples/ladybird/fd_census.py` does the classification from **outside** a running +browser — no patch, no rebuild, no pinned tree: + +``` +python3 examples/ladybird/fd_census.py --find WebContent +python3 examples/ladybird/fd_census.py --watch 30 +``` + +It prints the category census, peer DEAD/ALIVE per socket, retained-vs-in-flight by +age (so a freshly restarted process cannot be mistaken for a fix), which process holds +the live peers, and a verdict naming the class. Verified to agree with an +instrumented build on the same workload (143 DEAD / 4 ALIVE either way). The +in-process version is kept as +`examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt` for the few fields +only it can see (request id, `user_finish_called`, and a one-build A/B of the fix). + +Ulf's Bazel-built browser died overnight with + +``` +dup: Too many open files (errno=24) at Libraries/LibWebView/CompositorConnection.cpp:62 +``` + +Two `/proc//fd` censuses minutes apart showed **17,423 → 17,497 `socket:`**, with +only 18 `pipe:` and nothing else growing. Monotonic, unbounded, and *sockets only*. + +That last detail is what makes this report two bugs instead of one. My first diagnosis +(`MessagePort::entangle_with`) was **wrong for Ulf's crash**: it leaks 4 pipes for every +1 socket, so it cannot produce a sockets-only census. Counting fds by category, rather +than reasoning about which code looked suspicious, is what separated them — and the +per-request leak below matches his census exactly. + +--- + +## Bug 1 (the one that kills the browser): every completed HTTP request leaks one socket fd + +### Reproduction + +Serve a one-line `index.html` on localhost and load: + +```html + +``` + +`Build/full/bin/Ladybird --headless=screenshot --screenshot-delay 120 `, then census +the WebContent process: + +``` +ls -l /proc/$P/fd | awk '{print $NF}' | sed 's/\[[0-9]*\]//' | sort | uniq -c | sort -rn +``` + +Measured (clean build, no local patches): + +| requests served | WebContent `socket:` fds | `pipe:` fds | +|---|---|---| +| 0 (baseline) | 5 | 18 | +| 103 | 107 | 18 | +| 202 | 208 | 18 | +| 595 | 587 | 18 | +| 1164 | 1157 | 18 | + +**One socket per request, forever, and the pipe count never moves** — Ulf's census shape. +They are never reclaimed: the count stays flat for minutes after the loop stops, across +repeated explicit `internals.gc()` calls, and with `--disable-http-memory-cache +--disable-http-disk-cache`. `/proc/net/unix` shows them as connected `AF_UNIX` +`SOCK_STREAM` halves — RequestServer's response-body pipe (`RequestPipe::create`, +`Services/RequestServer/RequestPipe.cpp:46`). RequestServer closes its end (its own fd +count is flat at 4 sockets); **WebContent never closes its end.** + +It is not specific to `fetch()`: plain subresource loads leak identically — 50 `` +loads give 57 sockets, 200 give 207 (same +1/request against the same baseline of 5–7). + +### Cause: an explicit GC root anchoring a cycle that spans the GC heap and the refcount heap + +`internals.dumpGCGraph()` labels roots with their source location, which names the holder +outright. After 202 requests and two forced GCs: + +``` +808 Root nonstandard_resource_loader_file_or_http_network_fetch Libraries/LibWeb/Fetch/Fetching/Fetching.cpp:2338 +333 VM + 35 StackPointer +``` + +808 = **4 × 202**: the four callbacks passed to `ResourceLoader::load` at +`Fetching.cpp:2338`, one set per request, none ever released. Live cells scale in lockstep +(202 requests → 202 `FetchedDataReceiver`, 606 `Response`, 808 `PendingResponse`, 406 +`ReadableStream`). + +The loop: + +1. `nonstandard_resource_loader_file_or_http_network_fetch` creates four + `GC::Function`s (`Fetching.cpp:2261`–`2333`) and passes them as + `GC::Root<...>` into `ResourceLoader::load` + (`Libraries/LibWeb/Loader/ResourceLoader.h:43`). +2. `GC::Root` is a **strong, explicit root**: `RootImpl`'s constructor registers itself in + `Heap::m_roots` (`LibGC/Root.cpp:15`, `Heap.h:251`) and `gather_roots` marks every entry + (`Heap.cpp:841`). A `GC::Root` is uncollectable by construction — it is only released + when the `RootImpl` is destroyed. +3. `ResourceLoader::load` moves those roots into the lambdas it installs on the + **refcounted** `Requests::Request` (`ResourceLoader.cpp:469`–`513`, stored as + `on_headers_received` / `on_finish` / …). +4. `on_headers_received` puts the very same `Requests::Request` **back into the GC heap**: + `response->set_request_server_request({… .request = request_server_request})` + (`Fetching.cpp:2276`), and `Response` holds it as a + `RefPtr` (`Fetch/Infrastructure/HTTP/Responses.h:223`/`226`). + +So: `Heap::m_roots` → `GC::Function` → captured `pending_response` / `stream` / +`fetched_data_receiver` → `Response` → `RefPtr` → the lambdas holding +the `GC::Root`s. Neither mechanism can break it — the GC sees a live root, and the +refcount never reaches zero. `Requests::Request` owns the response fd (`m_fd`, closed only +in `~Request`, `LibRequests/Request.cpp:58`, and kept open meanwhile by the `ReadStream` / +`Core::Notifier`), so **the fd leaks with the cycle**. + +The only thing that ever clears those callbacks is `Request::defer_teardown()` +(`Request.cpp:284`), and its only callers are `stop()` (`:69`) and `did_transfer()` +(`:274`). **Normal completion never calls it**: `did_finish()` (`:251`) invokes +`on_finish` and returns, and `RequestClient::request_finished` (`RequestClient.cpp:205`) +only removes its map entry — the `RefPtr` inside `Response` keeps the request, its +callbacks, its roots and its fd alive for the lifetime of the process. + +### A/B that confirms it + +Same page, but each fetch is immediately `AbortController.abort()`ed — `abort` reaches +`Request::stop()`, which *does* call `defer_teardown()`: + +| variant | 200 requests | +|---|---| +| completed fetches | +200 sockets (207) | +| aborted fetches | **+0 sockets (flat 207 → 207)** | + +The path that tears down does not leak; the path that succeeds does. + +### Class B: the request that never finishes at all (survives the teardown fix) + +The teardown above hangs off the branch in `Request::set_up_internal_stream_data`'s +`on_finish` that decides the body is complete: + +```cpp +if (!user_finish_called && (!read_stream || read_stream->is_eof() || has_received_all_reported_bytes)) { +``` + +If a response never satisfies that *and* never reaches EOF, the block never runs, so +`user_on_finish` never runs, so **no teardown placed inside it can ever fire**. Repro: a +server that sends `Content-Length: 48000`, writes 100 bytes and then holds the socket +open. 40 such loads, censused with the diagnostic build after they have aged past 30s: + +``` +FDLEAK live: in_flight(<30s)=0 retained(>=30s)=40 +FDLEAK retained-bucket 40 x peer=ALIVE torn_down=false did_finish=false user_finish=false + request_done=false stream=true eof=false fd_open=true +``` + +Identical with `LADYBIRD_FDLEAK_TEARDOWN=1` (40 → 40): `did_finish=false` is the tell — +the `request_finished` IPC never arrived, so the fix is not even on the code path. And +`peer=ALIVE` distinguishes it from class A at a glance: RequestServer is still holding its +end, because as far as it knows the body is still being produced. + +By contrast every completed-request shape I can construct — plain fetch, XHR, POST, +`response.clone()`, unread bodies, ``/CSS/JS subresources, 404/302/cache hits, +truncated bodies, gzip/chunked, iframe navigations, mid-body `cancel()` — is flat at the +5-socket baseline with the fix on, and shows `peer=DEAD` retention without it. + +So the two classes need different fixes, and only one of them is "release the callbacks +when the body is delivered": + +- **Class A** is a *liveness* bug in the teardown path → the teardown patch. +- **Class B** is the cycle itself. As long as `Response` holds `Requests::Request` by + `RefPtr` while the request's callbacks hold `GC::Root`s back into the GC heap, ANY + request that never completes is retained forever, and there is no completion callback + left to hook. The fix has to break the cycle at the `Response` end — e.g. hold the + request weakly, or root the callbacks from something whose lifetime the GC can actually + end — rather than adding another teardown call site. + +### On fixing it — one obvious patch is wrong, and I verified that + +Calling `defer_teardown()` at the end of `did_finish()` looks like the one-line fix. **It +is not**: I applied exactly that, rebuilt `liblagom-requests`, and the leak went to zero +(flat 5 sockets) *because loading broke* — pages rendered blank, since the body may still +be draining when `did_finish` arrives (`on_finish` re-checks +`read_stream->is_eof()` / `user_finish_called` for precisely this reason). I reverted it +and re-verified the clean build renders and leaks again; the numbers above are all from +the unpatched build. + +A correct fix has to release the callbacks (and with them the `GC::Root`s) only once the +body is genuinely finished — e.g. at the point `on_finish` decides +`user_finish_called`/EOF, which is what the upstream patch and +`patches/0003-*.patch` do — **and** break the cycle at its other end so `Response` does +not keep the `Requests::Request` alive strongly, which is the only thing that can reach +class B. Doing only the first is measurable progress and not a fix: it is the same shape +as the `-fPIE` finding, where the change was necessary, verified, and still left the +symptom alive by another route. + +### The fix that actually closes class A everywhere: release the fd, not just the callbacks + +The teardown patch above was **necessary and insufficient**, and I only learned that from +Ulf's tree, not mine. With the upstream-equivalent teardown applied he still measured +**97 sockets/min** — 81→823 sockets, 73→815 dead-peer over 458 s — while every local +workload I could build was flat. Two instruments settled what was leaking: + +- `fd_census.py`: every leaked socket was **peer=DEAD**, i.e. RequestServer had already + closed its end. So these were completed requests, class A, on a tree carrying the class A + fix. +- `fdtrace.c`: every leaked fd was a **received SCM_RIGHTS attachment** whose sender was + **RequestServer** (`SO_PEERCRED`; his log said `from=?(pid=2261433)` because a renderer's + landlock policy grants only `/proc/self`, and `ps -p 2261433` named it). That pins the fd + exactly: the per-request response pipe from `RequestServer::RequestPipe::create()`, handed + over by `request_started`. + +The gap is **ownership, not liveness**. Dropping the callbacks unpins the GC cycle, but the +fd itself is released only by `~Request` — or by the `ReadStream` inside +`m_internal_stream_data`, which the teardown merely *nulls*. So any surviving reference to +the `Requests::Request` (the `RefPtr` in `Response`, a callback captured elsewhere, a +different retention path on a different tree) keeps **one fd per completed request** alive +even though the teardown ran. That is why the same patch zeroes the leak on my tree and +leaves 97/min on his: we differ in what else holds a reference, not in what the teardown +does. + +`0004` is the **next patch in the series**, applied on top of `0003` (or upstream's +equivalent teardown fix) — not an alternative to it. Two mistakes on the way there, both +caught by Ulf within minutes and both now pinned by tests: + +1. I generated it by diffing against the *clean* commit, so it silently carried `0003`'s + own hunk and could not apply to the tree it was written for (`patch does not apply`). +2. Correcting that, I shipped *two* variants — one for a tree with the teardown fix, one + without. That cannot work: `apply_overlay.sh` applies `patches/*.patch` **by glob**, so + one of two mutually-exclusive patches is guaranteed to fail. `patches/` is a series, + not a menu; an alternative belongs outside the glob, like `DIAGNOSTIC-*.patch.txt`. + The clean-tree variant was also strictly weaker — it closed the fd without dropping the + callbacks, leaving the GC cycle, and with it class B, in place. + +A test now reconstructs the pinned versions of every file the patches touch and applies the +whole series in glob order, exactly as the script does, so a patch that conflicts with its +predecessor fails in CI rather than on someone else's clone. + +`patches/0004-*.patch` closes the fd explicitly, at the point the code has *already proven* +the body is complete (the `user_finish_called` branch), deregistering the notifier before +closing so the event loop is never left polling a closed descriptor. It does **not** close +on `request_finished` alone — that truncates bodies (verified: blank pages), because +RequestServer finishes writing long before WebContent drains the pipe. + +Measured A/B, same binary, same 200-completed-request workload, only this function differing: + +| variant | sockets after 200 completed requests | peer DEAD | +|---|---|---| +| clean | 208 | 203 | +| `0004` applied | **6** | **0** | + +and body delivery is intact — `text=10 | stream=10/bytes=48000 | cancel=5` (plain fetch, +incremental `ReadableStream` reads, mid-body `cancel()`), plus a rendered 800x600 screenshot. +The generalisable lesson: **"the object is collectable" and "the fd is closed" are different +claims**, and for a descriptor received over IPC only the second one is the bug. A fix +verified through the object graph can pass while the resource still leaks. + +### Open: the leak persists on Ulf's machine after 0004, and my blind spot + +`0004` takes my measurements to 0 and his tree is still leaking. Two things I got wrong +about how I was measuring, both worth recording because they are the reason the loop +keeps failing: + +1. **Every census I requested was of WebContent.** That was my hypothesis, not a finding. + RequestServer creates the response pipes *and* the cache body files, and the UI + process and Compositor hold fds too — a leak in any of them is invisible to every + number I have collected so far. `fd_census.py --all` now censuses every + Ladybird-family process and ranks by growth rate, so the data names the process + instead of me naming it. +2. **My server never exercised the disk cache.** Ulf runs + `--http-disk-cache-mode enabled` against real sites; my test server sent no cache + headers at all, so `handle_read_cache_state` never ran in any A/B I did. That matters + because the large-cache-hit branch (`body_size >= PAGE_SIZE`) takes + `take_body_file()` → `send_transferred_body_file_to_client()`: it sends a **body + file** and never creates a response pipe, so it is a completed-request path that + `release_response_fd()` cannot reach. I built that workload (300 cache hits over + small/large/revalidated entries, disk cache on by default) and it stays flat at 5 + sockets here — so it is not sufficient on its own, but it is the first path found + that my fix structurally does not cover. + +What that means for the diagnosis: `0004` is verified to fix the response-pipe class +(208 → 5 sockets, 203 → 0 dead peers, bodies intact), and that class is real. It is +evidently not the whole of what Ulf is seeing, and the next measurement has to come from +his machine with `--all`, because I have now falsified every workload I can construct +locally — including the cache paths I had never tested. + +### Correction: it IS WebContent, and the instrument now reads its own build + +Ulf's next census settled the process question against me. His `--all` run (pid 19291, +4880 s, 4386 samples, flags +`--site-isolation=top-level --enable-http-memory-cache`): + +``` +fds: total=7514 socket:=7487 +sockets: 7487 peer DEAD=7479 peer ALIVE=6 unknown=2 +retained(>=30s)=7436, of retained: peer DEAD=7428 +growth over 4880s: sockets +7458 (+91.7/min), all peer DEAD +``` + +So the accumulation is in **WebContent**, it is **class A** (peer=DEAD = completed +requests), and 7428 of 7436 retained fds are *not* in flight. The `--all` detour above +was a wrong turn: censusing every process was the right instrument to build, and it +answered "WebContent", which is where I had been looking all along. + +But the number I could not interpret was the rate: **91.7/min against 97/min measured +before the fix**. That is equally consistent with two opposite conclusions — + +- `0004` does not address this leak, or +- `0004` was not in that binary. + +and they demand opposite next steps. My instinct was to *ask*. That would have been +another round trip, about a build that had already happened, answered from memory — +and it is the fourth time in this investigation that a number arrived without the +identity of the code that produced it. **A leak rate without its build provenance is +not a measurement anyone can reproduce, including me.** + +The answer was never in anyone's memory: it is in the binary, and the binary is still +mapped by the process being censused. `fd_census.py` now reads it: + +``` +$ python3 fd_census.py --build +build: HAS 0004 (release the response fd on completion) +build: HAS 0003 (tear down the request when the body is delivered) +``` + +and the same block is printed next to every verdict, because the verdict is only +interpretable together with it. Implementation: a pure-Python ELF reader over the +`.dynstr`/`.strtab` of the executable and every mapped `.so` (no binutils dependency +on someone else's machine), looking for `Requests::Request::release_response_fd` and +`::defer_teardown`. Statically linked builds — Ulf's is one — are covered by probing +the executable when no `lagom-requests` library is mapped. + +The load-bearing part is the **negative control**. `set_up_internal_stream_data` exists +in every build of that file, patched or not; if it is absent, the symbols were not +readable at all (stripped, LTO'd, or the code is somewhere we did not look) and the +probe reports *"cannot tell"* rather than *"fix absent"*. Without that control, a +stripped binary would read as an unpatched one, which would aim the next round of work +at exactly the wrong code — the same class of error as the two method mistakes above, +so it gets a guard rather than a caveat. + +Verified end-to-end against two genuinely different builds of the same library: with +`release_response_fd` renamed away and relinked, the probe reports `does NOT have 0004` +while the control stays present; restored and relinked, `HAS 0004`. A non-Ladybird +process reports "cannot tell". Both directions are tested, because a probe that can +only confirm a fix is present is useless for the question that prompted it. + +#### Correction: the probe was wrong, and Ulf was right + +Ulf's reply to the above was *"the tool says it's not, but I'm sure it was applied"* — +and then *"I have all the patches applied."* **He was right and the probe was wrong.** +The flaw is worth recording in full because it is the same class of error as the +measurements this whole document is about. + +Ladybird sets `ENABLE_LTO_FOR_RELEASE=ON` (`Meta/CMake/cmake_options.cmake:46`). In a +**static** build — Ulf's — LTO inlines a small internal-only method like +`release_response_fd()` into its single caller and leaves **no symbol and no string +behind at all**. Reproduced from first principles: a private method called only within +its TU, compiled `-O3 -flto` and linked, is absent from both `nm` and `strings` in the +linked executable. My own build kept it only because a **shared** library has to export +it. So the probe's answer depended on *how the binary was linked*, not on whether the +fix was in it — and the one build shape it had to get right was the one it got wrong. + +The negative control did not save it, and that is the instructive part. `set_up_internal_stream_data` +is vulnerable to the *same* optimisation: verified that a larger internal-only function +also vanishes under LTO. **A negative control only rules out "unreadable" if it cannot +disappear for the same reason as the thing it guards.** Mine failed in a different mode +than the one it was guarding against, so it certified readability it had not +established. I had written that the control was "the load-bearing part"; it was +load-bearing in the wrong direction. + +Two fixes, both verified: + +1. **`.debug_str` is read as well as `.dynstr`/`.strtab`.** Debug info names + inlined-away functions, and Ladybird's `RelWithDebInfo` compiles with `-g` (and + `-g1`, also verified sufficient), so the answer survives there for exactly the + LTO/static case. +2. **A fix is never reported MISSING unless a symbol that inlining cannot erase is + visible** (`UNINLINABLE_CONTROLS`: vtable/IPC-dispatched entry points such as + `request_started`, `headers_became_available`, and `Request`'s header-declared + out-of-line methods). If none is readable, the probe now says *"cannot tell whether + 0004 is present"* and names inlining as the reason. Tested both ways: an + Ulf-shaped blob (0004 inlined away, control present, nothing uninlinable) yields + "cannot tell"; a genuinely unpatched-but-readable binary is still called out as + missing 0004, re-verified end-to-end against a rebuilt shared library. + +**What this means for the diagnosis.** With `0004` confirmed applied, the ~92/min +class-A leak is *not* the response-pipe path `0004` closes. The fd has a **third +owner**, and the two candidates below are no longer "still open" — they are the +diagnosis. + +One more thing the probe caught on the way. Running the full census against a +*healthy* browser to check the new block, the verdict line read +`mixed DEAD/ALIVE -> both classes present` for 0 dead and 5 live sockets: both +classification branches required a 10x majority, so zero-of-a-class fell between them +and got reported as a class that had no members — while also reading the ordinary IPC +mesh as a leak. Fixed with explicit `dead == 0` / `alive == 0` cases. It only surfaced +because the healthy case finally got looked at; a verdict that is wrong on healthy +input will be believed when it is wrong on broken input too. + +### The remaining owner: two paths `0004` structurally cannot reach + +With `0004` confirmed present in the leaking binary, these are no longer speculative +alternatives — one of them is the leak. + +Reading the ownership chain for the class-A case again, with "what else holds a +`Requests::Request` after completion" as the question, turns up two paths that are +consistent with completed requests (peer=DEAD) retaining an fd: + +1. **`Response::m_request_server_request` holds the request by `RefPtr`** + (`Fetch/Infrastructure/HTTP/Responses.h:68`), and `clone()` *copies that struct* + (`Responses.cpp:209-210`) — so every clone of a response is another owner of the + same `Requests::Request`. `0004` closes the fd from inside the completion branch, so + it should still win for any request that reaches that branch; what it cannot cover + is a request that never does. +2. **Paused body delivery.** A document navigation starts with + `set_body_delivery_paused(true)` (`Fetching.cpp:2340`) and only + `resume_body_delivery()` re-enables the notifier. The completion branch in + `set_up_internal_stream_data` is driven by the read notifier and `request_done`; a + request whose delivery is paused and never resumed never reaches + `user_finish_called`, so `release_response_fd()` is never called on it — while + RequestServer, having finished writing, has already closed its end. **That produces + exactly the observed signature: peer=DEAD, retained, and unaffected by `0004`.** + `LocalNavigable` has ~8 separate resume/stop call sites for this + (`:488, :517, :564, :2271, :2299, :2310, :2328` plus + `stop_or_resume_response_body_delivery`), i.e. it is a + "every early return must remember to resume" contract — the shape that leaks on the + path nobody enumerated. Site isolation (`--site-isolation=top-level`, on in Ulf's + run and not in my earlier ones) adds cross-process navigation paths through exactly + this code. + +#### Falsified locally, and the measurement that replaces the guess + +I built the workload for (2): 330 top-level navigations to a URL whose body dribbles, +each abandoned 250 ms in (before the body completes), with `--site-isolation=top-level` +and the memory cache on. **Flat — 0 dead, no growth.** So (2) as I constructed it is not +sufficient either, and I am now two falsified hypotheses deep on a leak I cannot +reproduce. + +That is the point at which guessing again is the wrong move. The two candidates above +look identical in every column the census prints, but they differ in one field `ss` +already reports and I was throwing away: **`Recv-Q`, the bytes sitting unread in the +socket.** + +- `Recv-Q > 0` — the body was **never drained**. The consumer stopped reading, so the + completion branch that closes the fd was never reached. The fix is resume-or-close on + the abandoned path. +- `Recv-Q == 0` — the body **was fully read**; the descriptor is merely still *owned*. + The fix is dropping the surviving reference (the `RefPtr` in + `Response::m_request_server_request`, which `clone()` copies). + +Same signature, opposite fixes, one field. The census now prints +`of retained peer=DEAD: body UNREAD=N body drained=M` and names which mechanism the +numbers imply. Ulf's existing `--all` run already collects the `ss` line this comes +from, so this costs nothing beyond re-running the census he has run before. + +Of the two, (2) is the one that explains the *combination* Ulf measured — completed-and-closed by the peer, +retained, and indifferent to a fix that works locally — rather than only part of it. +The memory cache, which I suspected next because `--enable-http-memory-cache` was on +his command line and I had never tested it, is **ruled out** by reading +`HTTP::MemoryCache::Entry` (`Libraries/LibHTTP/Cache/MemoryCache.h:23`): it stores +status, headers and `Core::ImmutableBytes`, and never holds a `Requests::Request`, so +it cannot retain a descriptor. + +### Why it takes the whole browser down, not just a tab + +`EMFILE` in WebContent surfaces through `MUST()` on an encode +(`LibWebView/CompositorConnection.cpp:62`) — an abort, not a propagated error. WebContent +dies; the Compositor's `VERIFY(connection)` +(`Services/Compositor/ConnectionFromClient.cpp:68`) then aborts; and the UI process's +`MUST` in `initialize_client` follows. **One `EMFILE` kills three processes**, which is +also why the failure is unattributable after the fact: nothing in the log names the +resource that ran out. Two `/proc//fd` censuses a few minutes apart is what turned it +into a bug report. + +--- + +## Bug 2 (independent, smaller): a `MessagePort` dropped without `close()` leaks its socketpair + +Distinguishable by signature: this one leaks **2 pipes + 1 socket per port** (each +`TransportSocket` makes two `pipe2` pairs at `TransportSocket.cpp:165,170`), so a page doing +this shows a *pipe*-dominated census — not what Ulf saw. + +```html + +``` + +Result: WebContent reaches 1,628 `pipe:` + 409 `socket:` and dies with + +``` +UNEXPECTED ERROR: pipe2: Too many open files (errno=24) at Libraries/LibIPC/TransportSocket.cpp:165 +``` + +The same page with `c.port1.close(); c.port2.close()` stays flat at 18 pipes — so the leak +is scoped to ports dropped **without** `close()`, which is the ordinary case in real pages. + +`MessagePort::entangle_with` (`LibWeb/HTML/MessagePort.cpp:222`/`230` upstream) installs +read hooks capturing `GC::make_root(this)` and `GC::make_root(m_remote_port)` — again +strong explicit roots, stored in a callback owned by the transport owned by the port, so +the port roots itself. `close()` reaches `disentangle()` (`:472`), which is the only thing +that closes the transport; a port that is merely dropped never gets there. Same *shape* as +bug 1 (a `GC::Root` captured into something the rooted object owns), different site. + +--- + +## Method note + +Three of my earlier theories about Ulf's crash were wrong (lost fd acks; an +`AnonymousBuffer`/image-cache miss; leaked `TransportSocket`s), and one was wrong only +because my own census one-liner was broken — `sed 's/[0-9]*$//'` does not strip +`pipe:[123]`, and piping through `head` hid the answer. The census command that works: + +``` +ls -l /proc/$P/fd | awk '{print $NF}' | sed 's/\[[0-9]*\]//' | sort | uniq -c | sort -rn +``` + +Everything above is either a counted fd, a labelled GC root, or an A/B against a rebuilt +binary. + +--- + +## Resolved upstream: PR #11041 fixes all three classes, including the one we could not reach + +Ulf handed over `upload/11041.patch` — three commits by sideshowbarker, in review upstream. +All three **apply cleanly at our pin `71fb301a`** (`git apply --check`, RC 0). Read against +this document they close the investigation, and the mapping is worth stating precisely +because two of the three are things we had and one is the thing we had been unable to +build. + +| upstream | what it does | our state | +|---|---|---| +| 1/3 `LibRequests+LibWeb: Release response pipes when requests complete` | `defer_teardown()` at the user-finish callback | same call site as our `0001`, but **ordered differently — theirs is correct and ours has a latent bug**, see below | +| 2/3 `LibWeb: Release response pipes when fetches are canceled` | `abort()`/`terminate()` never told the network layer; both now release the request | a class we **never diagnosed** | +| 3/3 `LibWeb: Tear down a navigation parked for content sniffing` | a navigation with headers but not enough bytes to sniff has no document, so `Document::abort()` has no controller to stop and only the arrival callback releases the request — which never runs | **exactly the open lead** (`6a311f55`) | + +Four things I want on the record. + +**Patch 1 is not quite "the same fix as ours" — I nearly wrote that, and it is wrong in a +way that matters.** Both add `defer_teardown()` inside the same `if` in +`set_up_internal_stream_data`, but on opposite sides of the `user_on_finish(...)` call: + +```cpp +m_internal_stream_data->user_finish_called = true; +defer_teardown(); // upstream: BEFORE +user_on_finish(...); + // ours (0001): AFTER +``` + +Upstream's order is the safe one, and their commit message states the reason: *"the deferred +task also keeps the Request alive if the callback drops the last ref."* `defer_teardown()` +captures `NonnullRefPtr(*this)` **inside** the deferred lambda, so calling it *first* pins +the `Request` across `user_on_finish`. Ours runs after the callback returns — and +`user_on_finish` is the fetch completion path, which is exactly where the last reference can +go away (a `Response` holds its `Request` by `RefPtr` at `Responses.h:226`; drop the +response in the finish handler and the `Request` dies). Then our `defer_teardown()` runs on +a half-destroyed object, or is never reached at all. It has never fired for me, because on +my workloads something always outlived the callback — which is precisely the kind of latent +ordering bug that shows up as a rare crash on someone else's machine, not as a leak on mine. +That alone settles which version the overlay should carry: **theirs**. + +**Patch 3 is the hypothesis I had, and had failed to reproduce.** The open lead in this +document was "a navigation whose delivery is paused and never resumed never reaches +`user_finish_called`, so the completion path is never taken", and the reason it stayed open +is that I built two workloads for it (330 abandoned dribbling navigations with site +isolation + memory cache; 300 cache hits) and **both came out flat**. Upstream's reproducer +is more specific than either: an **iframe removed** while its response has headers but +fewer bytes than the sniff threshold, i.e. the leak needs the navigable *destroyed* while +parked, not merely the navigation abandoned. My workloads abandoned navigations without +destroying the navigable, so they exercised everything except the condition that matters. A +falsified workload was evidence about my workload, not about the hypothesis — and I came +close to treating it as the latter. + +**Patch 2 is a class my instrument could not have found.** `fd_census.py` ranks by growth +and splits `peer=DEAD`/`ALIVE`, which is what identified the completed-request class; but +`abort()`/`terminate()` leaking one fd per cycle looks identical in that view to any other +retained peer=DEAD socket. Nothing in the census points at the *cancel* path specifically, +and I never asked whether the two entry points that mark a fetch cancelled actually tell +the network layer. That is a reading-the-code finding, and no amount of my census data +would have produced it. + +**Our `0002` (`release_response_fd`) has no upstream counterpart, and that is a signal.** +Upstream fixes the leak by making sure the teardown *is reached* on each of the three paths; +our `0002` closes the fd defensively at a point where "EOF or every reported byte +delivered" is already proven, on the theory that some other reference to the `Request` +survives (`Response::m_request_server_request`, copied by `clone()`). If upstream's three +patches take Ulf's rate to zero, that theory is unnecessary — the fd was never being +retained by a surviving reference, it was that nothing had run the teardown at all. The +honest reading is that `0002` was a workaround for a missing call site, and it worked on my +box because my box only ever hit the class `0001` already covered. + +### What this changes in the overlay + +Our two patches become **pin artefacts** the moment #11041 lands, exactly like +`0001-libweb-bindings-deterministic-dictionary-order` did (todo `643ea99e`): they apply +only because our pin predates the fix. Both already carry `.effect-grep` files, so +`apply_overlay.sh --verify` accepts upstream's equivalent fix in place of our exact bytes — +which is the mechanism built for precisely this, and it means a repin past #11041 does not +break verification. The action on that repin is to **delete both patches**, drop them from +the README's patch table, and re-run the census to confirm the rate is zero with upstream's +version rather than ours. + +Not done here: #11041 is unmerged, so nothing is deleted yet. What I would *not* do is +carry our `0002` forward alongside upstream's three — two mechanisms closing the same fd, +one of them justified by a theory the other one falsifies, is how the next reader gets +misled. + +--- + +## Done: the overlay now carries upstream's three, because mine crashed his browser + +The section above was written while #11041 was still in review, and it ended with "not +done here: nothing is deleted yet". It is done now, and the trigger was not the merge. + +Ulf ran the Bazel-built browser with my two patches applied. It loaded pages, and the fd +census was flat — *"I also didn't see immediate socket leak, so that's good"* — and then +after a few minutes it died: + +``` +VERIFICATION FAILED: m_ptr at ./AK/OwnPtr.h:134 +#0 in AK::Function::CallableWrapper::call() +#1 in Core::Notifier::event(Core::Event&) +``` + +`{lambda()#2}` is the read notifier's `on_activation`. That is the frame which *calls* +`on_finish` — and my `release_response_fd()`, invoked from inside the completion branch of +`on_finish`, set `m_internal_stream_data->read_stream = nullptr` while `on_activation` was +still on the stack and still about to dereference it: + +```cpp + } while (true); + + if (m_internal_stream_data->read_stream->is_eof()) // Request.cpp:376 + m_internal_stream_data->read_notifier->close(); +``` + +`AK::OwnPtr::operator->` is `VERIFY(m_ptr)` (`AK/OwnPtr.h:134`), so the null became a +trap and then `SIGILL`. **A use-after-null one stack frame up from the code I changed.** + +Three things worth keeping from this. + +**My reasoning about the fix was upside down.** I argued `0002` was needed because +"collectable" and "closed" are different claims, and for an fd received over IPC only the +second is the bug. That sentence is still true, and it was the wrong frame: the question +is not *when is the fd closed* but *who owns it while the stack is still unwound*. The +completion branch runs **inside** the read loop's callback; anything it destroys, the +caller may still touch. Upstream's patch does not close the fd at all — it makes the +existing deferred teardown *reachable* on each of the three paths where it was missed, and +the teardown is deferred precisely so it cannot destroy state a live frame is using. The +mechanism I reached for (destroy it now, synchronously, at the point I can prove the body +is done) was in direct conflict with the one the code already had. + +**The ordering bug I had already spotted was the same bug, and I under-rated it.** I noted +that upstream calls `defer_teardown()` *before* `user_on_finish` while my `0001` called it +after, wrote that it was "the kind of latent ordering bug that shows up as a rare crash on +someone else's machine", and then carried the patch anyway because it had never fired for +me. It fired for him. "Never observed here" is a statement about my workloads. + +**One prediction verified, one falsified.** I predicted that if upstream's patches take the +rate to zero, `0002`'s surviving-reference theory was unnecessary — that holds, and it is +why `0002` is deleted rather than rebased. But I also wrote that the honest reading was +"`0002` was a workaround for a missing call site, and it worked on my box because my box +only ever hit the class `0001` already covered." That was too kind: it did not merely fail +to help on his box, it *broke* his box, and the reason was visible in the source the whole +time without any measurement at all. + +The overlay's `patches/` is now upstream's three commits, annotated in place with their +provenance, each with an `.effect-grep` so `--verify` recognises the merged fix on a tree +newer than our pin. `Internals.idl` gains `openResponsePipeCount()` in patch 1, which +regenerates the LibWeb bindings — the overlay's codegen picks that up correctly (verified: +the IDL change invalidates and rebuilds the binding set). diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 29dbac5..bba09d7 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -2,29 +2,368 @@ The Bazel workspace overlay produced by the any2bazel parity loop for [Ladybird](https://ladybird.org) (CMake + vcpkg, C++23). The narrative — why -each of these files looks the way it does, and the 34 findings the migration +each of these files looks the way it does, and the 36 findings the migration produced — is in [`docs/CASE-ladybird-migration.md`](../../docs/CASE-ladybird-migration.md). This directory is the *artifact*: what you would drop into a Ladybird checkout. -**What it achieves today:** all five processes (`ladybird` UI, WebContent, -Compositor, RequestServer, ImageDecoder) are Bazel-built, all 51 code +**What it achieves today:** all six processes (`ladybird` UI, WebContent, +WebWorker, Compositor, RequestServer, ImageDecoder) are Bazel-built, all 51 code generators run under Bazel with output **byte-identical to CMake's** (1,408/1,408 files, checked by `Meta/bazel_parity_harness.py`, which accounts for every one of the build's 586 ninja `CUSTOM_COMMAND`s — 51 covered, 535 excluded with a stated reason, **0 unhandled**), the result renders pages (`--headless=text` and `--headless=layout-tree` match the CMake reference byte for byte), **the 77 vcpkg dependencies are fetched and built by Bazel with -zero network access** (findings 30–33), and **so are the 10 Rust crates and -`flapc`** — 154 crates.io crates fetched from `Cargo.lock`, built by -network-blocked cargo actions (finding 34). - -**`git clone && bazel build //:ladybird` now builds the browser.** Verified by -removal, twice: with `Build/full/vcpkg_installed` gone, and then with -`Build/full/cargo` **and** `Build/full/bin/flapc` gone, a `bazel clean` build of -all six binaries renders `--headless=text` and `--headless=layout-tree` -byte-identically to the CMake reference. `Build/full` is still needed to -*regenerate* the BUILD files and to run the parity harness — a -converter-development dependency, not a build dependency. See +zero network access** (findings 30–33), and **so are the 8 Rust crates and the 4 +cargo binaries** — 155 crates.io crates fetched from `Cargo.lock`, +built by network-blocked cargo actions (findings 34–35). + +**Nothing in the emitted build reads CMake's build tree (`Build/full`).** Verified +by removal: with `Build/full/{Libraries,Services,UI,bin,cargo}` moved off the +machine — every Ladybird-generated header and every binary CMake produced — all six +binaries build from scratch and render `--headless=text` **and** +`--headless=layout-tree` byte-identically to the CMake reference, and `cquery` over +the closure of all six returns **0** targets under `Build/full` (down from 741). +`Build/full` is still needed to *regenerate* the BUILD files and to run the parity +harness — a converter-development dependency, not a build dependency. + +**A fresh clone now builds and renders — after one ordinary CMake build has run +once.** (Two inputs come from that build rather than from Bazel; a Ladybird +developer therefore stages nothing, and the gap is real only for a Bazel-only clone +— see [how a Ladybird developer gets them](#getting-the-two-remaining-inputs--with-and-without-cmake). +The third, the HSTS table, is now pinned and fetched by Bazel.) The +end-to-end test (clone into a new directory, drop in the overlay, nothing else) +was run for the first time and failed **six separate times**; `Build/full` was +the dependency I had removed, and **five of the six had nothing to do with it**. +All six are addressed below, and the result is verified on the clone rather than +asserted: `//:vcpkg_installed` builds all 76 ports offline, all six binaries +build (2,842 actions, RC=0), and `--headless=text`/`--headless=layout-tree` +are **byte-identical to the CMake reference on all three test pages**. Rows 1–2 +are still manual staging (a prefetch, `Meta/ladybird.py vcpkg`); row 3 is now +**fixed** — Bazel fetches the HSTS table from a pinned commit; #4 and #5 were +bugs and are fixed; #6 now fails loudly instead of 20 minutes later: + +| # | What is missing on a fresh clone | Why the dev machine hid it | +|---|---|---| +| 1 | **`Build/vcpkg`** — a git clone of microsoft/vcpkg at `vcpkg.json`'s `builtin-baseline`, which `//Build/vcpkg:tree` globs with `allow_empty = True`. Matches *one* file (its own `BUILD.bazel`) and the build fails with `/tmp/.../root/vcpkg: No such file or directory`. Ladybird's own `Meta/ladybird.py vcpkg` creates it; the recipe above never says so | `Meta/ladybird.py vcpkg` had been run months earlier | +| 2 | **`.git` inside that checkout** — vcpkg resolves versioned ports with `git read-tree`, so it is load-bearing (120 MB), and the filegroup *excludes* it. The action is `no-sandbox`, so it reads the real path and got it anyway: an **undeclared input the build needs** | `no-sandbox` + a real checkout on disk | +| 3 | **`Build/caches/HSTSPreload/transport_security_state_static.json`** (10 MB) — an *unversioned, unpinned* network download CMake does at configure time (`Meta/CMake/hsts_preload.cmake` fetches Chromium's `main`). `codegen_root.bzl` named it as a genrule `srcs`, so Bazel failed cleanly with `missing input file` — but nothing produced it. **Fixed:** pinned downstream to a Chromium commit + sha256 ([`hsts_preload.bzl`](workspace/hsts_preload.bzl)), fetched with `http_file`, and the generated table is byte-identical to CMake's — verified with the staged file *deleted* | the CMake configure had already downloaded it | +| 4 | **Two path bugs in `Meta/vcpkg_build.sh`** — the distfile index and its entries were passed **execroot-relative**, and vcpkg invokes the asset-cache script from its own cwd, so `awk`/`cp` looked in the wrong directory. Every asset lookup failed, reported as `no asset cache hits`, and `x-block-origin` then correctly refused the network. Fixed here (absolutize both) | the vcpkg checkout already had `downloads/tools/cmake-4.4.0-linux` from an earlier run, so vcpkg never *asked* the script for a tool | +| 5 | **The `angle` port runs `pip install ply`** (`x_vcpkg_get_python_packages`), which is **not** an asset-cache download and therefore not covered by the pin. A real hole in the "zero network access" claim: `vcpkg_tree` set `requires-network: "0"`, but that is a *scheduling hint*, and with `no-sandbox: "1"` **nothing enforced it** — with `use_default_shell_env = True` the action inherited `HTTP_PROXY`/`HTTPS_PROXY` and pip reached PyPI. **Fixed here:** the wheel is pinned by URL+sha256 (`vcpkg_python_packages.bzl`), staged into a find-links dir, and pip runs with `PIP_NO_INDEX` and the proxy variables unset | this sandbox exports a proxy, so pip silently succeeded through it | +| 6 | **The four `vcpkg_from_git` archives were pre-placed from a directory I had made by hand.** `vcpkg_git_archives.bzl` is generated, committed, listed in the table below — and **loaded by nothing**. The staging was `if [ -d ... ]; then cp ... 2>/dev/null \|\| true; fi`: three ways to succeed while copying nothing. Without them skia fails ~20 min in with `git fetch https://android.googlesource.com/.../piex.git … Error code: 128`, naming neither the directory nor the tarball. **Now a hard error in 4 seconds** naming both; reproducing them is a prefetch, not a rule — they are `git archive` output, so there is no URL to `http_file`, but cloning the pinned URL and `git archive`-ing the pinned ref reproduces the committed SHA512 exactly (verified on `libyuv`) | I created the directory by hand while building the pin, months before | + +Rows 1–3 are inputs the recipe must produce or document (step 1a). Rows 4–6 were +bugs, and 5 is the substantive one: **"zero network access" was measured with +`x-block-origin`, which only governs vcpkg's own downloader.** A portfile that +shells out to pip, git or curl bypasses it entirely, and nothing was enforcing the +absence of a network — `requires-network: "0"` is a hint, and `no-sandbox: "1"` +means there is no namespace to enforce. `ply` is now pinned like any other +dependency (URL + sha256 → `http_file` → find-links dir → `PIP_NO_INDEX`), which +makes an unpinned package a hard error rather than a download. The four +`vcpkg_from_git` archives are still staged from a directory rather than produced by +the recipe — they are `git archive` output with no URL, so reproducing them means +cloning each pinned URL and `git archive`-ing the pinned ref (8 lines of shell, and +the committed SHA512s check it). + +The lesson is the same one as finding 35, one layer out: **`Build/full` was the +dependency I went looking for, so it is the one I found.** The check that would +have caught all five is not a better `cquery` — it is doing the clone. See +[Known gaps](#known-gaps). + +### Getting the two remaining inputs — with and without CMake + +Two audiences, two answers. **A Ladybird developer stages nothing:** one ordinary +`./Meta/ladybird.py build` produces both as side effects, which is exactly why they +stayed invisible here for months. + +| Input | Who produces it in a normal build | +|---|---| +| the vcpkg checkout + its `.git` | `Meta/Utils/build_vcpkg.py`, called by `ladybird.py` **`build`** as well as `vcpkg`: clone, checkout `builtin-baseline`, bootstrap the tool at the tag+SHA512 in `scripts/vcpkg-tool-metadata.txt`. ~70 s | +| the four `git archive` tarballs | **vcpkg itself**, while building skia and angle | +| ~~the HSTS preload table~~ | **Bazel**, now: `@hsts_preload_json//file`, pinned in `hsts_preload.bzl` | + +**`apply_overlay.sh` runs both of these for you** — it did not always, and that +gap cost a 20-minute build: the script ran step 1 and *printed* step 2 in its closing +message, so the obvious next command was `bazel build`, which failed inside the vcpkg +action with `no git-sourced externals at ./Meta/CMake/vcpkg/git-archives`. A setup +script that stops one required step short of a working build has not set anything up, +and a closing message is not a substitute for doing the work. `--verify` now checks the +four tarballs by name against the committed pin too, since their absence is a +guaranteed build failure and finding that without a build is exactly what `--verify` +is for. + +**By hand, without ever running CMake — and it needs no CMake at all:** + +```sh +python3 Meta/ladybird.py vcpkg # 1. the checkout + .git (~70s). No CMake: + # `vcpkg` is a standalone subcommand. +python3 Meta/fetch_vcpkg_git_archives.py # 2. the four git archives (~80s), each + # reproduced with git clone + git archive + # and VERIFIED against the committed SHA512 + # (the HSTS table needs no step: Bazel + # fetches it from a pinned commit) +bazel build //:ladybird # the 5 services come with it (they are its `data`) +``` + +Step 2 is `Meta/fetch_vcpkg_git_archives.py`, added here. Verified: **4/4 reproduced +from scratch and byte-identical to the pinned SHA512s.** Two things about it are worth +knowing, because both were mistakes I made first: + +- **It takes the *list* from the committed pin, never from parsing portfiles.** A + first version derived the list by scanning skia's and angle's portfiles and was + wrong in both directions: 8 archives for skia where 4 are real, and libyuv missed + entirely. `declare_external_from_git` only *declares*; feature- and + platform-conditional `get_externals(${required_externals})` decides what is + actually fetched, and libyuv's comes from its own port. **Statically deciding the + set is unsound; statically resolving a name to a URL is fine**, and that is all the + script does. +- **Regenerating the pin uses vcpkg as the instrument**, not a parser: + `Meta/vcpkg_capture_git_archives.sh` runs `vcpkg install --only-downloads`, which + executes the portfiles' fetch phase and stops — ~6 min, no compilation, no CMake. + It produces **3 of the 4**: angle's zlib is fetched from angle's *build* phase, so + it never appears in a downloads-only run. That asymmetry is recorded in the script + rather than smoothed over. + +### Reproducing the tree on another machine + +The overlay is **not a fork**: it is a pinned upstream Ladybird commit + two patches ++ 45 Bazel files that sit alongside CMake's. [`apply_overlay.sh`](apply_overlay.sh) is +that sentence made executable, because *"copy `workspace/` over a clone"* has four +ways to be silently wrong — and every one of them was found by running it, not by +reading it: + +1. **The Ladybird commit.** The generated BUILD files name ~1,961 LibWeb compile + inputs and 665 IDL bindings *by path*, and were generated from one tree. + **Nothing in this repo recorded which one** until the script did + (`71fb301a`). Against a different tree the build fails on a moved file — or + worse, silently omits a new one. +2. **The patches**, which have to be applied or the browser leaks one socket fd per + completed HTTP request until it dies of `EMFILE` overnight. These are now + **upstream's own three commits from Ladybird PR #11041**, carried only because our + pin predates the merge + ([`0001`](patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch), + [`0002`](patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch), + [`0003`](patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch); + see [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md)). + They **replace two patches written here**, one of which crashed Ulf's browser — + see "Why upstream's patches replaced mine" below. There used to be four others: the + `PYTHONHASHSEED` determinism fix and the non-self-contained `UI/Qt/TabBar.h` are + both **fixed upstream** at this pin, so the repin from `f9e34731` deleted them. A + patch directory that only ever grows is a patch directory nobody re-checks against + upstream. +3. **The rename**: `bazelrc.txt` → `.bazelrc`. Stored under a different name so a + `cp -r` cannot be mistaken for a working build — and a rename a human does by + hand is a rename a human forgets. +4. **The order**, which is the one I would never have predicted and which the + `cp -r` recipe above gets wrong. `Build/vcpkg/BUILD.bazel` makes the *directory* + `Build/vcpkg` exist, and upstream's `Meta/Utils/build_vcpkg.py` treats "the + directory is there" as "the checkout is there": it skips the clone and runs + `git -C Build/vcpkg rev-parse HEAD`, which — there being no `.git` inside — + **walks up to Ladybird's own repo** and returns *Ladybird's* HEAD. It then tries + to check vcpkg's baseline out of the Ladybird repo: + `fatal: unable to read tree (40f3c709…)`. So that one file must be staged + **after** the prefetch; the script defers it and says so. + +`--verify` checks a tree without changing it, and checks the things a file copy +cannot: that HEAD is the pinned commit, that all 45 files are byte-identical, that +the patches are **applied** (`git apply --check -R` succeeding is the proof — a patch +that reverse-applies cleanly is already in the tree), and that the `.sh` files kept +their **executable bit**, which is tree state a careless copy drops and which then +fails deep inside a build action rather than at setup. + +Verified end to end: `apply_overlay.sh /tmp/lbfresh2` on an empty directory produced +a tree byte-identical to the one that renders (`diff -rq`, excluding prefetch +outputs), `Meta/fetch_vcpkg_git_archives.py` reproduced **4/4** archives verified +against the pinned SHA512s, and `bazel build` then built the 76 vcpkg ports offline +and linked `//:LibHTTP`. + +#### One thing the overlay cannot carry: host tools + +```sh +sudo apt install nasm autoconf automake libtool autoconf-archive libltdl-dev +# ...and, since the 71fb301a repin, four host packages upstream newly REQUIRES: +sudo apt install qt6-positioning-dev qt6-base-private-dev libxkbcommon-dev libglib2.0-dev +``` + +The second line is what a repin costs, and every one of the four was found by a +*configure or generate failure*, one at a time, because nothing derives this set: +`UI/Qt/CMakeLists.txt` turned `Positioning` from `OPTIONAL_COMPONENTS` into +`REQUIRED`, made `GuiPrivate` required on Linux (not just Apple/DirectX), and added +`pkg_check_modules(GIO REQUIRED gio-2.0 gio-unix-2.0)` for the new +`ExternalURLActivationToken`/`ExternalURLHandler` sources. `qt6-base-private-dev` and +`libxkbcommon-dev` are *transitive*: `Qt6GuiPrivate` reports itself NOT FOUND until +XKB is present, then names an `INTERFACE_INCLUDE_DIRECTORIES` path +(`/usr/include/.../QtGui/6.10.2`) that only the private-dev package ships — a +two-step failure where neither message mentions the package to install. That is the +finding-39 shape again, one layer out: the preflight covers *vcpkg's* host tools, so +Ladybird's own `find_package`/`pkg_check_modules` requirements are unchecked and cost +one failed configure each. + +**The Qt modules are now preflighted**, because their failure mode was the worst of +the set. `rules_qt`'s `qt.local_repo` *derives* its `cc_library` targets by listing +the host Qt's lib directory, so a module you do not have is never declared and Bazel +says: + +``` +ERROR: .../external/rules_qt++qt+qt/BUILD.bazel: no such target +'@@rules_qt++qt+qt//:QtPositioning': target 'QtPositioning' not declared in +package '' ... and referenced by '//:ladybird' +``` + +which mentions neither Qt nor apt, and points at a generated file in an output base +that you did not write. `qt_runtime.bzl` now checks the modules `//:ladybird` links +against that same lib directory — the input `qt.local_repo` derives from — and fails +with the package name instead: + +``` +qt_plugins: the Qt at /usr (version 6.10.2) is missing 1 module(s) + that //:ladybird links: + + QtPositioning (package: qt6-positioning-dev) + + Install them and re-run; on Debian/Ubuntu: + + sudo apt install qt6-positioning-dev + (If you installed it just now, Bazel may have the old @qt cached: + `bazel sync --configure` or `bazel clean --expunge` re-runs the probe.) +``` + +This sits next to the Qt *version* floor check, which was the right idea at the wrong +scope: it asked "is the SDK new enough" and never "does the SDK have the parts we +link". A test asserts the module list agrees with the `@qt//:Qt*` deps in +`BUILD.bazel`, so a module added by a future repin cannot silently skip its +preflight — the same drift that made `Positioning` a bug in the first place. + +**The advice depends on which Qt you have**, and getting that wrong is worse than +saying nothing. For a distro Qt the message says `apt install`; for a +**self-contained SDK** (aqt, a venv, the official installer) it says the opposite, +because `apt` installs into `/usr/lib/...` where that SDK never looks — you would +install the package, get the identical error, and reasonably conclude the message was +wrong. So the check reports the prefix and lib directory it probed, then names the +`aqt --modules` form instead. + +### Your Qt SDK path is yours + +`MODULE.bazel`'s `qt.local_repo(paths = ...)` is the one line in the overlay that is a +fact about **your machine** rather than about Ladybird at the pin — and the copy phase +used to overwrite it. If you build against Qt 6.9.2 in a venv while your system Qt is +6.4.2, that re-apply did not merely change a setting: it moved you *below* Ladybird's +6.9 floor, turning a working tree into a failing one, with the failure surfacing later +and elsewhere. Now it is resolved rather than imposed (first wins, and the script says +which rule won): + +1. `--qt-prefix DIR` +2. the `paths` line already in your `MODULE.bazel` — **so a re-apply preserves it** +3. the `qmake` first on your `PATH` (activating a venv is how you say which Qt you mean) +4. `/usr/lib/qt6`, the historical default + +`--verify` treats that line as expected-to-differ for the same reason, and if the +chosen Qt is readable the script prints its version and warns immediately when it is +below 6.9. + +Not a courtesy list — the set is derived from vcpkg's own scripts into +[`Meta/vcpkg_host_tools.tsv`](workspace/Meta/vcpkg_host_tools.tsv), and +`vcpkg_build.sh` checks all of it before building anything, printing the ports that +need each missing tool and this exact line. vcpkg has **no Linux download** for +these (its `nasm` URLs are inside `if(CMAKE_HOST_WIN32)`; `vcpkg-make` demands +autotools via `find_program` + `FATAL_ERROR`), so they cannot be pinned the way the +76 distfiles are — the gap is named rather than papered over. Before the check +existed, each one cost a full ~20-minute build to discover, and the error named the +wrong place: `nasm` surfaced from inside `libvpx`, autotools from `gperf` via a +helper port that neither mentions (finding 39). + +`glslangValidator` is the same class and still **not** covered: two genrules name +`/usr/bin/glslangValidator`, and `vcpkg_installed` does not ship it — `apt install +glslang-tools` for now (see [known gaps](#known-gaps)). + +### The HSTS table: pinned downstream, because upstream is not ours to fix + +`Meta/CMake/hsts_preload.cmake` downloads Chromium's +`net/http/transport_security_state_static.json` from **`main`** at configure time. +The generator turns it into a ~95,000-entry `constexpr Array` of domains LibHTTP +forces to HTTPS, so *the day you configured* decides a security-relevant table. +The right fix is one line in that `.cmake` file; we do not control it, so the +overlay pins it **downstream** and the upstream fetch is a bug report: + +- [`hsts_preload.bzl`](workspace/hsts_preload.bzl) — a module extension declaring + one `http_file` at an immutable commit URL with a `sha256`. `MODULE.bazel` names + it; `codegen_root.bzl`'s `gen_HSTSPreloadData` takes `@hsts_preload_json//file` + as `srcs` instead of a path under `Build/caches`. +- [`Meta/pin_hsts_preload.py`](workspace/Meta/pin_hsts_preload.py) — regenerates + that file: resolves the newest commit touching the path, downloads it, and writes + **the hash it measured**. `--expect-same-as ` refuses to write unless the + pinned bytes equal a file you already have, which is how this pin was shown not + to move the generated table. + +**Verified, on the fresh clone, with the CMake-downloaded file deleted** — so the +pinned fetch is the only possible source: `bazel build //:gen_HSTSPreloadData` +produces both outputs **byte-identical to the CMake reference** +(`HSTSPreloadData.cpp`, 4,873,678 bytes), and `bazel build //:LibHTTP` compiles and +links them (RC=0, 183 actions). + +Two things make this pin honest rather than convenient, and both are measurements: + +- **Pin a commit, not a release tag.** A tag is a pin to a *different table*: at + `139.0.7258.5` the file is 18.7 MB and generates **168,593** entries against this + commit's **94,626**. Pinning a tag would have traded a hermeticity gap for a + parity gap. The commit `main` pointed at when the reference build configured + serves bytes identical to what CMake downloaded — checked with `cmp`. +- **Unpinned would have been silently wrong.** `http_file` *does* work with no + `sha256` on the `main` URL (it builds, and Bazel prints the integrity it would + have used) — but then Bazel caches the first fetch forever: with a local origin, + changing the file upstream and rebuilding returned the **old** content in 0.3 s + with no warning. And the table moved during this work — `service.gov.scot` left + the list, 94,627 → 94,626 entries — so an unpinned fetch would have broken + byte-parity for a reason unrelated to the migration. + +The cost of pinning downstream only: **CMake still tracks `main`**, so a configure +newer than the pin disagrees with Bazel. That is now one pinned input versus one +unpinned one — a visible, dated disagreement with a commit sha to look at — instead +of two unpinned fetches that happened to agree. And it is closeable without touching +upstream: `download_file` is a no-op when the file already exists (verified with +`ENABLE_NETWORK_DOWNLOADS=OFF`), so copying Bazel's fetched file into +`Build/caches/HSTSPreload/` before configuring makes CMake consume the same pin. + +### Three shortcuts tested and rejected, all of which look like simplifications + +- **`--depth 1` on the vcpkg clone.** 8.7 MB instead of 121 MB, and `read-tree` + even succeeds for some ports — then resolution fails on ffmpeg and harfbuzz with + vcpkg's own `Try again with a full vcpkg clone`. The pinned versions' port trees + live in **history**; that is what a version database *is*. +- **Dropping `builtin-baseline`** so no `.git` is needed at all. It resolves happily + against the checked-out `ports/` — and **silently moves 10 dependency versions** + (ffmpeg 7.1.1#5 → 8.1.2#3, harfbuzz 10.2.0 → 14.2.1#2, mimalloc 2.2.7 → 3.4.3, + plus zlib, freetype, dbus, fontconfig, libedit, libwebp, cpptrace). A fix for a + hermeticity blocker that changes ten dependency versions is not a fix. +- **A git submodule** — git's own answer to this, and it *does* work mechanically + (a submodule's `.git` is a gitfile, and vcpkg's `read-tree` follows the + indirection: verified). It fails for a better reason: **a submodule pins a + checkout, and vcpkg pins history behind a baseline.** 14 of `vcpkg.json`'s 45 + `overrides` name a version that is *not* what `ports/` holds at the baseline + (ffmpeg's pinned port is git-tree `0988005f…`; `HEAD:ports/ffmpeg` is + `c40aaa40…`), so the bytes vcpkg builds exist at no single commit. A submodule + would deliver precisely the insufficient state, still need the full 119 MB of + history in `.git/modules`, sit in a `Build*/`-ignored directory vcpkg fills with + ~3 GB of scratch (permanently-dirty submodule), and still not produce the `vcpkg` + binary, which is bootstrapped from a tag+SHA512 in `scripts/vcpkg-tool-metadata.txt`. + **`Build/vcpkg` is not a vendored dependency; it is a package manager's cache that + happens to be a git checkout.** Finding 36. + +A fresh full clone at the baseline resolves all **78 ports to exactly the versions +this dev checkout resolves** (`diff`, 0 differences) — so the checkout carries no +local state beyond the ref, which is what makes the prefetch step sufficient. + +**This claim was false until recently, and the way it was false is the most +useful thing in this directory.** The README said it worked; it did not. Every +binary depended on **741 targets under `Build/full`**, the overlay shipped four +`BUILD.bazel` shims and *zero* headers, and `glob(..., allow_empty = True)` meant +a fresh clone got **no error at all** — just `fatal error: LibXML/Export.h: No +such file or directory` some 1,600 actions in. A glob over a foreign tree cannot +fail, so nothing ever said the tree was missing. Of the 741, **666 were merely +*shadowing* Bazel's own outputs** (the LibWeb bindings headers — Bazel generates +all 692, and the shim silently won or lost on include order); 31 were real gaps; +and closing them exposed **two genuine bugs the CMake tree had been masking** — +an FFI header collision and an entire missing Rust target. See [Known gaps](#known-gaps) for what is still owed. ## Layout @@ -37,20 +376,31 @@ converter-development dependency, not a build dependency. See | `Meta/vcpkg_assets.tsv` | **The dependency pin.** 76 `(url, sha512, dst)` rows captured from vcpkg itself via its `x-script` asset hook. Everything below is generated from this, with no vcpkg, no CMake and no network | | `vcpkg_distfiles.bzl`, `vcpkg_index.bzl`, `vcpkg_extension.bzl`, `vcpkg_git_archives.bzl` | One `http_file` per distfile, the sha512→label index the asset script resolves through, the module extension that creates the repos, and the 4 `vcpkg_from_git` archives. **Generated** by `Meta/emit_vcpkg_bazel.py` | | `vcpkg.bzl`, `Meta/vcpkg_build.sh` | `vcpkg_tree`: builds the whole dep tree as an ordinary Bazel action with `x-block-origin`, so it reaches the network zero times. Deliberately not a `repository_rule` | -| `Meta/vcpkg_capture_assets.sh` | Records the pin, by *being* vcpkg's asset cache. The one run allowed to fetch | +| `Meta/vcpkg_tool_assets.tsv` | vcpkg's OWN host tools (cmake, ninja) — url + sha512 + the filename vcpkg looks for. **Separate from the asset capture on purpose:** `vcpkg_find_acquire_program` probes the host first, so a tool the capturing machine already had is never downloaded and never captured (that is how ninja went unpinned; finding 38). Regenerate with `emit_vcpkg_bazel.py --capture-tools` | +| `Meta/vcpkg_host_tools.tsv` | The tools that must come **from the host**, because vcpkg has no Linux download for them at all — `nasm` (6 ports) and the autotools set. The third class of input, and the one that cannot be pinned: `vcpkg_find_acquire_program(NASM)` has URLs only inside `if(CMAKE_HOST_WIN32)`, and `vcpkg-make` demands `autoconf`/`automake`/`libtool` via bare `find_program` + `FATAL_ERROR`. So this file does not close the gap, it **names** it — and `vcpkg_build.sh` checks the whole list before building anything, so a machine missing three tools is told all three in one second instead of one per 20-minute build (finding 39). Regenerate with `emit_vcpkg_bazel.py --host-tools` | +| `Meta/vcpkg_capture_assets.sh` | Records the 76-distfile pin, by *being* vcpkg's asset cache. The one run allowed to fetch. Does a **full build** (~50 min) and passes `--binarysource=clear`, because only a port that actually runs its portfile requests its downloads: a Download-Mode halt, a cache hit, an already-installed port and a warm `downloads/` each lose rows *while vcpkg exits 0* (see Known gaps 13). The script judges its own completeness — vcpkg's exit code cannot | +| `Meta/fetch_vcpkg_git_archives.py` | Produces the 4 `vcpkg_from_git` tarballs **without CMake**: takes the list from the committed pin, resolves each clone URL out of the portfiles, then `git clone` + `git -c core.autocrlf=false archive ` and **verifies against the pinned SHA512**. 4/4 byte-identical | +| `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | +| `hsts_preload.bzl` | Chromium's HSTS preload table as one `http_file`, pinned to a **commit** + sha256 — the downstream pin for the one input upstream CMake fetches from `main`. **Generated** by `Meta/pin_hsts_preload.py` | +| `Meta/pin_hsts_preload.py` | Re-pins it: resolves the newest commit touching the path, downloads it, writes the hash it **measured**; `--expect-same-as` refuses to write unless the pinned bytes equal the file CMake downloaded (parity guard) | +| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the two patches, copy the 45 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | | `bazelrc.txt` | → `.bazelrc`. Global copts/defines/linkopts mirrored from `Meta/CMake/compile_options.cmake` | +| `qt_runtime.bzl` | Qt's **runtime** half: a repo rule that stages the plugins of the SDK `@qt` itself names (read out of @qt's generated `qtconf.bzl`, so plugins and libraries cannot come from different Qts), the private libraries an SDK bundles beside Qt (derived from DT_NEEDED), a generated `qt.conf` pointing the binary at them, and the Qt >= 6.9 floor `UI/Qt/CMakeLists.txt` declares. Without it Qt `dlopen`ed the HOST's plugin into Bazel's Qt — finding 40 | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | | `codegen_root.bzl` | Non-LibWeb generator genrules (IPC endpoints, LibJS Bytecode/Op, HSTS table, WebGL replayer, TIFF tag tables, the two SPIR-V shader headers, and the chained `generate_interpreter_layout` → `flapc` interpreter assembly). **Generated** by `Meta/emit_root_codegen_bazel.py` | -| `Libraries/LibWeb/BUILD.bazel`, `generated_srcs.bzl` | LibWeb (~1,961 compile inputs). **Generated** by `Meta/emit_libweb_bazel.py` | -| `Libraries/LibWeb/codegen.bzl` | LibWeb's 26 generator genrules + the bindings mega-genrule (661 `.idl` → 1,331 files). **Generated** by `Meta/emit_codegen_bazel.py` | +| `Libraries/LibWeb/BUILD.bazel` | LibWeb (~1,961 compile inputs). **Generated** by `Meta/emit_libweb_bazel.py` | +| `Libraries/LibWeb/generated_srcs.bzl` | Which of those inputs come from codegen: 692 `.cpp` (the reference build's compile list) + 693 headers (`codegen.bzl`'s `outs`). **Generated** by `Meta/emit_libweb_bazel.py --generated-srcs` — it claimed to be for a long time before it was (Known gaps 9) | +| `Libraries/LibWeb/codegen.bzl` | LibWeb's 26 generator genrules + the bindings mega-genrule (663 `.idl` → 1,340 files). **Generated** by `Meta/emit_codegen_bazel.py` | | `Meta/emit_*.py` | The emitters. They read CMake's `build.ninja` + the File API codemodel and write the four generated files above | | `Meta/bazel_parity_harness.py` | Buckets every ninja `CUSTOM_COMMAND` as covered / excluded-with-reason / unhandled, re-runs the covered ones and byte-compares against CMake's tree. Non-zero unhandled is a failure | | `Meta/BUILD.bazel` | `//Meta:generators` filegroup (the generator scripts, as genrule inputs) | | `Meta/vcpkg/BUILD.bazel` | The 41 `vcpkg_lib` targets Ladybird's libraries depend on, backed by `//:vcpkg_installed`. Hand-written and stable (one target per port); this is the whole interface between Ladybird and its dependencies | -| `cargo_crates.bzl`, `cargo_index.bzl`, `cargo_extension.bzl` | One `http_archive` per crates.io crate (154), the name/version/sha256 index the vendor staging resolves through, and the module extension that creates the repos + the 3 pinned Rust toolchain components. **Generated** by `Meta/emit_cargo_bazel.py` **from `Cargo.lock` alone** — no cargo, no network, no CMake | +| `cargo_crates.bzl`, `cargo_index.bzl`, `cargo_extension.bzl` | One `http_archive` per crates.io crate (155), the name/version/sha256 index the vendor staging resolves through, and the module extension that creates the repos + the 3 pinned Rust toolchain components. **Generated** by `Meta/emit_cargo_bazel.py` **from `Cargo.lock` alone** — no cargo, no network, no CMake | | `cargo.bzl`, `Meta/cargo_build.sh`, `Meta/cargo_binary_build.sh`, `Meta/cargo_vendor.sh` | `rust_sysroot` (the pinned 1.96.1 toolchain merged into one tree), `cargo_crate` / `cargo_binary` (offline, network-blocked build actions), and `cargo_lib` (one consumable `CcInfo` per crate: its archive + its FFI headers) | | `cargo_ring.bzl` | The 10 `cargo_crate` + 10 `cargo_lib` targets and `flapc`, as a macro for the root package (the crate sources are at the repo root and `glob()` is package-relative). **Generated** by `Meta/emit_cargo_bazel.py` | -| `Build/full/**/BUILD.bazel` | Shims over the *reference CMake build tree*: **only generated headers now.** The Rust archive and `flapc` shims are gone (finding 34), as are the vcpkg ones (finding 33) | +| `export_headers.bzl`, `Libraries/LibWeb/export_header.bzl` | The 15 `generate_export_header` `Export.h` files + AK's two `configure_file` headers, generated by Bazel from the same inputs CMake uses. **Generated** by `Meta/emit_export_headers_bazel.py`; `--check Build/full` byte-compares every one against CMake's | +| ~~`Build/full/**/BUILD.bazel`~~ | **Gone.** These shimmed the reference CMake build tree, and by the end they supplied *nothing*: the 709 headers they globbed were 21 Bazel generates and 688 LibWeb bindings headers Bazel also generates. Deleting them removed the *`Build/full`* dependency — not every clone-and-build blocker, as the table at the top now records | +| `Build/vcpkg/BUILD.bazel` | A filegroup over the microsoft/vcpkg checkout. **Still the same `allow_empty = True` glob over a foreign tree that finding 35 is about** — it just globs a *different* tree, so a fresh clone gets no diagnostic. Gap 7 | The four generated files are reproducible: re-running each emitter against the same `Build/full` reproduces them byte for byte — **on the same machine**. They @@ -61,19 +411,125 @@ adds `-I/usr/include/libdrm`), which is gap 3 below, not an emitter bug. ## Reproducing To **build the browser**, no CMake build is needed — Bazel fetches and builds the -vcpkg tree and the Rust crates itself, with zero network access in either: +vcpkg *ports* and the Rust crates itself, and vcpkg's own downloader reaches the +network zero times. **Two** inputs still have to be present first (step 1a; gaps +7–8), both obtainable **without CMake**. The third, the HSTS preload table, is now +fetched by Bazel: it is pinned downstream to a Chromium commit + sha256 in +[`hsts_preload.bzl`](workspace/hsts_preload.bzl): + +**One command does the whole tree**, and it is the recommended path because the +manual version below has four ways to be silently wrong (see +[Reproducing the tree](#reproducing-the-tree-on-another-machine)): + +```sh +./apply_overlay.sh ~/ladybird # branch + commits at the pinned commit, + # then BOTH prefetches, in the order that works +./apply_overlay.sh --verify ~/ladybird # check an existing tree, change nothing +``` + +### What you get: a branch with commits + +It leaves a **branch** — by default `ladybird-bazel-`, based on the pin — +holding **three commits**, and a **clean `git status`**: + +``` +aad224a Bazel overlay: build Ladybird with Bazel alongside CMake +5e5a46f LibRequests: close the response fd on completion, not just at ~Request +2d6a0a8 LibRequests: tear the request down once its body is delivered +``` + +One commit per patch (keeping the patch's own subject) plus one for the ~44 Bazel +files. That shape is the point: it is ordinary git, so it **composes** with what you +already have — + +```sh +git -C ~/ladybird rebase ladybird-bazel-71fb301a851e my-branch # overlay under your work +git -C ~/ladybird cherry-pick 5e5a46f # just the fd fix, no Bazel files +git -C ~/ladybird checkout my-branch # back to where you were +./apply_overlay.sh --onto-current ~/ladybird # overlay ON TOP of your HEAD +``` + +It used to leave a **detached HEAD with 45 untracked files**, which was the wrong +answer to "how do I get your changes into my tree": nothing was lost, but a floating +HEAD is not a place you can work (`rebase`/`merge`/`cherry-pick` all need a named +ref), 45 untracked files make `git status` permanently useless while `git diff` and +`git log` show nothing at all, a stray `git clean -fd` deletes the lot — and it +walked straight past the branch and commits you already had. + +Flags for the cases that differ: + +| Flag | Use | +|---|---| +| `--branch NAME` | build the overlay on a branch you name | +| `--onto-current` | base it on **your** HEAD instead of the pin (warns: the generated BUILD files name ~1,961 sources by path and were generated *from* the pin) | +| `--no-commit` | the old behaviour — mutate the working tree, commit nothing | + +**Already have a tree from an older pin?** The same command moves it forward. The +previous pin's overlay is a *commit* on its own branch, so it no longer collides with +the checkout at all, and your old branch still builds. A re-run is idempotent: it +resets the overlay branch to the pin and rebuilds its commits — but if that branch +holds a commit that is **not** the overlay's, it stops and names it rather than +resetting over your work: + +``` +error: branch 'ladybird-bazel-71fb301a851e' has commits that are not this overlay's: + 9570358 ulf: tweak bazelrc on the overlay branch + I will not reset a branch holding your work. [...] +``` + +Uncommitted changes to tracked files (from a run of the *old* script, or your own) +are **stashed, never discarded** — `git stash pop` restores them. That matters +because a repin can *delete* a patch, so those edits can no longer be +reverse-applied from anything the overlay carries and are indistinguishable from +your own. + +Or by hand: ```sh git clone https://github.com/LadybirdBrowser/ladybird && cd ladybird +git checkout 71fb301a851e4a098e863a7a67e6666599e1cab7 # the commit the generated + # BUILD files describe # 1. Drop in the overlay. cp -r .../examples/ladybird/workspace/. . && mv bazelrc.txt .bazelrc -git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt header - # that is not self-contained (gap 7) +git apply .../examples/ladybird/patches/*.patch # generator determinism, a Qt header + # that is not self-contained (gap 9), + # and the per-request fd leak +# 1a. THE TWO INPUTS THE OVERLAY DOES NOT CARRY -- both are produced by ONE +# ordinary Ladybird build. A Ladybird developer stages nothing by hand; +# these are blockers for a Bazel-ONLY clone, and each one is a thing CMake or +# vcpkg produces as a side effect (see finding 36): +# (a) Build/vcpkg + its .git <- python3 Meta/ladybird.py vcpkg (~70s) +# (b) the four Meta/CMake/vcpkg/git-archives/*.tar.gz +# <- vcpkg writes them into +# Build/vcpkg/downloads/ while building +# skia and angle. Verified: those files' +# SHA512s equal the committed ones in +# vcpkg_git_archives.bzl, i.e. the +# "hand-made" directory was a copy of +# vcpkg's own cache. +# WITHOUT CMAKE (the supported path for a Bazel-only clone): +python3 Meta/ladybird.py vcpkg # (a) checkout + .git, ~70s, no CMake +python3 Meta/fetch_vcpkg_git_archives.py # (b) 4/4 reproduced with git archive and + # verified against the pinned SHA512s +# The HSTS preload table needs NOTHING now: Bazel fetches it as +# @hsts_preload_json//file, pinned to a Chromium commit + sha256 in +# hsts_preload.bzl (below). Ditto `ply`: the wheel is pinned and pip runs +# with PIP_NO_INDEX (gap 8). +# WITH CMake, if you were building Ladybird anyway, (a) and (b) fall out of: +# ./Meta/ladybird.py build +# cp Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz Meta/CMake/vcpkg/git-archives/ +# +# NB vcpkg's buildtrees peak around 3 GB. They go next to the declared output +# (inside bazel-out) rather than $TMPDIR, so a small /tmp tmpfs is not a +# problem and no flag is needed -- see the note in Meta/vcpkg_build.sh for why +# the --action_env route is two ways wrong. # 2. Build. This includes the 77 vcpkg ports (~45 min cold) and the 10 Rust -# crates + flapc: the libraries depend on them through //Meta/vcpkg: and -# //:_lib, so there is no separate step. Build //:vcpkg_installed alone -# if you want to time it. -bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder //:Compositor //:WebWorker +# crates + flapc + cranelift-compiler: the libraries depend on them through +# //Meta/vcpkg: and //:_lib, so there is no separate step. Build +# //:vcpkg_installed alone if you want to time it. +# No CMake build is needed, and none is consulted -- see the removal test +# above. That was not true before finding 35. +bazel build //:ladybird # //:WebContent etc. are data of //:ladybird ``` To **regenerate or verify the BUILD files** you additionally need the reference @@ -90,6 +546,10 @@ python3 Meta/emit_build_bazel.py > BUILD.bazel python3 Meta/emit_codegen_bazel.py Libraries/LibWeb > Libraries/LibWeb/codegen.bzl python3 Meta/emit_root_codegen_bazel.py > codegen_root.bzl python3 Meta/emit_libweb_bazel.py > Libraries/LibWeb/BUILD.bazel +# ...and the generated-source lists that BUILD.bazel loads. A SECOND output, +# because the BUILD file load()s it, so the two cannot share one stdout. It used +# to say AUTO-GENERATED while nothing generated it (see Known gaps 9). +python3 Meta/emit_libweb_bazel.py --generated-srcs > Libraries/LibWeb/generated_srcs.bzl # The vcpkg rules regenerate from the committed capture alone — no vcpkg, no network. python3 Meta/emit_vcpkg_bazel.py --assets Meta/vcpkg_assets.tsv --distfiles > vcpkg_distfiles.bzl python3 Meta/emit_vcpkg_bazel.py --assets Meta/vcpkg_assets.tsv --index > vcpkg_index.bzl @@ -100,6 +560,13 @@ python3 Meta/emit_cargo_bazel.py --index > cargo_index.bzl python3 Meta/emit_cargo_bazel.py --ring > cargo_ring.bzl python3 Meta/emit_cargo_bazel.py --extension > cargo_extension.bzl python3 Meta/emit_cargo_bazel.py --check . # all four reproduce byte-for-byte +# The 15 Export.h + AK's 2 configure_file headers: emitted, and byte-compared +# against CMake's own copies. (AK/Backtrace.h is excluded on purpose -- it is a +# host PROBE, not a template, so comparing it to this machine's tree would only +# re-confirm this machine.) +python3 Meta/emit_export_headers_bazel.py > export_headers.bzl +python3 Meta/emit_export_headers_bazel.py --libweb > Libraries/LibWeb/export_header.bzl +python3 Meta/emit_export_headers_bazel.py --check Build/full # 17/17 identical # 5. Check every generator against CMake. python3 Meta/bazel_parity_harness.py # expects 1408/1408 identical, 0 UNHANDLED # Sweep hash seeds: a generator that iterates a set matches under some seeds only. @@ -109,17 +576,103 @@ for s in 0 1 7 42; do python3 Meta/bazel_parity_harness.py --seed $s; done The emitters locate the checkout via `$LADYBIRD_ROOT`, defaulting to the parent of `Meta/` — so they work from any checkout path. -Running the UI currently needs manual staging, which is itself a gap (see -below): +Running the UI currently needs the RESOURCE root staged by hand, which is itself +a gap (see below). It needs nothing else: the staging step for the helper +binaries was removed, because it was worse than the gap it papered over. + +Every path below is **derived, not written down**, because two of them used to be +stated as literal `k8-fastbuild` paths and one was simply wrong: the vcpkg tree is +built in the **exec** configuration (`vcpkg_lib` pins `cfg = "exec"`), so it is +under `k8-fastbuild-exec` and `bazel-bin/vcpkg_installed` does not exist. The +resource root is `/../share/Lagom`, not `/share/Lagom` -- +`LibWebView/Utilities.cpp`'s `find_prefix()` takes the PARENT of the binary's +directory and appends `share/Lagom`. Ask the build for the paths rather than +spelling any of them out: + +**`//:ladybird` declares the five services as `data`, so one target builds them all.** +This recipe used to name all six targets, and that was a workaround for a missing +dependency edge: the services are found by PATH at runtime +(`get_paths_for_helper_process`), not linked, so nothing in the graph said the browser +needs them and `bazel build //:ladybird` alone left whatever `WebContent` was already +in `bazel-bin`. Ulf hit the consequence: `ladybird` dated Aug 20 beside a `WebContent` +dated Aug 11 — this pin's browser talking to the previous pin's service. Upstream had +inserted ~3 IPC messages between the pins, shifting every id after them, so every +message failed to decode with `Can't read past the end of the stream memory` / +`Endpoint magic number mismatch, not my message!`, which reads like a codegen or ABI +bug and is nothing of the kind. The magic number in those frames is +`AK::string_hash("WebContentServer")` — the *correct* endpoint; the message +**numbering** is what disagreed (7/7 against the old pin, 0/7 against the new one). +`data` rather than `deps` because they are spawned processes, not link inputs — the +relationship `LibWasm` already has to `cranelift-compiler` — and the list is derived +from the `launch_server_process<>` call sites in `HelperProcess.cpp`, since upstream +adds services (`Compositor` is new since the previous pin). + +**Do NOT stage the services into a `libexec/` directory.** They are already +siblings of `ladybird` in `bazel-bin`, and `LibWebView/Utilities.cpp`'s +`get_paths_for_helper_process()` searches `/libexec/` **before** +`/bin/` -- so a `libexec` copy WINS over the fresh build, and only +the copy is ever run. This +recipe used to stage one, and after the 71fb301a repin the stale August copies +in `bazel-out/k8-fastbuild/libexec/` (from the OLD pin) were what actually +launched: the new UI talked to old-pin services, whose IPC message IDs had +shifted, and every message failed to parse with `Endpoint magic number +mismatch` / `Can't read past the end of the stream memory` while all 20 +generated `*Endpoint.h` were byte-identical to CMake's. Verified by removal: with +no `libexec/` at all, `--headless=text` and `--headless=layout-tree` are +byte-identical to the CMake reference. A staging step that shadows the build +output is a cache with no invalidation. ```sh export XDG_RUNTIME_DIR=/tmp/xdg-lb && mkdir -p $XDG_RUNTIME_DIR && chmod 700 $XDG_RUNTIME_DIR -ER=$(bazel info execution_root); mkdir -p "$ER/bazel-out/k8-fastbuild/libexec" -for b in WebContent RequestServer ImageDecoder Compositor; do cp -f bazel-bin/$b "$ER/bazel-out/k8-fastbuild/libexec/"; done -ln -sfn "$PWD/Build/full/share/Lagom" "$ER/bazel-out/k8-fastbuild/share" +BIN=$(bazel info bazel-bin) # target config: where ladybird is +# The services + cranelift-compiler need NO staging: they are siblings of +# ladybird in $BIN, which is the second entry in Ladybird's own lookup chain. +# If a libexec/ exists from an older recipe, it shadows them -- delete it. +rm -rf "$BIN/libexec" "$(dirname "$BIN")/libexec" +# The resource root, assembled from the CLONE and from Bazel's own vcpkg tree. +# This line used to read `ln -sfn "$PWD/Build/full/share/Lagom" ...` -- i.e. the +# recipe for running the Bazel build pointed at CMake's build tree, a seventh +# thing a fresh clone does not have (finding 36). Everything in it is either in +# Base/res or in //:vcpkg_installed; `chmod u+w` because Bazel's outputs are +# read-only and `cp -r` preserves that. +# +# $(dirname $BIN), not $BIN: find_prefix() resolves the resource root against the +# PARENT of the directory holding the binary. +L="$(dirname "$BIN")/share/Lagom" +# rm -f the SHARE dir too, not just Lagom: an older recipe left `share` as a +# SYMLINK into CMake's Build/full, so after that tree moved, `mkdir -p` failed +# with a "File exists"-shaped error on a path that does not exist. +rm -f "$(dirname "$BIN")/share" +chmod -R u+w "$L" 2>/dev/null; rm -rf "$L"; mkdir -p "$L/ladybird/pdfjs/web" +cp -r Base/res/. "$L/" +# ASK for the tree rather than guessing its configuration directory. +V=$(bazel cquery 'deps(//:ladybird)' --output=files 2>/dev/null | grep 'vcpkg_installed$' | head -1) +P="$V/x64-linux-dynamic/share/pdfjs" +cp -r --no-preserve=mode "$P/build" "$L/ladybird/pdfjs/" +cp -r --no-preserve=mode "$P/web/." "$L/ladybird/pdfjs/web/" +# UI/cmake/ResourceFiles.cmake stages this one file into pdfjs/web/, not pdfjs/. +mv "$L/ladybird/pdfjs/pdfjs-ladybird-transport.mjs" "$L/ladybird/pdfjs/web/" ./bazel-bin/ladybird --headless=text file:///tmp/test-page.html ``` +The assembled tree is byte-identical to CMake's `Build/full/share/Lagom` +(`diff -rq`, 0 differences), and with it a fresh clone renders `--headless=text` +and `--headless=layout-tree` byte-identically to the CMake reference on all +three test pages. Re-verified at 71fb301a against `Build/full71` with no +`libexec/` staged at all: `--headless=text` identical, `--headless=layout-tree` +identical (119 lines), `about:version` identical. + +Re-verified once more after the vcpkg pin was **re-captured** at this commit — the +change that moves `sdl3` from the old pin's 3.4.12 to the 3.2.28 `vcpkg.json` asks +for, so all 77 ports rebuilt from the new distfile set. `bazel build` of all six +binaries RC=0, Bazel's own tree carries `libSDL3.so.0.2.28` and +`sdl3_3.2.28_x64-linux-dynamic.list` where it used to carry 3.4.12, and both +`--headless=text` and `--headless=layout-tree` are byte-identical to the CMake +reference (same md5, 8 and 65 lines). Checked the way [gap 11](#known-gaps) says to: +the reference output was regenerated from `Build/full71/bin/Ladybird` on the same +page in the same session rather than compared against a remembered line count, and +the six running binaries were confirmed to be the ones just built. + ## Known gaps Honest inventory of what stops this from being a clone-and-build. @@ -170,7 +723,7 @@ Honest inventory of what stops this from being a clone-and-build. **Rust is closed too (finding 34), so the CMake build is no longer needed to build the browser at all.** The 260 MB prebuilt `librust_combined.a`, the hand-run `ar -M` that produced it and the reference build's `flapc` are gone. - `Cargo.lock` already pins a sha256 for all 154 crates.io crates and the URL is + `Cargo.lock` already pins a sha256 for all 155 crates.io crates and the URL is a pure function of (name, version), so — unlike vcpkg — **nothing had to be captured**: `Meta/emit_cargo_bazel.py` emits the fetch rules from `Cargo.lock` alone, with no cargo, no network and no CMake, and the three @@ -181,6 +734,14 @@ Honest inventory of what stops this from being a clone-and-build. claimed: deleting one crate from the vendor dir fails the action with "no matching package named `yuv` found" instead of downloading it. + The link/symbol measurements in the rest of this finding were taken at the + **previous pin** (`f9e34731`), where the tree had 10 staticlib crates and 14 + FFI headers; at `71fb301a` upstream consolidated `libweb_css_rust` and + `libweb_layout_rust` back into `libweb_rust`, so it is 8 crates and 19 + generated files. The counts below are left as measured rather than rescaled — + a number nobody re-measured is not a measurement — and the parity harness, + the emitter's `--report` and `tests/test_emit_cargo.py` carry the current ones. + Consumed one target per crate (`//:_lib` → its archive + its generated FFI headers), one-for-one with CMake's `target_link_libraries` — **not** a shared `--start-group` over all ten archives, which is what I built first and @@ -231,6 +792,47 @@ Honest inventory of what stops this from being a clone-and-build. rebuilds all 11 crates. Both trade a rebuild for correctness, which is the right way round — under-declaring silently reuses a stale artifact. Per-crate source sets need the path-dependency graph read out of the manifests. + + **Closing the header gap found two real bugs the CMake tree had been masking + (finding 35), and this is the argument for verify-by-removal.** Both were + *correctness* bugs in the Bazel build that a green build could not have shown, + because a stale include path from CMake's tree was quietly supplying the right + answer: + + * **An FFI header collision.** Eight of the ten crates emit a header literally + named `RustFFI.h`, and four TUs `#include ` with no directory. + CMake is unambiguous because `FFI_OUTPUT_DIR` defaults to the library's *own* + binary dir; Bazel puts every dep's include dirs on one command line, so + **LibRegex was compiling against LibUnicode's header** — masked only by a + leftover `-IBuild/full/Libraries/LibRegex` that shadowed both. Removing the + tree turned it into `'RustRegexFlags' has not been declared`. The fix needed + *two* steps, and the first alone looked sufficient: publish the unprefixed + dir on a separate target (`cargo_bare_include`) so it does not ride on + `cargo_lib`, **and** depend on it through `implementation_deps`. Include dirs + propagate along the C++ dep graph too, not just out of one rule — with a + plain `deps` edge, LibGfx inherited LibTextCodec's dir through + `LibGfx → LibTextCodec` and `YUVData.cpp` compiled against the wrong header + (`'FFI' does not name a type`). `implementation_deps` is Bazel's name for + exactly the scope CMake's `PRIVATE` include dir has, which is *why* a bare + include is unambiguous in CMake and had to be made unambiguous here. + * **An entire Rust target missing from the graph.** `Libraries/LibWasm` declares + its Cranelift crate with `build_rust_binary()`, not `import_rust_crate()`, and + the emitter parsed only the latter — so Cranelift was absent from Bazel + *entirely*. Two host escapes were covering for it: the FFI header sat in + `Build/full/Libraries/LibWasm` where a global `-I` reached it, and the + compiler binary was named by an **absolute path to my machine** baked into + `-DWASM_CRANELIFT_COMPILER_PATH=/home/ubuntu/...`, which alone would have + broken any other checkout. It is now a `cargo_binary` that also declares its + `cbindgen` header (byte-identical to CMake's), and the define is rewritten to + a bare file name so Ladybird's *own* lookup chain + (`resolve_cranelift_compiler_path`: env var → compile-time path → + sibling-of-self) finds it in Bazel's bin dir — the dependency is declared as + `data`, the path is not asserted. + + The generalizable part: **a shim that cannot fail cannot be trusted.** Both + bugs were invisible while a foreign tree was on the include path, and both + surfaced the moment it was removed. Neither would have been found by building + harder. 2. **The exec configuration's flags are a duplicated list.** Now that Bazel *runs* a tool it built (`//:generate_interpreter_layout`), that tool and the AK it links are built in the exec configuration, a separate flag namespace that @@ -249,51 +851,428 @@ Honest inventory of what stops this from being a clone-and-build. cross-compile, at which point the skew is silent. Bazel makes the two namespaces explicit and thereby makes the duplication visible; the right fix is a shared `.bzl` flag list or a custom toolchain, not more `--host_*` lines. -3. **`.bazelrc` still has host escapes:** `--action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm` - and `-L/usr/lib/x86_64-linux-gnu`, both because libdrm and Vulkan are - host-provided. (The `-L`/`-rpath` into `Build/full/vcpkg_installed/` are gone — - finding 33.) -4. **The generated-header shims over `Build/full` are what is left of the CMake - dependency.** `Build/full/**/BUILD.bazel` no longer shims any *binary* — the - vcpkg `.so`s (finding 33) and the Rust archives + `flapc` (finding 34) are all - Bazel-built, and `flapc` is now a real `cargo_binary` Bazel builds *and* runs - as a genrule tool, so `interpreter_x86_64.S` depends on no cargo artifact. What - remains is a handful of generated *headers* the emitters have not yet taught - Bazel to generate, plus `Build/full` itself as the model the emitters read and - the baseline the parity harness diffs against. That last one is a - converter-development dependency, not a build dependency. +3. **`.bazelrc` still has host escapes:** `--action_env=CPLUS_INCLUDE_PATH` naming + `/usr/include/libdrm` **and glib's two roots** (`/usr/include/glib-2.0`, + `/usr/lib/x86_64-linux-gnu/glib-2.0/include` — glib puts `glibconfig.h` under a + *libdir*, not an includedir), plus `-L/usr/lib/x86_64-linux-gnu`: libdrm, + Vulkan, glib and xkbcommon are all host-provided. glib joined at the 71fb301a + repin, when upstream added a `pkg_check_modules(GIO)` for UI/Qt's + `ExternalURLActivationToken`/`Handler`; xkbcommon arrives transitively with the + now-required `Qt6::GuiPrivate`. (The `-L`/`-rpath` into + `Build/full/vcpkg_installed/` are gone — finding 33.) + + **No absolute host path remains in the generated BUILD files.** Two did until + recently, and both were the same mistake in different clothing: a value that is + correct on the machine that generated it and meaningless anywhere else, sitting + in a checked-in file where nothing would ever contradict it. + `-DWASM_CRANELIFT_COMPILER_PATH="/home/ubuntu/.../bin/cranelift-compiler"` was + load-bearing (finding 35) and now resolves through Ladybird's own + sibling-of-self lookup; `vcpkg_tree`'s `cache_dir` was merely a resumability + affordance and is now empty, which is also the honest default (an empty cache + *is* the genuine from-source build). Grepping the emitted output for `/home/` + is a one-line check worth keeping in any migration — a hardcoded path that + happens to work is the failure mode that survives every test run on the + author's machine. +4. **The `Build/full` shims are gone; what remains is a *converter* dependency, + not a build dependency.** `Build/full/**/BUILD.bazel` no longer exists. The + vcpkg `.so`s (finding 33), the Rust archives + `flapc` (finding 34), the + Cranelift compiler (finding 35), the 15 `Export.h` + AK's two `configure_file` + headers (finding 35) and all 692 LibWeb generated headers are Bazel's own + outputs. `Build/full` is still read by the *emitters* (it is the model they + translate) and by the parity harness (it is the baseline they diff against) — + which is a dependency of *regenerating* the BUILD files, not of building the + browser from them. + + The distinction was invisible while the shims existed, which is exactly why + they lasted: `glob(["**/*.h"], allow_empty = True)` over a tree that is not + there yields an empty list and no diagnostic, so the build failed ~1,600 + actions later with a missing-header error naming neither the shim nor the + tree. **The lesson generalizes past this migration:** a shim over a foreign + build tree should fail loudly when the tree is absent, or it is indistinguishable + from a shim that is not needed. `allow_empty = True` on a glob whose emptiness + means "the thing you depend on is missing" converts a build error into a + mystery. 5. **Running the UI needs manual staging** (the commands above). Ladybird's UI spawns its service binaries by looking next to itself, and `share/Lagom` for resources. A real Bazel setup would express this with `data` + runfiles; doing so means teaching Ladybird's process-launch path about runfiles, so it - is a change to the target, not just to the BUILD files. + is a change to the target, not just to the BUILD files. **The staging block + itself was a clone-and-build blocker until now** — its last line symlinked + `Build/full/share/Lagom`, CMake's build tree, into place. Everything the + resource root needs is in `Base/res` plus `//:vcpkg_installed`'s pdf.js, and + the block above assembles it from those; the result is `diff -rq`-identical to + CMake's. Finding 36. **Two of its paths were also wrong, and are now derived + rather than written down** (finding 40): the vcpkg tree is built in the *exec* + configuration so `bazel-bin/vcpkg_installed` does not exist, and the resource + root is `/../share/Lagom` -- `find_prefix()` takes the PARENT of the + binary's directory -- not `/share/Lagom`. Both now come from + `bazel info` / `bazel cquery --output=files`. Qt's plugins USED to belong on + this list and no longer do: they are `data` of `//:ladybird` (`qt_runtime.bzl`), + which is what that "real Bazel setup" looks like for one of the three things. + + **The helper binaries no longer belong on this list either, and removing them + fixed a bug the staging step CAUSED.** The block used to `cp` the five + services into `$BIN/libexec/` — which was never needed (they are already + siblings of `ladybird` in `bazel-bin`, the second entry in + `get_paths_for_helper_process()`'s chain) and was actively wrong, because + `/libexec/` is searched **first**. That made the staged copy + shadow the build output: a directory Bazel does not own, that no `bazel clean` + removes and no rebuild refreshes. After the 71fb301a repin the fresh UI kept + launching WebContent binaries staged six weeks earlier from the *previous* + pin; upstream had inserted IPC messages, so message ids past the insertion + point had shifted and every message failed to decode — ~14,000 lines of + `Endpoint magic number mismatch` / `Can't read past the end of the stream + memory` from a build whose 20 generated `*Endpoint.h` were byte-identical to + CMake's. **Every check aimed at the code generator correctly said the build + was fine; the artifact that ran was not built by the build.** Fixed by + deleting the step and clearing any `libexec/` an older recipe left, guarded by + `tests/test_run_recipe.py`. The general shape: a convenience copy of a build + output, placed where the program looks first, is a cache with no + invalidation — and it fails as a *miscompile*, in the one direction the + parity checks cannot see. 6. **`rules_qt` is not on the BCR.** `MODULE.bazel` uses an `archive_override` pointing at kklochkov/rules_qt v2.0.1's release tarball (stock upstream, no patches). The BCR's `rules_qt` module is Vertexwahn's unrelated `rules_qt6`. Qt itself *is* host-portable: `qt.local_repo` discovers the host Qt via - `qmake -query`, so no Qt SDK is vendored. -7. **Three upstreamable Ladybird fixes.** The first is the `sorted()` determinism - fix in `Meta/Generators/libweb_bindings/to_idl_value.py`, filed as - [ladybird#10899](https://github.com/LadybirdBrowser/ladybird/issues/10899) and - fixed upstream. - - The second is a one-line `#include ` in `UI/Qt/TabBar.h`: - [`patches/0002-ui-qt-tabbar-self-contained-header.patch`](patches/0002-ui-qt-tabbar-self-contained-header.patch). - `TabBar.h` calls `as()` — a `dynamic_cast`, needing Tab's complete type — - while only forward-declaring `Tab`, and compiles under CMake purely by ordering - luck: AUTOMOC's unity `mocs_compilation.cpp` includes `moc_Tab.cpp` (hence - `Tab.h`) before `moc_TabBar.cpp`. Bazel mocs each header separately, so nothing - supplies the definition first. A latent upstream bug rather than a Bazel quirk — - any build that changes compile order (different unity bucketing, an IWYU pass) - hits it. Still to file upstream. - - A **third** one is now needed, in the same function as the first: the topological - sort in `dictionaries_in_dependency_order` iterates a *set* of dependency names, - so two dictionaries that do not depend on each other (`AudioConfiguration` and - `VideoConfiguration`, both reached from `MediaConfiguration`) emit in hash order - and `Bindings/MediaCapabilities.h` varies with `PYTHONHASHSEED`. A topological - sort constrains dependency-before-dependent; the order among independent - siblings must be pinned separately. Found by the harness's seed sweep, not by - any single run — the inherited-seed run was clean. The fix is in - [`patches/0001-libweb-bindings-deterministic-dictionary-order.patch`](patches/0001-libweb-bindings-deterministic-dictionary-order.patch); - apply it in the checkout before running the harness. + `qmake -query`, so no Qt SDK is vendored. Its **plugins** are host-portable too + now, and that took a fix rather than an observation: rules_qt wires up Qt's link + half only, so the binary linked @qt's libraries and then `dlopen`ed the HOST's + QPA plugin into them -- a SIGSEGV where the two Qt versions differ, and a silent + pass where they agree. `qt_runtime.bzl` stages the plugins of the SDK @qt itself + names, points a generated `qt.conf` at them, and enforces the Qt >= 6.9 floor + `UI/Qt/CMakeLists.txt` declares. Finding 40. + + The same crash then came back from the *other* direction, and it was not Qt's + doing: CMake puts `-fPIE` on every executable target, the generator copied it + into each `cc_binary`'s `copts`, and Bazel appends those AFTER the global + `--copt=-fPIC` — so the last flag won and the UI objects compiled `-fPIE`. That + lets GCC reference extern data PC-relative, the linker emits an + `R_X86_64_COPY`, and against a Qt built with `reduce_relocations` (every + official/aqt SDK; Debian's is built without it) `QCoreApplication::self` ends up + defined in the executable while `libQt6Core` writes its own copy: `qApp` is set + in one place and read, still null, in another. Qt's headers `#error` on exactly + this, but only when `__PIC__` is unset — Bazel passed both flags, so the check + never fired. The emitter now drops `-fPIE` (`DROPPED_TARGET_FLAGS`): 39 + `R_X86_64_COPY` relocations across the six executables, now 0. Finding 41. +7. **The fresh clone needs two inputs the overlay does not carry** (step 1a + above), and each stayed invisible for the same reason: it was already on the + machine. `Build/vcpkg` — a microsoft/vcpkg checkout at `vcpkg.json`'s + `builtin-baseline`, created by Ladybird's own `Meta/ladybird.py vcpkg` — is + globbed by `//Build/vcpkg:tree` with `allow_empty = True`, so on a clone it + matches one file and the failure names a missing `/tmp/.../root/vcpkg`. Its + `.git` is *load-bearing* (vcpkg resolves versioned ports with `git read-tree`) + yet the filegroup **excludes** it — an undeclared input the build reads anyway, + because the action is `no-sandbox`. The four `vcpkg_from_git` tarballs are the + other, reproduced by `Meta/fetch_vcpkg_git_archives.py`. Both are prefetches, + not rules: a `git_repository` strips `.git`, and `git archive` output has no URL + to `http_file`. + + **The third one — the HSTS table — is now closed.** It was the interesting case, + because the unpinned fetch is *upstream's* (`hsts_preload.cmake` tracks Chromium's + `main`) and we cannot change that. So it is pinned **downstream**: an `http_file` + at an immutable commit + `sha256` in `hsts_preload.bzl`, regenerated by + `Meta/pin_hsts_preload.py`, verified byte-identical to CMake's generated table + with the staged file deleted, and the upstream unpinned fetch filed as a bug. The + generalizable shape: **a converter cannot pin an input on the foreign build + system's behalf, but it can pin it for itself — provided it pins the revision the + foreign system is currently serving, and proves that with a byte comparison + rather than a hash it invented.** Pin the wrong revision (a release tag, say) and + the hermeticity gap becomes a parity gap. + + **And a fourth class, which no pin can close: host tools.** vcpkg has no Linux + download for `nasm` or the autotools set — its `nasm` URLs live inside + `if(CMAKE_HOST_WIN32)`, and `vcpkg-make` demands `autoconf`/`automake`/`libtool` + with a bare `find_program` and a `FATAL_ERROR`. So they are *named* instead: + `Meta/vcpkg_host_tools.tsv`, derived from vcpkg's own scripts, checked by + `vcpkg_build.sh` before anything builds (finding 39). Every one of these was + invisible for exactly the reason in this gap's first sentence — **it was already + on the machine** — and each cost a full ~20-minute build to find, in an error + naming the wrong place. `glslangValidator` is the same class and is **not** yet + named: two genrules in `codegen_root.bzl` hardcode `/usr/bin/glslangValidator` + and `vcpkg_installed` does not ship it, so it needs a pinned binary or a repo + rule that fails legibly. Fixing *how you find out* is not the same as fixing the + dependency, and this gap is only the former. + +8. **"Zero network access" is true of vcpkg's downloader, not of the vcpkg + action.** The pin + `x-block-origin` is verified — a distfile missing from the + index is a hard error, not a fetch. But `x-block-origin` only governs vcpkg's + *own* downloads, and the `angle` overlay-port calls + `x_vcpkg_get_python_packages`, which runs **`pip install ply`**. That is not an + asset-cache download, so nothing pins it and `x-block-origin` never sees it. + Nor is anything stopping it: `vcpkg_tree` sets `requires-network: "0"`, which is + a *scheduling hint* Bazel does not enforce, and `no-sandbox: "1"` means there is + no network namespace to enforce it in — so with `use_default_shell_env = True` + the action inherits `HTTP_PROXY`/`HTTPS_PROXY` and pip reaches PyPI. + + It went unnoticed because this sandbox exports a proxy, so pip silently + succeeded; a fresh clone in a network-free environment fails with + `No matching distribution found for ply`. Two things to fix, and the second + matters more than the first: pin `ply` as an `http_file` wheel staged into the + port's venv, **and stop taking `requires-network: "0"` as a claim** — the + verification has to be an environment with no route to the network, not a flag + that says there is none. Same shape as the shim that could not fail: a control + that is not enforced is indistinguishable from one that is not there. + +9. **A repin's first failure mode is a *loading-time* one, and no per-target check + can catch it.** Ulf's build at `71fb301a` did not fail to compile anything; it + failed to load: + + ``` + Error in glob: glob pattern 'Libraries/LibJS/BytecodeDef/**' didn't match + anything, but allow_empty is set to False + ``` + + Upstream `a32d9c9f` ("LibJS: Derive bytecodes from Flap handlers") deleted that + directory, and the pattern was a hardcoded string in a list literal in + `Meta/emit_cargo_bazel.py`. Three properties compound here: + + * `allow_empty = False` is **correct** and had to stay. Its opposite is the + reason the old `Build/full` shim packages could match nothing for weeks and + fail 1,600 actions later — a file list that *may* be empty proves nothing. + * A loading error has **no target to blame**, so nothing in the build graph can + report it and no test that builds or queries a target can reach it. + * The emitters that would have re-derived the pattern all ran green, because an + emitter that enumerates *kinds* of thing cannot notice a kind going away: + the same repin silently dropped `gen_Op` from `codegen_root.bzl`, and the + parity harness reported **`0 UNHANDLED`** while two generated headers had no + owner at all — its first-match-wins classifier bucketed the new Rust + generator as "resource staging", because that command's ninja rule is + ` … && cmake -E copy_if_different …` and the *exclusion* pattern + matched first. Every exclusion in a first-match classifier can silently + capture a command it was not written for, and the count that should have + caught it is computed after the capture. + + The fix is structural rather than a re-pin of the string: every crate directory + an `allow_empty = False` glob names is now **derived** — `Cargo.toml`'s + `members` closed over the manifests' `path =` dependencies (which is also how + the derivation reaches *outside* the workspace, since `libjs_rust` + build-depends on the `exclude`d `flapc`) — and `flapc`'s own extra inputs come + from scanning it for `include_str!`. So a deleted crate deletes its own glob + pattern. `tests/test_emit_cargo.py` guards it from both ends: the derivation + drops a pattern when the directory is removed, and **no module-level constant + in either emitter may hold a `/**` string** — the shape the bug took. + + The same consolidation (`libweb_css_rust` + `libweb_layout_rust` → + `libweb_rust`) also moved two things worth naming: two of the crate's 19 + generated files are **`.inc`, not `.h`**, and Bazel deletes an undeclared + `.inc` exactly as it deletes an undeclared header — so the suffix must not be + what decides whether a generated file is declared. And `build_rust_binary()` + takes `FEATURES` too (upstream's new `style-replay` is built + `FEATURES style-recording` from the *same crate* as `libweb_rust`'s staticlib), + which the parser only read on the `import_rust_crate` side: it would have built + a different binary than CMake does and said nothing. + + **Then, once loading was fixed, the build failed four more times — and every + one was the same bug in a different file.** The glob was only the first + *capture* to come due, i.e. a value derived once on the capturing machine, + written down, and thereafter believed. None of the four was found by reasoning + about the repin; each was found by the next error message. In the order the + build produced them: + + * `Libraries/LibWeb/generated_srcs.bzl` began with the line + `# AUTO-GENERATED by Meta/emit_libweb_bazel.py` and **no code path in that + emitter wrote it**. Its two lists (which of LibWeb's ~1,390 compile inputs + come from codegen) were hand-maintained, so upstream's five new generated + headers and four new `.cpp` were absent, and a *generated* header included a + *generated* header the `cc_library` did not declare: + `Bindings/Window.h:17: fatal error: LibWeb/Bindings/WindowGlobalMixin.h: No + such file or directory` — with the file present on disk beside the one that + included it. A false AUTO-GENERATED claim is worse than an honest + hand-written file, because it tells the next repin that re-running the + emitter will refresh it. Both lists are now derived (the reference build's + compile list; `codegen.bzl`'s own `outs`) and the file is emitted by + `--generated-srcs`. + * Fixing that exposed a **second bug it had been hiding**: `genrule_outputs()` + regexed the whole of `codegen.bzl` for anything path-shaped, so a genrule's + `srcs` counted as outputs too. Four *checked-in* headers that + `generate_dom_tree.py` reads (`HTML/TagNames.h` and friends) were classified + as generated — which is the `hdrs` `exclude=` list, so they were dropped from + the source glob and re-added as labels nothing produces. It never broke the + build only because the stale capture was consulted instead of the function's + answer. One capture concealing a live bug is the strongest argument against + keeping one. + * `QT_MAP` was a three-entry dict of the Qt modules Ladybird used when it was + measured. Upstream made `Qt6::Positioning` **required**; the dict had no key, + the dep fell through to `UNKNOWN`, and `//:ladybird` failed with + `QGeoPositionInfo: No such file or directory`. CMake's `Qt6` → + rules_qt's `@qt//:Qt` is a *rename*, so it is now a rule — and one + that still returns `None` for a non-Qt name, because reporting UNKNOWN is + right and inventing a label is not. + * `moc_headers()` ended with `not h.endswith("GeolocationProviderQt.h")`, + justified in its docstring by "Qt6::Positioning is not found in this + configuration" — a fact about the capturing machine, false at this pin. The + condition is now *does the reference build compile the sibling `.cpp`*, which + is in the model. Wrong in both directions: moc'ing a header CMake does not is + a target only Bazel builds; skipping one it does is a missing vtable at link. + + One of the five did not surface as an error at all, and would not have: the + extractor turned `/usr/lib/libgio-2.0.so` into the dep name `gio-2`, because it + cut the basename at the **first dot** instead of the extension. Invisible for + `libz.so` and `libQt6Widgets.so.6.10.2`; wrong for every library whose name + contains a dot. It only became visible because the three glib deps upstream + added arrived as unresolvable `UNKNOWN`s — had the emitter guessed a label + instead of reporting, `-lgio-2` would have failed at link time, far from here. + **An emitter that reports what it could not resolve converts a silent wrong + answer into a loud missing one**; that property found this bug, and it is the + same property that makes gap 9's `allow_empty = False` worth keeping. + + The vcpkg equivalent is still open, and is the cleanest specimen of the class. + `Meta/vcpkg_assets.tsv` is a capture that deliberately *replaces* the static + portfile parse (a portfile is a CMake program; the regex cannot see through its + platform branches). The reasoning was checked once, against three rows that + were genuinely Windows-only — and then frozen into the diagnostic, which + printed **every** casualty as an entry "vcpkg never asked for on this + platform". At this pin `vcpkg.json` moved `sdl3` to 3.2.28, the versions-db + derivation resolves that correctly, and the message reported the correct + current pin being discarded in favour of a stale captured 3.4.12 as a + Windows-only fetch. The classification needs no new input — a dropped row whose + *URL family* the capture also has is the same project at another version, i.e. a + stale capture — and the emitter now says so. The capture has since been re-taken + at this pin and `sdl3` **is** 3.2.28 in `vcpkg_distfiles.bzl`; of the 76 rows, + 75 reproduced the previous capture's `(url, sha512)` byte-for-byte and the one + difference is the intended pin move. **A replace-wins rule whose message asserts + the reason for the replacement instead of checking it will eventually be + confidently wrong**, and that sentence describes every item in this list. + + The check was one-directional, which cost a hand-fix within the hour of writing + it: it compared derived-against-captured only, so a capture with an **extra** row + was invisible. Re-capturing needed a supplementary run for `angle` alone, and a + one-port manifest resolves its dependencies from the vcpkg *baseline* rather than + from Ladybird's `vcpkg.json` overrides — so it fetched zlib 1.3.2 where Ladybird + pins 1.3.1, and the merged capture carried both. I deleted the row by hand, + which is precisely the move this list is about, and it is equally derivable: a + *captured* row in the same URL family at a version the derivation does not pin + came from the wrong resolution. `classify_capture_only` reports it as a LEAKED + CAPTURE ROW. + +10. **Two upstreamable Ladybird fixes — down from four, and that is the point of a + repin.** Two of the four are **fixed upstream** at this pin (`71fb301a`) and their + patches are deleted, which is worth stating because a patch directory is a debt + register that only shrinks if somebody re-reads it: + + * The `sorted()` determinism fix in `Meta/Generators/libweb_bindings/to_idl_value.py` + — filed as [ladybird#10899](https://github.com/LadybirdBrowser/ladybird/issues/10899) + and fixed upstream **better than my patch was**: upstream sorts inside + `dependency_names_for`, so no caller can receive a set, whereas my patch sorted at + the one call site I had found. Two of my four patches were the *same* bug in that + one function, and I filed the second as distinct; it was not. + * The one-line `#include ` in `UI/Qt/TabBar.h`. `TabBar.h` calls + `as()` — a `dynamic_cast`, needing Tab's complete type — while only + forward-declaring `Tab`, and compiled under CMake purely by ordering luck: + AUTOMOC's unity `mocs_compilation.cpp` includes `moc_Tab.cpp` (hence `Tab.h`) + before `moc_TabBar.cpp`. Bazel mocs each header separately, so nothing supplied + the definition first. Upstream now includes the header. + + Both survived in `patches/` only because the old pin (`f9e34731`) predated the + upstream fixes — a patch keeps applying long after it stops being needed, so + "it still applies" is not evidence that it is still a bug. + + ### Why upstream's patches replaced mine + + What remains is the fd leak, and the overlay now carries **upstream's three + commits** (Ladybird PR #11041, by sideshowbarker) rather than the two patches + written here. That swap was not housekeeping — **my `0002` crashed Ulf's browser**, + and the trace named the exact line: + + ``` + VERIFICATION FAILED: m_ptr at ./AK/OwnPtr.h:134 + #0 ...CallableWrapper::call() + ``` + + That lambda is the read notifier's `on_activation`, and it is the frame that *calls* + `on_finish`. My `release_response_fd()` ran from inside that completion branch and + set `m_internal_stream_data->read_stream = nullptr` — while the calling frame goes on + to dereference exactly that `OwnPtr`: + + ```cpp + if (m_internal_stream_data->read_stream->is_eof()) // Request.cpp:376 + m_internal_stream_data->read_notifier->close(); + ``` + + `OwnPtr::operator->` is `VERIFY(m_ptr)`. So the fix nulled a pointer its own caller + still owned: a use-after-null one stack frame up, invisible on my workloads and a + crash after a few minutes of real browsing on his. Upstream never touches + `read_stream`; it only ensures `defer_teardown()` is *reached*, and reaches it + **before** `user_on_finish` so the deferred lambda's `NonnullRefPtr` pins the + `Request` across the callback — where my `0001` called it *after*, which was a second + latent use-after-free in the same pair. + + The deeper lesson is about what the two patches were *for*. Mine closed the fd on the + theory that a surviving reference pinned it; upstream fixes the leak by running the + teardown on all three paths it can be missed — ordinary completion, `abort()`/ + `terminate()`, and a navigation parked for content sniffing. If reaching the teardown + is sufficient, the surviving-reference theory was never needed, and carrying both + would have meant two mechanisms closing one descriptor, one of them justified by a + theory the other disproves. Upstream also found a class my instrument could not: + `fd_census.py` ranks by growth and splits `peer=DEAD`/`ALIVE`, which identified the + completed-request class but says nothing specific about the *cancel* path. And their + patch 3 is the lead I had and could not reproduce — my two workloads abandoned + navigations without **destroying the navigable**, so they exercised everything except + the condition that matters. A falsified workload was evidence about my workload, not + about the hypothesis. + + All three are annotated in place with their provenance and carry `.effect-grep` + files, so `apply_overlay.sh --verify` recognises the merged upstream fix on a tree + newer than our pin instead of demanding a patch that would conflict. Delete all + three on the first repin past the merge. + + Diagnose a running browser with [`fd_census.py`](fd_census.py), which needs no + patch and no particular tree — `python3 fd_census.py --all --watch 30` ranks every + browser process by fd *growth*, so the data names the leaking process instead of + your hypothesis naming it. Each report also states **which fd fixes the running + binary contains** (`--build` for that alone), read out of the process's own mapped + ELF symbols: a leak rate is uninterpretable without it, since "still leaking at + 92/min" means "the fix does not work" or "the fix was not in this build" and those + need opposite next steps. It reports *"cannot tell"* rather than "fix absent" + whenever absence could be explained by inlining: under LTO in a **static** build a + small internal-only method is inlined into its only caller and leaves no symbol and + no string behind, so `.debug_str` is read too and a "missing" verdict requires a + symbol inlining cannot erase to be visible. This is a correction — the first version + told Ulf `0002` was absent from a binary that contained it, and the negative control + did not catch it because it was vulnerable to the same optimisation. A control only + rules out "unreadable" if it cannot vanish for the same reason as what it guards. + Since upstream has landed its own fix, + `--verify` also accepts an equivalent fix in place of our exact bytes: a patch we + carry only until upstream fixes it has an `.effect-grep` beside it, and verify + falls back to asking whether the *effect* is present before reporting the patch + missing. + +11. **A capture can only see the downloads vcpkg actually *requests*, and four + different things stop it requesting them — each while vcpkg exits 0.** This is + the same class as gap 10 but on the input side, and it is worse, because the + failure is a *pin that looks complete*. `Meta/vcpkg_capture_assets.sh` records + every `(url, sha512, dst)` by being vcpkg's asset cache, so a download nobody + asks for leaves no trace at all: + + - a **failed fetch** halts the rest of its portfile — the four WebKit files + `angle` downloads *after* its python venv step were lost this way, and vcpkg + reported "All requested installations completed successfully in: 49 min"; + - **`--only-downloads`** halts every portfile at its first executed step, which + is *before* those same four files (`portfile.cmake:86` vs 123–153). It was the + script's default, under a comment of mine claiming it "is enough because the + asset hook fires during resolution" — a sentence written from intuition. The + committed 76-row capture cannot have been produced that way, and the fast mode + is now opt-in via `CAPTURE_ONLY_DOWNLOADS=1`; + - a **binary-cache hit** unpacks an archive and never runs the portfile at all. + Measured on a zlib-only manifest against a warm cache: first run 3 rows, + second run **0 rows** and exit 0 in 1.54 ms. `--binarysource=clear` is now + mandatory here (`vcpkg_build.sh` had always passed it, for the adjacent + reason); + - an **already-installed port** or a **warm `downloads/`** likewise skips the + request (`-- Using cached gni-to-cmake.py` → no row) — and the warm downloads + dir is exactly what the *resume* rule asks you to share between runs, so the + property that makes a 50-minute job restartable is the property that makes its + output incomplete. + + So the script judges its own completeness, since vcpkg's exit code will not: + every loss lands in one sentinel that refuses to bless the file. The check that + does not depend on my having enumerated the list above is derived from vcpkg's + own log — every download it resolves is announced, so require a captured row for + each announcement, treating an absolute path in `-- Using cached ` as the + `vcpkg_from_git` case that legitimately bypasses the asset cache. What cannot be + done is *triage*: 58 of 77 ports halt harmlessly in download-only mode (at + `vcpkg_cmake_configure`, after every download), and nothing in the log + distinguishes those from `angle`'s mid-sequence halt, because only the portfile + knows whether a download follows. **Any halt therefore fails the capture** — the + alternative is a rule that is right about 57 ports and wrong about the one that + matters. diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh new file mode 100755 index 0000000..bae5d9a --- /dev/null +++ b/examples/ladybird/apply_overlay.sh @@ -0,0 +1,813 @@ +#!/usr/bin/env bash +# Reproduce the Ladybird tree this migration builds, on any machine. +# +# The overlay is not a fork: it is a pinned upstream Ladybird commit + two +# patches + a set of Bazel files that live alongside CMake's. This script is the +# executable form of that sentence, because "copy the workspace directory over a +# clone" is a recipe with three ways to be silently wrong: +# +# * the LADYBIRD COMMIT. The generated BUILD.bazel files name ~1,961 LibWeb +# compile inputs and 665 IDL bindings by path. They were generated from ONE +# upstream tree; against a different one the build fails on a moved file, or +# worse, silently omits a new one. Nothing in the overlay recorded which +# commit that was until this script did. +# * the PATCHES. Two upstream defects have to be fixed or the build is +# nondeterministic (generator emits dictionaries in PYTHONHASHSEED order) or +# does not compile at all under per-header moc (TabBar.h is not +# self-contained). Both are filed upstream; until they land they are patches. +# * the ORDER. Two overlay files must not exist before the vcpkg prefetch runs. +# Build/vcpkg/BUILD.bazel makes the directory Build/vcpkg EXIST, and upstream's +# Meta/Utils/build_vcpkg.py treats "the directory is there" as "the checkout is +# there": it skips the clone and runs `git -C Build/vcpkg rev-parse HEAD`, which +# -- there being no .git inside -- WALKS UP to Ladybird's own repo and returns +# Ladybird's HEAD. It then tries to check vcpkg's baseline out of the Ladybird +# repo and dies with `fatal: unable to read tree`. So the prefetch has to happen +# BEFORE that file is staged, and this script does the two in that order. +# +# * the RENAME. bazelrc.txt has to become .bazelrc. It is stored under a +# different name so that a `cp -r` of the overlay into a clone cannot be +# mistaken for a working build -- and a rename that a human does by hand is a +# rename a human forgets. +# +# Everything here is pinned and checkable, so a failure is a message rather than +# a mystery. Usage: +# +# ./apply_overlay.sh /path/to/ladybird # branch + commits (see below) +# ./apply_overlay.sh --verify /path/to/existing # check a tree, change nothing +# +# Both prefetches run here (Meta/ladybird.py vcpkg, then +# Meta/fetch_vcpkg_git_archives.py); after this, `bazel build` is the next command. +# The second one used to be PRINTED rather than run, which cost Ulf a 20-minute +# build: everything the script does succeeded, so the obvious next step was +# `bazel build`, and it failed inside the vcpkg action with "no git-sourced +# externals at ./Meta/CMake/vcpkg/git-archives". A setup script that stops one +# required step short of a working build has not set anything up -- the closing +# message is not a substitute for doing it (finding 35 again: an instruction the +# reader must remember is a step the script decided not to take). +# +# --------------------------------------------------------------------------- +# WHAT THIS LEAVES BEHIND: a BRANCH, with COMMITS. It used to leave a detached +# HEAD with 45 untracked files, which Ulf correctly called idiotic: +# +# * a detached HEAD is not a place you can work. Every git verb that composes +# -- rebase, merge, cherry-pick, pull --rebase -- needs a named ref, so the +# overlay could not be moved onto the branch the reader actually has. +# * 45 untracked files means `git status` is 45 lines of noise forever, `git +# diff` shows nothing (untracked files are not diffed), `git log` says +# nothing happened, and a stray `git clean -fd` deletes the entire overlay. +# * worst, it ignored what the reader already had. Their branch, their commits, +# their tree: the script walked past all of it to a floating checkout. +# +# So the overlay is now expressed the way every other change to a git repo is: +# as commits on a branch you can name, inspect, rebase and merge. +# +# patches/*.patch -> one commit each, keeping the patch's own subject +# workspace/* -> one commit ("Bazel overlay: ...") +# +# and by default they go on a branch named after the pin, based on the pin, so a +# repin gets its own branch and your previous one is untouched. Nothing is +# detached and `git status` is clean when it finishes. To put the overlay on top +# of work you already have instead, use --onto-current. +# +# --qt-prefix DIR the Qt SDK to build against. MODULE.bazel's qt.local_repo +# `paths` line is the ONE line in the overlay that is a fact +# about your machine, and copying the overlay over it is how a +# re-apply used to silently repoint a working build at the +# system Qt. Ulf builds against Qt 6.9.2 in a VENV while his +# system Qt is 6.4.2, i.e. the hardcoded /usr/lib/qt6 is not +# merely different for him, it is BELOW Ladybird's 6.9 floor -- +# so the re-apply turned a working tree into a failing one. +# Resolution order (first wins), all reported: +# 1. --qt-prefix DIR +# 2. the `paths` line already in the target's MODULE.bazel +# 3. the qmake first on PATH (a venv/aqt SDK puts its own +# there, which is exactly the right answer for one) +# 4. /usr/lib/qt6, the historical default +# --verify treats that line as expected-to-differ for the same +# reason: it is yours, not ours. +# --branch NAME the branch to build (default: ladybird-bazel-) +# --onto-current base it on your current HEAD, not on the pinned commit -- +# i.e. apply the overlay ON TOP of your own work. The pin +# check becomes a warning, because you are deliberately +# building against a tree the generated BUILD files were not +# generated from. +# --no-commit the OLD behaviour: mutate the working tree, commit nothing. +# Kept because a throwaway build directory does not want a +# branch, but it is no longer what you get by default. +# +# The one file that is NOT committed is Build/vcpkg/BUILD.bazel: Ladybird's own +# .gitignore ignores `Build*/`, so committing it would need -f and would fight +# upstream's intent. Being ignored, it also produces no `git status` noise, so +# leaving it out costs nothing. +set -euo pipefail + +# The upstream commit every generated BUILD file in this overlay was generated +# from, and the only tree they are known to describe. A tag would be wrong here +# for the same reason it was wrong for the HSTS table (see hsts_preload.bzl): a +# tag is a pin to a different tree than the one measured. +LADYBIRD_COMMIT="71fb301a851e4a098e863a7a67e6666599e1cab7" +LADYBIRD_REPO="https://github.com/LadybirdBrowser/ladybird.git" + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +WORKSPACE="$HERE/workspace" +PATCHES="$HERE/patches" + +die() { echo "error: $*" >&2; exit 1; } +note() { echo "==> $*"; } + +VERIFY=0 +PREFETCH=1 +COMMIT=1 +ONTO_CURRENT=0 +BRANCH="" +QT_PREFIX="" +# The value baked into workspace/MODULE.bazel, i.e. what a plain copy would impose. +QT_DEFAULT="/usr/lib/qt6" +USAGE="usage: $(basename "$0") [--verify] [--no-prefetch] [--branch NAME] + [--onto-current] [--no-commit] [--qt-prefix DIR] " +while [ $# -gt 1 ]; do + case "${1:-}" in + --verify) VERIFY=1; shift ;; + --no-prefetch) PREFETCH=0; shift ;; + --no-commit) COMMIT=0; shift ;; + --onto-current) ONTO_CURRENT=1; shift ;; + --branch) BRANCH="${2:-}"; [ -n "$BRANCH" ] || die "$USAGE"; shift 2 ;; + --branch=*) BRANCH="${1#--branch=}"; shift ;; + --qt-prefix) QT_PREFIX="${2:-}"; [ -n "$QT_PREFIX" ] || die "$USAGE"; shift 2 ;; + --qt-prefix=*) QT_PREFIX="${1#--qt-prefix=}"; shift ;; + -h|--help) echo "$USAGE"; exit 0 ;; + -*) die "unknown flag: $1 +$USAGE" ;; + *) break ;; + esac +done +# A lone --verify/--help with no path still has to be accepted by the loop above, +# which stops at $# -eq 1; catch the flag-only forms here. +case "${1:-}" in + --verify) VERIFY=1; shift ;; + -h|--help) echo "$USAGE"; exit 0 ;; +esac +[ $# -eq 1 ] || die "$USAGE" +TARGET="$1" + +# The default branch name carries the pin, so a repin lands on a NEW branch and +# the one you built last time still exists, still builds, and is still yours. +[ -n "$BRANCH" ] || BRANCH="ladybird-bazel-${LADYBIRD_COMMIT:0:12}" + +[ -d "$WORKSPACE" ] || die "no workspace/ next to this script ($WORKSPACE)" + +# --------------------------------------------------------------------------- +# The file list is derived from the overlay itself, never hand-maintained: a +# hand-kept list is how a newly added .bzl gets committed, documented, and then +# copied by nothing (which is exactly what happened to vcpkg_git_archives.bzl -- +# it was generated, committed, listed in the README table, and loaded by no one). +# Written to a temp file rather than consumed through process substitution: +# `< <(...)` needs /dev/fd, which some sandboxes and minimal shells do not give +# you, and this script has to run on the machine the reader has, not mine. +FILE_LIST="$(mktemp)" +trap 'rm -f "$FILE_LIST"' EXIT + +overlay_files() { + (cd "$WORKSPACE" && find . -type f ! -name '*.pyc' -printf '%P\n' | sort) > "$FILE_LIST" + cat "$FILE_LIST" +} + +# bazelrc.txt -> .bazelrc is the one path that differs between the overlay and +# the tree. Keep the mapping in ONE place; both copy and verify read it. +target_path() { + case "$1" in + bazelrc.txt) echo ".bazelrc" ;; + *) echo "$1" ;; + esac +} + +# --------------------------------------------------------------------------- +# The Qt prefix: the one line in the overlay that is a fact about YOUR machine. +# +# Everything else here is a fact about Ladybird at the pin, identical on every +# host. `qt.local_repo(paths = {"linux-x86_64": ...})` is not: it names an SDK, and +# copying the overlay over it silently replaced a working answer with the answer +# from the capturing machine. Ulf: "We're using Qt (6.9.2) from a VENV, and system +# Qt is 6.4.2" -- so the copy did not just change his configuration, it moved him +# BELOW Ladybird's 6.9 floor, turning a working tree into a failing one on re-apply. +_qt_line_path() { grep -nE '^\s*paths = \{"linux-x86_64":' "$1" 2>/dev/null | head -1; } + +# The prefix currently configured in a tree's MODULE.bazel, if any. +qt_prefix_in_tree() { + [ -f "$1/MODULE.bazel" ] || return 1 + sed -n 's|^[[:space:]]*paths = {"linux-x86_64": "\([^"]*\)".*|\1|p' \ + "$1/MODULE.bazel" 2>/dev/null | head -1 +} + +# The SDK whose qmake is first on PATH. For a venv/aqt Qt that is precisely the +# right answer -- activating the venv is how you say which Qt you mean -- and it is +# how rules_qt would find it if `paths` were not hardcoded. +qt_prefix_from_qmake() { + command -v qmake6 >/dev/null 2>&1 && q=qmake6 || q=qmake + command -v "$q" >/dev/null 2>&1 || return 1 + p="$("$q" -query QT_INSTALL_PREFIX 2>/dev/null)" || return 1 + [ -n "$p" ] && [ -d "$p" ] && echo "$p" +} + +qt_version_at() { + for q in "$1/bin/qmake6" "$1/bin/qmake"; do + [ -x "$q" ] && "$q" -query QT_VERSION 2>/dev/null && return 0 + done + return 1 +} + +# Resolve, in the documented order, and SAY which rule won: a silent default is +# what made this a bug in the first place. +# +# Sets the GLOBALS QT_RESOLVED/QT_SOURCE rather than echoing: called through +# `$(...)` the assignment to QT_SOURCE would happen in a subshell and be lost -- +# which it was, printing "(from )" on the first run of this code. +resolve_qt_prefix() { + local tree="$1" from + if [ -n "$QT_PREFIX" ]; then + QT_RESOLVED="$QT_PREFIX"; QT_SOURCE="--qt-prefix"; return + fi + if from="$(qt_prefix_in_tree "$tree")" && [ -n "$from" ]; then + QT_RESOLVED="$from"; QT_SOURCE="the paths line already in your MODULE.bazel"; return + fi + if from="$(qt_prefix_from_qmake)" && [ -n "$from" ]; then + QT_RESOLVED="$from"; QT_SOURCE="the qmake first on your PATH"; return + fi + QT_RESOLVED="$QT_DEFAULT"; QT_SOURCE="the overlay's default" +} + +# Rewrite the copied MODULE.bazel's paths line in place. +set_qt_prefix_in() { + local file="$1" prefix="$2" + grep -qE '^\s*paths = \{"linux-x86_64":' "$file" \ + || die "MODULE.bazel has no qt.local_repo \`paths\` line to point at your Qt. + The overlay's shape changed; --qt-prefix cannot be applied. (File left alone.)" + # | as the sed delimiter: a path contains / and must not need escaping. + sed -i "s|^\([[:space:]]*\)paths = {\"linux-x86_64\": \"[^\"]*\"|\1paths = {\"linux-x86_64\": \"$prefix\"|" "$file" +} + +# --------------------------------------------------------------------------- +overlay_files > /dev/null # populate $FILE_LIST once, for both modes + +# Resolved BEFORE the copy phase, because rule 2 reads the value in the target's +# MODULE.bazel and the copy is about to overwrite it. +QT_SOURCE="" +QT_RESOLVED="" +if [ "$VERIFY" -eq 0 ]; then + resolve_qt_prefix "$TARGET" +fi + +if [ "$VERIFY" -eq 1 ]; then + [ -d "$TARGET" ] || die "$TARGET does not exist" + cd "$TARGET" + rc=0 + + # The pin is now an ANCESTOR of HEAD, not HEAD itself: the overlay and the + # patches are commits on top of it. Asking `HEAD == pin` was right when the + # script left a detached checkout of the pin with everything uncommitted, and + # became wrong the moment the overlay became commits -- it reported MISMATCH + # for every correctly-built tree. So ask the question that is actually meant: + # is the tree BUILT ON the commit the BUILD files were generated from? + have="$(git rev-parse HEAD 2>/dev/null || echo none)" + if [ "$have" = "$LADYBIRD_COMMIT" ]; then + note "commit OK (at the pin $LADYBIRD_COMMIT)" + elif git merge-base --is-ancestor "$LADYBIRD_COMMIT" HEAD 2>/dev/null; then + extra="$(git rev-list --count "$LADYBIRD_COMMIT..HEAD")" + note "commit OK (pin $(git rev-parse --short "$LADYBIRD_COMMIT") + $extra commits on top)" + # Commits on top are the overlay itself, but they could also be upstream + # commits the reader merged -- which moves sources the BUILD files name. + # Distinguish, because only the second kind is a risk. + foreign="$(git log --format='%h %s' "$LADYBIRD_COMMIT..HEAD" \ + | grep -vE ' (Bazel overlay:|LibRequests:|Requests:)' || true)" + if [ -n "$foreign" ]; then + note " NOTE: $(echo "$foreign" | wc -l) of those are not overlay commits:" + echo "$foreign" | sed 's/^/ /' + note " if any of them move or add sources, the generated BUILD files" + note " (which name ~1,961 paths) may be stale against this tree." + fi + else + echo "MISMATCH commit: the pin $LADYBIRD_COMMIT is not an ancestor of HEAD ($have)." >&2 + echo " the generated BUILD files name sources by path; a different tree may" >&2 + echo " fail on a moved file or silently omit a new one." >&2 + rc=1 + fi + + missing=0; differ=0; same=0 + while IFS= read -r f; do + t="$(target_path "$f")" + if [ ! -e "$t" ]; then + # A file under Build/vcpkg is legitimately absent until the prefetch has + # run: staging it early is what breaks upstream's bootstrap. + case "$f" in + Build/vcpkg/*) + if [ ! -d "Build/vcpkg/.git" ]; then + note "pending (run the vcpkg prefetch, then re-apply): $t" + continue + fi ;; + esac + echo "MISSING $t" >&2; missing=$((missing + 1)) + elif ! cmp -s "$WORKSPACE/$f" "$t"; then + # MODULE.bazel's qt.local_repo `paths` line is EXPECTED to differ: it + # names the reader's Qt SDK, which is a fact about their machine and not + # part of the overlay. Reporting it as DIFFERS told people to overwrite + # their own correct configuration -- and following that advice is how a + # venv Qt 6.9.2 got replaced by a system Qt 6.4.2, below Ladybird's + # floor. So: compare with that ONE line normalised, and if the rest is + # identical, report the prefix instead of a failure. + if [ "$f" = "MODULE.bazel" ]; then + a="$(mktemp)"; b="$(mktemp)" + sed 's|^\([[:space:]]*\)paths = {"linux-x86_64": "[^"]*"|\1paths = {"linux-x86_64": "@@QT@@"|' \ + "$WORKSPACE/$f" > "$a" + sed 's|^\([[:space:]]*\)paths = {"linux-x86_64": "[^"]*"|\1paths = {"linux-x86_64": "@@QT@@"|' \ + "$t" > "$b" + if cmp -s "$a" "$b"; then + have_qt="$(qt_prefix_in_tree "$(dirname "$t")" || true)" + [ -n "$have_qt" ] || have_qt="$(sed -n 's|^[[:space:]]*paths = {"linux-x86_64": "\([^"]*\)".*|\1|p' "$t" | head -1)" + note "MODULE.bazel matches except the Qt SDK path (yours: ${have_qt:-?}) -- expected" + same=$((same + 1)) + rm -f "$a" "$b" + continue + fi + rm -f "$a" "$b" + fi + echo "DIFFERS $t" >&2; differ=$((differ + 1)) + else + same=$((same + 1)) + fi + done < "$FILE_LIST" + note "overlay files: $same identical, $differ differing, $missing missing" + [ "$missing" -eq 0 ] && [ "$differ" -eq 0 ] || rc=1 + + # The patches must be APPLIED, not merely present. `git apply --check -R` + # succeeding is the proof: a patch that reverse-applies cleanly is a patch + # already in the tree. + # + # But reverse-apply proves "MY EXACT BYTES are in the tree", which is a + # stronger claim than "the defect is fixed" -- and the difference is not + # hypothetical: upstream landed its own fix for the fd leak, so a tree that is + # CORRECT (and newer than the pin) was reported as PATCH NOT APPLIED, telling + # the user to apply a patch that would then conflict. A patch we carry only + # until upstream fixes it needs a second, weaker question: is the EFFECT there? + # `.effect-grep` next to a patch holds one extended regex per line; if every + # one matches the file the patch touches, the effect is present however it got + # there. Reverse-apply is still tried first, so the exact-bytes case keeps its + # precise answer. + # + # Asked of the SERIES first, for the reason spelled out at the apply loop + # below: 0002 rewrites 0001's context, so on a fully patched tree the + # per-patch reverse-check fails on 0001 and --verify reported PATCH NOT + # APPLIED about a correct tree. + if cat "$PATCHES"/*.patch | git apply --check -R - >/dev/null 2>&1; then + note "patch series applied (checked as a series)" + series_ok=1 + else + series_ok=0 + fi + for p in "$PATCHES"/*.patch; do + name="$(basename "$p")" + if [ "$series_ok" -eq 1 ] || git apply --check -R "$p" >/dev/null 2>&1; then + note "patch applied: $name" + continue + fi + effect="${p%.patch}.effect-grep" + if [ -f "$effect" ]; then + target="$(sed -n 's|^+++ b/||p' "$p" | head -1)" + if [ -f "$target" ]; then + # An `@window ` directive narrows the search to the n + # lines following the first match, because a whole-file grep is + # too weak to be worth anything here: `defer_teardown();` already + # occurs in stop() and did_transfer(), so an unpatched tree passed + # a whole-file check. A test pins that negative case. + win="$(grep -E '^@window ' "$effect" | head -1)" + slice="$target" + if [ -n "$win" ]; then + n="$(echo "$win" | awk '{print $2}')" + re="$(echo "$win" | cut -d' ' -f3-)" + start="$(grep -nE "$re" "$target" | head -1 | cut -d: -f1)" + if [ -z "$start" ]; then + echo "PATCH NOT APPLIED: $name (anchor not found: $re)" >&2; rc=1 + continue + fi + slice="$(mktemp)" + sed -n "${start},$((start + n))p" "$target" > "$slice" + fi + unmatched=0 + while IFS= read -r re; do + [ -n "$re" ] || continue + case "$re" in \#*|@*) continue ;; esac + grep -Eq "$re" "$slice" || unmatched=$((unmatched + 1)) + done < "$effect" + [ "$slice" = "$target" ] || rm -f "$slice" + if [ "$unmatched" -eq 0 ]; then + note "patch effect present (not our bytes -- fixed upstream?): $name" + continue + fi + fi + fi + echo "PATCH NOT APPLIED: $name" >&2; rc=1 + done + + # The four vcpkg_from_git tarballs. NOT overlay files (they are 200MB of git + # archive output, and they are .gitignored), so the file loop above cannot see + # them -- but their absence is a guaranteed build failure, which is exactly what + # --verify exists to find without a build. Ulf's tree passed --verify and then + # failed the build on this. + # + # Checked by NAME against the committed pin, not by count: a directory with + # three of the four in it is the interesting broken case, and `ls | wc -l` calls + # it fine. Hashes are not re-verified here (that is minutes of sha512 over + # ~200MB, and the fetcher already verified them at write time); --verify is the + # cheap check you run often. + archives_dir="Meta/CMake/vcpkg/git-archives" + want_archives="$(sed -n "s|^[[:space:]]*'\([^']*\.tar\.gz\)':.*|\1|p" \ + "$WORKSPACE/vcpkg_git_archives.bzl")" + if [ -z "$want_archives" ]; then + echo "MALFORMED: no archives parsed out of vcpkg_git_archives.bzl" >&2; rc=1 + else + missing_archives=0 + for a in $want_archives; do + [ -f "$archives_dir/$a" ] || { echo "MISSING $archives_dir/$a" >&2 + missing_archives=$((missing_archives + 1)); } + done + if [ "$missing_archives" -gt 0 ]; then + echo " $missing_archives of $(echo "$want_archives" | wc -w) vcpkg_from_git archives are absent." >&2 + echo " vcpkg_from_git bypasses the asset cache (it runs \`git fetch\`), so these" >&2 + echo " cannot be http_file'd; the build stages them from that directory and" >&2 + echo " fails without them. Fetch them (pure git, ~80s, verified against the pin):" >&2 + echo " cd $TARGET && python3 Meta/fetch_vcpkg_git_archives.py" >&2 + rc=1 + else + note "vcpkg_from_git archives: $(echo "$want_archives" | wc -w)/$(echo "$want_archives" | wc -w) present" + fi + fi + + # Executable bits are tree state git carries and a `cp` does not always: the + # vcpkg/cargo build scripts are run as actions, so a lost +x fails at action + # time, deep in a build, with a confusing message. + while IFS= read -r f; do + case "$f" in *.sh) [ -x "$(target_path "$f")" ] || { + echo "NOT EXECUTABLE: $(target_path "$f")" >&2; rc=1; }; ;; + esac + done < "$FILE_LIST" + + [ "$rc" -eq 0 ] && note "VERIFIED: this tree matches the overlay" \ + || echo "verification FAILED" >&2 + exit "$rc" +fi + +# --------------------------------------------------------------------------- +if [ -e "$TARGET/.git" ]; then + note "using existing clone at $TARGET" + cd "$TARGET" + git rev-parse --verify "$LADYBIRD_COMMIT^{commit}" >/dev/null 2>&1 \ + || git fetch --no-tags origin "$LADYBIRD_COMMIT" + # Where the reader IS, recorded before anything moves, so the summary at the + # end can tell them how to get back and so --onto-current has a base. + WAS_REF="$(git symbolic-ref --quiet --short HEAD || git rev-parse --short HEAD)" + note "you are on: $WAS_REF" + + # A tree that already HAS the overlay is the normal case for a REPIN. With the + # overlay committed on a branch this is no longer a dirty tree at all -- the + # previous run's work is a COMMIT, so it does not collide with a checkout, and + # this whole class of failure goes away. But a tree that has been through the + # OLD script (or that the reader has been editing) still has modified tracked + # files, and those must not be silently destroyed. + # + # STASH rather than `checkout --` / `reset --hard`: the modifications might not + # all be ours. A patch we no longer carry is indistinguishable from the + # reader's own debugging edit, and a script that silently discards the second + # kind is a script nobody should run on a tree they care about. A stash is + # recoverable and its name says who made it. + if [ -n "$(git status --porcelain --untracked-files=no)" ]; then + note "this tree has UNCOMMITTED changes to tracked files (a previous run of" + note "this script left them loose, or they are yours). Stashing -- nothing" + note "is discarded:" + git status --porcelain --untracked-files=no | sed 's/^/ /' + # No --include-untracked: the untracked files are the OVERLAY, which the + # copy below overwrites anyway, and stashing them would hide the staged + # Build/vcpkg state the prefetch ordering depends on. + git stash push --quiet \ + -m "apply_overlay.sh: tree state before $BRANCH" \ + || die "could not stash local changes; commit or stash them yourself, + then re-run. (This tree is left exactly as it was.)" + note "stashed as: $(git stash list | head -1)" + note " recover with: git -C $TARGET stash pop" + fi +else + note "cloning Ladybird (full history: vcpkg needs it, see README)" + git clone "$LADYBIRD_REPO" "$TARGET" + cd "$TARGET" + WAS_REF="$(git symbolic-ref --quiet --short HEAD || git rev-parse --short HEAD)" +fi + +# --------------------------------------------------------------------------- +# Get onto a BRANCH. Never a detached HEAD: see the header. Three cases, and each +# one ends with HEAD being a symbolic ref to $BRANCH. +if [ "$COMMIT" -eq 0 ]; then + # --no-commit keeps the old shape for a throwaway tree. Still not detached if + # we can avoid it: if the pin is already what HEAD resolves to, stay put. + note "--no-commit: mutating the working tree, committing nothing" + if [ "$(git rev-parse HEAD)" != "$LADYBIRD_COMMIT" ]; then + git checkout --detach "$LADYBIRD_COMMIT" + note "NOTE: this leaves a DETACHED HEAD (that is what --no-commit means)." + fi +elif [ "$ONTO_CURRENT" -eq 1 ]; then + # Apply the overlay on top of the reader's own work. Their HEAD is the base. + base="$(git rev-parse HEAD)" + if [ "$base" != "$LADYBIRD_COMMIT" ]; then + note "--onto-current: basing the overlay on YOUR HEAD ($(git rev-parse --short HEAD))," + note " not on the pinned commit $LADYBIRD_COMMIT." + note " WARNING: the generated BUILD files name ~1,961 sources by path and were" + note " generated from the pin. Against another tree the build can fail on a" + note " moved file or silently omit a new one. That is the trade you just made." + fi + if [ "$(git symbolic-ref --quiet --short HEAD || true)" = "$BRANCH" ]; then + note "already on $BRANCH" + else + git checkout -b "$BRANCH" 2>/dev/null \ + || die "branch '$BRANCH' already exists and is not what you are on. + Pick another with --branch NAME, or check it out yourself first." + note "created branch $BRANCH at $(git rev-parse --short HEAD)" + fi +else + # The default: a branch named after the pin, based on the pin. + # + # A re-run must be idempotent rather than an error, because "run it again" is + # what everyone does. If the branch exists we RESET it to the pin and rebuild + # the commits -- but only after checking it holds nothing but ours, since a + # reset would otherwise discard the reader's commits on that branch. + if git rev-parse --verify --quiet "refs/heads/$BRANCH" >/dev/null; then + git checkout --quiet "$BRANCH" + foreign="$(git log --format='%H %s' "$LADYBIRD_COMMIT..HEAD" 2>/dev/null \ + | grep -vE ' (Bazel overlay:|LibRequests:|Requests:)' || true)" + if [ -n "$foreign" ]; then + die "branch '$BRANCH' has commits that are not this overlay's: + +$(echo "$foreign" | sed 's/^/ /') + + I will not reset a branch holding your work. Either build the overlay on a + fresh branch: + $(basename "$0") --branch $BRANCH-new $TARGET + or put the overlay on TOP of those commits: + $(basename "$0") --onto-current $TARGET" + fi + note "re-running on existing branch $BRANCH: resetting to the pin to rebuild" + note " its commits (only overlay commits were on it; yours would have stopped this)" + git reset --quiet --hard "$LADYBIRD_COMMIT" + else + git checkout --quiet -b "$BRANCH" "$LADYBIRD_COMMIT" + note "created branch $BRANCH at the pin $LADYBIRD_COMMIT" + fi +fi +note "at $(git rev-parse HEAD)$([ "$COMMIT" -eq 1 ] && echo " (on $(git symbolic-ref --quiet --short HEAD || echo 'DETACHED'))")" + +note "applying $(ls "$PATCHES"/*.patch | wc -l) patches (all reported upstream)" +# patches/*.patch is a SERIES, applied in glob (numeric) order: a later patch may +# depend on an earlier one having landed. 0004 edits lines adjacent to 0003's inside +# the same branch, so it is generated against 0003 and only applies after it. +# +# That makes the glob load-bearing in a way it was not before. I briefly shipped two +# mutually-exclusive variants of 0004 -- one for a tree with the teardown fix, one for +# a tree without -- and since this loop applies EVERY *.patch, one of them could only +# ever fail. Ulf hit it immediately: "tries to apply both patches at the same time". +# Alternatives do not belong in this directory; a variant that is not part of the +# series goes beside DIAGNOSTIC-*.patch.txt, outside the glob. The series-applies-as-a- +# series property is now pinned by a test, so a future patch that silently conflicts +# with its predecessor fails in CI rather than on someone's clone. +# +# Each patch becomes its OWN COMMIT, keeping its own Subject: line, so `git log` +# reads as the series it is and any one of them can be reverted, rebased or +# cherry-picked on its own. `git apply` + `git commit` rather than `git am`: these +# are not mailbox files (no `From ` envelope line), and am on a non-mbox fails in +# a way that would send the reader looking for a mail bug. +# +# "Is the series already applied?" must be asked of the SERIES, not of each patch +# on its own -- and this is a bug the old script had too, found by running +# --onto-current against a tree that already had the overlay committed: +# +# error: patch failed: Libraries/LibRequests/Request.cpp:327 +# error: failed to apply 0001-...patch +# +# 0002 edits lines ADJACENT to 0001's inside the same function, so on a fully +# patched tree `git apply --check -R 0001` fails: 0001's context lines no longer +# exist, 0002 having rewritten them. The per-patch test therefore says "not +# applied" about a patch that IS applied, and the script tries to apply it again +# and dies. Reverse-checking the concatenation asks the right question -- is the +# whole series present -- and is verified both ways: OK on the patched tree, +# correctly failing on the pin. +if cat "$PATCHES"/*.patch | git apply --check -R - >/dev/null 2>&1; then + note " the whole series is already applied; nothing to do" + PATCH_LIST="" +else + PATCH_LIST="$(ls "$PATCHES"/*.patch)" +fi +for p in $PATCH_LIST; do + name="$(basename "$p")" + if git apply --check -R "$p" >/dev/null 2>&1; then + note " already applied: $name" + continue + fi + git apply "$p" || die "failed to apply $name -- patches/ is a + series applied in order; a patch here must apply on top of the ones before it. + If this is an ALTERNATIVE to another patch rather than an addition, it must not + live in patches/*.patch, which is globbed and applied in full." + note " applied: $name" + if [ "$COMMIT" -eq 1 ]; then + # The patch's own Subject:, minus the [PATCH] prefix. Falls back to the + # filename so a patch without a Subject still gets a legible commit. + subject="$(sed -n 's/^Subject: \(\[PATCH[^]]*\] \)\?//p' "$p" | head -1)" + [ -n "$subject" ] || subject="apply ${name%.patch}" + # Only the files this patch touched: never `add -A`, which would sweep in + # the reader's untracked files and the overlay that has not been copied yet. + sed -n 's|^+++ b/||p' "$p" | sort -u | while IFS= read -r f; do + [ -e "$f" ] && git add -- "$f" + done + git -c user.email="$(git config user.email || echo overlay@any2bazel)" \ + -c user.name="$(git config user.name || echo 'any2bazel overlay')" \ + commit --quiet --no-verify -m "$subject" \ + -m "From examples/ladybird/patches/$name in the any2bazel Ladybird +migration overlay. Reported upstream; see that file's header for the analysis." \ + || die "could not commit $name" + note " committed: $(git log -1 --format=%h) $subject" + fi +done + +# Phase 1: everything except the files under Build/vcpkg. Staging those creates the +# directory upstream's bootstrap reads as "already cloned" -- see the ORDER note at +# the top of this file. +note "copying the overlay" +n=0 +deferred=0 +while IFS= read -r f; do + case "$f" in Build/vcpkg/*) deferred=$((deferred + 1)); continue ;; esac + t="$(target_path "$f")" + mkdir -p "$(dirname "$t")" + cp -p "$WORKSPACE/$f" "$t" + n=$((n + 1)) +done < "$FILE_LIST" +note "$n files copied (bazelrc.txt -> .bazelrc), $deferred deferred until vcpkg exists" + +# The Qt prefix, restored/set AFTER the copy overwrote MODULE.bazel. Resolved +# before the copy so rule 2 can read the value the copy is about to destroy. +if [ -n "$QT_RESOLVED" ]; then + set_qt_prefix_in "$TARGET/MODULE.bazel" "$QT_RESOLVED" + if [ "$QT_RESOLVED" = "$QT_DEFAULT" ]; then + note "Qt SDK: $QT_RESOLVED (from $QT_SOURCE)" + else + note "Qt SDK: $QT_RESOLVED (from $QT_SOURCE) -- NOT the overlay's" + note " $QT_DEFAULT default; your choice was preserved, not overwritten." + fi + if v="$(qt_version_at "$QT_RESOLVED")" && [ -n "$v" ]; then + note " that Qt reports version $v" + case "$v" in + 6.[0-8].*|[0-5].*) + note " WARNING: Ladybird requires Qt >= 6.9 (UI/Qt/CMakeLists.txt)." + note " The build will stop in qt_runtime.bzl naming this prefix." ;; + esac + else + note " NOTE: no qmake under $QT_RESOLVED/bin -- if that is wrong, pass" + note " --qt-prefix DIR (rules_qt runs \`qmake -query\` there)." + fi +fi + +# ...and committed, so they are tracked files in a commit rather than 45 lines of +# untracked noise in `git status`. -f is needed for .bazelrc only if a future +# upstream .gitignore covers it; the deferred Build/vcpkg file is NOT committed at +# all (Ladybird ignores Build*/ and being ignored it makes no noise anyway). +if [ "$COMMIT" -eq 1 ]; then + while IFS= read -r f; do + case "$f" in Build/vcpkg/*) continue ;; esac + git add --force -- "$(target_path "$f")" + done < "$FILE_LIST" + if [ -n "$(git diff --cached --name-only)" ]; then + git -c user.email="$(git config user.email || echo overlay@any2bazel)" \ + -c user.name="$(git config user.name || echo 'any2bazel overlay')" \ + commit --quiet --no-verify \ + -m "Bazel overlay: build Ladybird with Bazel alongside CMake" \ + -m "Generated BUILD.bazel/*.bzl files plus their emitters, from the +any2bazel Ladybird migration at upstream commit $LADYBIRD_COMMIT. +Adds no CMake changes: the two build systems sit side by side. + +Build/vcpkg/BUILD.bazel is deliberately NOT in this commit -- Ladybird's +.gitignore covers Build*/, and it must not exist before the vcpkg +prefetch runs (it makes upstream's bootstrap read the bare directory as +an existing checkout)." \ + || die "could not commit the overlay" + note "committed the overlay: $(git log -1 --format=%h)" + else + note "overlay already committed and unchanged" + fi +fi + +# Phase 2: the vcpkg prefetch, then the deferred files. +# +# Deferring the copy is not sufficient on a REPIN. The trap at the top of this +# file is that the DIRECTORY Build/vcpkg existing (with no .git in it) makes +# upstream's bootstrap skip the clone -- and on a tree that already has an older +# overlay, Build/vcpkg/BUILD.bazel is ALREADY THERE, put there by the previous +# run. So phase 1 not creating it changes nothing, and the prefetch dies exactly +# as documented: +# +# fatal: unable to read tree (40f3c709...) +# subprocess.CalledProcessError: ['git','checkout','40f3c709...'] status 128 +# +# (it walks up to Ladybird's repo, gets Ladybird's HEAD, and tries to check +# vcpkg's baseline out of it). Found by running the repin against a replica of +# Ulf's tree; the deferral logic had only ever been tested on a fresh clone. +# +# So: if there is no .git, the directory is not a checkout, and anything in it is +# ours to move out of the way. Only the overlay's own deferred files are removed +# -- never a directory with a .git, and never anything the overlay does not own. +if [ ! -d "$TARGET/Build/vcpkg/.git" ] && [ -d "$TARGET/Build/vcpkg" ]; then + while IFS= read -r f; do + case "$f" in Build/vcpkg/*) ;; *) continue ;; esac + if [ -e "$TARGET/$f" ]; then + note "un-staging $f so the vcpkg bootstrap sees no checkout" + rm -f "$TARGET/$f" + fi + done < "$FILE_LIST" + # An empty Build/vcpkg is just as fatal as one with a file in it: the + # bootstrap tests for the DIRECTORY. rmdir, not rm -rf: if anything else is + # in there it is not ours and the failure should be loud. + rmdir "$TARGET/Build/vcpkg" 2>/dev/null && note "removed the empty Build/vcpkg" || true +fi + +if [ -d "$TARGET/Build/vcpkg/.git" ]; then + note "vcpkg checkout already present" +elif [ "$PREFETCH" -eq 1 ]; then + note "running the vcpkg prefetch (upstream's own bootstrap, ~70s, no CMake)" + python3 Meta/ladybird.py vcpkg +else + note "SKIPPING the vcpkg prefetch (--no-prefetch)" +fi + +if [ -d "$TARGET/Build/vcpkg/.git" ]; then + while IFS= read -r f; do + case "$f" in Build/vcpkg/*) ;; *) continue ;; esac + mkdir -p "$(dirname "$f")" + cp -p "$WORKSPACE/$f" "$f" + note "staged $f" + done < "$FILE_LIST" +else + cat >&2 <<'WARN' +==> NOT staged: Build/vcpkg/BUILD.bazel + Stage it only AFTER the vcpkg checkout exists, or upstream's bootstrap reads + the bare directory as an existing clone and fails with + `fatal: unable to read tree`. Do: + python3 Meta/ladybird.py vcpkg # creates Build/vcpkg + its .git + then re-run this script to stage the deferred file. +WARN +fi + +# Phase 3: the SECOND prefetch -- the four vcpkg_from_git tarballs. +# +# This has to run here, not in the closing message, because there is no build +# without it. `vcpkg_from_git` shells out to `git fetch`, which no asset source +# intercepts, so these four cannot be http_file'd like the other 76 distfiles; but +# vcpkg DOES honour a pre-placed downloads/-.tar.gz, which is what they +# become. Meta/vcpkg_build.sh hard-fails in four seconds when they are absent (it +# used to fail ~20 minutes in, inside skia's portfile, naming a googlesource URL) -- +# and that fast, clear failure was still a failure Ulf hit on a tree this script +# had just reported as done. The instruction was in the closing message. He, quite +# reasonably, ran `bazel build`. +# +# It needs the vcpkg checkout (it resolves each clone URL out of the portfiles), so +# it must come after phase 2 -- that ordering is the reason it was a message in the +# first place, and ordering is a thing a script can express. +if [ -d "$TARGET/Build/vcpkg/.git" ] && [ "$PREFETCH" -eq 1 ]; then + note "fetching the 4 vcpkg_from_git tarballs (~80s, verified against the pin)" + python3 Meta/fetch_vcpkg_git_archives.py \ + || die "the git-archive prefetch failed. It is pure git (clone + git archive, + checked against the SHA512s in vcpkg_git_archives.bzl), so this is a network + or a pin problem, not a build problem. Re-run just this step with: + cd $TARGET && python3 Meta/fetch_vcpkg_git_archives.py" +elif [ "$PREFETCH" -eq 0 ]; then + note "SKIPPING the git-archive prefetch (--no-prefetch)" + note " \`bazel build\` will fail without it: python3 Meta/fetch_vcpkg_git_archives.py" +fi + +cat < done. Both prefetches have run; the next command is the build: + + cd $TARGET + bazel build //:ladybird + + Re-check this tree at any time with: + $(basename "$0") --verify $TARGET +EOF + +if [ "$COMMIT" -eq 1 ] && git symbolic-ref -q HEAD >/dev/null; then + cat < the overlay is $(git rev-list --count "$LADYBIRD_COMMIT..HEAD" 2>/dev/null || echo '?') commits on branch '$(git symbolic-ref --short HEAD)': + +$(git log --oneline "$LADYBIRD_COMMIT..HEAD" 2>/dev/null | sed 's/^/ /') + + \`git status\` is clean. To put these on top of your own work instead: + git -C $TARGET rebase $(git symbolic-ref --short HEAD) $WAS_REF + or take just the fixes and none of the Bazel files: + git -C $TARGET cherry-pick + To go back to where you were: + git -C $TARGET checkout $WAS_REF +EOF +fi diff --git a/examples/ladybird/fd_census.py b/examples/ladybird/fd_census.py new file mode 100755 index 0000000..962e89c --- /dev/null +++ b/examples/ladybird/fd_census.py @@ -0,0 +1,780 @@ +#!/usr/bin/env python3 +"""Classify a process's leaking fds from OUTSIDE it: no patch, no rebuild, no pin. + +This is the fd-leak instrument, reachable from any tree. The earlier version was a +patch to `Libraries/LibRequests/Request.cpp`, which is the wrong delivery mechanism +for anyone who has their own commits: it assumes a tree at our pinned commit, it +conflicts with a tree that already carries the upstream fix, and it makes "run my +diagnostic" mean "reset your tree". Everything that build computed is visible from +outside, so this computes it from /proc and ss against a RUNNING process. + + python3 fd_census.py # one census + python3 fd_census.py --watch 30 # every 30s until interrupted + python3 fd_census.py --find WebContent # locate the pid yourself-free + python3 fd_census.py --all --watch 30 # rank EVERY browser process by growth + python3 fd_census.py --build # which fd fixes are IN this binary + +What it reports, and why each column exists: + + total / socket / pipe the census that separated the two leaks in the + first place: MessagePort leaks 4 pipes per socket, + so a sockets-only census falsifies it outright. + + peer DEAD vs ALIVE the discriminator between the two per-request + leaks. RequestServer closes its half of the + response socketpair when the request completes, so + `ss` shows peer inode 0: + DEAD -> the request COMPLETED and WebContent is + retaining a corpse. Fixed by the + teardown patch / upstream's equivalent. + ALIVE -> the peer still holds it: on_finish never + ran, so no teardown in that branch can + fire. Needs the GC-root/RefPtr cycle + broken at the Response end instead. + + retained vs in-flight (by age) an fd younger than the threshold is not a leak, it + is a request. Counting fds without ages is how a + freshly restarted process once looked like a fix. + Ages need >= 2 samples, so --watch earns them. + + peer process who holds the other end, when it is alive. Names + the producer instead of guessing. + + build WHICH FIX the running code contains, read out of the + process's own mapped binaries. A leak rate is + uninterpretable without it: "still leaking at + 92/min" means "the fix does not work" or "the fix + was not in this build", and those need opposite next + steps. Reports "cannot tell" rather than guessing. + +Exit status is 0 for a census, 1 if it could not read the process. +""" + +import argparse +import os +import re +import struct +import subprocess +import sys +import time + +SOCKET_RE = re.compile(r"^socket:\[(\d+)\]$") + +# A peer process holding at most this many of our sockets is the ordinary IPC mesh +# (each pair of browser processes keeps a couple of long-lived connections), not a +# producer sitting on unfinished responses. Ulf's healthy WebContent showed exactly +# 2 each to ladybird, Compositor, RequestServer and ImageDecoder. +IPC_MESH_MAX = 4 + + +def read_fd_targets(pid): + """{fd: symlink target} for a live pid. Racy by nature; skip what vanishes.""" + targets = {} + fd_dir = "/proc/%s/fd" % pid + for name in os.listdir(fd_dir): + try: + targets[int(name)] = os.readlink(os.path.join(fd_dir, name)) + except (OSError, ValueError): + continue # closed under us, or not a number + return targets + + +# The symbols that say which fix a RUNNING browser was built with. Read out of the +# binaries the process has mapped, so the census answers "was the patch in it?" +# instead of asking the person running it. +# +# WHY THIS EXISTS. Ulf reported ~92 leaked fds/min from a WebContent I could not tell +# had my patch in it, and the rate was close enough to the pre-patch 97/min to be +# consistent with EITHER "the patch is not in that binary" or "the patch is +# irrelevant to this leak". Those two demand opposite next moves, and I could not +# separate them, so the next step was going to be a question -- another round trip, +# answered from memory, about a build that had already happened. But the answer is +# not in anyone's memory: it is in the binary, and the binary is still mapped by the +# process being censused. A measurement that reports a leak rate without reporting +# WHICH CODE was running is not reproducible by anyone, including me. +# +# Each entry: symbol -> (what its presence means, what its absence means). +FIX_SYMBOLS = ( + # 0004: closes the response fd where the body is proven complete. + ("release_response_fd", "0004 (release the response fd on completion)"), + # 0003 / upstream's equivalent: drops the callbacks so the Request is collectable. + ("defer_teardown", "0003 (tear down the request when the body is delivered)"), +) + +# Present in EVERY build of this library, patched or not. Without it, a "symbol not +# found" result means the symbols were not readable at all (stripped, LTO-inlined, +# statically linked into a binary we did not look at) -- NOT that the fix is missing. +# Reporting "fix absent" for an unreadable binary would be a false negative that +# sends the next round of work in exactly the wrong direction, which is the error +# this whole probe exists to prevent. +CONTROL_SYMBOL = "set_up_internal_stream_data" + +# ...and the control ALONE IS NOT ENOUGH, which is the correction that matters here. +# +# Ulf said "I have all the patches applied" and this probe said 0004 was absent. He was +# right. The reason is a real flaw in the first version, not a mistake of his: +# Ladybird sets ENABLE_LTO_FOR_RELEASE=ON, and in a STATIC build (his) LTO inlines a +# small internal-only method like release_response_fd() into its single caller and +# leaves NO symbol and NO string behind at all. Reproduced from first principles: a +# private method called only within its TU, linked -O3 -flto, disappears from both +# `nm` and `strings` in the linked executable -- while my own build keeps it, because +# a SHARED library must export it. So the probe's answer depended on how the binary +# was linked, not on whether the fix was in it. +# +# And the control did not catch it, because the control is vulnerable to exactly the +# same optimisation: verified that a larger internal-only function also vanishes under +# LTO. A control only rules out "unreadable" if it CANNOT disappear for the same +# reason as the thing it guards. That is the general trap: a negative control has to +# fail in the failure mode you are guarding against, and mine failed in a different +# one. +# +# Two changes follow. (1) .debug_str is read as well as the symbol tables: debug info +# names inlined-away functions, and Ladybird's RelWithDebInfo compiles with -g, so the +# answer survives there for precisely the LTO/static case. (2) A symbol that is merely +# ABSENT is never reported as a missing fix unless something PROVES the binary is +# readable at a level inlining cannot erase -- see UNINLINABLE_CONTROLS. + +# Symbols that cannot be inlined away, so their presence proves the binary's names are +# readable: virtual/IPC-dispatched entry points reached through a vtable or across a +# library boundary. If NONE of these is visible, this binary cannot answer the +# question, whatever else is or is not in it. +UNINLINABLE_CONTROLS = ( + # Reached via the IPC endpoint dispatch table, not a direct call. + "request_started", + "headers_became_available", + # Request's own out-of-line, externally-called entry points. + "did_receive_headers", + "set_unbuffered_request_callbacks", +) + +# The Request code lives here; when Ladybird is built statically it is in the +# executable instead, so the executable is always probed too. +FIX_LIBRARY_HINT = "lagom-requests" + + +def elf_symbol_names(path): + """Every symbol name in an ELF file's string tables, or None if unreadable. + + Pure Python on purpose: this runs on someone else's machine, where `nm` and + `readelf` are a binutils install I should not require to answer a question the + file already contains. Reads .dynstr/.strtab wholesale rather than walking symbol + tables -- the question is only ever "does this name appear", and substring + matching a string table cannot report a name that is not in the file. + """ + try: + with open(path, "rb") as f: + header = f.read(64) + if len(header) < 64 or header[:4] != b"\x7fELF": + return None + if header[4] != 2: # not ELF64; nothing here is 32-bit + return None + shoff, = struct.unpack_from("= shnum: + return None + f.seek(shoff) + table = f.read(shentsize * shnum) + if len(table) < shentsize * shnum: + return None + + def entry(i): + off = i * shentsize + name_off, = struct.unpack_from("= 0 else None] + # .debug_str matters as much as the symbol tables: under LTO in a + # STATIC build, a small internal-only method like + # release_response_fd is inlined into its only caller and leaves NO + # symbol and NO string behind -- but the debug info still names it. + # Ladybird's RelWithDebInfo compiles with -g, so this is where the + # answer survives for exactly the build that prompted the question. + if name in (b".dynstr", b".strtab", b".debug_str") \ + and sec_size < 256 * 1024 * 1024: + f.seek(sec_off) + blob += f.read(sec_size) + return blob or None + except (OSError, struct.error, ValueError): + return None + + +def mapped_binaries(pid): + """The executable plus every mapped .so, as real paths we can open.""" + paths = [] + exe = "/proc/%d/exe" % pid + try: + paths.append(os.path.realpath(exe)) + except OSError: + pass + try: + with open("/proc/%d/maps" % pid) as f: + for line in f: + parts = line.split(None, 5) + if len(parts) < 6: + continue + path = parts[5].strip() + if path.startswith("/") and path not in paths: + paths.append(path) + except OSError: + pass + return paths + + +def probe_fixes(pid): + """Which fd fixes are compiled into the code this pid is RUNNING. + + Returns (findings, note). findings maps a description to True/False; note is set + when the answer is "cannot tell", so a caller never prints a missing fix it did + not actually establish is missing. + """ + candidates = [p for p in mapped_binaries(pid) if FIX_LIBRARY_HINT in p] + # Statically linked builds (Ulf's is one) have no such library: the code is in + # the executable. Probe it rather than reporting nothing. + if not candidates: + candidates = mapped_binaries(pid)[:1] + + blob = b"" + for path in candidates: + names = elf_symbol_names(path) + if names: + blob += names + if not blob: + return {}, ("could not read symbols from this process's binaries " + "(no readable ELF among %d mapped paths)" % len(candidates)) + if CONTROL_SYMBOL.encode() not in blob: + return {}, ("symbols unreadable: the control symbol %s is absent too, so a " + "missing fix here would prove nothing (stripped binary, LTO, or " + "the code is in a binary not probed)" % CONTROL_SYMBOL) + + findings = {desc: (sym.encode() in blob) for sym, desc in FIX_SYMBOLS} + if all(findings.values()): + return findings, None + + # Something looks missing. Before saying so, establish that absence MEANS + # anything here: an internal-only method inlined by LTO leaves no trace, so + # "not found" is only evidence if names that CANNOT be inlined away are visible. + # Without this check the probe told Ulf 0004 was absent from a binary that had it. + readable = [c for c in UNINLINABLE_CONTROLS if c.encode() in blob] + if not readable: + missing = [d for d, present in findings.items() if not present] + return {}, ("cannot tell whether %s is present: this binary's names are not " + "readable at a level inlining cannot erase (no uninlinable " + "control symbol found). Under LTO in a static build a small " + "internal-only method leaves no symbol AND no string, so its " + "absence here is not evidence. Check the source or build with " + "-g/shared libs." % "; ".join(missing)) + return findings, None + + +def fix_lines(pid): + """The build-provenance block: what code is actually running.""" + findings, note = probe_fixes(pid) + if note: + return ["build: %s" % note] + lines = [] + for desc, present in findings.items(): + lines.append("build: %s %s" % ("HAS" if present else "does NOT have", desc)) + if not all(findings.values()): + lines.append(" -> a leak measured on this binary does not test the missing " + "fix. Rebuild with it before concluding the fix does not work.") + return lines + + +def categorize(target): + """The category names are the ones the /proc census prints, deliberately.""" + if target.startswith("socket:"): + return "socket:" + if target.startswith("pipe:"): + return "pipe:" + if target.startswith("anon_inode:"): + return "anon_inode:" + if target.startswith("/memfd:"): + return "memfd:" + if target.startswith("/"): + return "file" + return "other" + + +def socket_inode(target): + m = SOCKET_RE.match(target) + return int(m.group(1)) if m else None + + +def parse_ss(text): + """Map socket inode -> (peer_inode, [holder names]) from `ss -np` output. + + ss lays out unix sockets as `... * users:(...)`, and + a peer inode of 0 means the far end is closed -- which is the whole point of + running ss rather than just counting /proc entries. + """ + sockets = {} + for line in text.splitlines(): + if "users:(" not in line: + continue + head, users = line.split("users:(", 1) + fields = head.split() + inodes = [f for f in fields if f.isdigit()] + if len(inodes) < 2: + continue + local_inode, peer_inode = int(inodes[-2]), int(inodes[-1]) + holders = re.findall(r'\("([^"]+)",pid=(\d+),fd=(\d+)\)', users) + # Recv-Q: bytes sitting UNREAD in this socket. It is the third column of + # `ss` output and it discriminates between the two mechanisms that both + # present as "retained, peer=DEAD", which nothing else in this census can: + # Recv-Q > 0 -> the body was never drained. The consumer stopped reading + # (e.g. delivery paused and never resumed), so the + # completion branch that closes the fd was never reached. + # Recv-Q == 0 -> the body WAS fully read and the fd is merely still owned. + # That is the ownership bug, not a delivery bug. + # Same field, opposite fixes, so it is worth one column. + recv_q = None + if len(fields) >= 3 and fields[2].isdigit(): + recv_q = int(fields[2]) + sockets[local_inode] = (peer_inode, holders, recv_q) + return sockets + + +def ss_sockets(): + try: + out = subprocess.run(["ss", "-np"], capture_output=True, text=True, + timeout=30).stdout + except (OSError, subprocess.SubprocessError): + return {} + return parse_ss(out) + + +def peer_state(inode, sockets): + """DEAD / ALIVE / unknown for one socket inode.""" + entry = sockets.get(inode) + if entry is None: + return "unknown", [] + peer_inode, holders = entry[0], entry[1] + if peer_inode == 0: + return "DEAD", holders + return "ALIVE", holders + + +def recv_queue(inode, sockets): + """Unread bytes in one socket, or None if unknown.""" + entry = sockets.get(inode) + if entry is None or len(entry) < 3: + return None + return entry[2] + + +BROWSER_PROCESS_NAMES = ("Ladybird", "ladybird", "WebContent", "RequestServer", + "ImageDecoder", "Compositor", "WebWorker") + + +def find_browser_pids(): + """Every Ladybird-family process, whatever it is called. + + Deliberately NOT just WebContent. Every census I asked Ulf for was of WebContent, + because that is where I had already decided the leak was -- so a leak in + RequestServer, the Compositor or the UI process would have been invisible to all + of them, and "still leaking" is consistent with that. An instrument should not + inherit my hypothesis: watch every process and let the growth say which one. + """ + out = [] + for entry in os.listdir("/proc"): + if not entry.isdigit(): + continue + try: + with open("/proc/%s/comm" % entry) as f: + comm = f.read().strip() + except OSError: + continue + if any(n in comm for n in BROWSER_PROCESS_NAMES): + out.append((int(entry), comm)) + return sorted(out) + + +def find_pids(name): + out = [] + for entry in os.listdir("/proc"): + if not entry.isdigit(): + continue + try: + with open("/proc/%s/comm" % entry) as f: + comm = f.read().strip() + except OSError: + continue + if name in comm: + try: + count = len(os.listdir("/proc/%s/fd" % entry)) + except OSError: + count = 0 + out.append((count, int(entry), comm)) + return sorted(out, reverse=True) + + +class Census: + """Samples a pid over time so fds can be aged. + + The ages are the reason this is a class and not a function: "retained" only + means anything relative to a first-seen time, and the first sample cannot know + one. Keyed by (fd, target) so fd reuse does not inherit an age. + """ + + def __init__(self, pid, retained_after=30.0): + self.pid = pid + self.retained_after = retained_after + self.first_seen = {} + # (timestamp, socket count, dead count) per sample. The RATE is the number + # that actually matters -- "is it still leaking, and how fast" -- and a + # single census cannot answer it. Reporting a raw count invites the mistake + # I already made once: reading a freshly restarted process as a fix. + self.history = [] + + def sample(self, now=None): + now = time.time() if now is None else now + targets = read_fd_targets(self.pid) + sockets = ss_sockets() + + live_keys = set() + by_category = {} + rows = [] + for fd, target in sorted(targets.items()): + key = (fd, target) + live_keys.add(key) + self.first_seen.setdefault(key, now) + age = now - self.first_seen[key] + + category = categorize(target) + by_category[category] = by_category.get(category, 0) + 1 + + inode = socket_inode(target) + state, holders = ("n/a", []) + rq = None + if inode is not None: + state, holders = peer_state(inode, sockets) + rq = recv_queue(inode, sockets) + rows.append({ + "fd": fd, "target": target, "category": category, "age": age, + "peer": state, "recv_q": rq, + "peers": [h[0] for h in holders if int(h[1]) != int(self.pid)], + }) + + for key in list(self.first_seen): + if key not in live_keys: + del self.first_seen[key] + + socks = [r for r in rows if r["category"] == "socket:"] + self.history.append((now, len(socks), + len([r for r in socks if r["peer"] == "DEAD"]))) + return {"rows": rows, "by_category": by_category, "now": now} + + def summarize(self, snapshot): + rows = snapshot["rows"] + out = [] + total = len(rows) + out.append("fds: total=%d %s" % ( + total, " ".join("%s=%d" % (k, v) for k, v in + sorted(snapshot["by_category"].items(), + key=lambda kv: -kv[1])))) + + socks = [r for r in rows if r["category"] == "socket:"] + dead = [r for r in socks if r["peer"] == "DEAD"] + alive = [r for r in socks if r["peer"] == "ALIVE"] + unknown = [r for r in socks if r["peer"] == "unknown"] + out.append("sockets: %d peer DEAD=%d peer ALIVE=%d unknown=%d" % ( + len(socks), len(dead), len(alive), len(unknown))) + + retained = [r for r in socks if r["age"] >= self.retained_after] + in_flight = [r for r in socks if r["age"] < self.retained_after] + aged = any(r["age"] > 0 for r in socks) + if aged: + out.append("sockets by age: in_flight(<%gs)=%d retained(>=%gs)=%d" % ( + self.retained_after, len(in_flight), + self.retained_after, len(retained))) + rd = len([r for r in retained if r["peer"] == "DEAD"]) + ra = len([r for r in retained if r["peer"] == "ALIVE"]) + out.append(" of retained: peer DEAD=%d peer ALIVE=%d" % (rd, ra)) + out.extend(self.recv_q_lines(retained)) + else: + # Every fd was first seen in THIS sample. With --watch that means the + # process was just attached to, not that the fds are new -- ages are + # relative to when the census started, and it cannot see further back. + out.append("sockets by age: all %d first seen this sample (ages start " + "now; the next sample will age them)" % len(socks)) + + out.extend(self.rate_lines()) + + # A live peer is NOT automatically a leak: every process in the browser is + # connected to its siblings by long-lived IPC sockets, so a healthy + # WebContent shows ~2 each to ladybird/Compositor/RequestServer/ + # ImageDecoder. Reporting those next to the leak counts invites reading the + # IPC mesh as evidence. Separate the two: N-of-a-kind at the baseline level + # is plumbing; a peer holding MANY is a producer that has not let go. + holders = {} + for r in alive: + for name in r["peers"]: + holders[name] = holders.get(name, 0) + 1 + if holders: + plumbing = {k: v for k, v in holders.items() if v <= IPC_MESH_MAX} + suspect = {k: v for k, v in holders.items() if v > IPC_MESH_MAX} + if suspect: + out.append("live peers RETAINING many: %s" % ", ".join( + "%s x%d" % (k, v) for k, v in + sorted(suspect.items(), key=lambda kv: -kv[1]))) + if plumbing: + out.append("live peers (normal IPC mesh, not the leak): %s" % + ", ".join("%s x%d" % (k, v) for k, v in + sorted(plumbing.items(), key=lambda kv: -kv[1]))) + + out.append(verdict(len(dead), len(alive))) + # LAST, next to the verdict, because the verdict is only interpretable + # together with which code produced it: "still leaking at 92/min" means + # "the fix does not work" or "the fix was not in the binary" depending on + # this line alone, and the two need opposite next steps. + out.extend(fix_lines(self.pid)) + return "\n".join(out) + + + def recv_q_lines(self, retained): + """Split the retained peer=DEAD fds by whether their body was ever drained. + + THE MEASUREMENT THAT DISCRIMINATES. With the completion-path fix (0004) + confirmed present in a binary that still leaks ~92/min of retained, + peer=DEAD sockets, the fd has an owner that fix does not reach -- and there + are two candidate mechanisms that look IDENTICAL in every column above: + + unread > 0 -> the body was never drained. Something stopped reading (a + navigation whose body delivery was paused and never + resumed), so the completion branch that closes the fd was + never reached. Fix: resume-or-close on every abandoned path. + unread == 0 -> the body WAS fully read; the descriptor is merely still + owned. Fix: drop the surviving reference (e.g. the RefPtr in + Response::m_request_server_request, which clone() copies). + + Same signature, opposite fixes. Guessing between them is what burned the last + several rounds, so this reads the answer off the socket instead. + """ + dead = [r for r in retained if r["peer"] == "DEAD"] + known = [r for r in dead if r["recv_q"] is not None] + if not known: + return [] + unread = [r for r in known if r["recv_q"] > 0] + drained = [r for r in known if r["recv_q"] == 0] + lines = [" of retained peer=DEAD: body UNREAD=%d body drained=%d" + % (len(unread), len(drained))] + if unread and len(unread) >= 5 * max(len(drained), 1): + total = sum(r["recv_q"] for r in unread) + lines.append(" -> bodies were NEVER DRAINED (%d bytes still queued). The " + "consumer stopped reading, so the completion path that " + "closes the fd was never reached: look at body delivery " + "being paused and not resumed, NOT at fd ownership." % total) + elif drained and len(drained) >= 5 * max(len(unread), 1): + lines.append(" -> bodies were FULLY READ, so delivery completed and the " + "fd is merely still OWNED. Look for a surviving reference to " + "the Requests::Request (Response::m_request_server_request is " + "a RefPtr, and clone() copies it), NOT at delivery.") + elif unread and drained: + lines.append(" -> MIXED: both mechanisms are present; they need separate " + "fixes and a re-census between them.") + return lines + + def rate_lines(self): + """Growth since the first sample, as a rate. The decisive measurement. + + Whether a fix works is a question about the SLOPE, not the level: the level + includes everything leaked before the census started, so a fixed browser + with 1500 already-leaked fds looks identical to a broken one until you + watch it. + """ + if len(self.history) < 2: + return ["growth: (first sample; the next one gives a rate)"] + t0, s0, d0 = self.history[0] + t1, s1, d1 = self.history[-1] + span = t1 - t0 + if span <= 0: + return [] + per_min = (s1 - s0) * 60.0 / span + dead_per_min = (d1 - d0) * 60.0 / span + lines = ["growth over %.0fs (%d samples): sockets %+d (%+.1f/min), " + "of which peer DEAD %+d (%+.1f/min)" + % (span, len(self.history), s1 - s0, per_min, d1 - d0, + dead_per_min)] + # No silent middle band. An earlier version called <0.5/min "flat" and only + # flagged >0.5/min, so a rate of exactly +0.5/min fell through reported as + # neither -- and +0.5/min is ~720 fds/day, which is precisely the overnight + # death being investigated. Any positive slope gets named, with the time to + # the fd limit as the unit that means something. + if s1 - s0 <= 0: + lines.append(" -> NOT GROWING in this window. A high count with a flat " + "rate is damage already done, not an active leak -- and " + "the process must be BUSY for that to mean anything " + "(load some pages, then re-census).") + else: + worst = max(dead_per_min, per_min) + to_limit = 1024.0 / worst if worst > 0 else float("inf") + unit = "min" if to_limit < 120 else "hours" + eta = to_limit if to_limit < 120 else to_limit / 60.0 + which = ("completed requests (peer DEAD)" if dead_per_min > 0 + else "sockets (peer still alive)") + lines.append(" -> STILL LEAKING %s at %.1f/min: a 1024-fd limit in " + "~%.0f %s. Slow is not safe; this is the shape that dies " + "overnight." % (which, worst, eta, unit)) + return lines + + +def verdict(dead, alive): + """Say what the counts MEAN, so the reply is a diagnosis and not a table.""" + if dead + alive == 0: + return "verdict: no unix sockets to classify." + # A class with ZERO members is not "present". The ratio tests below both need a + # 10x majority, so a healthy process (0 dead, ~5 live IPC sockets) fell through + # to "mixed DEAD/ALIVE -> both classes present" -- naming a class with no members + # and reading the ordinary IPC mesh as a leak. Observed on a working browser + # while testing something else, which is the only reason it was caught: a verdict + # that is wrong on healthy input will be believed when it is wrong on broken + # input too. + if dead == 0: + if alive <= IPC_MESH_MAX + 1: + return ("verdict: no retained corpses (0 peer=DEAD) and only %d live " + "socket(s) -- that is the normal IPC mesh, not a leak. Load " + "pages and watch the RATE before concluding anything." % alive) + return ("verdict: 0 peer=DEAD, %d peer=ALIVE -> nothing completed is being " + "retained; any leak here is class B (the producer still holds its " + "end, so on_finish never ran)." % alive) + if alive == 0: + return ("verdict: all %d socket(s) peer=DEAD -> completed requests retained " + "(class A), with no in-flight class B component." % dead) + if dead > 10 * max(alive, 1): + return ("verdict: overwhelmingly peer=DEAD -> completed requests retained " + "(class A). The teardown fix addresses exactly this; if it is " + "applied and this count still climbs, the fd has an owner other " + "than Requests::Request.") + if alive > 10 * max(dead, 1): + return ("verdict: overwhelmingly peer=ALIVE -> the producer still holds its " + "end, so on_finish never ran (class B). No teardown in that branch " + "can fire; the GC-root/RefPtr cycle has to be broken at the " + "Response end.") + return ("verdict: mixed DEAD/ALIVE -> both classes present; fix them " + "separately and re-census between.") + + +def watch_all(interval, retained_after): + """Watch every browser process at once and rank them by fd GROWTH. + + This exists because of a mistake worth naming: I spent the whole investigation + censusing WebContent, since that is where I believed the leak was. If the fds + accumulate in RequestServer -- which is the process that CREATES the response + pipes and the cache body files -- then every measurement I requested was blind to + it, and the leak would keep being reported as "still leaking" while my numbers + said fixed. Rank by growth, not by level: a process can hold many fds legitimately + (the IPC mesh) and the slope is what distinguishes a leak. + """ + censuses = {} + names = {} + first = True + while True: + for pid, comm in find_browser_pids(): + if pid not in censuses: + censuses[pid] = Census(pid, retained_after=retained_after) + names[pid] = comm + try: + censuses[pid].sample() + except OSError: + continue # exited under us; its history stays for the report + + print("=== %s (interval %gs) ===" % (time.strftime("%H:%M:%S"), interval)) + rows = [] + for pid, census in censuses.items(): + if not census.history: + continue + t0, s0, d0 = census.history[0] + t1, s1, d1 = census.history[-1] + total = len(read_fd_targets(pid)) if os.path.isdir("/proc/%d/fd" % pid) else 0 + span = t1 - t0 + rate = (s1 - s0) * 60.0 / span if span > 0 else 0.0 + dead_rate = (d1 - d0) * 60.0 / span if span > 0 else 0.0 + rows.append((rate, dead_rate, pid, names.get(pid, "?"), total, s1, d1, + len(census.history))) + rows.sort(reverse=True) + for rate, dead_rate, pid, comm, total, socks, dead, n in rows: + alive = "" if os.path.isdir("/proc/%d/fd" % pid) else " (EXITED)" + print(" %-14s pid=%-7d fds=%-5d sockets=%-5d dead=%-5d " + "%+.1f/min (dead %+.1f/min, %d samples)%s" + % (comm, pid, total, socks, dead, rate, dead_rate, n, alive)) + if first and len(rows): + print(" (first sample: rates are 0 until the second one)") + first = False + leakers = [r for r in rows if r[0] > 0.5] + if leakers: + print(" -> GROWING: %s. That process is the one to investigate; the fd " + "is accumulating THERE, whatever my hypothesis said." + % ", ".join("%s(pid=%d) %+.1f/min" % (r[3], r[2], r[0]) + for r in leakers)) + # Report the provenance of the growing process only: a rate without the + # code that produced it cannot distinguish "the fix failed" from "the + # fix was not in this build". + for r in leakers: + for line in fix_lines(r[2]): + print(" %s" % line) + sys.stdout.flush() + time.sleep(interval) + + +def main(argv=None): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("pid", nargs="?", type=int) + p.add_argument("--find", metavar="NAME", + help="list pids whose comm contains NAME, fd-heaviest first") + p.add_argument("--watch", type=float, metavar="SECONDS", + help="re-census every SECONDS (ages need >= 2 samples)") + p.add_argument("--all", action="store_true", + help="census EVERY Ladybird-family process and rank by growth " + "-- use this when you do not already know which process " + "leaks (i.e. always, at first)") + p.add_argument("--retained-after", type=float, default=30.0, + help="an fd older than this is retained, not in flight") + p.add_argument("--build", action="store_true", + help="report only which fd fixes are compiled into the running " + "process, and exit -- no leak measurement needed") + args = p.parse_args(argv) + + if args.build: + if args.pid is None: + p.error("--build needs a pid") + for line in fix_lines(args.pid): + print(line) + return 0 + + if args.find: + for count, pid, comm in find_pids(args.find): + print("pid=%-8d fds=%-6d %s" % (pid, count, comm)) + return 0 + + if args.all: + return watch_all(args.watch or 30.0, args.retained_after) + + if args.pid is None: + p.error("give a pid, --find NAME, or --all") + + census = Census(args.pid, retained_after=args.retained_after) + while True: + try: + snapshot = census.sample() + except OSError as e: + print("cannot read /proc/%d/fd: %s" % (args.pid, e), file=sys.stderr) + return 1 + print("=== %s pid=%d ===" % (time.strftime("%H:%M:%S"), args.pid)) + print(census.summarize(snapshot)) + if not args.watch: + return 0 + sys.stdout.flush() + time.sleep(args.watch) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/fdleak_workload.html b/examples/ladybird/fdleak_workload.html new file mode 100644 index 0000000..2791ebd --- /dev/null +++ b/examples/ladybird/fdleak_workload.html @@ -0,0 +1,74 @@ + + +

fdleak workload

starting
+ diff --git a/examples/ladybird/fdtrace.c b/examples/ladybird/fdtrace.c new file mode 100644 index 0000000..202ab2f --- /dev/null +++ b/examples/ladybird/fdtrace.c @@ -0,0 +1,418 @@ +/* + * fdtrace: name the code path that opened every fd a process never closed. + * + * Build once, LD_PRELOAD it into an UNMODIFIED browser. No patch, no rebuild of + * Ladybird, no particular commit -- which is the point: the fd census can say + * WHICH CLASS of fd is leaking, but not which code opened it, and every attempt to + * answer that so far has been me guessing at a tree I cannot reproduce. + * + * cc -shared -fPIC -O2 -g -o fdtrace.so fdtrace.c -ldl + * FDTRACE_OUT=/tmp/fdtrace.%d.log \ + * LD_PRELOAD=$PWD/fdtrace.so ./Build/full/bin/Ladybird # or the Bazel binary + * + * Then, after the leak has grown: + * + * python3 fd_census.py # how many, and which class + * python3 fdtrace_report.py /tmp/fdtrace..log + * + * The report leads with WHO SENT each still-open attachment (SO_PEERCRED on the + * receiving socket), because that is the field that discriminates. An SCM_RIGHTS fd + * is materialised by the kernel on the IPC read thread, so every attachment from + * every peer shares one identical acquisition stack -- grouping by stack alone + * produces a wall of identical frames and names nothing. `Requests::Request` fds and + * fds that never reach a Request look completely different here, which is the + * question the in-process census could not answer (it only ever saw fds that DID + * reach a Request). + * + * How it works: fds enter WebContent from RequestServer as SCM_RIGHTS attachments + * on recvmsg(), not via open(), so this wraps the acquiring calls (recvmsg, + * socketpair, socket, dup/dup2/dup3, pipe/pipe2, open/openat, accept) and the + * releasing ones (close). Each acquisition records a backtrace; each close drops + * it. Whatever is left at exit -- or whatever the report finds live -- is the leak, + * with its creation stack. + * + * Deliberately simple and allocation-light on the hot path: a fixed-size table + * indexed by fd, holding raw return addresses only. Symbolisation happens offline + * in the report script (addr2line), because doing it in-process would be slow and + * would perturb the very timing being measured. + */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MAX_FDS 65536 +#define MAX_FRAMES 24 + +struct entry { + int in_use; + int depth; + unsigned long seq; + void *frames[MAX_FRAMES]; + const char *how; +}; + +static struct entry g_table[MAX_FDS]; +static unsigned long g_seq; +static FILE *g_out; +static __thread int g_in_hook; /* re-entrancy: backtrace() itself may call libc */ + +/* Real symbols, resolved lazily. */ +static ssize_t (*real_recvmsg)(int, struct msghdr *, int); +static int (*real_close)(int); +static int (*real_socketpair)(int, int, int, int[2]); +static int (*real_socket)(int, int, int); +static int (*real_dup)(int); +static int (*real_dup2)(int, int); +static int (*real_dup3)(int, int, int); +static int (*real_pipe)(int[2]); +static int (*real_pipe2)(int[2], int); +static int (*real_accept)(int, struct sockaddr *, socklen_t *); +static int (*real_accept4)(int, struct sockaddr *, socklen_t *, int); + +#define BIND(name) \ + do { \ + if (!real_##name) \ + real_##name = dlsym(RTLD_NEXT, #name); \ + } while (0) + +static void trace_open(void) +{ + const char *pattern = getenv("FDTRACE_OUT"); + char path[4096]; + if (!pattern) + pattern = "/tmp/fdtrace.%d.log"; + /* one file per process: the browser is multi-process and they must not + * interleave, or the report cannot attribute a stack to a pid */ + snprintf(path, sizeof(path), pattern, (int)getpid()); + g_out = fopen(path, "w"); + if (!g_out) + return; + setvbuf(g_out, NULL, _IOLBF, 0); + /* The report needs the load addresses to turn return addresses into + * file:line, and they differ per run (PIE + ASLR). */ + fprintf(g_out, "# fdtrace pid=%d\n", (int)getpid()); + FILE *maps = fopen("/proc/self/maps", "r"); + if (maps) { + char line[1024]; + while (fgets(line, sizeof(line), maps)) + if (strstr(line, " r-xp ") || strstr(line, " r--p ")) + fprintf(g_out, "# map %s", line); + fclose(maps); + } +} + +/* Which IPC connection did an attachment arrive on, and WHO sent it? + * + * This is the field the first version lacked, and the reason it mattered: the + * acquisition stack of an SCM_RIGHTS fd is always the IPC read thread, identical + * for every attachment from every peer, so a log full of identical stacks says + * nothing about which subsystem is leaking. SO_PEERCRED on the receiving socket + * names the SENDER process, which does discriminate: a response pipe from + * RequestServer is the fd under investigation; an attachment from the Compositor or + * ImageDecoder is something else entirely. + * + * Cached per socket fd -- one getsockopt and one /proc read per connection, not per + * attachment, because this sits in the IPC hot path. + */ +#define MAX_PEER_CACHE 4096 +static struct { + int valid; + int pid; + char comm[32]; +} g_peer_cache[MAX_PEER_CACHE]; + +/* A snapshot of pid -> comm taken at LOAD TIME, which is the only moment a + * sandboxed process can take one. + * + * WHY THIS EXISTS -- the bug that produced `from=?(pid=2261433)`: + * WebContent installs a landlock policy that grants exactly `/proc/self` and no + * other path under /proc (Services/RendererSandboxLinux.cpp). So once the sandbox + * is up, reading /proc//comm fails with EACCES -- and so would + * /proc//cmdline and /proc//exe, because the barrier is per-path, not + * per-file. SO_PEERCRED still works, since it is a syscall on a socket rather than + * a path, which is why the log had a real pid and the name `?`. + * + * The constructor runs before main(), hence before the sandbox is installed, so a + * snapshot taken here can still name the siblings that already exist -- and the + * report resolves anything else from outside the sandbox. Every peer named `?` + * before this was a *permission* failure being read as an unknown process. */ +#define MAX_PROC_SNAPSHOT 8192 +static struct proc_name { + int pid; + char comm[32]; +} *g_proc_snapshot; +static int g_proc_snapshot_count; + +static int read_comm(int pid, char *out, size_t out_size) +{ + char path[64]; + snprintf(path, sizeof(path), "/proc/%d/comm", pid); + FILE *f = fopen(path, "r"); + if (!f) + return 0; + int ok = fgets(out, (int)out_size, f) != NULL; + fclose(f); + if (!ok) + return 0; + char *nl = strchr(out, '\n'); + if (nl) + *nl = 0; + return out[0] != 0; +} + +static void snapshot_proc_names(void) +{ + /* Best-effort and allocation is fine here: this is load time, single + * threaded, long before the first IPC message. */ + g_proc_snapshot = calloc(MAX_PROC_SNAPSHOT, sizeof(*g_proc_snapshot)); + if (!g_proc_snapshot) + return; + DIR *d = opendir("/proc"); + if (!d) + return; + struct dirent *ent; + while ((ent = readdir(d)) && g_proc_snapshot_count < MAX_PROC_SNAPSHOT) { + if (ent->d_name[0] < '1' || ent->d_name[0] > '9') + continue; + int pid = atoi(ent->d_name); + struct proc_name *slot = &g_proc_snapshot[g_proc_snapshot_count]; + if (read_comm(pid, slot->comm, sizeof(slot->comm))) { + slot->pid = pid; + g_proc_snapshot_count++; + } + } + closedir(d); +} + +static const char *snapshot_lookup(int pid) +{ + for (int i = 0; i < g_proc_snapshot_count; i++) + if (g_proc_snapshot[i].pid == pid) + return g_proc_snapshot[i].comm; + return NULL; +} + +static void peer_of(int sock, int *out_pid, const char **out_comm) +{ + *out_pid = -1; + *out_comm = "?"; + if (sock < 0 || sock >= MAX_PEER_CACHE) + return; + if (!g_peer_cache[sock].valid) { + struct ucred cred; + socklen_t len = sizeof(cred); + g_peer_cache[sock].valid = 1; + g_peer_cache[sock].pid = -1; + strcpy(g_peer_cache[sock].comm, "?"); + if (getsockopt(sock, SOL_SOCKET, SO_PEERCRED, &cred, &len) == 0) { + int pid = (int)cred.pid; + const char *how = "comm"; + g_peer_cache[sock].pid = pid; + if (!read_comm(pid, g_peer_cache[sock].comm, + sizeof(g_peer_cache[sock].comm))) { + /* Denied by the sandbox (or the peer already exited): fall back to + * the pre-sandbox snapshot. */ + const char *name = snapshot_lookup(pid); + how = "snapshot"; + if (name) { + snprintf(g_peer_cache[sock].comm, + sizeof(g_peer_cache[sock].comm), "%s", name); + } else { + strcpy(g_peer_cache[sock].comm, "?"); + how = "unresolved"; + } + } + /* One line per CONNECTION, so the report can resolve the pid itself -- + * from outside the sandbox, where /proc is readable -- even when this + * process could not. */ + if (g_out) + fprintf(g_out, "# peer sock=%d pid=%d comm=%s via=%s\n", sock, pid, + g_peer_cache[sock].comm, how); + } else if (g_out) { + fprintf(g_out, "# peer sock=%d pid=-1 comm=? via=no-peercred\n", sock); + } + } + *out_pid = g_peer_cache[sock].pid; + *out_comm = g_peer_cache[sock].comm; +} + +static void record_from(int fd, const char *how, int sock) +{ + if (fd < 0 || fd >= MAX_FDS) + return; + if (g_in_hook) + return; + g_in_hook = 1; + + struct entry *e = &g_table[fd]; + e->in_use = 1; + e->how = how; + e->seq = __sync_fetch_and_add(&g_seq, 1); + e->depth = backtrace(e->frames, MAX_FRAMES); + + if (g_out) { + fprintf(g_out, "+ fd=%d seq=%lu how=%s", fd, e->seq, how); + if (sock >= 0) { + int pid; + const char *comm; + peer_of(sock, &pid, &comm); + fprintf(g_out, " sock=%d from=%s(pid=%d)", sock, comm, pid); + } + fprintf(g_out, " stack:"); + for (int i = 0; i < e->depth; i++) + fprintf(g_out, " %p", e->frames[i]); + fprintf(g_out, "\n"); + } + g_in_hook = 0; +} + +static void record(int fd, const char *how) +{ + record_from(fd, how, -1); +} + +static void forget(int fd) +{ + if (fd < 0 || fd >= MAX_FDS) + return; + if (g_table[fd].in_use && g_out && !g_in_hook) { + g_in_hook = 1; + fprintf(g_out, "- fd=%d seq=%lu\n", fd, g_table[fd].seq); + g_in_hook = 0; + } + g_table[fd].in_use = 0; +} + +__attribute__((constructor)) static void fdtrace_init(void) +{ + trace_open(); + /* Before main(), so before the renderer's landlock policy narrows /proc down to + * /proc/self. After that point no peer name can be read from inside. */ + snapshot_proc_names(); +} + +ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags) +{ + BIND(recvmsg); + ssize_t r = real_recvmsg(sockfd, msg, flags); + if (r < 0 || !msg) + return r; + /* SCM_RIGHTS is how a response pipe arrives from RequestServer: the fd is + * created by the KERNEL here, so no open()-style hook can ever see it. This + * is the hook that matters for the Ladybird leak. */ + for (struct cmsghdr *c = CMSG_FIRSTHDR(msg); c; c = CMSG_NXTHDR(msg, c)) { + if (c->cmsg_level != SOL_SOCKET || c->cmsg_type != SCM_RIGHTS) + continue; + size_t payload = c->cmsg_len - CMSG_LEN(0); + size_t count = payload / sizeof(int); + int *fds = (int *)CMSG_DATA(c); + for (size_t i = 0; i < count; i++) + record_from(fds[i], "recvmsg/SCM_RIGHTS", sockfd); + } + return r; +} + +int close(int fd) +{ + BIND(close); + forget(fd); + return real_close(fd); +} + +int socketpair(int d, int t, int p, int sv[2]) +{ + BIND(socketpair); + int r = real_socketpair(d, t, p, sv); + if (r == 0) { + record(sv[0], "socketpair"); + record(sv[1], "socketpair"); + } + return r; +} + +int socket(int d, int t, int p) +{ + BIND(socket); + int r = real_socket(d, t, p); + if (r >= 0) + record(r, "socket"); + return r; +} + +int dup(int old) +{ + BIND(dup); + int r = real_dup(old); + if (r >= 0) + record(r, "dup"); + return r; +} + +int dup2(int old, int new_fd) +{ + BIND(dup2); + int r = real_dup2(old, new_fd); + if (r >= 0) + record(r, "dup2"); + return r; +} + +int dup3(int old, int new_fd, int flags) +{ + BIND(dup3); + int r = real_dup3(old, new_fd, flags); + if (r >= 0) + record(r, "dup3"); + return r; +} + +int pipe(int fds[2]) +{ + BIND(pipe); + int r = real_pipe(fds); + if (r == 0) { + record(fds[0], "pipe"); + record(fds[1], "pipe"); + } + return r; +} + +int pipe2(int fds[2], int flags) +{ + BIND(pipe2); + int r = real_pipe2(fds, flags); + if (r == 0) { + record(fds[0], "pipe2"); + record(fds[1], "pipe2"); + } + return r; +} + +int accept(int s, struct sockaddr *a, socklen_t *l) +{ + BIND(accept); + int r = real_accept(s, a, l); + if (r >= 0) + record(r, "accept"); + return r; +} + +int accept4(int s, struct sockaddr *a, socklen_t *l, int f) +{ + BIND(accept4); + int r = real_accept4(s, a, l, f); + if (r >= 0) + record(r, "accept4"); + return r; +} diff --git a/examples/ladybird/fdtrace_report.py b/examples/ladybird/fdtrace_report.py new file mode 100755 index 0000000..e5f0981 --- /dev/null +++ b/examples/ladybird/fdtrace_report.py @@ -0,0 +1,322 @@ +#!/usr/bin/env python3 +"""Turn an fdtrace log into a ranked list of the call sites that leak fds. + +Companion to fdtrace.c. Reads the `+ fd=... ` / `- fd=...` log, keeps the +acquisitions with no matching release, groups them by identical stack, and +symbolises each group's stack with addr2line -- so the output is "these 815 fds +were all created here", which is the question `fd_census.py` cannot answer. + + python3 fdtrace_report.py /tmp/fdtrace.12345.log + python3 fdtrace_report.py /tmp/fdtrace.12345.log --top 5 --frames 12 + +Why offline: symbolising in-process would perturb the timing being measured, and +the addresses are only meaningful together with the /proc/self/maps lines fdtrace +records at startup (PIE + ASLR move everything per run). +""" + +import argparse +import bisect +import collections +import os +import re +import subprocess +import sys + +MAP_RE = re.compile( + r"^# map ([0-9a-f]+)-([0-9a-f]+) \S+ ([0-9a-f]+) \S+ \d+\s+(/.*\S)\s*$") +PEER_RE = re.compile( + r"^# peer sock=(-?\d+) pid=(-?\d+) comm=(\S*) via=(\S+)\s*$") + + +def resolve_pid(pid): + """Name a pid from OUTSIDE the sandboxed process, which can read /proc. + + A renderer's landlock policy grants only /proc/self, so the tracer inside it + cannot read a sibling's /proc//comm and logs `?`. The report does not run + under that policy, so it can finish the job -- as long as the process is still + alive, which it usually is: RequestServer outlives the WebContent it serves. + """ + if pid is None or pid < 0: + return None + for name, transform in (("comm", lambda t: t.strip()), + ("cmdline", lambda t: os.path.basename( + t.split("\0")[0]) if t.strip("\0") else "")): + try: + with open("/proc/%d/%s" % (pid, name)) as f: + value = transform(f.read()) + except OSError: + continue + if value: + return value + try: + return os.path.basename(os.readlink("/proc/%d/exe" % pid)) + except OSError: + return None + + +class Maps: + """Address -> (object file, file-relative offset), from the recorded maps.""" + + def __init__(self): + self.starts = [] + self.entries = [] + + def add(self, start, end, offset, path): + self.starts.append(start) + self.entries.append((start, end, offset, path)) + + def finish(self): + order = sorted(range(len(self.starts)), key=lambda i: self.starts[i]) + self.starts = [self.starts[i] for i in order] + self.entries = [self.entries[i] for i in order] + + def resolve(self, addr): + i = bisect.bisect_right(self.starts, addr) - 1 + if i < 0: + return None + start, end, offset, path = self.entries[i] + if not (start <= addr < end): + return None + # addr2line on a shared object wants the offset within the FILE + return path, addr - start + offset + + +SENDER_RE = re.compile(r"^(?P.*)\(pid=(?P-?\d+)\)$") + + +def parse_peers(path): + """{sock: (pid, comm, via)} from the `# peer` header lines, if present. + + `via=snapshot` means the tracer could not read /proc at attachment time (the + renderer's sandbox) and fell back to its pre-sandbox snapshot; `via=unresolved` + means even that missed and only the pid is known. + """ + peers = {} + with open(path) as f: + for line in f: + if not line.startswith("# peer "): + continue + m = PEER_RE.match(line.rstrip("\n")) + if m: + peers[int(m.group(1))] = (int(m.group(2)), m.group(3), m.group(4)) + return peers + + +def name_sender(sender, resolver=resolve_pid): + """Turn a logged `NAME(pid=N)` into the best name available NOW. + + The tracer logs `?` when the sandbox denied /proc//comm. That is a + permission failure, not an unknown process, and it is recoverable here because + the report is not sandboxed -- so an unnamed sender gets one more chance from + outside, and if the process is gone the reader is told the exact command that + would have answered it rather than being left with a bare `?`. + """ + if not sender: + return sender + m = SENDER_RE.match(sender) + if not m: + return sender + name, pid = m.group("name"), int(m.group("pid")) + if name and name != "?": + return sender + resolved = resolver(pid) + if resolved: + return "%s(pid=%d)" % (resolved, pid) + return "?(pid=%d, gone -- was `ps -p %d -o comm=`)" % (pid, pid) + + +def parse(path): + maps = Maps() + open_fds = {} # fd -> (seq, how, [addrs]) + acquisitions = {} # seq -> (fd, how, [addrs]) + closed = set() + with open(path) as f: + for line in f: + if line.startswith("# map "): + m = MAP_RE.match(line.rstrip("\n")) + if m: + maps.add(int(m.group(1), 16), int(m.group(2), 16), + int(m.group(3), 16), m.group(4)) + continue + if line.startswith("+ "): + parts = line.split() + fd = int(parts[1].split("=")[1]) + seq = int(parts[2].split("=")[1]) + how = parts[3].split("=")[1] + sender = None + rest = parts[4:] + # `from=NAME(pid=N)` when the fd arrived as an IPC attachment. Old + # logs have no such field and no `stack:` marker; still readable. + for tok in list(rest): + if tok.startswith("from="): + sender = tok[len("from="):] + if tok == "stack:": + rest = rest[rest.index(tok) + 1:] + break + else: + rest = [t for t in rest if t.startswith("0x")] + addrs = [int(a, 16) for a in rest if a.startswith("0x")] + open_fds[fd] = seq + acquisitions[seq] = (fd, how, addrs, sender) + elif line.startswith("- "): + parts = line.split() + seq = int(parts[2].split("=")[1]) + closed.add(seq) + maps.finish() + live = {seq: v for seq, v in acquisitions.items() if seq not in closed} + return maps, acquisitions, live + + +def symbolize(maps, addrs, frames, skip_internal=True): + """addr2line per object file, batched. Missing tools degrade to raw addresses.""" + by_obj = collections.defaultdict(list) + order = [] + for a in addrs[:frames + 4]: + r = maps.resolve(a) + if r is None: + order.append((None, a)) + continue + path, off = r + by_obj[path].append(off) + order.append((path, off)) + + resolved = {} + for path, offs in by_obj.items(): + if not os.path.exists(path): + continue + try: + out = subprocess.run( + ["addr2line", "-f", "-C", "-e", path] + ["0x%x" % o for o in offs], + capture_output=True, text=True, timeout=60).stdout.splitlines() + except (OSError, subprocess.SubprocessError): + continue + # addr2line -f emits function then file:line, per address + for i, off in enumerate(offs): + fn = out[2 * i] if 2 * i < len(out) else "??" + loc = out[2 * i + 1] if 2 * i + 1 < len(out) else "??" + resolved[(path, off)] = (fn, loc) + + lines = [] + for key in order: + path, off = key + if path is None: + lines.append(" 0x%x (unmapped)" % off) + continue + fn, loc = resolved.get((path, off), ("??", "??")) + # The tracer's own frames are noise: they are in every stack by + # construction and push the real caller off the end of --frames. + if skip_internal and (fn.startswith("fdtrace") or fn.startswith("record") + or fn in ("recvmsg", "close", "socketpair", "forget", + "peer_of", "dup", "dup2", "dup3", "pipe", + "pipe2", "socket", "accept", "accept4")): + continue + short = os.path.basename(path) + if fn == "??" and loc == "??": + lines.append(" 0x%x in %s" % (off, short)) + else: + lines.append(" %s (%s)" % (fn, loc)) + if len(lines) >= frames: + break + return lines + + +def main(argv=None): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("log") + p.add_argument("--top", type=int, default=8, + help="how many leaking call sites to show") + p.add_argument("--frames", type=int, default=14, + help="stack frames per call site") + args = p.parse_args(argv) + + maps, acquisitions, live = parse(args.log) + if not acquisitions: + print("no fd acquisitions in %s -- was LD_PRELOAD actually in effect?" + % args.log, file=sys.stderr) + return 1 + + groups = collections.defaultdict(list) + senders = collections.defaultdict(list) + # Cache per sender string: a `?` costs a /proc read, and there are thousands of + # leaked fds sharing a handful of connections. + named = {} + for seq, (fd, how, addrs, sender) in live.items(): + groups[(how, tuple(addrs))].append((seq, fd)) + if sender: + if sender not in named: + named[sender] = name_sender(sender) + senders[named[sender]].append(fd) + + print("fdtrace: %d acquisitions, %d still open, %d distinct call sites" + % (len(acquisitions), len(live), len(groups))) + print("(still open = acquired and never closed, by THIS process)") + print() + + # An SCM_RIGHTS fd is CREATED BY THE KERNEL on the IPC read thread, so its + # acquisition stack is always TransportSocket::io_thread_loop -- true and + # useless on its own. What it does establish, precisely, is HOW MANY leaked fds + # entered as IPC attachments versus being opened locally, which is the split + # between "a received response pipe was never closed" and "something else". + # WHO SENT the leaked attachments. This is the discriminating field, because + # every attachment shares one acquisition stack (the IPC read thread) and so the + # stacks cannot tell two subsystems apart. A response pipe from RequestServer is + # the fd under investigation; one from Compositor or ImageDecoder is not. + if senders: + # If the tracer could not name a peer itself, say so once and say why -- + # otherwise a `?` reads as "unknown process" when it actually means "the + # renderer's landlock policy grants /proc/self only". + unresolved = [(sock, pid, via) for sock, (pid, _comm, via) + in sorted(parse_peers(args.log).items()) + if via in ("snapshot", "unresolved")] + if unresolved: + print("note: %d connection(s) could not be named INSIDE the process " + "(/proc is" % len(unresolved)) + print(" landlocked to /proc/self in the renderer); resolved here " + "from outside instead.") + for sock, pid, via in unresolved: + print(" sock=%d pid=%d (%s)" % (sock, pid, via)) + print() + print("still-open IPC attachments BY SENDER:") + for name, fds in sorted(senders.items(), key=lambda kv: -len(kv[1])): + print(" %6d from %s e.g. fd %s" % ( + len(fds), name, ", ".join(str(f) for f in sorted(fds)[:6]))) + top, top_fds = max(senders.items(), key=lambda kv: len(kv[1])) + if "RequestServer" in top: + print(" -> the leaked fds are RESPONSE PIPES from RequestServer: the") + print(" class this investigation is about. Every one is a request") + print(" whose response fd WebContent decoded and never closed.") + else: + print(" -> NOT RequestServer. The leak is attachments from %s, which is" + % top.split("(")[0]) + print(" a different bug from the per-request response-pipe leak --") + print(" look at what decodes IPC::File from that peer.") + print() + + ipc = sum(len(v) for (how, _), v in groups.items() if how.startswith("recvmsg")) + local = len(live) - ipc + print("still-open by origin: %d arrived over IPC (SCM_RIGHTS), %d opened locally" + % (ipc, local)) + if ipc > 10 * max(local, 1): + print(" -> the leaked fds are RECEIVED ATTACHMENTS. The creation stack is") + print(" the IPC read thread by construction; the bug is on the RECEIVING") + print(" side -- an attachment decoded into an owner that never closes it.") + print(" Cross-check with fd_census.py: peer=DEAD means the sender is") + print(" already gone, so nothing but this process can still close them.") + print() + + ranked = sorted(groups.items(), key=lambda kv: -len(kv[1])) + for (how, addrs), holders in ranked[:args.top]: + fds = sorted(f for _, f in holders) + print("%d still-open fds via %s e.g. fd %s" % ( + len(holders), how, ", ".join(str(f) for f in fds[:6]))) + for line in symbolize(maps, list(addrs), args.frames): + print(line) + print() + + if len(ranked) > args.top: + print("... %d more call sites" % (len(ranked) - args.top)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/patches/0001-libweb-bindings-deterministic-dictionary-order.patch b/examples/ladybird/patches/0001-libweb-bindings-deterministic-dictionary-order.patch deleted file mode 100644 index f0da082..0000000 --- a/examples/ladybird/patches/0001-libweb-bindings-deterministic-dictionary-order.patch +++ /dev/null @@ -1,20 +0,0 @@ -diff --git a/Meta/Generators/libweb_bindings/to_idl_value.py b/Meta/Generators/libweb_bindings/to_idl_value.py -index ef890dbb1b..9e9bb92767 100644 ---- a/Meta/Generators/libweb_bindings/to_idl_value.py -+++ b/Meta/Generators/libweb_bindings/to_idl_value.py -@@ -189,7 +189,14 @@ def dictionaries_in_dependency_order(dictionaries: List[Dictionary], context: Ge - raise RuntimeError(f"Dictionary '{dictionary.name}' depends on itself") - - visiting.add(dictionary.name) -- for dependency_name in dependency_names_for(dictionary): -+ # sorted(): dependency_names_for returns a SET, so iterating it visits -+ # sibling dependencies in hash order, and the emission order of two -+ # dictionaries that do not depend on each other (AudioConfiguration and -+ # VideoConfiguration, both reached from MediaConfiguration) then varies -+ # with PYTHONHASHSEED. A topological sort only constrains dependency -+ # before dependent; the order among independent siblings has to be pinned -+ # separately, or the generator is not reproducible. -+ for dependency_name in sorted(dependency_names_for(dictionary)): - dependency = local_dictionaries.get(dependency_name) - if dependency is not None: - visit(dependency) diff --git a/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.effect-grep b/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.effect-grep new file mode 100644 index 0000000..23cc47a --- /dev/null +++ b/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.effect-grep @@ -0,0 +1,24 @@ +# The EFFECT this patch must have, as extended regexes over the file it touches +# (Libraries/LibRequests/Request.cpp). Checked only when the patch's exact bytes +# are NOT in the tree, so a tree that is NEWER than our pin -- one where #11041 +# has merged -- verifies as "already fixed" instead of being told to apply a +# patch that would conflict. That is the whole reason this mechanism exists. +# +# The effect: the branch that decides the body is fully delivered (the one that +# calls the user's finish callback) must ALSO tear the request down, so the +# ReadStream holding the response-pipe fd is released instead of waiting for GC. +# +# The window matters. defer_teardown() already appears in stop() and +# did_transfer(), so a whole-file grep for it passes on a tree WITHOUT the fix -- +# verified, and pinned by a test. So anchor on the branch condition and require +# the call within the following few lines. +# +# NOTE the ORDER is deliberately NOT asserted here. Upstream calls +# defer_teardown() BEFORE user_on_finish (so the deferred lambda's NonnullRefPtr +# pins the Request across the callback); the patch this replaced called it after, +# which was a latent use-after-free. A grep cannot express "before" without +# becoming a byte comparison, and the exact-bytes check already covers our own +# version -- so this asserts the fix is PRESENT and leaves ordering to the patch. +@window 12 has_received_all_reported_bytes = +user_finish_called = true +defer_teardown\(\); diff --git a/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch b/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch new file mode 100644 index 0000000..d559d4c --- /dev/null +++ b/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch @@ -0,0 +1,234 @@ +From 63817a66757e4005383cd7c4e9508fa188e823fc Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Fri, 7 Aug 2026 09:44:33 -0700 +Subject: [PATCH 1/3] LibRequests+LibWeb: Release response pipes when requests + complete +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Pages issuing fetches in a tight loop accumulate one open Unix +socket per completed fetch in their WebContent process. The WPT scroll- +to-text-fragment/redirects.html test polls that way, and drove a local +WebContent process to ~21,900 socket/fileport attachments — at which +point the process crashed after receiving the next response pipe failed. +In a local repro driving about 1,500 fetches per second, WebContent grew +from 5,541 to 63,543 Unix-socket descriptors in 43 seconds. + +Cause: Requests::Request tears down its stream state only when a request +is stopped or transferred. On ordinary completion, the ReadStream with +the response-pipe descriptor, its notifier, and the internal buffers all +stay alive until the Request object is destroyed – and a fetch Response +holds a reference to its Request until GC. The scarce resource is file +descriptors — but GC pressure is measured in heap bytes. So, a tight +fetch loop retires thousands of completed requests without GC ever +having a reason to run — and so, the descriptor table fills up. + +Fix: Schedule the existing deferred teardown at the moment a request +delivers its user-finish callback (the single point every request passes +through on ordinary completion) — for buffered and unbuffered modes and +for network errors alike. That’s only reached once all response data has +been delivered. So streaming consumers and paused document loads are un- +affected — and stopped/transferred requests keep their existing teardown +paths. The teardown must stay deferred: It’s scheduled from inside the +callback chain it destroys, and the deferred task also keeps the Request +alive if the callback drops the last ref. With this change, the repro +holds steady at 0–2 open response pipes at an unchanged request rate. +--- + Libraries/LibRequests/Request.cpp | 20 +++++++++++ + Libraries/LibRequests/Request.h | 9 +++-- + Libraries/LibWeb/Internals/Internals.cpp | 6 ++++ + Libraries/LibWeb/Internals/Internals.h | 1 + + Libraries/LibWeb/Internals/Internals.idl | 1 + + ...sponse-pipes-released-after-completion.txt | 1 + + ...ponse-pipes-released-after-completion.html | 33 +++++++++++++++++++ + 7 files changed, 66 insertions(+), 5 deletions(-) + create mode 100644 Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt + create mode 100644 Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html + +| UPSTREAM PATCH -- Ladybird PR #11041 by sideshowbarker, commit 63817a66757e4005383cd7c4e9508fa188e823fc. +| NOT OURS. Carried only because our pin 71fb301a predates the merge; DELETE all +| three (and this note) on the first repin past it -- see the README patch table. +| +| Why upstream's and not the two patches this replaces: those were written here +| before #11041 existed, and one of them CRASHED Ulf's browser. Our 0002 +| (release_response_fd) nulled m_internal_stream_data->read_stream from inside the +| completion branch, while the read_notifier's on_activation lambda -- the frame +| that CALLS on_finish -- goes on to dereference that same OwnPtr: +| +| if (m_internal_stream_data->read_stream->is_eof()) <-- Request.cpp:376 +| m_internal_stream_data->read_notifier->close(); +| +| so the null landed on VERIFY(m_ptr) in AK::OwnPtr::operator-> (AK/OwnPtr.h:134), +| which is exactly the trace Ulf sent: CallableWrapper<...set_up_internal_stream_data +| ...{lambda#2}>::call() -> ak_verification_failed -> SIGILL. Upstream never nulls +| read_stream; it only ensures defer_teardown() is REACHED, and it reaches it BEFORE +| user_on_finish so the deferred lambda's NonnullRefPtr pins the Request across the +| callback. Our 0001 called it after, which is a second latent bug in the pair. +| +| The deeper point, recorded in docs/UPSTREAM-ladybird-fd-leaks.md: our 0002 existed +| to close the fd on the theory that a surviving reference pinned it. Upstream fixes +| the leak by running the teardown on all three paths -- so the theory was +| unnecessary, and carrying both would have been two mechanisms closing one fd, one +| of them justified by a theory the other disproves. + +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425a5bad5..16234917a4927 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -35,6 +35,25 @@ static Optional map_body_file(int fd, u64 offset, u64 size + return payload.release_value(); + } + ++static size_t s_live_read_stream_count = 0; ++ ++size_t ReadStream::live_count() ++{ ++ return s_live_read_stream_count; ++} ++ ++ReadStream::ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier) ++ : m_stream(move(stream)) ++ , m_notifier(move(notifier)) ++{ ++ ++s_live_read_stream_count; ++} ++ ++ReadStream::~ReadStream() ++{ ++ --s_live_read_stream_count; ++} ++ + ErrorOr> ReadStream::create(int reader_fd) + { + #if defined(AK_OS_WINDOWS) +@@ -326,6 +345,7 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + auto has_received_all_reported_bytes = m_internal_stream_data->request_done && m_internal_stream_data->delivered_size >= m_internal_stream_data->total_size; + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { + m_internal_stream_data->user_finish_called = true; ++ defer_teardown(); + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); + } + }; +diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h +index b4046fa8d114a..69cdf0d6d0398 100644 +--- a/Libraries/LibRequests/Request.h ++++ b/Libraries/LibRequests/Request.h +@@ -55,6 +55,9 @@ class ResponseData { + class ReadStream { + public: + static ErrorOr> create(int reader_fd); ++ ~ReadStream(); ++ ++ static size_t live_count(); + + NonnullRefPtr const& notifier() const { return m_notifier; } + +@@ -63,11 +66,7 @@ class ReadStream { + ErrorOr read_some(Bytes bytes) { return m_stream->read_some(bytes); } + + private: +- ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier) +- : m_stream(move(stream)) +- , m_notifier(move(notifier)) +- { +- } ++ ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier); + + NonnullOwnPtr m_stream; + NonnullRefPtr m_notifier; +diff --git a/Libraries/LibWeb/Internals/Internals.cpp b/Libraries/LibWeb/Internals/Internals.cpp +index d221019b6e8de..65cb4a9150f3d 100644 +--- a/Libraries/LibWeb/Internals/Internals.cpp ++++ b/Libraries/LibWeb/Internals/Internals.cpp +@@ -22,6 +22,7 @@ + #include + #include + #include ++#include + #include + #include + #include +@@ -614,6 +615,11 @@ void Internals::simulate_request_server_connection_loss() + page().client().page_did_lose_request_server_connection(); + } + ++WebIDL::UnsignedLongLong Internals::open_response_pipe_count() ++{ ++ return Requests::ReadStream::live_count(); ++} ++ + WebIDL::ExceptionOr Internals::set_content_blockers(Utf16String const& patterns_source) + { + Utf16StringBuilder patterns_builder; +diff --git a/Libraries/LibWeb/Internals/Internals.h b/Libraries/LibWeb/Internals/Internals.h +index b2df9d5c9e051..cd0a9bbf4a2dc 100644 +--- a/Libraries/LibWeb/Internals/Internals.h ++++ b/Libraries/LibWeb/Internals/Internals.h +@@ -106,6 +106,7 @@ class WEB_API Internals final : public InternalsBase { + + bool set_http_memory_cache_enabled(bool enabled); + void simulate_request_server_connection_loss(); ++ WebIDL::UnsignedLongLong open_response_pipe_count(); + WebIDL::ExceptionOr set_content_blockers(Utf16String const& patterns); + void set_content_blocking_enabled(bool enabled); + WebIDL::UnsignedLongLong partial_layout_count(); +diff --git a/Libraries/LibWeb/Internals/Internals.idl b/Libraries/LibWeb/Internals/Internals.idl +index e2634eadcac82..107b11fd08160 100644 +--- a/Libraries/LibWeb/Internals/Internals.idl ++++ b/Libraries/LibWeb/Internals/Internals.idl +@@ -85,6 +85,7 @@ interface Internals { + + boolean setHttpMemoryCacheEnabled(boolean enabled); + undefined simulateRequestServerConnectionLoss(); ++ unsigned long long openResponsePipeCount(); + undefined setContentBlockers(Utf16DOMString patterns); + undefined setContentBlockingEnabled(boolean enabled); + unsigned long long partialLayoutCount(); +diff --git a/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt b/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt +new file mode 100644 +index 0000000000000..836a4a872c4d4 +--- /dev/null ++++ b/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt +@@ -0,0 +1 @@ ++open response pipes after fetches: 0 +diff --git a/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html b/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html +new file mode 100644 +index 0000000000000..29c66bcf3cd40 +--- /dev/null ++++ b/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html +@@ -0,0 +1,33 @@ ++ ++ ++ + diff --git a/examples/ladybird/patches/0002-ui-qt-tabbar-self-contained-header.patch b/examples/ladybird/patches/0002-ui-qt-tabbar-self-contained-header.patch deleted file mode 100644 index 6e69b77..0000000 --- a/examples/ladybird/patches/0002-ui-qt-tabbar-self-contained-header.patch +++ /dev/null @@ -1,29 +0,0 @@ -From: any2bazel Ladybird migration -Subject: [PATCH] UI/Qt: make TabBar.h self-contained by including Tab.h - -TabBar.h calls as() inline -- a dynamic_cast, which needs Tab's complete -type -- while only forward-declaring Tab. It compiles today only by accident of -CMake's AUTOMOC: the unity mocs_compilation.cpp happens to #include moc_Tab.cpp -(which includes Tab.h) before moc_TabBar.cpp, so the definition is in scope by -the time TabBar.h is parsed. - -Bazel runs moc per header rather than through a unity file, so nothing supplies -Tab.h first and the header fails to compile on its own. That makes this a real -latent bug rather than a Bazel quirk: the header is not self-contained, and any -build that compiles it in a different order -- a unity build with different -bucketing, a header-only IWYU check, a different generator -- hits the same -thing. - -One line, and it makes the header stand alone. -diff --git a/UI/Qt/TabBar.h b/UI/Qt/TabBar.h -index 00269c5c..93963006 100644 ---- a/UI/Qt/TabBar.h -+++ b/UI/Qt/TabBar.h -@@ -10,6 +10,7 @@ - - #include - #include -+#include - - #include - #include diff --git a/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.effect-grep b/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.effect-grep new file mode 100644 index 0000000..9d6f61e --- /dev/null +++ b/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.effect-grep @@ -0,0 +1,18 @@ +# The EFFECT of upstream #11041 patch 2/3, over the file it touches +# (Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp). Checked only when +# the exact bytes are absent, so a tree where #11041 has MERGED verifies clean +# rather than being told to apply a patch that would conflict. +# +# The effect: abort() and terminate() must actually release the network request. +# stop_fetch() returns early once the state is aborted/terminated, so without an +# explicit release the response pipe stays open for the life of the process -- +# a whole leak class our fd census could not distinguish (it ranks by growth and +# splits peer=DEAD/ALIVE, which does not point at the CANCEL path specifically). +# +# Anchored on the helper's definition plus a call, rather than on a call alone: +# the name is the fix, and a definition with no caller would be the interesting +# broken state. No @window -- the two call sites are in different functions +# (abort, terminate) tens of lines apart, so a window would have to be so wide it +# proved nothing; requiring both the definition and the declaration is stronger. +void FetchController::stop_pending_request\(\) +stop_pending_request\(\); diff --git a/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch b/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch new file mode 100644 index 0000000..7177ee5 --- /dev/null +++ b/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch @@ -0,0 +1,136 @@ +From 2ae78115aecaae054532d8a8ba93fecb4c8a9a3c Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Mon, 17 Aug 2026 07:04:13 +0900 +Subject: [PATCH 2/3] LibWeb: Release response pipes when fetches are canceled +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Aborting or terminating a fetch left its response pipe open for +the lifetime of the process. A page that starts and cancels requests in +a loop (an EventSource closed and reopened, a fetch or XHR aborted while +the response is still hanging) leaked one file descriptor per cycle — +until it ran out of them. + +Cause: abort() and terminate() only set the controller state. The one +path that releases the request, stop_request(), is reached only from +stop_fetch() — which returns early once the state is already “aborted” +or “terminated”. So the two entry points that mark a fetch as canceled +were the two that never told the network layer — and RequestServer holds +a request alive until it either finishes or is stopped. + +Fix: Release the request from abort() and terminate() too — through a +helper that stop_request() now shares. + +One caller relied on terminate() leaving the request alone. When a +navigation response becomes a download, the request is handed to the UI +process and the fetch is then terminated. That path now drops its handle +to the request before terminating — so a download already under way +isn’t stopped. RequestServer reports the transfer back to the +WebContent process — and that’s what closes the reader end of the pipe. +--- + .../LibWeb/Fetch/Infrastructure/FetchController.cpp | 13 +++++++++++++ + .../LibWeb/Fetch/Infrastructure/FetchController.h | 5 +++++ + Libraries/LibWeb/HTML/LocalNavigable.cpp | 7 ++++++- + 3 files changed, 24 insertions(+), 1 deletion(-) + +| UPSTREAM PATCH -- Ladybird PR #11041 by sideshowbarker, commit 2ae78115aecaae054532d8a8ba93fecb4c8a9a3c. +| NOT OURS. Carried only because our pin 71fb301a predates the merge; DELETE all +| three (and this note) on the first repin past it -- see the README patch table. +| +| Why upstream's and not the two patches this replaces: those were written here +| before #11041 existed, and one of them CRASHED Ulf's browser. Our 0002 +| (release_response_fd) nulled m_internal_stream_data->read_stream from inside the +| completion branch, while the read_notifier's on_activation lambda -- the frame +| that CALLS on_finish -- goes on to dereference that same OwnPtr: +| +| if (m_internal_stream_data->read_stream->is_eof()) <-- Request.cpp:376 +| m_internal_stream_data->read_notifier->close(); +| +| so the null landed on VERIFY(m_ptr) in AK::OwnPtr::operator-> (AK/OwnPtr.h:134), +| which is exactly the trace Ulf sent: CallableWrapper<...set_up_internal_stream_data +| ...{lambda#2}>::call() -> ak_verification_failed -> SIGILL. Upstream never nulls +| read_stream; it only ensures defer_teardown() is REACHED, and it reaches it BEFORE +| user_on_finish so the deferred lambda's NonnullRefPtr pins the Request across the +| callback. Our 0001 called it after, which is a second latent bug in the pair. +| +| The deeper point, recorded in docs/UPSTREAM-ladybird-fd-leaks.md: our 0002 existed +| to close the fd on the theory that a surviving reference pinned it. Upstream fixes +| the leak by running the teardown on all three paths -- so the theory was +| unnecessary, and carrying both would have been two mechanisms closing one fd, one +| of them justified by a theory the other disproves. + +diff --git a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp +index 5e41472a1eb3b..6ba3de7c24e8c 100644 +--- a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp ++++ b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp +@@ -104,6 +104,10 @@ void FetchController::abort(JS::Realm& realm, Optional error) + : serialized_value_or_error.value(); + }; + m_serialized_abort_reason = structured_serialize(error.value(), fallback_error_value); ++ ++ // AD-HOC: stop_fetch() returns early once the state is "aborted". So, this is the last chance to release the ++ // network request. Without it the response pipe stays open for good. ++ stop_pending_request(); + } + + // https://fetch.spec.whatwg.org/#fetch-controller-terminate +@@ -111,6 +115,10 @@ void FetchController::terminate() + { + // To terminate a fetch controller controller, set controller’s state to "terminated". + m_state = State::Terminated; ++ ++ // AD-HOC: As in abort() above — stop_fetch() won’t release the request once the state is "terminated" — so, ++ // release it here. ++ stop_pending_request(); + } + + void FetchController::stop_fetch() +@@ -143,6 +151,11 @@ void FetchController::stop_fetch() + void FetchController::stop_request() + { + VERIFY(m_state == State::Stopped); ++ stop_pending_request(); ++} ++ ++void FetchController::stop_pending_request() ++{ + if (m_pending_request) { + m_pending_request->stop(); + m_pending_request = nullptr; +diff --git a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h +index fb27df8f53e55..480a764277c0d 100644 +--- a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h ++++ b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h +@@ -86,6 +86,11 @@ class WEB_API FetchController : public JS::Cell { + // Null or a fetch timing info. + GC::Ptr m_full_timing_info; + ++ // Releases the network request behind this controller. Every way a fetch stops early has to reach this: The ++ // response pipe is a socket pair, and RequestServer holds the request alive until it either finishes or is stopped. ++ // So, a request that's abandoned without being stopped would keep its descriptor for the lifetime of the process. ++ void stop_pending_request(); ++ + // https://fetch.spec.whatwg.org/#fetch-controller-report-timing-steps + // report timing steps (default null) + // Null or an algorithm accepting a global object. +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.cpp b/Libraries/LibWeb/HTML/LocalNavigable.cpp +index ff47e30dfa7c2..ee28f176bc547 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.cpp ++++ b/Libraries/LibWeb/HTML/LocalNavigable.cpp +@@ -540,8 +540,13 @@ static bool handle_navigation_response_as_download(GC::Ref nav + return true; + } + +- if (navigation_params->fetch_controller) ++ if (navigation_params->fetch_controller) { ++ // AD-HOC: The request now belongs to the UI process (it adopted it above). So, before terminating, drop our ++ // handle to it. Otherwise, fetch termination would stop a download already under way. RequestServer ++ // reports the transfer back to us; that's what tears down this process's end of the pipe. ++ navigation_params->fetch_controller->set_pending_request(nullptr); + navigation_params->fetch_controller->terminate(); ++ } + + return true; + } + diff --git a/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.effect-grep b/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.effect-grep new file mode 100644 index 0000000..b7cc71a --- /dev/null +++ b/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.effect-grep @@ -0,0 +1,22 @@ +# The EFFECT of upstream #11041 patch 3/3, over the file it touches +# (Libraries/LibWeb/HTML/LocalNavigable.cpp). Checked only when the exact bytes +# are absent, so a tree where #11041 has MERGED verifies clean. +# +# The effect: a navigation parked for content sniffing -- headers received, but +# fewer bytes than the sniff threshold -- must have a teardown that runs when the +# navigable is destroyed, when a newer navigation supersedes it, or when it is +# destroyed already. Such a navigation has no document, so Document::abort() has +# no controller to stop, and ONLY the arrival callback would release the request +# -- which never runs. This was the open lead in +# docs/UPSTREAM-ladybird-fd-leaks.md that I could not reproduce: both of my +# workloads abandoned navigations without DESTROYING the navigable, so they +# exercised everything except the condition that matters. +# +# Anchored on the pair of accessors by name: the mechanism IS the registration +# plus the ownership check on clearing (only the navigation that registered a +# teardown may clear it, or a late sniff callback clears a newer navigation's +# teardown and leaks that one instead). A grep for run_pending_navigation_teardown +# alone would pass on a tree that stores the teardown and never runs it. +void LocalNavigable::set_pending_navigation_teardown +void LocalNavigable::clear_pending_navigation_teardown +run_pending_navigation_teardown\(\); diff --git a/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch b/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch new file mode 100644 index 0000000..51a66bd --- /dev/null +++ b/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch @@ -0,0 +1,168 @@ +From 8636af4103410833243e5eb987d64e8baf076b1a Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Mon, 17 Aug 2026 07:28:12 +0900 +Subject: [PATCH 3/3] LibWeb: Tear down a navigation parked for content + sniffing +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Removing an iframe whose response headers had arrived but whose +body hadn’t arrived left the network request open for the lifetime of +the process. A page that adds and removes such iframes in a loop leaked +one file descriptor per iframe. + +Cause: If a navigation has a response but not yet enough bytes to sniff +its content type, it gets parked in wait_for_sniff_bytes. It has no +document at that point — so, Document::abort() has no fetch controller +to stop, and the only code that releases the request is the arrival +callback. That callback does handle a destroyed navigable — but it runs +only once bytes arrive. So, a server that sends headers and then stops +leaves nothing at all around to release the request. + +Fix: Register a teardown on the navigable before parking — and run it if +the navigable is destroyed first. The teardown calls the same helper the +arrival callback already uses — and destroying a navigable is where the +rest of its in-flight navigation state is released. + +A navigation superseded before its populate task runs leaked the same +way, for the same reason: That guard returned without releasing +anything. It now releases the response as the guard above it does. +--- + Libraries/LibWeb/HTML/LocalNavigable.cpp | 50 +++++++++++++++++++++++- + Libraries/LibWeb/HTML/LocalNavigable.h | 6 +++ + 2 files changed, 55 insertions(+), 1 deletion(-) + +| UPSTREAM PATCH -- Ladybird PR #11041 by sideshowbarker, commit 8636af4103410833243e5eb987d64e8baf076b1a. +| NOT OURS. Carried only because our pin 71fb301a predates the merge; DELETE all +| three (and this note) on the first repin past it -- see the README patch table. +| +| Why upstream's and not the two patches this replaces: those were written here +| before #11041 existed, and one of them CRASHED Ulf's browser. Our 0002 +| (release_response_fd) nulled m_internal_stream_data->read_stream from inside the +| completion branch, while the read_notifier's on_activation lambda -- the frame +| that CALLS on_finish -- goes on to dereference that same OwnPtr: +| +| if (m_internal_stream_data->read_stream->is_eof()) <-- Request.cpp:376 +| m_internal_stream_data->read_notifier->close(); +| +| so the null landed on VERIFY(m_ptr) in AK::OwnPtr::operator-> (AK/OwnPtr.h:134), +| which is exactly the trace Ulf sent: CallableWrapper<...set_up_internal_stream_data +| ...{lambda#2}>::call() -> ak_verification_failed -> SIGILL. Upstream never nulls +| read_stream; it only ensures defer_teardown() is REACHED, and it reaches it BEFORE +| user_on_finish so the deferred lambda's NonnullRefPtr pins the Request across the +| callback. Our 0001 called it after, which is a second latent bug in the pair. +| +| The deeper point, recorded in docs/UPSTREAM-ladybird-fd-leaks.md: our 0002 existed +| to close the fd on the theory that a surviving reference pinned it. Upstream fixes +| the leak by running the teardown on all three paths -- so the theory was +| unnecessary, and carrying both would have been two mechanisms closing one fd, one +| of them justified by a theory the other disproves. + +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.cpp b/Libraries/LibWeb/HTML/LocalNavigable.cpp +index ee28f176bc547..c908afbcee8df 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.cpp ++++ b/Libraries/LibWeb/HTML/LocalNavigable.cpp +@@ -681,6 +681,40 @@ void LocalNavigable::set_has_been_destroyed() + m_has_been_destroyed = true; + resolve_all_pending_async_scroll_operations(); + cancel_user_scroll_settlement(); ++ run_pending_navigation_teardown(); ++} ++ ++void LocalNavigable::set_pending_navigation_teardown(GC::Ptr> teardown) ++{ ++ // A navigation parking here while another is already parked supersedes it. So, tear the older one down — rather ++ // than dropping it and leaving its request open. ++ run_pending_navigation_teardown(); ++ ++ // Destruction has already run, so nothing would ever run this one. Tear it down now — instead of storing it forever. ++ if (m_has_been_destroyed) { ++ if (teardown) ++ teardown->function()(); ++ return; ++ } ++ ++ m_pending_navigation_teardown = teardown; ++} ++ ++void LocalNavigable::clear_pending_navigation_teardown(GC::Ptr> expected) ++{ ++ // Only the navigation that registered this teardown may clear it. A navigation whose sniff-byte callback ++ // arrives after a newer one has parked would otherwise clear the newer teardown — and leave that ++ // newer request with nothing to release it. ++ if (m_pending_navigation_teardown != expected) ++ return; ++ ++ m_pending_navigation_teardown = nullptr; ++} ++ ++void LocalNavigable::run_pending_navigation_teardown() ++{ ++ if (auto teardown = exchange(m_pending_navigation_teardown, nullptr)) ++ teardown->function()(); + } + + void LocalNavigable::remove_from_all_local_navigables() +@@ -711,6 +745,7 @@ void LocalNavigable::visit_edges(Cell::Visitor& visitor) + visitor.visit(m_active_document); + visitor.visit(m_input_method_composition_node); + visitor.visit(m_container); ++ visitor.visit(m_pending_navigation_teardown); + m_event_handler.visit_edges(visitor); + + for (auto& navigation_params : m_pending_navigations) { +@@ -2190,6 +2225,9 @@ void LocalNavigable::populate_session_history_entry_document( + + // 1. If navigable's ongoing navigation no longer equals navigationId, then run completionSteps and abort these steps. + if (navigation_id.has_value() && ongoing_navigation() != navigation_id) { ++ // AD-HOC: Nothing downstream will consume this response, and no document exists yet to abort — so, ++ // release its request here, as the active-window guard above does. ++ stop_or_resume_response_body_delivery(navigation_params); + if (completion_steps) { + completion_steps->function()(nullptr); + } +@@ -2303,8 +2341,18 @@ void LocalNavigable::populate_session_history_entry_document( + if (!sniff_bytes.has_value()) { + // Async path: bytes not yet available, wait for them + nav_params->response->resume_body_delivery_up_to(Fetch::Infrastructure::MAX_SNIFF_BYTES); ++ ++ // AD-HOC: The callback below runs only once bytes arrive — which never happens if the server sends ++ // headers and then stops. Hand the navigable a teardown — so that destroying it releases ++ // the request, instead of leaving the pipe open. ++ auto teardown = GC::create_function(heap(), [navigation_params] { ++ stop_or_resume_response_body_delivery(navigation_params); ++ }); ++ nav_params->navigable->set_pending_navigation_teardown(teardown); ++ + body->wait_for_sniff_bytes(GC::create_function(heap(), +- [output, nav_params, navigation_params, completion_steps, source_snapshot_params](ReadonlyBytes sniff_bytes) { ++ [output, nav_params, navigation_params, completion_steps, source_snapshot_params, teardown](ReadonlyBytes sniff_bytes) { ++ nav_params->navigable->clear_pending_navigation_teardown(teardown); + // AD-HOC: The document may have been destroyed between when the fetch started and when the + // bytes arrived. + if (nav_params->navigable->active_browsing_context()) { +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.h b/Libraries/LibWeb/HTML/LocalNavigable.h +index 8ebc9e99660ae..879da0f94f09c 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.h ++++ b/Libraries/LibWeb/HTML/LocalNavigable.h +@@ -95,6 +95,10 @@ class WEB_API LocalNavigable : public Navigable { + void set_navigation_load_event_guard(DOM::Document& parent_doc); + void clear_navigation_load_event_guard(); + ++ void set_pending_navigation_teardown(GC::Ptr>); ++ void clear_pending_navigation_teardown(GC::Ptr> expected); ++ void run_pending_navigation_teardown(); ++ + RefPtr active_session_history_entry() const; + void set_active_session_history_entry(RefPtr); + RefPtr current_session_history_entry() const; +@@ -406,6 +410,8 @@ class WEB_API LocalNavigable : public Navigable { + // AD-HOC: Guards the parent document's load event delay count during cross-document navigation. + Optional m_navigation_load_event_guard; + ++ GC::Ptr> m_pending_navigation_teardown; ++ + // Implied link between navigable and its container. + GC::Ptr m_container; + diff --git a/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt b/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt new file mode 100644 index 0000000..17ecb07 --- /dev/null +++ b/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt @@ -0,0 +1,470 @@ +NOT AN OVERLAY PATCH -- do not put this in patches/*.patch, apply_overlay.sh globs +those and this must never be part of a normal build. + +PREFER `examples/ladybird/fd_census.py`, WHICH NEEDS NO PATCH AT ALL. + + python3 examples/ladybird/fd_census.py --find WebContent + python3 examples/ladybird/fd_census.py --watch 30 + +It reports the same things this build does -- the /proc category census, peer +DEAD/ALIVE per socket, retained-vs-in-flight by age, and which process holds the +live peers -- by reading /proc and `ss` from outside a RUNNING browser. Verified +against this build on the same workload: 143 DEAD / 4 ALIVE from inside, 143 DEAD / +4 ALIVE from outside. + +Use it unless you specifically need something only the in-process build can see +(the request id, `user_finish_called`, or the `LADYBIRD_FDLEAK_TEARDOWN` A/B). +Patching a colleague's tree to run a diagnostic is the wrong trade when the +information is visible from outside: it assumes their tree is at our pinned commit, +it conflicts with a tree that already carries upstream's fix, and it means "reset +your tree" for anyone who has their own commits. + +This is a temporary DIAGNOSTIC for the remaining Ladybird fd leak (see +docs/UPSTREAM-ladybird-fd-leaks.md). Apply it to Libraries/LibRequests/Request.cpp +in a Ladybird checkout, rebuild only LibRequests, and run the browser normally: + + git apply .../DIAGNOSTIC-fdleak-census.patch.txt + ninja -C Build/full lib/liblagom-requests.so.0.1.0 # ~2 min warm + # or, for the Bazel build: bazel build //:ladybird (LibRequests is a cc_library, + # so a .so cannot be dropped in -- the source patch is the delivery mechanism) + +Environment knobs (all optional): + + LADYBIRD_FDLEAK_INTERVAL_SEC=30 census every N seconds (default 30) + LADYBIRD_FDLEAK_EVERY=64 also census every N requests started + LADYBIRD_FDLEAK_TEARDOWN=1 enable the completed-request fix, so ONE + build can A/B it (unset = upstream behaviour) + +What it prints, every interval, to WebContent's stderr: + + FDLEAK[timer]: live=... created=... destroyed=... did_finish=... + finish: delivered=... skipped=... | teardown=... stop=... transfer=... + FDLEAK[timer]: process fds: total=... sockets=... + FDLEAK live: in_flight(<30s)=... retained(>=30s)=... oldest_retained: id=... age=...s + FDLEAK retained-bucket N x peer=DEAD|ALIVE torn_down=... did_finish=... user_finish=... + request_done=... stream=... eof=... paused=... fd_open=... + +The two things it adds over an external /proc census: + +1. **in_flight vs retained.** A request younger than 30s is not a leak, it is a + request. Counting fds from outside cannot tell those apart, which is how a + freshly restarted process once looked like a fix. + +2. **peer=DEAD vs peer=ALIVE**, probed in-process with poll()+MSG_PEEK on the + response fd. This is the discriminator between the leak classes, and it is the + same signature as `ss -np`'s peer inode (`* 0` = dead): + + peer=DEAD -> RequestServer already closed its half: the request COMPLETED and + WebContent is retaining a corpse. This is the class the upstream + teardown patch fixes (verified locally: 143 dead-peer sockets + before, 0 after). + peer=ALIVE -> RequestServer is still holding its half: on_finish never ran, so + no teardown of any shape can fire. Reproduced locally with a + server that declares Content-Length and then stalls: 40 retained, + unchanged by the patch. + +So the buckets answer, with counts rather than inference, WHICH class a given +browsing session is actually leaking -- which is exactly what is unresolved now +that the upstream patch is applied and the leak continues. + +The census is O(retained) per interval and the whole thing is inert until the +first request is created. +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425a..abaa78cc 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -5,15 +5,273 @@ + */ + + #include ++#include ++#include + #include + #include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include + #include ++#include + #include + #include + #include + + namespace Requests { + ++// === FDLEAK instrumentation (temporary diagnostic build; NOT for upstream) === ++// Question it answers: for the response fds that are never closed, which exit ++// path did the request take? Prints a census every 64 requests started. ++namespace { ++ ++struct FdleakCounters { ++ size_t created { 0 }; ++ size_t destroyed { 0 }; ++ size_t teardown { 0 }; // defer_teardown() entered ++ size_t did_finish { 0 }; // request_finished IPC delivered ++ size_t finish_no_stream { 0 }; // on_finish with m_internal_stream_data == null ++ size_t finish_delivered { 0 }; // the if-block ran: user_on_finish + teardown ++ size_t finish_skipped { 0 }; // the if-block was SKIPPED: no teardown, fd kept ++ size_t stops { 0 }; ++ size_t transfers { 0 }; ++ size_t skip_reason_not_done { 0 }; // !request_done ++ size_t skip_reason_short { 0 }; // request_done but delivered < total ++ size_t fd_still_open_at_destroy { 0 }; ++}; ++ ++// The candidate fix is behind an env var so one build can A/B it: ++// LADYBIRD_FDLEAK_TEARDOWN=1 -> tear the request down once its body is delivered ++// unset -> upstream behaviour (leaks) ++size_t fdleak_report_interval() ++{ ++ static size_t interval = [] -> size_t { ++ auto const* value = getenv("LADYBIRD_FDLEAK_EVERY"); ++ if (!value || !*value) ++ return 64; ++ auto parsed = StringView { value, strlen(value) }.to_number(); ++ return parsed.value_or(64) ? parsed.value_or(64) : 64; ++ }(); ++ return interval; ++} ++ ++bool fdleak_teardown_enabled() ++{ ++ static bool enabled = [] { ++ auto const* value = getenv("LADYBIRD_FDLEAK_TEARDOWN"); ++ return value && *value && *value != '0'; ++ }(); ++ return enabled; ++} ++ ++FdleakCounters& fdleak() ++{ ++ static FdleakCounters counters; ++ return counters; ++} ++ ++// Snapshot of every request that is still alive right now. A request that stays ++// in here after its response finished is a leak: its callbacks (and the ++// response fd) are retained forever. ++struct FdleakLiveState { ++ u64 id { 0 }; ++ int fd { -1 }; ++ MonotonicTime created_at { MonotonicTime::now() }; ++ bool torn_down { false }; ++ bool did_finish { false }; ++ bool user_finish_called { false }; ++ bool request_done { false }; ++ bool has_stream { false }; ++ bool eof { false }; ++ bool paused { false }; ++ bool fd_open { false }; ++ u64 delivered { 0 }; ++ u64 total { 0 }; ++ unsigned skips { 0 }; ++}; ++ ++HashMap& fdleak_live() ++{ ++ static HashMap table; ++ return table; ++} ++ ++// How many fds does this process hold right now, and how many of them are ++// sockets? Cheap enough at once-per-64-requests. ++struct FdCounts { ++ size_t total { 0 }; ++ size_t sockets { 0 }; ++}; ++ ++FdCounts fdleak_fd_counts() ++{ ++ FdCounts counts; ++ Core::DirIterator it("/proc/self/fd", Core::DirIterator::SkipParentAndBaseDir); ++ while (it.has_next()) { ++ auto name = it.next_path(); ++ ++counts.total; ++ char target[64] = {}; ++ auto path = ByteString::formatted("/proc/self/fd/{}", name); ++ auto length = ::readlink(path.characters(), target, sizeof(target) - 1); ++ if (length > 0 && StringView { target, static_cast(length) }.starts_with("socket:"sv)) ++ ++counts.sockets; ++ } ++ return counts; ++} ++ ++// Is the OTHER end of this fd gone? That is the discriminator between the two ++// leak classes: RequestServer closes its pipe half when the request is complete ++// (`ss -np` shows peer inode `* 0`), and keeps it open while the body is still ++// being produced. Reading it in-process means the census classifies itself. ++enum class PeerState { ++ Unknown, ++ Alive, ++ Dead, ++}; ++ ++PeerState fdleak_peer_state(int fd) ++{ ++ if (fd < 0) ++ return PeerState::Unknown; ++ struct pollfd pfd = { .fd = fd, .events = POLLIN, .revents = 0 }; ++ if (::poll(&pfd, 1, 0) < 0) ++ return PeerState::Unknown; ++ if (pfd.revents & (POLLHUP | POLLERR)) ++ return PeerState::Dead; ++ if (!(pfd.revents & POLLIN)) ++ return PeerState::Alive; // readable-never: peer still holds it open ++ // Readable: a zero-byte peek is EOF, i.e. the peer closed after writing. ++ char scratch = 0; ++ auto peeked = ::recv(fd, &scratch, 1, MSG_PEEK | MSG_DONTWAIT); ++ if (peeked == 0) ++ return PeerState::Dead; ++ if (peeked < 0 && errno == EAGAIN) ++ return PeerState::Alive; ++ if (peeked < 0) ++ return PeerState::Unknown; ++ return PeerState::Alive; // unread data waiting ++} ++ ++StringView fdleak_peer_name(PeerState state) ++{ ++ switch (state) { ++ case PeerState::Alive: ++ return "peer=ALIVE"sv; ++ case PeerState::Dead: ++ return "peer=DEAD"sv; ++ default: ++ return "peer=?"sv; ++ } ++} ++ ++void fdleak_census() ++{ ++ // Anything alive for more than this is not "in flight", it is retained. ++ static constexpr auto retained_after = AK::Duration::from_seconds(30); ++ auto now = MonotonicTime::now(); ++ ++ HashMap buckets; ++ size_t in_flight = 0; ++ size_t retained = 0; ++ u64 oldest_retained_id = 0; ++ i64 oldest_age_ms = 0; ++ ++ for (auto const& [request, state] : fdleak_live()) { ++ auto age = now - state.created_at; ++ if (age < retained_after) { ++ ++in_flight; ++ continue; ++ } ++ ++retained; ++ if (age.to_milliseconds() > oldest_age_ms) { ++ oldest_age_ms = age.to_milliseconds(); ++ oldest_retained_id = state.id; ++ } ++ auto bucket = ByteString::formatted("{} torn_down={} did_finish={} user_finish={} request_done={} stream={} eof={} paused={} fd_open={} short={} skips={}", ++ fdleak_peer_name(fdleak_peer_state(state.fd)), ++ state.torn_down, state.did_finish, state.user_finish_called, state.request_done, ++ state.has_stream, state.eof, state.paused, state.fd_open, ++ state.did_finish && state.delivered < state.total, state.skips > 0); ++ ++buckets.ensure(bucket, [] { return 0u; }); ++ } ++ ++ dbgln("FDLEAK live: in_flight(<30s)={} retained(>=30s)={} oldest_retained: id={} age={}s", ++ in_flight, retained, oldest_retained_id, oldest_age_ms / 1000); ++ for (auto const& [bucket, count] : buckets) ++ dbgln("FDLEAK retained-bucket {:5} x {}", count, bucket); ++} ++ ++void fdleak_report(StringView tag) ++{ ++ auto& c = fdleak(); ++ dbgln("FDLEAK[{}]: live={} (created={} destroyed={}) did_finish={} finish: delivered={} skipped={} no_stream={} | skip_why: not_done={} short={} | teardown={} stop={} transfer={} fd_open_at_dtor={}", ++ tag, c.created - c.destroyed, c.created, c.destroyed, c.did_finish, ++ c.finish_delivered, c.finish_skipped, c.finish_no_stream, ++ c.skip_reason_not_done, c.skip_reason_short, ++ c.teardown, c.stops, c.transfers, c.fd_still_open_at_destroy); ++ auto fds = fdleak_fd_counts(); ++ dbgln("FDLEAK[{}]: process fds: total={} sockets={}", tag, fds.total, fds.sockets); ++ fdleak_census(); ++} ++ ++// Refresh the snapshot for `request` from live state. Called from every place ++// that can change it, so a retained request that never finished is still ++// described accurately (rather than showing the default-false fields). ++void fdleak_note(Request* request, bool torn_down_flag, bool has_stream, bool eof, bool paused, bool fd_open, bool request_done, u64 delivered, u64 total, int fd) ++{ ++ auto it = fdleak_live().find(request); ++ if (it == fdleak_live().end()) ++ return; ++ auto& st = it->value; ++ st.fd = fd; ++ if (torn_down_flag) ++ st.torn_down = true; ++ st.has_stream = has_stream; ++ st.eof = eof; ++ st.paused = paused; ++ st.fd_open = fd_open; ++ st.request_done = request_done; ++ st.delivered = delivered; ++ st.total = total; ++} ++ ++void fdleak_start_timer() ++{ ++ static RefPtr timer; ++ if (timer) ++ return; ++ auto seconds = [] -> int { ++ auto const* value = getenv("LADYBIRD_FDLEAK_INTERVAL_SEC"); ++ if (!value || !*value) ++ return 30; ++ return StringView { value, strlen(value) }.to_number().value_or(30); ++ }(); ++ timer = Core::Timer::create_repeating(seconds * 1000, [] { ++ fdleak_report("timer"sv); ++ }); ++ timer->start(); ++} ++ ++} ++ ++// Snapshot `this`'s current state into the live census. Usable anywhere inside a ++// Request member function. ++#define FDLEAK_NOTE(torn_down_flag) \ ++ fdleak_note(this, (torn_down_flag), \ ++ m_internal_stream_data && m_internal_stream_data->read_stream, \ ++ m_internal_stream_data && m_internal_stream_data->read_stream && m_internal_stream_data->read_stream->is_eof(), \ ++ m_body_delivery_paused, \ ++ m_fd != -1 && (!m_fd_is_owned_by_read_stream || (m_internal_stream_data && m_internal_stream_data->read_stream)), \ ++ m_internal_stream_data && m_internal_stream_data->request_done, \ ++ m_internal_stream_data ? m_internal_stream_data->delivered_size : 0, \ ++ m_internal_stream_data ? m_internal_stream_data->total_size : 0, \ ++ m_fd) ++ + static Optional map_body_file(int fd, u64 offset, u64 size) + { + ArmedScopeGuard close_fd = [fd] { +@@ -53,10 +311,21 @@ Request::Request(RequestClient& client, u64 request_id) + : m_client(client) + , m_request_id(request_id) + { ++ auto& c = fdleak(); ++ ++c.created; ++ fdleak_live().set(this, FdleakLiveState { .id = request_id }); ++ fdleak_start_timer(); ++ if (c.created % fdleak_report_interval() == 0) ++ fdleak_report("periodic"sv); + } + + Request::~Request() + { ++ auto& c = fdleak(); ++ ++c.destroyed; ++ fdleak_live().remove(this); ++ if (m_fd != -1) ++ ++c.fd_still_open_at_destroy; + if (m_fd != -1 && !m_fd_is_owned_by_read_stream) + (void)Core::System::close(m_fd); + } +@@ -68,6 +337,7 @@ int Request::request_server_client_id() const + + bool Request::stop() + { ++ ++fdleak().stops; + RefPtr keep_alive = *this; + + // The client may already be gone if the RequestServer connection was lost while this request was in flight. +@@ -250,6 +520,10 @@ void Request::set_stop_callback(RequestStopped on_stop) + + void Request::did_finish(Badge, u64 total_size, RequestTimingInfo const& timing_info, Optional const& network_error) + { ++ ++fdleak().did_finish; ++ if (auto it = fdleak_live().find(this); it != fdleak_live().end()) ++ it->value.did_finish = true; ++ FDLEAK_NOTE(false); + auto effective_network_error = m_body_delivery_error.has_value() ? m_body_delivery_error : network_error; + if (on_finish) + on_finish(total_size, timing_info, effective_network_error); +@@ -273,6 +547,7 @@ void Request::did_request_certificates(Badge) + + void Request::did_transfer(Badge) + { ++ ++fdleak().transfers; + auto on_stop = move(m_on_stop); + + defer_teardown(); +@@ -283,6 +558,14 @@ void Request::did_transfer(Badge) + + void Request::defer_teardown() + { ++ ++fdleak().teardown; ++ if (auto it = fdleak_live().find(this); it != fdleak_live().end()) { ++ it->value.torn_down = true; ++ // Teardown drops m_internal_stream_data, whose ReadStream owns and closes ++ // the response fd; m_fd keeps its (now stale) number, so stop reporting it. ++ it->value.has_stream = false; ++ it->value.fd_open = false; ++ } + if (m_internal_stream_data && m_internal_stream_data->read_notifier) + m_internal_stream_data->read_notifier->set_enabled(false); + m_mode = Mode::Unknown; +@@ -308,8 +591,10 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + auto user_on_finish = move(on_finish); + on_finish = [this](auto total_size, auto const& timing_info, auto network_error) { + // If the request was stopped while this IPC was in-flight, just bail. +- if (!m_internal_stream_data) ++ if (!m_internal_stream_data) { ++ ++fdleak().finish_no_stream; + return; ++ } + + m_internal_stream_data->total_size = total_size; + m_internal_stream_data->network_error = network_error; +@@ -325,8 +610,35 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + + auto has_received_all_reported_bytes = m_internal_stream_data->request_done && m_internal_stream_data->delivered_size >= m_internal_stream_data->total_size; + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { ++ ++fdleak().finish_delivered; ++ if (auto it = fdleak_live().find(this); it != fdleak_live().end()) ++ it->value.user_finish_called = true; ++ FDLEAK_NOTE(false); + m_internal_stream_data->user_finish_called = true; + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); ++ ++ // The body is fully delivered: drop the callbacks (and the GC::Roots they captured) ++ // and close the response fd, as stop()/did_transfer() do for the abandoned cases. ++ if (fdleak_teardown_enabled()) ++ defer_teardown(); ++ } else if (!m_internal_stream_data->user_finish_called) { ++ // FDLEAK: this is the path that never tears down. Name the failing condition. ++ auto& c = fdleak(); ++ ++c.finish_skipped; ++ if (auto it = fdleak_live().find(this); it != fdleak_live().end()) ++ ++it->value.skips; ++ FDLEAK_NOTE(false); ++ if (!m_internal_stream_data->request_done) ++ ++c.skip_reason_not_done; ++ else ++ ++c.skip_reason_short; ++ dbgln("FDLEAK skip: id={} mode={} request_done={} delivered={} total={} eof={} paused={} remaining={}", ++ m_request_id, m_mode == Mode::Buffered ? "buffered" : (m_mode == Mode::Unbuffered ? "unbuffered" : "unknown"), ++ m_internal_stream_data->request_done, m_internal_stream_data->delivered_size, ++ m_internal_stream_data->total_size, ++ m_internal_stream_data->read_stream ? m_internal_stream_data->read_stream->is_eof() : true, ++ m_body_delivery_paused, ++ m_internal_stream_data->body_delivery_remaining_byte_count.has_value() ? (i64)*m_internal_stream_data->body_delivery_remaining_byte_count : (i64)-1); + } + }; + +@@ -373,6 +685,8 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + } + } while (true); + ++ FDLEAK_NOTE(false); ++ + if (m_internal_stream_data->read_stream->is_eof()) + m_internal_stream_data->read_notifier->close(); + diff --git a/examples/ladybird/qt_runtime_diagnose.sh b/examples/ladybird/qt_runtime_diagnose.sh new file mode 100755 index 0000000..bdb6e66 --- /dev/null +++ b/examples/ladybird/qt_runtime_diagnose.sh @@ -0,0 +1,201 @@ +#!/usr/bin/env bash +# Diagnose a Qt runtime crash in the Bazel-built Ladybird, in one paste. +# +# WHY THIS EXISTS. Two unrelated failures present as "immediate crash", and +# LD_LIBRARY_PATH makes both go away -- which is exactly why reaching for it loses +# the information that says which one you had: +# +# A. THE LIBRARIES. An official/aqt Qt bundles its own ICU (aqt 6.9.2's +# libQt6Core needs libicui18n.so.73, which exists in the SDK's lib/ and +# nowhere else on a machine whose distro ICU is 76+). Bazel links @qt's +# libQt6Core out of a solib dir; libQt6Core resolves ICU through +# `RUNPATH $ORIGIN`, and $ORIGIN is the directory the loader OPENED it by -- +# the solib dir, which has no ICU. Death before main(). +# Fixed by @qt_plugins//:runtime_libs: the SDK's private libs become real link +# inputs, so BAZEL stages them and OUR runpath (the one glibc consults for our +# direct deps) finds them. +# +# B. THE PLUGINS. Qt dlopens the QPA plugin at QApplication construction from a +# prefix baked into libQt6Core, or from the executable's directory. Load the +# DISTRO's libqxcb.so into an SDK libQt6Core and you get SIGSEGV in +# QXcbConnection::initializeScreens (or, if the plugin is older, a clean +# "no Qt platform plugin could be initialized" abort). +# Fixed by //:qt_conf + //:qt_plugins staged beside the binary. +# +# Both fixes are in the tree. This script checks whether each one actually FIRED +# for the Qt you have, because a fix that silently degrades to "no private +# libraries" is indistinguishable from a Qt that needs none (finding 35). +# +# Usage: ./qt_runtime_diagnose.sh [/path/to/ladybird-tree] +# Reads only; runs nothing that can change the build. + +set -uo pipefail +TREE="${1:-$PWD}" +cd "$TREE" || { echo "no such tree: $TREE" >&2; exit 1; } + +say() { printf '\n=== %s\n' "$*"; } +BIN="bazel-bin/ladybird" + +say "0. the tree" +echo "tree: $TREE" +echo "commit: $(git rev-parse --short HEAD 2>/dev/null || echo '?')" + +say "0b. is a STALE libexec/ shadowing the services? (the most likely crash)" +# Checked FIRST, before anything about Qt, because this failure looks like a Qt +# crash, arrives as a SIGILL/VERIFICATION FAILED with a Qt-flavoured backtrace, +# and has nothing to do with Qt. It is also the failure that has now bitten twice. +# +# LibWebView/Utilities.cpp's get_paths_for_helper_process() searches +# /libexec/ FIRST +# /bin/ second +# so ANY libexec copy wins over the binaries Bazel just built, and only the copy +# is ever executed. After a repin the stale copies are from the OLD pin, whose IPC +# message IDs have shifted, so every message fails to parse: +# +# Failed to parse IPC message: +# Peer endpoint error: Endpoint magic number mismatch, not my message! +# IPC::ConnectionBase: Disconnecting misbehaving peer due to malformed message +# VERIFICATION FAILED: connection at Services/Compositor/ConnectionFromClient.cpp +# +# The tell is in the backtrace's PATHS, not in its frames: `ladybird` runs from +# bazel-out/.../bin/ while `Compositor` runs from bazel-out/.../libexec/. Todo +# 4a93a257 is exactly this lesson -- for a "built fine, behaves wrong" bug, ask +# what is EXECUTING before auditing what produced it. +stale=0 +for d in bazel-out/*/libexec bazel-bin/libexec; do + [ -d "$d" ] || continue + stale=$((stale + 1)) + echo " SHADOWING: $d" + ls "$d" 2>/dev/null | sed 's/^/ /' + echo " newest file here: $(find "$d" -type f -printf '%TY-%Tm-%Td %p\n' 2>/dev/null | sort | tail -1)" +done +if [ "$stale" -gt 0 ]; then + echo " compare against the FRESH build:" + for b in bazel-bin/Compositor bazel-bin/WebContent; do + [ -e "$b" ] && echo " $(find "$b" -printf '%TY-%Tm-%Td %p\n' 2>/dev/null)" + done + cat <<'FIX' + -> THIS IS ALMOST CERTAINLY YOUR BUG. libexec is searched BEFORE bin, so those + copies are what run, not what you just built. If their dates are older than + bazel-bin's, the UI is talking to services from a previous pin and every IPC + message fails with "Endpoint magic number mismatch". Delete them: +FIX + for d in bazel-out/*/libexec bazel-bin/libexec; do + [ -d "$d" ] && echo " rm -rf $TREE/$d" + done + echo " Nothing needs staging there: the services are already siblings of" + echo " ladybird in bazel-bin, which is the second entry in the lookup chain." +else + echo " none -- good (the services resolve to bazel-bin, the fresh build)" +fi + +say "1. which Qt the BUILD was told to use (MODULE.bazel)" +sed -n 's|^[[:space:]]*paths = {"linux-x86_64": "\([^"]*\)".*|\1|p' MODULE.bazel | head -1 + +say "2. which Qt @qt actually DISCOVERED (its own generated qtconf.bzl)" +# The output base, asked of bazel rather than guessed. +OB="$(bazel info output_base 2>/dev/null)" +QTCONF="$(find "$OB/external" -maxdepth 3 -name qtconf.bzl 2>/dev/null | head -1)" +if [ -n "$QTCONF" ]; then + grep -E '^(QT_VERSION|QT_INSTALL_PREFIX|QT_INSTALL_LIBS|QT_INSTALL_PLUGINS)' "$QTCONF" + LIBS="$(sed -n 's|^QT_INSTALL_LIBS="\(.*\)"|\1|p' "$QTCONF")" +else + echo "NOT FOUND -- @qt has not been fetched yet (build first)" + LIBS="" +fi + +say "3. did the PRIVATE-LIBRARY staging fire? (failure A)" +PLUG="$(find "$OB/external" -maxdepth 2 -name '*qt_plugins' -type d 2>/dev/null | head -1)" +if [ -z "$PLUG" ]; then + echo "@qt_plugins not fetched yet" +elif grep -q 'name = "runtime_libs"' "$PLUG/BUILD.bazel" 2>/dev/null; then + # grep -c prints one count PER FILE; with a glob that is "0\n0", which then + # fails `[ ... -eq 0 ]` with "integer expected". Count lines instead. + n="$(grep -h '^cc_import' "$PLUG/BUILD.bazel" 2>/dev/null | wc -l)" + echo "runtime_libs exists with $n cc_import(s):" + grep -A2 '^cc_import' "$PLUG/BUILD.bazel" 2>/dev/null | sed 's/^/ /' + if [ "$n" -eq 0 ]; then + cat <<'NOTE' + -> EMPTY. That is CORRECT for a distro Qt (its ICU is a distro package, + already on the loader's default search path) and WRONG for a + self-contained SDK. Check section 4: if the SDK's lib dir has libicu* + in it and this is empty, the derivation missed them -- that is the bug, + not your machine. +NOTE + fi +else + echo "no runtime_libs target at all (an OLD @qt_plugins? bazel sync --configure)" +fi + +say "4. what the SDK's lib dir actually ships (the input to that derivation)" +if [ -n "$LIBS" ] && [ -d "$LIBS" ]; then + echo "$LIBS:" + ls "$LIBS" | grep -vE '^libQt' | grep '\.so' | sed 's/^/ /' | head -20 + echo " (non-Qt .so files above; those DT_NEEDED by a libQt6*.so are what must be staged)" + echo " ICU the Qt libs ask for:" + objdump -p "$LIBS"/libQt6Core.so.6 2>/dev/null \ + | awk '/NEEDED/ && /icu/ {print " " $2}' | sort -u +else + echo "lib dir unknown or missing: '${LIBS:-}'" +fi + +say "5. did the PLUGIN staging fire? (failure B)" +if [ -f bazel-bin/qt.conf ]; then + sed 's/^/ /' bazel-bin/qt.conf + echo " plugins staged: $(find bazel-bin/plugins -name '*.so' 2>/dev/null | wc -l) .so" + echo " platform plugin -> $(readlink -f bazel-bin/plugins/platforms/libqxcb.so 2>/dev/null || echo MISSING)" + echo " (that path must be under the SAME SDK as section 2's QT_INSTALL_LIBS)" +else + echo "bazel-bin/qt.conf MISSING -- Qt will scan the compiled-in prefix instead" +fi + +say "6. what the BINARY resolves at load time (the actual answer)" +if [ -x "$BIN" ]; then + if ldd "$BIN" 2>&1 | grep -q "not found"; then + echo "UNRESOLVED libraries -- this is failure A:" + ldd "$BIN" 2>&1 | grep "not found" | sed 's/^/ /' + else + echo "all libraries resolve. Where the interesting ones come from:" + ldd "$BIN" 2>/dev/null | grep -iE 'icu|libQt6(Core|Gui|Widgets)' | sed 's/^/ /' + fi +else + echo "$BIN not built" +fi + +say "7. run it, with NO LD_LIBRARY_PATH, and keep the first failure" +echo "(unset deliberately: with it set, both failures disappear and this says nothing)" +env -u LD_LIBRARY_PATH "$BIN" --version 2>&1 | head -5 +rc=$? +echo "--version exit: $rc" + +say "8. what the helper processes would actually RESOLVE to" +# The question the backtrace answers and no build check does: for each service, +# which copy is first on Ladybird's lookup chain. Printed even when 0b found +# nothing, because "the fresh one" is the answer that makes the negative useful. +for svc in Compositor WebContent RequestServer ImageDecoder WebWorker; do + found="" + for d in bazel-out/*/libexec bazel-bin/libexec; do + [ -x "$d/$svc" ] && { found="$d/$svc <-- SHADOWS bazel-bin"; break; } + done + [ -n "$found" ] || { [ -x "bazel-bin/$svc" ] && found="bazel-bin/$svc"; } + printf ' %-14s %s\n' "$svc" "${found:-NOT FOUND}" +done + +cat <<'EOF' + +=== READ IT LIKE THIS + section 0b found a libexec/, or section 8 says SHADOWS + -> NOT a Qt problem at all, whatever the backtrace looks + like. Stale services from an older build are what run; + the IPC ids have shifted. `rm -rf` them and re-run. + section 6 shows "not found", or section 7 says "error while loading shared + libraries" -> failure A: the private-library staging. Section 3 is + empty while section 4 lists libicu*. Send me 3+4. + section 7 crashes with a BACKTRACE through QXcbConnection / QGuiApplication, + or says "no Qt platform plugin could be initialized" + -> failure B: the plugins. Compare section 5's resolved + libqxcb.so path against section 2's QT_INSTALL_LIBS; + if they are different SDKs, that is the bug. + neither, and it still crashes -> not the Qt runtime at all; get the backtrace: + gdb -q -batch -ex run -ex bt --args bazel-bin/ladybird +EOF diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index 8d1b5e2..ff1c103 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -2,18 +2,25 @@ load("@rules_qt//qt:defs.bzl", "qt_cc_moc", "qt_cc_rcc", "qt_qrc") load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library") load(":cargo_ring.bzl", "cargo_ring") load(":codegen_root.bzl", "root_codegen") +load(":export_headers.bzl", "export_headers") +load(":qt_runtime.bzl", "qt_conf", "qt_plugin_tree") load(":vcpkg.bzl", "vcpkg_tree", "vcpkg_tree_for_exec") load(":vcpkg_index.bzl", "VCPKG_DISTFILE_INDEX") package(default_visibility = ["//visibility:public"]) -# 24 generator genrules for this package: the IPC endpoints, LibJS Bytecode/Op, +# 25 generator genrules for this package: the IPC endpoints, LibJS Bytecode/Op, # LibHTTP's HSTS table, the Compositor WebGL replayer, the TIFF tag tables, the # two SPIR-V shader headers and the Flap interpreter assembly. Bazel now # GENERATES all of them instead of consuming CMake's prebuilt copies from # Build/full. root_codegen() +# CMake's generate_export_header() output (15 Export.h) and the two AK +# configure_file headers, generated by Bazel instead of read out of Build/full -- +# generated by Meta/emit_export_headers_bazel.py, byte-verified with --check. +export_headers() + # The Rust ring: 10 cargo staticlib crates + flapc, built by Bazel from crates # Bazel fetched, offline. This is what retired the prebuilt 260 MB # librust_combined.a and its hand-run `ar -M` merge (README step 1b). It lives in @@ -36,30 +43,41 @@ cc_library( "UI/**/*.h", ], allow_empty = True), deps = [ - # Bazel-generated headers (the genrules above) come FIRST so they win - # over the CMake copies; the Build/full roots below now only supply what - # Bazel does not yet generate (Rust FFI headers, CMake's - # generate_export_header Export.h). + # EVERY generated header is Bazel's own now. The two + # //Build/full/{Libraries,Services} header roots that used to be listed + # here are gone, and with them the last thing the BUILD read out of + # CMake's build tree. + # + # They were not supplying anything by the end, and that is the part worth + # recording, because it is why they survived so long: a `glob(["**/*.h"])` + # over a foreign tree cannot fail. The 709 headers under those roots were + # 21 that Bazel generates and 688 LibWeb bindings headers that Bazel ALSO + # generates -- so the roots were SHADOWING Bazel's own outputs, silently + # winning or losing on include order, and the only reason a fresh clone + # did not fail here was that it got no error either: allow_empty=True + # turns "the tree is not there" into an empty glob and the build dies + # ~1,600 actions later on a missing Export.h. Verified by removal: + # Build/full/{Libraries,Services,UI} moved off the machine, all six + # binaries rebuilt from scratch, --headless=text and + # --headless=layout-tree byte-identical to the CMake reference. ":generated_libraries_headers", ":generated_services_headers", ":generated_shader_headers", - "//Build/full/Libraries:generated_lib_headers", - "//Build/full/Services:generated_service_headers", + ":generated_export_headers", + ":generated_ak_headers", + "//Libraries/LibWeb:generated_export_header", ], ) VCPKG = "//Meta/vcpkg" -# Configure-generated headers live in Build/full/AK; expose as AK/*.h via a copy. -genrule( - name = "ak_gen_headers", - srcs = ["Build/full/AK/Debug.h", "Build/full/AK/Backtrace.h"], - outs = ["genroot/AK/Debug.h", "genroot/AK/Backtrace.h"], - cmd = "mkdir -p $(RULEDIR)/genroot/AK && " + - "cp $(location Build/full/AK/Debug.h) $(RULEDIR)/genroot/AK/Debug.h && " + - "cp $(location Build/full/AK/Backtrace.h) $(RULEDIR)/genroot/AK/Backtrace.h", -) +# NOTE: the ak_gen_headers genrule that COPIED AK/Debug.h and AK/Backtrace.h out +# of Build/full is gone. Bazel now generates both from their checked-in .in +# templates (export_headers.bzl): Debug.h by applying CMake's configure_file +# substitution, Backtrace.h by ASKING the host the same question +# find_package(Backtrace) asks. That removed the last AK dependency on a CMake +# build tree. cc_library( name = "AK", @@ -94,6 +112,7 @@ cc_library( 'AK/StringConversions.cpp', 'AK/StringUtils.cpp', 'AK/StringView.cpp', + 'AK/ThreadID.cpp', 'AK/Time.cpp', 'AK/Utf16FlyString.cpp', 'AK/Utf16String.cpp', @@ -103,7 +122,7 @@ cc_library( 'AK/Utf8View.cpp', 'AK/kmalloc.cpp', ], - hdrs = glob(["AK/*.h"]) + [":ak_gen_headers"], + hdrs = glob(["AK/*.h"]), # AK-private defines (CMake PRIVATE) — local_defines so they don't leak to # consumers. FMT_SHARED/AK_HAS_CPPTRACE only affect AK's own TUs. local_defines = [ @@ -111,8 +130,13 @@ cc_library( "AK_HAS_CPPTRACE=1", "FMT_SHARED", ], - includes = ["genroot"], deps = [ + # AK/Debug.h + AK/Backtrace.h, generated from their .in templates + # (export_headers.bzl). A cc_library belongs on a deps edge, not in hdrs: + # in hdrs it contributes no include path, which is why every consumer + # failed with "AK/Debug.h: No such file or directory" until this moved. + # The include root travels with the dep, so dependents get it too. + ":generated_ak_headers", VCPKG + ":fmt", VCPKG + ":simdutf", VCPKG + ":mimalloc", @@ -318,7 +342,7 @@ cc_library( ], ) -# === LibGC (shared_library, 19 TU) === +# === LibGC (shared_library, 20 TU) === cc_library( name = 'LibGC', srcs = [ @@ -330,6 +354,7 @@ cc_library( 'Libraries/LibGC/ConservativeRangeProvider.cpp', 'Libraries/LibGC/ConservativeVector.cpp', 'Libraries/LibGC/CrossHeapMember.cpp', + 'Libraries/LibGC/ExternalEntityTable.cpp', 'Libraries/LibGC/Heap.cpp', 'Libraries/LibGC/HeapBlock.cpp', 'Libraries/LibGC/HeapGroup.cpp', @@ -421,8 +446,8 @@ cc_library( ], hdrs = glob(["Libraries/LibGfx/**/*.h"], allow_empty = True), local_defines = ['LibGfx_EXPORTS', 'SK_CODEC_DECODES_BMP', 'SK_CODEC_DECODES_GIF', 'SK_CODEC_DECODES_WBMP', 'SK_DISABLE_TRACING', 'SK_ENABLE_AVX512_OPTS', 'SK_ENABLE_PRECOMPILE', 'SK_FONTMGR_ANDROID_AVAILABLE', 'SK_FONTMGR_FCI_AVAILABLE', 'SK_FONTMGR_FONTCONFIG_AVAILABLE', 'SK_FONTMGR_FREETYPE_DIRECTORY_AVAILABLE', 'SK_FONTMGR_FREETYPE_EMBEDDED_AVAILABLE', 'SK_FONTMGR_FREETYPE_EMPTY_AVAILABLE', 'SK_GAMMA_APPLY_TO_A8', 'SK_GANESH', 'SK_HAS_WUFFS_LIBRARY', 'SK_R32_SHIFT=16', 'SK_TYPEFACE_FACTORY_FREETYPE', 'SK_USE_PERFETTO', 'SK_VULKAN', 'SK_XML'], - copts = ['-IBuild/full/Libraries/LibGfx'], linkopts = ['-ldl', '-lpthread', '-lvulkan'], + implementation_deps = ['//:libgfx_rust_bare_include'], deps = [ '//:AK', '//:LibCompress', @@ -587,7 +612,6 @@ cc_library( cc_library( name = 'LibJS', srcs = [ - ':Libraries/LibJS/Bytecode/Op.cpp', ':Libraries/LibJS/Interpreter/interpreter_x86_64.S', 'Libraries/LibJS/Bytecode/Executable.cpp', 'Libraries/LibJS/Bytecode/IdentifierTable.cpp', @@ -603,6 +627,7 @@ cc_library( 'Libraries/LibJS/Contrib/Test262/GlobalObject.cpp', 'Libraries/LibJS/Contrib/Test262/IsHTMLDDA.cpp', 'Libraries/LibJS/CyclicModule.cpp', + 'Libraries/LibJS/Debugger.cpp', 'Libraries/LibJS/Heap/Cell.cpp', 'Libraries/LibJS/Interpreter/Interpreter.cpp', 'Libraries/LibJS/Interpreter/SlowPaths.cpp', @@ -853,7 +878,7 @@ cc_library( hdrs = glob(["Libraries/LibJS/**/*.h"], allow_empty = True), local_defines = ['LibJS_EXPORTS', 'SIMDJSON_DEVELOPMENT_CHECKS', 'SIMDJSON_THREADS_ENABLED=1'], additional_compiler_inputs = ['Libraries/LibJS/Runtime/JavaScriptImplementations/AbstractOperations.js', 'Libraries/LibJS/Runtime/JavaScriptImplementations/ArrayConstructor.js'], - copts = ['-fno-omit-frame-pointer', '-fvisibility=hidden', '-IBuild/full/Libraries/LibJS'], + copts = ['-fno-omit-frame-pointer', '-fvisibility=hidden'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -887,12 +912,12 @@ cc_library( ], ) -# === LibMedia (shared_library, 45 TU) === +# === LibMedia (shared_library, 54 TU) === cc_library( name = 'LibMedia', srcs = [ - 'Libraries/LibMedia/Audio/AudioBuffer.cpp', 'Libraries/LibMedia/Audio/AudioDevices.cpp', + 'Libraries/LibMedia/Audio/AudioRingBuffer.cpp', 'Libraries/LibMedia/Audio/NullPlaybackStream.cpp', 'Libraries/LibMedia/Audio/PlaybackStream.cpp', 'Libraries/LibMedia/Audio/PlaybackStreamPulseAudio.cpp', @@ -919,8 +944,9 @@ cc_library( 'Libraries/LibMedia/FFmpeg/FFmpegHelpers.cpp', 'Libraries/LibMedia/FFmpeg/FFmpegIOContext.cpp', 'Libraries/LibMedia/FFmpeg/FFmpegVideoDecoder.cpp', - 'Libraries/LibMedia/GenericTimeProvider.cpp', 'Libraries/LibMedia/IncrementallyPopulatedStream.cpp', + 'Libraries/LibMedia/MediaTime.cpp', + 'Libraries/LibMedia/MonotonicMediaClock.cpp', 'Libraries/LibMedia/PlaybackManager.cpp', 'Libraries/LibMedia/PlaybackStates/BufferingStateHandler.cpp', 'Libraries/LibMedia/PlaybackStates/PausedStateHandler.cpp', @@ -932,10 +958,18 @@ cc_library( 'Libraries/LibMedia/Processors/AudioTimeStretchProcessor.cpp', 'Libraries/LibMedia/Producers/DecodedAudioProducer.cpp', 'Libraries/LibMedia/Producers/DecodedVideoProducer.cpp', + 'Libraries/LibMedia/Producers/RemoteVideoProducer.cpp', 'Libraries/LibMedia/Sinks/AudioPlaybackSink.cpp', 'Libraries/LibMedia/Sinks/DisplayingVideoSink.cpp', + 'Libraries/LibMedia/Sinks/RemoteVideoSink.cpp', 'Libraries/LibMedia/TimeRanges.cpp', + 'Libraries/LibMedia/VideoEdgeQueue.cpp', 'Libraries/LibMedia/VideoFrame.cpp', + 'Libraries/LibMedia/VideoFrameHandle.cpp', + 'Libraries/LibMedia/VideoFramePool.cpp', + 'Libraries/LibMedia/VideoPresentation/PresentedFramePage.cpp', + 'Libraries/LibMedia/VideoPresentation/VideoPresentationClientConnection.cpp', + 'Libraries/LibMedia/VideoPresentation/VideoPresentationServerConnection.cpp', ], hdrs = glob(["Libraries/LibMedia/**/*.h"], allow_empty = True), local_defines = ['LIBMEDIA_AUDIO_BACKEND=1', 'LibMedia_EXPORTS', '_REENTRANT'], @@ -968,7 +1002,8 @@ cc_library( ], hdrs = glob(["Libraries/LibRegex/**/*.h"], allow_empty = True), local_defines = ['LibRegex_EXPORTS'], - copts = ['-fvisibility=hidden', '-IBuild/full/Libraries/LibRegex'], + copts = ['-fvisibility=hidden'], + implementation_deps = ['//:libregex_rust_bare_include'], deps = [ '//:AK', '//:LibUnicode', @@ -1076,7 +1111,8 @@ cc_library( ], hdrs = glob(["Libraries/LibTextCodec/**/*.h"], allow_empty = True), local_defines = ['LibTextCodec_EXPORTS'], - copts = ['-fvisibility=hidden', '-IBuild/full/Libraries/LibTextCodec'], + copts = ['-fvisibility=hidden'], + implementation_deps = ['//:libtextcodec_rust_bare_include'], deps = [ '//:AK', '//:all_source_headers', @@ -1116,7 +1152,6 @@ cc_library( ], hdrs = glob(["Libraries/LibURL/**/*.h"], allow_empty = True), local_defines = ['ENABLE_RUST', 'LibURL_EXPORTS'], - copts = ['-IBuild/full/Libraries/LibURL'], deps = [ '//:AK', '//:LibRegex', @@ -1159,7 +1194,6 @@ cc_library( hdrs = glob(["Libraries/LibUnicode/**/*.h"], allow_empty = True), alwayslink = True, local_defines = ['LibUnicode_EXPORTS'], - copts = ['-IBuild/full/Libraries/LibUnicode'], linkopts = ['-ldl'], deps = [ '//:AK', @@ -1201,9 +1235,11 @@ cc_library( 'Libraries/LibWasm/WASI/Wasi.cpp', ], hdrs = glob(["Libraries/LibWasm/**/*.h"], allow_empty = True), - local_defines = ['LibWasm_EXPORTS', 'WASM_COMPILED_FAULT_RECOVERY_SUPPORTED=1', 'WASM_CRANELIFT=1', 'WASM_CRANELIFT_COMPILER_PATH=\\"/home/ubuntu/ladybird-work/Build/full/bin/cranelift-compiler\\"'], - copts = ['-fvisibility=hidden', '-IBuild/full/Libraries/LibWasm'], + local_defines = ['LibWasm_EXPORTS', 'WASM_COMPILED_FAULT_RECOVERY_SUPPORTED=1', 'WASM_CRANELIFT=1', 'WASM_CRANELIFT_COMPILER_PATH=\\"cranelift-compiler\\"'], + copts = ['-fvisibility=hidden'], linkopts = ['-lvulkan'], + data = ['//:cranelift-compiler'], + implementation_deps = ['//:libwasm_cranelift_bare_include'], deps = [ '//:AK', '//:LibCore', @@ -1212,6 +1248,7 @@ cc_library( '//:LibSync', '//:LibThreading', '//:all_source_headers', + '//:libwasm_cranelift_lib', ], ) @@ -1260,7 +1297,7 @@ cc_library( ], ) -# === LibDevTools (shared_library, 35 TU) === +# === LibDevTools (shared_library, 36 TU) === cc_library( name = 'LibDevTools', srcs = [ @@ -1297,6 +1334,7 @@ cc_library( 'Libraries/LibDevTools/Actors/WatcherActor.cpp', 'Libraries/LibDevTools/Connection.cpp', 'Libraries/LibDevTools/DevToolsServer.cpp', + 'Libraries/LibDevTools/FirefoxClient.cpp', 'Libraries/LibDevTools/IndexedDBSerialization.cpp', 'Libraries/LibDevTools/StorageHelpers.cpp', ], @@ -1307,6 +1345,7 @@ cc_library( deps = [ '//:AK', '//:LibCore', + '//:LibFileSystem', '//:LibHTTP', '//:LibURL', '//:all_source_headers', @@ -1314,11 +1353,12 @@ cc_library( ], ) -# === LibWebView (shared_library, 57 TU) === +# === LibWebView (shared_library, 64 TU) === cc_library( name = 'LibWebView', srcs = [ 'Libraries/LibWebView/Application.cpp', + 'Libraries/LibWebView/ApplyHistoryStep.cpp', 'Libraries/LibWebView/Attribute.cpp', 'Libraries/LibWebView/Autocomplete.cpp', 'Libraries/LibWebView/AutocompleteMuxer.cpp', @@ -1336,6 +1376,9 @@ cc_library( 'Libraries/LibWebView/DOMNodeProperties.cpp', 'Libraries/LibWebView/DictionaryLookup.cpp', 'Libraries/LibWebView/DownloadPresentation.cpp', + 'Libraries/LibWebView/DownloadSegmentation.cpp', + 'Libraries/LibWebView/DownloadStore.cpp', + 'Libraries/LibWebView/ExternalURLHandler.cpp', 'Libraries/LibWebView/FileDownloader.cpp', 'Libraries/LibWebView/Geolocation.cpp', 'Libraries/LibWebView/HSTSStore.cpp', @@ -1356,6 +1399,9 @@ cc_library( 'Libraries/LibWebView/Profile.cpp', 'Libraries/LibWebView/SearchEngine.cpp', 'Libraries/LibWebView/SessionHistory.cpp', + 'Libraries/LibWebView/SessionHistorySnapshotStorage.cpp', + 'Libraries/LibWebView/SessionHistoryTraversalQueue.cpp', + 'Libraries/LibWebView/SessionStore.cpp', 'Libraries/LibWebView/Settings.cpp', 'Libraries/LibWebView/SiteIsolation.cpp', 'Libraries/LibWebView/SiteIsolationManager.cpp', @@ -1422,7 +1468,6 @@ cc_library( 'Services/WebContent/WebUIConnection.cpp', ], hdrs = glob(["Services/WebContent/**/*.h"], allow_empty = True), - copts = ['-IBuild/full'], deps = [ '//:AK', '//:LibCore', @@ -1463,7 +1508,6 @@ cc_library( 'Services/RequestServer/WebSocketImplCurl.cpp', ], hdrs = glob(["Services/RequestServer/**/*.h"], allow_empty = True), - copts = ['-IBuild/full'], deps = [ '//:AK', '//:LibCore', @@ -1491,7 +1535,6 @@ cc_library( 'Services/ImageDecoder/ConnectionFromClient.cpp', ], hdrs = glob(["Services/ImageDecoder/**/*.h"], allow_empty = True), - copts = ['-IBuild/full'], deps = [ '//:AK', '//:LibCore', @@ -1526,7 +1569,6 @@ cc_library( ], hdrs = glob(["Services/Compositor/**/*.h"], allow_empty = True), local_defines = ['SK_CODEC_DECODES_BMP', 'SK_CODEC_DECODES_GIF', 'SK_CODEC_DECODES_WBMP', 'SK_DISABLE_TRACING', 'SK_ENABLE_AVX512_OPTS', 'SK_ENABLE_PRECOMPILE', 'SK_FONTMGR_ANDROID_AVAILABLE', 'SK_FONTMGR_FCI_AVAILABLE', 'SK_FONTMGR_FONTCONFIG_AVAILABLE', 'SK_FONTMGR_FREETYPE_DIRECTORY_AVAILABLE', 'SK_FONTMGR_FREETYPE_EMBEDDED_AVAILABLE', 'SK_FONTMGR_FREETYPE_EMPTY_AVAILABLE', 'SK_GAMMA_APPLY_TO_A8', 'SK_GANESH', 'SK_HAS_WUFFS_LIBRARY', 'SK_R32_SHIFT=16', 'SK_TYPEFACE_FACTORY_FREETYPE', 'SK_USE_PERFETTO', 'SK_VULKAN', 'SK_XML'], - copts = ['-IBuild/full'], deps = [ '//:LibCore', '//:LibGfx', @@ -1548,7 +1590,6 @@ cc_library( 'Services/WebWorker/WorkerHost.cpp', ], hdrs = glob(["Services/WebWorker/**/*.h"], allow_empty = True), - copts = ['-IBuild/full'], deps = [ '//:AK', '//:LibCore', @@ -1578,7 +1619,6 @@ cc_binary( 'Services/ImageDecoder/SandboxLinux.cpp', 'Services/ImageDecoder/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1602,7 +1642,6 @@ cc_binary( srcs = [ 'Services/RequestServer/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1635,7 +1674,6 @@ cc_binary( 'Services/Compositor/SandboxLinux.cpp', 'Services/Compositor/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-ldl', '-lpthread', '-lvulkan'], deps = [ '//:AK', @@ -1666,7 +1704,6 @@ cc_binary( 'Services/RendererSandboxLinux.cpp', 'Services/WebWorker/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1701,7 +1738,6 @@ cc_binary( 'Services/RendererSandboxLinux.cpp', 'Services/WebContent/main.cpp', ], - copts = ['-fPIE', '-IBuild/full'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1742,7 +1778,6 @@ cc_binary( 'Libraries/LibJS/Interpreter/GenerateLayout.cpp', ], local_defines = ['private=public', 'protected=public'], - copts = ['-fPIE'], deps = [ '//:AK', '//:all_source_headers', @@ -1756,7 +1791,7 @@ cc_binary( # or header filegroups are hand-maintained here. qt_cc_rcc compiles the .qrc. qt_cc_moc( name = "qt_moc", - hdrs = ["UI/Qt/Autocomplete.h", "UI/Qt/BookmarksBar.h", "UI/Qt/BrowserWindow.h", "UI/Qt/DevToolsBanner.h", "UI/Qt/EventLoopImplementationQtEventTarget.h", "UI/Qt/FindInPageWidget.h", "UI/Qt/LocationEdit.h", "UI/Qt/Settings.h", "UI/Qt/Tab.h", "UI/Qt/TabBar.h", "UI/Qt/WebContentView.h"], + hdrs = ["UI/Qt/Autocomplete.h", "UI/Qt/BookmarksBar.h", "UI/Qt/BrowserWindow.h", "UI/Qt/DevToolsBanner.h", "UI/Qt/EventLoopImplementationQtEventTarget.h", "UI/Qt/FindInPageWidget.h", "UI/Qt/GeolocationProviderQt.h", "UI/Qt/LocationEdit.h", "UI/Qt/Settings.h", "UI/Qt/Tab.h", "UI/Qt/TabBar.h", "UI/Qt/WebContentView.h"], ) qt_qrc( @@ -1770,7 +1805,25 @@ qt_cc_rcc( srcs = [":qt_qrc"], ) -# === ladybird (executable, 21 TU) === + +# === Qt6 RUNTIME: the plugins Qt dlopens, and the qt.conf that finds them === +# Not a CMake target -- CMake does not need one, because its binary links a Qt +# whose baked-in prefix already points at the plugins of that same Qt. Bazel's +# does not: it links @qt's libraries and then Qt looks for plugins next to the +# executable, finds none, and falls back to the HOST's plugin directory. Loading +# another Qt build's QPA plugin into this one is the Ubuntu 24.04 SIGSEGV in +# QXcbConnection::initializeScreens; on a box where the versions agree it "works". +# qt_runtime.bzl has the full account (finding 40); both targets below are data of +# //:ladybird, so they are staged in bazel-bin AND in the runfiles tree. +qt_plugin_tree( + name = "qt_plugins", + plugins = ["@qt_plugins//:plugins"], +) + +qt_conf( + name = "qt_conf", +) +# === ladybird (executable, 24 TU) === cc_binary( name = 'ladybird', srcs = [ @@ -1784,7 +1837,10 @@ cc_binary( 'UI/Qt/DevToolsBanner.cpp', 'UI/Qt/EventLoopImplementationQt.cpp', 'UI/Qt/EventLoopImplementationQtEventTarget.cpp', + 'UI/Qt/ExternalURLActivationToken.cpp', + 'UI/Qt/ExternalURLHandler.cpp', 'UI/Qt/FindInPageWidget.cpp', + 'UI/Qt/GeolocationProviderQt.cpp', 'UI/Qt/Icon.cpp', 'UI/Qt/LocationEdit.cpp', 'UI/Qt/Menu.cpp', @@ -1798,9 +1854,10 @@ cc_binary( 'UI/Qt/WindowControlButton.cpp', 'UI/Qt/main.cpp', ], - local_defines = ['QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_WIDGETS_LIB'], - copts = ['-fPIE', '-IBuild/full/UI', '-IBuild/full/UI/Qt'], - linkopts = ['-lGLX', '-lOpenGL', '-lvulkan'], + local_defines = ['LADYBIRD_QT_HAVE_POSITIONING=1', 'QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_POSITIONING_LIB', 'QT_WIDGETS_LIB'], + copts = ['-pthread'], + linkopts = ['-lGLX', '-lOpenGL', '-lgio-2.0', '-lglib-2.0', '-lgobject-2.0', '-lvulkan', '-lxkbcommon'], + data = [':Compositor', ':ImageDecoder', ':RequestServer', ':WebContent', ':WebWorker', ':qt_conf', ':qt_plugins'], deps = [ '//:AK', '//:LibCore', @@ -1812,6 +1869,7 @@ cc_binary( '//:LibMain', '//:LibRequests', '//:LibSync', + '//:LibThreading', '//:LibURL', '//:LibUnicode', '//:LibWakeLock', @@ -1822,7 +1880,9 @@ cc_binary( '//Meta/vcpkg:ssl', '@qt//:QtCore', '@qt//:QtGui', + '@qt//:QtPositioning', '@qt//:QtWidgets', + '@qt_plugins//:runtime_libs', ], ) @@ -1841,13 +1901,24 @@ cc_binary( vcpkg_tree( name = "vcpkg_installed", distfiles = VCPKG_DISTFILE_INDEX, + # The wheels for the Python packages a portfile pip-installs (angle asks for + # `ply`). Pinned separately from the 76 distfiles because pip bypasses vcpkg's + # asset cache entirely, so the capture cannot see them and x-block-origin + # cannot block them -- see vcpkg_python_packages.bzl and finding 36. + python_wheels = ['@vcpkg_pywheel_ply//file'], source_dir = ".", source_root = ":vcpkg_source_inputs", triplet = "x64-linux-dynamic", - # Resume cache: makes a killed 45-minute build cheap to restart. Absolute by - # necessity (the action's cwd is the execroot) and therefore a host escape -- - # it is a build-speed affordance, not part of the dependency graph. - cache_dir = "/home/ubuntu/.cache/vcpkg-bazel", + # Resume cache: OPT-IN, and empty here on purpose. Setting it makes a killed + # 45-minute vcpkg build cheap to restart, but the path must be absolute (the + # action's cwd is the execroot), so any value here is one developer's home + # directory in a file everyone checks out -- it was + # /home/ubuntu/.cache/vcpkg-bazel, the last absolute host path left in the + # emitted build. Empty is also the honest default: it is the genuine + # from-source build. Set this one attribute locally if you want resumability. + # It is a build-speed affordance and no part of the dependency graph, which is + # precisely why it must not be a checked-in constant. + cache_dir = "", vcpkg_root = "Build/vcpkg", vcpkg_tree = "//Build/vcpkg:tree", ) diff --git a/examples/ladybird/workspace/Build/full/Libraries/BUILD.bazel b/examples/ladybird/workspace/Build/full/Libraries/BUILD.bazel deleted file mode 100644 index d788bef..0000000 --- a/examples/ladybird/workspace/Build/full/Libraries/BUILD.bazel +++ /dev/null @@ -1,29 +0,0 @@ -load("@rules_cc//cc:defs.bzl", "cc_library") -package(default_visibility = ["//visibility:public"]) - -# NOTE: no rust_ffi_headers here any more. The 14 Rust FFI headers used to be -# globbed out of CMake's build tree (cargo writes them into -# Build/full/Libraries// via FFI_OUTPUT_DIR), so every Rust-consuming TU -# read a CMake artifact. Bazel now BUILDS the crates and declares those headers -# as the cargo_crate action's own outputs, consumed one target per crate -# (//:libweb_css_rust_lib etc., cargo_ring.bzl) -- so the include path arrives on -# the dep edge instead of a glob over a foreign tree. Verified by removal: with -# this gone AND Build/full/cargo + Build/full/bin/flapc moved off the machine, all -# six binaries still build and render identically. - -# All CMake-generated per-library headers materialized under Build/full/Libraries -# (generate_export_header -> /Export.h, plus other generated .h). Exposed as -# a header root so etc. resolve in the sandbox. -cc_library( - name = "generated_lib_headers", - hdrs = glob(["**/*.h"], allow_empty = True), - includes = ["."], -) - -# NOTE: no exports_files here any more. TIFFTagHandler.cpp, HSTSPreloadData.cpp, -# Op.cpp and interpreter_x86_64.S used to be exported from CMake's build tree and -# COMPILED straight into Bazel's libraries -- so the migration silently depended -# on CMake having produced those .cpp/.S files. Bazel now generates all four -# itself (codegen_root.bzl: gen_TIFFMetadata, gen_HSTSPreloadData, gen_Op, -# gen_interpreter_asm) and the consumers reference the genrule outputs. Verified -# by removal: deleting these entries keeps the build green. diff --git a/examples/ladybird/workspace/Build/full/Services/BUILD.bazel b/examples/ladybird/workspace/Build/full/Services/BUILD.bazel deleted file mode 100644 index fa12d04..0000000 --- a/examples/ladybird/workspace/Build/full/Services/BUILD.bazel +++ /dev/null @@ -1,15 +0,0 @@ -load("@rules_cc//cc:defs.bzl", "cc_library") -package(default_visibility = ["//visibility:public"]) - -# CMake-generated IPC endpoint headers materialized under Build/full/Services. -# Exposed as a header root so etc. -# resolve in the Bazel sandbox (matches CMake's -IBuild/full/Services). -cc_library( - name = "generated_service_headers", - hdrs = glob(["**/*.h"], allow_empty = True), - includes = ["."], -) -# NOTE: no exports_files here any more. WebGLCommandReplayer.cpp used to be -# compiled straight out of CMake's tree; it is now a genrule output in the root -# package (`:Services/Compositor/WebGLCommandReplayer.cpp`, see codegen_root.bzl). -# Verified by removal: deleting this line keeps //:Compositor green. diff --git a/examples/ladybird/workspace/Build/full/UI/BUILD.bazel b/examples/ladybird/workspace/Build/full/UI/BUILD.bazel deleted file mode 100644 index 78f2065..0000000 --- a/examples/ladybird/workspace/Build/full/UI/BUILD.bazel +++ /dev/null @@ -1,14 +0,0 @@ -package(default_visibility = ["//visibility:public"]) - -# Nothing from CMake's UI build dir is consumed any more. -# -# Qt moc/qrc are generated by Bazel via rules_qt (see //:qt_moc / //:qt_rcc in -# the root BUILD), and the two glslang shader headers -# (WebContentViewLinux{Frag,Vert}Shader.h) are now generated by Bazel too -# (//:gen_WebContentViewLinuxFragShader + :generated_shader_headers in -# codegen_root.bzl), so the qt_autogen_headers cc_library that used to glob them -# out of Build/full/UI/Qt is gone. Verified by removal: the build is green -# without it. -# -# This package is kept (empty) only because .bazelrc/others may reference the -# path; it declares no targets. diff --git a/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel b/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel index 5b3b6a7..18af3fe 100644 --- a/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel +++ b/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel @@ -1,48 +1,58 @@ load("@rules_cc//cc:defs.bzl", "cc_library") load(":codegen.bzl", "libweb_codegen", "libweb_bindings_codegen") load(":generated_srcs.bzl", "LIBWEB_GENERATED_SRCS", "LIBWEB_GENERATED_HDRS") +load(":export_header.bzl", "libweb_export_header") package(default_visibility = ["//visibility:public"]) +# LibWeb's generate_export_header output. It is generated HERE rather than in the +# root package because Bazel include dirs cannot escape a package: LibWeb is its +# own package, so its Export.h has to be an output of this package (at +# genroot/LibWeb/Export.h, with includes=["genroot"] -- includes=["../.."] is +# rejected by Bazel, and rightly so). Emitted by +# Meta/emit_export_headers_bazel.py --libweb. +libweb_export_header() + libweb_codegen() libweb_bindings_codegen() -# The four Rust crates that live INSIDE this package (LibWeb/Rust, -# LibWeb/CSS/Rust, LibWeb/Layout/Rust, LibWeb/ContentBlocker/Rust, plus -# HTML/Parser/Rust), exposed so the root package's cargo_ring() can declare them -# as cargo inputs. They are one cargo WORKSPACE with the crates at the repo root, -# but Bazel packages cut across it: glob() is package-relative, so the root -# package cannot see files under Libraries/LibWeb/ at all. Hence a filegroup on -# this side of the boundary rather than a glob on that side -- the alternative -# (making the root package own these files) would mean deleting this package. +# The Rust crates that live INSIDE this package, exposed so the root package's +# cargo_ring() can declare them as cargo inputs. They are one cargo WORKSPACE +# with the crates at the repo root, but Bazel packages cut across it: glob() is +# package-relative, so the root package cannot see files under Libraries/LibWeb/ +# at all. Hence a filegroup on this side of the boundary rather than a glob on +# that side -- the alternative (making the root package own these files) would +# mean deleting this package. +# +# The patterns are DERIVED from Cargo.toml (Meta/emit_cargo_bazel.py's +# crate_dirs(), the same list the root package globs), not written down. They +# used to be written down, and that broke the whole build: upstream consolidated +# libweb_css_rust and libweb_layout_rust into libweb_rust, so `CSS/Rust/**` and +# `Layout/Rust/**` matched nothing, and an allow_empty = False glob that matches +# nothing fails at LOADING time -- before any target exists to blame. filegroup( name = "rust_crate_srcs", - srcs = glob([ - "Rust/**", - "CSS/Rust/**", - "Layout/Rust/**", - "ContentBlocker/Rust/**", - "HTML/Parser/Rust/**", - ], allow_empty = False) + [ - # Non-Rust build-script inputs that live here too: libweb_css_rust's - # build.rs GENERATES Rust from these CSS data files, and libweb_rust's - # reads the HTML name headers + Entities.json. Taken from the reference - # build's cargo depfiles, so the list is measured rather than predicted. - "CSS/Enums.json", - "CSS/Keywords.json", - "CSS/LogicalPropertyGroups.json", - "CSS/Properties.json", - "CSS/PseudoClasses.json", - "CSS/PseudoElementPropertyGroups.txt", - "CSS/PseudoElements.json", - "CSS/Units.json", - "HTML/AttributeNames.h", - "HTML/Parser/Entities.json", - "HTML/TagNames.h", + srcs = glob(['ContentBlocker/Rust/**', 'HTML/Parser/Rust/**', 'Rust/**'], exclude = ['ContentBlocker/Rust/target/**', 'HTML/Parser/Rust/target/**', 'Rust/target/**'], allow_empty = False) + [ + # Non-Rust build-script inputs that live here too: libweb_rust's build.rs + # GENERATES Rust from the CSS data files and reads the HTML name headers + + # Entities.json. Taken from the reference build's cargo depfiles, so the + # list is measured rather than predicted. + 'CSS/Enums.json', + 'CSS/Keywords.json', + 'CSS/LogicalPropertyGroups.json', + 'CSS/Properties.json', + 'CSS/PseudoClasses.json', + 'CSS/PseudoElementPropertyGroups.txt', + 'CSS/PseudoElements.json', + 'CSS/TransformFunctions.json', + 'CSS/Units.json', + 'HTML/AttributeNames.h', + 'HTML/Parser/Entities.json', + 'HTML/TagNames.h', ], ) -# === LibWeb (shared_library, 1961 TU: 1273 checked-in + 688 generated) === +# === LibWeb (shared_library, 1922 TU: 1230 checked-in + 692 generated) === # SYSTEM libs (linkopts on binary): ['dl', 'pthread', 'vulkan'] cc_library( name = 'LibWeb', @@ -72,6 +82,8 @@ cc_library( 'Bindings/OptionConstructor.cpp', 'Bindings/PlatformObject.cpp', 'Bindings/PrincipalHostDefined.cpp', + 'Bindings/Wrappable.cpp', + 'Bindings/WrapperWorld.cpp', 'CSS/Angle.cpp', 'CSS/AnimationEvent.cpp', 'CSS/BooleanExpression.cpp', @@ -139,7 +151,7 @@ cc_library( 'CSS/CascadedProperties.cpp', 'CSS/Clip.cpp', 'CSS/ColorFunctionDescriptor.cpp', - 'CSS/ComputedProperties.cpp', + 'CSS/ComputedStyleWorkingSet.cpp', 'CSS/ComputedValues.cpp', 'CSS/ContainerQuery.cpp', 'CSS/CounterStyle.cpp', @@ -153,7 +165,6 @@ cc_library( 'CSS/EdgeRect.cpp', 'CSS/FeatureQuery.cpp', 'CSS/Fetch.cpp', - 'CSS/Filter.cpp', 'CSS/Flex.cpp', 'CSS/FontComputer.cpp', 'CSS/FontFace.cpp', @@ -162,28 +173,23 @@ cc_library( 'CSS/FontFeatureData.cpp', 'CSS/FontLoading.cpp', 'CSS/Frequency.cpp', + 'CSS/GeneratedContent.cpp', 'CSS/GridTrackPlacement.cpp', 'CSS/GridTrackSize.cpp', 'CSS/HypotheticalElement.cpp', 'CSS/Invalidation/AdoptedStyleSheetInvalidator.cpp', 'CSS/Invalidation/AttributeInvalidator.cpp', + 'CSS/Invalidation/ContainerQueryInvalidator.cpp', 'CSS/Invalidation/CustomElementInvalidator.cpp', 'CSS/Invalidation/ElementStateInvalidator.cpp', 'CSS/Invalidation/EmbeddedContentInvalidator.cpp', 'CSS/Invalidation/FormControlInvalidator.cpp', - 'CSS/Invalidation/HasMutationFeatureCollector.cpp', - 'CSS/Invalidation/HasMutationInvalidator.cpp', - 'CSS/Invalidation/InvalidationSetMatcher.cpp', 'CSS/Invalidation/LanguageInvalidator.cpp', 'CSS/Invalidation/LinkInvalidator.cpp', 'CSS/Invalidation/MediaQueryInvalidator.cpp', - 'CSS/Invalidation/NodeInvalidator.cpp', 'CSS/Invalidation/PartInvalidator.cpp', 'CSS/Invalidation/PseudoClassInvalidator.cpp', 'CSS/Invalidation/SlotInvalidator.cpp', - 'CSS/Invalidation/StructuralMutationInvalidator.cpp', - 'CSS/Invalidation/StyleInvalidator.cpp', - 'CSS/InvalidationSet.cpp', 'CSS/Length.cpp', 'CSS/LengthBox.cpp', 'CSS/MediaList.cpp', @@ -217,6 +223,7 @@ cc_library( 'CSS/PreferredColorScheme.cpp', 'CSS/Ratio.cpp', 'CSS/Resolution.cpp', + 'CSS/ResolvedTransform.cpp', 'CSS/RustStyleBridge.cpp', 'CSS/Screen.cpp', 'CSS/ScreenOrientation.cpp', @@ -226,8 +233,9 @@ cc_library( 'CSS/Serialize.cpp', 'CSS/Sizing.cpp', 'CSS/StyleComputer.cpp', + 'CSS/StyleEngineBridge.cpp', + 'CSS/StyleEngineInput.cpp', 'CSS/StyleInvalidation.cpp', - 'CSS/StyleInvalidationData.cpp', 'CSS/StyleProperty.cpp', 'CSS/StylePropertyMap.cpp', 'CSS/StylePropertyMapReadOnly.cpp', @@ -375,7 +383,6 @@ cc_library( 'Crypto/CryptoAlgorithms.cpp', 'Crypto/CryptoBindings.cpp', 'Crypto/CryptoKey.cpp', - 'Crypto/KeyAlgorithms.cpp', 'Crypto/SubtleCrypto.cpp', 'DOM/AbortController.cpp', 'DOM/AbortSignal.cpp', @@ -475,12 +482,10 @@ cc_library( 'Encoding/TextEncoderStream.cpp', 'EncryptedMediaExtensions/Algorithms.cpp', 'EncryptedMediaExtensions/MediaKeySystemAccess.cpp', - 'EncryptedMediaExtensions/NavigatorEncryptedMediaExtensionsPartial.cpp', 'EntriesAPI/FileSystemEntry.cpp', 'EventTiming/PerformanceEventTiming.cpp', 'Fetch/Body.cpp', 'Fetch/BodyInit.cpp', - 'Fetch/Enums.cpp', 'Fetch/FetchMethod.cpp', 'Fetch/Fetching/Checks.cpp', 'Fetch/Fetching/FetchedDataReceiver.cpp', @@ -573,9 +578,11 @@ cc_library( 'HTML/CrossOrigin/AbstractOperations.cpp', 'HTML/CrossOrigin/Reporting.cpp', 'HTML/CrossProcessId.cpp', + 'HTML/CustomElements/CustomElementAlgorithms.cpp', 'HTML/CustomElements/CustomElementDefinition.cpp', 'HTML/CustomElements/CustomElementName.cpp', 'HTML/CustomElements/CustomElementReactionNames.cpp', + 'HTML/CustomElements/CustomElementReactions.cpp', 'HTML/CustomElements/CustomElementRegistry.cpp', 'HTML/CustomElements/CustomStateSet.cpp', 'HTML/DOMParser.cpp', @@ -688,6 +695,7 @@ cc_library( 'HTML/HTMLVideoElement.cpp', 'HTML/HashChangeEvent.cpp', 'HTML/History.cpp', + 'HTML/HistoryOperation.cpp', 'HTML/HyperlinkElementUtils.cpp', 'HTML/ImageBitmap.cpp', 'HTML/ImageData.cpp', @@ -708,6 +716,7 @@ cc_library( 'HTML/NavigableContainer.cpp', 'HTML/NavigateEvent.cpp', 'HTML/Navigation.cpp', + 'HTML/NavigationActivation.cpp', 'HTML/NavigationCurrentEntryChangeEvent.cpp', 'HTML/NavigationDestination.cpp', 'HTML/NavigationHistoryEntry.cpp', @@ -724,6 +733,7 @@ cc_library( 'HTML/OffscreenCanvas.cpp', 'HTML/OffscreenCanvasRenderingContext2D.cpp', 'HTML/POSTResource.cpp', + 'HTML/PageSwapEvent.cpp', 'HTML/PageTransitionEvent.cpp', 'HTML/Parser/Entities.cpp', 'HTML/Parser/HTMLEncodingDetection.cpp', @@ -760,13 +770,13 @@ cc_library( 'HTML/Scripting/SimilarOriginWindowAgent.cpp', 'HTML/Scripting/TemporaryExecutionContext.cpp', 'HTML/Scripting/WindowEnvironmentSettingsObject.cpp', + 'HTML/Scripting/WindowRealm.cpp', 'HTML/Scripting/WorkerAgent.cpp', 'HTML/Scripting/WorkerEnvironmentSettingsObject.cpp', 'HTML/SelectItem.cpp', 'HTML/SelectedFile.cpp', 'HTML/SerializedPolicyContainer.cpp', 'HTML/SessionHistoryEntry.cpp', - 'HTML/SessionHistoryTraversalQueue.cpp', 'HTML/SharedResourceRequest.cpp', 'HTML/SharedWorker.cpp', 'HTML/SharedWorkerGlobalScope.cpp', @@ -843,59 +853,16 @@ cc_library( 'Internals/XRTest.cpp', 'IntersectionObserver/IntersectionObserver.cpp', 'IntersectionObserver/IntersectionObserverEntry.cpp', - 'Layout/AudioBox.cpp', - 'Layout/AvailableSpace.cpp', 'Layout/BlockContainer.cpp', - 'Layout/BlockFormattingContext.cpp', 'Layout/Box.cpp', - 'Layout/BreakNode.cpp', - 'Layout/CanvasBox.cpp', - 'Layout/CheckBox.cpp', - 'Layout/FieldSetBox.cpp', - 'Layout/FlexFormattingContext.cpp', - 'Layout/FormattingContext.cpp', - 'Layout/GridFormattingContext.cpp', - 'Layout/ImageBox.cpp', 'Layout/ImageProvider.cpp', - 'Layout/InlineFormattingContext.cpp', - 'Layout/InlineLevelIterator.cpp', - 'Layout/InlineNode.cpp', - 'Layout/LayoutState.cpp', - 'Layout/LegendBox.cpp', - 'Layout/LineBox.cpp', - 'Layout/LineBoxFragment.cpp', - 'Layout/LineBuilder.cpp', - 'Layout/ListItemBox.cpp', - 'Layout/ListItemMarkerBox.cpp', - 'Layout/NavigableContainerViewport.cpp', + 'Layout/LayoutRustBridge.cpp', 'Layout/Node.cpp', 'Layout/NodeArena.cpp', - 'Layout/RadioButton.cpp', - 'Layout/RangeInputBox.cpp', - 'Layout/ReplacedBox.cpp', - 'Layout/ReplacedWithChildrenFormattingContext.cpp', - 'Layout/SVGBox.cpp', - 'Layout/SVGClipBox.cpp', - 'Layout/SVGForeignObjectBox.cpp', - 'Layout/SVGFormattingContext.cpp', - 'Layout/SVGGeometryBox.cpp', - 'Layout/SVGGraphicsBox.cpp', - 'Layout/SVGImageBox.cpp', - 'Layout/SVGMaskBox.cpp', - 'Layout/SVGPatternBox.cpp', - 'Layout/SVGSVGBox.cpp', - 'Layout/SVGTextBox.cpp', - 'Layout/SVGTextPathBox.cpp', 'Layout/ScrollableOverflow.cpp', - 'Layout/TableFormattingContext.cpp', - 'Layout/TableGrid.cpp', - 'Layout/TableWrapper.cpp', - 'Layout/TextAreaBox.cpp', - 'Layout/TextInputBox.cpp', 'Layout/TextNode.cpp', 'Layout/TextOffsetMapping.cpp', 'Layout/TreeBuilder.cpp', - 'Layout/VideoBox.cpp', 'Layout/Viewport.cpp', 'Loader/ContentBlocker.cpp', 'Loader/DownloadFilename.cpp', @@ -968,7 +935,6 @@ cc_library( 'Painting/HitTestDisplayList.cpp', 'Painting/ImagePaintable.cpp', 'Painting/InlinePaintable.cpp', - 'Painting/MarkerPaintable.cpp', 'Painting/NavigableContainerViewportPaintable.cpp', 'Painting/PaintStyle.cpp', 'Painting/Paintable.cpp', @@ -1069,6 +1035,7 @@ cc_library( 'SVG/SVGImageElement.cpp', 'SVG/SVGLength.cpp', 'SVG/SVGLengthList.cpp', + 'SVG/SVGLengthValue.cpp', 'SVG/SVGLineElement.cpp', 'SVG/SVGLinearGradientElement.cpp', 'SVG/SVGList.cpp', @@ -1350,8 +1317,6 @@ cc_library( '//:LibXML', '//:all_source_headers', '//:libweb_content_blocker_rust_lib', - '//:libweb_css_rust_lib', - '//:libweb_layout_rust_lib', '//:libweb_rust_lib', '//Meta/vcpkg:SDL3', '//Meta/vcpkg:crypto', diff --git a/examples/ladybird/workspace/Libraries/LibWeb/codegen.bzl b/examples/ladybird/workspace/Libraries/LibWeb/codegen.bzl index 0fa57f7..0b6273a 100644 --- a/examples/ladybird/workspace/Libraries/LibWeb/codegen.bzl +++ b/examples/ladybird/workspace/Libraries/LibWeb/codegen.bzl @@ -160,13 +160,13 @@ def libweb_codegen(): ) def libweb_bindings_codegen(): - # Mega-genrule: 661 IDL args (659 static + 2 generated) -> 1331 files. - # Byte-identical to CMake, proven 1331/1331 by Meta/bazel_parity_harness.py. - # srcs is the FULL DEPENDS closure (661 .idl), not just the 661 args: + # Mega-genrule: 663 IDL args (661 static + 2 generated) -> 1340 files. + # Byte-identical to CMake, proven 1340/1340 by Meta/bazel_parity_harness.py. + # srcs is the FULL DEPENDS closure (663 .idl), not just the 663 args: # the generator follows `includes`/partial interfaces between .idl files. native.genrule( name = 'gen_bindings', - srcs = ['ARIA/ARIAMixin.idl', 'Animations/Animatable.idl', 'Animations/Animation.idl', 'Animations/AnimationEffect.idl', 'Animations/AnimationPlaybackEvent.idl', 'Animations/AnimationTimeline.idl', 'Animations/DocumentTimeline.idl', 'Animations/KeyframeEffect.idl', 'Animations/ScrollTimeline.idl', 'CSS/AnimationEvent.idl', 'CSS/CSS.idl', 'CSS/CSSAnimation.idl', 'CSS/CSSConditionRule.idl', 'CSS/CSSContainerRule.idl', 'CSS/CSSCounterStyleRule.idl', 'CSS/CSSFontFaceDescriptors.idl', 'CSS/CSSFontFaceRule.idl', 'CSS/CSSFontFeatureValuesMap.idl', 'CSS/CSSFontFeatureValuesRule.idl', 'CSS/CSSFunctionDeclarations.idl', 'CSS/CSSFunctionDescriptors.idl', 'CSS/CSSFunctionRule.idl', 'CSS/CSSGroupingRule.idl', 'CSS/CSSImageValue.idl', 'CSS/CSSImportRule.idl', 'CSS/CSSKeyframeRule.idl', 'CSS/CSSKeyframesRule.idl', 'CSS/CSSKeywordValue.idl', 'CSS/CSSLayerBlockRule.idl', 'CSS/CSSLayerStatementRule.idl', 'CSS/CSSMarginRule.idl', 'CSS/CSSMathClamp.idl', 'CSS/CSSMathInvert.idl', 'CSS/CSSMathMax.idl', 'CSS/CSSMathMin.idl', 'CSS/CSSMathNegate.idl', 'CSS/CSSMathProduct.idl', 'CSS/CSSMathSum.idl', 'CSS/CSSMathValue.idl', 'CSS/CSSMatrixComponent.idl', 'CSS/CSSMediaRule.idl', 'CSS/CSSNamespaceRule.idl', 'CSS/CSSNestedDeclarations.idl', 'CSS/CSSNumericArray.idl', 'CSS/CSSNumericValue.idl', 'CSS/CSSPageDescriptors.idl', 'CSS/CSSPageRule.idl', 'CSS/CSSPerspective.idl', 'CSS/CSSPropertyRule.idl', 'CSS/CSSRotate.idl', 'CSS/CSSRule.idl', 'CSS/CSSRuleList.idl', 'CSS/CSSScale.idl', 'CSS/CSSScopeRule.idl', 'CSS/CSSSkew.idl', 'CSS/CSSSkewX.idl', 'CSS/CSSSkewY.idl', 'CSS/CSSStyleDeclaration.idl', 'CSS/CSSStyleProperties.idl', 'CSS/CSSStyleRule.idl', 'CSS/CSSStyleSheet.idl', 'CSS/CSSStyleValue.idl', 'CSS/CSSSupportsRule.idl', 'CSS/CSSTransformComponent.idl', 'CSS/CSSTransformValue.idl', 'CSS/CSSTransition.idl', 'CSS/CSSTranslate.idl', 'CSS/CSSUnitValue.idl', 'CSS/CSSUnparsedValue.idl', 'CSS/CSSVariableReferenceValue.idl', 'CSS/ElementCSSInlineStyle.idl', 'CSS/FontFace.idl', 'CSS/FontFaceSet.idl', 'CSS/FontFaceSetLoadEvent.idl', 'CSS/LinkStyle.idl', 'CSS/MediaList.idl', 'CSS/MediaQueryList.idl', 'CSS/MediaQueryListEvent.idl', 'CSS/Screen.idl', 'CSS/ScreenOrientation.idl', 'CSS/StylePropertyMap.idl', 'CSS/StylePropertyMapReadOnly.idl', 'CSS/StyleSheet.idl', 'CSS/StyleSheetList.idl', 'CSS/TransitionEvent.idl', 'CSS/VisualViewport.idl', 'Clipboard/Clipboard.idl', 'Clipboard/ClipboardEvent.idl', 'Clipboard/ClipboardItem.idl', 'Compression/CompressionStream.idl', 'Compression/DecompressionStream.idl', 'ContentSecurityPolicy/SecurityPolicyViolationEvent.idl', 'CookieStore/CookieChangeEvent.idl', 'CookieStore/CookieStore.idl', 'CredentialManagement/Credential.idl', 'CredentialManagement/CredentialsContainer.idl', 'CredentialManagement/FederatedCredential.idl', 'CredentialManagement/PasswordCredential.idl', 'Crypto/Crypto.idl', 'Crypto/CryptoKey.idl', 'Crypto/SubtleCrypto.idl', 'DOM/AbortController.idl', 'DOM/AbortSignal.idl', 'DOM/AbstractRange.idl', 'DOM/Attr.idl', 'DOM/CDATASection.idl', 'DOM/CaretPosition.idl', 'DOM/CharacterData.idl', 'DOM/ChildNode.idl', 'DOM/Comment.idl', 'DOM/CustomEvent.idl', 'DOM/DOMImplementation.idl', 'DOM/DOMTokenList.idl', 'DOM/Document.idl', 'DOM/DocumentFragment.idl', 'DOM/DocumentOrShadowRoot.idl', 'DOM/DocumentType.idl', 'DOM/Element.idl', 'DOM/Event.idl', 'DOM/EventHandler.idl', 'DOM/EventListener.idl', 'DOM/EventTarget.idl', 'DOM/HTMLCollection.idl', 'DOM/MutationObserver.idl', 'DOM/MutationRecord.idl', 'DOM/NamedNodeMap.idl', 'DOM/Node.idl', 'DOM/NodeFilter.idl', 'DOM/NodeIterator.idl', 'DOM/NodeList.idl', 'DOM/ParentNode.idl', 'DOM/ProcessingInstruction.idl', 'DOM/Range.idl', 'DOM/ShadowRoot.idl', 'DOM/Slottable.idl', 'DOM/StaticRange.idl', 'DOM/Text.idl', 'DOM/TreeWalker.idl', 'DOM/XMLDocument.idl', 'DOMURL/DOMURL.idl', 'DOMURL/Origin.idl', 'DOMURL/URLSearchParams.idl', 'Encoding/TextDecoder.idl', 'Encoding/TextDecoderCommon.idl', 'Encoding/TextDecoderStream.idl', 'Encoding/TextEncoder.idl', 'Encoding/TextEncoderCommon.idl', 'Encoding/TextEncoderStream.idl', 'EncryptedMediaExtensions/MediaKeySystemAccess.idl', 'EntriesAPI/FileSystemEntry.idl', 'EventTiming/PerformanceEventTiming.idl', 'Fetch/Body.idl', 'Fetch/BodyInit.idl', 'Fetch/Headers.idl', 'Fetch/Request.idl', 'Fetch/Response.idl', 'FileAPI/Blob.idl', 'FileAPI/File.idl', 'FileAPI/FileList.idl', 'FileAPI/FileReader.idl', 'FileAPI/FileReaderSync.idl', 'Fullscreen/DocumentExtensions.idl', 'Fullscreen/DocumentOrShadowRootExtensions.idl', 'Fullscreen/ElementExtensions.idl', 'GPC/GlobalPrivacyControl.idl', 'Gamepad/Gamepad.idl', 'Gamepad/GamepadButton.idl', 'Gamepad/GamepadEvent.idl', 'Gamepad/GamepadHapticActuator.idl', 'Geolocation/Geolocation.idl', 'Geolocation/GeolocationCoordinates.idl', 'Geolocation/GeolocationPosition.idl', 'Geolocation/GeolocationPositionError.idl', 'Geometry/DOMMatrix.idl', 'Geometry/DOMMatrixReadOnly.idl', 'Geometry/DOMPoint.idl', 'Geometry/DOMPointReadOnly.idl', 'Geometry/DOMQuad.idl', 'Geometry/DOMRect.idl', 'Geometry/DOMRectList.idl', 'Geometry/DOMRectReadOnly.idl', 'HTML/AbstractWorker.idl', 'HTML/AnimationFrameProvider.idl', 'HTML/AudioTrack.idl', 'HTML/AudioTrackList.idl', 'HTML/BarProp.idl', 'HTML/BeforeUnloadEvent.idl', 'HTML/BroadcastChannel.idl', 'HTML/Canvas/CanvasCompositing.idl', 'HTML/Canvas/CanvasDrawImage.idl', 'HTML/Canvas/CanvasDrawPath.idl', 'HTML/Canvas/CanvasFillStrokeStyles.idl', 'HTML/Canvas/CanvasFilters.idl', 'HTML/Canvas/CanvasImageData.idl', 'HTML/Canvas/CanvasImageSmoothing.idl', 'HTML/Canvas/CanvasPath.idl', 'HTML/Canvas/CanvasPathDrawingStyles.idl', 'HTML/Canvas/CanvasRect.idl', 'HTML/Canvas/CanvasSettings.idl', 'HTML/Canvas/CanvasShadowStyles.idl', 'HTML/Canvas/CanvasState.idl', 'HTML/Canvas/CanvasText.idl', 'HTML/Canvas/CanvasTextDrawingStyles.idl', 'HTML/Canvas/CanvasTransform.idl', 'HTML/Canvas/CanvasUserInterface.idl', 'HTML/Canvas/OffscreenCanvasBase.idl', 'HTML/CanvasGradient.idl', 'HTML/CanvasPattern.idl', 'HTML/CanvasRenderingContext2D.idl', 'HTML/CanvasRenderingContext2DSettings.idl', 'HTML/CloseEvent.idl', 'HTML/CloseWatcher.idl', 'HTML/CommandEvent.idl', 'HTML/CustomElements/CustomElementRegistry.idl', 'HTML/CustomElements/CustomStateSet.idl', 'HTML/DOMParser.idl', 'HTML/DOMStringList.idl', 'HTML/DOMStringMap.idl', 'HTML/DataTransfer.idl', 'HTML/DataTransferItem.idl', 'HTML/DataTransferItemList.idl', 'HTML/DedicatedWorkerGlobalScope.idl', 'HTML/DragEvent.idl', 'HTML/ElementInternals.idl', 'HTML/ErrorEvent.idl', 'HTML/EventSource.idl', 'HTML/External.idl', 'HTML/FormDataEvent.idl', 'HTML/HTMLAllCollection.idl', 'HTML/HTMLAnchorElement.idl', 'HTML/HTMLAreaElement.idl', 'HTML/HTMLAudioElement.idl', 'HTML/HTMLBRElement.idl', 'HTML/HTMLBaseElement.idl', 'HTML/HTMLBodyElement.idl', 'HTML/HTMLButtonElement.idl', 'HTML/HTMLCanvasElement.idl', 'HTML/HTMLDListElement.idl', 'HTML/HTMLDataElement.idl', 'HTML/HTMLDataListElement.idl', 'HTML/HTMLDetailsElement.idl', 'HTML/HTMLDialogElement.idl', 'HTML/HTMLDirectoryElement.idl', 'HTML/HTMLDivElement.idl', 'HTML/HTMLDocument.idl', 'HTML/HTMLElement.idl', 'HTML/HTMLEmbedElement.idl', 'HTML/HTMLFieldSetElement.idl', 'HTML/HTMLFontElement.idl', 'HTML/HTMLFormControlsCollection.idl', 'HTML/HTMLFormElement.idl', 'HTML/HTMLFrameElement.idl', 'HTML/HTMLFrameSetElement.idl', 'HTML/HTMLHRElement.idl', 'HTML/HTMLHeadElement.idl', 'HTML/HTMLHeadingElement.idl', 'HTML/HTMLHtmlElement.idl', 'HTML/HTMLHyperlinkElementUtils.idl', 'HTML/HTMLIFrameElement.idl', 'HTML/HTMLImageElement.idl', 'HTML/HTMLInputElement.idl', 'HTML/HTMLLIElement.idl', 'HTML/HTMLLabelElement.idl', 'HTML/HTMLLegendElement.idl', 'HTML/HTMLLinkElement.idl', 'HTML/HTMLMapElement.idl', 'HTML/HTMLMarqueeElement.idl', 'HTML/HTMLMediaElement.idl', 'HTML/HTMLMenuElement.idl', 'HTML/HTMLMetaElement.idl', 'HTML/HTMLMeterElement.idl', 'HTML/HTMLModElement.idl', 'HTML/HTMLOListElement.idl', 'HTML/HTMLObjectElement.idl', 'HTML/HTMLOptGroupElement.idl', 'HTML/HTMLOptionElement.idl', 'HTML/HTMLOptionsCollection.idl', 'HTML/HTMLOrSVGOrMathMLElement.idl', 'HTML/HTMLOutputElement.idl', 'HTML/HTMLParagraphElement.idl', 'HTML/HTMLParamElement.idl', 'HTML/HTMLPictureElement.idl', 'HTML/HTMLPreElement.idl', 'HTML/HTMLProgressElement.idl', 'HTML/HTMLQuoteElement.idl', 'HTML/HTMLScriptElement.idl', 'HTML/HTMLSelectElement.idl', 'HTML/HTMLSelectedContentElement.idl', 'HTML/HTMLSlotElement.idl', 'HTML/HTMLSourceElement.idl', 'HTML/HTMLSpanElement.idl', 'HTML/HTMLStyleElement.idl', 'HTML/HTMLTableCaptionElement.idl', 'HTML/HTMLTableCellElement.idl', 'HTML/HTMLTableColElement.idl', 'HTML/HTMLTableElement.idl', 'HTML/HTMLTableRowElement.idl', 'HTML/HTMLTableSectionElement.idl', 'HTML/HTMLTemplateElement.idl', 'HTML/HTMLTextAreaElement.idl', 'HTML/HTMLTimeElement.idl', 'HTML/HTMLTitleElement.idl', 'HTML/HTMLTrackElement.idl', 'HTML/HTMLUListElement.idl', 'HTML/HTMLUnknownElement.idl', 'HTML/HTMLVideoElement.idl', 'HTML/HashChangeEvent.idl', 'HTML/History.idl', 'HTML/HyperlinkElementUtils.idl', 'HTML/ImageBitmap.idl', 'HTML/ImageData.idl', 'HTML/Location.idl', 'HTML/MediaError.idl', 'HTML/MessageChannel.idl', 'HTML/MessageEvent.idl', 'HTML/MessagePort.idl', 'HTML/MimeType.idl', 'HTML/MimeTypeArray.idl', 'HTML/NavigateEvent.idl', 'HTML/Navigation.idl', 'HTML/NavigationCurrentEntryChangeEvent.idl', 'HTML/NavigationDestination.idl', 'HTML/NavigationHistoryEntry.idl', 'HTML/NavigationTransition.idl', 'HTML/NavigationType.idl', 'HTML/Navigator.idl', 'HTML/NavigatorBeacon.idl', 'HTML/NavigatorConcurrentHardware.idl', 'HTML/NavigatorDeviceMemory.idl', 'HTML/NavigatorID.idl', 'HTML/NavigatorLanguage.idl', 'HTML/NavigatorOnLine.idl', 'HTML/OffscreenCanvas.idl', 'HTML/OffscreenCanvasRenderingContext2D.idl', 'HTML/PageTransitionEvent.idl', 'HTML/Path2D.idl', 'HTML/Plugin.idl', 'HTML/PluginArray.idl', 'HTML/PopStateEvent.idl', 'HTML/PopoverTargetAttributes.idl', 'HTML/PredefinedColorSpace.idl', 'HTML/PromiseRejectionEvent.idl', 'HTML/RadioNodeList.idl', 'HTML/Scripting/Fetching.idl', 'HTML/SharedWorker.idl', 'HTML/SharedWorkerGlobalScope.idl', 'HTML/Storage.idl', 'HTML/StorageEvent.idl', 'HTML/SubmitEvent.idl', 'HTML/TextMetrics.idl', 'HTML/TextTrack.idl', 'HTML/TextTrackCue.idl', 'HTML/TextTrackCueList.idl', 'HTML/TextTrackList.idl', 'HTML/TimeRanges.idl', 'HTML/ToggleEvent.idl', 'HTML/TrackEvent.idl', 'HTML/UniversalGlobalScope.idl', 'HTML/UserActivation.idl', 'HTML/ValidityState.idl', 'HTML/VideoTrack.idl', 'HTML/VideoTrackList.idl', 'HTML/Window.idl', 'HTML/WindowDeprecated.idl', 'HTML/WindowLocalStorage.idl', 'HTML/WindowOrWorkerGlobalScope.idl', 'HTML/WindowSessionStorage.idl', 'HTML/Worker.idl', 'HTML/WorkerGlobalScope.idl', 'HTML/WorkerLocation.idl', 'HTML/WorkerNavigator.idl', 'HTML/WorkletGlobalScope.idl', 'HTML/XMLSerializer.idl', 'HighResolutionTime/DOMHighResTimeStamp.idl', 'HighResolutionTime/EpochTimeStamp.idl', 'HighResolutionTime/Performance.idl', 'IndexedDB/IDBCursor.idl', 'IndexedDB/IDBCursorWithValue.idl', 'IndexedDB/IDBDatabase.idl', 'IndexedDB/IDBFactory.idl', 'IndexedDB/IDBIndex.idl', 'IndexedDB/IDBKeyRange.idl', 'IndexedDB/IDBObjectStore.idl', 'IndexedDB/IDBOpenDBRequest.idl', 'IndexedDB/IDBRecord.idl', 'IndexedDB/IDBRequest.idl', 'IndexedDB/IDBTransaction.idl', 'IndexedDB/IDBVersionChangeEvent.idl', 'Internals/FakeXRDevice.idl', 'Internals/InternalAnimationTimeline.idl', 'Internals/InternalGamepad.idl', 'Internals/Internals.idl', 'Internals/WebUI.idl', 'Internals/XRTest.idl', 'IntersectionObserver/IntersectionObserver.idl', 'IntersectionObserver/IntersectionObserverEntry.idl', 'MathML/MathMLAnchorElement.idl', 'MathML/MathMLElement.idl', 'MediaCapabilitiesAPI/MediaCapabilities.idl', 'MediaCapture/MediaDeviceInfo.idl', 'MediaCapture/MediaDevices.idl', 'MediaCapture/MediaStream.idl', 'MediaCapture/MediaStreamConstraints.idl', 'MediaCapture/MediaStreamTrack.idl', 'MediaCapture/MediaStreamTrackEvent.idl', 'MediaSourceExtensions/BufferedChangeEvent.idl', 'MediaSourceExtensions/ManagedMediaSource.idl', 'MediaSourceExtensions/ManagedSourceBuffer.idl', 'MediaSourceExtensions/MediaSource.idl', 'MediaSourceExtensions/MediaSourceHandle.idl', 'MediaSourceExtensions/SourceBuffer.idl', 'MediaSourceExtensions/SourceBufferList.idl', 'NavigationTiming/PerformanceExtensions.idl', 'NavigationTiming/PerformanceNavigation.idl', 'NavigationTiming/PerformanceTiming.idl', 'NotificationsAPI/Notification.idl', 'PerformanceTimeline/PerformanceEntry.idl', 'PerformanceTimeline/PerformanceObserver.idl', 'PerformanceTimeline/PerformanceObserverEntryList.idl', 'PermissionsAPI/PermissionStatus.idl', 'PermissionsAPI/Permissions.idl', 'RequestIdleCallback/IdleDeadline.idl', 'RequestIdleCallback/IdleRequest.idl', 'ResizeObserver/ResizeObserver.idl', 'ResizeObserver/ResizeObserverEntry.idl', 'ResizeObserver/ResizeObserverSize.idl', 'ResourceTiming/PerformanceResourceTiming.idl', 'SVG/SVGAElement.idl', 'SVG/SVGAnimatedEnumeration.idl', 'SVG/SVGAnimatedInteger.idl', 'SVG/SVGAnimatedLength.idl', 'SVG/SVGAnimatedLengthList.idl', 'SVG/SVGAnimatedNumber.idl', 'SVG/SVGAnimatedNumberList.idl', 'SVG/SVGAnimatedRect.idl', 'SVG/SVGAnimatedString.idl', 'SVG/SVGAnimatedTransformList.idl', 'SVG/SVGAnimationElement.idl', 'SVG/SVGCircleElement.idl', 'SVG/SVGClipPathElement.idl', 'SVG/SVGComponentTransferFunctionElement.idl', 'SVG/SVGDefsElement.idl', 'SVG/SVGDescElement.idl', 'SVG/SVGElement.idl', 'SVG/SVGEllipseElement.idl', 'SVG/SVGFEBlendElement.idl', 'SVG/SVGFEColorMatrixElement.idl', 'SVG/SVGFEComponentTransferElement.idl', 'SVG/SVGFECompositeElement.idl', 'SVG/SVGFEDisplacementMapElement.idl', 'SVG/SVGFEDropShadowElement.idl', 'SVG/SVGFEFloodElement.idl', 'SVG/SVGFEFuncAElement.idl', 'SVG/SVGFEFuncBElement.idl', 'SVG/SVGFEFuncGElement.idl', 'SVG/SVGFEFuncRElement.idl', 'SVG/SVGFEGaussianBlurElement.idl', 'SVG/SVGFEImageElement.idl', 'SVG/SVGFEMergeElement.idl', 'SVG/SVGFEMergeNodeElement.idl', 'SVG/SVGFEMorphologyElement.idl', 'SVG/SVGFEOffsetElement.idl', 'SVG/SVGFETurbulenceElement.idl', 'SVG/SVGFilterElement.idl', 'SVG/SVGFilterPrimitiveStandardAttributes.idl', 'SVG/SVGFitToViewBox.idl', 'SVG/SVGForeignObjectElement.idl', 'SVG/SVGGElement.idl', 'SVG/SVGGeometryElement.idl', 'SVG/SVGGradientElement.idl', 'SVG/SVGGraphicsElement.idl', 'SVG/SVGImageElement.idl', 'SVG/SVGLength.idl', 'SVG/SVGLengthList.idl', 'SVG/SVGLineElement.idl', 'SVG/SVGLinearGradientElement.idl', 'SVG/SVGMaskElement.idl', 'SVG/SVGMetadataElement.idl', 'SVG/SVGNumber.idl', 'SVG/SVGNumberList.idl', 'SVG/SVGPathElement.idl', 'SVG/SVGPatternElement.idl', 'SVG/SVGPolygonElement.idl', 'SVG/SVGPolylineElement.idl', 'SVG/SVGRadialGradientElement.idl', 'SVG/SVGRectElement.idl', 'SVG/SVGSVGElement.idl', 'SVG/SVGScriptElement.idl', 'SVG/SVGStopElement.idl', 'SVG/SVGStyleElement.idl', 'SVG/SVGSwitchElement.idl', 'SVG/SVGSymbolElement.idl', 'SVG/SVGTSpanElement.idl', 'SVG/SVGTextContentElement.idl', 'SVG/SVGTextElement.idl', 'SVG/SVGTextPathElement.idl', 'SVG/SVGTextPositioningElement.idl', 'SVG/SVGTitleElement.idl', 'SVG/SVGTransform.idl', 'SVG/SVGTransformList.idl', 'SVG/SVGURIReference.idl', 'SVG/SVGUnitTypes.idl', 'SVG/SVGUseElement.idl', 'SVG/SVGViewElement.idl', 'Selection/Selection.idl', 'Serial/Serial.idl', 'Serial/SerialPort.idl', 'ServiceWorker/Cache.idl', 'ServiceWorker/CacheStorage.idl', 'ServiceWorker/ServiceWorker.idl', 'ServiceWorker/ServiceWorkerContainer.idl', 'ServiceWorker/ServiceWorkerGlobalScope.idl', 'ServiceWorker/ServiceWorkerRegistration.idl', 'Speech/SpeechGrammar.idl', 'Speech/SpeechGrammarList.idl', 'Speech/SpeechRecognition.idl', 'Speech/SpeechRecognitionAlternative.idl', 'Speech/SpeechRecognitionEvent.idl', 'Speech/SpeechRecognitionPhrase.idl', 'Speech/SpeechRecognitionResult.idl', 'Speech/SpeechRecognitionResultList.idl', 'Speech/SpeechSynthesis.idl', 'Speech/SpeechSynthesisUtterance.idl', 'Speech/SpeechSynthesisVoice.idl', 'StorageAPI/NavigatorStorage.idl', 'StorageAPI/StorageManager.idl', 'Streams/ByteLengthQueuingStrategy.idl', 'Streams/CountQueuingStrategy.idl', 'Streams/GenericTransformStream.idl', 'Streams/QueuingStrategy.idl', 'Streams/QueuingStrategyInit.idl', 'Streams/ReadableByteStreamController.idl', 'Streams/ReadableStream.idl', 'Streams/ReadableStreamBYOBReader.idl', 'Streams/ReadableStreamBYOBRequest.idl', 'Streams/ReadableStreamDefaultController.idl', 'Streams/ReadableStreamDefaultReader.idl', 'Streams/ReadableStreamGenericReader.idl', 'Streams/TransformStream.idl', 'Streams/TransformStreamDefaultController.idl', 'Streams/Transformer.idl', 'Streams/UnderlyingSink.idl', 'Streams/UnderlyingSource.idl', 'Streams/WritableStream.idl', 'Streams/WritableStreamDefaultController.idl', 'Streams/WritableStreamDefaultWriter.idl', 'TrustedTypes/TrustedHTML.idl', 'TrustedTypes/TrustedScript.idl', 'TrustedTypes/TrustedScriptURL.idl', 'TrustedTypes/TrustedTypePolicy.idl', 'TrustedTypes/TrustedTypePolicyFactory.idl', 'UIEvents/CompositionEvent.idl', 'UIEvents/EventModifier.idl', 'UIEvents/FocusEvent.idl', 'UIEvents/InputEvent.idl', 'UIEvents/KeyboardEvent.idl', 'UIEvents/MouseEvent.idl', 'UIEvents/PointerEvent.idl', 'UIEvents/PointerEventHandlers.idl', 'UIEvents/TextEvent.idl', 'UIEvents/UIEvent.idl', 'UIEvents/WheelEvent.idl', 'URLPattern/URLPattern.idl', 'UserTiming/PerformanceMark.idl', 'UserTiming/PerformanceMeasure.idl', 'ViewTransition/ViewTransition.idl', 'WebAssembly/Global.idl', 'WebAssembly/Instance.idl', 'WebAssembly/Memory.idl', 'WebAssembly/Module.idl', 'WebAssembly/Table.idl', 'WebAssembly/WebAssembly.idl', 'WebAudio/AnalyserNode.idl', 'WebAudio/AudioBuffer.idl', 'WebAudio/AudioBufferSourceNode.idl', 'WebAudio/AudioContext.idl', 'WebAudio/AudioDestinationNode.idl', 'WebAudio/AudioListener.idl', 'WebAudio/AudioNode.idl', 'WebAudio/AudioParam.idl', 'WebAudio/AudioScheduledSourceNode.idl', 'WebAudio/BaseAudioContext.idl', 'WebAudio/BiquadFilterNode.idl', 'WebAudio/ChannelMergerNode.idl', 'WebAudio/ChannelSplitterNode.idl', 'WebAudio/ConstantSourceNode.idl', 'WebAudio/DelayNode.idl', 'WebAudio/DynamicsCompressorNode.idl', 'WebAudio/GainNode.idl', 'WebAudio/MediaElementAudioSourceNode.idl', 'WebAudio/OfflineAudioCompletionEvent.idl', 'WebAudio/OfflineAudioContext.idl', 'WebAudio/OscillatorNode.idl', 'WebAudio/PannerNode.idl', 'WebAudio/PeriodicWave.idl', 'WebAudio/ScriptProcessorNode.idl', 'WebAudio/StereoPannerNode.idl', 'WebGL/Extensions/ANGLEInstancedArrays.idl', 'WebGL/Extensions/EXTBlendMinMax.idl', 'WebGL/Extensions/EXTColorBufferFloat.idl', 'WebGL/Extensions/EXTRenderSnorm.idl', 'WebGL/Extensions/EXTTextureFilterAnisotropic.idl', 'WebGL/Extensions/EXTTextureNorm16.idl', 'WebGL/Extensions/OESElementIndexUint.idl', 'WebGL/Extensions/OESStandardDerivatives.idl', 'WebGL/Extensions/OESVertexArrayObject.idl', 'WebGL/Extensions/WebGLCompressedTextureS3tc.idl', 'WebGL/Extensions/WebGLCompressedTextureS3tcSrgb.idl', 'WebGL/Extensions/WebGLDebugRendererInfo.idl', 'WebGL/Extensions/WebGLDrawBuffers.idl', 'WebGL/Extensions/WebGLVertexArrayObjectOES.idl', 'WebGL/Types.idl', 'WebGL/WebGL2RenderingContext.idl', 'WebGL/WebGL2RenderingContextBase.idl', 'WebGL/WebGL2RenderingContextOverloads.idl', 'WebGL/WebGLActiveInfo.idl', 'WebGL/WebGLBuffer.idl', 'WebGL/WebGLContextEvent.idl', 'WebGL/WebGLFramebuffer.idl', 'WebGL/WebGLObject.idl', 'WebGL/WebGLProgram.idl', 'WebGL/WebGLQuery.idl', 'WebGL/WebGLRenderbuffer.idl', 'WebGL/WebGLRenderingContext.idl', 'WebGL/WebGLRenderingContextBase.idl', 'WebGL/WebGLRenderingContextOverloads.idl', 'WebGL/WebGLSampler.idl', 'WebGL/WebGLShader.idl', 'WebGL/WebGLShaderPrecisionFormat.idl', 'WebGL/WebGLSync.idl', 'WebGL/WebGLTexture.idl', 'WebGL/WebGLTransformFeedback.idl', 'WebGL/WebGLUniformLocation.idl', 'WebGL/WebGLVertexArrayObject.idl', 'WebIDL/Buffers.idl', 'WebIDL/DOMException.idl', 'WebIDL/Function.idl', 'WebIDL/QuotaExceededError.idl', 'WebLocks/Lock.idl', 'WebLocks/LockManager.idl', 'WebLocks/NavigatorLocks.idl', 'WebSockets/WebSocket.idl', 'WebVTT/VTTCue.idl', 'WebVTT/VTTRegion.idl', 'WebXR/XRLayer.idl', 'WebXR/XRRenderState.idl', 'WebXR/XRSession.idl', 'WebXR/XRSessionEvent.idl', 'WebXR/XRSystem.idl', 'WebXR/XRWebGLLayer.idl', 'XHR/FormData.idl', 'XHR/ProgressEvent.idl', 'XHR/XMLHttpRequest.idl', 'XHR/XMLHttpRequestEventTarget.idl', 'XHR/XMLHttpRequestUpload.idl', 'XPath/XPathEvaluator.idl', 'XPath/XPathExpression.idl', 'XPath/XPathNSResolver.idl', 'XPath/XPathResult.idl'] + [ + srcs = ['ARIA/ARIAMixin.idl', 'Animations/Animatable.idl', 'Animations/Animation.idl', 'Animations/AnimationEffect.idl', 'Animations/AnimationPlaybackEvent.idl', 'Animations/AnimationTimeline.idl', 'Animations/DocumentTimeline.idl', 'Animations/KeyframeEffect.idl', 'Animations/ScrollTimeline.idl', 'CSS/AnimationEvent.idl', 'CSS/CSS.idl', 'CSS/CSSAnimation.idl', 'CSS/CSSConditionRule.idl', 'CSS/CSSContainerRule.idl', 'CSS/CSSCounterStyleRule.idl', 'CSS/CSSFontFaceDescriptors.idl', 'CSS/CSSFontFaceRule.idl', 'CSS/CSSFontFeatureValuesMap.idl', 'CSS/CSSFontFeatureValuesRule.idl', 'CSS/CSSFunctionDeclarations.idl', 'CSS/CSSFunctionDescriptors.idl', 'CSS/CSSFunctionRule.idl', 'CSS/CSSGroupingRule.idl', 'CSS/CSSImageValue.idl', 'CSS/CSSImportRule.idl', 'CSS/CSSKeyframeRule.idl', 'CSS/CSSKeyframesRule.idl', 'CSS/CSSKeywordValue.idl', 'CSS/CSSLayerBlockRule.idl', 'CSS/CSSLayerStatementRule.idl', 'CSS/CSSMarginRule.idl', 'CSS/CSSMathClamp.idl', 'CSS/CSSMathInvert.idl', 'CSS/CSSMathMax.idl', 'CSS/CSSMathMin.idl', 'CSS/CSSMathNegate.idl', 'CSS/CSSMathProduct.idl', 'CSS/CSSMathSum.idl', 'CSS/CSSMathValue.idl', 'CSS/CSSMatrixComponent.idl', 'CSS/CSSMediaRule.idl', 'CSS/CSSNamespaceRule.idl', 'CSS/CSSNestedDeclarations.idl', 'CSS/CSSNumericArray.idl', 'CSS/CSSNumericValue.idl', 'CSS/CSSPageDescriptors.idl', 'CSS/CSSPageRule.idl', 'CSS/CSSPerspective.idl', 'CSS/CSSPropertyRule.idl', 'CSS/CSSRotate.idl', 'CSS/CSSRule.idl', 'CSS/CSSRuleList.idl', 'CSS/CSSScale.idl', 'CSS/CSSScopeRule.idl', 'CSS/CSSSkew.idl', 'CSS/CSSSkewX.idl', 'CSS/CSSSkewY.idl', 'CSS/CSSStyleDeclaration.idl', 'CSS/CSSStyleProperties.idl', 'CSS/CSSStyleRule.idl', 'CSS/CSSStyleSheet.idl', 'CSS/CSSStyleValue.idl', 'CSS/CSSSupportsRule.idl', 'CSS/CSSTransformComponent.idl', 'CSS/CSSTransformValue.idl', 'CSS/CSSTransition.idl', 'CSS/CSSTranslate.idl', 'CSS/CSSUnitValue.idl', 'CSS/CSSUnparsedValue.idl', 'CSS/CSSVariableReferenceValue.idl', 'CSS/ElementCSSInlineStyle.idl', 'CSS/FontFace.idl', 'CSS/FontFaceSet.idl', 'CSS/FontFaceSetLoadEvent.idl', 'CSS/LinkStyle.idl', 'CSS/MediaList.idl', 'CSS/MediaQueryList.idl', 'CSS/MediaQueryListEvent.idl', 'CSS/Screen.idl', 'CSS/ScreenOrientation.idl', 'CSS/StylePropertyMap.idl', 'CSS/StylePropertyMapReadOnly.idl', 'CSS/StyleSheet.idl', 'CSS/StyleSheetList.idl', 'CSS/TransitionEvent.idl', 'CSS/VisualViewport.idl', 'Clipboard/Clipboard.idl', 'Clipboard/ClipboardEvent.idl', 'Clipboard/ClipboardItem.idl', 'Compression/CompressionStream.idl', 'Compression/DecompressionStream.idl', 'ContentSecurityPolicy/SecurityPolicyViolationEvent.idl', 'CookieStore/CookieChangeEvent.idl', 'CookieStore/CookieStore.idl', 'CredentialManagement/Credential.idl', 'CredentialManagement/CredentialsContainer.idl', 'CredentialManagement/FederatedCredential.idl', 'CredentialManagement/PasswordCredential.idl', 'Crypto/Crypto.idl', 'Crypto/CryptoKey.idl', 'Crypto/SubtleCrypto.idl', 'DOM/AbortController.idl', 'DOM/AbortSignal.idl', 'DOM/AbstractRange.idl', 'DOM/Attr.idl', 'DOM/CDATASection.idl', 'DOM/CaretPosition.idl', 'DOM/CharacterData.idl', 'DOM/ChildNode.idl', 'DOM/Comment.idl', 'DOM/CustomEvent.idl', 'DOM/DOMImplementation.idl', 'DOM/DOMTokenList.idl', 'DOM/Document.idl', 'DOM/DocumentFragment.idl', 'DOM/DocumentOrShadowRoot.idl', 'DOM/DocumentType.idl', 'DOM/Element.idl', 'DOM/Event.idl', 'DOM/EventHandler.idl', 'DOM/EventListener.idl', 'DOM/EventTarget.idl', 'DOM/HTMLCollection.idl', 'DOM/MutationObserver.idl', 'DOM/MutationRecord.idl', 'DOM/NamedNodeMap.idl', 'DOM/Node.idl', 'DOM/NodeFilter.idl', 'DOM/NodeIterator.idl', 'DOM/NodeList.idl', 'DOM/ParentNode.idl', 'DOM/ProcessingInstruction.idl', 'DOM/Range.idl', 'DOM/ShadowRoot.idl', 'DOM/Slottable.idl', 'DOM/StaticRange.idl', 'DOM/Text.idl', 'DOM/TreeWalker.idl', 'DOM/XMLDocument.idl', 'DOMURL/DOMURL.idl', 'DOMURL/Origin.idl', 'DOMURL/URLSearchParams.idl', 'Encoding/TextDecoder.idl', 'Encoding/TextDecoderCommon.idl', 'Encoding/TextDecoderStream.idl', 'Encoding/TextEncoder.idl', 'Encoding/TextEncoderCommon.idl', 'Encoding/TextEncoderStream.idl', 'EncryptedMediaExtensions/MediaKeySystemAccess.idl', 'EntriesAPI/FileSystemEntry.idl', 'EventTiming/PerformanceEventTiming.idl', 'Fetch/Body.idl', 'Fetch/BodyInit.idl', 'Fetch/Headers.idl', 'Fetch/Request.idl', 'Fetch/Response.idl', 'FileAPI/Blob.idl', 'FileAPI/File.idl', 'FileAPI/FileList.idl', 'FileAPI/FileReader.idl', 'FileAPI/FileReaderSync.idl', 'Fullscreen/DocumentExtensions.idl', 'Fullscreen/DocumentOrShadowRootExtensions.idl', 'Fullscreen/ElementExtensions.idl', 'GPC/GlobalPrivacyControl.idl', 'Gamepad/Gamepad.idl', 'Gamepad/GamepadButton.idl', 'Gamepad/GamepadEvent.idl', 'Gamepad/GamepadHapticActuator.idl', 'Geolocation/Geolocation.idl', 'Geolocation/GeolocationCoordinates.idl', 'Geolocation/GeolocationPosition.idl', 'Geolocation/GeolocationPositionError.idl', 'Geometry/DOMMatrix.idl', 'Geometry/DOMMatrixReadOnly.idl', 'Geometry/DOMPoint.idl', 'Geometry/DOMPointReadOnly.idl', 'Geometry/DOMQuad.idl', 'Geometry/DOMRect.idl', 'Geometry/DOMRectList.idl', 'Geometry/DOMRectReadOnly.idl', 'HTML/AbstractWorker.idl', 'HTML/AnimationFrameProvider.idl', 'HTML/AudioTrack.idl', 'HTML/AudioTrackList.idl', 'HTML/BarProp.idl', 'HTML/BeforeUnloadEvent.idl', 'HTML/BroadcastChannel.idl', 'HTML/Canvas/CanvasCompositing.idl', 'HTML/Canvas/CanvasDrawImage.idl', 'HTML/Canvas/CanvasDrawPath.idl', 'HTML/Canvas/CanvasFillStrokeStyles.idl', 'HTML/Canvas/CanvasFilters.idl', 'HTML/Canvas/CanvasImageData.idl', 'HTML/Canvas/CanvasImageSmoothing.idl', 'HTML/Canvas/CanvasPath.idl', 'HTML/Canvas/CanvasPathDrawingStyles.idl', 'HTML/Canvas/CanvasRect.idl', 'HTML/Canvas/CanvasSettings.idl', 'HTML/Canvas/CanvasShadowStyles.idl', 'HTML/Canvas/CanvasState.idl', 'HTML/Canvas/CanvasText.idl', 'HTML/Canvas/CanvasTextDrawingStyles.idl', 'HTML/Canvas/CanvasTransform.idl', 'HTML/Canvas/CanvasUserInterface.idl', 'HTML/Canvas/OffscreenCanvasBase.idl', 'HTML/CanvasGradient.idl', 'HTML/CanvasPattern.idl', 'HTML/CanvasRenderingContext2D.idl', 'HTML/CanvasRenderingContext2DSettings.idl', 'HTML/CloseEvent.idl', 'HTML/CloseWatcher.idl', 'HTML/CommandEvent.idl', 'HTML/CustomElements/CustomElementRegistry.idl', 'HTML/CustomElements/CustomStateSet.idl', 'HTML/DOMParser.idl', 'HTML/DOMStringList.idl', 'HTML/DOMStringMap.idl', 'HTML/DataTransfer.idl', 'HTML/DataTransferItem.idl', 'HTML/DataTransferItemList.idl', 'HTML/DedicatedWorkerGlobalScope.idl', 'HTML/DragEvent.idl', 'HTML/ElementInternals.idl', 'HTML/ErrorEvent.idl', 'HTML/EventSource.idl', 'HTML/External.idl', 'HTML/FormDataEvent.idl', 'HTML/HTMLAllCollection.idl', 'HTML/HTMLAnchorElement.idl', 'HTML/HTMLAreaElement.idl', 'HTML/HTMLAudioElement.idl', 'HTML/HTMLBRElement.idl', 'HTML/HTMLBaseElement.idl', 'HTML/HTMLBodyElement.idl', 'HTML/HTMLButtonElement.idl', 'HTML/HTMLCanvasElement.idl', 'HTML/HTMLDListElement.idl', 'HTML/HTMLDataElement.idl', 'HTML/HTMLDataListElement.idl', 'HTML/HTMLDetailsElement.idl', 'HTML/HTMLDialogElement.idl', 'HTML/HTMLDirectoryElement.idl', 'HTML/HTMLDivElement.idl', 'HTML/HTMLDocument.idl', 'HTML/HTMLElement.idl', 'HTML/HTMLEmbedElement.idl', 'HTML/HTMLFieldSetElement.idl', 'HTML/HTMLFontElement.idl', 'HTML/HTMLFormControlsCollection.idl', 'HTML/HTMLFormElement.idl', 'HTML/HTMLFrameElement.idl', 'HTML/HTMLFrameSetElement.idl', 'HTML/HTMLHRElement.idl', 'HTML/HTMLHeadElement.idl', 'HTML/HTMLHeadingElement.idl', 'HTML/HTMLHtmlElement.idl', 'HTML/HTMLHyperlinkElementUtils.idl', 'HTML/HTMLIFrameElement.idl', 'HTML/HTMLImageElement.idl', 'HTML/HTMLInputElement.idl', 'HTML/HTMLLIElement.idl', 'HTML/HTMLLabelElement.idl', 'HTML/HTMLLegendElement.idl', 'HTML/HTMLLinkElement.idl', 'HTML/HTMLMapElement.idl', 'HTML/HTMLMarqueeElement.idl', 'HTML/HTMLMediaElement.idl', 'HTML/HTMLMenuElement.idl', 'HTML/HTMLMetaElement.idl', 'HTML/HTMLMeterElement.idl', 'HTML/HTMLModElement.idl', 'HTML/HTMLOListElement.idl', 'HTML/HTMLObjectElement.idl', 'HTML/HTMLOptGroupElement.idl', 'HTML/HTMLOptionElement.idl', 'HTML/HTMLOptionsCollection.idl', 'HTML/HTMLOrSVGOrMathMLElement.idl', 'HTML/HTMLOutputElement.idl', 'HTML/HTMLParagraphElement.idl', 'HTML/HTMLParamElement.idl', 'HTML/HTMLPictureElement.idl', 'HTML/HTMLPreElement.idl', 'HTML/HTMLProgressElement.idl', 'HTML/HTMLQuoteElement.idl', 'HTML/HTMLScriptElement.idl', 'HTML/HTMLSelectElement.idl', 'HTML/HTMLSelectedContentElement.idl', 'HTML/HTMLSlotElement.idl', 'HTML/HTMLSourceElement.idl', 'HTML/HTMLSpanElement.idl', 'HTML/HTMLStyleElement.idl', 'HTML/HTMLTableCaptionElement.idl', 'HTML/HTMLTableCellElement.idl', 'HTML/HTMLTableColElement.idl', 'HTML/HTMLTableElement.idl', 'HTML/HTMLTableRowElement.idl', 'HTML/HTMLTableSectionElement.idl', 'HTML/HTMLTemplateElement.idl', 'HTML/HTMLTextAreaElement.idl', 'HTML/HTMLTimeElement.idl', 'HTML/HTMLTitleElement.idl', 'HTML/HTMLTrackElement.idl', 'HTML/HTMLUListElement.idl', 'HTML/HTMLUnknownElement.idl', 'HTML/HTMLVideoElement.idl', 'HTML/HashChangeEvent.idl', 'HTML/History.idl', 'HTML/HyperlinkElementUtils.idl', 'HTML/ImageBitmap.idl', 'HTML/ImageData.idl', 'HTML/Location.idl', 'HTML/MediaError.idl', 'HTML/MessageChannel.idl', 'HTML/MessageEvent.idl', 'HTML/MessagePort.idl', 'HTML/MimeType.idl', 'HTML/MimeTypeArray.idl', 'HTML/NavigateEvent.idl', 'HTML/Navigation.idl', 'HTML/NavigationActivation.idl', 'HTML/NavigationCurrentEntryChangeEvent.idl', 'HTML/NavigationDestination.idl', 'HTML/NavigationHistoryEntry.idl', 'HTML/NavigationTransition.idl', 'HTML/NavigationType.idl', 'HTML/Navigator.idl', 'HTML/NavigatorBeacon.idl', 'HTML/NavigatorConcurrentHardware.idl', 'HTML/NavigatorDeviceMemory.idl', 'HTML/NavigatorID.idl', 'HTML/NavigatorLanguage.idl', 'HTML/NavigatorOnLine.idl', 'HTML/OffscreenCanvas.idl', 'HTML/OffscreenCanvasRenderingContext2D.idl', 'HTML/PageSwapEvent.idl', 'HTML/PageTransitionEvent.idl', 'HTML/Path2D.idl', 'HTML/Plugin.idl', 'HTML/PluginArray.idl', 'HTML/PopStateEvent.idl', 'HTML/PopoverTargetAttributes.idl', 'HTML/PredefinedColorSpace.idl', 'HTML/PromiseRejectionEvent.idl', 'HTML/RadioNodeList.idl', 'HTML/Scripting/Fetching.idl', 'HTML/SharedWorker.idl', 'HTML/SharedWorkerGlobalScope.idl', 'HTML/Storage.idl', 'HTML/StorageEvent.idl', 'HTML/SubmitEvent.idl', 'HTML/TextMetrics.idl', 'HTML/TextTrack.idl', 'HTML/TextTrackCue.idl', 'HTML/TextTrackCueList.idl', 'HTML/TextTrackList.idl', 'HTML/TimeRanges.idl', 'HTML/ToggleEvent.idl', 'HTML/TrackEvent.idl', 'HTML/UniversalGlobalScope.idl', 'HTML/UserActivation.idl', 'HTML/ValidityState.idl', 'HTML/VideoTrack.idl', 'HTML/VideoTrackList.idl', 'HTML/Window.idl', 'HTML/WindowDeprecated.idl', 'HTML/WindowLocalStorage.idl', 'HTML/WindowOrWorkerGlobalScope.idl', 'HTML/WindowSessionStorage.idl', 'HTML/Worker.idl', 'HTML/WorkerGlobalScope.idl', 'HTML/WorkerLocation.idl', 'HTML/WorkerNavigator.idl', 'HTML/WorkletGlobalScope.idl', 'HTML/XMLSerializer.idl', 'HighResolutionTime/DOMHighResTimeStamp.idl', 'HighResolutionTime/EpochTimeStamp.idl', 'HighResolutionTime/Performance.idl', 'IndexedDB/IDBCursor.idl', 'IndexedDB/IDBCursorWithValue.idl', 'IndexedDB/IDBDatabase.idl', 'IndexedDB/IDBFactory.idl', 'IndexedDB/IDBIndex.idl', 'IndexedDB/IDBKeyRange.idl', 'IndexedDB/IDBObjectStore.idl', 'IndexedDB/IDBOpenDBRequest.idl', 'IndexedDB/IDBRecord.idl', 'IndexedDB/IDBRequest.idl', 'IndexedDB/IDBTransaction.idl', 'IndexedDB/IDBVersionChangeEvent.idl', 'Internals/FakeXRDevice.idl', 'Internals/InternalAnimationTimeline.idl', 'Internals/InternalGamepad.idl', 'Internals/Internals.idl', 'Internals/WebUI.idl', 'Internals/XRTest.idl', 'IntersectionObserver/IntersectionObserver.idl', 'IntersectionObserver/IntersectionObserverEntry.idl', 'MathML/MathMLAnchorElement.idl', 'MathML/MathMLElement.idl', 'MediaCapabilitiesAPI/MediaCapabilities.idl', 'MediaCapture/MediaDeviceInfo.idl', 'MediaCapture/MediaDevices.idl', 'MediaCapture/MediaStream.idl', 'MediaCapture/MediaStreamConstraints.idl', 'MediaCapture/MediaStreamTrack.idl', 'MediaCapture/MediaStreamTrackEvent.idl', 'MediaSourceExtensions/BufferedChangeEvent.idl', 'MediaSourceExtensions/ManagedMediaSource.idl', 'MediaSourceExtensions/ManagedSourceBuffer.idl', 'MediaSourceExtensions/MediaSource.idl', 'MediaSourceExtensions/MediaSourceHandle.idl', 'MediaSourceExtensions/SourceBuffer.idl', 'MediaSourceExtensions/SourceBufferList.idl', 'NavigationTiming/PerformanceExtensions.idl', 'NavigationTiming/PerformanceNavigation.idl', 'NavigationTiming/PerformanceTiming.idl', 'NotificationsAPI/Notification.idl', 'PerformanceTimeline/PerformanceEntry.idl', 'PerformanceTimeline/PerformanceObserver.idl', 'PerformanceTimeline/PerformanceObserverEntryList.idl', 'PermissionsAPI/PermissionStatus.idl', 'PermissionsAPI/Permissions.idl', 'RequestIdleCallback/IdleDeadline.idl', 'RequestIdleCallback/IdleRequest.idl', 'ResizeObserver/ResizeObserver.idl', 'ResizeObserver/ResizeObserverEntry.idl', 'ResizeObserver/ResizeObserverSize.idl', 'ResourceTiming/PerformanceResourceTiming.idl', 'SVG/SVGAElement.idl', 'SVG/SVGAnimatedEnumeration.idl', 'SVG/SVGAnimatedInteger.idl', 'SVG/SVGAnimatedLength.idl', 'SVG/SVGAnimatedLengthList.idl', 'SVG/SVGAnimatedNumber.idl', 'SVG/SVGAnimatedNumberList.idl', 'SVG/SVGAnimatedRect.idl', 'SVG/SVGAnimatedString.idl', 'SVG/SVGAnimatedTransformList.idl', 'SVG/SVGAnimationElement.idl', 'SVG/SVGCircleElement.idl', 'SVG/SVGClipPathElement.idl', 'SVG/SVGComponentTransferFunctionElement.idl', 'SVG/SVGDefsElement.idl', 'SVG/SVGDescElement.idl', 'SVG/SVGElement.idl', 'SVG/SVGEllipseElement.idl', 'SVG/SVGFEBlendElement.idl', 'SVG/SVGFEColorMatrixElement.idl', 'SVG/SVGFEComponentTransferElement.idl', 'SVG/SVGFECompositeElement.idl', 'SVG/SVGFEDisplacementMapElement.idl', 'SVG/SVGFEDropShadowElement.idl', 'SVG/SVGFEFloodElement.idl', 'SVG/SVGFEFuncAElement.idl', 'SVG/SVGFEFuncBElement.idl', 'SVG/SVGFEFuncGElement.idl', 'SVG/SVGFEFuncRElement.idl', 'SVG/SVGFEGaussianBlurElement.idl', 'SVG/SVGFEImageElement.idl', 'SVG/SVGFEMergeElement.idl', 'SVG/SVGFEMergeNodeElement.idl', 'SVG/SVGFEMorphologyElement.idl', 'SVG/SVGFEOffsetElement.idl', 'SVG/SVGFETurbulenceElement.idl', 'SVG/SVGFilterElement.idl', 'SVG/SVGFilterPrimitiveStandardAttributes.idl', 'SVG/SVGFitToViewBox.idl', 'SVG/SVGForeignObjectElement.idl', 'SVG/SVGGElement.idl', 'SVG/SVGGeometryElement.idl', 'SVG/SVGGradientElement.idl', 'SVG/SVGGraphicsElement.idl', 'SVG/SVGImageElement.idl', 'SVG/SVGLength.idl', 'SVG/SVGLengthList.idl', 'SVG/SVGLineElement.idl', 'SVG/SVGLinearGradientElement.idl', 'SVG/SVGMaskElement.idl', 'SVG/SVGMetadataElement.idl', 'SVG/SVGNumber.idl', 'SVG/SVGNumberList.idl', 'SVG/SVGPathElement.idl', 'SVG/SVGPatternElement.idl', 'SVG/SVGPolygonElement.idl', 'SVG/SVGPolylineElement.idl', 'SVG/SVGRadialGradientElement.idl', 'SVG/SVGRectElement.idl', 'SVG/SVGSVGElement.idl', 'SVG/SVGScriptElement.idl', 'SVG/SVGStopElement.idl', 'SVG/SVGStyleElement.idl', 'SVG/SVGSwitchElement.idl', 'SVG/SVGSymbolElement.idl', 'SVG/SVGTSpanElement.idl', 'SVG/SVGTextContentElement.idl', 'SVG/SVGTextElement.idl', 'SVG/SVGTextPathElement.idl', 'SVG/SVGTextPositioningElement.idl', 'SVG/SVGTitleElement.idl', 'SVG/SVGTransform.idl', 'SVG/SVGTransformList.idl', 'SVG/SVGURIReference.idl', 'SVG/SVGUnitTypes.idl', 'SVG/SVGUseElement.idl', 'SVG/SVGViewElement.idl', 'Selection/Selection.idl', 'Serial/Serial.idl', 'Serial/SerialPort.idl', 'ServiceWorker/Cache.idl', 'ServiceWorker/CacheStorage.idl', 'ServiceWorker/ServiceWorker.idl', 'ServiceWorker/ServiceWorkerContainer.idl', 'ServiceWorker/ServiceWorkerGlobalScope.idl', 'ServiceWorker/ServiceWorkerRegistration.idl', 'Speech/SpeechGrammar.idl', 'Speech/SpeechGrammarList.idl', 'Speech/SpeechRecognition.idl', 'Speech/SpeechRecognitionAlternative.idl', 'Speech/SpeechRecognitionEvent.idl', 'Speech/SpeechRecognitionPhrase.idl', 'Speech/SpeechRecognitionResult.idl', 'Speech/SpeechRecognitionResultList.idl', 'Speech/SpeechSynthesis.idl', 'Speech/SpeechSynthesisUtterance.idl', 'Speech/SpeechSynthesisVoice.idl', 'StorageAPI/NavigatorStorage.idl', 'StorageAPI/StorageManager.idl', 'Streams/ByteLengthQueuingStrategy.idl', 'Streams/CountQueuingStrategy.idl', 'Streams/GenericTransformStream.idl', 'Streams/QueuingStrategy.idl', 'Streams/QueuingStrategyInit.idl', 'Streams/ReadableByteStreamController.idl', 'Streams/ReadableStream.idl', 'Streams/ReadableStreamBYOBReader.idl', 'Streams/ReadableStreamBYOBRequest.idl', 'Streams/ReadableStreamDefaultController.idl', 'Streams/ReadableStreamDefaultReader.idl', 'Streams/ReadableStreamGenericReader.idl', 'Streams/TransformStream.idl', 'Streams/TransformStreamDefaultController.idl', 'Streams/Transformer.idl', 'Streams/UnderlyingSink.idl', 'Streams/UnderlyingSource.idl', 'Streams/WritableStream.idl', 'Streams/WritableStreamDefaultController.idl', 'Streams/WritableStreamDefaultWriter.idl', 'TrustedTypes/TrustedHTML.idl', 'TrustedTypes/TrustedScript.idl', 'TrustedTypes/TrustedScriptURL.idl', 'TrustedTypes/TrustedTypePolicy.idl', 'TrustedTypes/TrustedTypePolicyFactory.idl', 'UIEvents/CompositionEvent.idl', 'UIEvents/EventModifier.idl', 'UIEvents/FocusEvent.idl', 'UIEvents/InputEvent.idl', 'UIEvents/KeyboardEvent.idl', 'UIEvents/MouseEvent.idl', 'UIEvents/PointerEvent.idl', 'UIEvents/PointerEventHandlers.idl', 'UIEvents/TextEvent.idl', 'UIEvents/UIEvent.idl', 'UIEvents/WheelEvent.idl', 'URLPattern/URLPattern.idl', 'UserTiming/PerformanceMark.idl', 'UserTiming/PerformanceMeasure.idl', 'ViewTransition/ViewTransition.idl', 'WebAssembly/Global.idl', 'WebAssembly/Instance.idl', 'WebAssembly/Memory.idl', 'WebAssembly/Module.idl', 'WebAssembly/Table.idl', 'WebAssembly/WebAssembly.idl', 'WebAudio/AnalyserNode.idl', 'WebAudio/AudioBuffer.idl', 'WebAudio/AudioBufferSourceNode.idl', 'WebAudio/AudioContext.idl', 'WebAudio/AudioDestinationNode.idl', 'WebAudio/AudioListener.idl', 'WebAudio/AudioNode.idl', 'WebAudio/AudioParam.idl', 'WebAudio/AudioScheduledSourceNode.idl', 'WebAudio/BaseAudioContext.idl', 'WebAudio/BiquadFilterNode.idl', 'WebAudio/ChannelMergerNode.idl', 'WebAudio/ChannelSplitterNode.idl', 'WebAudio/ConstantSourceNode.idl', 'WebAudio/DelayNode.idl', 'WebAudio/DynamicsCompressorNode.idl', 'WebAudio/GainNode.idl', 'WebAudio/MediaElementAudioSourceNode.idl', 'WebAudio/OfflineAudioCompletionEvent.idl', 'WebAudio/OfflineAudioContext.idl', 'WebAudio/OscillatorNode.idl', 'WebAudio/PannerNode.idl', 'WebAudio/PeriodicWave.idl', 'WebAudio/ScriptProcessorNode.idl', 'WebAudio/StereoPannerNode.idl', 'WebGL/Extensions/ANGLEInstancedArrays.idl', 'WebGL/Extensions/EXTBlendMinMax.idl', 'WebGL/Extensions/EXTColorBufferFloat.idl', 'WebGL/Extensions/EXTRenderSnorm.idl', 'WebGL/Extensions/EXTTextureFilterAnisotropic.idl', 'WebGL/Extensions/EXTTextureNorm16.idl', 'WebGL/Extensions/OESElementIndexUint.idl', 'WebGL/Extensions/OESStandardDerivatives.idl', 'WebGL/Extensions/OESVertexArrayObject.idl', 'WebGL/Extensions/WebGLCompressedTextureS3tc.idl', 'WebGL/Extensions/WebGLCompressedTextureS3tcSrgb.idl', 'WebGL/Extensions/WebGLDebugRendererInfo.idl', 'WebGL/Extensions/WebGLDrawBuffers.idl', 'WebGL/Extensions/WebGLVertexArrayObjectOES.idl', 'WebGL/Types.idl', 'WebGL/WebGL2RenderingContext.idl', 'WebGL/WebGL2RenderingContextBase.idl', 'WebGL/WebGL2RenderingContextOverloads.idl', 'WebGL/WebGLActiveInfo.idl', 'WebGL/WebGLBuffer.idl', 'WebGL/WebGLContextEvent.idl', 'WebGL/WebGLFramebuffer.idl', 'WebGL/WebGLObject.idl', 'WebGL/WebGLProgram.idl', 'WebGL/WebGLQuery.idl', 'WebGL/WebGLRenderbuffer.idl', 'WebGL/WebGLRenderingContext.idl', 'WebGL/WebGLRenderingContextBase.idl', 'WebGL/WebGLRenderingContextOverloads.idl', 'WebGL/WebGLSampler.idl', 'WebGL/WebGLShader.idl', 'WebGL/WebGLShaderPrecisionFormat.idl', 'WebGL/WebGLSync.idl', 'WebGL/WebGLTexture.idl', 'WebGL/WebGLTransformFeedback.idl', 'WebGL/WebGLUniformLocation.idl', 'WebGL/WebGLVertexArrayObject.idl', 'WebIDL/Buffers.idl', 'WebIDL/DOMException.idl', 'WebIDL/Function.idl', 'WebIDL/QuotaExceededError.idl', 'WebLocks/Lock.idl', 'WebLocks/LockManager.idl', 'WebLocks/NavigatorLocks.idl', 'WebSockets/WebSocket.idl', 'WebVTT/VTTCue.idl', 'WebVTT/VTTRegion.idl', 'WebXR/XRLayer.idl', 'WebXR/XRRenderState.idl', 'WebXR/XRSession.idl', 'WebXR/XRSessionEvent.idl', 'WebXR/XRSystem.idl', 'WebXR/XRWebGLLayer.idl', 'XHR/FormData.idl', 'XHR/ProgressEvent.idl', 'XHR/XMLHttpRequest.idl', 'XHR/XMLHttpRequestEventTarget.idl', 'XHR/XMLHttpRequestUpload.idl', 'XPath/XPathEvaluator.idl', 'XPath/XPathExpression.idl', 'XPath/XPathNSResolver.idl', 'XPath/XPathResult.idl'] + [ 'CSS/GeneratedCSSNumericFactoryMethods.idl', # generated in this package 'CSS/GeneratedCSSStyleProperties.idl', # generated in this package ] + ["//Meta:generators"], @@ -505,6 +505,7 @@ def libweb_bindings_codegen(): 'Bindings/DedicatedWorkerExposedInterfaces.h', 'Bindings/DedicatedWorkerGlobalScope.cpp', 'Bindings/DedicatedWorkerGlobalScope.h', + 'Bindings/DedicatedWorkerGlobalScopeGlobalMixin.h', 'Bindings/DelayNode.cpp', 'Bindings/DelayNode.h', 'Bindings/Document.cpp', @@ -910,6 +911,8 @@ def libweb_bindings_codegen(): 'Bindings/NavigateEvent.h', 'Bindings/Navigation.cpp', 'Bindings/Navigation.h', + 'Bindings/NavigationActivation.cpp', + 'Bindings/NavigationActivation.h', 'Bindings/NavigationCurrentEntryChangeEvent.cpp', 'Bindings/NavigationCurrentEntryChangeEvent.h', 'Bindings/NavigationDestination.cpp', @@ -968,6 +971,8 @@ def libweb_bindings_codegen(): 'Bindings/Origin.h', 'Bindings/OscillatorNode.cpp', 'Bindings/OscillatorNode.h', + 'Bindings/PageSwapEvent.cpp', + 'Bindings/PageSwapEvent.h', 'Bindings/PageTransitionEvent.cpp', 'Bindings/PageTransitionEvent.h', 'Bindings/PannerNode.cpp', @@ -1246,6 +1251,7 @@ def libweb_bindings_codegen(): 'Bindings/SharedWorkerExposedInterfaces.h', 'Bindings/SharedWorkerGlobalScope.cpp', 'Bindings/SharedWorkerGlobalScope.h', + 'Bindings/SharedWorkerGlobalScopeGlobalMixin.h', 'Bindings/Slottable.cpp', 'Bindings/Slottable.h', 'Bindings/SourceBuffer.cpp', @@ -1284,6 +1290,7 @@ def libweb_bindings_codegen(): 'Bindings/StorageEvent.h', 'Bindings/StorageManager.cpp', 'Bindings/StorageManager.h', + 'Bindings/StructuredSerializeBindings.cpp', 'Bindings/StylePropertyMap.cpp', 'Bindings/StylePropertyMap.h', 'Bindings/StylePropertyMapReadOnly.cpp', @@ -1448,6 +1455,7 @@ def libweb_bindings_codegen(): 'Bindings/WindowDeprecated.h', 'Bindings/WindowExposedInterfaces.cpp', 'Bindings/WindowExposedInterfaces.h', + 'Bindings/WindowGlobalMixin.h', 'Bindings/WindowLocalStorage.cpp', 'Bindings/WindowLocalStorage.h', 'Bindings/WindowOrWorkerGlobalScope.cpp', @@ -1464,6 +1472,7 @@ def libweb_bindings_codegen(): 'Bindings/WorkerNavigator.h', 'Bindings/WorkletGlobalScope.cpp', 'Bindings/WorkletGlobalScope.h', + 'Bindings/WrapperFactory.cpp', 'Bindings/WritableStream.cpp', 'Bindings/WritableStream.h', 'Bindings/WritableStreamDefaultController.cpp', @@ -1503,5 +1512,5 @@ def libweb_bindings_codegen(): 'Bindings/XRWebGLLayer.cpp', 'Bindings/XRWebGLLayer.h', ], - cmd = 'PYTHONHASHSEED=0 python3 Meta/Generators/generate_libweb_bindings.py -o $(RULEDIR)/Bindings $(location Animations/Animatable.idl) $(location Animations/Animation.idl) $(location Animations/AnimationEffect.idl) $(location Animations/AnimationPlaybackEvent.idl) $(location Animations/AnimationTimeline.idl) $(location Animations/DocumentTimeline.idl) $(location Animations/KeyframeEffect.idl) $(location Animations/ScrollTimeline.idl) $(location ARIA/ARIAMixin.idl) $(location Clipboard/Clipboard.idl) $(location Clipboard/ClipboardEvent.idl) $(location Clipboard/ClipboardItem.idl) $(location Compression/CompressionStream.idl) $(location Compression/DecompressionStream.idl) $(location ContentSecurityPolicy/SecurityPolicyViolationEvent.idl) $(location CookieStore/CookieChangeEvent.idl) $(location CookieStore/CookieStore.idl) $(location CredentialManagement/Credential.idl) $(location CredentialManagement/CredentialsContainer.idl) $(location CredentialManagement/FederatedCredential.idl) $(location CredentialManagement/PasswordCredential.idl) $(location Crypto/Crypto.idl) $(location Crypto/CryptoKey.idl) $(location Crypto/SubtleCrypto.idl) $(location CSS/AnimationEvent.idl) $(location CSS/CSS.idl) $(location CSS/CSSAnimation.idl) $(location CSS/CSSConditionRule.idl) $(location CSS/CSSContainerRule.idl) $(location CSS/CSSCounterStyleRule.idl) $(location CSS/CSSFontFaceDescriptors.idl) $(location CSS/CSSFontFaceRule.idl) $(location CSS/CSSFontFeatureValuesMap.idl) $(location CSS/CSSFontFeatureValuesRule.idl) $(location CSS/CSSFunctionDeclarations.idl) $(location CSS/CSSFunctionDescriptors.idl) $(location CSS/CSSFunctionRule.idl) $(location CSS/CSSGroupingRule.idl) $(location CSS/CSSImageValue.idl) $(location CSS/CSSImportRule.idl) $(location CSS/CSSKeyframeRule.idl) $(location CSS/CSSKeyframesRule.idl) $(location CSS/CSSKeywordValue.idl) $(location CSS/CSSLayerBlockRule.idl) $(location CSS/CSSLayerStatementRule.idl) $(location CSS/CSSMarginRule.idl) $(location CSS/CSSMathClamp.idl) $(location CSS/CSSMathInvert.idl) $(location CSS/CSSMathMax.idl) $(location CSS/CSSMathMin.idl) $(location CSS/CSSMathNegate.idl) $(location CSS/CSSMathProduct.idl) $(location CSS/CSSMathSum.idl) $(location CSS/CSSMathValue.idl) $(location CSS/CSSMatrixComponent.idl) $(location CSS/CSSMediaRule.idl) $(location CSS/CSSNamespaceRule.idl) $(location CSS/CSSNestedDeclarations.idl) $(location CSS/CSSNumericArray.idl) $(location CSS/CSSNumericValue.idl) $(location CSS/CSSPageDescriptors.idl) $(location CSS/CSSPageRule.idl) $(location CSS/CSSPerspective.idl) $(location CSS/CSSPropertyRule.idl) $(location CSS/CSSRotate.idl) $(location CSS/CSSRule.idl) $(location CSS/CSSRuleList.idl) $(location CSS/CSSScale.idl) $(location CSS/CSSScopeRule.idl) $(location CSS/CSSSkew.idl) $(location CSS/CSSSkewX.idl) $(location CSS/CSSSkewY.idl) $(location CSS/CSSStyleDeclaration.idl) $(location CSS/CSSStyleProperties.idl) $(location CSS/CSSStyleRule.idl) $(location CSS/CSSStyleSheet.idl) $(location CSS/CSSStyleValue.idl) $(location CSS/CSSSupportsRule.idl) $(location CSS/CSSTransformComponent.idl) $(location CSS/CSSTransformValue.idl) $(location CSS/CSSTransition.idl) $(location CSS/CSSTranslate.idl) $(location CSS/CSSUnitValue.idl) $(location CSS/CSSUnparsedValue.idl) $(location CSS/CSSVariableReferenceValue.idl) $(location CSS/ElementCSSInlineStyle.idl) $(location CSS/FontFace.idl) $(location CSS/FontFaceSet.idl) $(location CSS/FontFaceSetLoadEvent.idl) $(location CSS/GeneratedCSSNumericFactoryMethods.idl) $(location CSS/GeneratedCSSStyleProperties.idl) $(location CSS/LinkStyle.idl) $(location CSS/MediaList.idl) $(location CSS/MediaQueryList.idl) $(location CSS/MediaQueryListEvent.idl) $(location CSS/Screen.idl) $(location CSS/ScreenOrientation.idl) $(location CSS/StylePropertyMap.idl) $(location CSS/StylePropertyMapReadOnly.idl) $(location CSS/StyleSheet.idl) $(location CSS/StyleSheetList.idl) $(location CSS/TransitionEvent.idl) $(location CSS/VisualViewport.idl) $(location DOM/AbortController.idl) $(location DOM/AbortSignal.idl) $(location DOM/AbstractRange.idl) $(location DOM/Attr.idl) $(location DOM/CDATASection.idl) $(location DOM/CaretPosition.idl) $(location DOM/CharacterData.idl) $(location DOM/ChildNode.idl) $(location DOM/Comment.idl) $(location DOM/CustomEvent.idl) $(location DOM/Document.idl) $(location DOM/DocumentFragment.idl) $(location DOM/DocumentOrShadowRoot.idl) $(location DOM/DocumentType.idl) $(location DOM/DOMImplementation.idl) $(location DOM/DOMTokenList.idl) $(location DOM/Element.idl) $(location DOM/Event.idl) $(location DOM/EventHandler.idl) $(location DOM/EventListener.idl) $(location DOM/EventTarget.idl) $(location DOM/HTMLCollection.idl) $(location DOM/MutationObserver.idl) $(location DOM/MutationRecord.idl) $(location DOM/NamedNodeMap.idl) $(location DOM/Node.idl) $(location DOM/NodeFilter.idl) $(location DOM/NodeIterator.idl) $(location DOM/NodeList.idl) $(location DOM/ParentNode.idl) $(location DOM/ProcessingInstruction.idl) $(location DOM/Range.idl) $(location DOM/ShadowRoot.idl) $(location DOM/Slottable.idl) $(location DOM/StaticRange.idl) $(location DOM/Text.idl) $(location DOM/TreeWalker.idl) $(location DOM/XMLDocument.idl) $(location DOMURL/DOMURL.idl) $(location DOMURL/Origin.idl) $(location DOMURL/URLSearchParams.idl) $(location Encoding/TextDecoder.idl) $(location Encoding/TextDecoderCommon.idl) $(location Encoding/TextDecoderStream.idl) $(location Encoding/TextEncoder.idl) $(location Encoding/TextEncoderCommon.idl) $(location Encoding/TextEncoderStream.idl) $(location EncryptedMediaExtensions/MediaKeySystemAccess.idl) $(location EntriesAPI/FileSystemEntry.idl) $(location EventTiming/PerformanceEventTiming.idl) $(location Fetch/Body.idl) $(location Fetch/BodyInit.idl) $(location Fetch/Headers.idl) $(location Fetch/Request.idl) $(location Fetch/Response.idl) $(location FileAPI/Blob.idl) $(location FileAPI/File.idl) $(location FileAPI/FileList.idl) $(location FileAPI/FileReader.idl) $(location FileAPI/FileReaderSync.idl) $(location Fullscreen/DocumentExtensions.idl) $(location Fullscreen/DocumentOrShadowRootExtensions.idl) $(location Fullscreen/ElementExtensions.idl) $(location Gamepad/Gamepad.idl) $(location Gamepad/GamepadButton.idl) $(location Gamepad/GamepadEvent.idl) $(location Gamepad/GamepadHapticActuator.idl) $(location Geolocation/Geolocation.idl) $(location Geolocation/GeolocationCoordinates.idl) $(location Geolocation/GeolocationPosition.idl) $(location Geolocation/GeolocationPositionError.idl) $(location Geometry/DOMMatrix.idl) $(location Geometry/DOMMatrixReadOnly.idl) $(location Geometry/DOMPoint.idl) $(location Geometry/DOMPointReadOnly.idl) $(location Geometry/DOMQuad.idl) $(location Geometry/DOMRect.idl) $(location Geometry/DOMRectList.idl) $(location Geometry/DOMRectReadOnly.idl) $(location GPC/GlobalPrivacyControl.idl) $(location HighResolutionTime/DOMHighResTimeStamp.idl) $(location HighResolutionTime/EpochTimeStamp.idl) $(location HighResolutionTime/Performance.idl) $(location HTML/AbstractWorker.idl) $(location HTML/AnimationFrameProvider.idl) $(location HTML/AudioTrack.idl) $(location HTML/AudioTrackList.idl) $(location HTML/BarProp.idl) $(location HTML/BeforeUnloadEvent.idl) $(location HTML/BroadcastChannel.idl) $(location HTML/Canvas/CanvasCompositing.idl) $(location HTML/Canvas/CanvasDrawImage.idl) $(location HTML/Canvas/CanvasDrawPath.idl) $(location HTML/Canvas/CanvasFillStrokeStyles.idl) $(location HTML/Canvas/CanvasFilters.idl) $(location HTML/Canvas/CanvasImageData.idl) $(location HTML/Canvas/CanvasImageSmoothing.idl) $(location HTML/Canvas/CanvasPath.idl) $(location HTML/Canvas/CanvasPathDrawingStyles.idl) $(location HTML/Canvas/CanvasRect.idl) $(location HTML/Canvas/CanvasSettings.idl) $(location HTML/Canvas/CanvasShadowStyles.idl) $(location HTML/Canvas/CanvasState.idl) $(location HTML/Canvas/CanvasText.idl) $(location HTML/Canvas/CanvasTextDrawingStyles.idl) $(location HTML/Canvas/CanvasTransform.idl) $(location HTML/Canvas/CanvasUserInterface.idl) $(location HTML/Canvas/OffscreenCanvasBase.idl) $(location HTML/CanvasGradient.idl) $(location HTML/CanvasPattern.idl) $(location HTML/CanvasRenderingContext2D.idl) $(location HTML/CanvasRenderingContext2DSettings.idl) $(location HTML/CloseEvent.idl) $(location HTML/CloseWatcher.idl) $(location HTML/CommandEvent.idl) $(location HTML/CustomElements/CustomElementRegistry.idl) $(location HTML/CustomElements/CustomStateSet.idl) $(location HTML/DataTransfer.idl) $(location HTML/DataTransferItem.idl) $(location HTML/DataTransferItemList.idl) $(location HTML/DedicatedWorkerGlobalScope.idl) $(location HTML/DOMParser.idl) $(location HTML/DOMStringList.idl) $(location HTML/DOMStringMap.idl) $(location HTML/DragEvent.idl) $(location HTML/ElementInternals.idl) $(location HTML/ErrorEvent.idl) $(location HTML/EventSource.idl) $(location HTML/External.idl) $(location HTML/FormDataEvent.idl) $(location HTML/HashChangeEvent.idl) $(location HTML/History.idl) $(location HTML/HTMLAllCollection.idl) $(location HTML/HTMLAnchorElement.idl) $(location HTML/HTMLAreaElement.idl) $(location HTML/HTMLAudioElement.idl) $(location HTML/HTMLBaseElement.idl) $(location HTML/HTMLBodyElement.idl) $(location HTML/HTMLBRElement.idl) $(location HTML/HTMLButtonElement.idl) $(location HTML/HTMLCanvasElement.idl) $(location HTML/HTMLDataElement.idl) $(location HTML/HTMLDataListElement.idl) $(location HTML/HTMLDetailsElement.idl) $(location HTML/HTMLDialogElement.idl) $(location HTML/HTMLDirectoryElement.idl) $(location HTML/HTMLDivElement.idl) $(location HTML/HTMLDListElement.idl) $(location HTML/HTMLDocument.idl) $(location HTML/HTMLElement.idl) $(location HTML/HTMLEmbedElement.idl) $(location HTML/HTMLFieldSetElement.idl) $(location HTML/HTMLFontElement.idl) $(location HTML/HTMLFormControlsCollection.idl) $(location HTML/HTMLFormElement.idl) $(location HTML/HTMLFrameElement.idl) $(location HTML/HTMLFrameSetElement.idl) $(location HTML/HTMLHeadElement.idl) $(location HTML/HTMLHeadingElement.idl) $(location HTML/HTMLHRElement.idl) $(location HTML/HTMLHtmlElement.idl) $(location HTML/HTMLHyperlinkElementUtils.idl) $(location HTML/HTMLIFrameElement.idl) $(location HTML/HTMLImageElement.idl) $(location HTML/HTMLInputElement.idl) $(location HTML/HTMLLabelElement.idl) $(location HTML/HTMLLegendElement.idl) $(location HTML/HTMLLIElement.idl) $(location HTML/HTMLLinkElement.idl) $(location HTML/HTMLMapElement.idl) $(location HTML/HTMLMarqueeElement.idl) $(location HTML/HTMLMediaElement.idl) $(location HTML/HTMLMenuElement.idl) $(location HTML/HTMLMetaElement.idl) $(location HTML/HTMLMeterElement.idl) $(location HTML/HTMLModElement.idl) $(location HTML/HTMLObjectElement.idl) $(location HTML/HTMLOListElement.idl) $(location HTML/HTMLOptGroupElement.idl) $(location HTML/HTMLOptionElement.idl) $(location HTML/HTMLOptionsCollection.idl) $(location HTML/HTMLOrSVGOrMathMLElement.idl) $(location HTML/HTMLOutputElement.idl) $(location HTML/HTMLParagraphElement.idl) $(location HTML/HTMLParamElement.idl) $(location HTML/HTMLPictureElement.idl) $(location HTML/HTMLPreElement.idl) $(location HTML/HTMLProgressElement.idl) $(location HTML/HTMLQuoteElement.idl) $(location HTML/HTMLScriptElement.idl) $(location HTML/HTMLSelectedContentElement.idl) $(location HTML/HTMLSelectElement.idl) $(location HTML/HTMLSlotElement.idl) $(location HTML/HTMLSourceElement.idl) $(location HTML/HTMLSpanElement.idl) $(location HTML/HTMLStyleElement.idl) $(location HTML/HTMLTableCaptionElement.idl) $(location HTML/HTMLTableCellElement.idl) $(location HTML/HTMLTableColElement.idl) $(location HTML/HTMLTableElement.idl) $(location HTML/HTMLTableRowElement.idl) $(location HTML/HTMLTableSectionElement.idl) $(location HTML/HTMLTemplateElement.idl) $(location HTML/HTMLTextAreaElement.idl) $(location HTML/HTMLTimeElement.idl) $(location HTML/HTMLTitleElement.idl) $(location HTML/HTMLTrackElement.idl) $(location HTML/HTMLUListElement.idl) $(location HTML/HTMLUnknownElement.idl) $(location HTML/HTMLVideoElement.idl) $(location HTML/HyperlinkElementUtils.idl) $(location HTML/ImageBitmap.idl) $(location HTML/ImageData.idl) $(location HTML/Location.idl) $(location HTML/MediaError.idl) $(location HTML/MessageChannel.idl) $(location HTML/MessageEvent.idl) $(location HTML/MessagePort.idl) $(location HTML/MimeType.idl) $(location HTML/MimeTypeArray.idl) $(location HTML/NavigateEvent.idl) $(location HTML/Navigation.idl) $(location HTML/NavigationCurrentEntryChangeEvent.idl) $(location HTML/NavigationDestination.idl) $(location HTML/NavigationHistoryEntry.idl) $(location HTML/NavigationTransition.idl) $(location HTML/NavigationType.idl) $(location HTML/Navigator.idl) $(location HTML/NavigatorBeacon.idl) $(location HTML/NavigatorConcurrentHardware.idl) $(location HTML/NavigatorDeviceMemory.idl) $(location HTML/NavigatorID.idl) $(location HTML/NavigatorLanguage.idl) $(location HTML/NavigatorOnLine.idl) $(location HTML/OffscreenCanvas.idl) $(location HTML/OffscreenCanvasRenderingContext2D.idl) $(location HTML/PageTransitionEvent.idl) $(location HTML/Path2D.idl) $(location HTML/Plugin.idl) $(location HTML/PluginArray.idl) $(location HTML/PopoverTargetAttributes.idl) $(location HTML/PopStateEvent.idl) $(location HTML/PredefinedColorSpace.idl) $(location HTML/PromiseRejectionEvent.idl) $(location HTML/RadioNodeList.idl) $(location HTML/Scripting/Fetching.idl) $(location HTML/SharedWorker.idl) $(location HTML/SharedWorkerGlobalScope.idl) $(location HTML/Storage.idl) $(location HTML/StorageEvent.idl) $(location HTML/SubmitEvent.idl) $(location HTML/TextMetrics.idl) $(location HTML/TextTrack.idl) $(location HTML/TextTrackCue.idl) $(location HTML/TextTrackCueList.idl) $(location HTML/TextTrackList.idl) $(location HTML/TimeRanges.idl) $(location HTML/ToggleEvent.idl) $(location HTML/TrackEvent.idl) $(location HTML/UniversalGlobalScope.idl) $(location HTML/UserActivation.idl) $(location HTML/ValidityState.idl) $(location HTML/VideoTrack.idl) $(location HTML/VideoTrackList.idl) $(location HTML/Window.idl) $(location HTML/WindowDeprecated.idl) $(location HTML/WindowLocalStorage.idl) $(location HTML/WindowOrWorkerGlobalScope.idl) $(location HTML/WindowSessionStorage.idl) $(location HTML/Worker.idl) $(location HTML/WorkerGlobalScope.idl) $(location HTML/WorkerLocation.idl) $(location HTML/WorkerNavigator.idl) $(location HTML/WorkletGlobalScope.idl) $(location HTML/XMLSerializer.idl) $(location IndexedDB/IDBCursor.idl) $(location IndexedDB/IDBCursorWithValue.idl) $(location IndexedDB/IDBDatabase.idl) $(location IndexedDB/IDBFactory.idl) $(location IndexedDB/IDBIndex.idl) $(location IndexedDB/IDBKeyRange.idl) $(location IndexedDB/IDBObjectStore.idl) $(location IndexedDB/IDBOpenDBRequest.idl) $(location IndexedDB/IDBRecord.idl) $(location IndexedDB/IDBRequest.idl) $(location IndexedDB/IDBTransaction.idl) $(location IndexedDB/IDBVersionChangeEvent.idl) $(location Internals/FakeXRDevice.idl) $(location Internals/InternalAnimationTimeline.idl) $(location Internals/InternalGamepad.idl) $(location Internals/Internals.idl) $(location Internals/WebUI.idl) $(location Internals/XRTest.idl) $(location IntersectionObserver/IntersectionObserver.idl) $(location IntersectionObserver/IntersectionObserverEntry.idl) $(location MathML/MathMLAnchorElement.idl) $(location MathML/MathMLElement.idl) $(location MediaCapabilitiesAPI/MediaCapabilities.idl) $(location MediaCapture/MediaDeviceInfo.idl) $(location MediaCapture/MediaDevices.idl) $(location MediaCapture/MediaStream.idl) $(location MediaCapture/MediaStreamConstraints.idl) $(location MediaCapture/MediaStreamTrack.idl) $(location MediaCapture/MediaStreamTrackEvent.idl) $(location MediaSourceExtensions/BufferedChangeEvent.idl) $(location MediaSourceExtensions/ManagedMediaSource.idl) $(location MediaSourceExtensions/ManagedSourceBuffer.idl) $(location MediaSourceExtensions/MediaSource.idl) $(location MediaSourceExtensions/MediaSourceHandle.idl) $(location MediaSourceExtensions/SourceBuffer.idl) $(location MediaSourceExtensions/SourceBufferList.idl) $(location NavigationTiming/PerformanceExtensions.idl) $(location NavigationTiming/PerformanceNavigation.idl) $(location NavigationTiming/PerformanceTiming.idl) $(location NotificationsAPI/Notification.idl) $(location PerformanceTimeline/PerformanceEntry.idl) $(location PerformanceTimeline/PerformanceObserver.idl) $(location PerformanceTimeline/PerformanceObserverEntryList.idl) $(location PermissionsAPI/Permissions.idl) $(location PermissionsAPI/PermissionStatus.idl) $(location RequestIdleCallback/IdleDeadline.idl) $(location RequestIdleCallback/IdleRequest.idl) $(location ResizeObserver/ResizeObserver.idl) $(location ResizeObserver/ResizeObserverEntry.idl) $(location ResizeObserver/ResizeObserverSize.idl) $(location ResourceTiming/PerformanceResourceTiming.idl) $(location Selection/Selection.idl) $(location Serial/Serial.idl) $(location Serial/SerialPort.idl) $(location ServiceWorker/Cache.idl) $(location ServiceWorker/CacheStorage.idl) $(location ServiceWorker/ServiceWorker.idl) $(location ServiceWorker/ServiceWorkerContainer.idl) $(location ServiceWorker/ServiceWorkerGlobalScope.idl) $(location ServiceWorker/ServiceWorkerRegistration.idl) $(location Speech/SpeechGrammar.idl) $(location Speech/SpeechGrammarList.idl) $(location Speech/SpeechRecognition.idl) $(location Speech/SpeechRecognitionAlternative.idl) $(location Speech/SpeechRecognitionEvent.idl) $(location Speech/SpeechRecognitionPhrase.idl) $(location Speech/SpeechRecognitionResult.idl) $(location Speech/SpeechRecognitionResultList.idl) $(location Speech/SpeechSynthesis.idl) $(location Speech/SpeechSynthesisUtterance.idl) $(location Speech/SpeechSynthesisVoice.idl) $(location StorageAPI/NavigatorStorage.idl) $(location StorageAPI/StorageManager.idl) $(location Streams/ByteLengthQueuingStrategy.idl) $(location Streams/CountQueuingStrategy.idl) $(location Streams/GenericTransformStream.idl) $(location Streams/QueuingStrategy.idl) $(location Streams/QueuingStrategyInit.idl) $(location Streams/ReadableByteStreamController.idl) $(location Streams/ReadableStream.idl) $(location Streams/ReadableStreamBYOBReader.idl) $(location Streams/ReadableStreamBYOBRequest.idl) $(location Streams/ReadableStreamDefaultController.idl) $(location Streams/ReadableStreamDefaultReader.idl) $(location Streams/ReadableStreamGenericReader.idl) $(location Streams/Transformer.idl) $(location Streams/TransformStream.idl) $(location Streams/TransformStreamDefaultController.idl) $(location Streams/UnderlyingSink.idl) $(location Streams/UnderlyingSource.idl) $(location Streams/WritableStream.idl) $(location Streams/WritableStreamDefaultController.idl) $(location Streams/WritableStreamDefaultWriter.idl) $(location SVG/SVGAElement.idl) $(location SVG/SVGAnimatedEnumeration.idl) $(location SVG/SVGAnimatedInteger.idl) $(location SVG/SVGAnimatedLength.idl) $(location SVG/SVGAnimatedLengthList.idl) $(location SVG/SVGAnimatedNumber.idl) $(location SVG/SVGAnimatedNumberList.idl) $(location SVG/SVGAnimatedRect.idl) $(location SVG/SVGAnimatedString.idl) $(location SVG/SVGAnimatedTransformList.idl) $(location SVG/SVGAnimationElement.idl) $(location SVG/SVGCircleElement.idl) $(location SVG/SVGClipPathElement.idl) $(location SVG/SVGComponentTransferFunctionElement.idl) $(location SVG/SVGDefsElement.idl) $(location SVG/SVGDescElement.idl) $(location SVG/SVGElement.idl) $(location SVG/SVGEllipseElement.idl) $(location SVG/SVGFEBlendElement.idl) $(location SVG/SVGFEColorMatrixElement.idl) $(location SVG/SVGFEComponentTransferElement.idl) $(location SVG/SVGFECompositeElement.idl) $(location SVG/SVGFEDisplacementMapElement.idl) $(location SVG/SVGFEDropShadowElement.idl) $(location SVG/SVGFEFloodElement.idl) $(location SVG/SVGFEFuncAElement.idl) $(location SVG/SVGFEFuncBElement.idl) $(location SVG/SVGFEFuncGElement.idl) $(location SVG/SVGFEFuncRElement.idl) $(location SVG/SVGFEGaussianBlurElement.idl) $(location SVG/SVGFEImageElement.idl) $(location SVG/SVGFEMergeElement.idl) $(location SVG/SVGFEMergeNodeElement.idl) $(location SVG/SVGFEMorphologyElement.idl) $(location SVG/SVGFEOffsetElement.idl) $(location SVG/SVGFETurbulenceElement.idl) $(location SVG/SVGFilterElement.idl) $(location SVG/SVGFilterPrimitiveStandardAttributes.idl) $(location SVG/SVGFitToViewBox.idl) $(location SVG/SVGForeignObjectElement.idl) $(location SVG/SVGGElement.idl) $(location SVG/SVGGeometryElement.idl) $(location SVG/SVGGradientElement.idl) $(location SVG/SVGGraphicsElement.idl) $(location SVG/SVGImageElement.idl) $(location SVG/SVGLength.idl) $(location SVG/SVGLengthList.idl) $(location SVG/SVGLinearGradientElement.idl) $(location SVG/SVGLineElement.idl) $(location SVG/SVGMaskElement.idl) $(location SVG/SVGMetadataElement.idl) $(location SVG/SVGNumber.idl) $(location SVG/SVGNumberList.idl) $(location SVG/SVGPathElement.idl) $(location SVG/SVGPatternElement.idl) $(location SVG/SVGPolygonElement.idl) $(location SVG/SVGPolylineElement.idl) $(location SVG/SVGRadialGradientElement.idl) $(location SVG/SVGRectElement.idl) $(location SVG/SVGScriptElement.idl) $(location SVG/SVGStopElement.idl) $(location SVG/SVGStyleElement.idl) $(location SVG/SVGSVGElement.idl) $(location SVG/SVGSwitchElement.idl) $(location SVG/SVGSymbolElement.idl) $(location SVG/SVGTextContentElement.idl) $(location SVG/SVGTextElement.idl) $(location SVG/SVGTextPathElement.idl) $(location SVG/SVGTextPositioningElement.idl) $(location SVG/SVGTitleElement.idl) $(location SVG/SVGTransform.idl) $(location SVG/SVGTransformList.idl) $(location SVG/SVGTSpanElement.idl) $(location SVG/SVGUnitTypes.idl) $(location SVG/SVGURIReference.idl) $(location SVG/SVGUseElement.idl) $(location SVG/SVGViewElement.idl) $(location TrustedTypes/TrustedHTML.idl) $(location TrustedTypes/TrustedScript.idl) $(location TrustedTypes/TrustedScriptURL.idl) $(location TrustedTypes/TrustedTypePolicy.idl) $(location TrustedTypes/TrustedTypePolicyFactory.idl) $(location UIEvents/CompositionEvent.idl) $(location UIEvents/EventModifier.idl) $(location UIEvents/FocusEvent.idl) $(location UIEvents/InputEvent.idl) $(location UIEvents/KeyboardEvent.idl) $(location UIEvents/MouseEvent.idl) $(location UIEvents/PointerEvent.idl) $(location UIEvents/PointerEventHandlers.idl) $(location UIEvents/TextEvent.idl) $(location UIEvents/UIEvent.idl) $(location UIEvents/WheelEvent.idl) $(location URLPattern/URLPattern.idl) $(location UserTiming/PerformanceMark.idl) $(location UserTiming/PerformanceMeasure.idl) $(location ViewTransition/ViewTransition.idl) $(location WebLocks/Lock.idl) $(location WebLocks/LockManager.idl) $(location WebLocks/NavigatorLocks.idl) $(location WebAssembly/Global.idl) $(location WebAssembly/Instance.idl) $(location WebAssembly/Memory.idl) $(location WebAssembly/Module.idl) $(location WebAssembly/Table.idl) $(location WebAssembly/WebAssembly.idl) $(location WebAudio/AnalyserNode.idl) $(location WebAudio/AudioBuffer.idl) $(location WebAudio/AudioBufferSourceNode.idl) $(location WebAudio/AudioContext.idl) $(location WebAudio/AudioDestinationNode.idl) $(location WebAudio/AudioListener.idl) $(location WebAudio/AudioNode.idl) $(location WebAudio/AudioParam.idl) $(location WebAudio/AudioScheduledSourceNode.idl) $(location WebAudio/BaseAudioContext.idl) $(location WebAudio/BiquadFilterNode.idl) $(location WebAudio/ChannelMergerNode.idl) $(location WebAudio/ChannelSplitterNode.idl) $(location WebAudio/ConstantSourceNode.idl) $(location WebAudio/DelayNode.idl) $(location WebAudio/DynamicsCompressorNode.idl) $(location WebAudio/GainNode.idl) $(location WebAudio/MediaElementAudioSourceNode.idl) $(location WebAudio/OfflineAudioCompletionEvent.idl) $(location WebAudio/OfflineAudioContext.idl) $(location WebAudio/OscillatorNode.idl) $(location WebAudio/PannerNode.idl) $(location WebAudio/PeriodicWave.idl) $(location WebAudio/ScriptProcessorNode.idl) $(location WebAudio/StereoPannerNode.idl) $(location WebGL/Extensions/ANGLEInstancedArrays.idl) $(location WebGL/Extensions/EXTBlendMinMax.idl) $(location WebGL/Extensions/EXTColorBufferFloat.idl) $(location WebGL/Extensions/EXTRenderSnorm.idl) $(location WebGL/Extensions/EXTTextureFilterAnisotropic.idl) $(location WebGL/Extensions/EXTTextureNorm16.idl) $(location WebGL/Extensions/OESElementIndexUint.idl) $(location WebGL/Extensions/OESStandardDerivatives.idl) $(location WebGL/Extensions/OESVertexArrayObject.idl) $(location WebGL/Extensions/WebGLCompressedTextureS3tc.idl) $(location WebGL/Extensions/WebGLCompressedTextureS3tcSrgb.idl) $(location WebGL/Extensions/WebGLDebugRendererInfo.idl) $(location WebGL/Extensions/WebGLDrawBuffers.idl) $(location WebGL/Extensions/WebGLVertexArrayObjectOES.idl) $(location WebGL/Types.idl) $(location WebGL/WebGL2RenderingContext.idl) $(location WebGL/WebGL2RenderingContextBase.idl) $(location WebGL/WebGL2RenderingContextOverloads.idl) $(location WebGL/WebGLActiveInfo.idl) $(location WebGL/WebGLBuffer.idl) $(location WebGL/WebGLContextEvent.idl) $(location WebGL/WebGLFramebuffer.idl) $(location WebGL/WebGLObject.idl) $(location WebGL/WebGLProgram.idl) $(location WebGL/WebGLQuery.idl) $(location WebGL/WebGLRenderbuffer.idl) $(location WebGL/WebGLRenderingContext.idl) $(location WebGL/WebGLRenderingContextBase.idl) $(location WebGL/WebGLRenderingContextOverloads.idl) $(location WebGL/WebGLSampler.idl) $(location WebGL/WebGLShader.idl) $(location WebGL/WebGLShaderPrecisionFormat.idl) $(location WebGL/WebGLSync.idl) $(location WebGL/WebGLTexture.idl) $(location WebGL/WebGLTransformFeedback.idl) $(location WebGL/WebGLUniformLocation.idl) $(location WebGL/WebGLVertexArrayObject.idl) $(location WebIDL/Buffers.idl) $(location WebIDL/DOMException.idl) $(location WebIDL/Function.idl) $(location WebIDL/QuotaExceededError.idl) $(location WebSockets/WebSocket.idl) $(location WebVTT/VTTCue.idl) $(location WebVTT/VTTRegion.idl) $(location WebXR/XRLayer.idl) $(location WebXR/XRRenderState.idl) $(location WebXR/XRSession.idl) $(location WebXR/XRSessionEvent.idl) $(location WebXR/XRSystem.idl) $(location WebXR/XRWebGLLayer.idl) $(location XHR/FormData.idl) $(location XHR/ProgressEvent.idl) $(location XHR/XMLHttpRequest.idl) $(location XHR/XMLHttpRequestEventTarget.idl) $(location XHR/XMLHttpRequestUpload.idl) $(location XPath/XPathEvaluator.idl) $(location XPath/XPathExpression.idl) $(location XPath/XPathNSResolver.idl) $(location XPath/XPathResult.idl)', + cmd = 'PYTHONHASHSEED=0 python3 Meta/Generators/generate_libweb_bindings.py -o $(RULEDIR)/Bindings $(location Animations/Animatable.idl) $(location Animations/Animation.idl) $(location Animations/AnimationEffect.idl) $(location Animations/AnimationPlaybackEvent.idl) $(location Animations/AnimationTimeline.idl) $(location Animations/DocumentTimeline.idl) $(location Animations/KeyframeEffect.idl) $(location Animations/ScrollTimeline.idl) $(location ARIA/ARIAMixin.idl) $(location Clipboard/Clipboard.idl) $(location Clipboard/ClipboardEvent.idl) $(location Clipboard/ClipboardItem.idl) $(location Compression/CompressionStream.idl) $(location Compression/DecompressionStream.idl) $(location ContentSecurityPolicy/SecurityPolicyViolationEvent.idl) $(location CookieStore/CookieChangeEvent.idl) $(location CookieStore/CookieStore.idl) $(location CredentialManagement/Credential.idl) $(location CredentialManagement/CredentialsContainer.idl) $(location CredentialManagement/FederatedCredential.idl) $(location CredentialManagement/PasswordCredential.idl) $(location Crypto/Crypto.idl) $(location Crypto/CryptoKey.idl) $(location Crypto/SubtleCrypto.idl) $(location CSS/AnimationEvent.idl) $(location CSS/CSS.idl) $(location CSS/CSSAnimation.idl) $(location CSS/CSSConditionRule.idl) $(location CSS/CSSContainerRule.idl) $(location CSS/CSSCounterStyleRule.idl) $(location CSS/CSSFontFaceDescriptors.idl) $(location CSS/CSSFontFaceRule.idl) $(location CSS/CSSFontFeatureValuesMap.idl) $(location CSS/CSSFontFeatureValuesRule.idl) $(location CSS/CSSFunctionDeclarations.idl) $(location CSS/CSSFunctionDescriptors.idl) $(location CSS/CSSFunctionRule.idl) $(location CSS/CSSGroupingRule.idl) $(location CSS/CSSImageValue.idl) $(location CSS/CSSImportRule.idl) $(location CSS/CSSKeyframeRule.idl) $(location CSS/CSSKeyframesRule.idl) $(location CSS/CSSKeywordValue.idl) $(location CSS/CSSLayerBlockRule.idl) $(location CSS/CSSLayerStatementRule.idl) $(location CSS/CSSMarginRule.idl) $(location CSS/CSSMathClamp.idl) $(location CSS/CSSMathInvert.idl) $(location CSS/CSSMathMax.idl) $(location CSS/CSSMathMin.idl) $(location CSS/CSSMathNegate.idl) $(location CSS/CSSMathProduct.idl) $(location CSS/CSSMathSum.idl) $(location CSS/CSSMathValue.idl) $(location CSS/CSSMatrixComponent.idl) $(location CSS/CSSMediaRule.idl) $(location CSS/CSSNamespaceRule.idl) $(location CSS/CSSNestedDeclarations.idl) $(location CSS/CSSNumericArray.idl) $(location CSS/CSSNumericValue.idl) $(location CSS/CSSPageDescriptors.idl) $(location CSS/CSSPageRule.idl) $(location CSS/CSSPerspective.idl) $(location CSS/CSSPropertyRule.idl) $(location CSS/CSSRotate.idl) $(location CSS/CSSRule.idl) $(location CSS/CSSRuleList.idl) $(location CSS/CSSScale.idl) $(location CSS/CSSScopeRule.idl) $(location CSS/CSSSkew.idl) $(location CSS/CSSSkewX.idl) $(location CSS/CSSSkewY.idl) $(location CSS/CSSStyleDeclaration.idl) $(location CSS/CSSStyleProperties.idl) $(location CSS/CSSStyleRule.idl) $(location CSS/CSSStyleSheet.idl) $(location CSS/CSSStyleValue.idl) $(location CSS/CSSSupportsRule.idl) $(location CSS/CSSTransformComponent.idl) $(location CSS/CSSTransformValue.idl) $(location CSS/CSSTransition.idl) $(location CSS/CSSTranslate.idl) $(location CSS/CSSUnitValue.idl) $(location CSS/CSSUnparsedValue.idl) $(location CSS/CSSVariableReferenceValue.idl) $(location CSS/ElementCSSInlineStyle.idl) $(location CSS/FontFace.idl) $(location CSS/FontFaceSet.idl) $(location CSS/FontFaceSetLoadEvent.idl) $(location CSS/GeneratedCSSNumericFactoryMethods.idl) $(location CSS/GeneratedCSSStyleProperties.idl) $(location CSS/LinkStyle.idl) $(location CSS/MediaList.idl) $(location CSS/MediaQueryList.idl) $(location CSS/MediaQueryListEvent.idl) $(location CSS/Screen.idl) $(location CSS/ScreenOrientation.idl) $(location CSS/StylePropertyMap.idl) $(location CSS/StylePropertyMapReadOnly.idl) $(location CSS/StyleSheet.idl) $(location CSS/StyleSheetList.idl) $(location CSS/TransitionEvent.idl) $(location CSS/VisualViewport.idl) $(location DOM/AbortController.idl) $(location DOM/AbortSignal.idl) $(location DOM/AbstractRange.idl) $(location DOM/Attr.idl) $(location DOM/CDATASection.idl) $(location DOM/CaretPosition.idl) $(location DOM/CharacterData.idl) $(location DOM/ChildNode.idl) $(location DOM/Comment.idl) $(location DOM/CustomEvent.idl) $(location DOM/Document.idl) $(location DOM/DocumentFragment.idl) $(location DOM/DocumentOrShadowRoot.idl) $(location DOM/DocumentType.idl) $(location DOM/DOMImplementation.idl) $(location DOM/DOMTokenList.idl) $(location DOM/Element.idl) $(location DOM/Event.idl) $(location DOM/EventHandler.idl) $(location DOM/EventListener.idl) $(location DOM/EventTarget.idl) $(location DOM/HTMLCollection.idl) $(location DOM/MutationObserver.idl) $(location DOM/MutationRecord.idl) $(location DOM/NamedNodeMap.idl) $(location DOM/Node.idl) $(location DOM/NodeFilter.idl) $(location DOM/NodeIterator.idl) $(location DOM/NodeList.idl) $(location DOM/ParentNode.idl) $(location DOM/ProcessingInstruction.idl) $(location DOM/Range.idl) $(location DOM/ShadowRoot.idl) $(location DOM/Slottable.idl) $(location DOM/StaticRange.idl) $(location DOM/Text.idl) $(location DOM/TreeWalker.idl) $(location DOM/XMLDocument.idl) $(location DOMURL/DOMURL.idl) $(location DOMURL/Origin.idl) $(location DOMURL/URLSearchParams.idl) $(location Encoding/TextDecoder.idl) $(location Encoding/TextDecoderCommon.idl) $(location Encoding/TextDecoderStream.idl) $(location Encoding/TextEncoder.idl) $(location Encoding/TextEncoderCommon.idl) $(location Encoding/TextEncoderStream.idl) $(location EncryptedMediaExtensions/MediaKeySystemAccess.idl) $(location EntriesAPI/FileSystemEntry.idl) $(location EventTiming/PerformanceEventTiming.idl) $(location Fetch/Body.idl) $(location Fetch/BodyInit.idl) $(location Fetch/Headers.idl) $(location Fetch/Request.idl) $(location Fetch/Response.idl) $(location FileAPI/Blob.idl) $(location FileAPI/File.idl) $(location FileAPI/FileList.idl) $(location FileAPI/FileReader.idl) $(location FileAPI/FileReaderSync.idl) $(location Fullscreen/DocumentExtensions.idl) $(location Fullscreen/DocumentOrShadowRootExtensions.idl) $(location Fullscreen/ElementExtensions.idl) $(location Gamepad/Gamepad.idl) $(location Gamepad/GamepadButton.idl) $(location Gamepad/GamepadEvent.idl) $(location Gamepad/GamepadHapticActuator.idl) $(location Geolocation/Geolocation.idl) $(location Geolocation/GeolocationCoordinates.idl) $(location Geolocation/GeolocationPosition.idl) $(location Geolocation/GeolocationPositionError.idl) $(location Geometry/DOMMatrix.idl) $(location Geometry/DOMMatrixReadOnly.idl) $(location Geometry/DOMPoint.idl) $(location Geometry/DOMPointReadOnly.idl) $(location Geometry/DOMQuad.idl) $(location Geometry/DOMRect.idl) $(location Geometry/DOMRectList.idl) $(location Geometry/DOMRectReadOnly.idl) $(location GPC/GlobalPrivacyControl.idl) $(location HighResolutionTime/DOMHighResTimeStamp.idl) $(location HighResolutionTime/EpochTimeStamp.idl) $(location HighResolutionTime/Performance.idl) $(location HTML/AbstractWorker.idl) $(location HTML/AnimationFrameProvider.idl) $(location HTML/AudioTrack.idl) $(location HTML/AudioTrackList.idl) $(location HTML/BarProp.idl) $(location HTML/BeforeUnloadEvent.idl) $(location HTML/BroadcastChannel.idl) $(location HTML/Canvas/CanvasCompositing.idl) $(location HTML/Canvas/CanvasDrawImage.idl) $(location HTML/Canvas/CanvasDrawPath.idl) $(location HTML/Canvas/CanvasFillStrokeStyles.idl) $(location HTML/Canvas/CanvasFilters.idl) $(location HTML/Canvas/CanvasImageData.idl) $(location HTML/Canvas/CanvasImageSmoothing.idl) $(location HTML/Canvas/CanvasPath.idl) $(location HTML/Canvas/CanvasPathDrawingStyles.idl) $(location HTML/Canvas/CanvasRect.idl) $(location HTML/Canvas/CanvasSettings.idl) $(location HTML/Canvas/CanvasShadowStyles.idl) $(location HTML/Canvas/CanvasState.idl) $(location HTML/Canvas/CanvasText.idl) $(location HTML/Canvas/CanvasTextDrawingStyles.idl) $(location HTML/Canvas/CanvasTransform.idl) $(location HTML/Canvas/CanvasUserInterface.idl) $(location HTML/Canvas/OffscreenCanvasBase.idl) $(location HTML/CanvasGradient.idl) $(location HTML/CanvasPattern.idl) $(location HTML/CanvasRenderingContext2D.idl) $(location HTML/CanvasRenderingContext2DSettings.idl) $(location HTML/CloseEvent.idl) $(location HTML/CloseWatcher.idl) $(location HTML/CommandEvent.idl) $(location HTML/CustomElements/CustomElementRegistry.idl) $(location HTML/CustomElements/CustomStateSet.idl) $(location HTML/DataTransfer.idl) $(location HTML/DataTransferItem.idl) $(location HTML/DataTransferItemList.idl) $(location HTML/DedicatedWorkerGlobalScope.idl) $(location HTML/DOMParser.idl) $(location HTML/DOMStringList.idl) $(location HTML/DOMStringMap.idl) $(location HTML/DragEvent.idl) $(location HTML/ElementInternals.idl) $(location HTML/ErrorEvent.idl) $(location HTML/EventSource.idl) $(location HTML/External.idl) $(location HTML/FormDataEvent.idl) $(location HTML/HashChangeEvent.idl) $(location HTML/History.idl) $(location HTML/HTMLAllCollection.idl) $(location HTML/HTMLAnchorElement.idl) $(location HTML/HTMLAreaElement.idl) $(location HTML/HTMLAudioElement.idl) $(location HTML/HTMLBaseElement.idl) $(location HTML/HTMLBodyElement.idl) $(location HTML/HTMLBRElement.idl) $(location HTML/HTMLButtonElement.idl) $(location HTML/HTMLCanvasElement.idl) $(location HTML/HTMLDataElement.idl) $(location HTML/HTMLDataListElement.idl) $(location HTML/HTMLDetailsElement.idl) $(location HTML/HTMLDialogElement.idl) $(location HTML/HTMLDirectoryElement.idl) $(location HTML/HTMLDivElement.idl) $(location HTML/HTMLDListElement.idl) $(location HTML/HTMLDocument.idl) $(location HTML/HTMLElement.idl) $(location HTML/HTMLEmbedElement.idl) $(location HTML/HTMLFieldSetElement.idl) $(location HTML/HTMLFontElement.idl) $(location HTML/HTMLFormControlsCollection.idl) $(location HTML/HTMLFormElement.idl) $(location HTML/HTMLFrameElement.idl) $(location HTML/HTMLFrameSetElement.idl) $(location HTML/HTMLHeadElement.idl) $(location HTML/HTMLHeadingElement.idl) $(location HTML/HTMLHRElement.idl) $(location HTML/HTMLHtmlElement.idl) $(location HTML/HTMLHyperlinkElementUtils.idl) $(location HTML/HTMLIFrameElement.idl) $(location HTML/HTMLImageElement.idl) $(location HTML/HTMLInputElement.idl) $(location HTML/HTMLLabelElement.idl) $(location HTML/HTMLLegendElement.idl) $(location HTML/HTMLLIElement.idl) $(location HTML/HTMLLinkElement.idl) $(location HTML/HTMLMapElement.idl) $(location HTML/HTMLMarqueeElement.idl) $(location HTML/HTMLMediaElement.idl) $(location HTML/HTMLMenuElement.idl) $(location HTML/HTMLMetaElement.idl) $(location HTML/HTMLMeterElement.idl) $(location HTML/HTMLModElement.idl) $(location HTML/HTMLObjectElement.idl) $(location HTML/HTMLOListElement.idl) $(location HTML/HTMLOptGroupElement.idl) $(location HTML/HTMLOptionElement.idl) $(location HTML/HTMLOptionsCollection.idl) $(location HTML/HTMLOrSVGOrMathMLElement.idl) $(location HTML/HTMLOutputElement.idl) $(location HTML/HTMLParagraphElement.idl) $(location HTML/HTMLParamElement.idl) $(location HTML/HTMLPictureElement.idl) $(location HTML/HTMLPreElement.idl) $(location HTML/HTMLProgressElement.idl) $(location HTML/HTMLQuoteElement.idl) $(location HTML/HTMLScriptElement.idl) $(location HTML/HTMLSelectedContentElement.idl) $(location HTML/HTMLSelectElement.idl) $(location HTML/HTMLSlotElement.idl) $(location HTML/HTMLSourceElement.idl) $(location HTML/HTMLSpanElement.idl) $(location HTML/HTMLStyleElement.idl) $(location HTML/HTMLTableCaptionElement.idl) $(location HTML/HTMLTableCellElement.idl) $(location HTML/HTMLTableColElement.idl) $(location HTML/HTMLTableElement.idl) $(location HTML/HTMLTableRowElement.idl) $(location HTML/HTMLTableSectionElement.idl) $(location HTML/HTMLTemplateElement.idl) $(location HTML/HTMLTextAreaElement.idl) $(location HTML/HTMLTimeElement.idl) $(location HTML/HTMLTitleElement.idl) $(location HTML/HTMLTrackElement.idl) $(location HTML/HTMLUListElement.idl) $(location HTML/HTMLUnknownElement.idl) $(location HTML/HTMLVideoElement.idl) $(location HTML/HyperlinkElementUtils.idl) $(location HTML/ImageBitmap.idl) $(location HTML/ImageData.idl) $(location HTML/Location.idl) $(location HTML/MediaError.idl) $(location HTML/MessageChannel.idl) $(location HTML/MessageEvent.idl) $(location HTML/MessagePort.idl) $(location HTML/MimeType.idl) $(location HTML/MimeTypeArray.idl) $(location HTML/NavigateEvent.idl) $(location HTML/Navigation.idl) $(location HTML/NavigationActivation.idl) $(location HTML/NavigationCurrentEntryChangeEvent.idl) $(location HTML/NavigationDestination.idl) $(location HTML/NavigationHistoryEntry.idl) $(location HTML/NavigationTransition.idl) $(location HTML/NavigationType.idl) $(location HTML/Navigator.idl) $(location HTML/NavigatorBeacon.idl) $(location HTML/NavigatorConcurrentHardware.idl) $(location HTML/NavigatorDeviceMemory.idl) $(location HTML/NavigatorID.idl) $(location HTML/NavigatorLanguage.idl) $(location HTML/NavigatorOnLine.idl) $(location HTML/OffscreenCanvas.idl) $(location HTML/OffscreenCanvasRenderingContext2D.idl) $(location HTML/PageSwapEvent.idl) $(location HTML/PageTransitionEvent.idl) $(location HTML/Path2D.idl) $(location HTML/Plugin.idl) $(location HTML/PluginArray.idl) $(location HTML/PopoverTargetAttributes.idl) $(location HTML/PopStateEvent.idl) $(location HTML/PredefinedColorSpace.idl) $(location HTML/PromiseRejectionEvent.idl) $(location HTML/RadioNodeList.idl) $(location HTML/Scripting/Fetching.idl) $(location HTML/SharedWorker.idl) $(location HTML/SharedWorkerGlobalScope.idl) $(location HTML/Storage.idl) $(location HTML/StorageEvent.idl) $(location HTML/SubmitEvent.idl) $(location HTML/TextMetrics.idl) $(location HTML/TextTrack.idl) $(location HTML/TextTrackCue.idl) $(location HTML/TextTrackCueList.idl) $(location HTML/TextTrackList.idl) $(location HTML/TimeRanges.idl) $(location HTML/ToggleEvent.idl) $(location HTML/TrackEvent.idl) $(location HTML/UniversalGlobalScope.idl) $(location HTML/UserActivation.idl) $(location HTML/ValidityState.idl) $(location HTML/VideoTrack.idl) $(location HTML/VideoTrackList.idl) $(location HTML/Window.idl) $(location HTML/WindowDeprecated.idl) $(location HTML/WindowLocalStorage.idl) $(location HTML/WindowOrWorkerGlobalScope.idl) $(location HTML/WindowSessionStorage.idl) $(location HTML/Worker.idl) $(location HTML/WorkerGlobalScope.idl) $(location HTML/WorkerLocation.idl) $(location HTML/WorkerNavigator.idl) $(location HTML/WorkletGlobalScope.idl) $(location HTML/XMLSerializer.idl) $(location IndexedDB/IDBCursor.idl) $(location IndexedDB/IDBCursorWithValue.idl) $(location IndexedDB/IDBDatabase.idl) $(location IndexedDB/IDBFactory.idl) $(location IndexedDB/IDBIndex.idl) $(location IndexedDB/IDBKeyRange.idl) $(location IndexedDB/IDBObjectStore.idl) $(location IndexedDB/IDBOpenDBRequest.idl) $(location IndexedDB/IDBRecord.idl) $(location IndexedDB/IDBRequest.idl) $(location IndexedDB/IDBTransaction.idl) $(location IndexedDB/IDBVersionChangeEvent.idl) $(location Internals/FakeXRDevice.idl) $(location Internals/InternalAnimationTimeline.idl) $(location Internals/InternalGamepad.idl) $(location Internals/Internals.idl) $(location Internals/WebUI.idl) $(location Internals/XRTest.idl) $(location IntersectionObserver/IntersectionObserver.idl) $(location IntersectionObserver/IntersectionObserverEntry.idl) $(location MathML/MathMLAnchorElement.idl) $(location MathML/MathMLElement.idl) $(location MediaCapabilitiesAPI/MediaCapabilities.idl) $(location MediaCapture/MediaDeviceInfo.idl) $(location MediaCapture/MediaDevices.idl) $(location MediaCapture/MediaStream.idl) $(location MediaCapture/MediaStreamConstraints.idl) $(location MediaCapture/MediaStreamTrack.idl) $(location MediaCapture/MediaStreamTrackEvent.idl) $(location MediaSourceExtensions/BufferedChangeEvent.idl) $(location MediaSourceExtensions/ManagedMediaSource.idl) $(location MediaSourceExtensions/ManagedSourceBuffer.idl) $(location MediaSourceExtensions/MediaSource.idl) $(location MediaSourceExtensions/MediaSourceHandle.idl) $(location MediaSourceExtensions/SourceBuffer.idl) $(location MediaSourceExtensions/SourceBufferList.idl) $(location NavigationTiming/PerformanceExtensions.idl) $(location NavigationTiming/PerformanceNavigation.idl) $(location NavigationTiming/PerformanceTiming.idl) $(location NotificationsAPI/Notification.idl) $(location PerformanceTimeline/PerformanceEntry.idl) $(location PerformanceTimeline/PerformanceObserver.idl) $(location PerformanceTimeline/PerformanceObserverEntryList.idl) $(location PermissionsAPI/Permissions.idl) $(location PermissionsAPI/PermissionStatus.idl) $(location RequestIdleCallback/IdleDeadline.idl) $(location RequestIdleCallback/IdleRequest.idl) $(location ResizeObserver/ResizeObserver.idl) $(location ResizeObserver/ResizeObserverEntry.idl) $(location ResizeObserver/ResizeObserverSize.idl) $(location ResourceTiming/PerformanceResourceTiming.idl) $(location Selection/Selection.idl) $(location Serial/Serial.idl) $(location Serial/SerialPort.idl) $(location ServiceWorker/Cache.idl) $(location ServiceWorker/CacheStorage.idl) $(location ServiceWorker/ServiceWorker.idl) $(location ServiceWorker/ServiceWorkerContainer.idl) $(location ServiceWorker/ServiceWorkerGlobalScope.idl) $(location ServiceWorker/ServiceWorkerRegistration.idl) $(location Speech/SpeechGrammar.idl) $(location Speech/SpeechGrammarList.idl) $(location Speech/SpeechRecognition.idl) $(location Speech/SpeechRecognitionAlternative.idl) $(location Speech/SpeechRecognitionEvent.idl) $(location Speech/SpeechRecognitionPhrase.idl) $(location Speech/SpeechRecognitionResult.idl) $(location Speech/SpeechRecognitionResultList.idl) $(location Speech/SpeechSynthesis.idl) $(location Speech/SpeechSynthesisUtterance.idl) $(location Speech/SpeechSynthesisVoice.idl) $(location StorageAPI/NavigatorStorage.idl) $(location StorageAPI/StorageManager.idl) $(location Streams/ByteLengthQueuingStrategy.idl) $(location Streams/CountQueuingStrategy.idl) $(location Streams/GenericTransformStream.idl) $(location Streams/QueuingStrategy.idl) $(location Streams/QueuingStrategyInit.idl) $(location Streams/ReadableByteStreamController.idl) $(location Streams/ReadableStream.idl) $(location Streams/ReadableStreamBYOBReader.idl) $(location Streams/ReadableStreamBYOBRequest.idl) $(location Streams/ReadableStreamDefaultController.idl) $(location Streams/ReadableStreamDefaultReader.idl) $(location Streams/ReadableStreamGenericReader.idl) $(location Streams/Transformer.idl) $(location Streams/TransformStream.idl) $(location Streams/TransformStreamDefaultController.idl) $(location Streams/UnderlyingSink.idl) $(location Streams/UnderlyingSource.idl) $(location Streams/WritableStream.idl) $(location Streams/WritableStreamDefaultController.idl) $(location Streams/WritableStreamDefaultWriter.idl) $(location SVG/SVGAElement.idl) $(location SVG/SVGAnimatedEnumeration.idl) $(location SVG/SVGAnimatedInteger.idl) $(location SVG/SVGAnimatedLength.idl) $(location SVG/SVGAnimatedLengthList.idl) $(location SVG/SVGAnimatedNumber.idl) $(location SVG/SVGAnimatedNumberList.idl) $(location SVG/SVGAnimatedRect.idl) $(location SVG/SVGAnimatedString.idl) $(location SVG/SVGAnimatedTransformList.idl) $(location SVG/SVGAnimationElement.idl) $(location SVG/SVGCircleElement.idl) $(location SVG/SVGClipPathElement.idl) $(location SVG/SVGComponentTransferFunctionElement.idl) $(location SVG/SVGDefsElement.idl) $(location SVG/SVGDescElement.idl) $(location SVG/SVGElement.idl) $(location SVG/SVGEllipseElement.idl) $(location SVG/SVGFEBlendElement.idl) $(location SVG/SVGFEColorMatrixElement.idl) $(location SVG/SVGFEComponentTransferElement.idl) $(location SVG/SVGFECompositeElement.idl) $(location SVG/SVGFEDisplacementMapElement.idl) $(location SVG/SVGFEDropShadowElement.idl) $(location SVG/SVGFEFloodElement.idl) $(location SVG/SVGFEFuncAElement.idl) $(location SVG/SVGFEFuncBElement.idl) $(location SVG/SVGFEFuncGElement.idl) $(location SVG/SVGFEFuncRElement.idl) $(location SVG/SVGFEGaussianBlurElement.idl) $(location SVG/SVGFEImageElement.idl) $(location SVG/SVGFEMergeElement.idl) $(location SVG/SVGFEMergeNodeElement.idl) $(location SVG/SVGFEMorphologyElement.idl) $(location SVG/SVGFEOffsetElement.idl) $(location SVG/SVGFETurbulenceElement.idl) $(location SVG/SVGFilterElement.idl) $(location SVG/SVGFilterPrimitiveStandardAttributes.idl) $(location SVG/SVGFitToViewBox.idl) $(location SVG/SVGForeignObjectElement.idl) $(location SVG/SVGGElement.idl) $(location SVG/SVGGeometryElement.idl) $(location SVG/SVGGradientElement.idl) $(location SVG/SVGGraphicsElement.idl) $(location SVG/SVGImageElement.idl) $(location SVG/SVGLength.idl) $(location SVG/SVGLengthList.idl) $(location SVG/SVGLinearGradientElement.idl) $(location SVG/SVGLineElement.idl) $(location SVG/SVGMaskElement.idl) $(location SVG/SVGMetadataElement.idl) $(location SVG/SVGNumber.idl) $(location SVG/SVGNumberList.idl) $(location SVG/SVGPathElement.idl) $(location SVG/SVGPatternElement.idl) $(location SVG/SVGPolygonElement.idl) $(location SVG/SVGPolylineElement.idl) $(location SVG/SVGRadialGradientElement.idl) $(location SVG/SVGRectElement.idl) $(location SVG/SVGScriptElement.idl) $(location SVG/SVGStopElement.idl) $(location SVG/SVGStyleElement.idl) $(location SVG/SVGSVGElement.idl) $(location SVG/SVGSwitchElement.idl) $(location SVG/SVGSymbolElement.idl) $(location SVG/SVGTextContentElement.idl) $(location SVG/SVGTextElement.idl) $(location SVG/SVGTextPathElement.idl) $(location SVG/SVGTextPositioningElement.idl) $(location SVG/SVGTitleElement.idl) $(location SVG/SVGTransform.idl) $(location SVG/SVGTransformList.idl) $(location SVG/SVGTSpanElement.idl) $(location SVG/SVGUnitTypes.idl) $(location SVG/SVGURIReference.idl) $(location SVG/SVGUseElement.idl) $(location SVG/SVGViewElement.idl) $(location TrustedTypes/TrustedHTML.idl) $(location TrustedTypes/TrustedScript.idl) $(location TrustedTypes/TrustedScriptURL.idl) $(location TrustedTypes/TrustedTypePolicy.idl) $(location TrustedTypes/TrustedTypePolicyFactory.idl) $(location UIEvents/CompositionEvent.idl) $(location UIEvents/EventModifier.idl) $(location UIEvents/FocusEvent.idl) $(location UIEvents/InputEvent.idl) $(location UIEvents/KeyboardEvent.idl) $(location UIEvents/MouseEvent.idl) $(location UIEvents/PointerEvent.idl) $(location UIEvents/PointerEventHandlers.idl) $(location UIEvents/TextEvent.idl) $(location UIEvents/UIEvent.idl) $(location UIEvents/WheelEvent.idl) $(location URLPattern/URLPattern.idl) $(location UserTiming/PerformanceMark.idl) $(location UserTiming/PerformanceMeasure.idl) $(location ViewTransition/ViewTransition.idl) $(location WebLocks/Lock.idl) $(location WebLocks/LockManager.idl) $(location WebLocks/NavigatorLocks.idl) $(location WebAssembly/Global.idl) $(location WebAssembly/Instance.idl) $(location WebAssembly/Memory.idl) $(location WebAssembly/Module.idl) $(location WebAssembly/Table.idl) $(location WebAssembly/WebAssembly.idl) $(location WebAudio/AnalyserNode.idl) $(location WebAudio/AudioBuffer.idl) $(location WebAudio/AudioBufferSourceNode.idl) $(location WebAudio/AudioContext.idl) $(location WebAudio/AudioDestinationNode.idl) $(location WebAudio/AudioListener.idl) $(location WebAudio/AudioNode.idl) $(location WebAudio/AudioParam.idl) $(location WebAudio/AudioScheduledSourceNode.idl) $(location WebAudio/BaseAudioContext.idl) $(location WebAudio/BiquadFilterNode.idl) $(location WebAudio/ChannelMergerNode.idl) $(location WebAudio/ChannelSplitterNode.idl) $(location WebAudio/ConstantSourceNode.idl) $(location WebAudio/DelayNode.idl) $(location WebAudio/DynamicsCompressorNode.idl) $(location WebAudio/GainNode.idl) $(location WebAudio/MediaElementAudioSourceNode.idl) $(location WebAudio/OfflineAudioCompletionEvent.idl) $(location WebAudio/OfflineAudioContext.idl) $(location WebAudio/OscillatorNode.idl) $(location WebAudio/PannerNode.idl) $(location WebAudio/PeriodicWave.idl) $(location WebAudio/ScriptProcessorNode.idl) $(location WebAudio/StereoPannerNode.idl) $(location WebGL/Extensions/ANGLEInstancedArrays.idl) $(location WebGL/Extensions/EXTBlendMinMax.idl) $(location WebGL/Extensions/EXTColorBufferFloat.idl) $(location WebGL/Extensions/EXTRenderSnorm.idl) $(location WebGL/Extensions/EXTTextureFilterAnisotropic.idl) $(location WebGL/Extensions/EXTTextureNorm16.idl) $(location WebGL/Extensions/OESElementIndexUint.idl) $(location WebGL/Extensions/OESStandardDerivatives.idl) $(location WebGL/Extensions/OESVertexArrayObject.idl) $(location WebGL/Extensions/WebGLCompressedTextureS3tc.idl) $(location WebGL/Extensions/WebGLCompressedTextureS3tcSrgb.idl) $(location WebGL/Extensions/WebGLDebugRendererInfo.idl) $(location WebGL/Extensions/WebGLDrawBuffers.idl) $(location WebGL/Extensions/WebGLVertexArrayObjectOES.idl) $(location WebGL/Types.idl) $(location WebGL/WebGL2RenderingContext.idl) $(location WebGL/WebGL2RenderingContextBase.idl) $(location WebGL/WebGL2RenderingContextOverloads.idl) $(location WebGL/WebGLActiveInfo.idl) $(location WebGL/WebGLBuffer.idl) $(location WebGL/WebGLContextEvent.idl) $(location WebGL/WebGLFramebuffer.idl) $(location WebGL/WebGLObject.idl) $(location WebGL/WebGLProgram.idl) $(location WebGL/WebGLQuery.idl) $(location WebGL/WebGLRenderbuffer.idl) $(location WebGL/WebGLRenderingContext.idl) $(location WebGL/WebGLRenderingContextBase.idl) $(location WebGL/WebGLRenderingContextOverloads.idl) $(location WebGL/WebGLSampler.idl) $(location WebGL/WebGLShader.idl) $(location WebGL/WebGLShaderPrecisionFormat.idl) $(location WebGL/WebGLSync.idl) $(location WebGL/WebGLTexture.idl) $(location WebGL/WebGLTransformFeedback.idl) $(location WebGL/WebGLUniformLocation.idl) $(location WebGL/WebGLVertexArrayObject.idl) $(location WebIDL/Buffers.idl) $(location WebIDL/DOMException.idl) $(location WebIDL/Function.idl) $(location WebIDL/QuotaExceededError.idl) $(location WebSockets/WebSocket.idl) $(location WebVTT/VTTCue.idl) $(location WebVTT/VTTRegion.idl) $(location WebXR/XRLayer.idl) $(location WebXR/XRRenderState.idl) $(location WebXR/XRSession.idl) $(location WebXR/XRSessionEvent.idl) $(location WebXR/XRSystem.idl) $(location WebXR/XRWebGLLayer.idl) $(location XHR/FormData.idl) $(location XHR/ProgressEvent.idl) $(location XHR/XMLHttpRequest.idl) $(location XHR/XMLHttpRequestEventTarget.idl) $(location XHR/XMLHttpRequestUpload.idl) $(location XPath/XPathEvaluator.idl) $(location XPath/XPathExpression.idl) $(location XPath/XPathNSResolver.idl) $(location XPath/XPathResult.idl)', ) diff --git a/examples/ladybird/workspace/Libraries/LibWeb/export_header.bzl b/examples/ladybird/workspace/Libraries/LibWeb/export_header.bzl new file mode 100644 index 0000000..799e31c --- /dev/null +++ b/examples/ladybird/workspace/Libraries/LibWeb/export_header.bzl @@ -0,0 +1,17 @@ +# AUTO-GENERATED by Meta/emit_export_headers_bazel.py --libweb — do not edit. +# LibWeb's generate_export_header output. It lives here rather than in the +# root package because Libraries/LibWeb is its own Bazel package, so only +# this package may declare an output inside it. +load("@rules_cc//cc:defs.bzl", "cc_library") + +def libweb_export_header(): + native.genrule( + name = 'gen_LibWeb_Export_h', + outs = ['genroot/LibWeb/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef WEB_API_H\n#define WEB_API_H\n\n#ifdef LIBWEB_STATIC_DEFINE\n# define WEB_API\n# define LIBWEB_NO_EXPORT\n#else\n# ifndef WEB_API\n# ifdef LibWeb_EXPORTS\n /* We are building this library */\n# define WEB_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define WEB_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBWEB_NO_EXPORT\n# define LIBWEB_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBWEB_DEPRECATED\n# define LIBWEB_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBWEB_DEPRECATED_EXPORT\n# define LIBWEB_DEPRECATED_EXPORT WEB_API LIBWEB_DEPRECATED\n#endif\n\n#ifndef LIBWEB_DEPRECATED_NO_EXPORT\n# define LIBWEB_DEPRECATED_NO_EXPORT LIBWEB_NO_EXPORT LIBWEB_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBWEB_NO_DEPRECATED\n# define LIBWEB_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* WEB_API_H */\nLADYBIRD_EOF\n", + ) + cc_library( + name = 'generated_export_header', + hdrs = ['genroot/LibWeb/Export.h'], + includes = ['genroot'], + ) diff --git a/examples/ladybird/workspace/Libraries/LibWeb/generated_srcs.bzl b/examples/ladybird/workspace/Libraries/LibWeb/generated_srcs.bzl index d777c16..6cd311b 100644 --- a/examples/ladybird/workspace/Libraries/LibWeb/generated_srcs.bzl +++ b/examples/ladybird/workspace/Libraries/LibWeb/generated_srcs.bzl @@ -1,5 +1,7 @@ -# AUTO-GENERATED by Meta/emit_libweb_bazel.py — genrule outputs consumed by //Libraries/LibWeb:LibWeb -# Generated .cpp SRCS = the 688 the CMake reference compiles into LibWeb (GLFunctions.cpp is emitted but not compiled). +# AUTO-GENERATED by Meta/emit_libweb_bazel.py --generated-srcs. +# Do not edit: both lists are DERIVED (see generated_srcs_bzl there). +# Generated .cpp SRCS = the 692 the CMake reference compiles into LibWeb +# (a generated .cpp CMake does not compile is emitted but not listed). LIBWEB_GENERATED_SRCS = [ 'ARIA/AriaRoles.cpp', 'Bindings/ANGLEInstancedArrays.cpp', @@ -371,6 +373,7 @@ LIBWEB_GENERATED_SRCS = [ 'Bindings/NamedNodeMap.cpp', 'Bindings/NavigateEvent.cpp', 'Bindings/Navigation.cpp', + 'Bindings/NavigationActivation.cpp', 'Bindings/NavigationCurrentEntryChangeEvent.cpp', 'Bindings/NavigationDestination.cpp', 'Bindings/NavigationHistoryEntry.cpp', @@ -400,6 +403,7 @@ LIBWEB_GENERATED_SRCS = [ 'Bindings/OffscreenCanvasRenderingContext2D.cpp', 'Bindings/Origin.cpp', 'Bindings/OscillatorNode.cpp', + 'Bindings/PageSwapEvent.cpp', 'Bindings/PageTransitionEvent.cpp', 'Bindings/PannerNode.cpp', 'Bindings/ParentNode.cpp', @@ -558,6 +562,7 @@ LIBWEB_GENERATED_SRCS = [ 'Bindings/Storage.cpp', 'Bindings/StorageEvent.cpp', 'Bindings/StorageManager.cpp', + 'Bindings/StructuredSerializeBindings.cpp', 'Bindings/StylePropertyMap.cpp', 'Bindings/StylePropertyMapReadOnly.cpp', 'Bindings/StyleSheet.cpp', @@ -648,6 +653,7 @@ LIBWEB_GENERATED_SRCS = [ 'Bindings/WorkerLocation.cpp', 'Bindings/WorkerNavigator.cpp', 'Bindings/WorkletGlobalScope.cpp', + 'Bindings/WrapperFactory.cpp', 'Bindings/WritableStream.cpp', 'Bindings/WritableStreamDefaultController.cpp', 'Bindings/WritableStreamDefaultWriter.cpp', @@ -691,7 +697,7 @@ LIBWEB_GENERATED_SRCS = [ 'WebGL/WebGLContextProxy.cpp', ] -# All generated headers so resolves (includes=[".."]). +# All 693 generated headers, so resolves (includes=[".."]). LIBWEB_GENERATED_HDRS = [ 'ARIA/AriaRoles.h', 'Bindings/ANGLEInstancedArrays.h', @@ -861,6 +867,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/DecompressionStream.h', 'Bindings/DedicatedWorkerExposedInterfaces.h', 'Bindings/DedicatedWorkerGlobalScope.h', + 'Bindings/DedicatedWorkerGlobalScopeGlobalMixin.h', 'Bindings/DelayNode.h', 'Bindings/Document.h', 'Bindings/DocumentExtensions.h', @@ -1064,6 +1071,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/NamedNodeMap.h', 'Bindings/NavigateEvent.h', 'Bindings/Navigation.h', + 'Bindings/NavigationActivation.h', 'Bindings/NavigationCurrentEntryChangeEvent.h', 'Bindings/NavigationDestination.h', 'Bindings/NavigationHistoryEntry.h', @@ -1093,6 +1101,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/OffscreenCanvasRenderingContext2D.h', 'Bindings/Origin.h', 'Bindings/OscillatorNode.h', + 'Bindings/PageSwapEvent.h', 'Bindings/PageTransitionEvent.h', 'Bindings/PannerNode.h', 'Bindings/ParentNode.h', @@ -1232,6 +1241,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/SharedWorker.h', 'Bindings/SharedWorkerExposedInterfaces.h', 'Bindings/SharedWorkerGlobalScope.h', + 'Bindings/SharedWorkerGlobalScopeGlobalMixin.h', 'Bindings/Slottable.h', 'Bindings/SourceBuffer.h', 'Bindings/SourceBufferList.h', @@ -1333,6 +1343,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/Window.h', 'Bindings/WindowDeprecated.h', 'Bindings/WindowExposedInterfaces.h', + 'Bindings/WindowGlobalMixin.h', 'Bindings/WindowLocalStorage.h', 'Bindings/WindowOrWorkerGlobalScope.h', 'Bindings/WindowSessionStorage.h', @@ -1374,12 +1385,8 @@ LIBWEB_GENERATED_HDRS = [ 'CSS/PseudoElement.h', 'CSS/TransformFunctions.h', 'CSS/Units.h', - 'HTML/AttributeNames.h', 'HTML/MediaControlsDOM.h', 'HTML/Parser/NamedCharacterReferences.h', - 'HTML/TagNames.h', - 'SVG/AttributeNames.h', - 'SVG/TagNames.h', 'WebGL/GLFunctions.h', 'WebGL/WebGLCommands.h', 'WebGL/WebGLContextProxy.h', diff --git a/examples/ladybird/workspace/MODULE.bazel b/examples/ladybird/workspace/MODULE.bazel index 11b3f27..8d0cbe1 100644 --- a/examples/ladybird/workspace/MODULE.bazel +++ b/examples/ladybird/workspace/MODULE.bazel @@ -1,6 +1,6 @@ module(name = "ladybird", version = "0.0.0") -bazel_dep(name = "rules_cc", version = "0.2.17") +bazel_dep(name = "rules_cc", version = "0.2.19") bazel_dep(name = "platforms", version = "1.0.0") # Bazel 9 removed the native sh_binary; //Meta:vcpkg_build is one. @@ -30,6 +30,26 @@ qt.active_sdk(name = "qt", repo = "qt6_local") use_repo(qt, "qt") register_toolchains("@qt//:qt_linux_x86_64_toolchain") +# Qt's RUNTIME half. rules_qt wires up linking; nothing wired up the plugins Qt +# dlopens at QApplication construction, so the binary linked @qt's libQt6Core and +# then loaded the HOST's platform plugin into it -- two Qt builds in one process. +# On a box where SDK and distro versions differ that is a SIGSEGV in +# QXcbConnection::initializeScreens (reported on Ubuntu 24.04); where they agree it +# works by accident. @qt_plugins takes the plugins from the same SDK @qt names, and +# //:qt_conf points the binary at them. See qt_runtime.bzl for the whole story. +qt_runtime = use_extension("//:qt_runtime.bzl", "qt_runtime") +use_repo(qt_runtime, "qt_plugins") + +# --------------------------------------------------------------------------- +# Chromium's HSTS preload table: the one input CMake downloads UNPINNED (from +# `main`, at configure time) and the one thing that made a Bazel-only clone +# non-hermetic. Pinned downstream to a commit + sha256 in hsts_preload.bzl -- +# which also records why the commit is that one and not a release tag -- and +# regenerated by Meta/pin_hsts_preload.py. Consumed by :gen_HSTSPreloadData in +# codegen_root.bzl as @hsts_preload_json//file. +hsts = use_extension("//:hsts_preload.bzl", "hsts_preload") +use_repo(hsts, "hsts_preload_json") + # --------------------------------------------------------------------------- # Ring 2: the vcpkg dependency tree. Bazel owns FETCHING (here); vcpkg is still # the recipe that builds them (//:vcpkg_installed, see vcpkg.bzl). @@ -95,7 +115,7 @@ use_repo( 'vcpkg_libpng_1_6_58_apng_patch_gz_95a6f5bb7148', 'vcpkg_libproxy_libproxy_0_4_18_tar_gz_1148d688a9f0', 'vcpkg_libpsl_public_suffix_list_0ed17e_dat_7969c40b0600', - 'vcpkg_libsdl_org_SDL_release_3_4_12_tar_gz_fc0a55ca01c3', + 'vcpkg_libsdl_org_SDL_release_3_2_28_tar_gz_9e188c992caa', 'vcpkg_libtiff_libtiff_v4_7_2_tar_gz_c4dcde3c79e5', 'vcpkg_libtom_libtommath_v1_3_0_tar_gz_3dbd7053a670', 'vcpkg_libunistring_1_2_tar_xz_5fbb5a0a864d', @@ -108,6 +128,14 @@ use_repo( 'vcpkg_ngtcp2_nghttp3_v1_17_0_tar_gz_23d85a2abfa8', 'vcpkg_ngtcp2_ngtcp2_v1_24_0_tar_gz_04a5762d6eac', 'vcpkg_ngtcp2_sfparse_f2046eaa1acba7c5467399b1e1e1f354d22d1f48_tar_gz_b3cbcce6d96d', + # A host tool vcpkg fetches for ITSELF, pinned from Meta/vcpkg_tool_assets.tsv + # (derived from vcpkg scripts/vcpkg-tools.json) rather than from the download + # capture: vcpkg does not download a tool the machine already has, so a + # capturing machine carrying /usr/bin/ninja at exactly the required 1.13.2 kept + # ninja out of the pin, and a machine without it hit + # `distfile MISSING FROM INDEX` (finding 38). cmake is in this list already, + # but only by luck -- the capturing host had the wrong version. + 'vcpkg_ninja_linux_1_13_2_zip_714b900cf10b', 'vcpkg_openssl_openssl_openssl_3_6_3_tar_gz_a89c08101fa1', 'vcpkg_patchelf_0_19_0_x86_64_tar_gz_2a65c9cbdddc', 'vcpkg_pdfjs_5_6_205_dist_zip_66fecdb8a80d', @@ -125,6 +153,14 @@ use_repo( 'vcpkg_xiph_opus_v1_5_2_tar_gz_4ffefd9c0356', 'vcpkg_xiph_theora_v1_2_0_tar_gz_b2aac15528f0', 'vcpkg_xiph_vorbis_v1_3_7_tar_gz_bfb6f5dbfd49', + # NOT from the capture: the wheel for a Python package a portfile + # pip-installs. pip does not go through the vcpkg asset cache, so + # `--x-asset-sources=x-script+x-block-origin` never sees it -- which is how + # `pip install ply` in the angle port went on reaching PyPI through an + # inherited HTTP_PROXY (finding 36). Hand-maintained in + # vcpkg_python_packages.bzl for exactly that reason: no instrument can + # produce it, so nothing would regenerate it. + 'vcpkg_pywheel_ply', ) # --------------------------------------------------------------------------- @@ -195,6 +231,7 @@ use_repo( 'crate_fastrand_2_4_0', 'crate_flatbuffers_25_12_19', 'crate_foldhash_0_1_5', + 'crate_foldhash_0_2_0', 'crate_form_urlencoded_1_2_2', 'crate_getrandom_0_4_2', 'crate_gimli_0_31_1', diff --git a/examples/ladybird/workspace/Meta/BUILD.bazel b/examples/ladybird/workspace/Meta/BUILD.bazel index 49675f6..7052042 100644 --- a/examples/ladybird/workspace/Meta/BUILD.bazel +++ b/examples/ladybird/workspace/Meta/BUILD.bazel @@ -21,6 +21,11 @@ sh_binary( srcs = ["vcpkg_build.sh"], ) +# The host-prerequisite list, read by vcpkg_build.sh at action time and named as +# an input by vcpkg.bzl (NOT as data on the sh_binary above: runfiles are not +# beside the wrapper Bazel execs, so the script cannot find it by dirname $0). +exports_files(["vcpkg_host_tools.tsv"]) + # Ring 2 part 3: the driver for the cargo build actions (see //:cargo.bzl). It # assembles the vendor directory from crates Bazel fetched, points CARGO_HOME at # action-local scratch, and runs `cargo rustc --offline --locked` -- so the build diff --git a/examples/ladybird/workspace/Meta/bazel_parity_harness.py b/examples/ladybird/workspace/Meta/bazel_parity_harness.py index 828df1e..b92927a 100644 --- a/examples/ladybird/workspace/Meta/bazel_parity_harness.py +++ b/examples/ladybird/workspace/Meta/bazel_parity_harness.py @@ -32,7 +32,13 @@ import os, re, subprocess, sys, shutil, filecmp ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -FULL = os.path.join(ROOT, "Build/full") +# The reference CMake build tree the emitter reads. An env var, not a +# hardcoded "Build/full": a repin needs a SECOND reference build side by side +# with the old one (you cannot delete the tree you are still diffing against), +# and CMake bakes the build dir's absolute path into build.ninja -- so +# "just rename the directory afterwards" corrupts it. Found by doing exactly +# that during the 71fb301a repin. +FULL = os.environ.get("LADYBIRD_BUILD_DIR") or (ROOT + "/Build/full") SCRATCH = os.environ.get("PARITY_SCRATCH", os.path.join(ROOT, "Build/parity_out")) # --------------------------------------------------------------------------- @@ -56,6 +62,20 @@ (r'(?:^|&& )\S*bin/generate_interpreter_layout\b', 'generate_interpreter_layout (self-built)'), (r'(?:^|&& )\S*bin/flapc\b', 'flapc (self-built)'), + # generate-libjs-bytecode: upstream a32d9c9f replaced the Python bytecode + # generator with a Rust BINARY (a second bin of the flapc crate) that emits + # Bytecode/Op.h + OpCodes.h from interpreter.flap. + # + # It has to be listed HERE, above EXCLUDED, and that is the whole lesson: its + # ninja command is ` --input ... && cmake -E copy_if_different ...`, so + # the `copy_if_different` exclusion matched it first and the harness bucketed a + # GENERATOR as "resource staging" -- reporting 0 UNHANDLED while two generated + # headers had no owner at all. A first-match-wins classifier makes every + # exclusion pattern a potential silent capture of a command it was never + # written for, and the count that is supposed to catch that cannot, because the + # command was counted. + (r'(?:^|&& )\S*bin/generate-libjs-bytecode\b', + 'generate-libjs-bytecode (self-built)'), ] # --------------------------------------------------------------------------- diff --git a/examples/ladybird/workspace/Meta/cargo_binary_build.sh b/examples/ladybird/workspace/Meta/cargo_binary_build.sh old mode 100644 new mode 100755 index 807db7e..7c710d6 --- a/examples/ladybird/workspace/Meta/cargo_binary_build.sh +++ b/examples/ladybird/workspace/Meta/cargo_binary_build.sh @@ -1,15 +1,25 @@ #!/bin/bash -# Build a cargo BINARY crate offline: flapc, the Flap-DSL -> interpreter-assembly -# compiler. Driver for the cargo_binary build action (see cargo.bzl). +# Build a cargo BINARY crate offline. Driver for the cargo_binary build action +# (see cargo.bzl). There are two, and they are two different SHAPES of problem: # -# flapc was the last artifact this migration still took from the reference CMake -# build. Bazel already RAN it as a declared genrule tool, so the interpreter -# assembly was Bazel's own output -- but the compiler itself came out of -# Build/full/bin/, because Libraries/LibJS/Flap is a Rust crate. It gets exactly -# the same treatment as the staticlib crates, and needs no extra machinery: its -# workspace is `exclude`d from the root one and has its own lock with 3 packages, -# whose single registry crate is the same smallvec 1.15.1 (same checksum, checked -# by the emitter) that the big workspace already pins. +# * **flapc**, the Flap-DSL -> interpreter-assembly compiler: a pure build +# tool. It was the last artifact this migration still took from the reference +# CMake build -- Bazel already RAN it as a declared genrule tool, so the +# interpreter assembly was Bazel's own output, but the compiler itself came +# out of Build/full/bin/ because Libraries/LibJS/Flap is a Rust crate. Its +# workspace is `exclude`d from the root one and has its own lock with 3 +# packages, whose single registry crate is the same smallvec 1.15.1 (same +# checksum, checked by the emitter) the big workspace already pins. +# * **cranelift-compiler**, LibWasm's AOT WebAssembly compiler: a RUNTIME +# tool, spawned by the browser (Core::Process::spawn in CraneliftBridge.cpp), +# and one that ALSO emits an FFI header its C++ caller includes. It was +# absent from the Bazel graph entirely -- CMake declares it with +# build_rust_binary(), which the emitter did not parse, so every browser +# binary failed on `CraneliftFFI.h: No such file or directory`. +# +# The second one is why this driver takes FFI arguments at all: a binary crate's +# build script runs cbindgen exactly as a staticlib crate's does, and the header +# is a declared output for the same reason (Bazel deletes what nothing declares). # # Contract: # $1 crate name (the cargo package) @@ -19,6 +29,8 @@ # $5 the crate index: " " per line # $6 output binary path # $7 the --bin name +# $8 output dir for the FFI headers, or "" if the crate emits none +# $9+ the FFI header paths to check for, relative to $8 # # See cargo_vendor.sh for why the staging looks the way it does. set -euo pipefail @@ -30,6 +42,9 @@ SYSROOT="${4:?rust sysroot}" INDEX="${5:?crate index}" OUT_BIN="${6:?output binary}" BIN="${7:?bin name}" +FFI_OUT="${8-}" +shift $(( $# < 8 ? $# : 8 )) +FFI_HEADERS=("$@") # A binary output is a single file with no triple in its path (unlike the # staticlib, whose declared output mirrors cargo's own layout), so the triple @@ -58,3 +73,9 @@ cd "$SRC" cp "$TARGET_DIR/$TRIPLE/release/$BIN" "$OUT_BIN" chmod +x "$OUT_BIN" + +# The declared headers, resolved out of the OWNING crate's OUT_DIR -- the same +# collision-safe copy the staticlib driver does, shared in cargo_vendor.sh. +if [ -n "$FFI_OUT" ]; then + sync_ffi_headers "$FFI_OUT" "${FFI_HEADERS[@]+"${FFI_HEADERS[@]}"}" +fi diff --git a/examples/ladybird/workspace/Meta/cargo_build.sh b/examples/ladybird/workspace/Meta/cargo_build.sh old mode 100644 new mode 100755 index 5382d24..fc68fca --- a/examples/ladybird/workspace/Meta/cargo_build.sh +++ b/examples/ladybird/workspace/Meta/cargo_build.sh @@ -32,7 +32,8 @@ # and fails an action that does not write a declared one. (That check is how # HTMLTokenizerRustFFI.h -- written by a dependency's build script, declared # by nobody -- turned up.) Which crate's copy of a colliding header name -# wins is note 4, and it is the subtlest thing in this file. +# wins is sync_ffi_headers in cargo_vendor.sh, and it is the subtlest thing +# in this ring. # 3. **The cargo invocation mirrors CMake's exactly**: same subcommand # (`cargo rustc --lib`), same --target/--release, same trailing rustc flags # (-Cdefault-linker-libraries=yes -D warnings). Not for tidiness -- those @@ -60,11 +61,7 @@ source "${CARGO_VENDOR_LIB:?path to cargo_vendor.sh}" FEATURE_FLAGS=() [ -n "$FEATURES" ] && FEATURE_FLAGS=("--features=$FEATURES") -# FFI_OUTPUT_DIR is a SCRATCH dir, not the declared output dir -- see note 4 -# below for why that distinction is load-bearing. -FFI_SCRATCH="$WORK/ffi" -mkdir -p "$FFI_SCRATCH" "$FFI_OUT" -export FFI_OUTPUT_DIR="$FFI_SCRATCH" +mkdir -p "$FFI_OUT" cd "$SRC" "$SYSROOT/bin/cargo" rustc \ @@ -82,49 +79,7 @@ cd "$SRC" cp "$TARGET_DIR/$TRIPLE/release/lib$CRATE.a" "$OUT_LIB" -# --- note 4: resolve each header from the OWNING crate's OUT_DIR ------------- -# -# $FFI_OUTPUT_DIR is shared by every build script in the crate's dependency -# graph, and the header names COLLIDE: six crates each emit a file called -# `RustFFI.h`. liburl_rust path-depends on libregex_rust, so building liburl_rust -# runs BOTH build scripts against the same FFI_OUTPUT_DIR and the surviving -# RustFFI.h is whichever ran last -- we shipped libregex_rust's URL header for -# one build and it was only caught by byte-comparing against CMake's tree. -# -# CMake hits this too, and has a whole script for it -# (Meta/CMake/sync_rust_ffi_header.cmake): after cargo runs it copies the header -# out of the OWNING crate's own OUT_DIR, found via -# `build/-*/root-output`, and that copy is what the compiler sees. Mirror -# that: prefer the crate's own OUT_DIR, and fall back to the shared scratch dir -# for a header written by a DEPENDENCY's build script (libweb_rust's -# HTMLTokenizerRustFFI.h comes from libweb_html_tokenizer and collides with -# nothing, which is exactly why CMake never noticed it was undeclared). -BUILD_DIR="$TARGET_DIR/$TRIPLE/release/build" -missing=() -for h in "${FFI_HEADERS[@]+"${FFI_HEADERS[@]}"}"; do - mkdir -p "$FFI_OUT/$(dirname "$h")" - src="" - # The crate's own OUT_DIR, newest first: a rebuild can leave several. - for ro in $(ls -t "$BUILD_DIR/$CRATE"-*/root-output 2>/dev/null); do - cand="$(cat "$ro")/$h" - [ -f "$cand" ] && { src="$cand"; break; } - done - # Else a dependency's build script wrote it to the shared dir. - [ -z "$src" ] && [ -f "$FFI_SCRATCH/$h" ] && src="$FFI_SCRATCH/$h" - if [ -z "$src" ]; then - missing+=("$h") - continue - fi - cp "$src" "$FFI_OUT/$h" -done -if [ ${#missing[@]} -gt 0 ]; then - echo "cargo_build: $CRATE declared FFI headers cargo never wrote: ${missing[*]}" >&2 - echo "cargo_build: what it DID write under $FFI_SCRATCH:" >&2 - (cd "$FFI_SCRATCH" && find . -type f | sed 's|^\./| |') >&2 - echo "cargo_build: and in its own OUT_DIRs:" >&2 - for ro in "$BUILD_DIR/$CRATE"-*/root-output; do - [ -f "$ro" ] || continue - (cd "$(cat "$ro")" && find . -type f -name "*.h" | sed 's|^\./| |') >&2 - done - exit 1 -fi +# The declared headers, resolved out of the OWNING crate's OUT_DIR -- see +# sync_ffi_headers in cargo_vendor.sh for why $FFI_OUTPUT_DIR alone is not enough +# (eight crates emit a file called RustFFI.h into one shared directory). +sync_ffi_headers "$FFI_OUT" "${FFI_HEADERS[@]+"${FFI_HEADERS[@]}"}" diff --git a/examples/ladybird/workspace/Meta/cargo_vendor.sh b/examples/ladybird/workspace/Meta/cargo_vendor.sh old mode 100644 new mode 100755 index c27b4a2..1a686c5 --- a/examples/ladybird/workspace/Meta/cargo_vendor.sh +++ b/examples/ladybird/workspace/Meta/cargo_vendor.sh @@ -87,3 +87,67 @@ export AR_${TRIPLE_US}="${AR:-ar}" export CARGO_TARGET_${TRIPLE_UP}_LINKER="${CC:-cc}" TARGET_DIR="$WORK/target" + +# --- FFI_OUTPUT_DIR is a SCRATCH dir, not the declared output dir ------------ +# Shared by both drivers because the reason is the same for both: cbindgen runs +# from a build script, every build script in the crate's dependency graph writes +# into the SAME $FFI_OUTPUT_DIR, and the header names COLLIDE (eight crates emit +# a file called `RustFFI.h`). So the declared outputs are resolved out of the +# OWNING crate's own OUT_DIR afterwards -- see sync_ffi_headers. +FFI_SCRATCH="$WORK/ffi" +mkdir -p "$FFI_SCRATCH" +export FFI_OUTPUT_DIR="$FFI_SCRATCH" + +# sync_ffi_headers [header...] +# +# Copy each declared FFI header into the declared output dir, resolved from the +# crate that OWNS it. Mirrors Meta/CMake/sync_rust_ffi_header.cmake, which exists +# for exactly this reason: liburl_rust path-depends on libregex_rust, so building +# liburl_rust runs BOTH build scripts against one FFI_OUTPUT_DIR and the +# surviving RustFFI.h is whichever ran last -- we shipped libregex_rust's URL +# header for one build and only caught it by byte-comparing against CMake's tree. +# +# CMake finds the owning crate's OUT_DIR via `build/-*/root-output`; so +# does this, preferring it and falling back to the shared scratch dir for a +# header written by a DEPENDENCY's build script (libweb_rust's +# HTMLTokenizerRustFFI.h comes from libweb_html_tokenizer and collides with +# nothing, which is exactly why CMake never noticed it was undeclared). +# +# Failing loudly is the point: Bazel deletes an undeclared output and fails an +# action that does not write a declared one, so a missing header is reported here +# with what cargo DID write rather than surfacing as a file-not-found in a C++ +# compile a thousand actions later. +sync_ffi_headers() { + local ffi_out="$1"; shift + [ $# -gt 0 ] || return 0 + local build_dir="$TARGET_DIR/$TRIPLE/release/build" + local missing=() h ro cand src + mkdir -p "$ffi_out" + for h in "$@"; do + mkdir -p "$ffi_out/$(dirname "$h")" + src="" + # The crate's own OUT_DIR, newest first: a rebuild can leave several. + for ro in $(ls -t "$build_dir/$CRATE"-*/root-output 2>/dev/null); do + cand="$(cat "$ro")/$h" + [ -f "$cand" ] && { src="$cand"; break; } + done + # Else a dependency's build script wrote it to the shared dir. + [ -z "$src" ] && [ -f "$FFI_SCRATCH/$h" ] && src="$FFI_SCRATCH/$h" + if [ -z "$src" ]; then + missing+=("$h") + continue + fi + cp "$src" "$ffi_out/$h" + done + if [ ${#missing[@]} -gt 0 ]; then + echo "cargo: $CRATE declared FFI headers cargo never wrote: ${missing[*]}" >&2 + echo "cargo: what it DID write under $FFI_SCRATCH:" >&2 + (cd "$FFI_SCRATCH" && find . -type f | sed 's|^\./| |') >&2 + echo "cargo: and in its own OUT_DIRs:" >&2 + for ro in "$build_dir/$CRATE"-*/root-output; do + [ -f "$ro" ] || continue + (cd "$(cat "$ro")" && find . -type f -name "*.h" | sed 's|^\./| |') >&2 + done + return 1 + fi +} diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index 5de48a1..9e1ae75 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -14,7 +14,14 @@ from collections import defaultdict ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -MODEL = os.path.join(ROOT, "model.cmake.full.json") +MODEL = os.environ.get("LADYBIRD_MODEL") or os.path.join(ROOT, "model.cmake.full.json") +# The reference build tree, RELATIVE to the checkout: the model records source +# and include paths relative to the repo root, so every "is this a generated +# file?" test is a string prefix on this. An env var rather than a literal +# "Build/full" because a repin needs the new reference build to coexist with the +# old one, and CMake bakes the build dir into build.ninja -- so renaming the +# directory after the fact is not an option (learned the hard way at 71fb301a). +BUILD_REL = (os.environ.get("LADYBIRD_BUILD_REL") or "Build/full").strip("/") + "/" VCPKG = "//Meta/vcpkg" # Ring 2 part 3: the Rust crates are BUILT BY BAZEL now (cargo_ring.bzl in this # same package), so the labels are local -- nothing points into Build/full/cargo. @@ -25,6 +32,49 @@ # all, and grouping them makes the linker pull one crate's objects into a target # that never asked for them. RUST_LIB_FMT = "//:%s_lib" +# Crates whose FFI header the OWNING library includes with no directory prefix +# (`#include `). Those libraries additionally depend on +# //:_bare_include, which carries ONLY that crate's ffi/ dir. +# It is a separate label from //:_lib on purpose: CcInfo include dirs +# propagate transitively, and 8 of the 10 crates emit a header named RustFFI.h, +# so a shared unprefixed dir let one crate's header satisfy another library's +# bare include (LibGfx compiled against LibRegex's). +RUST_BARE_INCLUDE_FMT = "//:%s_bare_include" + + +def _cargo(): + sys.path.insert(0, os.path.join(ROOT, "Meta")) + import emit_cargo_bazel + return emit_cargo_bazel + + +def rust_bare_include_crates(): + """The crates whose header is included bare -- IMPORTED, not mirrored. + + This used to be a hand-kept tuple beside the cargo emitter's derived set, + with a comment saying the two were "kept in sync". They were not: the cargo + emitter DERIVES the set by scanning the tree for a directory-less + `#include `, and a hand-kept copy of a derived set is the finding-23 bug in + miniature. So it is read from the one place that computes it, and adding a + bare include anywhere in Ladybird needs no edit in either file. + """ + cargo = _cargo() + return {c for c, on in + cargo.crates_included_bare(cargo._bare_scan_specs()).items() if on} + + +def rust_binary_crates(): + """{CMake target name -> crate} for the build_rust_binary() crates. + + CMake's dependency edge on a binary crate is a custom target named + `-build` (rust_crate.cmake), and that name is what turns up in + the model as a dep -- `cranelift-compiler-build` on LibWasm. Mapping it back + to the crate is what lets the emitter translate that edge instead of dropping + it, which is precisely what it was doing: LibWasm's dep on the Cranelift + crate was silently discarded, and the build failed 1,600 actions later on + `CraneliftFFI.h: No such file or directory`. + """ + return {b["bin"] + "-build": b for b in _cargo().binary_specs()} # Global defines already set in .bazelrc; do not re-emit per target. GLOBAL_DEFINES = { @@ -33,13 +83,38 @@ "NDEBUG", } # System libs with no vcpkg .so (linked via linkopts on the final binary). -SYSTEM_LIBS = {"dl", "m", "pthread", "vulkan", "pulse"} -# Qt6 CMake target -> rules_qt label. -QT_MAP = { - "Qt6Core": "@qt//:QtCore", - "Qt6Gui": "@qt//:QtGui", - "Qt6Widgets": "@qt//:QtWidgets", -} +# glib/gio/gobject and xkbcommon joined at 71fb301a: upstream added a +# pkg_check_modules(GIO) for UI/Qt/ExternalURLActivationToken + ExternalURLHandler, +# and xkbcommon arrives transitively with the now-required Qt6::GuiPrivate. Their +# include roots (/usr/include/glib-2.0, /usr/lib/*/glib-2.0/include) are absolute, +# so like libdrm's they cannot be per-target copts and live in .bazelrc's +# CPLUS_INCLUDE_PATH -- see the host-tool preflight todo: this whole set is a +# configure-time host probe that neither build system derives on our side yet. +SYSTEM_LIBS = {"dl", "m", "pthread", "vulkan", "pulse", + "gio-2.0", "gobject-2.0", "glib-2.0", "xkbcommon"} + + +def qt_label(nm): + """CMake's Qt6 -> rules_qt's @qt//:Qt, by RULE not by table. + + This was a three-entry dict {Qt6Core, Qt6Gui, Qt6Widgets}, which is the same + shape as every other capture in this tree: correct for the pin that was + measured, silent about the next one. At 71fb301a upstream made + Qt6::Positioning REQUIRED (UI/Qt/GeolocationProviderQt.cpp), the dict had no + Qt6Positioning key, so the dep fell through to UNKNOWN and //:ladybird failed + to compile with + + UI/Qt/GeolocationProviderQt.h:11:10: fatal error: QGeoPositionInfo: + No such file or directory + + rules_qt names one cc_library per module in the discovered SDK, so the + mapping is a rename, and every module the SDK has is already a label. A + module the SDK does NOT have must still fail -- as an UNKNOWN dep naming it, + which is why this returns None rather than a label it has not checked. + """ + if not nm.startswith("Qt6") or len(nm) <= 3: + return None + return "@qt//:Qt" + nm[len("Qt6"):] # CMake's AUTOMOC/AUTORCC output, prebuilt under Build/full/_autogen. # Bazel runs moc/rcc itself (qt_cc_moc / qt_cc_rcc), so these are dropped from @@ -60,12 +135,104 @@ def global_flags(): GLOBAL_FLAGS = global_flags() + +def bazelrc_host_includes(): + """The include roots .bazelrc hands every action via CPLUS_INCLUDE_PATH. + + These are the host escapes README gap 3 inventories. Read here so the + emitter can CHECK the ones it drops against them rather than dropping them + silently. + """ + rc = open(os.path.join(ROOT, ".bazelrc")).read() + out = set() + for m in re.finditer(r"--(?:host_)?action_env=CPLUS_INCLUDE_PATH=(\S+)", rc): + out |= {p for p in m.group(1).split(":") if p} + return out + + +BAZELRC_HOST_INCLUDES = bazelrc_host_includes() +# Absolute include roots the model uses that .bazelrc does NOT carry. Collected +# during emission and reported at the end: a per-target message would repeat +# once per TU, and the useful unit is "which roots does this configuration need +# that the build does not provide". +MISSING_HOST_INCLUDES = set() +# Roots that arrive with a dep edge instead of an env var, so their absence from +# CPLUS_INCLUDE_PATH is correct rather than missing. Qt comes from rules_qt as a +# real Bazel dep (@qt//:Qt carries its own include dirs) and the vcpkg +# tree rides on //Meta/vcpkg: as system_includes -- both deliberately NOT +# global -isystem any more, which is what finding 33 bought. +HOST_INCLUDE_EXEMPT = ("/usr/include/x86_64-linux-gnu/qt6", + "/usr/lib/x86_64-linux-gnu/qt6", + "vcpkg_installed") + + +def record_host_include(path): + if any(x in path for x in HOST_INCLUDE_EXEMPT): + return + if path not in BAZELRC_HOST_INCLUDES: + MISSING_HOST_INCLUDES.add(path) + + +def report_host_includes(): + """Print the shortfall to stderr. Not fatal: see record_host_include.""" + if not MISSING_HOST_INCLUDES: + return + sys.stderr.write( + "WARNING: %d absolute include root(s) CMake compiles with are absent " + "from\n .bazelrc's CPLUS_INCLUDE_PATH, so a TU that needs one " + "will fail with\n 'No such file or directory' far from here:\n" + % len(MISSING_HOST_INCLUDES)) + for p in sorted(MISSING_HOST_INCLUDES): + sys.stderr.write(" %s\n" % p) + sys.stderr.write( + " Add them there (they cannot be per-target copts: Bazel " + "rejects a path\n outside the execution root), or give the dep " + "a Bazel label that carries them.\n") + +# Flags CMake puts on a target that must NOT be copied into a Bazel copt. +# +# -fPIE is the one that matters, and it is a crash, not a nit. CMake adds it to +# every executable target (CMAKE_POSITION_INDEPENDENT_CODE + an exe => -fPIE), +# and the capture faithfully recorded it -- so the generated cc_binary carried +# `copts = ['-fPIE']`. Bazel appends per-target copts AFTER the .bazelrc's +# --copt=-fPIC, and for GCC the LAST of -fPIC/-fPIE wins. So the UI/Qt objects +# were compiled -fPIE while every library around them was -fPIC. +# +# Under -fPIE, GCC may reference extern data DIRECTLY (PC-relative) instead of +# through the GOT, and the linker then materialises the definition in the +# executable with an R_X86_64_COPY relocation. For libraries built with Qt's +# `reduce_relocations` (every official/aqt Qt SDK; Debian's is built without it) +# that is fatal: QtCore accesses its own `QCoreApplication::self` PC-relative -- +# its own BSS copy -- while QtGui reads the same symbol through the GOT, which +# the copy relocation has repointed at the EXECUTABLE's BSS. QApplication's +# constructor then sets qApp in one place and QtGui reads the other, still null. +# The first emit through a null sender segfaults: QGuiApplication::screenAdded +# from QWindowSystemInterface::handleScreenAdded, inside doActivate, on +# `mov 0x8(%rdi),%rbx` with rdi = 0. +# +# Qt's own headers say so and are right: qcompilerdetection.h #errors with +# "-fPIE is not sufficient ... Compile your code with -fPIC and without -fPIE" +# -- but only when __PIC__ is unset, and Bazel passes BOTH flags, so __PIC__ is +# defined at preprocess time and the guard never fires. The build was clean and +# the binary was broken. +# +# Dropping -fPIE is not a divergence from CMake's semantics: Bazel's own +# toolchain compiles the objects of a cc_binary PIC and links -pie, which is +# what CMake was asking for. Verified by removal: with -fPIE gone the binary has +# ZERO R_X86_64_COPY relocations (39 before, incl. qApp) and the GUI starts +# against an aqt 6.9.2 SDK, where before it segfaulted for both the xcb and the +# offscreen QPA plugin. -Wl,-z,nocopyreloc is NOT an alternative: it turns the +# same defect into a link error ("causes overflow in R_X86_64_PC32"). +DROPPED_TARGET_FLAGS = {"-fPIE"} + def target_flags(t): """Per-target compile flags (feature/warning) not covered by .bazelrc.""" flags = set() for a in t["actions"]: if a["mnemonic"] != "CppCompile": continue for x in a["arguments"]: + if x in DROPPED_TARGET_FLAGS: + continue if x.startswith(("-f", "-m", "-p")) and x not in GLOBAL_FLAGS: flags.add(x) return sorted(flags) @@ -104,13 +271,48 @@ def target_srcs(t): srcs += [i for i in a["inputs"] if i.endswith((".cpp", ".c", ".cc", ".S"))] return sorted(s for s in set(srcs) if AUTOGEN_MARKER not in s) +def rewrite_host_path_define(d): + """Rewrite a define whose VALUE is an absolute path to a Bazel-built binary. + + There is exactly one, and it was the last hardcoded + /home/ubuntu/... in the emitted build: + + -DWASM_CRANELIFT_COMPILER_PATH="/bin/cranelift-compiler" + + CMake bakes the reference build's absolute output path in, which is fine for + CMake (the build tree does not move) and fatal for a checkout on any other + machine -- the define alone made `git clone && bazel build` unusable even once + the crate itself built. + + The fix is not "point it at bazel-bin" (that is the same escape with a + different prefix). Ladybird already resolves the compiler through a lookup + CHAIN (resolve_cranelift_compiler_path in CraneliftBridge.cpp): + $LADYBIRD_CRANELIFT_COMPILER, then the compile-time path, then + SIBLING-OF-SELF. Bazel puts every root-package output in ONE bin directory, + so cranelift-compiler is already a sibling of WebContent, ladybird and the + rest -- the third link in Ladybird's own chain finds it with no path baked in + at all. So the define becomes the bare file name (link 2 is then a + cwd-relative probe that harmlessly misses), and the binary is attached as + `data` on LibWasm so it is really THERE, in the runfiles of everything that + links LibWasm. The dependency is declared; the path is not asserted. + """ + key, _, value = d.partition("=") + literal = value.strip('"') + if not literal.startswith("/"): + return d + for b in rust_binary_crates().values(): + if os.path.basename(literal) == b["output_name"]: + return '%s="%s"' % (key, b["output_name"]) + return d + + def target_defines(t, name): defs = set() for a in t["actions"]: if a["mnemonic"] != "CppCompile": continue for x in a["arguments"]: if x.startswith("-D"): - d = x[2:] + d = rewrite_host_path_define(x[2:]) if d not in GLOBAL_DEFINES: # Bazel needs embedded quotes escaped in (local_)defines. defs.add(d.replace('"', '\\"')) @@ -118,18 +320,28 @@ def target_defines(t, name): def target_private_includes(t): """Per-target -isystem/-I under Build/full (the target's own gendir) not in - the 6 global roots.""" + the 6 global roots. + + Paths are NORMALIZED before the comparison, which is not cosmetic. CMake emits + a target's own binary dir relative to itself, so the five service targets get + `-IBuild/full/Services/WebContent/../..` -- the same directory as the global + `-IBuild/full`, spelled differently. Compared as strings it is not in + globalroots, so it came out as a per-target `-IBuild/full` copt on five + targets plus WebContent: six copts pointing into CMake's build tree that + supplied nothing any global root did not already supply, and which made the + emitted build look like it needed Build/full when it did not. + """ globalroots = {ROOT, ROOT+"/Libraries", ROOT+"/Services", - ROOT+"/Build/full", ROOT+"/Build/full/Libraries", - ROOT+"/Build/full/Services", - ROOT+"/Build/full/vcpkg_installed/x64-linux-dynamic/include"} + ROOT+"/"+BUILD_REL.rstrip("/"), ROOT+"/"+BUILD_REL+"Libraries", + ROOT+"/"+BUILD_REL+"Services", + ROOT+"/"+BUILD_REL+"vcpkg_installed/x64-linux-dynamic/include"} incs = [] for a in t["actions"]: if a["mnemonic"] != "CppCompile": continue args = a["arguments"]; i = 0 while i < len(args): if args[i] in ("-I", "-isystem"): - p = args[i+1] + p = os.path.normpath(args[i+1]) if p in globalroots: pass elif p.startswith(ROOT): @@ -160,29 +372,101 @@ def target_embed_inputs(t): embeds.add(rel) return sorted(embeds) +def rust_dep_labels(crate): + """(deps, implementation_deps) for one Rust crate dep. + + `deps` is always //:_lib -- the archive plus the crate's PREFIXED + headers, exactly CMake's target_link_libraries edge, and safe to propagate + because the spelling is unique per crate. + + `implementation_deps` is //:_bare_include when the owning library + spells the header with NO directory, and the distinction is the whole point. + Eight of the ten crates emit a header literally named `RustFFI.h`, so an + unprefixed include dir that reaches a second library makes a bare + `#include ` bind to whichever dir sorted first on the command line + -- silently, since both files exist. + + Splitting it into its own TARGET (cargo_bare_include) was necessary but NOT + sufficient, and this is the second half of that lesson. Include dirs + propagate along the C++ dep graph, not just out of one rule: LibGfx depends on + LibTextCodec, which owns libtextcodec_rust's bare dir, so LibGfx's compiles + received /libtextcodec_rust/ffi/LibTextCodec BEFORE its own + /libgfx_rust/ffi/LibGfx and YUVData.cpp compiled against LibTextCodec's + header ("'FFI' does not name a type"). A separate target only stops the dir + leaking out of cargo_lib; it does not stop it leaking out of LibTextCodec. + + `implementation_deps` is Bazel's name for exactly the scope CMake's + `target_include_directories(... PRIVATE)` has: the dep is used to COMPILE this + library and is not part of its interface, so its include dirs stop here. That + is a one-for-one translation of the CMake this build is a port of -- CMake's + FFI_OUTPUT_DIR is added PRIVATE, to the owning library's own binary dir -- and + it is the reason a bare include is unambiguous there and had to be made + unambiguous here. + """ + impl = [] + if crate in rust_bare_include_crates(): + impl.append(RUST_BARE_INCLUDE_FMT % crate) + return [RUST_LIB_FMT % crate], impl + + +# How one CMake dep is translated. Four kinds, kept as named fields rather than +# as "a string, or a list, or a 2-tuple whose first element is a magic word" -- +# which is what this was, and it stopped being safe the moment a translation +# needed to produce BOTH deps and implementation_deps (a 2-tuple, i.e. exactly +# the shape the ("SYS", name) sentinel already used). +class Dep: + def __init__(self, deps=(), impl=(), sys=(), unknown=()): + self.deps, self.impl = list(deps), list(impl) + self.sys, self.unknown = list(sys), list(unknown) + + def dep_label(d, targets, so, ar): nm = d["name"] if nm.startswith("lagom-"): tgt = lagom_to_target(nm, targets) - if tgt == "LibWeb": return "//Libraries/LibWeb:LibWeb" - return "//:%s" % tgt if tgt else None + if tgt == "LibWeb": return Dep(deps=["//Libraries/LibWeb:LibWeb"]) + return Dep(deps=["//:%s" % tgt]) if tgt else None if nm.endswith("_rust"): - return RUST_LIB_FMT % nm + deps, impl = rust_dep_labels(nm) + return Dep(deps=deps, impl=impl) + # A build_rust_binary() crate. CMake's edge is on the custom target + # `-build`, and it means two separate things that Bazel splits: + # the generated FFI HEADER (a dep, via //:_lib) and the BINARY itself, + # which is not linked at all -- LibWasm spawns it -- so it becomes runfiles + # (see the `data` handling in emit_target). + binaries = rust_binary_crates() + if nm in binaries: + b = binaries[nm] + if not b["ffi_headers"]: + return None + deps, impl = rust_dep_labels(b["crate"]) + return Dep(deps=deps, impl=impl) + # A staticlib crate's redundant `-build` custom target (CMake emits + # both it and the imported-library dep); the library dep carries everything. + if nm.endswith("-build"): + return None if nm in so or nm in ar: - return VCPKG + ":" + nm + return Dep(deps=[VCPKG + ":" + nm]) if nm in SYSTEM_LIBS: - return ("SYS", nm) # linkopt on final binary + return Dep(sys=[nm]) # linkopt on final binary # Qt6 + GL: system .so under /usr/lib; CMake finds them via find_package. # Map the CMake target name to the -l library name; the /usr/lib search # path is a global -L in .bazelrc. # Qt6 is a real Bazel dep via rules_qt: its qt.local_repo discovers the host # SDK through qmake, so moc/rcc and the Qt cc_librarys all come from one SDK. - if nm in QT_MAP: - return QT_MAP[nm] + qt = qt_label(nm) + if qt: + # Plus @qt_plugins//:runtime_libs -- the PRIVATE libraries an SDK ships + # beside Qt (aqt bundles ICU 73) which rules_qt does not stage, and which + # libQt6Core cannot find on its own because its RUNPATH $ORIGIN expands to + # Bazel's solib dir. Making them link inputs is what removed the need for + # LD_LIBRARY_PATH=/lib. Empty for a distro Qt. See qt_runtime.bzl / + # finding 40. + return Dep(deps=[qt, "@qt_plugins//:runtime_libs"]) SYS_MAP = {"GLX": "GLX", "OpenGL": "OpenGL"} if nm in SYS_MAP: - return ("SYS", SYS_MAP[nm]) - return ("UNKNOWN", nm) + return Dep(sys=[SYS_MAP[nm]]) + return Dep(unknown=[nm]) # Generated sources that BAZEL now produces itself (a genrule output in the root # package, listed in codegen_root.bzl). A model src under Build/full naming one @@ -208,21 +492,21 @@ def bazel_generated_root_srcs(): def _emit_srcs(srcs): print(" srcs = [") for s in srcs: - rel = s[len("Build/full/"):] if s.startswith("Build/full/") else None + rel = s[len(BUILD_REL):] if s.startswith(BUILD_REL) else None if rel is not None and rel in GENERATED_BY_BAZEL: # Bazel generates this file; consume its genrule output. print(f" {':' + rel!r},") - elif s.startswith("Build/full/Libraries/LibWeb/"): - lab = "//Libraries/LibWeb:" + s[len("Build/full/Libraries/LibWeb/"):] + elif s.startswith(BUILD_REL + "Libraries/LibWeb/"): + lab = "//Libraries/LibWeb:" + s[len(BUILD_REL + "Libraries/LibWeb/"):] print(f" {lab!r},") - elif s.startswith("Build/full/Libraries/"): - lab = "//Build/full/Libraries:" + s[len("Build/full/Libraries/"):] + elif s.startswith(BUILD_REL + "Libraries/"): + lab = "//Build/full/Libraries:" + s[len(BUILD_REL + "Libraries/"):] print(f" {lab!r},") - elif s.startswith("Build/full/Services/"): - lab = "//Build/full/Services:" + s[len("Build/full/Services/"):] + elif s.startswith(BUILD_REL + "Services/"): + lab = "//Build/full/Services:" + s[len(BUILD_REL + "Services/"):] print(f" {lab!r},") - elif s.startswith("Build/full/UI/"): - lab = "//Build/full/UI:" + s[len("Build/full/UI/"):] + elif s.startswith(BUILD_REL + "UI/"): + lab = "//Build/full/UI:" + s[len(BUILD_REL + "UI/"):] print(f" {lab!r},") else: print(f" {s!r},") @@ -256,6 +540,55 @@ def _emit_srcs(srcs): # Emitted after the Qt autogen rules, since it consumes them. QT_TARGETS = ["ladybird"] + +def spawned_services(): + """The helper binaries the UI SPAWNS at runtime, read out of Ladybird's source. + + WHY THIS IS A DEPENDENCY AT ALL, and the bug it fixes. + `bazel build //:ladybird` built the browser and NOTHING ELSE, because nothing in + the graph said the browser needs its services -- they are found by PATH at + runtime (get_paths_for_helper_process), not linked. So the recipe told people to + name all six targets, and a build of just //:ladybird left whatever WebContent + happened to be in bazel-bin from a previous build. + + Ulf hit the consequence in its most confusing form: ladybird dated Aug 20 next + to a WebContent dated Aug 11, i.e. a browser from THIS pin talking to a service + from the PREVIOUS one. Upstream had inserted ~3 IPC messages between the pins, + shifting every id after them, so every message failed to decode: + + Failed to parse IPC message: + Local endpoint error: Can't read past the end of the stream memory + Peer endpoint error: Endpoint magic number mismatch, not my message! + + which reads like a codegen or ABI bug and is nothing of the kind. (The magic + number in those frames, 0xffa5367a, is AK::string_hash("WebContentServer") -- + the CORRECT endpoint. The ids are what disagreed: his WebContent's numbering + matched f9e34731 7/7 and 71fb301a 0/7.) His diagnosis: declare them, so they + rebuild with the thing that spawns them. Exactly right. + + `data`, not `deps`: they are separate processes, not link inputs -- the same + relationship LibWasm already has to cranelift-compiler, which this emitter + already gets right (see rewrite_binary_path_define). data also puts them in the + runfiles tree, which is what `bazel run` needs. + + DERIVED from HelperProcess.cpp, never hand-listed. The names are the string + literals passed to launch_server_process<>, i.e. the same source of truth the + runtime lookup uses; a hand-kept list here would be a sixth service away from + silently reintroducing the bug (and upstream adds services -- Compositor is new + since the previous pin). If the parse finds nothing, that is a hard failure + rather than a build that quietly omits them again. + """ + src = os.path.join(ROOT, "Libraries", "LibWebView", "HelperProcess.cpp") + with open(src) as f: + text = f.read() + names = sorted(set(re.findall(r'launch_server_process<[^>]*>\(\s*"(\w+)"sv', text))) + if not names: + sys.exit("emit_build_bazel: found no launch_server_process<> calls in %s -- " + "the spawned-service list is DERIVED from them, and an empty list " + "would silently rebuild //:ladybird without its services (the Aug 11 " + "WebContent bug). Has the launch helper been rewritten?" % src) + return names + # Ring 2's own targets: the vcpkg tree build action and its inputs. Emitted # rather than hand-appended, because a hand-appended tail is a file the emitter # would silently truncate on the next run -- exactly the drift this project keeps @@ -276,13 +609,24 @@ def _emit_srcs(srcs): vcpkg_tree( name = "vcpkg_installed", distfiles = VCPKG_DISTFILE_INDEX, + # The wheels for the Python packages a portfile pip-installs (angle asks for + # `ply`). Pinned separately from the 76 distfiles because pip bypasses vcpkg's + # asset cache entirely, so the capture cannot see them and x-block-origin + # cannot block them -- see vcpkg_python_packages.bzl and finding 36. + python_wheels = ['@vcpkg_pywheel_ply//file'], source_dir = ".", source_root = ":vcpkg_source_inputs", triplet = "x64-linux-dynamic", - # Resume cache: makes a killed 45-minute build cheap to restart. Absolute by - # necessity (the action's cwd is the execroot) and therefore a host escape -- - # it is a build-speed affordance, not part of the dependency graph. - cache_dir = "/home/ubuntu/.cache/vcpkg-bazel", + # Resume cache: OPT-IN, and empty here on purpose. Setting it makes a killed + # 45-minute vcpkg build cheap to restart, but the path must be absolute (the + # action's cwd is the execroot), so any value here is one developer's home + # directory in a file everyone checks out -- it was + # /home/ubuntu/.cache/vcpkg-bazel, the last absolute host path left in the + # emitted build. Empty is also the honest default: it is the genuine + # from-source build. Set this one attribute locally if you want resumability. + # It is a build-speed affordance and no part of the dependency graph, which is + # precisely why it must not be a checked-in constant. + cache_dir = "", vcpkg_root = "Build/vcpkg", vcpkg_tree = "//Build/vcpkg:tree", ) @@ -323,6 +667,8 @@ def _emit_srcs(srcs): load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library") load(":cargo_ring.bzl", "cargo_ring") load(":codegen_root.bzl", "root_codegen") +load(":export_headers.bzl", "export_headers") +load(":qt_runtime.bzl", "qt_conf", "qt_plugin_tree") load(":vcpkg.bzl", "vcpkg_tree", "vcpkg_tree_for_exec") load(":vcpkg_index.bzl", "VCPKG_DISTFILE_INDEX") @@ -335,6 +681,11 @@ def _emit_srcs(srcs): # Build/full. root_codegen() +# CMake's generate_export_header() output (15 Export.h) and the two AK +# configure_file headers, generated by Bazel instead of read out of Build/full -- +# generated by Meta/emit_export_headers_bazel.py, byte-verified with --check. +export_headers() + # The Rust ring: 10 cargo staticlib crates + flapc, built by Bazel from crates # Bazel fetched, offline. This is what retired the prebuilt 260 MB # librust_combined.a and its hand-run `ar -M` merge (README step 1b). It lives in @@ -359,15 +710,29 @@ def _emit_srcs(srcs): "UI/**/*.h", ], allow_empty = True), deps = [ - # Bazel-generated headers (the genrules above) come FIRST so they win - # over the CMake copies; the Build/full roots below now only supply what - # Bazel does not yet generate (Rust FFI headers, CMake's - # generate_export_header Export.h). + # EVERY generated header is Bazel's own now. The two + # //Build/full/{Libraries,Services} header roots that used to be listed + # here are gone, and with them the last thing the BUILD read out of + # CMake's build tree. + # + # They were not supplying anything by the end, and that is the part worth + # recording, because it is why they survived so long: a `glob(["**/*.h"])` + # over a foreign tree cannot fail. The 709 headers under those roots were + # 21 that Bazel generates and 688 LibWeb bindings headers that Bazel ALSO + # generates -- so the roots were SHADOWING Bazel's own outputs, silently + # winning or losing on include order, and the only reason a fresh clone + # did not fail here was that it got no error either: allow_empty=True + # turns "the tree is not there" into an empty glob and the build dies + # ~1,600 actions later on a missing Export.h. Verified by removal: + # Build/full/{Libraries,Services,UI} moved off the machine, all six + # binaries rebuilt from scratch, --headless=text and + # --headless=layout-tree byte-identical to the CMake reference. ":generated_libraries_headers", ":generated_services_headers", ":generated_shader_headers", - "//Build/full/Libraries:generated_lib_headers", - "//Build/full/Services:generated_service_headers", + ":generated_export_headers", + ":generated_ak_headers", + "//Libraries/LibWeb:generated_export_header", ], ) @@ -375,21 +740,18 @@ def _emit_srcs(srcs): ''' AK_BLOCK_HEAD = ''' -# Configure-generated headers live in Build/full/AK; expose as AK/*.h via a copy. -genrule( - name = "ak_gen_headers", - srcs = ["Build/full/AK/Debug.h", "Build/full/AK/Backtrace.h"], - outs = ["genroot/AK/Debug.h", "genroot/AK/Backtrace.h"], - cmd = "mkdir -p $(RULEDIR)/genroot/AK && " + - "cp $(location Build/full/AK/Debug.h) $(RULEDIR)/genroot/AK/Debug.h && " + - "cp $(location Build/full/AK/Backtrace.h) $(RULEDIR)/genroot/AK/Backtrace.h", -) +# NOTE: the ak_gen_headers genrule that COPIED AK/Debug.h and AK/Backtrace.h out +# of Build/full is gone. Bazel now generates both from their checked-in .in +# templates (export_headers.bzl): Debug.h by applying CMake's configure_file +# substitution, Backtrace.h by ASKING the host the same question +# find_package(Backtrace) asks. That removed the last AK dependency on a CMake +# build tree. cc_library( name = "AK", ''' -AK_BLOCK_TAIL = ''' hdrs = glob(["AK/*.h"]) + [":ak_gen_headers"], +AK_BLOCK_TAIL = ''' hdrs = glob(["AK/*.h"]), # AK-private defines (CMake PRIVATE) — local_defines so they don't leak to # consumers. FMT_SHARED/AK_HAS_CPPTRACE only affect AK's own TUs. local_defines = [ @@ -397,8 +759,13 @@ def _emit_srcs(srcs): "AK_HAS_CPPTRACE=1", "FMT_SHARED", ], - includes = ["genroot"], deps = [ + # AK/Debug.h + AK/Backtrace.h, generated from their .in templates + # (export_headers.bzl). A cc_library belongs on a deps edge, not in hdrs: + # in hdrs it contributes no include path, which is why every consumer + # failed with "AK/Debug.h: No such file or directory" until this moved. + # The include root travels with the dep, so dependents get it too. + ":generated_ak_headers", VCPKG + ":fmt", VCPKG + ":simdutf", VCPKG + ":mimalloc", @@ -432,27 +799,71 @@ def emit_qt_autogen(moc_hdrs): print() -def moc_headers(): +QT_RUNTIME_BLOCK = ''' +# === Qt6 RUNTIME: the plugins Qt dlopens, and the qt.conf that finds them === +# Not a CMake target -- CMake does not need one, because its binary links a Qt +# whose baked-in prefix already points at the plugins of that same Qt. Bazel's +# does not: it links @qt's libraries and then Qt looks for plugins next to the +# executable, finds none, and falls back to the HOST's plugin directory. Loading +# another Qt build's QPA plugin into this one is the Ubuntu 24.04 SIGSEGV in +# QXcbConnection::initializeScreens; on a box where the versions agree it "works". +# qt_runtime.bzl has the full account (finding 40); both targets below are data of +# //:ladybird, so they are staged in bazel-bin AND in the runfiles tree. +qt_plugin_tree( + name = "qt_plugins", + plugins = ["@qt_plugins//:plugins"], +) + +qt_conf( + name = "qt_conf", +) +''' + + +def emit_qt_runtime(): + print(QT_RUNTIME_BLOCK, end="") + + +def moc_headers(exes=None): """UI/Qt headers with Q_OBJECT, i.e. the ones CMake's AUTOMOC would moc. - GeolocationProviderQt.h is excluded: it is only compiled when Qt6::Positioning - is found, which this configuration does not have (the model shows no - GeolocationProviderQt.cpp compile), so mocking it would be a target Bazel - builds and CMake does not. + Conditionally-compiled headers are filtered by whether the reference build + COMPILES the matching .cpp, not by name. This used to end with + + return [h for h in hdrs if not h.endswith("GeolocationProviderQt.h")] + + justified by "Qt6::Positioning is not found in this configuration" -- true + when written, false at 71fb301a, where upstream made Positioning required and + CMake compiles GeolocationProviderQt.cpp. A name-based exclusion cannot + notice that; asking the model can. Mocking a header CMake does not moc is a + target Bazel builds and CMake does not; NOT mocking one it does is a missing + vtable at link time, so the condition has to be the measurement. """ qt_dir = os.path.join(ROOT, "UI/Qt") + compiled = set() + if exes: + for t in exes.values(): + for a in t["actions"]: + if a["mnemonic"] == "CppCompile": + compiled |= {os.path.basename(i) for i in a["inputs"]} hdrs = [] for f in sorted(os.listdir(qt_dir)): if not f.endswith(".h"): continue if "Q_OBJECT" not in open(os.path.join(qt_dir, f), errors="ignore").read(): continue + # A Q_OBJECT header with a sibling .cpp is compiled-or-not with it. One + # with NO sibling .cpp (a header-only QObject) has nothing to measure, so + # it is moc'd -- AUTOMOC would. + cpp = f[:-2] + ".cpp" + if exes and os.path.exists(os.path.join(qt_dir, cpp)) and cpp not in compiled: + continue hdrs.append("UI/Qt/" + f) - return [h for h in hdrs if not h.endswith("GeolocationProviderQt.h")] + return hdrs def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, - extra_srcs=()): + extra_srcs=(), extra_data=()): """Emit one cc_library/cc_binary from the model. body_only: emit only `name =` + `srcs =` (AK's hand-written block supplies @@ -468,26 +879,50 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, defs = target_defines(t, name) incs = target_private_includes(t) flags = target_flags(t) - deps, rustdeps, sysdeps, unknown = [], [], [], [] + deps, impl_deps, data, sysdeps, unknown = [], [], [], [], [] + binaries = rust_binary_crates() for d in t.get("deps", []): if not d.get("external"): if d["name"] == "LibWeb": deps.append("//Libraries/LibWeb:LibWeb") elif d["name"] in libs: deps.append("//:%s" % d["name"]) # exe deps on service static libs / other production libs elif d["name"] in exes: pass # exe->exe (spawn at runtime, not a link dep) + elif d["name"] in binaries: + # A build_rust_binary() crate. For the STATICLIB crates CMake + # emits both this `-build` custom target AND an + # external dep on the imported library, so the -build edge is + # redundant and dropping it is right. For a BINARY crate it is + # the only edge there is -- and dropping it is what left + # Cranelift out of the Bazel graph entirely, so every browser + # binary died on `CraneliftFFI.h: No such file or directory` + # some 1,600 actions in. + # + # It carries two things Bazel keeps apart: + # deps -- the generated FFI header, via //:_lib + # (headers-only: there is no archive to link). + # data -- the EXECUTABLE, which is not a link input at all. + # LibWasm spawns it at run time, so it belongs in + # the runfiles of everything that links LibWasm. + # That, plus rewrite_host_path_define turning the + # baked-in absolute path into a bare file name, is + # what makes Ladybird's own sibling-of-self lookup + # find it in any checkout instead of only in mine. + b = binaries[d["name"]] + if b["ffi_headers"]: + rdeps, rimpl = rust_dep_labels(b["crate"]) + deps.extend(rdeps) + impl_deps.extend(rimpl) + data.append("//:" + b["bin"]) continue lab = dep_label(d, targets, so, ar) if lab is None: continue - if isinstance(lab, list): - deps.extend(lab) - elif isinstance(lab, tuple): - (sysdeps if lab[0]=="SYS" else unknown).append(lab[1]) - else: - deps.append(lab) + deps.extend(lab.deps) + impl_deps.extend(lab.impl) + sysdeps.extend(lab.sys) + unknown.extend(lab.unknown) rule = "cc_binary" if is_exe else "cc_library" if header: print(f"# === {name} ({t['kind']}, {len(srcs)} TU) ===") - if rustdeps: print(f"# RUST deps (deferred): {rustdeps}") if unknown: print(f"# UNKNOWN deps: {unknown}") if not body_only: print(f"{rule}(") @@ -506,12 +941,32 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, copt_toks = list(flags) for i in incs: if i.startswith("/"): - # System include roots (/usr Qt6, libdrm) can't be per-target - # copts: Bazel rejects a path outside the execution root even - # with -isystem. They live as global -isystem in .bazelrc - # (mirroring CMake's find_package include dirs). + # System include roots (/usr Qt6, libdrm, glib) can't be + # per-target copts: Bazel rejects a path outside the execution + # root even with -isystem. They live in .bazelrc's + # CPLUS_INCLUDE_PATH (mirroring CMake's find_package dirs) -- + # and CHECKED against it, not just skipped. A silent `continue` + # here means a root CMake uses and .bazelrc lacks produces no + # output at all: I transcribed three of glib's four roots by + # hand and the build failed 3,800 actions later on + # `gio/gdesktopappinfo.h: No such file or directory`, which is + # the same hand-copied-fact bug as everything else in this + # repin. Reported, not fixed automatically: which roots are + # acceptable host escapes is a judgement (see README gap 3), + # so the emitter's job is to refuse to hide the difference. + record_host_include(i) continue - if "vcpkg_installed" in i: + if i.startswith(BUILD_REL + "Libraries/"): + # CMake's per-library FFI dir (FFI_OUTPUT_DIR defaults to the + # library's own binary dir), which is where its crate's + # RustFFI.h lands. Pointing this at the CMake tree is what kept + # the build dependent on Build/full -- and it also MASKED a real + # bug: it shadowed the ambiguous ffi/ roots, so a bare + # silently resolved here instead of to another + # crate's header. Bazel's equivalent dir travels with the + # //:_lib dep (cargo.bzl), so this copt is simply dropped. + pass + elif "vcpkg_installed" in i: # Not a copt any more. The vcpkg include dirs (include/, # include/skia, include/harfbuzz, include/libxml2) are carried by # the //Meta/vcpkg: dep as system_includes, so the include @@ -519,6 +974,16 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, # without depending on skia now fails to compile, # which is the whole point of declaring inputs. pass + elif i in (BUILD_REL.rstrip("/") + "/UI", BUILD_REL.rstrip("/") + "/UI/Qt"): + # CMake's UI gendir, whose only non-autogen contents are the two + # SPIR-V shader headers (WebContentViewLinux{Frag,Vert}Shader.h). + # Bazel generates both itself and carries them on + # :generated_shader_headers with includes=["UI/Qt"], so the + # include path arrives through the dep graph. Dropped, not + # relocated -- and checked by removal: with Build/full/UI moved + # off the machine, //:ladybird still builds and renders + # byte-identically. + pass else: copt_toks.append("-I" + i) if copt_toks: @@ -528,6 +993,24 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, # where the lib's INTERFACE/PRIVATE system deps flow to the executable). if sysdeps: print(" linkopts = [%s]," % ", ".join("%r" % ("-l"+l) for l in sorted(set(sysdeps)))) + # Runtime tools this target SPAWNS: not link inputs, but real inputs. + # On a cc_library `data` propagates into the runfiles of every binary + # that links it, which is exactly the reach cranelift-compiler needs + # (LibWasm is linked by WebContent, WebWorker and ladybird). + data = list(data) + list(extra_data) + if data: + print(" data = [%s]," % ", ".join("%r" % x for x in sorted(set(data)))) + # PRIVATE deps: used to compile this library, not part of its interface, + # so their include dirs stop here. Bazel's name for CMake's PRIVATE, and + # the only thing that keeps a crate's unprefixed FFI dir from reaching a + # library that has its OWN crate's RustFFI.h to find -- see + # rust_dep_labels(). Not valid on a cc_binary (nothing depends on it, so + # everything it has is already private). + if impl_deps and not is_exe: + print(" implementation_deps = [%s]," % + ", ".join("%r" % x for x in sorted(set(impl_deps)))) + elif impl_deps: + deps.extend(impl_deps) deps.append("//:all_source_headers") alldeps = sorted(set(deps)) if alldeps: @@ -562,11 +1045,21 @@ def main(): print(AK_BLOCK_TAIL, end="") for name in ROOT_TARGETS: emit_target(name, targets, libs, exes, so, ar) - emit_qt_autogen(moc_headers()) + emit_qt_autogen(moc_headers(exes)) + emit_qt_runtime() for name in QT_TARGETS: emit_target(name, targets, libs, exes, so, ar, - extra_srcs=[":qt_moc", ":qt_rcc"]) + extra_srcs=[":qt_moc", ":qt_rcc"], + # The Qt plugins + qt.conf: runtime inputs of the GUI, in the + # same sense as the cranelift-compiler binary LibWasm spawns. + # Plus the five helper processes the UI spawns by path: see + # spawned_services() for why `bazel build //:ladybird` used to + # leave a stale WebContent behind, and what that looks like. + extra_data=[":qt_conf", ":qt_plugins"] + + [":" + s for s in spawned_services()]) print(VCPKG_TAIL, end="") + # Last, so it is the final thing on stderr rather than buried in the middle. + report_host_includes() def lib_hdr_glob(name, srcs): diff --git a/examples/ladybird/workspace/Meta/emit_cargo_bazel.py b/examples/ladybird/workspace/Meta/emit_cargo_bazel.py index 6ec56b1..f7210ab 100644 --- a/examples/ladybird/workspace/Meta/emit_cargo_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_cargo_bazel.py @@ -51,8 +51,8 @@ enumerate; Bazel deletes undeclared outputs. So the header list is the union of CMake's declaration and the observed set, and the observed set wins where they differ (see FFI_HEADERS_OBSERVED). - * **The registry/workspace split is the whole reason this is cheap.** 154 of - the 167 `[[package]]` entries have a checksum; the other 13 are the in-tree + * **The registry/workspace split is the whole reason this is cheap.** 155 of + the 166 `[[package]]` entries have a checksum; the other 11 are the in-tree workspace members, which have no checksum because they are *sources*. A crate with no checksum must never become a fetch rule (it would 404), and a registry crate that silently drops out is a build failure a long way from its @@ -81,10 +81,14 @@ HEADER = "# AUTO-GENERATED by Meta/emit_cargo_bazel.py — do not edit.\n" -# The two cargo workspaces. Flap is `exclude`d from the root workspace and has -# its own lock with exactly 3 packages (flapc, in-tree bytecode_def, and -# smallvec from crates.io), so it needs the same treatment at 1/50th the size. +# The two cargo workspaces. Flap is `exclude`d from the root workspace and +# resolves its own lock, so it needs the same treatment at a fraction of the +# size. What is IN that lock is deliberately not written down here: this comment +# used to name its three packages, one of which upstream deleted (bytecode_def, +# in a32d9c9f), leaving the emitter to print a confident description of a +# package that no longer existed. emit_ring() reads the lock instead. LOCKS = ["Cargo.lock", "Libraries/LibJS/Flap/Cargo.lock"] +FLAP_LOCK = "Libraries/LibJS/Flap/Cargo.lock" # The host triple cargo builds for. Not a guess: `rustc -vV`'s `host:` line is # what CMake reads (Meta/CMake/rust_crate.cmake), and it names the directory the @@ -109,6 +113,20 @@ def toolchain_channel(): return _toml(os.path.join(ROOT, "rust-toolchain.toml"))["toolchain"]["channel"] +def root_workspace_members(): + """The root cargo workspace's member directories, read from Cargo.toml. + + Which workspace a crate belongs to decides its SOURCE SET, and it is the one + thing here that must not be guessed. `Libraries/LibWasm/Rust` is member #7 of + the root workspace, so cargo resolves the whole workspace when asked for it + and its input set is the shared `crate_srcs`; `Libraries/LibJS/Flap` is in + `exclude` and has its own lock, so its input set is its own subtree. Reading + the member list means adding a crate to either workspace needs no edit here. + """ + ws = _toml(os.path.join(ROOT, "Cargo.toml"))["workspace"] + return {os.path.normpath(m) for m in ws.get("members", [])} + + # --------------------------------------------------------------------------- # The lock files. # --------------------------------------------------------------------------- @@ -201,11 +219,21 @@ def repo_name(name, version): "libtextcodec_rust": ["RustFFI.h"], "liburl_rust": ["RustFFI.h"], "libunicode_rust": ["RustFFI.h"], - "libweb_rust": ["HTML/Parser/RustFFI.h", "HTMLTokenizerRustFFI.h"], - "libweb_layout_rust": ["Layout/TreeBuilderRustFFI.h"], - "libweb_css_rust": ["ComputedValuesRustFFI.h", "RustFFI.h", - "SelectorRustFFI.h", "StyleValueRustFFI.h"], + # Upstream consolidated libweb_css_rust and libweb_layout_rust BACK into + # libweb_rust, so this one crate now writes what three used to -- eleven + # files of its own plus HTMLTokenizerRustFFI.h from its path-dependency + # libweb_html_tokenizer, which lands in the same FFI_OUTPUT_DIR. Two of them + # are .inc, not .h: an undeclared .inc is deleted by Bazel exactly like an + # undeclared header, and it is #included the same way. + "libweb_rust": ["ComputedValuesRustFFI.h", "HTML/Parser/RustFFI.h", + "HTMLTokenizerRustFFI.h", "Layout/LayoutRustFFI.h", + "Layout/TreeBuilderRustFFI.h", "RustFFI.h", + "SelectorRustFFI.h", "StyleEngineBridgeGenerated.h", + "StyleEngineBridgeGenerated.inc", "StyleEngineRustFFI.h", + "StyleEngineStateFactsGenerated.inc", "StyleValueRustFFI.h"], "libweb_content_blocker_rust": ["ContentBlockerRustFFI.h"], + # A BINARY crate that also emits a header -- see BINARY_CMAKELISTS below. + "libwasm_cranelift": ["CraneliftFFI.h"], } # Where each crate's headers are included FROM. CMake sets FFI_OUTPUT_DIR to the @@ -221,9 +249,8 @@ def repo_name(name, version): "liburl_rust": "LibURL", "libunicode_rust": "LibUnicode", "libweb_rust": "LibWeb", - "libweb_layout_rust": "LibWeb", - "libweb_css_rust": "LibWeb", "libweb_content_blocker_rust": "LibWeb", + "libwasm_cranelift": "LibWasm", } # The non-Rust files each crate's build script reads, taken from the DEPFILES the @@ -235,62 +262,223 @@ def repo_name(name, version): # found most of these; the depfile finds all of them, including the ones reached # through a `manifest_dir.parent()` join. CRATE_EXTRA_INPUTS = { - "libjs_rust": ["Libraries/LibJS/Bytecode/Bytecode.def"], - # The LibWeb ones (8 CSS data files for libweb_css_rust, TagNames.h + - # AttributeNames.h + Entities.json for libweb_rust) are NOT here: they are - # inside the Libraries/LibWeb package, so they ride in that package's - # rust_crate_srcs filegroup instead. Same measurement, different side of a - # package boundary. + # libjs_rust's build script derives its instruction types from the Flap + # interpreter (upstream a32d9c9f: `include_str!("../../Interpreter/ + # interpreter.flap")` via flapc::metadata), so the .flap file is a compile + # input reached by a `manifest_dir.parent()` join -- exactly the shape the + # depfile finds and reading build.rs nearly misses. It replaced + # Libraries/LibJS/Bytecode/Bytecode.def, which that commit DELETED. + "libjs_rust": ["Libraries/LibJS/Interpreter/interpreter.flap"], + # libwasm_cranelift's build.rs reads `manifest_dir.join("../Opcode.h")` and + # generates a Rust constant per `M(name, value, ...)` line, so the WASM + # opcode table is a compile input to the cranelift compiler binary. + "libwasm_cranelift": ["Libraries/LibWasm/Opcode.h"], + # The LibWeb ones are NOT here: they are inside the Libraries/LibWeb package, + # so they ride in that package's rust_crate_srcs filegroup instead -- + # PACKAGE_EXTRA_INPUTS below. Same measurement, different side of a package + # boundary. +} + +# The same measurement for the files that live inside ANOTHER Bazel package, so +# the root package cannot glob them: libweb_rust's build script generates Rust +# from the CSS data files and reads the HTML name headers + Entities.json. Kept +# here rather than in emit_libweb_bazel.py so ONE place holds the measurement and +# ONE --report checks it against the reference build's depfiles. +PACKAGE_EXTRA_INPUTS = { + "Libraries/LibWeb": [ + "CSS/Enums.json", + "CSS/Keywords.json", + "CSS/LogicalPropertyGroups.json", + "CSS/Properties.json", + "CSS/PseudoClasses.json", + "CSS/PseudoElementPropertyGroups.txt", + "CSS/PseudoElements.json", + "CSS/TransformFunctions.json", + "CSS/Units.json", + "HTML/AttributeNames.h", + "HTML/Parser/Entities.json", + "HTML/TagNames.h", + ], } # The crate source trees. A crate depends on more than its own directory: every # one of them `#[path = "../../../RustAllocator.rs"]`s the shared allocator shim, # and several are path-dependencies of each other (libweb_rust -> -# libweb_html_tokenizer, liburl_rust -> libregex_rust), which cargo resolves -# through the workspace. So the source set is "all the in-tree crates plus the -# shim", declared once. Over-declaring costs a rebuild of 11 crates when any .rs -# changes; under-declaring silently reuses a stale archive. For a first landing -# that trade is the right way round, and it is honest debt to name. +# libweb_html_tokenizer, liburl_rust -> libregex_rust, libjs_rust -> flapc), +# which cargo resolves through the workspace. So the source set is "all the +# in-tree crates plus the shim", declared once. Over-declaring costs a rebuild of +# every crate when any .rs changes; under-declaring silently reuses a stale +# archive. For a first landing that trade is the right way round, and it is +# honest debt to name (todo 6f39dd58). # -# Split in two because BAZEL PACKAGES CUT ACROSS THE CARGO WORKSPACE: five of the +# DERIVED from Cargo.toml's member list and the manifests' `path =` +# dependencies, never written down -- because a directory written down here is a +# glob that outlives the directory. Upstream a32d9c9f deleted +# Libraries/LibJS/BytecodeDef/, which was a hardcoded `allow_empty = False` glob +# pattern in this list, and the whole build then failed at LOADING time with +# "glob pattern didn't match anything" -- before any target could report why. +# Deriving the patterns means a deleted crate deletes its own pattern. +# +# Split in two because BAZEL PACKAGES CUT ACROSS THE CARGO WORKSPACE: some of the # crates live under Libraries/LibWeb, which is its own package, and glob() is # package-relative -- the root package cannot see them at all. So they arrive as # a filegroup label that LibWeb's own BUILD file defines -# (//Libraries/LibWeb:rust_crate_srcs, emitted by emit_libweb_bazel.py). This is -# the first place in this migration where the two systems' unit of grouping -# genuinely disagree: cargo says "one workspace", Bazel says "two packages", and -# a label is the only thing that crosses. -CRATE_SRC_GLOBS = [ +# (//Libraries/LibWeb:rust_crate_srcs, emitted by emit_libweb_bazel.py from +# package_crate_globs() HERE, so the two sides cannot disagree about which +# directories exist). This is the first place in this migration where the two +# systems' unit of grouping genuinely disagree: cargo says "one workspace", Bazel +# says "two packages", and a label is the only thing that crosses. +FOREIGN_PACKAGES = {"Libraries/LibWeb": "//Libraries/LibWeb:rust_crate_srcs"} + +# Files at the repo root that every crate needs: the two workspace files, the +# toolchain pin and the shared allocator shim. Not a directory between them, so +# nothing here can outlive a crate. +CRATE_SRC_ROOT_FILES = [ "Cargo.toml", "Cargo.lock", "rust-toolchain.toml", "Libraries/RustAllocator.rs", - "Libraries/*/Rust/**", - "Libraries/LibJS/BytecodeDef/**", ] -CRATE_SRC_LABELS = ["//Libraries/LibWeb:rust_crate_srcs"] - -# flapc's own workspace: excluded from the root one, its own lock, 3 packages. -FLAPC_SRC_GLOBS = [ - "Libraries/LibJS/Flap/Cargo.toml", - "Libraries/LibJS/Flap/Cargo.lock", - "Libraries/LibJS/Flap/src/**", - # benches/ and tests/ are not built by `cargo rustc --bin flapc`, but they - # ARE manifest inputs: Cargo.toml declares `[[bench]] name = "compiler"`, and - # cargo validates every declared target while PARSING the manifest, before it - # decides what to build ("can't find `compiler` bench at - # benches/compiler.rs"). The bench also include_str!s - # tests/interpreter-layout.conf. So the manifest's target list is part of the - # input set whether or not those targets are compiled -- a distinction CMake - # never makes, and one only a sandbox reveals. - "Libraries/LibJS/Flap/benches/**", - "Libraries/LibJS/Flap/tests/**", - "Libraries/LibJS/BytecodeDef/**", - "Libraries/LibJS/Bytecode/Bytecode.def", - "Libraries/LibJS/Interpreter/interpreter.flap", - "rust-toolchain.toml", -] +# `target/` is cargo's own output dir: never an input, and a developer who has run +# cargo by hand in the tree would otherwise have it globbed into every crate's +# action inputs (the reference build sets CARGO_TARGET_DIR into the build dir, so +# it does not exist here -- but that is a property of how we invoke cargo, not of +# the tree). +# +# ANCHORED to each crate root, NOT `**/target/**`. The unanchored form is wrong +# and fails loudly, which is the only good thing about it: flapc has a Rust MODULE +# at `src/target/` (the code generator's per-architecture backends), so +# `**/target/**` dropped 40 source files and the build died with +# `error[E0583]: file not found for module 'target'`. A glob exclusion is a +# pattern over paths and knows nothing about what a directory MEANS; "the dir +# cargo writes to" is `/target`, and only the anchored form says that. +def crate_src_glob_excludes(dirs): + return [d + "/target/**" for d in dirs] + + +def _manifest_path_deps(manifest_rel): + """The `path = "..."` dependency dirs declared by one manifest, ROOT-relative. + + Read from the manifest rather than from the workspace member list, because a + path dependency can point OUTSIDE the workspace: libjs_rust build-depends on + `flapc = { path = "../Flap" }`, and Libraries/LibJS/Flap is `exclude`d from + the root workspace. Miss it and touching a flapc source does not rebuild the + crate whose instruction types are derived from it. + """ + path = os.path.join(ROOT, manifest_rel) + if not os.path.exists(path): + return [] + d = os.path.dirname(manifest_rel) + out = [] + for m in re.finditer(r'path\s*=\s*"([^"]+)"', open(path).read()): + p = os.path.normpath(os.path.join(d, m.group(1))) + if os.path.isdir(os.path.join(ROOT, p)): + out.append(p) + return out + + +def crate_dirs(): + """Every in-tree crate DIRECTORY the root workspace resolves, ROOT-relative. + + Starts from Cargo.toml's `members` and closes over `path =` dependencies, so + a crate that is added, moved or deleted upstream needs no edit here. + """ + seen = set() + queue = sorted(root_workspace_members()) + while queue: + d = queue.pop() + if d in seen or not os.path.isdir(os.path.join(ROOT, d)): + continue + seen.add(d) + queue += _manifest_path_deps(os.path.join(d, "Cargo.toml")) + return sorted(seen) + + +def _in_foreign_package(d): + for pkg in FOREIGN_PACKAGES: + if d == pkg or d.startswith(pkg + "/"): + return pkg + return None + + +def crate_src_globs(): + """The ROOT package's glob patterns for the shared crate source set.""" + return CRATE_SRC_ROOT_FILES + [ + d + "/**" for d in crate_dirs() if not _in_foreign_package(d)] + + +def crate_src_labels(): + """One filegroup label per foreign package that actually holds a crate.""" + return sorted({FOREIGN_PACKAGES[p] for p in + (_in_foreign_package(d) for d in crate_dirs()) if p}) + + +def package_crate_globs(pkg): + """The crate globs for a foreign package, PACKAGE-relative. + + Called by emit_libweb_bazel.py, so the filegroup on that side of the boundary + and the glob on this side are derived from one list. + """ + return sorted(os.path.relpath(d, pkg) + "/**" + for d in crate_dirs() if _in_foreign_package(d) == pkg) + + +def _excluded_dirs(): + """The workspaces Cargo.toml `exclude`s, which resolve their own locks.""" + ws = _toml(os.path.join(ROOT, "Cargo.toml"))["workspace"] + return [os.path.normpath(d) for d in ws.get("exclude", []) + if os.path.isdir(os.path.join(ROOT, d))] + + +def flapc_src_globs(): + """The `exclude`d flapc workspace: its own subtree, plus what it reads out of + the tree above it. + + One `/**` rather than a pattern per subdirectory, because the manifest's + TARGET list is part of the input set whether or not those targets are built: + Cargo.toml declares `[[bench]] name = "compiler"`, and cargo validates every + declared target while PARSING the manifest, before it decides what to build + ("can't find `compiler` bench at benches/compiler.rs"). That bench also + include_str!s tests/interpreter-layout.conf. A distinction CMake never makes, + and one only a sandbox reveals -- and a whole-subtree glob cannot lose a + subdirectory the way an enumerated list of them can. + """ + dirs = _excluded_dirs() + extra = ["rust-toolchain.toml"] + for d in dirs: + # src/metadata.rs include_str!s the interpreter definition from two + # directories up: flapc DERIVES the bytecode from it (upstream a32d9c9f), + # so it is an input, and it lives outside the excluded subtree. + extra += [x for x in _include_str_paths(d) if not x.startswith(d + "/")] + return sorted(d + "/**" for d in dirs) + sorted(set(extra)) + + +def _include_str_paths(d): + """ROOT-relative paths a crate subtree include_str!s, derived by scanning it. + + Derived rather than listed for the same reason as everything else here: the + one file flapc reads from outside its own subtree changed name and location + in the commit this repin follows. + """ + out = set() + pat = re.compile(r'include_str!\s*\(\s*"([^"]+)"') + for dirpath, dirnames, filenames in os.walk(os.path.join(ROOT, d)): + dirnames[:] = [x for x in dirnames if x != "target"] + for fn in filenames: + if not fn.endswith(".rs"): + continue + full = os.path.join(dirpath, fn) + try: + text = open(full, errors="replace").read() + except OSError: + continue + for m in pat.finditer(text): + p = os.path.normpath(os.path.join( + os.path.relpath(dirpath, ROOT), m.group(1))) + if os.path.exists(os.path.join(ROOT, p)): + out.add(p) + return sorted(out) # Where each crate's Cargo.toml lives, relative to the source root. Parsed from # the import_rust_crate() call site: MANIFEST_PATH is relative to the CMakeLists @@ -321,10 +509,15 @@ def _multi(block, key, stops): "Libraries/LibURL/CMakeLists.txt", "Libraries/LibUnicode/CMakeLists.txt", "Libraries/LibWeb/CMakeLists.txt", + # LibWasm calls build_rust_binary(), not import_rust_crate() -- see + # parse_cmake_binaries() for why that distinction is load-bearing and why it + # nevertheless has to be in the SAME list. + "Libraries/LibWasm/CMakeLists.txt", ] _KEYS = ["MANIFEST_PATH", "CRATE_NAME", "FFI_OUTPUT_DIR", "FFI_HEADER", - "FFI_HEADERS", "FEATURES"] + "FFI_HEADERS", "FEATURES", "BINARY_NAME", "OUTPUT_NAME", + "OUTPUT_PATH_VAR"] def parse_cmake_crates(): @@ -361,7 +554,10 @@ def parse_cmake_crates(): # var -> features, for the `if (NOT BUILD_SHARED_LIBS)` idiom above. vars_ = {} for m in re.finditer( - r"set\(\s*(\w+)\s+FEATURES\s+([\w\s]+?)\s*\)", text): + # `-` is legal in a cargo feature name (`style-recording`), so it + # has to be in the character class -- a class that excludes it + # truncates the feature rather than failing to match it. + r"set\(\s*(\w+)\s+FEATURES\s+([\w\s-]+?)\s*\)", text): vars_[m.group(1)] = m.group(2).split() for block in RE_IMPORT.findall(text): crate = _arg(block, "CRATE_NAME") @@ -384,6 +580,179 @@ def parse_cmake_crates(): return sorted(out, key=lambda c: c["crate"]) +def parse_cmake_binaries(): + """[{crate, bin, manifest, ffi_headers}] from build_rust_binary() calls. + + The SECOND kind of Rust target CMake has, and the one this migration missed + for a long time, with a consequence worth recording: because + `Libraries/LibWasm/CMakeLists.txt` was absent from CMAKELISTS and + `build_rust_binary` was not parsed, the Cranelift crate was **entirely absent + from the Bazel graph** -- and every browser binary failed with + `CraneliftFFI.h: No such file or directory`, 1,600 actions into a build that + could not have worked. The reference build hid it twice over: the header was + sitting in `Build/full/Libraries/LibWasm`, which a global `-I` reached, and + the compiler binary was reached by an ABSOLUTE PATH baked into a + `-DWASM_CRANELIFT_COMPILER_PATH=` -- i.e. two host escapes covering for one + missing target. + + Two things make a build_rust_binary crate different from an + import_rust_crate one, and both matter here: + + * There is NO archive to link. `cargo rustc --bin` is the whole build, and + what the C++ side consumes is the *executable*, spawned at run time. + * It can STILL emit an FFI header. libwasm_cranelift's build.rs runs + cbindgen exactly like the staticlib crates do, and + `Libraries/LibWasm/CraneliftBridge.cpp` includes the result -- bare + (`#include `), because CMake's FFI_OUTPUT_DIR is + LibWasm's own binary dir. + + Everything else it shares with import_rust_crate, FEATURES included, and that + is parsed here for the same reason: upstream's `style-replay` binary is built + `FEATURES style-recording` out of the same crate as libweb_rust's staticlib. + A parser that only read FEATURES on the staticlib side would build a + different binary than CMake does and say nothing -- the finding-23 shape, + where the two build systems agree on the target list and disagree on the ABI. + + So a binary crate needs both halves: a cargo_binary for the executable, and + the same declared-header treatment the staticlib crates get. The two are + reported separately for the same reason import_rust_crate's list is parsed + rather than hard-coded: which kind a crate is, and whether it emits a header, + is a property of the CMake call site. + """ + out = [] + for rel in CMAKELISTS: + path = os.path.join(ROOT, rel) + if not os.path.exists(path): + continue + with open(path) as f: + text = f.read() + pkgdir = os.path.dirname(rel) + # The same `if (NOT BUILD_SHARED_LIBS)` variable form the staticlib side + # resolves -- see parse_cmake_crates() for why that branch is the one the + # Bazel overlay mirrors. + vars_ = {} + for m in re.finditer(r"set\(\s*(\w+)\s+FEATURES\s+([\w\s-]+?)\s*\)", text): + vars_[m.group(1)] = m.group(2).split() + for block in RE_BINARY.findall(text): + crate = _arg(block, "CRATE_NAME") + manifest = _arg(block, "MANIFEST_PATH") + binary = _arg(block, "BINARY_NAME") + if not (crate and manifest and binary): + continue + # FFI_OUTPUT_DIR present at the call site means the build script is + # expected to write a header. Which header it writes is the + # observation (FFI_HEADERS_OBSERVED), exactly as for the staticlibs: + # build_rust_binary has no FFI_HEADERS argument at all, so CMake + # declares nothing and there is nothing to compare against. + emits_ffi = _arg(block, "FFI_OUTPUT_DIR") is not None + features = _multi(block, "FEATURES", _KEYS) + for m in re.finditer(r"\$\{(\w+)\}", block): + features += vars_.get(m.group(1), []) + out.append({ + "crate": crate, + "bin": binary, + "manifest": os.path.normpath(os.path.join(pkgdir, manifest)), + "output_name": _arg(block, "OUTPUT_NAME") or binary, + "features": sorted(set(features)), + "emits_ffi": emits_ffi, + }) + return sorted(out, key=lambda c: c["crate"]) + + +def binary_specs(): + """The per-binary-crate recipe, with the FFI headers each one emits. + + `ffi_bare_include` is derived the same way the staticlib crates' is (by + scanning the tree for a directory-less `#include`), not asserted: LibWasm + spells it `#include `, and it must be that scan that says so. + """ + specs = [] + for b in parse_cmake_binaries(): + headers = sorted(FFI_HEADERS_OBSERVED.get(b["crate"], [])) \ + if b["emits_ffi"] else [] + specs.append(dict(b, ffi_headers=headers, + ffi_prefix=FFI_PREFIX.get(b["crate"], ""))) + bare = crates_included_bare(_bare_scan_specs()) + for sp in specs: + sp["ffi_bare_include"] = bool(bare.get(sp["crate"])) + return specs + + +def _bare_scan_specs(): + """Every Rust crate that emits a header, staticlib and binary alike. + + ONE list, because the bare-include scan is a question about the whole tree: + it looks for a directory-less `#include ` where X is any crate's header + basename, and a scan that only knew about the staticlibs would answer "no" + for CraneliftFFI.h no matter how many TUs include it that way. + """ + return [{"crate": c["crate"], "manifest": c["manifest"], + "ffi_headers": sorted(set(c["cmake_ffi_headers"]) | + set(FFI_HEADERS_OBSERVED.get(c["crate"], [])))} + for c in parse_cmake_crates()] + \ + [{"crate": b["crate"], "manifest": b["manifest"], + "ffi_headers": sorted(FFI_HEADERS_OBSERVED.get(b["crate"], [])) + if b["emits_ffi"] else []} + for b in parse_cmake_binaries()] + + +def crates_included_bare(specs): + """Crates whose FFI header is #included with NO directory prefix. + + Derived by scanning the source tree for `#include ` where X is exactly a + crate's FFI header basename, rather than hard-coding a list: which TUs spell + the include bare is a property of Ladybird's source and changes when someone + edits an include line. + + Why it matters: 8 of the 10 crates emit a header named `RustFFI.h`. CMake + gives each library only its OWN crate's binary dir, so a bare + `#include ` is unambiguous there. Bazel propagates every dep's + include dirs onto one command line, so exposing an unprefixed dir for all + crates lets one crate's RustFFI.h satisfy another's include -- silently, since + both exist and only ORDER decides. So the unprefixed dir is opt-in per crate, + and only for crates that actually need it. + """ + bare = {} + pat = re.compile(r'^\s*#\s*include\s*<([^/>]+)>\s*$', re.M) + wanted = {} + owner_dir = {} + for sp in specs: + # The crate's manifest lives at /[sub/]Rust/Cargo.toml, so the + # owning library directory is the manifest's path with the trailing + # Rust/Cargo.toml removed -- that is the tree whose TUs may include this + # crate's header bare. + d = os.path.dirname(sp["manifest"]) + while d and os.path.basename(d) in ("Rust", "rust"): + d = os.path.dirname(d) + owner_dir[sp["crate"]] = d + for h in sp["ffi_headers"]: + wanted.setdefault(os.path.basename(h), set()).add(sp["crate"]) + for dirpath, dirnames, filenames in os.walk(os.path.join(ROOT, "Libraries")): + dirnames[:] = [d for d in dirnames if d not in ("Rust", "target")] + for fn_ in filenames: + if not fn_.endswith((".cpp", ".h")): + continue + fpath = os.path.join(dirpath, fn_) + try: + with open(fpath, errors="replace") as f: + text = f.read() + except OSError: + continue + for m in pat.finditer(text): + inc = m.group(1) + if inc not in wanted: + continue + # Attribute the bare include to the crate owned by the library + # this file lives in, so we never enable the unprefixed dir for + # an unrelated crate that happens to emit the same basename. + rel = os.path.relpath(fpath, ROOT) + for crate in wanted[inc]: + owner = owner_dir.get(crate) + if owner and rel.startswith(owner + os.sep): + bare[crate] = True + return bare + + def crate_specs(): """The per-crate build recipe the Bazel rule consumes. @@ -403,6 +772,13 @@ def crate_specs(): "ffi_headers": headers, "ffi_prefix": FFI_PREFIX.get(c["crate"], ""), }) + # Derived from the source tree, not hard-coded: which crates' headers are + # included without a directory prefix. Scanned over EVERY crate that emits a + # header (binaries included), so the answer does not depend on which emitter + # asked. + bare = crates_included_bare(_bare_scan_specs()) + for sp in specs: + sp["ffi_bare_include"] = bool(bare.get(sp["crate"])) return specs @@ -486,6 +862,7 @@ def emit_index(crates, specs): print(" \"features\": %r," % s["features"]) print(" \"ffi_headers\": %r," % s["ffi_headers"]) print(" \"ffi_prefix\": %r," % s["ffi_prefix"]) + print(" \"ffi_bare_include\": %r," % s["ffi_bare_include"]) print(" },") print("}") print() @@ -495,7 +872,7 @@ def emit_index(crates, specs): print("RUST_TRIPLE = %r" % TRIPLE) -def emit_ring(crates, specs): +def emit_ring(crates, specs, binaries): """cargo_ring.bzl: the whole Rust ring as a macro for the ROOT package. A macro instantiated in the root package rather than its own `Meta/cargo` @@ -511,20 +888,20 @@ def emit_ring(crates, specs): and a hand-kept copy of them would be the bug (finding 23). """ sys.stdout.write(HEADER) - print('load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "rust_sysroot")') + print('load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "cargo_bare_include", "rust_sysroot")') print('load(":cargo_index.bzl", "CARGO_CRATE_FILES", "CARGO_CRATE_SPECS")') print() - print("# Ladybird's %d production Rust crates and flapc, BUILT BY BAZEL from the" - % len(specs)) - print("# %d crates.io crates Bazel fetched -- replacing the prebuilt 260 MB" % len(crates)) - print("# librust_combined.a that was copied out of Build/full/cargo/, the `ar -M`") - print("# merge that produced it (README step 1b), and the reference build's flapc") - print("# binary. Nothing here names Build/full.") + print("# Ladybird's %d production Rust crates and %d binary crates, BUILT BY BAZEL" + % (len(specs), len(binaries))) + print("# from the %d crates.io crates Bazel fetched -- replacing the prebuilt" % len(crates)) + print("# 260 MB librust_combined.a that was copied out of Build/full/cargo/, the") + print("# `ar -M` merge that produced it (README step 1b), and the reference build's") + print("# flapc and cranelift-compiler binaries. Nothing here names Build/full.") print("#") print("# Every attribute comes from CARGO_CRATE_SPECS, which") - print("# Meta/emit_cargo_bazel.py parses out of import_rust_crate() in CMake, so a") - print("# feature or header list cannot drift from the reference build without the") - print("# emitter's --report saying so.") + print("# Meta/emit_cargo_bazel.py parses out of import_rust_crate() and") + print("# build_rust_binary() in CMake, so a feature or header list cannot drift from") + print("# the reference build without the emitter's --report saying so.") print() print("def cargo_ring():") print(" # The pinned toolchain (rust-toolchain.toml says %s), assembled from" @@ -543,12 +920,19 @@ def emit_ring(crates, specs): print(" # cargo WORKSPACE. Each of them path-depends on") print(" # Libraries/RustAllocator.rs and several are path-dependencies of each") print(" # other, so cargo resolves the whole workspace whichever crate you ask") - print(" # for. Over-declaring costs a rebuild of 11 crates when any .rs changes;") - print(" # under-declaring silently reuses a stale archive. The trade is the right") - print(" # way round, and it is real debt: per-crate source sets need the") - print(" # path-dependency graph read out of the manifests.") - print(" crate_srcs = native.glob(%r, allow_empty = False) + %r" - % (CRATE_SRC_GLOBS, CRATE_SRC_LABELS)) + print(" # for. Over-declaring costs a rebuild of every crate when any .rs") + print(" # changes; under-declaring silently reuses a stale archive. The trade is") + print(" # the right way round, and it is real debt: per-crate source sets need") + print(" # the path-dependency graph read out of the manifests.") + print(" #") + print(" # The patterns are DERIVED from Cargo.toml's member list closed over the") + print(" # manifests' `path =` deps, not written down: a directory written down") + print(" # here is an allow_empty=False glob that outlives the directory, and") + print(" # loading then fails before any target can say why.") + root_dirs = [d for d in crate_dirs() if not _in_foreign_package(d)] + print(" crate_srcs = native.glob(%r, exclude = %r, allow_empty = False) + %r" + % (crate_src_globs(), crate_src_glob_excludes(root_dirs), + crate_src_labels())) for s in specs: extra = CRATE_EXTRA_INPUTS.get(s["crate"], []) srcs = "crate_srcs" @@ -562,6 +946,8 @@ def emit_ring(crates, specs): print(" crate_features = CARGO_CRATE_SPECS[%r][\"features\"]," % s["crate"]) print(" ffi_headers = CARGO_CRATE_SPECS[%r][\"ffi_headers\"]," % s["crate"]) print(" ffi_prefix = CARGO_CRATE_SPECS[%r][\"ffi_prefix\"]," % s["crate"]) + print(" ffi_bare_include = CARGO_CRATE_SPECS[%r][\"ffi_bare_include\"]," + % s["crate"]) print(" manifest = CARGO_CRATE_SPECS[%r][\"manifest\"]," % s["crate"]) if extra: print(" # Build-script inputs, taken from the reference build's DEPFILES") @@ -584,21 +970,107 @@ def emit_ring(crates, specs): print(" name = %r," % (s["crate"] + "_lib")) print(" crate = %r," % (":" + s["crate"])) print(" )") + if s["ffi_bare_include"]: + # The owning library's TUs spell this crate's header with no + # directory. That dir must NOT ride on cargo_lib (it propagates + # transitively and 8 crates ship a RustFFI.h), so it is its own + # target, depended on by exactly one library. + print(" cargo_bare_include(") + print(" name = %r," % (s["crate"] + "_bare_include")) + print(" crate = %r," % (":" + s["crate"])) + print(" )") print() - print(" # flapc, the Flap-DSL -> interpreter-assembly compiler. Its workspace is") - print(" # `exclude`d from the root one and has its own lock with exactly 3") - print(" # packages (flapc, in-tree bytecode_def, and smallvec from crates.io") - print(" # pinned =1.15.1 -- the same version AND checksum the big workspace pins,") - print(" # so it is the same fetch rule, not a second one).") - print(" cargo_binary(") - print(" name = \"flapc\",") - print(" bin = \"flapc\",") - print(" crate = \"flapc\",") - print(" crates = CARGO_CRATE_FILES,") - print(" manifest = \"Libraries/LibJS/Flap/Cargo.toml\",") - print(" srcs = native.glob(%r, allow_empty = False)," % FLAPC_SRC_GLOBS) - print(" sysroot = \":rust_sysroot\",") - print(" )") + print(" # The BINARY crates: cargo `--bin` targets, declared in CMake with") + print(" # build_rust_binary() rather than import_rust_crate(). Two of them, and") + print(" # they are two different shapes:") + print(" #") + # The Flap lock's contents are DESCRIBED, not restated. The previous version + # of this comment said "exactly 3 packages (flapc, in-tree bytecode_def, and + # smallvec ... =1.15.1)", all four facts hand-copied -- and upstream's + # a32d9c9f deleted bytecode_def, so the generated file confidently documented + # a package that no longer exists. Same class as the SYSTEM_LIBS and glib + # include-root drift: a fact worth stating in generated output is a fact worth + # reading from the input. + flap_pkgs = lock_packages(FLAP_LOCK) + flap_reg, flap_ws = split_packages(flap_pkgs) + shared = [(n, v) for (n, v, c) in flap_reg + if (n, v, c) in lock_packages("Cargo.lock")] + print(" # flapc a pure build TOOL -- Bazel runs it in a genrule to") + print(" # produce the interpreter assembly. Its workspace is") + print(" # `exclude`d from the root one and has its own lock") + print(" # with %d package%s: %s" % ( + len(flap_pkgs), "" if len(flap_pkgs) == 1 else "s", + ", ".join("%s %s%s" % (n, v, " (in-tree)" if (n, v) in flap_ws else "") + for (n, v) in [(n, v) for (n, v, _) in flap_pkgs]))) + if shared: + print(" # %s %s %s pinned at the same version AND" + % (", ".join(n for (n, _) in shared), + "is" if len(shared) == 1 else "are", + "also" if len(shared) == 1 else "also")) + print(" # checksum as the big workspace, so %s the same" + % ("it is" if len(shared) == 1 else "they are")) + print(" # fetch rule%s, not a second one." + % ("" if len(shared) == 1 else "s")) + print(" # cranelift-compiler a RUNTIME tool -- LibWasm spawns it to AOT-compile") + print(" # WebAssembly, and its build script ALSO emits the") + print(" # CraneliftFFI.h that LibWasm's CraneliftBridge.cpp") + print(" # includes. It is a root-workspace member, so its") + print(" # source set is the shared crate_srcs.") + for b in binaries: + member = os.path.dirname(b["manifest"]) + in_root_ws = member in root_workspace_members() + extra = CRATE_EXTRA_INPUTS.get(b["crate"], []) + print(" cargo_binary(") + print(" name = %r," % b["bin"]) + print(" bin = %r," % b["bin"]) + print(" crate = %r," % b["crate"]) + print(" crates = CARGO_CRATE_FILES,") + print(" manifest = %r," % b["manifest"]) + if b["features"]: + print(" # build_rust_binary() takes FEATURES too, and this one does:") + print(" # the same crate builds a staticlib and this binary, and only") + print(" # the features distinguish what each gets.") + print(" crate_features = %r," % b["features"]) + if b["ffi_headers"]: + print(" # This binary crate's build script runs cbindgen too, so the") + print(" # header is a DECLARED output -- Bazel deletes what nothing") + print(" # declares, and LibWasm includes this one bare.") + print(" ffi_headers = %r," % b["ffi_headers"]) + print(" ffi_prefix = %r," % b["ffi_prefix"]) + print(" ffi_bare_include = %r," % b["ffi_bare_include"]) + if in_root_ws: + print(" # A member of the ROOT cargo workspace (Cargo.toml lists it), so") + print(" # cargo resolves the whole workspace whichever crate you ask for") + print(" # and the source set is the shared one -- same over-declaration,") + print(" # same reason, as the staticlib crates.") + srcs = "crate_srcs" + if extra: + srcs += " + %r" % extra + print(" srcs = %s," % srcs) + else: + print(" # Its OWN workspace (`exclude`d from the root one), so its source") + print(" # set is its own subtree plus what it include_str!s from above") + print(" # it -- both derived, the subtree from Cargo.toml's `exclude`") + print(" # and the rest by scanning for include_str!.") + print(" srcs = native.glob(%r, exclude = %r, allow_empty = False)," + % (flapc_src_globs(), crate_src_glob_excludes(_excluded_dirs()))) + print(" sysroot = \":rust_sysroot\",") + print(" )") + if b["ffi_headers"]: + print() + print(" # The consumable target, IDENTICAL in shape to a staticlib crate's:") + print(" # this crate's generated header to include. There is no archive to") + print(" # link -- what C++ consumes from a binary crate is the EXECUTABLE,") + print(" # spawned at run time -- so cargo_lib yields a headers-only CcInfo.") + print(" cargo_lib(") + print(" name = %r," % (b["crate"] + "_lib")) + print(" crate = %r," % (":" + b["bin"])) + print(" )") + if b["ffi_bare_include"]: + print(" cargo_bare_include(") + print(" name = %r," % (b["crate"] + "_bare_include")) + print(" crate = %r," % (":" + b["bin"])) + print(" )") def emit_extension(crates): @@ -674,10 +1146,27 @@ def report(crates): if missing: print(" declared by CMake but never written: %s" % ", ".join(missing)) rc = 1 - for crate in sorted(set(FFI_HEADERS_OBSERVED) - - {c["crate"] for c in parse_cmake_crates()}): - print("%-30s observed but no import_rust_crate found -- did a crate get " - "renamed or moved?" % crate) + print() + # The binary crates are reported separately, because the two kinds fail + # differently: a missing staticlib crate is a link error, a missing binary + # crate is a header that never appears (CraneliftFFI.h) plus a tool that has + # to be found at run time. + for b in binary_specs(): + print("%-30s bin=%-20s manifest=%s" + % (b["crate"], b["bin"], b["manifest"])) + if b["emits_ffi"] and not b["ffi_headers"]: + print(" FFI_OUTPUT_DIR is set at the call site but no header is " + "listed in FFI_HEADERS_OBSERVED -- run the reference cargo " + "build with FFI_OUTPUT_DIR set and add what it wrote") + rc = 1 + for h in b["ffi_headers"]: + print(" emits %s (bare-included: %s)" + % (h, "yes" if b["ffi_bare_include"] else "no")) + known = ({c["crate"] for c in parse_cmake_crates()} | + {b["crate"] for b in parse_cmake_binaries()}) + for crate in sorted(set(FFI_HEADERS_OBSERVED) - known): + print("%-30s observed but no import_rust_crate/build_rust_binary found " + "-- did a crate get renamed or moved?" % crate) rc = 1 return rc @@ -707,7 +1196,7 @@ def check(dirpath): {"--crates": lambda: emit_crates(crates), "--index": lambda: emit_index(crates, specs), "--extension": lambda: emit_extension(crates), - "--ring": lambda: emit_ring(crates, specs)}[flag]() + "--ring": lambda: emit_ring(crates, specs, binary_specs())}[flag]() finally: sys.stdout = real path = os.path.join(dirpath, fn) @@ -737,7 +1226,7 @@ def main(): elif "--extension" in argv: emit_extension(crates) elif "--ring" in argv: - emit_ring(crates, crate_specs()) + emit_ring(crates, crate_specs(), binary_specs()) elif "--use-repo" in argv: emit_use_repo(crates) else: diff --git a/examples/ladybird/workspace/Meta/emit_codegen_bazel.py b/examples/ladybird/workspace/Meta/emit_codegen_bazel.py index fdc5efe..8e2f6da 100644 --- a/examples/ladybird/workspace/Meta/emit_codegen_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_codegen_bazel.py @@ -11,7 +11,13 @@ import re, os, sys, shlex ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -FULL = ROOT + "/Build/full" +# The reference CMake build tree the emitter reads. An env var, not a +# hardcoded "Build/full": a repin needs a SECOND reference build side by side +# with the old one (you cannot delete the tree you are still diffing against), +# and CMake bakes the build dir's absolute path into build.ninja -- so +# "just rename the directory afterwards" corrupts it. Found by doing exactly +# that during the 71fb301a repin. +FULL = os.environ.get("LADYBIRD_BUILD_DIR") or (ROOT + "/Build/full") def parse(lib_build_dir): txt = open(FULL + "/build.ninja").read() diff --git a/examples/ladybird/workspace/Meta/emit_export_headers_bazel.py b/examples/ladybird/workspace/Meta/emit_export_headers_bazel.py new file mode 100644 index 0000000..923e3d5 --- /dev/null +++ b/examples/ladybird/workspace/Meta/emit_export_headers_bazel.py @@ -0,0 +1,421 @@ +#!/usr/bin/env python3 +"""Emit Bazel genrules for CMake's `generate_export_header` output. + +WHY THIS EXISTS +--------------- +Ladybird's `ladybird_lib()` calls `ladybird_generate_export_header(name fs_name)` +(Meta/CMake/targets.cmake), which is CMake's `GenerateExportHeader` module with +`EXPORT_MACRO_NAME _API` and `EXPORT_FILE_NAME "Export.h"`. That +writes `Build//Libraries//Export.h` -- a *configure*-time artifact of +the reference build. + +Until now Bazel did not generate these 15 headers at all: it globbed them out of +`Build/full/Libraries/**` via the `//Build/full/Libraries:generated_lib_headers` +shim. That made `bazel build //:ladybird` silently depend on a completed CMake +build -- and because the shim uses `glob(..., allow_empty = True)`, a fresh +checkout produced NO error from the glob, just `fatal error: LibXML/Export.h: No +such file or directory` ~1,600 actions into the build. Generating them here is +what lets a clean `git clone` build. + +WHY A TEMPLATE IS THE RIGHT ANSWER HERE +--------------------------------------- +Reproducing another build system's output by re-implementing its generator is +normally the wrong move -- it is a fork that silently drifts. Two things make this +the exception, and both were checked rather than assumed: + +1. **The output is a pure function of one token.** All 15 checked-in headers + normalize to a SINGLE byte-identical template under substitution of + (``, ``, `_EXPORTS`), all three derived from the library name. + Verified by normalizing each of the 15 and comparing: 1 distinct template. +2. **The alternative is worse.** The only other faithful option is running CMake, + which is the dependency being removed. + +So the risk is a future CMake version changing the template. That is caught, not +hoped about: `--check` byte-compares this emitter's output for every library +against the reference tree's copy, and the parity harness diffs generated output +tree-wide. If CMake's template changes, `--check` fails loudly instead of the +build succeeding with a subtly wrong header. + +The template is CMake's own (Modules/GenerateExportHeader.cmake) for the +GNU/Clang, non-Windows case -- the only case Ladybird's Linux build exercises. The +`#if 0 /* DEFINE_NO_DEPRECATED */` block and the leading blank line are +reproduced deliberately: byte parity includes the parts that look like noise. + +ALSO: THE TWO `configure_file` HEADERS +-------------------------------------- +`AK/CMakeLists.txt` runs `configure_file(Debug.h.in Debug.h @ONLY)` and the same +for `Backtrace.h.in`. These were the last two headers Bazel read out of +`Build/full/AK/`, and they are a different shape from Export.h: the TEMPLATE is +checked into the source tree, so nothing is being re-implemented here except +CMake's substitution rules (`#cmakedefine01` -> `#define X 0|1`, `#cmakedefine` +-> `#define X` or a comment, `@VAR@` -> value). + +`Debug.h` is 79 `#cmakedefine01` lines and every one of them is `0` in the +reference build -- they are opt-in debug spew, off unless someone passes +`-DFOO_DEBUG=ON`. So an unconfigured emit is the faithful answer. + +`Backtrace.h` is NOT a template substitution, it is a HOST PROBE: +`find_package(Backtrace)` decides whether `execinfo.h` exists and what it is +called. Pasting this machine's answer (`execinfo.h`) into a checked-in file is +exactly the "test the variable, not the value" mistake from the Dolphin case +study -- it would survive every check on glibc and break on musl, where backtrace +lives elsewhere or not at all. So the genrule ASKS the question at build time by +compiling a probe, the same question CMake asks, and emits whichever answer this +host gives. + +Usage: + python3 Meta/emit_export_headers_bazel.py # emit export_headers.bzl + python3 Meta/emit_export_headers_bazel.py --check # byte-verify vs reference +""" + +import argparse +import os +import re +import sys + +# The libraries CMake generates an Export.h for: those built with +# LADYBIRD_LIB_EXPLICIT_SYMBOL_EXPORT, so that ladybird_lib() ran +# ladybird_generate_export_header(). Kept as an explicit list (not a glob over +# Libraries/) precisely so a library gaining or losing an export header shows up +# as a --check failure rather than silently changing the build. +EXPORT_LIBS = [ + "LibCore", + "LibDNS", + "LibDatabase", + "LibDevTools", + "LibGC", + "LibJS", + "LibMedia", + "LibRegex", + "LibSync", + "LibTest", + "LibTextCodec", + "LibWasm", + "LibWeb", + "LibWebView", + "LibXML", +] + +# CMake's GenerateExportHeader output for GCC/Clang on non-Windows. +# @API@ = EXPORT_MACRO_NAME (upper(fs_name) + "_API") +# @PRE@ = upper(target name), used for the NO_EXPORT/DEPRECATED macros +# @EXP@ = "_EXPORTS", the macro CMake defines while building the target +TEMPLATE = """ +#ifndef @API@_H +#define @API@_H + +#ifdef @PRE@_STATIC_DEFINE +# define @API@ +# define @PRE@_NO_EXPORT +#else +# ifndef @API@ +# ifdef @EXP@ + /* We are building this library */ +# define @API@ __attribute__((visibility("default"))) +# else + /* We are using this library */ +# define @API@ __attribute__((visibility("default"))) +# endif +# endif + +# ifndef @PRE@_NO_EXPORT +# define @PRE@_NO_EXPORT __attribute__((visibility("hidden"))) +# endif +#endif + +#ifndef @PRE@_DEPRECATED +# define @PRE@_DEPRECATED __attribute__ ((__deprecated__)) +#endif + +#ifndef @PRE@_DEPRECATED_EXPORT +# define @PRE@_DEPRECATED_EXPORT @API@ @PRE@_DEPRECATED +#endif + +#ifndef @PRE@_DEPRECATED_NO_EXPORT +# define @PRE@_DEPRECATED_NO_EXPORT @PRE@_NO_EXPORT @PRE@_DEPRECATED +#endif + +/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */ +#if 0 /* DEFINE_NO_DEPRECATED */ +# ifndef @PRE@_NO_DEPRECATED +# define @PRE@_NO_DEPRECATED +# endif +#endif + +#endif /* @API@_H */ +""" + + +def tokens(lib): + """The three substitution tokens for a library, derived exactly as + Meta/CMake/targets.cmake derives them: fs_name is the library name minus its + "Lib" prefix, and EXPORT_MACRO_NAME is upper(fs_name) + "_API".""" + fs_name = lib[3:] if lib.startswith("Lib") else lib + return ("%s_API" % fs_name.upper(), lib.upper(), "%s_EXPORTS" % lib) + + +def render(lib): + api, pre, exp = tokens(lib) + # Substitute @EXP@ first: it is the only token whose replacement text + # ("LibGC_EXPORTS") contains a substring that a later pattern could match. + # Each replace() runs over the whole string, so an earlier substitution's + # OUTPUT is visible to a later one -- e.g. replacing @PRE@ with "LIBGC" then + # looking for "GC_API" is fine here (disjoint), but the ordering is load + # bearing in general and must not be permuted casually. + return TEMPLATE.replace("@EXP@", exp).replace("@API@", api).replace("@PRE@", pre) + + +def configure_file(text, defines): + """Apply CMake's `configure_file(... @ONLY)` substitution rules. + + Only the three directives Ladybird's two templates actually use, because a + partial re-implementation that silently ignores an unknown directive is how + you get a header that looks configured and is not. Anything unrecognized is + left alone and reported by `--check` as a byte mismatch. + + * `#cmakedefine01 X` -> `#define X 1` if X is truthy else `#define X 0` + * `#cmakedefine X` -> `#define X` if defined, else `/* #undef X */` + * `@VAR@` -> its value (@ONLY means ONLY @VAR@, not ${VAR}) + """ + def sub01(m): + pad, name = m.group(1), m.group(2) + return "#%sdefine %s %s" % (pad, name, "1" if defines.get(name) else "0") + + def subdef(m): + pad, name = m.group(1), m.group(2) + if defines.get(name): + return "#%sdefine %s" % (pad, name) + return "#%s/* #undef %s */" % (pad, name) + + # The `#` sits at column 0 and the indentation comes AFTER it + # (`# cmakedefine01 FOO`), which is how these templates are written -- so + # the padding to preserve is between the hash and the directive, not before + # the hash. Two earlier attempts got this wrong: `(\s*)#cmakedefine01` never + # matched, and --check caught it both times rather than emitting a header with + # the directive left verbatim. + text = re.sub(r"(?m)^#([ \t]*)cmakedefine01 (\w+)[ \t]*$", sub01, text) + text = re.sub(r"(?m)^#([ \t]*)cmakedefine (\w+)[ \t]*$", subdef, text) + text = re.sub(r"@(\w+)@", lambda m: str(defines.get(m.group(1), "")), text) + return text + + +# Debug.h: every `#cmakedefine01 *_DEBUG` is OFF in a default configure. Passing +# an empty define map is therefore the faithful reproduction, and --check proves +# it against the reference tree rather than trusting this comment. +DEBUG_H_DEFINES = {} + +# Backtrace.h is deliberately NOT emitted from a define map -- see the module +# docstring. The genrule below compiles a probe for and substitutes +# whatever THIS host answers, so the header is a question, not a pasted value. +BACKTRACE_PROBE_CMD = r""" +set -e +tmp=$$(mktemp -d) +printf '#include \nint main(){void*b[1];backtrace(b,1);return 0;}\n' > $$tmp/p.c +if $${CC:-cc} -o $$tmp/p $$tmp/p.c 2>/dev/null; then + found=1 +else + found=0 +fi +rm -rf $$tmp +if [ "$$found" = 1 ]; then + sed -e 's|^#cmakedefine Backtrace_FOUND$$|#define Backtrace_FOUND|' \ + -e 's|@Backtrace_HEADER@|execinfo.h|' $< > $@ +else + sed -e 's|^#cmakedefine Backtrace_FOUND$$|/* #undef Backtrace_FOUND */|' \ + -e 's|@Backtrace_HEADER@||' $< > $@ +fi +""" + + +# Libraries/LibWeb is its own Bazel PACKAGE, so the root package cannot declare +# an output inside it ("Label '//:Libraries/LibWeb/Export.h' is invalid because +# 'Libraries/LibWeb' is a subpackage"). Its Export.h is emitted into LibWeb's own +# package instead, via --libweb. Any other library that gains a BUILD.bazel will +# hit the same wall and belongs in this set. +SUBPACKAGE_LIBS = {"LibWeb"} + + +def root_libs(): + return [l for l in EXPORT_LIBS if l not in SUBPACKAGE_LIBS] + + +def heredoc(body): + """A shell heredoc that writes `body` EXACTLY, with no added trailing newline. + + `cat > $@ <<'EOF'\n\nEOF` appends a newline of its own, so a body that + already ends in "\n" gains a second one -- 1 byte of drift that `--check` + (which compares render() to the reference) could never see, because it is + introduced by the emitted SHELL, not by render(). It was caught only by + byte-comparing the BUILT artifact. Strip one trailing newline before wrapping, + since the heredoc puts it back. + """ + if body.endswith("\n"): + body = body[:-1] + esc = body.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + return "cat > $@ <<'LADYBIRD_EOF'\\n%s\\nLADYBIRD_EOF\\n" % esc + + +def emit_libweb(): + """The LibWeb-package half: its own Export.h genrule + header target.""" + lib = "LibWeb" + print("# AUTO-GENERATED by Meta/emit_export_headers_bazel.py --libweb — do not edit.") + print("# LibWeb's generate_export_header output. It lives here rather than in the") + print("# root package because Libraries/LibWeb is its own Bazel package, so only") + print("# this package may declare an output inside it.") + print('load("@rules_cc//cc:defs.bzl", "cc_library")') + print() + print("def libweb_export_header():") + # Emitted at genroot/LibWeb/Export.h, NOT at Export.h. All 289 consumers spell + # it , so the include root has to be a directory CONTAINING a + # "LibWeb" dir. Using the package dir itself would need includes=['../..'], + # which Bazel rejects outright ("resolves to the workspace root, which would + # allow this rule and all of its transitive dependents to include any file in + # your workspace") -- and it is right to: that would put the entire repo on + # every dependent's include path. A private genroot/ subdir keeps the exported + # path exactly while the include root stays inside this + # package. + print(" native.genrule(") + print(" name = 'gen_%s_Export_h'," % lib) + print(" outs = ['genroot/LibWeb/Export.h'],") + print(" cmd = \"%s\"," % heredoc(render(lib))) + print(" )") + print(" cc_library(") + print(" name = 'generated_export_header',") + print(" hdrs = ['genroot/LibWeb/Export.h'],") + print(" includes = ['genroot'],") + print(" )") + + +def emit(): + print("# AUTO-GENERATED by Meta/emit_export_headers_bazel.py — do not edit.") + print("# CMake's generate_export_header() output for the %d libraries that" + % len(EXPORT_LIBS)) + print("# opt into explicit symbol export, emitted as genrules so building the") + print("# browser needs no CMake build. Byte-verified against the reference") + print("# tree by `Meta/emit_export_headers_bazel.py --check Build/full`.") + print('load("@rules_cc//cc:defs.bzl", "cc_library")') + print() + print("def export_headers():") + for lib in root_libs(): + # A quoted heredoc ('LADYBIRD_EOF') so the shell interprets NOTHING in + # the body: it contains #, $, quotes, parentheses and backslashes. + print(" native.genrule(") + print(" name = 'gen_%s_Export_h'," % lib) + print(" outs = ['Libraries/%s/Export.h']," % lib) + print(" cmd = \"%s\"," % heredoc(render(lib))) + print(" )") + print() + # AK/Debug.h -- configure_file over a checked-in template, all flags off. + # Emitted by substituting here rather than shelling sed 79 times. + print(" # AK/Debug.h: configure_file(Debug.h.in) with every *_DEBUG off,") + print(" # which is what a default configure produces (--check verifies it).") + print(" native.genrule(") + print(" name = 'gen_AK_Debug_h',") + print(" srcs = ['AK/Debug.h.in'],") + print(" outs = ['genroot/AK/Debug.h'],") + debug_in = os.path.join("AK", "Debug.h.in") + if os.path.exists(debug_in): + with open(debug_in) as f: + body = configure_file(f.read(), DEBUG_H_DEFINES) + print(" cmd = \"%s\"," % heredoc(body)) + else: + print(" cmd = \"# AK/Debug.h.in not found at emit time\",") + print(" )") + # AK/Backtrace.h -- a host probe, asked at build time. See module docstring. + print(" # AK/Backtrace.h: find_package(Backtrace) is a HOST QUESTION, so the") + print(" # genrule compiles a probe instead of baking in this machine's answer.") + print(" native.genrule(") + print(" name = 'gen_AK_Backtrace_h',") + print(" srcs = ['AK/Backtrace.h.in'],") + print(" outs = ['genroot/AK/Backtrace.h'],") + print(" cmd = %r," % BACKTRACE_PROBE_CMD) + print(" )") + print() + # One header root exposing all 15 as /Export.h, mirroring the shim it + # replaces (//Build/full/Libraries:generated_lib_headers). + print(" cc_library(") + print(" name = 'generated_export_headers',") + print(" hdrs = [%s]," % ", ".join( + "'Libraries/%s/Export.h'" % lib for lib in root_libs())) + print(" includes = ['Libraries'],") + print(" )") + # AK/*.h arrive under genroot/ so resolves, mirroring the + # ak_gen_headers genrule in BUILD.bazel that copied them out of Build/full. + print(" cc_library(") + print(" name = 'generated_ak_headers',") + print(" hdrs = ['genroot/AK/Debug.h', 'genroot/AK/Backtrace.h'],") + print(" includes = ['genroot'],") + print(" )") + + +def check(tree): + """Byte-compare this emitter's output against the reference CMake tree.""" + bad = 0 + for lib in EXPORT_LIBS: + ref = os.path.join(tree, "Libraries", lib, "Export.h") + if not os.path.exists(ref): + print("MISSING reference: %s" % ref) + bad += 1 + continue + with open(ref) as f: + want = f.read() + got = render(lib) + if got != want: + print("MISMATCH %s" % lib) + # Show the first differing line so template drift is diagnosable. + for i, (a, b) in enumerate(zip(got.splitlines(), want.splitlines())): + if a != b: + print(" line %d: emitted %r" % (i + 1, a)) + print(" cmake %r" % b) + break + bad += 1 + print("%d/%d export headers byte-identical to %s" + % (len(EXPORT_LIBS) - bad, len(EXPORT_LIBS), tree)) + + # AK/Debug.h: the configure_file path, checked the same way. (Backtrace.h is + # deliberately excluded: its content is a function of the HOST, so comparing + # it to this machine's reference tree would only ever re-confirm this machine. + # The probe is verified by building, not by diffing.) + dbg_in, dbg_ref = "AK/Debug.h.in", os.path.join(tree, "AK", "Debug.h") + if os.path.exists(dbg_in) and os.path.exists(dbg_ref): + with open(dbg_in) as f: + got = configure_file(f.read(), DEBUG_H_DEFINES) + with open(dbg_ref) as f: + want = f.read() + if got == want: + print("AK/Debug.h byte-identical to %s" % dbg_ref) + else: + bad += 1 + print("MISMATCH AK/Debug.h") + for i, (a, b) in enumerate(zip(got.splitlines(), want.splitlines())): + if a != b: + print(" line %d: emitted %r" % (i + 1, a)) + print(" cmake %r" % b) + break + else: + print(" (differing line count: %d emitted vs %d cmake)" + % (len(got.splitlines()), len(want.splitlines()))) + else: + print("SKIP AK/Debug.h (template or reference missing)") + return 1 if bad else 0 + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--libweb", action="store_true", + help="emit the Libraries/LibWeb package half instead") + ap.add_argument("--check", metavar="BUILD_DIR", + help="byte-compare against a reference CMake build tree " + "(e.g. Build/full)") + args = ap.parse_args() + if args.check: + return check(args.check) + if args.libweb: + emit_libweb() + return 0 + emit() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py index 963e5d8..2b8db4d 100644 --- a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py @@ -16,22 +16,44 @@ import json, os, re, sys ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -MODEL = os.path.join(ROOT, "model.cmake.full.json") +MODEL = os.environ.get("LADYBIRD_MODEL") or os.path.join(ROOT, "model.cmake.full.json") PKG_PREFIX = "Libraries/LibWeb/" -GEN_PREFIX = "Build/full/Libraries/LibWeb/" +# see emit_build_bazel.BUILD_REL: the reference build dir is an env var, because a +# repin needs two reference trees side by side and build.ninja cannot be moved. +BUILD_REL = (os.environ.get("LADYBIRD_BUILD_REL") or "Build/full").strip("/") + "/" +GEN_PREFIX = BUILD_REL + "Libraries/LibWeb/" VCPKG = "//Meta/vcpkg" # See emit_build_bazel.py: the crates are Bazel-built now, and each crate is ONE # target (//:_lib) carrying its own archive and its own generated FFI # headers -- one for one with CMake's per-library edge, so LibWeb links the four -# crates it uses and nothing else. -RUST_LIB_FMT = "//:%s_lib" - -GLOBAL_DEFINES = { - "USE_VULKAN=1", "ENABLE_COMPILETIME_FORMAT_CHECK", "USE_FONTCONFIG=1", - "_FORTIFY_SOURCE=3", "USE_VULKAN_DMABUF_IMAGES=1", "_FILE_OFFSET_BITS=64", - "NDEBUG", -} -SYSTEM_LIBS = {"dl", "m", "pthread", "vulkan"} +# crates it uses and nothing else. Plus, for a crate whose header LibWeb includes +# with no directory, an implementation_deps edge -- IMPORTED from +# emit_build_bazel rather than reimplemented, so the two emitters cannot disagree +# about which crates those are (none of LibWeb's four today; the mechanism is here +# because "none today" is a measurement, not an invariant). +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import emit_build_bazel +from emit_build_bazel import rust_dep_labels +# The crate directories inside THIS package, and the non-Rust build-script inputs +# that live here, both derived by emit_cargo_bazel from Cargo.toml and the +# reference build's depfiles. Imported rather than restated for one concrete +# reason: this filegroup used to hardcode `CSS/Rust/**` and `Layout/Rust/**` with +# allow_empty = False, upstream consolidated both crates into LibWeb/Rust, and the +# glob then matched nothing -- which is a LOADING-time failure, so no target could +# report it. The root package's glob over the same crates is one list; so is this. +import emit_cargo_bazel + +# IMPORTED, not restated. Both of these describe the BUILD, not this target: a +# define is global because .bazelrc sets it for every TU, and a lib is a "system +# lib" because no vcpkg port provides it. Two copies of one global fact is the +# shape that broke this repin four times over -- and these two had ALREADY +# diverged: the 71fb301a repin added glib/gio/gobject/xkbcommon to +# emit_build_bazel's SYSTEM_LIBS (upstream's new pkg_check_modules(GIO)) and this +# copy kept the old four. Harmless only because LibWeb happens not to depend on +# glib; "happens not to" is not a property worth relying on, and the failure it +# would produce is an UNKNOWN dep that silently drops a link input. +GLOBAL_DEFINES = emit_build_bazel.GLOBAL_DEFINES +SYSTEM_LIBS = emit_build_bazel.SYSTEM_LIBS # LibWeb exports extern "C" FFI that the prebuilt Rust archive consumes and also # consumes it back (a static-archive <-> static-archive cycle GNU ld cannot @@ -40,50 +62,63 @@ # emit_build_bazel.py's ALWAYSLINK_LIBS. ALWAYSLINK = True -PRELUDE = '''load("@rules_cc//cc:defs.bzl", "cc_library") + +def prelude(): + """The head of the BUILD file: loads, package(), codegen macros, filegroup. + + A function rather than a constant because the rust_crate_srcs glob is + DERIVED (see the emit_cargo_bazel import above), and a constant is exactly + what let it name two directories that upstream had deleted. + """ + pkg = PKG_PREFIX.rstrip("/") + globs = emit_cargo_bazel.package_crate_globs(pkg) + excludes = emit_cargo_bazel.crate_src_glob_excludes( + [g[:-len("/**")] for g in globs]) + extra = emit_cargo_bazel.PACKAGE_EXTRA_INPUTS.get(pkg, []) + return '''load("@rules_cc//cc:defs.bzl", "cc_library") load(":codegen.bzl", "libweb_codegen", "libweb_bindings_codegen") load(":generated_srcs.bzl", "LIBWEB_GENERATED_SRCS", "LIBWEB_GENERATED_HDRS") +load(":export_header.bzl", "libweb_export_header") package(default_visibility = ["//visibility:public"]) +# LibWeb's generate_export_header output. It is generated HERE rather than in the +# root package because Bazel include dirs cannot escape a package: LibWeb is its +# own package, so its Export.h has to be an output of this package (at +# genroot/LibWeb/Export.h, with includes=["genroot"] -- includes=["../.."] is +# rejected by Bazel, and rightly so). Emitted by +# Meta/emit_export_headers_bazel.py --libweb. +libweb_export_header() + libweb_codegen() libweb_bindings_codegen() -# The four Rust crates that live INSIDE this package (LibWeb/Rust, -# LibWeb/CSS/Rust, LibWeb/Layout/Rust, LibWeb/ContentBlocker/Rust, plus -# HTML/Parser/Rust), exposed so the root package's cargo_ring() can declare them -# as cargo inputs. They are one cargo WORKSPACE with the crates at the repo root, -# but Bazel packages cut across it: glob() is package-relative, so the root -# package cannot see files under Libraries/LibWeb/ at all. Hence a filegroup on -# this side of the boundary rather than a glob on that side -- the alternative -# (making the root package own these files) would mean deleting this package. +# The Rust crates that live INSIDE this package, exposed so the root package's +# cargo_ring() can declare them as cargo inputs. They are one cargo WORKSPACE +# with the crates at the repo root, but Bazel packages cut across it: glob() is +# package-relative, so the root package cannot see files under Libraries/LibWeb/ +# at all. Hence a filegroup on this side of the boundary rather than a glob on +# that side -- the alternative (making the root package own these files) would +# mean deleting this package. +# +# The patterns are DERIVED from Cargo.toml (Meta/emit_cargo_bazel.py's +# crate_dirs(), the same list the root package globs), not written down. They +# used to be written down, and that broke the whole build: upstream consolidated +# libweb_css_rust and libweb_layout_rust into libweb_rust, so `CSS/Rust/**` and +# `Layout/Rust/**` matched nothing, and an allow_empty = False glob that matches +# nothing fails at LOADING time -- before any target exists to blame. filegroup( name = "rust_crate_srcs", - srcs = glob([ - "Rust/**", - "CSS/Rust/**", - "Layout/Rust/**", - "ContentBlocker/Rust/**", - "HTML/Parser/Rust/**", - ], allow_empty = False) + [ - # Non-Rust build-script inputs that live here too: libweb_css_rust's - # build.rs GENERATES Rust from these CSS data files, and libweb_rust's - # reads the HTML name headers + Entities.json. Taken from the reference - # build's cargo depfiles, so the list is measured rather than predicted. - "CSS/Enums.json", - "CSS/Keywords.json", - "CSS/LogicalPropertyGroups.json", - "CSS/Properties.json", - "CSS/PseudoClasses.json", - "CSS/PseudoElementPropertyGroups.txt", - "CSS/PseudoElements.json", - "CSS/Units.json", - "HTML/AttributeNames.h", - "HTML/Parser/Entities.json", - "HTML/TagNames.h", + srcs = glob(%r, exclude = %r, allow_empty = False) + [ + # Non-Rust build-script inputs that live here too: libweb_rust's build.rs + # GENERATES Rust from the CSS data files and reads the HTML name headers + + # Entities.json. Taken from the reference build's cargo depfiles, so the + # list is measured rather than predicted. +%s ], ) -''' +''' % (globs, excludes, + "\n".join(" %r," % x for x in extra)) def global_flags(): @@ -99,8 +134,76 @@ def load(): def genrule_outputs(): + """Every file codegen.bzl declares as a genrule OUTPUT, package-relative. + + Parsed out of the `outs = [...]` lists specifically, not by scanning the + file for anything that looks like a path. The loose scan was the bug: a + genrule's `srcs` names checked-in headers too (gen_MediaControlsDOM reads + HTML/TagNames.h, HTML/AttributeNames.h, SVG/TagNames.h, + SVG/AttributeNames.h), so those four SOURCE headers were classified as + generated -- which puts them in the hdrs exclude= list, i.e. drops them + from the library's headers entirely and re-adds them as labels that no + genrule produces. It only ever worked because the stale checked-in list + was consulted instead of this function's answer. + """ bz = open(os.path.join(ROOT, "Libraries/LibWeb/codegen.bzl")).read() - return set(re.findall(r"'([A-Za-z0-9_/]+\.(?:cpp|h|cc))'", bz)) + # Drop comment lines: a commented-out out= entry is not an output. + bz = "\n".join(l for l in bz.splitlines() if not l.lstrip().startswith("#")) + outs = set() + for m in re.finditer(r"\bouts\s*=\s*(\[[^\]]*\])", bz, re.S): + outs |= set(re.findall(r"'([^']+)'", m.group(1))) + return outs + + +# The generated headers a consumer can #include. Everything a genrule declares +# as an output with a header extension: a header that is generated but absent +# from this list is not in the cc_library's hdrs, so any TU that includes it +# fails with "No such file or directory" pointing at the GENERATED file that +# does exist on disk -- exactly how the 71fb301a repin broke (upstream added +# Bindings/WindowGlobalMixin.h and four siblings; Bindings/Window.h includes +# it; LibWeb/DOM/Document.cpp failed). +HDR_EXTS = (".h", ".hh", ".hpp", ".inc", ".def") + + +def generated_srcs_bzl(t, gen_outs): + """Emit Libraries/LibWeb/generated_srcs.bzl -- both lists DERIVED. + + This file used to be a capture: it said AUTO-GENERATED at the top but no + emitter wrote it, so it froze one pin's answer and silently overrode the + derivable one. Both lists come from facts already in hand: + + * SRCS = the generated members of the CMake reference's compile list for + LibWeb (target_srcs under GEN_PREFIX), i.e. what CMake actually + compiles -- the same source of truth as the checked-in srcs. + * HDRS = every header-extension genrule output, i.e. what the codegen + actually produces. + + They are deliberately NOT the same set: a generated .cpp that CMake does + not compile (WebGL/GLFunctions.cpp) is emitted but absent from SRCS, and + HDRS is a superset of the headers that pair with a compiled .cpp. + """ + gen_srcs = sorted(s[len(GEN_PREFIX):] for s in target_srcs(t) + if s.startswith(GEN_PREFIX)) + for s in gen_srcs: + assert s in gen_outs, f"generated src not a genrule output: {s}" + gen_hdrs = sorted(o for o in gen_outs if o.endswith(HDR_EXTS)) + out = [ + "# AUTO-GENERATED by Meta/emit_libweb_bazel.py --generated-srcs.", + "# Do not edit: both lists are DERIVED (see generated_srcs_bzl there).", + "# Generated .cpp SRCS = the %d the CMake reference compiles into LibWeb" + % len(gen_srcs), + "# (a generated .cpp CMake does not compile is emitted but not listed).", + "LIBWEB_GENERATED_SRCS = [", + ] + out += [" %r," % s for s in gen_srcs] + out.append("]") + out.append("") + out.append("# All %d generated headers, so resolves" + " (includes=[\"..\"])." % len(gen_hdrs)) + out.append("LIBWEB_GENERATED_HDRS = [") + out += [" %r," % h for h in gen_hdrs] + out.append("]") + return "\n".join(out) + "\n" def vcpkg_available(): @@ -146,9 +249,9 @@ def target_flags(t): def target_private_includes(t): globalroots = {ROOT, ROOT + "/Libraries", ROOT + "/Services", - ROOT + "/Build/full", ROOT + "/Build/full/Libraries", - ROOT + "/Build/full/Services", - ROOT + "/Build/full/vcpkg_installed/x64-linux-dynamic/include"} + ROOT + "/" + BUILD_REL.rstrip("/"), ROOT + "/" + BUILD_REL + "Libraries", + ROOT + "/" + BUILD_REL + "Services", + ROOT + "/" + BUILD_REL + "vcpkg_installed/x64-linux-dynamic/include"} incs = [] for a in t["actions"]: if a["mnemonic"] != "CppCompile": @@ -182,6 +285,11 @@ def lagom_to_target(name, targets): def main(): targets = load() + if "--generated-srcs" in sys.argv[1:]: + # The second output of this emitter. Separate because BUILD.bazel LOADS + # it, so the two cannot be one stdout stream. + sys.stdout.write(generated_srcs_bzl(targets["LibWeb"], genrule_outputs())) + return so, ar = vcpkg_available() libs = {n: t for n, t in targets.items() if t.get("role") == "production" @@ -204,7 +312,7 @@ def main(): flags = target_flags(t) incs = target_private_includes(t) - deps, sysdeps, unknown = [], [], [] + deps, impl_deps, sysdeps, unknown = [], [], [], [] for d in t.get("deps", []): nm = d["name"] if not d.get("external"): @@ -217,7 +325,9 @@ def main(): deps.append("//:%s" % tgt) continue if nm.endswith("_rust"): - deps.append(RUST_LIB_FMT % nm) + rdeps, rimpl = rust_dep_labels(nm) + deps.extend(rdeps) + impl_deps.extend(rimpl) elif nm in so or nm in ar: deps.append(VCPKG + ":" + nm) elif nm in SYSTEM_LIBS: @@ -244,7 +354,7 @@ def main(): copt_toks.append("-I" + i) deps.append("//:all_source_headers") - out = [PRELUDE] + out = [prelude()] out.append(f"# === LibWeb ({t['kind']}, {len(all_srcs)} TU: " f"{len(checked_in)} checked-in + {len(all_srcs)-len(checked_in)} generated) ===") if unknown: @@ -270,6 +380,11 @@ def main(): out.append(" local_defines = %r," % defs) if copt_toks: out.append(" copts = [%s]," % ", ".join("%r" % x for x in copt_toks)) + # PRIVATE deps -- see rust_dep_labels() in emit_build_bazel.py for why an + # unprefixed FFI include dir must not propagate. + if impl_deps: + out.append(" implementation_deps = [%s]," % + ", ".join("%r" % x for x in sorted(set(impl_deps)))) out.append(" deps = [") for d in sorted(set(deps)): out.append(f" {d!r},") diff --git a/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py b/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py index 26cc40b..c4cab78 100644 --- a/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py @@ -46,7 +46,13 @@ import re, os, sys, shlex ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -FULL = ROOT + "/Build/full" +# The reference CMake build tree the emitter reads. An env var, not a +# hardcoded "Build/full": a repin needs a SECOND reference build side by side +# with the old one (you cannot delete the tree you are still diffing against), +# and CMake bakes the build dir's absolute path into build.ninja -- so +# "just rename the directory afterwards" corrupts it. Found by doing exactly +# that during the 71fb301a repin. +FULL = os.environ.get("LADYBIRD_BUILD_DIR") or (ROOT + "/Build/full") LIBWEB_CD = FULL + "/Libraries/LibWeb" VCPKG_PKG = "//:vcpkg_installed_exec" @@ -118,6 +124,75 @@ def parse_glslang(): return sorted(out, key=lambda d: d['out']) +def parse_bytecode(): + """upstream a32d9c9f's generate-libjs-bytecode: interpreter.flap -> Op.h + OpCodes.h. + + A THIRD self-built tool, and the one that broke Ulf's build. It replaced + Meta/Generators/generate_libjs_bytecode_def_derived.py (a Python generator this + emitter picked up for free, via the Meta/Generators/ pattern) with a second + binary of the flapc crate -- so the generator moved out of every bucket the + emitter knew about, and gen_Op simply disappeared from the output. Nothing said + so: an emitter that enumerates the generator KINDS it understands cannot notice + a kind going away, only the parity harness can, and only if the harness does not + silently bucket the command as something else (it did; see COVERED there). + + The ninja command is ` --input F --op-header X.tmp --opcodes-header Y.tmp + && copy_if_different ... && remove ...`; CMake writes .tmp files and copies them + so a byte-identical regeneration does not retrigger downstream compiles. Bazel + genrules are hermetic and declare their outputs, so the .tmp dance is dropped and + the tool writes $(location ...) directly -- verified byte-identical by the harness. + """ + for cmd, cd, _declared, _produced in _blocks(): + if not re.search(r'(?:^|&& )\S*bin/generate-libjs-bytecode\b', cmd): + continue + # Find the tool in the WHOLE argv, not in the first `&&` segment: CMake + # emits `cd && ... && copy_if_different ...`, so segment 0 is + # the cd and the tool is in segment 1. Locate by name, then stop at the + # next `&&` -- position is not something to assume here (the same mistake + # the glslang parser documents). + toks = shlex.split(cmd.replace('&&', ' && ')) + i = next(i for i, x in enumerate(toks) + if x.endswith('bin/generate-libjs-bytecode')) + argv = toks[i + 1:] + if '&&' in argv: + argv = argv[:argv.index('&&')] + def val(flag): + v = argv[argv.index(flag) + 1] + # CMake passes the outputs relative to the command's cd, and with a + # .tmp suffix it then copies away; both are CMake's business, not ours. + v = v[:-4] if v.endswith('.tmp') else v + return os.path.relpath(os.path.normpath( + v if v.startswith('/') else os.path.join(cd, v)), FULL) + return dict( + input=os.path.relpath(argv[argv.index('--input') + 1], ROOT), + op_header=val('--op-header'), + opcodes_header=val('--opcodes-header'), + ) + return None + + +def emit_bytecode(bc): + if not bc: + return [] + print(' # The LibJS bytecode headers, generated by a tool THIS BUILD MAKES') + print(' # (//:generate-libjs-bytecode, the flapc crate\'s second bin --') + print(' # cargo_ring.bzl). Upstream a32d9c9f made interpreter.flap the sole') + print(' # authority for the instruction set and deleted both Bytecode.def') + print(' # and the Python generator that read it, which is why the overlay') + print(' # pinned at f9e34731 failed on a newer checkout: its glob for') + print(' # Libraries/LibJS/BytecodeDef/** matched nothing.') + print(' native.genrule(') + print(' name = %r,' % 'gen_Op') + print(' srcs = [%r],' % bc['input']) + print(' outs = [%r, %r],' % (bc['op_header'], bc['opcodes_header'])) + print(' tools = ["//:generate-libjs-bytecode"],') + print(' cmd = "$(location //:generate-libjs-bytecode) --input $(location %s) ' + '--op-header $(location %s) --opcodes-header $(location %s)",' + % (bc['input'], bc['op_header'], bc['opcodes_header'])) + print(' )') + return [bc['op_header'], bc['opcodes_header']] + + def parse_flap(): """The two chained self-built-tool commands: layout.conf, then the .S. @@ -154,11 +229,19 @@ def parse_flap(): def _rel_full(p): return os.path.relpath(os.path.normpath( p if p.startswith('/') else os.path.join(cd, p)), FULL) + # --bytecode-def is OPTIONAL, and its absence is the normal case + # now: upstream a32d9c9f ("LibJS: Derive bytecodes from Flap + # handlers") made interpreter.flap the sole authority, deleted + # Bytecode/Bytecode.def and dropped the flag. Read it out of the + # argv rather than requiring it -- this KeyError'd on the 71fb301a + # repin, which is the emitter doing its job (a hardcoded flag list + # is a claim about upstream that upstream can falsify). flap = dict( arch=d['arch'], object_format=d['object-format'], out=_rel_full(d['output']), constants=_rel_full(d['constants']), - bytecode_def=os.path.relpath(d['bytecode-def'], ROOT), + bytecode_def=(os.path.relpath(d['bytecode-def'], ROOT) + if 'bytecode-def' in d else None), dsl=os.path.relpath(d['input'], ROOT), extra=extra, ) @@ -169,9 +252,28 @@ def _rel_full(p): # referenced as //pkg:path, not as a root-package-relative path. SUBPACKAGES = ("Libraries/LibWeb", "Meta") +# Generator inputs that are NOT in the repo: CMake downloads them at configure +# time into the build tree, so a Bazel-only clone has no file at the path the +# ninja command line names. Each maps to the label of a pinned fetch. +# +# There is exactly one, and it is the HSTS preload table: +# hsts_preload.cmake downloads Chromium's `main` -- unversioned -- so the path +# below only exists if someone ran a CMake configure, and its CONTENT depends on +# the day they ran it. hsts_preload.bzl pins a commit + sha256 downstream (the +# upstream unpinned fetch is filed as a bug we cannot fix from here), so the +# genrule takes the pinned file instead of the configure's leftovers. Mapping it +# here rather than post-editing codegen_root.bzl keeps the emitter the single +# source of truth for that file. +DOWNLOADED_INPUTS = { + 'Build/caches/HSTSPreload/transport_security_state_static.json': + '@hsts_preload_json//file', +} + def _label(rel): """Root-relative repo path -> a label the root package can reference.""" + if rel in DOWNLOADED_INPUTS: + return DOWNLOADED_INPUTS[rel] for pkg in SUBPACKAGES: if rel == pkg or rel.startswith(pkg + '/'): return '//%s:%s' % (pkg, os.path.relpath(rel, pkg)) @@ -330,17 +432,22 @@ def emit_flap(layout, flap): print(' cmd = "$(location //:generate_interpreter_layout) > $@",') print(' )') extra = (' ' + ' '.join(flap['extra'])) if flap['extra'] else '' + srcs = [flap['dsl']] + if flap['bytecode_def']: + srcs.append(flap['bytecode_def']) + srcs.append(':' + layout['out']) + bcd = (' --bytecode-def $(location %s)' % flap['bytecode_def'] + if flap['bytecode_def'] else '') print(' native.genrule(') print(' name = %r,' % 'gen_interpreter_asm') - print(' srcs = [%r, %r, %r],' - % (flap['dsl'], flap['bytecode_def'], ':' + layout['out'])) + print(' srcs = [%s],' % ', '.join(repr(s) for s in srcs)) print(' outs = [%r],' % flap['out']) print(' tools = [%r],' % FLAPC_TOOL) print(' cmd = "$(location %s) --arch %s --object-format %s ' - '--constants $(location %s) --bytecode-def $(location %s) ' + '--constants $(location %s)%s ' '--input $(location %s) --output $@%s",' % (FLAPC_TOOL, flap['arch'], flap['object_format'], - ':' + layout['out'], flap['bytecode_def'], flap['dsl'], extra)) + ':' + layout['out'], bcd, flap['dsl'], extra)) print(' )') return [flap['out']] @@ -357,12 +464,13 @@ def main(): rules = parse() shaders = parse_glslang() layout, flap = parse_flap() + bytecode = parse_bytecode() print('# AUTO-GENERATED by Meta/emit_root_codegen_bazel.py — do not edit.') # native.cc_library was removed from Bazel; the rule must be loaded. print('load("@rules_cc//cc:defs.bzl", "cc_library")') print('# %d Python-generator genrules for the root package,' % len(rules)) print('# %d glslang shader headers, %d self-built-tool genrules' - % (len(shaders), 2 if (layout and flap) else 0)) + % (len(shaders), (2 if (layout and flap) else 0) + (1 if bytecode else 0))) print('# (byte-parity: Meta/bazel_parity_harness.py).\n') print('def root_codegen():') seen = set() @@ -388,6 +496,7 @@ def main(): % (mk, script_ref, d['args'])) print(' )') all_outs.extend(d['outs']) + all_outs.extend(emit_bytecode(bytecode)) all_outs.extend(emit_flap(layout, flap)) emit_header_roots(all_outs) emit_glslang(shaders) diff --git a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py index 844ddfa..d2718f7 100644 --- a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py @@ -391,6 +391,425 @@ def load_capture(path): return out +# The host tools vcpkg fetches for ITSELF (cmake, ninja, ...), as opposed to the +# distfiles it fetches for ports. Values are (url, sha512, filename). +# +# WHY THIS IS NOT PART OF THE CAPTURE. The capture records what vcpkg actually +# downloaded on the capturing machine, and vcpkg does not download a tool it can +# already find: `vcpkg_find_acquire_program` probes the host first. My machine has +# /usr/bin/ninja at exactly 1.13.2 -- the version vcpkg wants -- so ninja was never +# fetched, never captured, and never pinned; a machine WITHOUT it got +# `distfile MISSING FROM INDEX ... ninja-linux.zip` followed by x-block-origin +# correctly refusing the network. cmake made it into the pin only by luck: the host +# cmake is 4.2.3 against the required 4.4.0, so that one WAS downloaded. +# +# So the capture is the wrong instrument for this class: what it observes depends on +# what the capturing machine happened to have installed. The right source is vcpkg's +# own tool metadata -- scripts/vcpkg-tools.json, versioned inside the checkout at the +# baseline commit, carrying url + sha512 + archive name for every tool on every +# platform. That is a PIN, not an observation, so it is complete regardless of what +# is installed anywhere. +# The tools this build's ports can actually ask for. vcpkg-tools.json also pins +# dotnet, node, powershell-core, azcopy, gsutil, coscli and nuget -- ~400 MB of +# downloads for a build that never invokes any of them (only the unrelated +# vbs-enclave-tooling-codegen port does, and it is not in this closure). Pinning +# those would trade one failure mode for a slower, larger version of the same +# hermeticity story, so the set is scoped and the scoping is stated: BUILD_TOOLS is +# what vcpkg needs to configure and build a port at all. +# +# cmake and ninja are needed by scripts/detect_compiler -- which runs before any +# port does, for every triplet -- so they are not optional for anyone. +BUILD_TOOLS = ("cmake", "ninja") + + +def tool_distfiles(triplet_os="linux", arches=(None, "x64", "amd64"), + want=BUILD_TOOLS, vcpkg=None): + """vcpkg's own host tools for one platform, read from its tool metadata. + + Keyed by sha512 like every other distfile, so these merge straight into the + index the asset-cache script resolves through. + """ + path = os.path.join(vcpkg or VCPKG, "scripts", "vcpkg-tools.json") + if not os.path.exists(path): + raise VcpkgUnavailable("no %s (needed for vcpkg's own tool pins)" % path) + with open(path) as f: + meta = json.load(f) + out = {} + for t in meta.get("tools", []): + if want is not None and t.get("name") not in want: + continue + if t.get("os") != triplet_os or t.get("arch") not in arches: + continue + url, sha = t.get("url"), t.get("sha512") + if not url or not sha: + continue # a tool vcpkg expects from the system, not a download + # vcpkg stores the archive under `archive` when it differs from the URL's + # basename (ninja-linux.zip -> ninja-linux-1.13.2.zip): the asset cache is + # asked for the name vcpkg will look for, so prefer it. + name = t.get("archive") or url.rsplit("/", 1)[-1] + out[sha.lower()] = (url, name, "vcpkg-tool:" + t.get("name", "?"), "tools.json") + return out + + +# The tool pins are COMMITTED next to the asset capture, in the same 3-column TSV +# format, and that is not redundancy -- it is what keeps the emitter's central +# promise true: the committed pin regenerates every Bazel file with no vcpkg +# checkout, no CMake and no network. Deriving the tools from +# scripts/vcpkg-tools.json at emit time would have quietly made a vcpkg checkout a +# requirement again (two tests caught exactly that). So the derivation is a +# separate, deliberate step -- `--capture-tools` -- and its output is reviewed and +# committed like any other pin. +TOOLS_TSV = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "vcpkg_tool_assets.tsv") + + +def load_tool_pins(path=None): + """The committed tool pins -> {sha512: (url, filename, port, source)}.""" + path = path or TOOLS_TSV + if not os.path.exists(path): + return {} + out = {} + with open(path) as f: + for line in f: + line = line.split("#", 1)[0].strip() + if not line: + continue + parts = line.split("\t") + if len(parts) != 3: + continue + url, sha, name = parts + if not re.fullmatch(r'[0-9a-fA-F]{128}', sha): + continue + out[sha.lower()] = (url, name, "vcpkg-tool", "tools.tsv") + return out + + +# --------------------------------------------------------------------------- +# The third class of input: tools vcpkg CANNOT download at all. +# +# Finding 38 pinned the tools vcpkg fetches for itself (cmake, ninja) after ninja +# went unpinned because the capturing machine already had it. The same blind spot +# has a strictly worse case behind it, and nasm is it: on Linux +# `vcpkg_find_acquire_program(NASM)` has NO download_urls -- the Windows branch +# has three URLs and a sha512, the Linux branch has an apt package name and +# nothing else. So there is no pin to add. vcpkg probes the host, does not find +# it, and stops: +# +# CMake Error at scripts/cmake/vcpkg_find_acquire_program.cmake:201 (message): +# Could not find nasm. Please install it via your package manager +# +# ~20 minutes into `bazel build //:vcpkg_installed`, from inside libvpx, naming a +# scratch path. Six ports in this closure ask for it (dav1d, ffmpeg, +# libjpeg-turbo, libvpx, openh264, openssl), and the machine this was developed on +# has nasm, perl, pkg-config and python3 all preinstalled -- which is exactly why +# neither the capture NOR the tools.json pin could ever have revealed this. +# +# These are HOST PREREQUISITES, and calling them that is the point: they are not a +# hermeticity gap we can close by pinning a URL, they are the current, honest +# boundary of the port. Two things follow, and both are implemented here: +# +# 1. The set is DERIVED from vcpkg's own scripts (which programs the closure's +# portfiles invoke, and which of those have no Linux download), not from a +# hand-written list that rots at the next baseline bump. +# 2. It is CHECKED BEFORE the build, all at once, by name, with the ports that +# need each one -- so a machine missing three of them learns all three in one +# second, instead of one per 20-minute build. +# +# The real fix -- pinning these as Bazel-fetched binaries so the build needs no +# host tools at all -- is the same open work as glslangValidator, and is filed +# rather than pretended. +HOST_TOOLS_TSV = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "vcpkg_host_tools.tsv") + + +def _acquire_script(program, vcpkg=None): + p = os.path.join(vcpkg or VCPKG, "scripts", "cmake", + "vcpkg_find_acquire_program(%s).cmake" % program) + return p if os.path.exists(p) else None + + +# A condition that selects Windows. `MSVC` counts: an +# `ASM_COMPILER_ID STREQUAL "MSVC"` branch is Windows-only in practice, and that +# branch is the ONLY thing in this closure that asks for CLANG -- read naively it +# reports clang as a Linux prerequisite, which it is not. +_WIN_TOKEN = re.compile( + r'\b(CMAKE_HOST_WIN32|VCPKG_HOST_IS_WINDOWS|VCPKG_TARGET_IS_WINDOWS|WIN32|MSVC|' + r'VCPKG_TARGET_IS_UWP|VCPKG_TARGET_IS_MINGW)\b') + + +def _classify(cond): + """'win' / 'notwin' / None for a CMake if-condition.""" + if not _WIN_TOKEN.search(cond): + return None + return "notwin" if re.match(r'\s*NOT\b', cond) else "win" + + +def strip_windows_blocks(text): + """CMake source -> only the lines reachable on a non-Windows host. + + Both halves of this analysis need it, which is why it is one function. When + reading an acquire-script, every download URL lives in the Windows branch, so + NOT skipping it is exactly the mistake that hides the gap. When reading a + portfile, a `vcpkg_find_acquire_program` call inside a Windows branch is not a + prerequisite here at all. + + `else()` is the subtle case and is handled rather than guessed: the else of + `if(NOT VCPKG_TARGET_IS_WINDOWS)` IS the Windows branch (dav1d's is written + exactly that way), so each if-chain remembers whether any of its conditions + was a negated Windows test and suppresses its else accordingly. + """ + out, stack = [], [] + for raw in text.splitlines(): + s = raw.strip() + m = re.match(r'if\s*\((.*)', s, re.S) + if m: + cls = _classify(m.group(1)) + stack.append({"suppressed": cls == "win", "saw_notwin": cls == "notwin"}) + continue + m = re.match(r'elseif\s*\((.*)', s, re.S) + if m and stack: + cls = _classify(m.group(1)) + stack[-1]["suppressed"] = cls == "win" + stack[-1]["saw_notwin"] = stack[-1]["saw_notwin"] or cls == "notwin" + continue + if re.match(r'else\s*\(', s) and stack: + stack[-1]["suppressed"] = stack[-1]["saw_notwin"] + continue + if re.match(r'endif\b', s): + if stack: + stack.pop() + continue + if not any(f["suppressed"] for f in stack): + out.append(raw) + return "\n".join(out) + + +def _has_non_windows_download(text): + """Does this acquire-script offer a download vcpkg can use on Linux? + + A tool with no such download is one vcpkg can only take from the host. + """ + body = strip_windows_blocks(text) + # `z_use_vcpkg_fetch` delegates to `vcpkg fetch`, which IS covered by the + # tools.json pin (that is how ninja is handled) -- so it is not a host + # prerequisite even though it sets no download_urls itself. + if "z_use_vcpkg_fetch" in body: + return True + return bool(re.search(r'set\s*\(\s*download_urls', body)) + + +def _port_cmake_files(port, vcpkg=None): + """Every .cmake/.json in a port that is reachable on a non-Windows host.""" + vc = vcpkg or VCPKG + for base in (OVERLAY, os.path.join(vc, "ports")): + d = os.path.join(base, port) + if not os.path.isdir(d): + continue + for root, _dirs, files in os.walk(d): + # A port can split itself by platform at the FILE level rather than + # with an if(): openssl's portfile.cmake picks between + # `unix/portfile.cmake` and `windows/portfile.cmake`, and the windows + # one asks for CLANG and NASM at TOP LEVEL, guarded by nothing this + # file can see. Skipping windows/ directories is therefore not + # cosmetic -- without it openssl reports clang as a Linux + # prerequisite. Conservative by construction: a Windows-only + # directory can only hold Windows-only requirements. + rel = os.path.relpath(root, d).split(os.sep) + if any(p.lower() in ("windows", "win32", "uwp", "mingw") for p in rel): + continue + for fn in sorted(files): + if fn.endswith((".cmake", ".json")): + yield os.path.join(root, fn) + return + + +# Programs a port probes with a bare `find_program` and then hard-fails on. This +# is a SECOND mechanism, found the hard way after the first one shipped: gperf +# died on `autoconf autoconf-archive automake libtoolize` from +# vcpkg-make/vcpkg_make.cmake, which never calls vcpkg_find_acquire_program at +# all -- it calls find_program(AUTORECONF NAMES autoreconf) and raises +# FATAL_ERROR listing apt packages. And it does it from a HELPER port +# (vcpkg-make), so the error names gperf while the requirement lives somewhere +# gperf does not mention. Enumerating only the acquire-program calls would have +# missed every one of these, which is why the preflight scans for both. + + +# Which binary proves an apt package is installed. Only needed where the two +# names differ; anything absent is probed under its own name, and a package that +# ships no executable at all maps to "" and is reported as unprobeable. +_PKG_BINARY = { + "autoconf": ["autoreconf", "autoconf"], + "automake": ["aclocal", "automake"], + "libtool": ["libtoolize", "glibtoolize"], + "gettext": ["autopoint"], + "gtk-doc-tools": ["gtkdocize"], + "autoconf-archive": [], # m4 macros, no binary + "libltdl-dev": [], # headers + "pkg-config": ["pkg-config", "pkgconf"], +} + + +def _fatal_package_requirements(text): + """Packages a file DEMANDS from the system package manager, and how to probe. + + Anchored on the text of the `message(FATAL_ERROR ...)` itself, not on the file + containing a FATAL_ERROR somewhere. That distinction is the difference between + a useful preflight and a noisy one: angle's portfile has both a FATAL_ERROR + (about an unsupported architecture) and a WARNING recommending + mesa-common-dev, and a file-level check staples them together and demands a + package no build step actually requires. A WARNING is advice; only a + FATAL_ERROR is a prerequisite. + + Returns {apt-package: [binaries that prove it, possibly empty]}. + """ + out = {} + for m in re.finditer(r'message\(\s*FATAL_ERROR\s+(.*?)\)\s*$', + strip_windows_blocks(text), re.S | re.M): + msg = m.group(1) + if not re.search(r'package manager|apt(?:-get)? install', msg): + continue + for am in re.finditer(r'apt(?:-get)? install ([^\n"\\]*)', msg): + for pkg in am.group(1).split(): + if not re.fullmatch(r'[\w.+-]+', pkg) or pkg == "sudo": + continue + out.setdefault(pkg, _PKG_BINARY.get(pkg, [pkg])) + return out + + +def host_tool_requirements(ports=None, vcpkg=None): + """-> {key: (binary, apt-package, [ports that need it], [alternatives])}. + + The programs the closure's portfiles need FROM THE HOST, by both mechanisms: + a `vcpkg_find_acquire_program` for a tool with no Linux download, and a bare + `find_program` in a file that hard-fails naming a package manager. + + This is a static parse and inherits its limits (a program named through a + variable is invisible). Unlike the distfile parse there is no instrument that + does better, because the failure only manifests on a machine that LACKS the + tool -- so a capture on a machine that has everything sees nothing, which is + precisely how nasm and autoconf each got to fail at minute 20. Under-reporting + degrades to that same late error; it never invents a prerequisite. + """ + vc = vcpkg or VCPKG + if ports is None: + ports = sorted(pinned_versions()) + out, asked = {}, {} + for port in ports: + for path in _port_cmake_files(port, vc): + with open(path, errors="replace") as f: + text = f.read() + # Mechanism 1: only the calls REACHABLE on a non-Windows host. openssl + # and vcpkg-make both ask for CLANG, but only inside an MSVC/Windows + # branch -- counting those made clang a prerequisite of a Linux build, + # which is a false alarm, and a preflight that cries wolf gets deleted. + for m in re.finditer( + r'vcpkg_find_acquire_program\(\s*([A-Z0-9_]+)', + strip_windows_blocks(text)): + asked.setdefault(m.group(1), set()).add(port) + # Mechanism 2: a FATAL_ERROR naming system packages directly. + for apt, binaries in _fatal_package_requirements(text).items(): + cur = out.setdefault(apt, [binaries[0] if binaries else "", apt, + set(), binaries[1:]]) + cur[2].add(port) + for program, users in asked.items(): + script = _acquire_script(program, vc) + if not script: + continue + with open(script, errors="replace") as f: + text = f.read() + if _has_non_windows_download(text): + continue + # program_name from the NON-Windows branch: PYTHON3 sets `python` for + # Windows and `python3` for everything else, and probing for `python` on + # Linux asks for a binary that has not existed by default for years. + body = strip_windows_blocks(text) + name = re.search(r'set\(program_name\s+"?([\w.+-]+)', body) + apt = re.search(r'set\(apt_package_name\s+"?([\w.+-]+)', body) + binary = name.group(1) if name else program.lower() + cur = out.setdefault(binary, [binary, apt.group(1) if apt else "", + set(), []]) + cur[2] |= users + return {k: (v[0], v[1], sorted(v[2]), v[3]) for k, v in out.items()} + + +def emit_host_tools(reqs): + """Write the host-prerequisite TSV: program, binary, apt package, ports.""" + print("# Tools the build needs FROM THE HOST, because vcpkg cannot supply them") + print("# on Linux at all. GENERATED by emit_vcpkg_bazel.py --host-tools.") + print("#") + print("# NOT the same class as vcpkg_tool_assets.tsv. Those are tools vcpkg") + print("# fetches for ITSELF, and the fix there was a pin. There is no URL to") + print("# pin for these, by either of the two mechanisms that produce them:") + print("#") + print("# 1. vcpkg_find_acquire_program() whose download URLs are ONLY") + print("# inside `if(CMAKE_HOST_WIN32)`. On Linux vcpkg probes the host and") + print("# hard-fails: 'Could not find nasm. Please install it via your") + print("# package manager'.") + print("# 2. a message(FATAL_ERROR ...) naming apt packages outright, after a") + print("# bare find_program. gperf died this way on autoconf/automake/") + print("# libtool -- and the requirement lives in the vcpkg-make HELPER") + print("# port, so the error names gperf while the cause names neither.") + print("#") + print("# So this file does not pretend to close the gap: it NAMES the gap, and") + print("# vcpkg_build.sh checks the whole list up front -- a machine missing") + print("# three tools is told about three tools in one second, instead of one") + print("# per 20-minute build (nasm surfaced from libvpx at minute ~20).") + print("#") + print("# ports-that-need-it is where the requirement is WRITTEN, which for") + print("# vcpkg-make is not where it fails: every autotools port using it can.") + print("#") + print("# An empty binary means the package ships no executable to probe for") + print("# (autoconf-archive is m4 macros, libltdl-dev is headers), so the") + print("# preflight can only name it -- it cannot verify it.") + print("#") + print("# binary-or-alternatives\tapt-package\tports-that-need-it") + for program in sorted(reqs): + binary, apt, users, alts = reqs[program] + names = "|".join([binary] + list(alts)) if binary else "-" + print("%s\t%s\t%s" % (names, apt or program, ",".join(users))) + + +def load_host_tools(path=None): + """The committed host-prerequisite list -> [(alternatives, apt, ports)]. + + `alternatives` is empty for a package with no binary to probe. + """ + path = path or HOST_TOOLS_TSV + if not os.path.exists(path): + return [] + out = [] + with open(path) as f: + for line in f: + line = line.split("#", 1)[0].strip() + if not line: + continue + parts = line.split("\t") + if len(parts) != 3: + continue + names, apt, users = parts + out.append(([n for n in names.split("|") if n and n != "-"], apt, + [p for p in users.split(",") if p])) + return out + + +def emit_tool_pins(tools): + """Write the tool pin TSV: url, sha512, the filename vcpkg looks for.""" + print("# vcpkg's OWN host tools, pinned from its scripts/vcpkg-tools.json at the") + print("# builtin-baseline. GENERATED by emit_vcpkg_bazel.py --capture-tools.") + print("#") + print("# Separate from vcpkg_assets.tsv because the asset capture CANNOT see") + print("# these: vcpkg_find_acquire_program probes the host first, so a tool the") + print("# capturing machine already has is never downloaded and never captured.") + print("# That is how ninja went unpinned -- the capturing machine had") + print("# /usr/bin/ninja at exactly the required version -- while cmake was pinned") + print("# only because the host's was too old. What a pin contains must not depend") + print("# on what happens to be installed on one machine.") + print("#") + print("# url\tsha512\tfilename-vcpkg-looks-for") + for sha, (url, name, port, _src) in sorted(tools.items(), key=lambda kv: kv[1][1]): + print("%s\t%s\t%s" % (url, sha, name)) + + def collect(): """-> (distfiles, git_externals, unresolved). Ports come from the manifest.""" pins = pinned_versions() @@ -412,6 +831,91 @@ def collect(): return distfiles, externals, unresolved, unexpanded +def _url_family(url): + """The URL with its final path component removed: 'where this comes from'. + + Two distfiles of the SAME upstream project at different versions share a + family (.../libsdl-org/SDL/archive/release-3.2.28.tar.gz and + .../release-3.4.12.tar.gz); two unrelated projects do not. That is all the + identity needed to tell "the capture never fetched this" apart from "the + capture fetched a DIFFERENT VERSION of this". + """ + return url.rsplit("/", 1)[0] + + +def classify_static_only(distfiles, capture): + """Split derived-but-not-captured rows into (platform_only, stale_capture). + + The capture REPLACES the static parse (see main()), for good reasons. But the + diagnostic that reported the casualties asserted its reason instead of + checking it: it printed every one of them as an entry "vcpkg never asked for + on this platform", which was true of the three it was written against + (libiconv/pthreads4w/dirent, all behind `if(VCPKG_TARGET_IS_WINDOWS)`) and + became false the moment a pin MOVED. + + At Ladybird's 71fb301a, vcpkg.json pins sdl3 3.2.28 -- confirmed by the + reference build's vcpkg_installed/vcpkg/info/sdl3_3.2.28_*.list -- and the + versions-db derivation gets that right. The capture, taken at the previous + pin, has release-3.4.12, so the correct current pin was dropped in favour of + a stale one and the message called it a Windows-only fetch. The checked-in + vcpkg_distfiles.bzl fetches 3.4.12 to this day. + + A distfile whose URL FAMILY the capture also has is not platform-only: it is + the same upstream project at a different version, i.e. the capture is stale + with respect to the manifest. Returns: + + platform_only [sha] -- genuinely absent from the capture + stale_capture [(sha, capture_sha)] -- derived pin vs the captured one + """ + fams = {} + for sha, (url, _name, _port, _src) in capture.items(): + fams.setdefault(_url_family(url), []).append(sha) + platform_only, stale = [], [] + for sha in sorted(set(distfiles) - set(capture)): + same = fams.get(_url_family(distfiles[sha][0])) + if same: + stale.append((sha, sorted(same)[0])) + else: + platform_only.append(sha) + return platform_only, stale + + +def classify_capture_only(distfiles, capture): + """Captured rows the DERIVATION contradicts: the mirror of the above. + + classify_static_only looks one way only -- derived-but-not-captured -- and so + cannot see a row the capture has too MANY of. That happened immediately: the + 71fb301a re-capture needed a supplementary run for angle alone (the one port + whose downloads a Download Mode halt had eaten), and a one-port manifest + resolves its own dependencies from the vcpkg BASELINE rather than from + Ladybird's vcpkg.json overrides. So it pulled zlib 1.3.2 where Ladybird pins + 1.3.1, and the capture came back with BOTH. + + I dropped that row by hand and then noticed I was doing the very thing this + session has been about: a hand-fix nothing checks, which the next capture + silently repeats. It is derivable -- the versions-db derivation knows the + pinned version, and a captured row in the same URL family at a version the + derivation does not pin is a row from the wrong resolution. + + Note the asymmetry with classify_static_only: there the CAPTURE wins, because + only vcpkg can expand a portfile's variables. Here the capture is a superset, + and the extra row is not a fetch this build makes -- so it is a leak, not a + pin. Reported, not dropped silently, because "vcpkg asked for it" is exactly + what makes it plausible enough to keep. + + Returns [(capture_sha, derived_sha)] -- the leaked row and the row it shadows. + """ + fams = {} + for sha, (url, _name, _port, _src) in distfiles.items(): + fams.setdefault(_url_family(url), []).append(sha) + leaked = [] + for sha in sorted(set(capture) - set(distfiles)): + same = fams.get(_url_family(capture[sha][0])) + if same: + leaked.append((sha, sorted(same)[0])) + return leaked + + def observed_git_archives(downloads_dir, distfiles): """The archives in a completed run's downloads/ that the asset cache never saw. @@ -531,9 +1035,15 @@ def emit_extension(distfiles): """ sys.stdout.write(HEADER) print('load(":vcpkg_distfiles.bzl", "vcpkg_distfiles")') + # The pip-installed Python packages a PORTFILE asks for. Hand-written, not + # captured -- pip does not go through vcpkg's asset cache, so the instrument + # that produced the 76 distfiles cannot see them (finding 36). Loaded here so + # they are created by the same extension and named by the same use_repo. + print('load(":vcpkg_python_packages.bzl", "vcpkg_python_wheels")') print() print("def _vcpkg_deps_impl(_ctx):") print(" vcpkg_distfiles()") + print(" vcpkg_python_wheels()") print() print("vcpkg_deps = module_extension(implementation = _vcpkg_deps_impl)") @@ -547,6 +1057,13 @@ def emit_use_repo(distfiles): "no such repository", far from its cause).""" names = sorted(bazel_name(name, sha) for sha, (_u, name, _p, _s) in distfiles.items()) + # The pip wheels are created by the SAME extension, so they must be in the + # same use_repo -- and they are NOT in the capture, because pip does not go + # through vcpkg's asset cache (finding 36). Read them out of the hand-written + # vcpkg_python_packages.bzl rather than restating them, so pasting this output + # into MODULE.bazel cannot silently drop the wheel and leave a "no such + # repository" a long way from its cause. + names += ["vcpkg_pywheel_" + n for n in sorted(python_wheel_names())] print("use_repo(") print(" vcpkg_deps,") for n in names: @@ -554,6 +1071,24 @@ def emit_use_repo(distfiles): print(")") +def python_wheel_names(): + """The keys of VCPKG_PYTHON_WHEELS, parsed out of the .bzl that declares them. + + Parsed rather than duplicated: that file is the pin, and a second copy of the + list here would be one more thing to drift (finding 23's rule, applied to the + one vcpkg input no instrument can capture). + """ + path = os.path.join(ROOT, "vcpkg_python_packages.bzl") + if not os.path.exists(path): + sys.stderr.write("WARNING: %s is missing; no pip wheels will be named, so " + "any port that pip-installs will fail\n" % path) + return [] + with open(path) as f: + body = f.read() + block = body.split("VCPKG_PYTHON_WHEELS = {", 1)[1].split("\n}", 1)[0] + return re.findall(r'^\s*"([A-Za-z0-9_.\-]+)":', block, re.M) + + def emit_index(distfiles): sys.stdout.write(HEADER) print("# sha512 (hex) -> the label of the file Bazel fetched for it.") @@ -628,18 +1163,98 @@ def main(): # that can break the build when an unrelated upstream URL rots, in # exchange for nothing. Meanwhile the union covered none of the 5 files # the capture genuinely misses -- the static parse misses those too. So - # the instrument wins outright, and the shortfall is REPORTED. - static_only = sorted(set(distfiles) - set(cap)) + # the instrument wins outright, and the shortfall is REPORTED -- but + # CLASSIFIED, not asserted. See classify_static_only: a dropped row whose + # URL family the capture also has is the same project at a different + # version, i.e. a STALE CAPTURE, and calling that "a fetch vcpkg never + # asked for on this platform" is how sdl3 3.2.28 got silently replaced by + # a captured 3.4.12 for a whole pin. + platform_only, stale = classify_static_only(distfiles, cap) + leaked = classify_capture_only(distfiles, cap) sys.stderr.write("capture: %d distfiles (static parse had %d)\n" % (len(cap), len(distfiles))) - if static_only: + if platform_only: sys.stderr.write( - " dropping %d static-only entries vcpkg never asked for on this " - "platform:\n" % len(static_only)) - for sha in static_only: + " dropping %d entries vcpkg never asked for on this platform " + "(no capture row for that upstream at all):\n" % len(platform_only)) + for sha in platform_only: sys.stderr.write(" %-40s %s\n" % (distfiles[sha][1], distfiles[sha][2])) + if stale: + sys.stderr.write( + " STALE CAPTURE: %d port(s) where vcpkg.json's pin and the " + "capture disagree.\n" + " The capture wins (it is the only exact source), so the " + "EMITTED rules fetch the\n" + " captured version, NOT the pinned one. Re-run " + "Meta/vcpkg_capture_assets.sh.\n" % len(stale)) + for sha, cap_sha in stale: + sys.stderr.write(" %-12s pinned %s\n" % (distfiles[sha][2], + distfiles[sha][1])) + sys.stderr.write(" %-12s captured %s\n" % ("", cap[cap_sha][1])) + if leaked: + sys.stderr.write( + " LEAKED CAPTURE ROWS: %d row(s) at a version this manifest does " + "not pin.\n" + " The capture has BOTH versions of these, so a run in it resolved " + "against a\n" + " different manifest -- a one-port supplementary capture takes its " + "deps from the\n" + " vcpkg BASELINE, not from Ladybird's vcpkg.json overrides. The " + "emitted rules\n" + " below therefore fetch a distfile no port in this build asks for. " + "Delete the\n" + " captured row, or re-capture with the real manifest.\n" % len(leaked)) + for cap_sha, sha in leaked: + sys.stderr.write(" %-12s captured %s\n" % (distfiles[sha][2], + cap[cap_sha][1])) + sys.stderr.write(" %-12s pinned %s\n" % ("", distfiles[sha][1])) distfiles, unexpanded = cap, [] + # vcpkg's OWN tools are unioned in from their COMMITTED pin, not replaced + # and not re-derived: they are a different class from port distfiles, and + # the capture cannot see them at all (finding 38). + tools = load_tool_pins() + if not tools: + sys.stderr.write( + "error: no %s\n" + "vcpkg's own host tools (cmake, ninja) are pinned there, because the\n" + "capture cannot see a tool the capturing machine already had. Without\n" + "them the index works only on machines that happen to have the same\n" + "tools installed. Regenerate with:\n" + " emit_vcpkg_bazel.py --capture-tools > Meta/vcpkg_tool_assets.tsv\n" + % TOOLS_TSV) + return 2 + added = [sha for sha in tools if sha not in distfiles] + sys.stderr.write("vcpkg host tools: %d pinned from %s (%d not in the capture)\n" + % (len(tools), os.path.basename(TOOLS_TSV), len(added))) + for sha in sorted(added): + sys.stderr.write(" added %-34s %s\n" % (tools[sha][1], tools[sha][2])) + # Cross-check against vcpkg's metadata WHEN a checkout is at hand: a stale + # committed pin (baseline bumped, tool version moved) is otherwise invisible + # until someone without that tool tries to build. + try: + live = tool_distfiles() + except VcpkgUnavailable: + pass + else: + stale = set(live) - set(tools) + gone = set(tools) - set(live) + for sha in sorted(stale): + sys.stderr.write( + " WARNING: %s is in vcpkg-tools.json but NOT in the committed" + " pin -- re-run --capture-tools\n" % live[sha][1]) + for sha in sorted(gone): + sys.stderr.write( + " note: %s is pinned but no longer in vcpkg-tools.json\n" + % tools[sha][1]) + for sha, row in tools.items(): + distfiles.setdefault(sha, row) + if "--capture-tools" in sys.argv: + emit_tool_pins(tool_distfiles()) + return 0 + if "--host-tools" in sys.argv: + emit_host_tools(host_tool_requirements()) + return 0 if "--git-archives" in sys.argv: dl = sys.argv[sys.argv.index("--git-archives") + 1] archives, byproducts = observed_git_archives(dl, distfiles) diff --git a/examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py b/examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py new file mode 100644 index 0000000..64f0d2a --- /dev/null +++ b/examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Fetch the four vcpkg_from_git archives, WITHOUT running CMake or vcpkg. + +WHY THIS EXISTS +--------------- +Four of vcpkg's inputs are not distfiles. skia and angle pull sub-dependencies +with `vcpkg_from_git`, which -- unlike every other fetch vcpkg makes -- bypasses +the asset cache entirely (`x-asset-sources` never sees it, so +`Meta/vcpkg_capture_assets.sh` cannot record it and `x-block-origin` does not +govern it). What vcpkg_from_git *does* honour is a pre-placed +`downloads/-.tar.gz`, and that is the hook this uses. + +For a long time those four tarballs came from a directory on the author's +machine, staged by a `cp ... 2>/dev/null || true` that could not fail (finding +36). They were in fact a copy of `Build/vcpkg/downloads/`, i.e. of vcpkg's own +cache -- which means the only way to get them was to have run CMake. This script +is the answer to "how do you get them WITHOUT running CMake": + +It takes the LIST of archives from the committed pin (VCPKG_GIT_ARCHIVES in +vcpkg_git_archives.bzl), resolves each one's clone URL out of the portfiles, then +for each: `git clone` + `git -c core.autocrlf=false archive ` -- byte-for-byte +what vcpkg_from_git does internally -- and **verifies the result against the pinned +SHA512, failing on any mismatch**. So the pin stops being trusted: the hashes came +from vcpkg, and reproducing them from scratch with git is the proof that git and +vcpkg agree. Needs the vcpkg checkout (`Meta/ladybird.py vcpkg`, ~70s) for the +portfiles, and nothing else. + +WHY NOT A REPO RULE: these are `git archive` output, so there is no URL for +`http_file`; a repository_rule could shell out to git, but it would then be an +un-cacheable, single-threaded network fetch at load time, and the same eight +lines of git. Keep it as a prefetch that Bazel *verifies* (vcpkg_build.sh +hard-fails in 4s when the tarballs are absent), rather than machinery that hides +a clone inside analysis. +""" + +import argparse +import hashlib +import os +import re +import subprocess +import sys +import tempfile +from pathlib import Path + +LADYBIRD_ROOT = Path(os.environ.get("LADYBIRD_ROOT", Path(__file__).resolve().parent.parent)) + +# WHICH tarballs are needed is NOT decided here -- it is read from the committed +# pin (VCPKG_GIT_ARCHIVES), because that list came from vcpkg itself. This module +# only answers "given this archive name, what URL do I clone?", by scanning +# portfiles for the literal URL/REF arguments. +# +# That split is the whole correctness argument, and it is the second thing I got +# wrong here. A first version *derived* the list by parsing skia's and angle's +# portfiles: it found 8 for skia (against 4 truly fetched) and missed libyuv +# entirely. Both errors have the same cause -- `declare_external_from_git` only +# DECLARES, and `get_externals(${required_externals})` picks from it under +# feature/platform `if()`s, so the set is decided by CMake evaluation, not by the +# text -- and libyuv is a separate port whose portfile calls vcpkg_from_git +# directly. Statically deciding the SET is therefore unsound; statically +# resolving a NAME to a URL is not. + + +def all_portfiles(vcpkg_root: Path) -> list[Path]: + """Overlay first: --overlay-ports shadows the checkout's ports/.""" + overlay = sorted((LADYBIRD_ROOT / "Meta" / "CMake" / "vcpkg" / "overlay-ports").glob("*/portfile.cmake")) + builtin = sorted((vcpkg_root / "ports").glob("*/portfile.cmake")) + shadowed = {p.parent.name for p in overlay} + return overlay + [p for p in builtin if p.parent.name not in shadowed] + + +# Three syntaxes reach vcpkg_from_git, all with literal arguments: +# declare_external_from_git(name URL "..." REF "...") -- skia +# checkout_in_path("" "" "") -- angle +# vcpkg_from_git(URL REF ) -- libyuv, directly +_PATTERNS = ( + re.compile(r'declare_external_from_git\s*\(\s*\w+\s+URL\s+"(?P[^"]+)"\s+REF\s+"(?P[^"]+)"'), + re.compile(r'checkout_in_path\s*\(\s*"[^"]+"\s+"(?P[^"]+)"\s+"(?P[^"]+)"\s*\)'), + re.compile(r'vcpkg_from_git\s*\((?P[^)]*)\)'), +) +_FROM_GIT_URL = re.compile(r'URL\s+"?(?P[^"\s]+)"?') +_FROM_GIT_REF = re.compile(r'\bREF\s+"?(?P[^"\s]+)"?') +_SET_RE = re.compile(r'set\s*\(\s*(?P\w+)\s+"?(?P[0-9a-f]{40})"?\s*\)') + + +def _resolve(value: str, variables: dict) -> str | None: + """Expand ${VAR} against `set(VAR )` in the same portfile. + + Only 40-hex values are collected, so this cannot silently expand to a branch + name: vcpkg_from_git REQUIRES a commit SHA (it rev-parses and compares), and + a ref that is not one should surface here rather than later. + """ + m = re.fullmatch(r"\$\{(\w+)\}", value) + if m: + return variables.get(m.group(1)) + return value if re.fullmatch(r"[0-9a-f]{40}", value) else None + + +def refs_in_portfile(path: Path) -> dict: + """-> {(port, ref): url} for every literal vcpkg_from_git call in one portfile.""" + text = path.read_text() + port = path.parent.name + variables = {m.group("var"): m.group("val") for m in _SET_RE.finditer(text)} + out = {} + for pat in _PATTERNS: + for m in pat.finditer(text): + if "body" in m.groupdict() and m.groupdict().get("body") is not None: + body = m.group("body") + mu, mr = _FROM_GIT_URL.search(body), _FROM_GIT_REF.search(body) + if not (mu and mr): + continue + url, raw_ref = mu.group("url"), mr.group("ref") + else: + url, raw_ref = m.group("url"), m.group("ref") + ref = _resolve(raw_ref, variables) + if ref: + # vcpkg names the archive after the PORT, not the dependency: + # DOWNLOADS/${PORT}-${sanitized_ref}.tar.gz in vcpkg_from_git.cmake. + out[f"{port}-{ref}.tar.gz"] = url + return out + + +def resolve_urls(vcpkg_root: Path, wanted: list) -> dict: + """Find the clone URL for each WANTED archive name. Every name must resolve.""" + index: dict = {} + for pf in all_portfiles(vcpkg_root): + for name, url in refs_in_portfile(pf).items(): + index.setdefault(name, url) + missing = [n for n in wanted if n not in index] + if missing: + raise SystemExit( + "fetch_vcpkg_git_archives: no vcpkg_from_git call found for:\n " + + "\n ".join(missing) + + "\n(the pin lists it, but no portfile in the checkout or overlay declares it --\n" + " the checkout may be at the wrong baseline)" + ) + return {n: index[n] for n in wanted} + + +def committed_hashes() -> dict[str, str]: + bzl = LADYBIRD_ROOT / "vcpkg_git_archives.bzl" + if not bzl.is_file(): + return {} + return dict(re.findall(r"'([^']+\.tar\.gz)':\s*'([0-9a-f]{128})'", bzl.read_text())) + + +def fetch(vcpkg_root: Path, out_dir: Path) -> int: + """Reproduce each pinned tarball with git clone + git archive, verify, install.""" + pinned = committed_hashes() + if not pinned: + raise SystemExit( + "fetch_vcpkg_git_archives: no pinned archives found in vcpkg_git_archives.bzl\n" + " That file IS the list of what to fetch; regenerate the pin first with\n" + " Meta/vcpkg_capture_git_archives.sh (vcpkg install --only-downloads)." + ) + urls = resolve_urls(vcpkg_root, sorted(pinned)) + out_dir.mkdir(parents=True, exist_ok=True) + failures = [] + + for name, want in sorted(pinned.items()): + ref = name.rsplit("-", 1)[1][: -len(".tar.gz")] + dest = out_dir / name + if dest.is_file() and sha512(dest) == want: + print(f" ok (cached) {name}") + continue + with tempfile.TemporaryDirectory(prefix="vcpkg-git-archive-") as tmp: + repo = Path(tmp) / "repo" + print(f" cloning {urls[name]}") + # A full clone: the pinned ref is usually not the tip, and a shallow + # clone cannot archive an arbitrary commit -- the same reason vcpkg + # needs full history for its own registry. + subprocess.run(["git", "clone", "--quiet", urls[name], str(repo)], check=True) + tmp_out = Path(tmp) / name + # `-c core.autocrlf=false` is not optional: it is what + # vcpkg_from_git.cmake passes, and it changes the bytes of any file + # git would otherwise translate. + subprocess.run( + ["git", "-c", "core.autocrlf=false", "archive", ref, "-o", str(tmp_out)], + cwd=repo, + check=True, + ) + got = sha512(tmp_out) + if got != want: + failures.append(f"{name}: sha512 {got[:16]}... != pinned {want[:16]}...") + print(f" MISMATCH {name}") + continue + tmp_out.replace(dest) + print(f" verified {name}") + + for f in failures: + print(f"fetch_vcpkg_git_archives: {f}", file=sys.stderr) + if failures: + return 1 + print(f"{len(pinned)}/{len(pinned)} git-sourced externals present and verified in {out_dir}") + return 0 + + +def sha512(path: Path) -> str: + h = hashlib.sha512() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--vcpkg-root", type=Path, default=LADYBIRD_ROOT / "Build" / "vcpkg") + ap.add_argument( + "--out", + type=Path, + default=LADYBIRD_ROOT / "Meta" / "CMake" / "vcpkg" / "git-archives", + help="where the tarballs are written (the directory Meta/vcpkg_build.sh stages from)", + ) + args = ap.parse_args() + + if not args.vcpkg_root.is_dir(): + raise SystemExit( + f"fetch_vcpkg_git_archives: no vcpkg checkout at {args.vcpkg_root}\n" + " Get one with: python3 Meta/ladybird.py vcpkg (no CMake configure needed)" + ) + return fetch(args.vcpkg_root, args.out) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/workspace/Meta/pin_hsts_preload.py b/examples/ladybird/workspace/Meta/pin_hsts_preload.py new file mode 100644 index 0000000..7df6a55 --- /dev/null +++ b/examples/ladybird/workspace/Meta/pin_hsts_preload.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +"""Re-pin Chromium's HSTS preload table, and write hsts_preload.bzl. + +CMake fetches net/http/transport_security_state_static.json from Chromium's +**main** at configure time (Meta/CMake/hsts_preload.cmake), unversioned. That is +an upstream defect we cannot fix from the overlay, so the overlay pins it +downstream: one immutable commit URL plus the sha256 this script MEASURED by +downloading it. + +Two properties this script exists to preserve, both of them learned the hard way: + + * Pin a COMMIT, not a release tag. The tag 139.0.7258.5 serves an 18.7 MB file + generating 168,593 entries; the commit main pointed at when the reference + CMake build configured serves 10.5 MB and 94,626. A tag is a pin, but a pin + to a *different table* -- so pinning to one trades a hermeticity gap for a + parity gap. `--commit` defaults to the newest commit touching the path, which + is the one `main` serves. + * Never write a hash you did not measure. The sha256 in the output is computed + from the bytes downloaded from the URL being pinned. `--expect-sha256` + additionally asserts the pinned bytes equal a file you already have (the one + CMake downloaded), which is how the current pin was shown to keep byte-parity + with the reference build. + +Usage: + # re-pin to the newest commit touching the path + python3 Meta/pin_hsts_preload.py > hsts_preload.bzl + + # re-pin, and refuse unless it matches what CMake already downloaded + python3 Meta/pin_hsts_preload.py \ + --expect-same-as Build/caches/HSTSPreload/transport_security_state_static.json \ + > hsts_preload.bzl + + # just report what the pin would be, without writing + python3 Meta/pin_hsts_preload.py --check +""" +import argparse +import hashlib +import json +import shutil +import subprocess +import sys + +PATH = "net/http/transport_security_state_static.json" +COMMITS_API = "https://api.github.com/repos/chromium/chromium/commits?path={path}&per_page=1" +COMMIT_API = "https://api.github.com/repos/chromium/chromium/commits/{commit}" +RAW_URL = "https://raw.githubusercontent.com/chromium/chromium/{commit}/" + PATH + +TEMPLATE = '''\ +# Chromium's HSTS preload table, pinned to a commit. GENERATED by +# Meta/pin_hsts_preload.py -- see that script to re-pin; do not hand-edit. +# +# WHY THIS FILE EXISTS. `Meta/CMake/hsts_preload.cmake` downloads +# net/http/transport_security_state_static.json from Chromium's **main** at +# configure time -- an unversioned ref. The generator turns it into a ~95,000 +# entry `constexpr Array` of domains LibHTTP forces to HTTPS, so "whatever main +# served the day you configured" decides a security-relevant table, and two +# developers who configured on different days build different browsers. +# +# The fix belongs upstream (pin the URL in hsts_preload.cmake). We cannot make +# that change from here, so we pin DOWNSTREAM: Bazel fetches one immutable +# commit URL with a sha256, and the upstream unpinned fetch is filed as a bug. +# +# Pinning downstream is only honest if it does not break byte-parity with CMake, +# and this pin does not: the bytes this commit serves are byte-identical to the +# ones CMake's `main` fetch downloaded for the reference build ({size:,} bytes, +# checked with --expect-same-as, which refuses to write this file otherwise). +# Pinning to a Chromium *release tag* would NOT have that property: 139.0.7258.5 +# serves 18.7 MB and generates 168,593 entries against this commit's 94,626. +# +# Two consequences worth knowing: +# +# * CMake still tracks `main`, so a CMake configure NEWER than this pin will +# disagree with Bazel. That is now a visible, dated disagreement between one +# pinned input and one unpinned one, instead of two unpinned fetches that +# happened to agree. `download_file` is a no-op when the file already exists +# (verified with ENABLE_NETWORK_DOWNLOADS=OFF), so staging the Bazel-fetched +# file into Build/caches/HSTSPreload/ before configuring makes CMake consume +# this same pin -- see README, "Getting the three inputs". +# * Bumping the pin is a deliberate, reviewable act: re-run +# Meta/pin_hsts_preload.py, which resolves the newest commit touching the +# path, downloads it, and writes this file with the hash it measured. +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + +# The pinned commit: "{subject}" ({date}). +# raw.githubusercontent.com paths are immutable for a full commit sha, unlike the +# `main` path CMake uses. +HSTS_PRELOAD_COMMIT = "{commit}" + +HSTS_PRELOAD_SHA256 = "{sha256}" + +HSTS_PRELOAD_URL = "https://raw.githubusercontent.com/chromium/chromium/{{}}/net/http/transport_security_state_static.json".format( + HSTS_PRELOAD_COMMIT, +) + +def _hsts_preload_impl(_ctx): + http_file( + name = "hsts_preload_json", + urls = [HSTS_PRELOAD_URL], + sha256 = HSTS_PRELOAD_SHA256, + downloaded_file_path = "transport_security_state_static.json", + ) + +hsts_preload = module_extension(implementation = _hsts_preload_impl) +''' + + +def _get(url, timeout): + """GET url, as bytes. + + Shells out to curl rather than using urllib: this runs behind + TLS-intercepting proxies (urllib rejected ours with CERTIFICATE_VERIFY_FAILED + while curl, which reads the system CA bundle the same way git and Bazel do, + succeeded). A pin script that cannot run in the environment the build runs in + is not a pin script. + """ + if shutil.which("curl") is None: + sys.exit("curl not found; it is how this script reaches the network") + p = subprocess.run(["curl", "-sSL", "--fail", "--max-time", str(timeout), url], + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + sys.exit("fetch failed for %s: %s" % (url, p.stderr.decode().strip())) + return p.stdout + + +def newest_commit(): + """The newest commit touching the path -- i.e. what `main` currently serves.""" + commits = json.loads(_get(COMMITS_API.format(path=PATH), 60)) + if not commits: + sys.exit("no commits returned for %s" % PATH) + c = commits[0] + return c["sha"], c["commit"]["message"].splitlines()[0], c["commit"]["committer"]["date"][:10] + + +def describe(commit): + """Subject line + date for a commit, so the pin records WHAT it pinned.""" + c = json.loads(_get(COMMIT_API.format(commit=commit), 60)) + return (c["commit"]["message"].splitlines()[0], + c["commit"]["committer"]["date"][:10]) + + +def fetch(commit): + blob = _get(RAW_URL.format(commit=commit), 300) + return blob, hashlib.sha256(blob).hexdigest() + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--commit", help="pin this commit instead of the newest one touching the path") + ap.add_argument("--expect-same-as", metavar="FILE", + help="fail unless the pinned bytes equal FILE (e.g. the file CMake downloaded)") + ap.add_argument("--check", action="store_true", + help="report the pin on stderr and write nothing") + args = ap.parse_args(argv) + + if args.commit: + commit = args.commit + subject, date = describe(commit) + else: + commit, subject, date = newest_commit() + + blob, sha256 = fetch(commit) + + if args.expect_same_as: + want = open(args.expect_same_as, "rb").read() + if want != blob: + sys.exit( + "PARITY: the pinned bytes differ from %s (%d vs %d bytes).\n" + "Chromium's main has moved since that file was downloaded. Pinning this\n" + "commit would make Bazel's generated table differ from CMake's; either\n" + "re-run CMake's configure (or stage the pinned file into\n" + "Build/caches/HSTSPreload/), or pin the older commit with --commit." + % (args.expect_same_as, len(want), len(blob)) + ) + print("parity OK: pinned bytes identical to %s" % args.expect_same_as, file=sys.stderr) + + print("commit %s %s %s %d bytes sha256 %s" + % (commit[:12], date, subject, len(blob), sha256), file=sys.stderr) + if args.check: + return 0 + + sys.stdout.write(TEMPLATE.format( + commit=commit, sha256=sha256, size=len(blob), + subject=subject, date=date, + )) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/workspace/Meta/vcpkg_assets.tsv b/examples/ladybird/workspace/Meta/vcpkg_assets.tsv index f5a52bb..335d95f 100644 --- a/examples/ladybird/workspace/Meta/vcpkg_assets.tsv +++ b/examples/ladybird/workspace/Meta/vcpkg_assets.tsv @@ -1,76 +1,76 @@ -https://chrome-infra-packages.appspot.com/dl/gn/gn/linux-amd64/+/fj2NZKMkIYZNH6uYG0bn8OsW_lZB5JKz3JeScMCLAGQC d49575bd383b6aace1257a6e9439ce0a206173ec2cab94d5312f06db412e09c89aa75b1f4c69f5dca4389d15a489c211a73439a66f437c34b18bc90eefa0b775 gn-linux-amd64.zip -https://downloads.sourceforge.net/project/libpng-apng/libpng16/1.6.58/libpng-1.6.58-apng.patch.gz 95a6f5bb7148b5c48dccd73811d7bcf9752a631a7bb4f4856670a7da12a7159581ac1bce1749318343794e0f5cb86972711ba2ec0f523c168f0991fa940687d5 libpng-1.6.58-apng.patch.gz.25974.part -https://ftpmirror.gnu.org/gnu/automake/automake-1.17.tar.gz 11357dfab8cbf4b5d94d9d06e475732ca01df82bef1284888a34bd558afc37b1a239bed1b5eb18a9dbcc326344fb7b1b301f77bb8385131eb8e1e118b677883a automake-1.17.tar.gz -https://ftpmirror.gnu.org/gnu/gperf/gperf-3.3.tar.gz 246b75b8ce7d77d6a8725cd15f1cf2e68da404812573af1d5bf32dbe6ad4228f48757baefc77bcb1f5597c2397043c04d31d8a04ab507bfa7a80f85e1ab6045f gperf-3.3.tar.gz -https://github.com/AOMediaCodec/libavif/archive/v1.4.2.tar.gz cbf31827884058acc54d3b1fa0f2059f022691609a76e0f913981d5b8c1be60f52069a25b61c888c09e963161d1f4c9bf692c0716ab0ee067a4ccaa4e36d9ce1 AOMediaCodec-libavif-v1.4.2.tar.gz -https://github.com/GNOME/libxml2/archive/v2.15.3.tar.gz f65df793fca5e46552afbaa56b04c4774829a95e012a6dc4dc3d10e6884a6118e30a426e422516e1d21e91c4a1d34cb00a5cee61af35cab03c71fb9b5c09e138 GNOME-libxml2-v2.15.3.tar.gz -https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator/archive/v3.4.0.tar.gz 563acbcd8912d10d92c23715eba7acf0e7c1683af36021f415b36f359c2cce065f3906e395c32282a8410ec5c8179fbcb6412935c6629a49357475d4b4410e2a GPUOpen-LibrariesAndSDKs-VulkanMemoryAllocator-v3.4.0.tar.gz -https://github.com/KhronosGroup/EGL-Registry/archive/3ae2b7c48690d2ce13cc6db3db02dfc0572be65e.tar.gz c7b09ded4964fa427546bd345a29325105b79079b59642214dc8f04de113f42de2bc4272dbbbd4a801d92afc20297442fdfa12043a0900cf1e2b1cd83f260883 KhronosGroup-EGL-Registry-3ae2b7c48690d2ce13cc6db3db02dfc0572be65e.tar.gz.21812.part -https://github.com/KhronosGroup/OpenGL-Registry/archive/0b449b97cdf1043eef5e1f0e235cbbab6ec10c86.tar.gz 148e1bfe4cc199bcc2c23b22d0b3e4988a29389d7f510ba4a6340672dbb7ab99bb836d2c08587499484df704d51a1adf4f0dc3a30d5ad8977ee0ad339163b17e KhronosGroup-OpenGL-Registry-0b449b97cdf1043eef5e1f0e235cbbab6ec10c86.tar.gz.21854.part -https://github.com/KhronosGroup/Vulkan-Headers/archive/vulkan-sdk-1.4.350.1.tar.gz eee702ee5ff447986901e44989d96a5e38f7ef7f67e2015c8db6280251cb23b69cb9e1a0568b0a43dc3df820685079ed31998a1dc4704f5649768e26db5147b1 KhronosGroup-Vulkan-Headers-vulkan-sdk-1.4.350.1.tar.gz -https://github.com/Kitware/CMake/releases/download/v4.4.0/cmake-4.4.0-linux-x86_64.tar.gz 3df4aaa128a438ed48dcac7065fd355ff538eed8f394491298d0db63a891d671da247c8fa262e4fa6bf99429d630abab317d5a0248168fe203d1ca4978dab4da cmake-4.4.0-linux-x86_64.tar.gz.20596.part -https://github.com/NixOS/patchelf/releases/download/0.19.0/patchelf-0.19.0-x86_64.tar.gz 2a65c9cbdddcc7952cdbd6e98a2cf3da01386cf0f0b927a6bbcfe8131ecf0bfb17c534246635b5e6a090652ee54c903f9f9c4f3f1d2412dba59f287ae2ae8070 patchelf-0.19.0-x86_64.tar.gz.21729.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/ThirdParty/ANGLE/gni-to-cmake.py cf9dab1b59cd44b9ce05a2dc4636115e770dfe7d5cfbd90c3fef659628ac8155c57b866480f7cfe9a0afeb31ff5ce5eed74473f386120a7fc5910e8b032bd61d gni-to-cmake.py.22348.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/ThirdParty/ANGLE/include/CMakeLists.txt a7ddf3c6df7565e232f87ec651cc4fd84240b8866609e23e3e6e41d22532fd34c70e0f3b06120fd3d6d930ca29c1d0d470d4c8cb7003a66f8c1a840a42f32949 include_CMakeLists.txt.22445.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/DetectSSE2.cmake 219a4c8591ee31d11eb3d1e4803cc3c9d4573984bb25ecac6f2c76e6a3dab598c00b0157d0f94b18016de6786e49d8b29a161693a5ce23d761c8fe6a798c1bca DetectSSE2.cmake.22467.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/WebKitCompilerFlags.cmake 8b281ffcf9209c845a5fdae48a4e05f08ca677c37a7fb00d9270de81bd103160d26e091724ce8df8d428ad604900b5202b221fed5bafffd4bf00025718ef9d8e WebKitCompilerFlags.cmake.22460.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/WebKitMacros.cmake 565175443d5d1b8119af504164bf93840e8c786fc479e45feb98ca542351b91d2ea00265d7f8dfa6960975de81802bc43b2a2af2c90fc44bda1ae46d96c89247 WebKitMacros.cmake.22482.part -https://github.com/abseil/abseil-cpp/archive/20260107.1.tar.gz f5012885d6b6844a9cf5ed92ad5468b8757db33dfe1364bfb232fff928e06c550c7eb4557f45186a8ac4d18b178df9be267681abab4a6de40823b574afbe9960 abseil-abseil-cpp-20260107.1.tar.gz -https://github.com/cisco/openh264/archive/v2.6.0.tar.gz 26a03acde7153a6b40b99f00641772433a244c72a3cc4bca6d903cf3b770174d028369a2fb73b2f0774e1124db0e269758eed6d88975347a815e0366c820d247 cisco-openh264-v2.6.0.tar.gz -https://github.com/curl/curl/archive/curl-8_21_0.tar.gz 0ab6c99c3d5b86fb65c526db517c3159b11db2f8d82552d635c4887059c0602288603c93b754ce0ec543ea2f275122ccec2c8dcd866c2611b5b949c728ee72df curl-curl-curl-8_21_0.tar.gz.8039.part -https://github.com/davea42/libdwarf-code/archive/v2.3.1.tar.gz e8eb74c622cedb512d82dff313ce9c5ac2064a7a7a0691c776213b384c1c64d0f549aaab1ef732dcba8c72d52048cea641d9909d3b2503cf96760ad8e81ae77c davea42-libdwarf-code-v2.3.1.tar.gz.30028.part -https://github.com/facebook/zstd/archive/v1.5.7.tar.gz 26e441267305f6e58080460f96ab98645219a90d290a533410b1b0b1d2f870721c95f8384e342ee647c5e968385a5b7e30c2d04340c37f59b3e6d86762c3260c facebook-zstd-v1.5.7.tar.gz.28957.part -https://github.com/fastfloat/fast_float/archive/v8.2.10.tar.gz eec563358117b593e882a9069074a269d811e8989457a0d57e50c5d2f4d534b6820308082bb70c0a8a8388513e92b64f12c5213389eff726ad0483eacba961ff fastfloat-fast_float-v8.2.10.tar.gz.19662.part -https://github.com/ffmpeg/ffmpeg/archive/n7.1.1.tar.gz 6b9a5ee501be41d6abc7579a106263b31f787321cbc45dedee97abf992bf8236cdb2394571dd256a74154f4a20018d429ae7e7f0409611ddc4d6f529d924d175 ffmpeg-ffmpeg-n7.1.1.tar.gz -https://github.com/fmtlib/fmt/archive/12.2.0.tar.gz 5ac2ba0f54a484999ed5407d82b77aad170cea49a267decd2c0eedadf3b14413e2a83fcc8e9ca9c16640595e019b8636e160f72314d8be50653324e82ac745eb fmtlib-fmt-12.2.0.tar.gz -https://github.com/fmtlib/fmt/commit/588b3a0f8f6a8bcf2a959cae882d5b2703e86737.patch?full_index=1 afda8fdfcdcb4b0dd5df4d4dae96a57a85fb9c4b65d0b49d51258f0913d4aed93ed146ebf96ed7b277490b1dde6c7117f43332013071441a96c3147520de8368 fmt-backport-4813.patch -https://github.com/google/angle/archive/79ac1a8cd767a32cce6401203e20c4bd4ca4d539.tar.gz a3d5d09460f05f1f1c081411bf15f585bb8add5f3db7c4854dc2cf2ef1fd22dd5b2dd5d3de78bfb8bcdae8459dd9581f7dce331acd34053c87c84cfe6c3d134c google-angle-79ac1a8cd767a32cce6401203e20c4bd4ca4d539.tar.gz.22306.part -https://github.com/google/brotli/archive/v1.2.0.tar.gz f94542afd2ecd96cc41fd21a805a3da314281ae558c10650f3e6d9ca732b8425bba8fde312823f0a564c7de3993bdaab5b43378edab65ebb798cefb6fd702256 google-brotli-v1.2.0.tar.gz.28349.part -https://github.com/google/highway/archive/1.4.0.tar.gz 819422857d6a74e3a936c402698e078db5b7b88fb43767e62429ec7bd954fe93b017e75029a4df4a1a97ef3a2486107eef5247da751eb487640dea409f3f2fa2 google-highway-1.4.0.tar.gz -https://github.com/google/skia/archive/e7c90ecca9444fe09598f1630ab7cee2c0ee027a.tar.gz f52286fcac1d1b2b45046d7df72cee384b664f0b512c4b4a76bca743747f2113fef6991059b4e71ad804d0209103003ba0f01aa90afac786de20a52e8484018d google-skia-e7c90ecca9444fe09598f1630ab7cee2c0ee027a.tar.gz -https://github.com/google/woff2/archive/v1.0.2.tar.gz c788bba1530aec463e755e901f9342f4b599e3a07f54645fef1dc388ab5d5c30625535e5dd38e9e792e04a640574baa50eeefb6b7338ab403755f4a4e0c3044d google-woff2-v1.0.2.tar.gz -https://github.com/google/wuffs-mirror-release-c/archive/v0.3.4.tar.gz d22136a1adf337573944eed917142c6bde877a09bd65738010c6c367c7a3fc9e4573fd4dc8469c93799fe1e3247760e65e64829e830871ee7b333fd72ebc629d google-wuffs-mirror-release-c-v0.3.4.tar.gz -https://github.com/harfbuzz/harfbuzz/archive/10.2.0.tar.gz 697205a571bb3d52d83598e8511e2e21e7cd15630aac32d8deb4354e462efda6a5ce46510cf4a1c18365dffe935cf2e4f1fda65d1779f17c9bb60c503315bf5c harfbuzz-harfbuzz-10.2.0.tar.gz -https://github.com/jeremy-rifkin/cpptrace/archive/v1.0.2.tar.gz 4ae394fb3c21149bf2441a754eebe639e6a5534927426b6507806c7bee0b1c982e047c972904d472f1c660adb5be3881e7e3a6eddd18e4e9d376ae3855d50a7c jeremy-rifkin-cpptrace-v1.0.2.tar.gz.31337.part -https://github.com/libexpat/libexpat/archive/R_2_8_2.tar.gz e60e6d6ae9d0115f41186f06f3854008054863f0b29a58d44142f5b30057a494337145d820b5e18270b8ca3e779e318a757fcc6bf64d6d204e5498df5eeb2195 libexpat-libexpat-R_2_8_2.tar.gz.13196.part -https://github.com/libjpeg-turbo/libjpeg-turbo/archive/3.2.0.tar.gz 1ce063e9e126d55019385da3a6ff4521a9a3958edeab15e2465ae3435026dac6e598277eea066173ada47da37159d0f7812b2227869c10dd141dcfe2ddeab720 libjpeg-turbo-libjpeg-turbo-3.2.0.tar.gz.21972.part -https://github.com/libjxl/libjxl/archive/v0.11.2.tar.gz a7e1f7d060b358f4382e84367d66aa2850aef3b4524a0fdfe3f22dd258fb9e35dda7540f859d8bf4c32f31c61a7a03db677f4490a9f472cd25869a9d00797336 libjxl-libjxl-v0.11.2.tar.gz -https://github.com/libproxy/libproxy/archive/0.4.18.tar.gz 1148d688a9f070273a1a2b110a788561789799089660292bbba59fbf0a9caf7d28cb039a9ccdcb935f752e1e34739b2d2f4c784b1bb3bbaa03d108e7b38a4754 libproxy-libproxy-0.4.18.tar.gz -https://github.com/libsdl-org/SDL/archive/release-3.4.12.tar.gz fc0a55ca01c32f613b9cd8c6cffad17ee0855ee542f03fa455632043c99a0a259599557a46c7c636032444260e64476867d9af43a1da689837f99c22942cd863 libsdl-org-SDL-release-3.4.12.tar.gz -https://github.com/libtom/libtommath/archive/v1.3.0.tar.gz 3dbd7053a670afa563a069a9785f1aa4cab14a210bcd05d8fc7db25bd3dcce36b10a3f4f54ca92d75a694f891226f01bdf6ac15bacafeb93a8be6b04c579beb3 libtom-libtommath-v1.3.0.tar.gz -https://github.com/madler/zlib/archive/v1.3.1.tar.gz 8c9642495bafd6fad4ab9fb67f09b268c69ff9af0f4f20cf15dfc18852ff1f312bd8ca41de761b3f8d8e90e77d79f2ccacd3d4c5b19e475ecf09d021fdfe9088 madler-zlib-v1.3.1.tar.gz.21336.part -https://github.com/mesonbuild/meson/archive/1.9.0.tar.gz ecd69b6734be14c467f7db67dd88c0e57ebfad83ce3ddada131ff3e43ac964523e1083d7c7f3803033a9a76adbc32ad26dd2e3aca69884269000ca64130bde07 meson-1.9.0.tar.gz -https://github.com/microsoft/mimalloc/archive/v2.2.7.tar.gz 19f5481c98822c6a279e2f2d26d6c54918e18bcafd515a76ca3a3845f3a5e599236ea042f9cadaf793dce0c1ede347d76cc2fc5503094288362a78034d4d8f71 microsoft-mimalloc-v2.2.7.tar.gz -https://github.com/mm2/Little-CMS/archive/lcms2.19.1.tar.gz 1b2781ed8898e65f15be17cf0130a1500ec0bf5ca5159f871dff5692e387747be9526feef0bc7c370200656fc0aabe3036746041285a3978e90adec200d685f2 mm2-Little-CMS-lcms2.19.1.tar.gz -https://github.com/mozilla/pdf.js/releases/download/v5.6.205/pdfjs-5.6.205-dist.zip 66fecdb8a80d013b592c361e85abd7eeea5bc35f0131b771cc25a1878d7737704458dc715670aace3c7d3437f85388dff04796fe117eef01be81fd0d192f73a2 pdfjs-5.6.205-dist.zip -https://github.com/nghttp2/nghttp2/archive/v1.69.0.tar.gz 1029fb86935a88fc518cc2d976dff5253277f97e01f32b1f73c5df96dcf7fe0280a83a9e9d676c3d96caa542300dc7cfa61f2a40b1b11d2c2b527e870a974b53 nghttp2-nghttp2-v1.69.0.tar.gz.7004.part -https://github.com/ngtcp2/nghttp3/archive/v1.17.0.tar.gz 23d85a2abfa81433049d7b1b0440b5b04ae3515830db0347da335a30b93c8be8e25d2f73e198cdb9207e2b51ce924f133bab7314cde76d80f73c51c1fd1c36c4 ngtcp2-nghttp3-v1.17.0.tar.gz.6300.part -https://github.com/ngtcp2/ngtcp2/archive/v1.24.0.tar.gz 04a5762d6eac7227431eb8e293d2786dee9f7d0467a584bdd35f4e33c7c6ff6e7b010e6e894fabdf780f6377373629c10a4d41effa374e2129d9f9b1e537873e ngtcp2-ngtcp2-v1.24.0.tar.gz.5038.part -https://github.com/ngtcp2/sfparse/archive/f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz b3cbcce6d96dc731d21a67940a05c1603d43ee9766819e1d174ca93c7afc7fa6247aa818368cafbb31444e0198ab5bba45808772a3ca89e459f8d2c0ffae6f5d ngtcp2-sfparse-f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz.6318.part -https://github.com/openssl/openssl/archive/openssl-3.6.3.tar.gz a89c08101fa1d7e3c09b14f4a90d450bcf336a4f6a3e6e4ea990e4deddcd9ce250472f9114438fd134ff4b47fe93dd47232308567088b2b1c0b2eb50e3b56bdf openssl-openssl-openssl-3.6.3.tar.gz.32544.part -https://github.com/pkgconf/pkgconf/archive/pkgconf-3.0.3.tar.gz 4e49bb7b10c6fdebfa1175b4c33138b8aeca2582e49adb5c7ba66d08e8614771ef060b0c2673bd098c1aed821c8ade1a6dd276c1f4ac6e29bd7efcb8f4c0402e pkgconf-pkgconf-pkgconf-3.0.3.tar.gz -https://github.com/pnggroup/libpng/archive/v1.6.58.tar.gz 65f54d805e1f7c46a5fc335b984e4cbd4f934e0f02fbf6673c13800b49a4c11fbeb4098eebfb33079527a56c3d933e97631f91ab68dbb31442982784f9241ace pnggroup-libpng-v1-65f54d80.6.58.tar.gz -https://github.com/rockdaboot/libpsl/archive/0.21.5.tar.gz d8e224b2ce5d9a6ac78700eb8975d09aef4e5af7db29539e5e339c5cd100f1272371fe45757ab5383ddbcd569bdf9d697a78932ea9fdf43ff48d3cea02f644cd rockdaboot-libpsl-0.21.5.tar.gz -https://github.com/simdjson/simdjson/archive/v4.6.4.tar.gz 003b96daab30ccaefdac60a9676cf623af5a8662016f988fc0ecaa2f36d8b48ba97f2431e6498dd16fc2b1d841798b2d06dabb0b7487efd4520c0de260e49056 simdjson-simdjson-v4.6.4.tar.gz -https://github.com/simdutf/simdutf/archive/v9.0.0.tar.gz 0c74226247cbe95368efa87ab84f5217485f16bcdf7a9def8741c6086cb86e6c378f0c437030d2be0934726e3ea9c28b5df2e593d0c654c78291c455a8d1e103 simdutf-simdutf-v9.0.0.tar.gz -https://github.com/tukaani-project/xz/archive/v5.8.3.tar.gz 8fb5e6a13397d259d8ff7484f9b63f8a6752ff1c63e1a4601170ad8175aadefb5126a1cae7f73370bfc6c2a0b4e1c0bad57a58fc5b781d3f7d45e5a483c091cc tukaani-project-xz-v5.8.3.tar.gz.19746.part -https://github.com/unicode-org/icu/releases/download/release-78.3/icu4c-78.3-sources.tgz 04a49455e1489030c520a4bfd2664fa2171e7938d08f2acdbbcb1fda976639fd8b1f0704f2eec89ba59a7b6d118ceaab6ec5a096e40d9085a0895d91ce225245 icu4c-78.3-sources.tgz -https://github.com/videolan/dav1d/archive/1.5.3.tar.gz 8d976b93135213d41385c20205475269a6826a68ebfd716c4d9a7a3ff2a79703e8df0573e43207c81b5db44807d2721db18ec84c0fc6bef98efab86a2cccb6cc videolan-dav1d-1.5.3.tar.gz -https://github.com/webmproject/libvpx/archive/v1.16.0.tar.gz 07f5e352411d6c0be331706d1835ac89bafbeddcbbac5542b473323766e9e974f4f68b33590f2aa50a7d8d69468a642b508cbb0a7c49a82c9933b07820f9c9d9 webmproject-libvpx-v1-07f5e352.16.0.tar.gz -https://github.com/webmproject/libwebp/archive/v1.6.0.tar.gz 298e0ad4c09392213baf5abb69d330c6203b618800073fe2df91d01d35034197c5d3e29a74573b06971473c52c74514f0e6e0f6c8162f923e2dd15cb1a692aef webmproject-libwebp-v1-298e0ad4.6.0.tar.gz -https://github.com/xiph/ogg/archive/v1.3.6.tar.gz c247e1da8b12f8b33272fafb6d7c171a1a2687c3632977439fa60b96ccc2ad751d88a2931bb3e18e1ddf2eea2e82cdd0aab087b2ec5393a9228c703476fa0167 xiph-ogg-v1.3.6.tar.gz -https://github.com/xiph/opus/archive/v1.5.2.tar.gz 4ffefd9c035671024f9720c5129bfe395dea04f0d6b730041c2804e89b1db6e4d19633ad1ae58855afc355034233537361e707f26dc53adac916554830038fab xiph-opus-v1.5.2.tar.gz -https://github.com/xiph/theora/archive/v1.2.0.tar.gz b2aac15528f0ef8258c0902e33e8211e8858c3c7e6e9eeb708cce5922de5f0e412255ddaf540a50c0ebf601df6c4376fd24a0bdd7f8de4432c4ae6e5d6ffe2b6 xiph-theora-v1.2.0.tar.gz -https://github.com/xiph/vorbis/archive/v1.3.7.tar.gz bfb6f5dbfd49ed38b2b08b3667c06d02e68f649068a050f21a3cc7e1e56b27afd546aaa3199c4f6448f03f6e66a82f9a9dc2241c826d3d1d4acbd38339b9e9fb xiph-vorbis-v1.3.7.tar.gz -https://gitlab.com/libtiff/libtiff/-/archive/v4.7.2/libtiff-v4.7.2.tar.gz c4dcde3c79e5d69c7231f8862e2e5a83d90d9cce694fb2a4804800b2f8f1bc9db504b9252d81dce872eec8358b33a3a1dbdddcbb6181f6fb8d1d7fc0e9a9fc6a libtiff-libtiff-v4.7.2.tar.gz.24666.part -https://gitlab.freedesktop.org//dbus/dbus/-/archive/dbus-1.16.2/dbus-dbus-1.16.2.tar.gz 8ad3ab55bf6e2bbe6ff871302c2840c0cb82b4ec785b05f146c577ca1e931825084012ac90251e28c30e44d111e5ca5711b29349f4f0e68a09ba49392e63ac89 dbus-dbus-dbus-1.16.2.tar.gz.14029.part -https://gitlab.freedesktop.org//freetype/freetype/-/archive/VER-2-13-3/freetype-VER-2-13-3.tar.gz fccfaa15eb79a105981bf634df34ac9ddf1c53550ec0b334903a1b21f9f8bf5eb2b3f9476e554afa112a0fca58ec85ab212d674dfd853670efec876bacbe8a53 freetype-freetype-VER-2-13-3.tar.gz -https://gitlab.freedesktop.org/fontconfig/fontconfig/-/archive/2.17.1/fontconfig-2.17.1.tar.gz 8e05cad63cd0c5ca15d1359e19a605912198fcc0ec6ecc11d5a0ef596d72e795cd8128e4d350716e63cbc01612c3807b1455b8153901333790316170c9ef8e75 fontconfig-fontconfig-2.17.1.tar.gz -https://raw.githubusercontent.com/publicsuffix/list/0ed17ee161ed2ae551c78f3b399ac8f2724d2154/public_suffix_list.dat 7969c40b0600baf2786af0e6503b4282d487b6603418c41f28c3b39e9cd9320ac66c0d2e8fbfa2b794e461f26843e3479d60ec24ac5c0990fe8f0c6bfaeee69d libpsl-public_suffix_list-0ed17e.dat -https://sourceforge.net/projects/giflib/files/giflib-6.1.3.tar.gz/download fb1d6319694745e8cdac7c57e96bd3a87dbfd978f2bfd00e826742db53398011c43f9a6e7f4375b0e77b162358ddfa14d85bce652680fb5967b72c46775c0edb giflib-6-fb1d6319.1.3.tar.gz -https://sourceforge.net/projects/libuuid/files/libuuid-1.0.3.tar.gz/download 77488caccc66503f6f2ded7bdfc4d3bc2c20b24a8dc95b2051633c695e99ec27876ffbafe38269b939826e1fdb06eea328f07b796c9e0aaca12331a787175507 libuuid-1.0.3.tar.gz -https://sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz 083f5e675d73f3233c7930ebe20425a533feedeaaa9d8cc86831312a6581cefbe6ed0d08d2fa89be81082f2a5abdabca8b3c080bf97218a1bd59dc118a30b9f3 bzip2-1.0.8.tar.gz -https://sqlite.org/2026/sqlite-autoconf-3530300.tar.gz 355a8db490ec2a68c2801644e56178a26416c355792586a6c1c904de116e26f8602bc344e7172181c9d92c4c9e696319243e16405460fad87b23ee997a3ef9da sqlite-autoconf-3530300.tar.gz -https://thrysoee.dk/editline/libedit-20240808-3.1.tar.gz b11d64947f9484bb2320b0fbcfdc94466993af1dfa0d853853b73c222e95d6c1e78d88d0c305929b95bf7a85009129475c9fef0ac8595b43d75543d85052a4ff libedit-20240808-3.1.tar.gz -https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/libidn/libidn2-2.3.7.tar.gz eab5702bc0baed45492f8dde43a4d2ea3560ad80645e5f9e0cfa8d3b57bccd7fd782d04638e000ba07924a5d9f85e760095b55189188c4017b94705bef9b4a66 libidn2-2.3.7.tar.gz -https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/libunistring/libunistring-1.2.tar.xz 5fbb5a0a864db73a6d18cdea7b31237da907fff0ef288f3a8db6ebdba8ef61ad8855e5fc780c2bbf632218d8fa59dd119734e5937ca64dc77f53f30f13b80b17 libunistring-1.2.tar.xz +https://chrome-infra-packages.appspot.com/dl/gn/gn/linux-amd64/+/fj2NZKMkIYZNH6uYG0bn8OsW_lZB5JKz3JeScMCLAGQC d49575bd383b6aace1257a6e9439ce0a206173ec2cab94d5312f06db412e09c89aa75b1f4c69f5dca4389d15a489c211a73439a66f437c34b18bc90eefa0b775 /home/ubuntu/vcap-downloads/gn-linux-amd64.zip.5420.part +https://downloads.sourceforge.net/project/libpng-apng/libpng16/1.6.58/libpng-1.6.58-apng.patch.gz 95a6f5bb7148b5c48dccd73811d7bcf9752a631a7bb4f4856670a7da12a7159581ac1bce1749318343794e0f5cb86972711ba2ec0f523c168f0991fa940687d5 /home/ubuntu/vcap-downloads/libpng-1.6.58-apng.patch.gz.23712.part +https://ftpmirror.gnu.org/gnu/automake/automake-1.17.tar.gz 11357dfab8cbf4b5d94d9d06e475732ca01df82bef1284888a34bd558afc37b1a239bed1b5eb18a9dbcc326344fb7b1b301f77bb8385131eb8e1e118b677883a /home/ubuntu/vcap-downloads/automake-1.17.tar.gz.29261.part +https://ftpmirror.gnu.org/gnu/gperf/gperf-3.3.tar.gz 246b75b8ce7d77d6a8725cd15f1cf2e68da404812573af1d5bf32dbe6ad4228f48757baefc77bcb1f5597c2397043c04d31d8a04ab507bfa7a80f85e1ab6045f /home/ubuntu/vcap-downloads/gperf-3.3.tar.gz.29537.part +https://github.com/AOMediaCodec/libavif/archive/v1.4.2.tar.gz cbf31827884058acc54d3b1fa0f2059f022691609a76e0f913981d5b8c1be60f52069a25b61c888c09e963161d1f4c9bf692c0716ab0ee067a4ccaa4e36d9ce1 /home/ubuntu/vcap-downloads/AOMediaCodec-libavif-v1.4.2.tar.gz.32757.part +https://github.com/GNOME/libxml2/archive/v2.15.3.tar.gz f65df793fca5e46552afbaa56b04c4774829a95e012a6dc4dc3d10e6884a6118e30a426e422516e1d21e91c4a1d34cb00a5cee61af35cab03c71fb9b5c09e138 /home/ubuntu/vcap-downloads/GNOME-libxml2-v2.15.3.tar.gz.2924.part +https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator/archive/v3.4.0.tar.gz 563acbcd8912d10d92c23715eba7acf0e7c1683af36021f415b36f359c2cce065f3906e395c32282a8410ec5c8179fbcb6412935c6629a49357475d4b4410e2a /home/ubuntu/vcap-downloads/GPUOpen-LibrariesAndSDKs-VulkanMemoryAllocator-v3.4.0.tar.gz.5104.part +https://github.com/KhronosGroup/EGL-Registry/archive/3ae2b7c48690d2ce13cc6db3db02dfc0572be65e.tar.gz c7b09ded4964fa427546bd345a29325105b79079b59642214dc8f04de113f42de2bc4272dbbbd4a801d92afc20297442fdfa12043a0900cf1e2b1cd83f260883 /tmp/vcap/downloads/KhronosGroup-EGL-Registry-3ae2b7c48690d2ce13cc6db3db02dfc0572be65e.tar.gz.4769.part +https://github.com/KhronosGroup/OpenGL-Registry/archive/0b449b97cdf1043eef5e1f0e235cbbab6ec10c86.tar.gz 148e1bfe4cc199bcc2c23b22d0b3e4988a29389d7f510ba4a6340672dbb7ab99bb836d2c08587499484df704d51a1adf4f0dc3a30d5ad8977ee0ad339163b17e /tmp/vcap/downloads/KhronosGroup-OpenGL-Registry-0b449b97cdf1043eef5e1f0e235cbbab6ec10c86.tar.gz.4954.part +https://github.com/KhronosGroup/Vulkan-Headers/archive/vulkan-sdk-1.4.350.1.tar.gz eee702ee5ff447986901e44989d96a5e38f7ef7f67e2015c8db6280251cb23b69cb9e1a0568b0a43dc3df820685079ed31998a1dc4704f5649768e26db5147b1 /home/ubuntu/vcap-downloads/KhronosGroup-Vulkan-Headers-vulkan-sdk-1.4.350.1.tar.gz.5226.part +https://github.com/Kitware/CMake/releases/download/v4.4.0/cmake-4.4.0-linux-x86_64.tar.gz 3df4aaa128a438ed48dcac7065fd355ff538eed8f394491298d0db63a891d671da247c8fa262e4fa6bf99429d630abab317d5a0248168fe203d1ca4978dab4da /tmp/vcap/downloads/cmake-4.4.0-linux-x86_64.tar.gz.3336.part +https://github.com/NixOS/patchelf/releases/download/0.19.0/patchelf-0.19.0-x86_64.tar.gz 2a65c9cbdddcc7952cdbd6e98a2cf3da01386cf0f0b927a6bbcfe8131ecf0bfb17c534246635b5e6a090652ee54c903f9f9c4f3f1d2412dba59f287ae2ae8070 /tmp/vcap/downloads/patchelf-0.19.0-x86_64.tar.gz.4596.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/ThirdParty/ANGLE/gni-to-cmake.py cf9dab1b59cd44b9ce05a2dc4636115e770dfe7d5cfbd90c3fef659628ac8155c57b866480f7cfe9a0afeb31ff5ce5eed74473f386120a7fc5910e8b032bd61d /home/ubuntu/vcap-downloads/gni-to-cmake.py.18491.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/ThirdParty/ANGLE/include/CMakeLists.txt a7ddf3c6df7565e232f87ec651cc4fd84240b8866609e23e3e6e41d22532fd34c70e0f3b06120fd3d6d930ca29c1d0d470d4c8cb7003a66f8c1a840a42f32949 /home/ubuntu/vcap-downloads/include_CMakeLists.txt.21622.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/DetectSSE2.cmake 219a4c8591ee31d11eb3d1e4803cc3c9d4573984bb25ecac6f2c76e6a3dab598c00b0157d0f94b18016de6786e49d8b29a161693a5ce23d761c8fe6a798c1bca /home/ubuntu/vcap-downloads/DetectSSE2.cmake.21654.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/WebKitCompilerFlags.cmake 8b281ffcf9209c845a5fdae48a4e05f08ca677c37a7fb00d9270de81bd103160d26e091724ce8df8d428ad604900b5202b221fed5bafffd4bf00025718ef9d8e /home/ubuntu/vcap-downloads/WebKitCompilerFlags.cmake.21638.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/WebKitMacros.cmake 565175443d5d1b8119af504164bf93840e8c786fc479e45feb98ca542351b91d2ea00265d7f8dfa6960975de81802bc43b2a2af2c90fc44bda1ae46d96c89247 /home/ubuntu/vcap-downloads/WebKitMacros.cmake.21670.part +https://github.com/abseil/abseil-cpp/archive/20260107.1.tar.gz f5012885d6b6844a9cf5ed92ad5468b8757db33dfe1364bfb232fff928e06c550c7eb4557f45186a8ac4d18b178df9be267681abab4a6de40823b574afbe9960 /home/ubuntu/vcap-downloads/abseil-abseil-cpp-20260107.1.tar.gz.5435.part +https://github.com/cisco/openh264/archive/v2.6.0.tar.gz 26a03acde7153a6b40b99f00641772433a244c72a3cc4bca6d903cf3b770174d028369a2fb73b2f0774e1124db0e269758eed6d88975347a815e0366c820d247 /home/ubuntu/vcap-downloads/cisco-openh264-v2.6.0.tar.gz.25254.part +https://github.com/curl/curl/archive/curl-8_21_0.tar.gz 0ab6c99c3d5b86fb65c526db517c3159b11db2f8d82552d635c4887059c0602288603c93b754ce0ec543ea2f275122ccec2c8dcd866c2611b5b949c728ee72df /home/ubuntu/vcap-downloads/curl-curl-curl-8_21_0.tar.gz.22159.part +https://github.com/davea42/libdwarf-code/archive/v2.3.1.tar.gz e8eb74c622cedb512d82dff313ce9c5ac2064a7a7a0691c776213b384c1c64d0f549aaab1ef732dcba8c72d52048cea641d9909d3b2503cf96760ad8e81ae77c /home/ubuntu/vcap-downloads/davea42-libdwarf-code-v2.3.1.tar.gz.18747.part +https://github.com/facebook/zstd/archive/v1.5.7.tar.gz 26e441267305f6e58080460f96ab98645219a90d290a533410b1b0b1d2f870721c95f8384e342ee647c5e968385a5b7e30c2d04340c37f59b3e6d86762c3260c /home/ubuntu/vcap-downloads/facebook-zstd-v1.5.7.tar.gz.18550.part +https://github.com/fastfloat/fast_float/archive/v8.2.10.tar.gz eec563358117b593e882a9069074a269d811e8989457a0d57e50c5d2f4d534b6820308082bb70c0a8a8388513e92b64f12c5213389eff726ad0483eacba961ff /home/ubuntu/vcap-downloads/fastfloat-fast_float-v8.2.10.tar.gz.22767.part +https://github.com/ffmpeg/ffmpeg/archive/n7.1.1.tar.gz 6b9a5ee501be41d6abc7579a106263b31f787321cbc45dedee97abf992bf8236cdb2394571dd256a74154f4a20018d429ae7e7f0409611ddc4d6f529d924d175 /home/ubuntu/vcap-downloads/ffmpeg-ffmpeg-n7.1.1.tar.gz.28355.part +https://github.com/fmtlib/fmt/archive/12.2.0.tar.gz 5ac2ba0f54a484999ed5407d82b77aad170cea49a267decd2c0eedadf3b14413e2a83fcc8e9ca9c16640595e019b8636e160f72314d8be50653324e82ac745eb /home/ubuntu/vcap-downloads/fmtlib-fmt-12.2.0.tar.gz.29208.part +https://github.com/fmtlib/fmt/commit/588b3a0f8f6a8bcf2a959cae882d5b2703e86737.patch?full_index=1 afda8fdfcdcb4b0dd5df4d4dae96a57a85fb9c4b65d0b49d51258f0913d4aed93ed146ebf96ed7b277490b1dde6c7117f43332013071441a96c3147520de8368 /home/ubuntu/vcap-downloads/fmt-backport-4813.patch.29193.part +https://github.com/google/angle/archive/79ac1a8cd767a32cce6401203e20c4bd4ca4d539.tar.gz a3d5d09460f05f1f1c081411bf15f585bb8add5f3db7c4854dc2cf2ef1fd22dd5b2dd5d3de78bfb8bcdae8459dd9581f7dce331acd34053c87c84cfe6c3d134c /home/ubuntu/vcap-downloads/google-angle-79ac1a8cd767a32cce6401203e20c4bd4ca4d539.tar.gz.17652.part +https://github.com/google/brotli/archive/v1.2.0.tar.gz f94542afd2ecd96cc41fd21a805a3da314281ae558c10650f3e6d9ca732b8425bba8fde312823f0a564c7de3993bdaab5b43378edab65ebb798cefb6fd702256 /home/ubuntu/vcap-downloads/google-brotli-v1.2.0.tar.gz.18515.part +https://github.com/google/highway/archive/1.4.0.tar.gz 819422857d6a74e3a936c402698e078db5b7b88fb43767e62429ec7bd954fe93b017e75029a4df4a1a97ef3a2486107eef5247da751eb487640dea409f3f2fa2 /home/ubuntu/vcap-downloads/google-highway-1.4.0.tar.gz.1576.part +https://github.com/google/skia/archive/e7c90ecca9444fe09598f1630ab7cee2c0ee027a.tar.gz f52286fcac1d1b2b45046d7df72cee384b664f0b512c4b4a76bca743747f2113fef6991059b4e71ad804d0209103003ba0f01aa90afac786de20a52e8484018d /home/ubuntu/vcap-downloads/google-skia-e7c90ecca9444fe09598f1630ab7cee2c0ee027a.tar.gz.5591.part +https://github.com/google/woff2/archive/v1.0.2.tar.gz c788bba1530aec463e755e901f9342f4b599e3a07f54645fef1dc388ab5d5c30625535e5dd38e9e792e04a640574baa50eeefb6b7338ab403755f4a4e0c3044d /home/ubuntu/vcap-downloads/google-woff2-v1.0.2.tar.gz.10089.part +https://github.com/google/wuffs-mirror-release-c/archive/v0.3.4.tar.gz d22136a1adf337573944eed917142c6bde877a09bd65738010c6c367c7a3fc9e4573fd4dc8469c93799fe1e3247760e65e64829e830871ee7b333fd72ebc629d /home/ubuntu/vcap-downloads/google-wuffs-mirror-release-c-v0.3.4.tar.gz.10113.part +https://github.com/harfbuzz/harfbuzz/archive/10.2.0.tar.gz 697205a571bb3d52d83598e8511e2e21e7cd15630aac32d8deb4354e462efda6a5ce46510cf4a1c18365dffe935cf2e4f1fda65d1779f17c9bb60c503315bf5c /home/ubuntu/vcap-downloads/harfbuzz-harfbuzz-10.2.0.tar.gz.30771.part +https://github.com/jeremy-rifkin/cpptrace/archive/v1.0.2.tar.gz 4ae394fb3c21149bf2441a754eebe639e6a5534927426b6507806c7bee0b1c982e047c972904d472f1c660adb5be3881e7e3a6eddd18e4e9d376ae3855d50a7c /home/ubuntu/vcap-downloads/jeremy-rifkin-cpptrace-v1.0.2.tar.gz.18974.part +https://github.com/libexpat/libexpat/archive/R_2_8_2.tar.gz e60e6d6ae9d0115f41186f06f3854008054863f0b29a58d44142f5b30057a494337145d820b5e18270b8ca3e779e318a757fcc6bf64d6d204e5498df5eeb2195 /home/ubuntu/vcap-downloads/libexpat-libexpat-R_2_8_2.tar.gz.22302.part +https://github.com/libjpeg-turbo/libjpeg-turbo/archive/3.2.0.tar.gz 1ce063e9e126d55019385da3a6ff4521a9a3958edeab15e2465ae3435026dac6e598277eea066173ada47da37159d0f7812b2227869c10dd141dcfe2ddeab720 /home/ubuntu/vcap-downloads/libjpeg-turbo-libjpeg-turbo-3.2.0.tar.gz.23084.part +https://github.com/libjxl/libjxl/archive/v0.11.2.tar.gz a7e1f7d060b358f4382e84367d66aa2850aef3b4524a0fdfe3f22dd258fb9e35dda7540f859d8bf4c32f31c61a7a03db677f4490a9f472cd25869a9d00797336 /home/ubuntu/vcap-downloads/libjxl-libjxl-v0.11.2.tar.gz.1858.part +https://github.com/libproxy/libproxy/archive/0.4.18.tar.gz 1148d688a9f070273a1a2b110a788561789799089660292bbba59fbf0a9caf7d28cb039a9ccdcb935f752e1e34739b2d2f4c784b1bb3bbaa03d108e7b38a4754 /home/ubuntu/vcap-downloads/libproxy-libproxy-0.4.18.tar.gz.1996.part +https://github.com/libsdl-org/SDL/archive/release-3.2.28.tar.gz 9e188c992caa7f7ff030789f7926007d2272f51b4c3aa7fc94f58f6823810bde71ce149990c78eee47f26471df2a7b87d4fc25881c339ed2026b1e59052bce39 /home/ubuntu/vcap-downloads/libsdl-org-SDL-release-3.2.28.tar.gz.3504.part +https://github.com/libtom/libtommath/archive/v1.3.0.tar.gz 3dbd7053a670afa563a069a9785f1aa4cab14a210bcd05d8fc7db25bd3dcce36b10a3f4f54ca92d75a694f891226f01bdf6ac15bacafeb93a8be6b04c579beb3 /home/ubuntu/vcap-downloads/libtom-libtommath-v1.3.0.tar.gz.2886.part +https://github.com/madler/zlib/archive/v1.3.1.tar.gz 8c9642495bafd6fad4ab9fb67f09b268c69ff9af0f4f20cf15dfc18852ff1f312bd8ca41de761b3f8d8e90e77d79f2ccacd3d4c5b19e475ecf09d021fdfe9088 /tmp/vcap/downloads/madler-zlib-v1.3.1.tar.gz.4636.part +https://github.com/mesonbuild/meson/archive/1.9.0.tar.gz ecd69b6734be14c467f7db67dd88c0e57ebfad83ce3ddada131ff3e43ac964523e1083d7c7f3803033a9a76adbc32ad26dd2e3aca69884269000ca64130bde07 /home/ubuntu/vcap-downloads/meson-1.9.0.tar.gz.24958.part +https://github.com/microsoft/mimalloc/archive/v2.2.7.tar.gz 19f5481c98822c6a279e2f2d26d6c54918e18bcafd515a76ca3a3845f3a5e599236ea042f9cadaf793dce0c1ede347d76cc2fc5503094288362a78034d4d8f71 /home/ubuntu/vcap-downloads/microsoft-mimalloc-v2.2.7.tar.gz.3120.part +https://github.com/mm2/Little-CMS/archive/lcms2.19.1.tar.gz 1b2781ed8898e65f15be17cf0130a1500ec0bf5ca5159f871dff5692e387747be9526feef0bc7c370200656fc0aabe3036746041285a3978e90adec200d685f2 /home/ubuntu/vcap-downloads/mm2-Little-CMS-lcms2.19.1.tar.gz.1262.part +https://github.com/mozilla/pdf.js/releases/download/v5.6.205/pdfjs-5.6.205-dist.zip 66fecdb8a80d013b592c361e85abd7eeea5bc35f0131b771cc25a1878d7737704458dc715670aace3c7d3437f85388dff04796fe117eef01be81fd0d192f73a2 /home/ubuntu/vcap-downloads/pdfjs-5.6.205-dist.zip.3253.part +https://github.com/nghttp2/nghttp2/archive/v1.69.0.tar.gz 1029fb86935a88fc518cc2d976dff5253277f97e01f32b1f73c5df96dcf7fe0280a83a9e9d676c3d96caa542300dc7cfa61f2a40b1b11d2c2b527e870a974b53 /home/ubuntu/vcap-downloads/nghttp2-nghttp2-v1.69.0.tar.gz.22093.part +https://github.com/ngtcp2/nghttp3/archive/v1.17.0.tar.gz 23d85a2abfa81433049d7b1b0440b5b04ae3515830db0347da335a30b93c8be8e25d2f73e198cdb9207e2b51ce924f133bab7314cde76d80f73c51c1fd1c36c4 /home/ubuntu/vcap-downloads/ngtcp2-nghttp3-v1.17.0.tar.gz.22018.part +https://github.com/ngtcp2/ngtcp2/archive/v1.24.0.tar.gz 04a5762d6eac7227431eb8e293d2786dee9f7d0467a584bdd35f4e33c7c6ff6e7b010e6e894fabdf780f6377373629c10a4d41effa374e2129d9f9b1e537873e /home/ubuntu/vcap-downloads/ngtcp2-ngtcp2-v1.24.0.tar.gz.21976.part +https://github.com/ngtcp2/sfparse/archive/f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz b3cbcce6d96dc731d21a67940a05c1603d43ee9766819e1d174ca93c7afc7fa6247aa818368cafbb31444e0198ab5bba45808772a3ca89e459f8d2c0ffae6f5d /home/ubuntu/vcap-downloads/ngtcp2-sfparse-f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz.22045.part +https://github.com/openssl/openssl/archive/openssl-3.6.3.tar.gz a89c08101fa1d7e3c09b14f4a90d450bcf336a4f6a3e6e4ea990e4deddcd9ce250472f9114438fd134ff4b47fe93dd47232308567088b2b1c0b2eb50e3b56bdf /home/ubuntu/vcap-downloads/openssl-openssl-openssl-3.6.3.tar.gz.19041.part +https://github.com/pkgconf/pkgconf/archive/pkgconf-3.0.3.tar.gz 4e49bb7b10c6fdebfa1175b4c33138b8aeca2582e49adb5c7ba66d08e8614771ef060b0c2673bd098c1aed821c8ade1a6dd276c1f4ac6e29bd7efcb8f4c0402e /home/ubuntu/vcap-downloads/pkgconf-pkgconf-pkgconf-3.0.3.tar.gz.28330.part +https://github.com/pnggroup/libpng/archive/v1.6.58.tar.gz 65f54d805e1f7c46a5fc335b984e4cbd4f934e0f02fbf6673c13800b49a4c11fbeb4098eebfb33079527a56c3d933e97631f91ab68dbb31442982784f9241ace /home/ubuntu/vcap-downloads/pnggroup-libpng-v1.6.58.tar.gz.23741.part +https://github.com/rockdaboot/libpsl/archive/0.21.5.tar.gz d8e224b2ce5d9a6ac78700eb8975d09aef4e5af7db29539e5e339c5cd100f1272371fe45757ab5383ddbcd569bdf9d697a78932ea9fdf43ff48d3cea02f644cd /home/ubuntu/vcap-downloads/rockdaboot-libpsl-0.21.5.tar.gz.2439.part +https://github.com/simdjson/simdjson/archive/v4.6.4.tar.gz 003b96daab30ccaefdac60a9676cf623af5a8662016f988fc0ecaa2f36d8b48ba97f2431e6498dd16fc2b1d841798b2d06dabb0b7487efd4520c0de260e49056 /home/ubuntu/vcap-downloads/simdjson-simdjson-v4.6.4.tar.gz.4530.part +https://github.com/simdutf/simdutf/archive/v9.0.0.tar.gz 0c74226247cbe95368efa87ab84f5217485f16bcdf7a9def8741c6086cb86e6c378f0c437030d2be0934726e3ea9c28b5df2e593d0c654c78291c455a8d1e103 /home/ubuntu/vcap-downloads/simdutf-simdutf-v9.0.0.tar.gz.4952.part +https://github.com/tukaani-project/xz/archive/v5.8.3.tar.gz 8fb5e6a13397d259d8ff7484f9b63f8a6752ff1c63e1a4601170ad8175aadefb5126a1cae7f73370bfc6c2a0b4e1c0bad57a58fc5b781d3f7d45e5a483c091cc /home/ubuntu/vcap-downloads/tukaani-project-xz-v5.8.3.tar.gz.22800.part +https://github.com/unicode-org/icu/releases/download/release-78.3/icu4c-78.3-sources.tgz 04a49455e1489030c520a4bfd2664fa2171e7938d08f2acdbbcb1fda976639fd8b1f0704f2eec89ba59a7b6d118ceaab6ec5a096e40d9085a0895d91ce225245 /home/ubuntu/vcap-downloads/icu4c-78.3-sources.tgz.30283.part +https://github.com/videolan/dav1d/archive/1.5.3.tar.gz 8d976b93135213d41385c20205475269a6826a68ebfd716c4d9a7a3ff2a79703e8df0573e43207c81b5db44807d2721db18ec84c0fc6bef98efab86a2cccb6cc /home/ubuntu/vcap-downloads/videolan-dav1d-1.5.3.tar.gz.28218.part +https://github.com/webmproject/libvpx/archive/v1.16.0.tar.gz 07f5e352411d6c0be331706d1835ac89bafbeddcbbac5542b473323766e9e974f4f68b33590f2aa50a7d8d69468a642b508cbb0a7c49a82c9933b07820f9c9d9 /home/ubuntu/vcap-downloads/webmproject-libvpx-v1.16.0.tar.gz.24198.part +https://github.com/webmproject/libwebp/archive/v1.6.0.tar.gz 298e0ad4c09392213baf5abb69d330c6203b618800073fe2df91d01d35034197c5d3e29a74573b06971473c52c74514f0e6e0f6c8162f923e2dd15cb1a692aef /home/ubuntu/vcap-downloads/webmproject-libwebp-v1.6.0.tar.gz.23963.part +https://github.com/xiph/ogg/archive/v1.3.6.tar.gz c247e1da8b12f8b33272fafb6d7c171a1a2687c3632977439fa60b96ccc2ad751d88a2931bb3e18e1ddf2eea2e82cdd0aab087b2ec5393a9228c703476fa0167 /home/ubuntu/vcap-downloads/xiph-ogg-v1.3.6.tar.gz.24475.part +https://github.com/xiph/opus/archive/v1.5.2.tar.gz 4ffefd9c035671024f9720c5129bfe395dea04f0d6b730041c2804e89b1db6e4d19633ad1ae58855afc355034233537361e707f26dc53adac916554830038fab /home/ubuntu/vcap-downloads/xiph-opus-v1.5.2.tar.gz.24725.part +https://github.com/xiph/theora/archive/v1.2.0.tar.gz b2aac15528f0ef8258c0902e33e8211e8858c3c7e6e9eeb708cce5922de5f0e412255ddaf540a50c0ebf601df6c4376fd24a0bdd7f8de4432c4ae6e5d6ffe2b6 /home/ubuntu/vcap-downloads/xiph-theora-v1.2.0.tar.gz.24593.part +https://github.com/xiph/vorbis/archive/v1.3.7.tar.gz bfb6f5dbfd49ed38b2b08b3667c06d02e68f649068a050f21a3cc7e1e56b27afd546aaa3199c4f6448f03f6e66a82f9a9dc2241c826d3d1d4acbd38339b9e9fb /home/ubuntu/vcap-downloads/xiph-vorbis-v1.3.7.tar.gz.24507.part +https://gitlab.com/libtiff/libtiff/-/archive/v4.7.2/libtiff-v4.7.2.tar.gz c4dcde3c79e5d69c7231f8862e2e5a83d90d9cce694fb2a4804800b2f8f1bc9db504b9252d81dce872eec8358b33a3a1dbdddcbb6181f6fb8d1d7fc0e9a9fc6a /home/ubuntu/vcap-downloads/libtiff-libtiff-v4.7.2.tar.gz.23688.part +https://gitlab.freedesktop.org//dbus/dbus/-/archive/dbus-1.16.2/dbus-dbus-1.16.2.tar.gz 8ad3ab55bf6e2bbe6ff871302c2840c0cb82b4ec785b05f146c577ca1e931825084012ac90251e28c30e44d111e5ca5711b29349f4f0e68a09ba49392e63ac89 /home/ubuntu/vcap-downloads/dbus-dbus-dbus-1.16.2.tar.gz.22755.part +https://gitlab.freedesktop.org//freetype/freetype/-/archive/VER-2-13-3/freetype-VER-2-13-3.tar.gz fccfaa15eb79a105981bf634df34ac9ddf1c53550ec0b334903a1b21f9f8bf5eb2b3f9476e554afa112a0fca58ec85ab212d674dfd853670efec876bacbe8a53 /home/ubuntu/vcap-downloads/freetype-freetype-VER-2-13-3.tar.gz.30169.part +https://gitlab.freedesktop.org/fontconfig/fontconfig/-/archive/2.17.1/fontconfig-2.17.1.tar.gz 8e05cad63cd0c5ca15d1359e19a605912198fcc0ec6ecc11d5a0ef596d72e795cd8128e4d350716e63cbc01612c3807b1455b8153901333790316170c9ef8e75 /home/ubuntu/vcap-downloads/fontconfig-fontconfig-2.17.1.tar.gz.30257.part +https://raw.githubusercontent.com/publicsuffix/list/0ed17ee161ed2ae551c78f3b399ac8f2724d2154/public_suffix_list.dat 7969c40b0600baf2786af0e6503b4282d487b6603418c41f28c3b39e9cd9320ac66c0d2e8fbfa2b794e461f26843e3479d60ec24ac5c0990fe8f0c6bfaeee69d /home/ubuntu/vcap-downloads/libpsl-public_suffix_list-0ed17e.dat.2866.part +https://sourceforge.net/projects/giflib/files/giflib-6.1.3.tar.gz/download fb1d6319694745e8cdac7c57e96bd3a87dbfd978f2bfd00e826742db53398011c43f9a6e7f4375b0e77b162358ddfa14d85bce652680fb5967b72c46775c0edb /home/ubuntu/vcap-downloads/giflib-6.1.3.tar.gz.23839.part +https://sourceforge.net/projects/libuuid/files/libuuid-1.0.3.tar.gz/download 77488caccc66503f6f2ded7bdfc4d3bc2c20b24a8dc95b2051633c695e99ec27876ffbafe38269b939826e1fdb06eea328f07b796c9e0aaca12331a787175507 /home/ubuntu/vcap-downloads/libuuid-1.0.3.tar.gz.29486.part +https://sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz 083f5e675d73f3233c7930ebe20425a533feedeaaa9d8cc86831312a6581cefbe6ed0d08d2fa89be81082f2a5abdabca8b3c080bf97218a1bd59dc118a30b9f3 /home/ubuntu/vcap-downloads/bzip2-1.0.8.tar.gz.29977.part +https://sqlite.org/2026/sqlite-autoconf-3530300.tar.gz 355a8db490ec2a68c2801644e56178a26416c355792586a6c1c904de116e26f8602bc344e7172181c9d92c4c9e696319243e16405460fad87b23ee997a3ef9da /home/ubuntu/vcap-downloads/sqlite-autoconf-3530300.tar.gz.9370.part +https://thrysoee.dk/editline/libedit-20240808-3.1.tar.gz b11d64947f9484bb2320b0fbcfdc94466993af1dfa0d853853b73c222e95d6c1e78d88d0c305929b95bf7a85009129475c9fef0ac8595b43d75543d85052a4ff /home/ubuntu/vcap-downloads/libedit-20240808-3.1.tar.gz.1211.part +https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/libidn/libidn2-2.3.7.tar.gz eab5702bc0baed45492f8dde43a4d2ea3560ad80645e5f9e0cfa8d3b57bccd7fd782d04638e000ba07924a5d9f85e760095b55189188c4017b94705bef9b4a66 /home/ubuntu/vcap-downloads/libidn2-2.3.7.tar.gz.2269.part +https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/libunistring/libunistring-1.2.tar.xz 5fbb5a0a864db73a6d18cdea7b31237da907fff0ef288f3a8db6ebdba8ef61ad8855e5fc780c2bbf632218d8fa59dd119734e5937ca64dc77f53f30f13b80b17 /home/ubuntu/vcap-downloads/libunistring-1.2.tar.xz.2021.part diff --git a/examples/ladybird/workspace/Meta/vcpkg_build.sh b/examples/ladybird/workspace/Meta/vcpkg_build.sh index d81257d..6151b06 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_build.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_build.sh @@ -51,9 +51,57 @@ INDEX="${2:?distfile index}" VCPKG_TREE="${3:?vcpkg checkout}" SRC="${4:?ladybird source root}" TRIPLET="${5:-x64-linux-dynamic}" +WHEELS="${6-}" +HOST_TOOLS="${7-}" -WORK="$(mktemp -d)" -trap 'rm -rf "$WORK"' EXIT +# vcpkg's buildtrees peak around 3 GB and $TMPDIR is frequently a tmpfs sized well +# under that (7.9 GB here, shared with everything else), which surfaces as +# `cp: error writing ...: No space left on device` from the ASSET SCRIPT -- a +# message that blames the pin for a full disk. +# +# So do not use $TMPDIR at all: put the scratch dir NEXT TO THE DECLARED OUTPUT, +# which is inside bazel-out and therefore on whatever real filesystem the output +# base lives on. That also removes a flag from the recipe -- pointing $TMPDIR +# somewhere bigger needs BOTH --action_env and --host_action_env (this action runs +# in the target AND exec configurations, finding 26's duplication again), and an +# `env =` on the rule silently beats --action_env anyway, so the flag route is two +# ways wrong. Report the free space either way: a 3 GB build on a 2 GB disk should +# say so in its own voice rather than 20 minutes later in someone else's. +# mkdir first: Bazel creates the declared output dir but not necessarily its parent +# before the action runs, and mktemp does not create intermediate dirs. Absolute, +# because vcpkg rejects a relative $HOME outright ("was not an absolute path") and +# $OUT arrives execroot-relative -- the same relative-vs-absolute distinction that +# broke the asset index above, in a third place. +mkdir -p "$(dirname "$OUT")" +OUT_PARENT="$(cd "$(dirname "$OUT")" && pwd)" +WORK="$(mktemp -d "$OUT_PARENT/vcpkg-scratch.XXXXXX")" +echo "vcpkg_build: scratch dir $WORK ($(df -h "$WORK" | awk 'NR==2{print $4}') free)" >&2 +# Cleaned up on the way out INCLUDING on failure -- an interrupted run used to +# leave a multi-GB tree behind, and several of them is how the disk filled. +trap 'rm -rf "$WORK"' EXIT INT TERM + +# The index arrives as an EXECROOT-RELATIVE path (bazel-out/.../x.index), and the +# asset-cache script below is run by vcpkg from vcpkg's own working directory, not +# ours -- so a relative path there resolves against the wrong dir and awk reports +# "cannot open". Absolutize it here, once, while we are still in the execroot. +# +# Why this survived so long: on the machine this was developed on, the vcpkg +# checkout already had downloads/tools/cmake-4.4.0-linux populated by an earlier +# `Meta/ladybird.py vcpkg` run, so vcpkg found its tools locally and never invoked +# the asset script for one. Only a clone with a pristine checkout makes vcpkg ask +# for cmake -- and then the script fails, is reported as "no asset cache hits", +# and x-block-origin correctly refuses to reach the network. +# +# The index's VALUES are execroot-relative for the same reason, so rewrite them +# to absolute here too: awk finding the row is only half the job, and `cp` from a +# relative path fails identically once vcpkg has chdir'd. Both halves have to be +# absolutized in the same place, or the failure just moves one line down. +EXECROOT="$PWD" +INDEX_ABS="$WORK/index" +awk -v root="$EXECROOT" \ + '{ if ($2 ~ /^\//) print $1, $2; else print $1, root "/" $2 }' \ + "$INDEX" > "$INDEX_ABS" +INDEX="$INDEX_ABS" # vcpkg reads $HOME (for its own config/telemetry dirs) and hard-fails "unable to # read $HOME" without it. Bazel deliberately does not pass HOME through to @@ -64,6 +112,40 @@ trap 'rm -rf "$WORK"' EXIT export HOME="$WORK/home" mkdir -p "$HOME" +# --- pip: offline, from Bazel-fetched wheels only ---------------------------- +# +# vcpkg's asset cache does NOT cover pip. The angle overlay-port calls +# x_vcpkg_get_python_packages, which runs `pip install ply` inside a venv, and +# neither x-script nor x-block-origin ever sees the request -- so the 76-distfile +# pin says nothing about it and, with this action running `no-sandbox` and +# inheriting the shell env, pip happily used the caller's HTTP_PROXY for months +# (finding 36). `requires-network: "0"` did not stop it: that is a scheduling hint, +# not a namespace. +# +# pip's own offline switches are the fix, so no portfile patch is needed: +# PIP_NO_INDEX forbids talking to an index at all, PIP_FIND_LINKS points at a +# directory of wheels Bazel fetched by URL+hash. A package that is not pinned then +# fails with "No matching distribution found", which is the pip-side equivalent of +# x-block-origin -- an error instead of a download. +# +# Unset the proxy variables too. Leaving them would make the *failure* mode depend +# on the caller's environment: --no-index means pip should not reach an index, and +# an inherited proxy is exactly how this went unnoticed. Belt and braces, because +# the lesson of finding 36 is that one unenforced control is not a control. +FINDLINKS="$WORK/wheels" +mkdir -p "$FINDLINKS" +if [ -n "$WHEELS" ]; then + IFS=',' read -r -a _wheels <<< "$WHEELS" + for w in "${_wheels[@]}"; do + [ -n "$w" ] && cp "$w" "$FINDLINKS/" + done +fi +export PIP_NO_INDEX=1 +export PIP_FIND_LINKS="$FINDLINKS" +export PIP_DISABLE_PIP_VERSION_CHECK=1 +unset HTTP_PROXY HTTPS_PROXY http_proxy https_proxy ALL_PROXY all_proxy +echo "vcpkg_build: pip is offline; $(ls "$FINDLINKS" | wc -l) pinned wheel(s)" >&2 + # --- the asset-cache script: resolve by hash, never fetch -------------------- cat > "$WORK/fetch.sh" < /dev/null 2>&1; then found=1; break; fi + IFS='|' + done + IFS="$old_ifs" + if [ -z "$found" ]; then + echo "vcpkg_build: MISSING host tool: $names (apt: $apt)" >&2 + echo "vcpkg_build: needed by: $users" >&2 + missing_pkgs="$missing_pkgs $apt" + fi + done < "$HOST_TOOLS" + if [ -n "$missing_pkgs" ]; then + echo "vcpkg_build:" >&2 + echo "vcpkg_build: vcpkg has NO download for these on Linux -- it probes the" >&2 + echo "vcpkg_build: host and hard-fails partway through the build. Install them:" >&2 + echo "vcpkg_build:" >&2 + echo "vcpkg_build: sudo apt install$missing_pkgs${unprobeable}" >&2 + echo "vcpkg_build:" >&2 + echo "vcpkg_build: (list: Meta/vcpkg_host_tools.tsv, regenerate with" >&2 + echo "vcpkg_build: Meta/emit_vcpkg_bazel.py --host-tools)" >&2 + exit 1 + fi + [ -n "$unprobeable" ] && echo \ + "vcpkg_build: note: cannot verify$unprobeable (no binary to probe for)" >&2 +fi + # --- a writable vcpkg root (vcpkg writes buildtrees/downloads/packages) ----- ROOT="$WORK/root" mkdir -p "$ROOT" @@ -85,9 +230,35 @@ rm -rf "$ROOT/installed" "$ROOT/packages" "$ROOT/buildtrees" mkdir -p "$ROOT/downloads" # Pre-place the git-sourced externals (see note 5). -if [ -d "$SRC/Meta/CMake/vcpkg/git-archives" ]; then - cp "$SRC/Meta/CMake/vcpkg/git-archives/"*.tar.gz "$ROOT/downloads/" 2>/dev/null || true +# +# This used to be `if [ -d ... ]; then cp ... 2>/dev/null || true; fi` -- three +# separate ways to succeed while copying nothing, in four lines. On the machine +# that developed this, the directory existed because I had created it by hand; on a +# fresh clone it does not, so all four archives were silently absent and the build +# failed ~20 minutes later inside skia's portfile with +# `git fetch https://android.googlesource.com/.../piex.git ... Error code: 128`, +# naming neither this directory nor the tarball. Exactly the finding-35 shape: a +# copy that cannot fail is indistinguishable from a copy that is not needed. +# +# So: fail here, naming what is missing and where it comes from. VCPKG_GIT_ARCHIVES +# in vcpkg_git_archives.bzl lists the four expected names -- that file is GENERATED +# and checked in, and (finding 36) is currently loaded by nothing at all, which is +# the deeper bug this only reports. These are `git archive` output, so unlike the 76 +# distfiles they have no URL to http_file -- but reproducing them is a PREFETCH, not a +# repo rule: clone the pinned URL and `git archive` the pinned ref, which reproduces +# the committed SHA512 exactly (verified on libyuv). A repo rule here would be a fork +# of tooling the project already owns (gap 7). +GIT_ARCHIVES="$SRC/Meta/CMake/vcpkg/git-archives" +if ! compgen -G "$GIT_ARCHIVES/*.tar.gz" > /dev/null; then + echo "vcpkg_build: no git-sourced externals at $GIT_ARCHIVES" >&2 + echo "vcpkg_build: 4 are required (see VCPKG_GIT_ARCHIVES in" >&2 + echo "vcpkg_build: vcpkg_git_archives.bzl). vcpkg_from_git bypasses the asset" >&2 + echo "vcpkg_build: cache entirely, so without them skia and angle will try to" >&2 + echo "vcpkg_build: 'git fetch' and fail ~20 minutes into the build with an" >&2 + echo "vcpkg_build: error naming a googlesource URL rather than this directory." >&2 + exit 1 fi +cp "$GIT_ARCHIVES"/*.tar.gz "$ROOT/downloads/" # --- manifest, verbatim (note 2) -------------------------------------------- MANIFEST="$WORK/manifest" diff --git a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh index 9755d74..1e0d19a 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh @@ -21,11 +21,32 @@ # the pin. # # Usage: vcpkg_capture_assets.sh [vcpkg install args...] +# +# This does a FULL vcpkg build (~50 min), because a download-only run cannot reach +# every download -- see the CAPTURE_ONLY_DOWNLOADS comment below the recorder. set -euo pipefail OUT="${1:?usage: vcpkg_capture_assets.sh [vcpkg args...]}" shift -: > "$OUT" +# APPEND to an existing capture rather than truncating it, and share vcpkg's +# downloads/ across runs (pass --downloads-root at a stable path). The run takes +# tens of minutes and reaches the network for every byte, so it WILL be +# interrupted -- a sandbox restart, a dead mirror, a timeout. Truncating means +# every interruption costs the whole run, which is how the first two attempts +# died (todo c2affe6b: long jobs must be resumable and append-only). The final +# `sort -u` dedupes, so a tuple recorded twice is free; a tuple recorded once and +# then thrown away is another 40 minutes. +touch "$OUT" +echo "capture: appending to $OUT ($(wc -l < "$OUT") rows already recorded)" >&2 + +# Where the recorder reports downloads it could not complete. A file rather than a +# counter, because the recorder runs as a separate PROCESS per download: nothing +# it sets in a variable can reach this script. +FAILED=$(mktemp /tmp/vcpkg-capture-failed-XXXXXX) +# vcpkg's own output, kept so the driver can detect a HALTED portfile -- which +# loses downloads exactly like a failed fetch, but produces no asset-script call +# (the step was never reached) and no nonzero exit. +VCPKG_LOG=$(mktemp /tmp/vcpkg-capture-log-XXXXXX) REC=$(mktemp /tmp/vcpkg-record-XXXXXX.sh) cat > "$REC" <> "$OUT" # Record the tuple regardless (the SHA512 is what matters and it is # mirror-independent), and try the known GNU mirrors before giving up. Hit live: # ftpmirror.gnu.org returned 502 for ~13 minutes and wedged the capture. -if curl -sSL --fail --max-time 120 -o "\$3" "\$1"; then exit 0; fi +# Bound STALLS, not total transfer time. This was --max-time 120, which is a cap +# on how long a download may legitimately take -- so it killed OpenGL-Registry at +# 22MB of a working transfer, then reported it as "FAILED to fetch", then fell +# through to the origin, on a repeat, forever: a capture that cannot finish and +# blames the mirror. --speed-time/--speed-limit is the property actually wanted +# ("no progress for 60s"), and it cannot mistake a big file for a dead one. +if curl -sSL --fail --speed-time 60 --speed-limit 1024 -o "\$3" "\$1"; then exit 0; fi alt=\$(printf '%s' "\$1" | sed \ -e 's|https://ftpmirror.gnu.org/gnu/|https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/|' \ -e 's|https://ftp.gnu.org/pub/gnu/|https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/|') if [ "\$alt" != "\$1" ]; then echo "capture: primary failed, trying mirror \$alt" >&2 - curl -sSL --fail --max-time 120 -o "\$3" "\$alt" && exit 0 + curl -sSL --fail --speed-time 60 --speed-limit 1024 -o "\$3" "\$alt" && exit 0 fi # Leave NO partial file behind. curl -o creates the destination before it knows # the transfer will fail, and vcpkg's downloads/ is keyed by name: a surviving @@ -61,25 +88,206 @@ fi # renamed four unrelated distfiles in the capture. (finding 30) rm -f "\$3" echo "capture: FAILED to fetch \$1" >&2 +# Tell the DRIVER, not just the log. A failed download halts its portfile, so +# every vcpkg_download_distfile after it in that port is never requested and so +# never captured -- and vcpkg still exits 0 saying "All requested installations +# completed successfully". Without this file the driver cannot know. +printf '%s\n' "\$1" >> "$FAILED" exit 1 EOF chmod +x "$REC" # Deliberately NO x-block-origin here: this is the one run allowed to fetch. # -# --only-downloads: a capture wants the FETCHES, not the 45-minute build. Without -# it the capture takes ~50 minutes and every interruption costs the whole run -# (which is how the first two attempts died); with it the same 76 distfiles come -# down in 2 minutes. vcpkg has no fetch-only mode for the *ports* -- this is the -# closest thing, and it is enough because the asset hook fires during resolution. +# This run does the FULL BUILD, and that is not a preference -- `--only-downloads` +# cannot produce a complete capture, which cost a re-capture to learn. In Download +# Mode vcpkg refuses to execute anything, and a portfile that stops executing +# stops downloading: +# +# CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:23 +# This command cannot be executed in Download Mode. +# Halting portfile execution. +# ... x_vcpkg_get_python_packages(...) angle/portfile.cmake:86 +# +# angle downloads gni-to-cmake.py, sets up a python venv to run it, and THEN +# downloads four more WebKit files (include_CMakeLists.txt, +# WebKitCompilerFlags.cmake, DetectSSE2.cmake, WebKitMacros.cmake) at lines +# 123-153. The venv is line 86, so in Download Mode those four URLs are +# unreachable -- which is exactly the five-row shortfall (72 vs 76) that a +# download-only re-capture produced while reporting success. The committed 76-row +# capture could not have been made this way; it came from a full build, and the +# comment that used to sit here claiming --only-downloads "is enough because the +# asset hook fires during resolution" was invented, not measured. +# +# Note how WIDESPREAD the halting is: in that run 58 of the 77 ports halted, most +# of them at vcpkg_cmake_configure (harmless -- every download precedes the +# configure) but some, like angle, mid-download-sequence. Nothing in the log +# distinguishes the two cases; only the portfile knows whether a download follows +# the step that halted. So "halted" cannot be triaged into safe and unsafe, and a +# capture containing any halt is not emittable. +# +# CAPTURE_ONLY_DOWNLOADS=1 therefore exists only to refresh URLs you already know +# are reachable without running anything (it is minutes instead of ~50). It is +# opt-in, and the halt check below will refuse to bless its output -- as it must, +# since that mode's whole speed advantage IS the skipped execution. +if [ -n "${CAPTURE_ONLY_DOWNLOADS:-}" ]; then + echo "capture: --only-downloads requested: FAST but structurally" >&2 + echo "capture: INCOMPLETE. Ports halt where they would execute, losing every" >&2 + echo "capture: download after that point. Use it to refresh known-reachable" >&2 + echo "capture: URLs only; the result will be reported as not emittable." >&2 + set -- --only-downloads "$@" +fi +# --binarysource=clear is MANDATORY for a capture, and it is the third way this +# script silently lost rows. vcpkg's binary cache is keyed by each port's ABI +# hash; on a hit it unpacks the archive and NEVER RUNS THE PORTFILE, so the port +# asks for none of its downloads and contributes nothing to the capture. Unlike a +# failed fetch or a halt, this leaves no trace at all: no error, no halt, exit 0. +# +# Measured, because I did not believe it was this bad: two runs of a zlib-only +# manifest against a warm ~/.cache/vcpkg/archives, second run with a fresh install +# root. Run 1 built zlib and captured 3 rows. Run 2 printed "Restored 3 +# package(s)", "All requested installations completed successfully in: 1.54 ms", +# exited 0 -- and captured ZERO. The old capture had no --binarysource at all, so +# it inherited whatever cache the machine had; that is enough on its own to +# explain a re-capture that cannot reproduce the committed rows. +# +# It goes BEFORE "$@" so a caller can still override it deliberately (last flag +# wins), and the floor check below is what catches it if they do. +# +# `set -euo pipefail` is on, so the pipeline's status is vcpkg's when vcpkg fails +# and tee's only when tee does -- without pipefail the tee would swallow a vcpkg +# failure outright. "${VCPKG_ROOT:?set VCPKG_ROOT}/vcpkg" install \ - --only-downloads \ --x-asset-sources="clear;x-script,$REC {url} {sha512} {dst}" \ - "$@" + --binarysource=clear \ + "$@" 2>&1 | tee "$VCPKG_LOG" + +# A halted portfile loses its later downloads exactly like a failed fetch does, +# and vcpkg exits 0 for it too ("Downloaded sources for angle", then "All +# requested installations completed successfully"). There is no exit code to read +# and no asset-script callback for a step never reached, so the only witness is +# the log -- hence the tee. +# +# Report the PORT, not the CMake line: the halt message names a helper script +# (vcpkg_execute_required_process.cmake:23) that is the same for every port, and +# the portfile in the call stack is a versioned path under bt/versioning_. The +# port name comes from the "Installing N/M :@" line above the +# halt, which is also the identifier a reader can act on. +# +# Match the halt case-INSENSITIVELY and on the short phrase: vcpkg has (at least) +# two spellings, "Halting portfile execution." for a refused step and "Download +# failed, halting portfile." for a fetch that did not produce the file. The second +# is normally also reported by the recorder's $FAILED, but only for fetches the +# recorder itself ran -- a SHA512 mismatch, say, is vcpkg rejecting a file the +# recorder fetched happily, and then the log is the only witness again. +awk ' + /^Installing [0-9]+\/[0-9]+ / { split($3, f, ":"); port = f[1] } + tolower($0) ~ /halting portfile/ && port \ + { print "port " port " halted before finishing its portfile" } +' "$VCPKG_LOG" | sort -u >> "$FAILED" + +# The two ways a port can skip its portfile ENTIRELY, and so contribute nothing +# while looking fine. Both are silent -- no error, no halt, exit 0 -- so the log +# is again the only witness. +# +# "Restored N package(s) from " -- a binary-cache hit unpacks an archive +# instead of running the portfile. Guarded above with --binarysource=clear, +# but a caller can override it, so verify the OUTCOME rather than trusting +# the flag. +# "The following packages are already installed" -- an install root that +# already has the port. Nothing is rebuilt, nothing is downloaded. This is +# why a capture wants a FRESH --x-install-root: the 71fb301a capture's +# second run had 7 ports already installed and could not have captured any +# of their distfiles. +if grep -q "^Restored [0-9]* package" "$VCPKG_LOG"; then + grep -o "^Restored [0-9]* package(s) from [^ ]*" "$VCPKG_LOG" \ + | sed 's/$/ -- a cache hit never runs the portfile, so those downloads were never requested/' \ + >> "$FAILED" +fi +# And the check that does not depend on my enumerating the loss modes: ask vcpkg +# what it NEEDED and require a row for each. Every download it resolves is +# announced in one of two ways -- +# +# "Trying to download using asset cache script" -- it called the +# recorder, so there must be a row (unless the fetch failed, which $FAILED +# already covers). +# "-- Using cached " -- the file was ALREADY +# in --downloads-root, so vcpkg skipped the asset script entirely and the +# recorder never saw it. FOURTH loss mode, and it contradicts the resume +# advice above: sharing downloads/ across runs is what makes a re-run cheap, +# and it is also what makes a re-run's capture incomplete. Measured on the +# angle re-capture: "-- Using cached gni-to-cmake.py" produced no row. +# +# An ABSOLUTE path in the second form is different and must NOT be required: that +# is vcpkg_from_git pre-placing its own archive (libyuv and skia's two, here), +# which bypasses asset caching entirely and is pinned by vcpkg_git_archives.bzl +# instead. Derived, not listed: relative name = asset download, absolute = git. +# +# Compare on the {dst} basename, undoing the two manglings the recorder is +# deliberately dumb about: the "..part" suffix, and the 8-hex disambiguator +# vcpkg splices in when a wrong-hash file is already present. +{ sed -n 's/^Trying to download \(.*\) using asset cache script$/\1/p' "$VCPKG_LOG" + sed -n 's|^-- Using cached \([^/].*\)$|\1|p' "$VCPKG_LOG" +} | sort -u > "$VCPKG_LOG.need" +{ cut -f3 "$OUT" | sed 's|.*/||; s|\.[0-9]\{1,\}\.part$||' + cut -f3 "$OUT" | sed 's|.*/||; s|\.[0-9]\{1,\}\.part$||' \ + | sed -E 's/-[0-9a-f]{8}(\.[^.]+(\.[^.]+)?)$/\1/' +} | sort -u > "$VCPKG_LOG.have" +comm -23 "$VCPKG_LOG.need" "$VCPKG_LOG.have" \ + | sed 's/$/ -- vcpkg resolved this download but no row was captured (already in downloads\/?)/' \ + >> "$FAILED" +rm -f "$VCPKG_LOG.need" "$VCPKG_LOG.have" + +if grep -q "The following packages are already installed" "$VCPKG_LOG"; then + n=$(sed -n '/The following packages are already installed/,/^The following packages will be/p' \ + "$VCPKG_LOG" | grep -c "^ *[*]* *[a-z0-9]" || true) + echo "$n package(s) were ALREADY INSTALLED -- their portfiles did not run," \ + "so their downloads were never requested; use a fresh --x-install-root" \ + >> "$FAILED" +fi rm -f "$REC" -# Dedupe on (url, sha512) -- NOT the whole line. The third column is the raw -# {dst}, which carries a pid and so differs on every retry; sorting whole lines -# leaves the same distfile in the file several times over. +# Keep $VCPKG_LOG until the capture is blessed: if it is INCOMPLETE the log is the +# evidence for WHY (which port halted, at which portfile line), and deleting it +# would leave a reader with a port name and nothing to read. +trap 'rm -f "$FAILED"' EXIT + +# A capture that lost a download is INCOMPLETE, and vcpkg does not tell you so in +# its exit code: with --only-downloads it printed "All requested installations +# completed successfully in: 49 min" and exited 0 having FAILED to download +# angle's gni-to-cmake.py (a transient TLS error -- this sandbox's clock was +# briefly behind the certificate's validity window, "certificate is not yet +# valid"). That is not a cosmetic loss: the failure HALTED angle's portfile, so +# the FOUR vcpkg_download_distfile calls after it were never made, never +# requested, and so never captured. The result is a pin that is missing five URLs +# and looks complete -- the worst possible shape, since the emitted rules would +# then fetch nothing for angle and the failure would surface much later as a +# build error inside a port. +# +# The recorder logs the tuple BEFORE fetching, so a failed download is still in +# the capture; what is lost is everything the halted portfile would have asked for +# next. The recorder therefore also appends each failed URL to a sentinel file, +# which is the thing checked here -- not vcpkg's exit code, which lies, and not a +# grep of stderr, which depends on how the caller redirected it. +# +# Dedupe FIRST so the row count reported below is the real one. Dedupe on +# (url, sha512) -- NOT the whole line: the third column is the raw {dst}, which +# carries a pid and so differs on every retry, and sorting whole lines leaves the +# same distfile in the file several times over. sort -u -t$'\t' -k1,2 -o "$OUT" "$OUT" +if [ -s "$FAILED" ]; then + echo "capture: INCOMPLETE -- $(wc -l < "$FAILED") loss(es):" >&2 + sed 's/^/capture: /' "$FAILED" >&2 + echo "capture: Every one of these means some vcpkg_download_distfile was never" >&2 + echo "capture: REQUESTED, and only requested downloads can be captured: a failed" >&2 + echo "capture: fetch and a refused step both halt the rest of their portfile, and" >&2 + echo "capture: a cache hit or an already-installed port skips the portfile whole." >&2 + echo "capture: So $OUT is missing rows and looks complete. Do not emit from it." >&2 + echo "capture: Re-run; it appends and shares downloads/, so it resumes." >&2 + echo "capture: A halt on EVERY port means CAPTURE_ONLY_DOWNLOADS was set --" >&2 + echo "capture: that mode cannot produce a complete capture; drop it." >&2 + echo "capture: vcpkg's own output is kept at $VCPKG_LOG" >&2 + exit 1 +fi +rm -f "$VCPKG_LOG" echo "captured $(wc -l < "$OUT") distfiles -> $OUT" >&2 diff --git a/examples/ladybird/workspace/Meta/vcpkg_capture_git_archives.sh b/examples/ladybird/workspace/Meta/vcpkg_capture_git_archives.sh new file mode 100755 index 0000000..dcb9f24 --- /dev/null +++ b/examples/ladybird/workspace/Meta/vcpkg_capture_git_archives.sh @@ -0,0 +1,96 @@ +#!/bin/bash +# Regenerate the vcpkg_from_git pin, without CMake and without a full build. +# +# THE PROBLEM. Four of vcpkg's inputs are not distfiles: skia and angle pull +# sub-dependencies with `vcpkg_from_git`, which bypasses the asset cache +# entirely. `x-asset-sources` never sees them, so Meta/vcpkg_capture_assets.sh +# (which records the other 76 by BEING the asset cache) cannot record them, and +# `x-block-origin` does not govern them either -- they are a plain `git fetch` +# inside a portfile. +# +# WHY NOT JUST READ THE PORTFILES. Because which git externals are used is +# decided by CMake *evaluation*, not by the text. skia declares ten with +# `declare_external_from_git` and then calls +# `get_externals(${required_externals})`, where `required_externals` is built up +# under feature and platform `if()`s -- so a static scan of skia's portfile +# yields 8 where 4 are real. And libyuv's archive comes from the libyuv *port* +# calling vcpkg_from_git directly, which a scan of skia+angle misses entirely. I +# wrote that scanner first; it was wrong in both directions at once. +# +# THE INSTRUMENT. `vcpkg install --only-downloads` runs the portfiles' *fetch* +# phase and stops. That is enough to make vcpkg_from_git produce its tarballs, at +# the refs the real resolution picks, in ~6 minutes with no compilation and no +# CMake configure of Ladybird. Same tactic as the asset capture and as +# scripts/npm_instrument: do not predict what the foreign build system will ask +# for -- run it and record the answer. +# +# THE ONE THAT ESCAPES EVEN THIS. angle's zlib is fetched by `checkout_in_path` +# from angle's *build* phase, not its fetch phase, so --only-downloads does not +# produce it (verified: 3 of the 4 appear). Its (url, ref) is a literal in the +# overlay portfile, so Meta/fetch_vcpkg_git_archives.py resolves and reproduces +# it with git, and the SHA512 below is what proves the two agree. Recording that +# asymmetry here is the point: "--only-downloads gets them all" would be the +# false version of this comment. +# +# Usage: Meta/vcpkg_capture_git_archives.sh [outfile] +# Requires: Build/vcpkg (python3 Meta/ladybird.py vcpkg) and network access. +# Prints namesha512 for every PORT-<40hex>.tar.gz vcpkg produced. +set -uo pipefail + +SRC="${LADYBIRD_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +OUT="${1:-/dev/stdout}" +ROOT="$SRC/Build/vcpkg" + +if [ ! -x "$ROOT/vcpkg" ]; then + echo "capture: no bootstrapped vcpkg at $ROOT" >&2 + echo "capture: get one with 'python3 Meta/ladybird.py vcpkg' (no CMake needed)" >&2 + exit 1 +fi + +SCRATCH=$(mktemp -d "${TMPDIR:-/tmp}/vcpkg-git-capture-XXXXXX") +trap 'rm -rf "$SCRATCH"' EXIT INT TERM + +mkdir -p "$SCRATCH/m/Meta/CMake/vcpkg" +cp "$SRC/vcpkg.json" "$SCRATCH/m/" +cp "$SRC/vcpkg-configuration.json" "$SCRATCH/m/" 2>/dev/null +cp -r "$SRC/Meta/CMake/vcpkg/overlay-ports" "$SCRATCH/m/Meta/CMake/vcpkg/" + +# The manifest is copied VERBATIM, never reconstructed: the baseline plus the 45 +# overrides are what select the port versions, and a reconstruction that drifts +# would silently capture refs for different versions than the build uses. +cd "$SCRATCH/m" || exit 1 +VCPKG_ROOT="$ROOT" "$ROOT/vcpkg" install \ + --only-downloads \ + --x-manifest-root="$SCRATCH/m" \ + --overlay-ports="$SCRATCH/m/Meta/CMake/vcpkg/overlay-ports" \ + --triplet=x64-linux-dynamic \ + --host-triplet=x64-linux \ + --x-install-root="$SCRATCH/out" \ + --x-buildtrees-root="$SCRATCH/bt" \ + --downloads-root="$SCRATCH/dl" >&2 +rc=$? +if [ $rc -ne 0 ]; then + echo "capture: vcpkg install --only-downloads failed (rc=$rc)" >&2 + exit $rc +fi + +# vcpkg names these DOWNLOADS/${PORT}-${sanitized_ref}.tar.gz, which is exactly +# the shape below; every other file in downloads/ is an ordinary distfile already +# covered by Meta/vcpkg_assets.tsv. +found=0 +: > "$OUT" +for f in "$SCRATCH/dl"/*.tar.gz; do + name=$(basename "$f") + [[ "$name" =~ ^[a-z0-9]+-[0-9a-f]{40}\.tar\.gz$ ]] || continue + printf '%s\t%s\n' "$name" "$(sha512sum "$f" | cut -d' ' -f1)" >> "$OUT" + found=$((found + 1)) +done + +if [ "$found" -eq 0 ]; then + echo "capture: no PORT-.tar.gz produced -- vcpkg_from_git may have changed" >&2 + exit 1 +fi +echo "capture: recorded $found git-sourced externals" >&2 +echo "capture: NB angle's zlib is fetched in angle's BUILD phase, so it does not" >&2 +echo "capture: appear here; Meta/fetch_vcpkg_git_archives.py reproduces it from" >&2 +echo "capture: the portfile's literal (url, ref) and verifies it against the pin." >&2 diff --git a/examples/ladybird/workspace/Meta/vcpkg_host_tools.tsv b/examples/ladybird/workspace/Meta/vcpkg_host_tools.tsv new file mode 100644 index 0000000..bcec671 --- /dev/null +++ b/examples/ladybird/workspace/Meta/vcpkg_host_tools.tsv @@ -0,0 +1,26 @@ +# Tools the build needs FROM THE HOST, because vcpkg cannot download +# them on Linux at all. GENERATED by emit_vcpkg_bazel.py --host-tools. +# +# NOT the same class as vcpkg_tool_assets.tsv. Those are tools vcpkg +# fetches for itself, and the fix there was a pin. For these, +# vcpkg_find_acquire_program() has download URLs ONLY inside its +# `if(CMAKE_HOST_WIN32)` branch -- on Linux it probes the host and +# hard-fails with 'Could not find '. There is no URL to pin, so this +# file does not pretend to close the gap: it NAMES the gap, and +# vcpkg_build.sh checks the whole list up front so a machine missing +# three tools is told about three tools in one second, rather than one +# per 20-minute build (nasm surfaced from libvpx at minute ~20). +# +# An empty BINARY means the package ships no executable to probe for +# (autoconf-archive is m4 macros), so the preflight can only name it. +# +# binary-or-alternatives apt-package ports-that-need-it +autoreconf|autoconf autoconf vcpkg-make +- autoconf-archive vcpkg-make +aclocal|automake automake vcpkg-make +- libltdl-dev vcpkg-make +libtoolize|glibtoolize libtool vcpkg-make +nasm nasm dav1d,ffmpeg,libjpeg-turbo,libvpx,openh264 +perl perl libvpx,openssl +pkg-config pkg-config curl,ffmpeg,libavif,libxml2,vcpkg-make,vcpkg-tool-meson +python3 python3 icu,skia,vcpkg-gn,vcpkg-tool-meson diff --git a/examples/ladybird/workspace/Meta/vcpkg_tool_assets.tsv b/examples/ladybird/workspace/Meta/vcpkg_tool_assets.tsv new file mode 100644 index 0000000..8952843 --- /dev/null +++ b/examples/ladybird/workspace/Meta/vcpkg_tool_assets.tsv @@ -0,0 +1,14 @@ +# vcpkg's OWN host tools, pinned from its scripts/vcpkg-tools.json at the +# builtin-baseline. GENERATED by emit_vcpkg_bazel.py --capture-tools. +# +# Separate from vcpkg_assets.tsv because the asset capture CANNOT see +# these: vcpkg_find_acquire_program probes the host first, so a tool the +# capturing machine already has is never downloaded and never captured. +# That is how ninja went unpinned -- the capturing machine had +# /usr/bin/ninja at exactly the required version -- while cmake was pinned +# only because the host's was too old. What a pin contains must not depend +# on what happens to be installed on one machine. +# +# url sha512 filename-vcpkg-looks-for +https://github.com/Kitware/CMake/releases/download/v4.4.0/cmake-4.4.0-linux-x86_64.tar.gz 3df4aaa128a438ed48dcac7065fd355ff538eed8f394491298d0db63a891d671da247c8fa262e4fa6bf99429d630abab317d5a0248168fe203d1ca4978dab4da cmake-4.4.0-linux-x86_64.tar.gz +https://github.com/ninja-build/ninja/releases/download/v1.13.2/ninja-linux.zip 714b900cf10b7ecb1b641c91f4ef696250c64984e5955a8088e4a538d6e8077f43e55f6da47efcedbe316c68d51a9e98feff51734eb0eac1b17aa85af5698753 ninja-linux-1.13.2.zip diff --git a/examples/ladybird/workspace/bazelrc.txt b/examples/ladybird/workspace/bazelrc.txt index b9792ee..5ad71df 100644 --- a/examples/ladybird/workspace/bazelrc.txt +++ b/examples/ladybird/workspace/bazelrc.txt @@ -1,5 +1,12 @@ common --enable_bzlmod +# A declared version that MVS does not actually resolve to is inert: the floor +# is usually bazel_tools' (built into Bazel), so .bazelversion is part of the +# dependency specification. Make the drift fail the build instead of warning -- +# it caught rules_cc 0.2.17 being declared while 0.2.19 resolved on Bazel 9.2.0. +# See docs/BAZEL-RULES.md, "Versions are resolved, not remembered". +common --check_direct_dependencies=error + # C++23 everywhere (Ladybird global). build --cxxopt=-std=c++23 --host_cxxopt=-std=c++23 @@ -70,9 +77,37 @@ build --linkopt=-Wl,--allow-shlib-undefined build --linkopt=-Wl,--allow-multiple-definition # System library search path for find_package deps (Qt6, GLX/OpenGL). build --linkopt=-L/usr/lib/x86_64-linux-gnu -# System includes for find_package deps that are still host-provided (libdrm). -# Qt6 no longer appears here: it comes from rules_qt as a real Bazel dep. -build --action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm +# System includes for find_package/pkg_check_modules deps that are still +# host-provided: libdrm, and (since 71fb301a) glib -- upstream added a +# pkg_check_modules(GIO) for UI/Qt's ExternalURLActivationToken/Handler. +# +# SIX roots, not the three that look obvious, and the list is DERIVED: it is +# every absolute -I/-isystem in the CMake reference that no Bazel dep edge +# carries, and emit_build_bazel.py now WARNS when one is missing here instead of +# dropping it silently. I first transcribed three of them by hand and the build +# failed 3,800 actions later on `gio/gdesktopappinfo.h: No such file or +# directory`. glibconfig.h lives under a LIBDIR rather than an includedir; +# gio-unix-2.0 is a separate root from glib-2.0 (the UNIX-only GIO headers); +# blkid/libmount/sysprof-6 arrive transitively through glib's own pkg-config. +# Qt6 does not appear here: it comes from rules_qt as a real Bazel dep, and the +# vcpkg include dirs ride on //Meta/vcpkg: (finding 33). +build --action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include:/usr/include/gio-unix-2.0:/usr/include/blkid:/usr/include/libmount:/usr/include/sysprof-6 +# ...and the SAME value as a --repo_env, which is not redundant. --action_env +# makes the compiler FIND the header; Bazel then rejects it anyway: +# +# Compiling UI/Qt/ExternalURLHandler.cpp failed: absolute path inclusion(s) +# found in rule '//:ladybird': ... includes the following non-builtin files +# with absolute paths: '/usr/lib/.../glib-2.0/include/glibconfig.h' +# +# -- a correctness check, and a fair one: a header outside the execution root +# that no toolchain declares is an undeclared input, so Bazel cannot know when +# it changed. rules_cc computes the builtin include dirs by running `cc -E -v` +# in a REPOSITORY rule, and CPLUS_INCLUDE_PATH appears in that output -- so +# passing it as --repo_env is what puts these roots in +# cxx_builtin_include_directories, i.e. declares them to the toolchain rather +# than sneaking past the check. Only glibconfig.h needed it: every other root +# here is under /usr/include, which gcc already reports as builtin. +build --repo_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include:/usr/include/gio-unix-2.0:/usr/include/blkid:/usr/include/libmount:/usr/include/sysprof-6 # --- exec (host) configuration --------------------------------------------- # Every flag above is a --cxxopt/--copt/--linkopt, which Bazel applies only to @@ -119,4 +154,4 @@ build --host_copt=-fPIC build --host_linkopt=-Wl,--allow-shlib-undefined build --host_linkopt=-Wl,--allow-multiple-definition build --host_linkopt=-L/usr/lib/x86_64-linux-gnu -build --host_action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm +build --host_action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include:/usr/include/gio-unix-2.0:/usr/include/blkid:/usr/include/libmount:/usr/include/sysprof-6 diff --git a/examples/ladybird/workspace/cargo.bzl b/examples/ladybird/workspace/cargo.bzl index ffa0322..7a85416 100644 --- a/examples/ladybird/workspace/cargo.bzl +++ b/examples/ladybird/workspace/cargo.bzl @@ -184,11 +184,20 @@ rust_sysroot = rule( # one crate's .rs files rebuilds one crate. # --------------------------------------------------------------------------- CargoCrateInfo = provider( - doc = "One built Rust staticlib plus the FFI headers its build script wrote.", + doc = "One built Rust crate plus the FFI headers its build script wrote.", fields = { - "archive": "The lib.a File.", + "archive": "The lib.a File, or None for a --bin crate, which " + + "has no archive to link: what its C++ consumer uses is the " + + "EXECUTABLE, spawned at run time (libwasm_cranelift).", "headers": "The generated FFI header Files.", - "include_dirs": "Dirs to put on the include path for those headers.", + "include_dirs": "Dirs to put on the include path for those headers. " + + "Exported through CcInfo, so TRANSITIVE -- only ever the " + + "ffi/ root, whose spellings are unique per crate.", + "bare_include_dir": "The ffi/ dir, for crates whose header is " + + "#included with no directory. Deliberately NOT in " + + "CcInfo: 8 crates ship a RustFFI.h, so this must stay " + + "LOCAL to the owning library (its own copts), the way " + + "CMake's per-library FFI_OUTPUT_DIR is. Empty otherwise.", }, ) @@ -280,17 +289,64 @@ def _cargo_crate_impl(ctx): }, use_default_shell_env = True, ) - include_dirs = [ - "%s/%s/%s" % (archive.root.path, ctx.label.package, ffi_root), - ] - if ctx.attr.ffi_prefix: - include_dirs.append(include_dirs[0] + "/" + ctx.attr.ffi_prefix) + # The include root is the directory that makes each header resolve at the + # spelling its consumers use, and NOTHING more. + # + # 8 of the 10 crates emit a header literally named RustFFI.h (CMake: + # `FFI_HEADER RustFFI.h`), so a shared, unprefixed root is AMBIGUOUS: with + # both /ffi dirs on one command line, a bare `#include ` + # binds to whichever -isystem came first. That is not hypothetical -- it was + # happening: LibRegex's compile received libunicode_rust/ffi BEFORE + # libregex_rust/ffi, so `` in RustRegex.h resolved to LibUnicode's + # header, and the build only worked because a leftover + # -IBuild/full/Libraries/LibRegex shadowed both. Removing the CMake tree + # exposed it as "'RustRegexFlags' has not been declared". + # + # CMake has no such ambiguity: FFI_OUTPUT_DIR defaults to + # CMAKE_CURRENT_BINARY_DIR, so a library sees ONLY its own crate's dir. The + # faithful translation is therefore the PREFIXED root only (ffi/LibRegex), so + # resolves and a bare does not resolve to + # someone else's header. The 4 TUs that spell it bare get the unprefixed dir + # too -- but only from their OWN crate, via ffi_bare_include below. + # BOTH roots are needed, and they are not interchangeable: + # ffi/ -> makes the PREFIXED spelling work + # (32 TUs; the header sits at ffi/LibUnicode/RustFFI.h) + # ffi// -> makes the BARE spelling work (4 TUs) + # Dropping ffi/ broke LibUnicode ("LibUnicode/RustFFI.h: No such file"), and + # exposing ffi/ for EVERY crate is what made the bare spelling ambiguous in + # the first place (8 crates all ship a RustFFI.h, so first -isystem wins). + # + # The resolution: every crate gets ffi/ (needed for its own prefixed spelling, + # and harmless because that spelling is unique per crate), but only the crates + # whose header is included bare ALSO get ffi//. That keeps exactly one + # unprefixed candidate per compile for a bare include -- the consuming + # library's own -- which is precisely the guarantee CMake gets from + # FFI_OUTPUT_DIR defaulting to the library's own binary dir. + # Only the PREFIXED-capable root goes in the provider, because CcInfo's + # system_includes propagate TRANSITIVELY: every dir here lands on every + # downstream library's command line. That is correct for ffi/ (the spelling + # is unique per crate) and wrong for ffi// (8 + # crates ship a RustFFI.h, so a bare would bind to whichever + # crate's dir sorted first). LibGfx proved it: it inherited LibRegex's and + # LibTextCodec's bare dirs through the dep graph and compiled against + # LibRegex's header ("'FFI' does not name a type"). + # + # So the bare-include dir is NOT exported here. It is published separately as + # `bare_include_dir` for the OWNING library to put on its own copts, where it + # is local and cannot leak downstream -- which is exactly the scope CMake's + # FFI_OUTPUT_DIR (the library's own binary dir) has. + ffi_base = "%s/%s/%s" % (archive.root.path, ctx.label.package, ffi_root) + include_dirs = [ffi_base] + bare_dir = "" + if ctx.attr.ffi_prefix and ctx.attr.ffi_bare_include: + bare_dir = ffi_base + "/" + ctx.attr.ffi_prefix return [ DefaultInfo(files = depset([archive] + headers)), CargoCrateInfo( archive = archive, headers = headers, include_dirs = include_dirs, + bare_include_dir = bare_dir, ), ] @@ -303,6 +359,13 @@ cargo_crate = rule( undeclared output is deleted by Bazel, which is how a header CMake never declared (HTMLTokenizerRustFFI.h) turned up.""", attrs = { + "ffi_bare_include": attr.bool( + default = False, + doc = "Deprecated-by-construction escape hatch: also expose this " + + "crate's unprefixed ffi dir, for the TUs that spell the " + + "header with no directory. Per-crate so it can " + + "never make two crates' headers collide.", + ), "crate": attr.string(mandatory = True, doc = "The cargo package name."), "manifest": attr.string( mandatory = True, @@ -363,19 +426,38 @@ cargo_crate = rule( ) def _cargo_binary_impl(ctx): - """A cargo `--bin` crate, e.g. flapc. + """A cargo `--bin` crate: flapc, and cranelift-compiler. Same action, different cargo subcommand, and the output is executable so a genrule can name it in `tools`. flapc's own 3-package workspace gets the identical treatment: its lock file has exactly one registry crate (smallvec, pinned `=1.15.1`, the same version and checksum as the big workspace's), so it needs no separate machinery at all. + + A binary crate can ALSO emit an FFI header, and the second one does: + libwasm_cranelift's build.rs runs cbindgen exactly as the staticlib crates' + do, and LibWasm's CraneliftBridge.cpp includes the result. So `ffi_headers` + is the same declared-output contract cargo_crate has, for the same reason (an + undeclared output is deleted by Bazel) -- what differs is only that there is + no archive: what C++ consumes from this crate is the *executable*, spawned at + run time via WASM_CRANELIFT_COMPILER_PATH. """ out = ctx.actions.declare_file(ctx.label.name) + ffi_root = "%s.ffi" % ctx.label.name + headers = [ + ctx.actions.declare_file("%s/%s/%s" % (ffi_root, ctx.attr.ffi_prefix, h)) + for h in ctx.attr.ffi_headers + ] + ffi_out = "" + if headers: + ffi_out = "%s/%s/%s" % (out.root.path, ctx.label.package, ffi_root) + if ctx.attr.ffi_prefix: + ffi_out += "/" + ctx.attr.ffi_prefix + index, crate_inputs = _crate_index(ctx) sysroot = ctx.file.sysroot ctx.actions.run( - outputs = [out], + outputs = [out] + headers, inputs = depset([index, sysroot, ctx.file._vendor] + ctx.files.srcs + crate_inputs), executable = ctx.executable._build, @@ -387,7 +469,8 @@ def _cargo_binary_impl(ctx): index.path, out.path, ctx.attr.bin, - ], + ffi_out, + ] + ctx.attr.ffi_headers, mnemonic = "CargoBinary", progress_message = "Building Rust binary %s (offline)" % ctx.attr.bin, execution_requirements = {"block-network": "1"}, @@ -399,21 +482,52 @@ def _cargo_binary_impl(ctx): }, use_default_shell_env = True, ) - return [DefaultInfo( - files = depset([out]), - executable = out, - runfiles = ctx.runfiles(files = [out]), - )] + ffi_base = "%s/%s/%s" % (out.root.path, ctx.label.package, ffi_root) + bare_dir = "" + if ctx.attr.ffi_prefix and ctx.attr.ffi_bare_include: + bare_dir = ffi_base + "/" + ctx.attr.ffi_prefix + return [ + DefaultInfo( + files = depset([out]), + executable = out, + runfiles = ctx.runfiles(files = [out]), + ), + # Carried even with no headers (flapc), so a consumer asking for this + # crate's headers gets an empty set rather than a missing provider. + CargoCrateInfo( + archive = None, + headers = headers, + include_dirs = [ffi_base] if headers else [], + bare_include_dir = bare_dir, + ), + ] cargo_binary = rule( implementation = _cargo_binary_impl, executable = True, - doc = "Build a cargo binary crate offline (flapc), runnable as a genrule tool.", + doc = """Build a cargo binary crate offline, runnable as a genrule tool. + + Two of them: flapc (a pure tool) and cranelift-compiler (a tool Ladybird + SPAWNS at run time, which additionally emits the FFI header LibWasm + includes).""", attrs = { "crate": attr.string(mandatory = True), "bin": attr.string(mandatory = True, doc = "The --bin name."), "manifest": attr.string(mandatory = True), "crate_features": attr.string_list(), + "ffi_headers": attr.string_list( + doc = "Headers this crate's build script writes, relative to " + + "ffi_prefix. Declared so Bazel keeps them.", + ), + "ffi_prefix": attr.string( + doc = "Directory the headers are staged under, so the include " + + "spelling resolves.", + ), + "ffi_bare_include": attr.bool( + doc = "Whether the owning library includes the header with no " + + "directory. Derived by scanning the source, see " + + "emit_cargo_bazel.crates_included_bare().", + ), "srcs": attr.label_list(allow_files = True), "crates": attr.string_keyed_label_dict(allow_files = True), "sysroot": attr.label(allow_single_file = True, mandatory = True), @@ -478,16 +592,29 @@ cargo_binary = rule( def _cargo_lib_impl(ctx): info = ctx.attr.crate[CargoCrateInfo] archive = info.archive + compilation = cc_common.create_compilation_context( + headers = depset(info.headers), + # -isystem, not -I: these are cbindgen output, and Ladybird's + # -Werror should not fire inside generated code (the same reason + # the vcpkg headers are -isystem). + system_includes = depset(info.include_dirs), + ) + if archive == None: + # A --bin crate (libwasm_cranelift): headers to include, NOTHING to link. + # The binary itself is not a link input at all -- LibWasm SPAWNS it, so it + # belongs in the consumer's data/runfiles, not in its linking context. + # Handled here rather than with a second rule so a binary crate that emits + # a header is consumed with exactly the same two labels a staticlib crate + # is (//:_lib and //:_bare_include), and the ring emitter + # needs no special case. + return [ + DefaultInfo(files = depset(info.headers)), + CcInfo(compilation_context = compilation), + ] return [ DefaultInfo(files = depset([archive] + info.headers)), CcInfo( - compilation_context = cc_common.create_compilation_context( - headers = depset(info.headers), - # -isystem, not -I: these are cbindgen output, and Ladybird's - # -Werror should not fire inside generated code (the same reason - # the vcpkg headers are -isystem). - system_includes = depset(info.include_dirs), - ), + compilation_context = compilation, linking_context = cc_common.create_linking_context( linker_inputs = depset([cc_common.create_linker_input( owner = ctx.label, @@ -506,6 +633,44 @@ def _cargo_lib_impl(ctx): ), ] +def _cargo_bare_include_impl(ctx): + """A CcInfo carrying ONLY the crate's ffi/ dir. + + Depended on by the ONE library that owns the crate, so its TUs can spell the + header `` with no directory -- the spelling CMake allows because + FFI_OUTPUT_DIR is the library's own binary dir. + + This is a separate target rather than part of cargo_lib because CcInfo's + system_includes propagate transitively: folded into cargo_lib, the dir would + reach every downstream library and a bare would bind to whichever + of the 8 crates' dirs came first on the command line. As its own target it is + added by exactly one library and travels no further, because nothing depends + on that library's copts. Verified by removal: with this folded into cargo_lib, + LibGfx compiled against LibRegex's header ('FFI' does not name a type). + """ + info = ctx.attr.crate[CargoCrateInfo] + if not info.bare_include_dir: + fail("cargo_bare_include on a crate with no bare_include_dir: %s" % + ctx.attr.crate.label) + return [ + DefaultInfo(files = depset(info.headers)), + CcInfo( + compilation_context = cc_common.create_compilation_context( + headers = depset(info.headers), + system_includes = depset([info.bare_include_dir]), + ), + ), + ] + +cargo_bare_include = rule( + implementation = _cargo_bare_include_impl, + doc = "The unprefixed FFI include dir for the crate's OWNING library only.", + attrs = { + "crate": attr.label(providers = [CargoCrateInfo], mandatory = True, + cfg = "exec"), + }, +) + cargo_lib = rule( implementation = _cargo_lib_impl, doc = """One crate: its archive to link, its generated FFI headers to include. diff --git a/examples/ladybird/workspace/cargo_crates.bzl b/examples/ladybird/workspace/cargo_crates.bzl index ce9e11f..9a3e7b4 100644 --- a/examples/ladybird/workspace/cargo_crates.bzl +++ b/examples/ladybird/workspace/cargo_crates.bzl @@ -1,7 +1,7 @@ # AUTO-GENERATED by Meta/emit_cargo_bazel.py — do not edit. load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") -# 154 crates.io crates, every sha256 lifted verbatim out of the two +# 155 crates.io crates, every sha256 lifted verbatim out of the two # Cargo.lock files. Nothing is re-hashed here and nothing was captured: # the lock file is the pin and the URL is a function of (name, version), # so this regenerates with no cargo, no network and no CMake. @@ -382,6 +382,14 @@ def cargo_crates(): type = "tgz", build_file_content = _CRATE_BUILD, ) + http_archive( + name = 'crate_foldhash_0_2_0', + urls = ['https://static.crates.io/crates/foldhash/foldhash-0.2.0.crate'], + sha256 = '77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb', + strip_prefix = 'foldhash-0.2.0', + type = "tgz", + build_file_content = _CRATE_BUILD, + ) http_archive( name = 'crate_form_urlencoded_1_2_2', urls = ['https://static.crates.io/crates/form_urlencoded/form_urlencoded-1.2.2.crate'], diff --git a/examples/ladybird/workspace/cargo_extension.bzl b/examples/ladybird/workspace/cargo_extension.bzl index 671c988..bc432b3 100644 --- a/examples/ladybird/workspace/cargo_extension.bzl +++ b/examples/ladybird/workspace/cargo_extension.bzl @@ -1,7 +1,7 @@ # AUTO-GENERATED by Meta/emit_cargo_bazel.py — do not edit. # http_archive is a REPOSITORY rule, so under bzlmod it cannot be # called from MODULE.bazel -- it has to come from a module extension. -# That indirection is also what lets one use_repo() name 157 repos +# That indirection is also what lets one use_repo() name 158 repos # without MODULE.bazel enumerating a single URL or hash. load(":cargo_crates.bzl", "cargo_crates") load(":cargo.bzl", "rust_toolchain_archives") diff --git a/examples/ladybird/workspace/cargo_index.bzl b/examples/ladybird/workspace/cargo_index.bzl index c6e7e8d..6d7c36a 100644 --- a/examples/ladybird/workspace/cargo_index.bzl +++ b/examples/ladybird/workspace/cargo_index.bzl @@ -54,6 +54,7 @@ CARGO_CRATE_FILES = { 'fastrand 2.4.0 a043dc74da1e37d6afe657061213aa6f425f855399a11d3463c6ecccc4dfda1f': '@crate_fastrand_2_4_0//:srcs', 'flatbuffers 25.12.19 35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3': '@crate_flatbuffers_25_12_19//:srcs', 'foldhash 0.1.5 d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2': '@crate_foldhash_0_1_5//:srcs', + 'foldhash 0.2.0 77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb': '@crate_foldhash_0_2_0//:srcs', 'form_urlencoded 1.2.2 cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf': '@crate_form_urlencoded_1_2_2//:srcs', 'getrandom 0.4.2 0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555': '@crate_getrandom_0_4_2//:srcs', 'gimli 0.31.1 07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f': '@crate_gimli_0_31_1//:srcs', @@ -174,63 +175,59 @@ CARGO_CRATE_FILES = { CARGO_CRATE_SPECS = { 'libgfx_rust': { "manifest": 'Libraries/LibGfx/Rust/Cargo.toml', - "features": [], + "features": ['allocator'], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibGfx', + "ffi_bare_include": True, }, 'libjs_rust': { "manifest": 'Libraries/LibJS/Rust/Cargo.toml', "features": [], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibJS', + "ffi_bare_include": False, }, 'libregex_rust': { "manifest": 'Libraries/LibRegex/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibRegex', + "ffi_bare_include": True, }, 'libtextcodec_rust': { "manifest": 'Libraries/LibTextCodec/Rust/Cargo.toml', "features": [], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibTextCodec', + "ffi_bare_include": True, }, 'libunicode_rust': { "manifest": 'Libraries/LibUnicode/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibUnicode', + "ffi_bare_include": False, }, 'liburl_rust': { "manifest": 'Libraries/LibURL/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibURL', + "ffi_bare_include": False, }, 'libweb_content_blocker_rust': { "manifest": 'Libraries/LibWeb/ContentBlocker/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['ContentBlockerRustFFI.h'], "ffi_prefix": 'LibWeb', - }, - 'libweb_css_rust': { - "manifest": 'Libraries/LibWeb/CSS/Rust/Cargo.toml', - "features": ['allocator'], - "ffi_headers": ['ComputedValuesRustFFI.h', 'RustFFI.h', 'SelectorRustFFI.h', 'StyleValueRustFFI.h'], - "ffi_prefix": 'LibWeb', - }, - 'libweb_layout_rust': { - "manifest": 'Libraries/LibWeb/Layout/Rust/Cargo.toml', - "features": ['allocator'], - "ffi_headers": ['Layout/TreeBuilderRustFFI.h'], - "ffi_prefix": 'LibWeb', + "ffi_bare_include": False, }, 'libweb_rust': { "manifest": 'Libraries/LibWeb/Rust/Cargo.toml', - "features": [], - "ffi_headers": ['HTML/Parser/RustFFI.h', 'HTMLTokenizerRustFFI.h'], + "features": ['style-recording'], + "ffi_headers": ['ComputedValuesRustFFI.h', 'HTML/Parser/RustFFI.h', 'HTMLTokenizerRustFFI.h', 'Layout/LayoutRustFFI.h', 'Layout/TreeBuilderRustFFI.h', 'RustFFI.h', 'SelectorRustFFI.h', 'StyleEngineBridgeGenerated.h', 'StyleEngineBridgeGenerated.inc', 'StyleEngineRustFFI.h', 'StyleEngineStateFactsGenerated.inc', 'StyleValueRustFFI.h'], "ffi_prefix": 'LibWeb', + "ffi_bare_include": False, }, } diff --git a/examples/ladybird/workspace/cargo_ring.bzl b/examples/ladybird/workspace/cargo_ring.bzl index 2ae7229..fadf5fe 100644 --- a/examples/ladybird/workspace/cargo_ring.bzl +++ b/examples/ladybird/workspace/cargo_ring.bzl @@ -1,17 +1,17 @@ # AUTO-GENERATED by Meta/emit_cargo_bazel.py — do not edit. -load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "rust_sysroot") +load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "cargo_bare_include", "rust_sysroot") load(":cargo_index.bzl", "CARGO_CRATE_FILES", "CARGO_CRATE_SPECS") -# Ladybird's 10 production Rust crates and flapc, BUILT BY BAZEL from the -# 154 crates.io crates Bazel fetched -- replacing the prebuilt 260 MB -# librust_combined.a that was copied out of Build/full/cargo/, the `ar -M` -# merge that produced it (README step 1b), and the reference build's flapc -# binary. Nothing here names Build/full. +# Ladybird's 8 production Rust crates and 4 binary crates, BUILT BY BAZEL +# from the 155 crates.io crates Bazel fetched -- replacing the prebuilt +# 260 MB librust_combined.a that was copied out of Build/full/cargo/, the +# `ar -M` merge that produced it (README step 1b), and the reference build's +# flapc and cranelift-compiler binaries. Nothing here names Build/full. # # Every attribute comes from CARGO_CRATE_SPECS, which -# Meta/emit_cargo_bazel.py parses out of import_rust_crate() in CMake, so a -# feature or header list cannot drift from the reference build without the -# emitter's --report saying so. +# Meta/emit_cargo_bazel.py parses out of import_rust_crate() and +# build_rust_binary() in CMake, so a feature or header list cannot drift from +# the reference build without the emitter's --report saying so. def cargo_ring(): # The pinned toolchain (rust-toolchain.toml says 1.96.1), assembled from @@ -29,11 +29,16 @@ def cargo_ring(): # cargo WORKSPACE. Each of them path-depends on # Libraries/RustAllocator.rs and several are path-dependencies of each # other, so cargo resolves the whole workspace whichever crate you ask - # for. Over-declaring costs a rebuild of 11 crates when any .rs changes; - # under-declaring silently reuses a stale archive. The trade is the right - # way round, and it is real debt: per-crate source sets need the - # path-dependency graph read out of the manifests. - crate_srcs = native.glob(['Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', 'Libraries/RustAllocator.rs', 'Libraries/*/Rust/**', 'Libraries/LibJS/BytecodeDef/**'], allow_empty = False) + ['//Libraries/LibWeb:rust_crate_srcs'] + # for. Over-declaring costs a rebuild of every crate when any .rs + # changes; under-declaring silently reuses a stale archive. The trade is + # the right way round, and it is real debt: per-crate source sets need + # the path-dependency graph read out of the manifests. + # + # The patterns are DERIVED from Cargo.toml's member list closed over the + # manifests' `path =` deps, not written down: a directory written down + # here is an allow_empty=False glob that outlives the directory, and + # loading then fails before any target can say why. + crate_srcs = native.glob(['Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', 'Libraries/RustAllocator.rs', 'Libraries/LibGfx/Rust/**', 'Libraries/LibJS/Flap/**', 'Libraries/LibJS/Rust/**', 'Libraries/LibRegex/Rust/**', 'Libraries/LibTextCodec/Rust/**', 'Libraries/LibURL/Rust/**', 'Libraries/LibUnicode/Rust/**', 'Libraries/LibWasm/Rust/**'], exclude = ['Libraries/LibGfx/Rust/target/**', 'Libraries/LibJS/Flap/target/**', 'Libraries/LibJS/Rust/target/**', 'Libraries/LibRegex/Rust/target/**', 'Libraries/LibTextCodec/Rust/target/**', 'Libraries/LibURL/Rust/target/**', 'Libraries/LibUnicode/Rust/target/**', 'Libraries/LibWasm/Rust/target/**'], allow_empty = False) + ['//Libraries/LibWeb:rust_crate_srcs'] cargo_crate( name = 'libgfx_rust', @@ -42,6 +47,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libgfx_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libgfx_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libgfx_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libgfx_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libgfx_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -60,6 +66,10 @@ def cargo_ring(): name = 'libgfx_rust_lib', crate = ':libgfx_rust', ) + cargo_bare_include( + name = 'libgfx_rust_bare_include', + crate = ':libgfx_rust', + ) cargo_crate( name = 'libjs_rust', @@ -68,11 +78,12 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libjs_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libjs_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libjs_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libjs_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libjs_rust']["manifest"], # Build-script inputs, taken from the reference build's DEPFILES # rather than from reading build.rs: this crate's build script # generates Rust from these, so they are compile inputs. - srcs = crate_srcs + ['Libraries/LibJS/Bytecode/Bytecode.def'], + srcs = crate_srcs + ['Libraries/LibJS/Interpreter/interpreter.flap'], sysroot = ":rust_sysroot", ) @@ -97,6 +108,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libregex_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libregex_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libregex_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libregex_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libregex_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -115,6 +127,10 @@ def cargo_ring(): name = 'libregex_rust_lib', crate = ':libregex_rust', ) + cargo_bare_include( + name = 'libregex_rust_bare_include', + crate = ':libregex_rust', + ) cargo_crate( name = 'libtextcodec_rust', @@ -123,6 +139,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libtextcodec_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libtextcodec_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libtextcodec_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libtextcodec_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libtextcodec_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -141,6 +158,10 @@ def cargo_ring(): name = 'libtextcodec_rust_lib', crate = ':libtextcodec_rust', ) + cargo_bare_include( + name = 'libtextcodec_rust_bare_include', + crate = ':libtextcodec_rust', + ) cargo_crate( name = 'libunicode_rust', @@ -149,6 +170,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libunicode_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libunicode_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libunicode_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libunicode_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libunicode_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -175,6 +197,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['liburl_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['liburl_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['liburl_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['liburl_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['liburl_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -201,6 +224,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -221,13 +245,14 @@ def cargo_ring(): ) cargo_crate( - name = 'libweb_css_rust', - crate = 'libweb_css_rust', + name = 'libweb_rust', + crate = 'libweb_rust', crates = CARGO_CRATE_FILES, - crate_features = CARGO_CRATE_SPECS['libweb_css_rust']["features"], - ffi_headers = CARGO_CRATE_SPECS['libweb_css_rust']["ffi_headers"], - ffi_prefix = CARGO_CRATE_SPECS['libweb_css_rust']["ffi_prefix"], - manifest = CARGO_CRATE_SPECS['libweb_css_rust']["manifest"], + crate_features = CARGO_CRATE_SPECS['libweb_rust']["features"], + ffi_headers = CARGO_CRATE_SPECS['libweb_rust']["ffi_headers"], + ffi_prefix = CARGO_CRATE_SPECS['libweb_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libweb_rust']["ffi_bare_include"], + manifest = CARGO_CRATE_SPECS['libweb_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", ) @@ -242,73 +267,98 @@ def cargo_ring(): # crate's C++ FFI into a target that never linked it. See the # block comment in cargo.bzl. cargo_lib( - name = 'libweb_css_rust_lib', - crate = ':libweb_css_rust', + name = 'libweb_rust_lib', + crate = ':libweb_rust', ) - cargo_crate( - name = 'libweb_layout_rust', - crate = 'libweb_layout_rust', + # The BINARY crates: cargo `--bin` targets, declared in CMake with + # build_rust_binary() rather than import_rust_crate(). Two of them, and + # they are two different shapes: + # + # flapc a pure build TOOL -- Bazel runs it in a genrule to + # produce the interpreter assembly. Its workspace is + # `exclude`d from the root one and has its own lock + # with 2 packages: flapc 0.1.0 (in-tree), smallvec 1.15.1 + # smallvec is also pinned at the same version AND + # checksum as the big workspace, so it is the same + # fetch rule, not a second one. + # cranelift-compiler a RUNTIME tool -- LibWasm spawns it to AOT-compile + # WebAssembly, and its build script ALSO emits the + # CraneliftFFI.h that LibWasm's CraneliftBridge.cpp + # includes. It is a root-workspace member, so its + # source set is the shared crate_srcs. + cargo_binary( + name = 'generate-libjs-bytecode', + bin = 'generate-libjs-bytecode', + crate = 'flapc', crates = CARGO_CRATE_FILES, - crate_features = CARGO_CRATE_SPECS['libweb_layout_rust']["features"], - ffi_headers = CARGO_CRATE_SPECS['libweb_layout_rust']["ffi_headers"], - ffi_prefix = CARGO_CRATE_SPECS['libweb_layout_rust']["ffi_prefix"], - manifest = CARGO_CRATE_SPECS['libweb_layout_rust']["manifest"], - srcs = crate_srcs, + manifest = 'Libraries/LibJS/Flap/Cargo.toml', + # Its OWN workspace (`exclude`d from the root one), so its source + # set is its own subtree plus what it include_str!s from above + # it -- both derived, the subtree from Cargo.toml's `exclude` + # and the rest by scanning for include_str!. + srcs = native.glob(['Libraries/LibJS/Flap/**', 'Libraries/LibJS/Interpreter/interpreter.flap', 'rust-toolchain.toml'], exclude = ['Libraries/LibJS/Flap/target/**'], allow_empty = False), sysroot = ":rust_sysroot", ) - - # The consumable target: this crate's archive to link and this - # crate's generated FFI headers to include, one for one with - # CMake's per-library edge. NOT a shared --start-group over all - # ten archives: the crates have no true cross-crate symbol - # references (measured: 0), the shared symbols are each crate's - # own copy of rust-std, and a group makes ld satisfy one crate's - # std symbol from another crate's object -- dragging that - # crate's C++ FFI into a target that never linked it. See the - # block comment in cargo.bzl. - cargo_lib( - name = 'libweb_layout_rust_lib', - crate = ':libweb_layout_rust', + cargo_binary( + name = 'flapc', + bin = 'flapc', + crate = 'flapc', + crates = CARGO_CRATE_FILES, + manifest = 'Libraries/LibJS/Flap/Cargo.toml', + # Its OWN workspace (`exclude`d from the root one), so its source + # set is its own subtree plus what it include_str!s from above + # it -- both derived, the subtree from Cargo.toml's `exclude` + # and the rest by scanning for include_str!. + srcs = native.glob(['Libraries/LibJS/Flap/**', 'Libraries/LibJS/Interpreter/interpreter.flap', 'rust-toolchain.toml'], exclude = ['Libraries/LibJS/Flap/target/**'], allow_empty = False), + sysroot = ":rust_sysroot", ) - - cargo_crate( - name = 'libweb_rust', - crate = 'libweb_rust', + cargo_binary( + name = 'cranelift-compiler', + bin = 'cranelift-compiler', + crate = 'libwasm_cranelift', crates = CARGO_CRATE_FILES, - crate_features = CARGO_CRATE_SPECS['libweb_rust']["features"], - ffi_headers = CARGO_CRATE_SPECS['libweb_rust']["ffi_headers"], - ffi_prefix = CARGO_CRATE_SPECS['libweb_rust']["ffi_prefix"], - manifest = CARGO_CRATE_SPECS['libweb_rust']["manifest"], - srcs = crate_srcs, + manifest = 'Libraries/LibWasm/Rust/Cargo.toml', + # This binary crate's build script runs cbindgen too, so the + # header is a DECLARED output -- Bazel deletes what nothing + # declares, and LibWasm includes this one bare. + ffi_headers = ['CraneliftFFI.h'], + ffi_prefix = 'LibWasm', + ffi_bare_include = True, + # A member of the ROOT cargo workspace (Cargo.toml lists it), so + # cargo resolves the whole workspace whichever crate you ask for + # and the source set is the shared one -- same over-declaration, + # same reason, as the staticlib crates. + srcs = crate_srcs + ['Libraries/LibWasm/Opcode.h'], sysroot = ":rust_sysroot", ) - # The consumable target: this crate's archive to link and this - # crate's generated FFI headers to include, one for one with - # CMake's per-library edge. NOT a shared --start-group over all - # ten archives: the crates have no true cross-crate symbol - # references (measured: 0), the shared symbols are each crate's - # own copy of rust-std, and a group makes ld satisfy one crate's - # std symbol from another crate's object -- dragging that - # crate's C++ FFI into a target that never linked it. See the - # block comment in cargo.bzl. + # The consumable target, IDENTICAL in shape to a staticlib crate's: + # this crate's generated header to include. There is no archive to + # link -- what C++ consumes from a binary crate is the EXECUTABLE, + # spawned at run time -- so cargo_lib yields a headers-only CcInfo. cargo_lib( - name = 'libweb_rust_lib', - crate = ':libweb_rust', + name = 'libwasm_cranelift_lib', + crate = ':cranelift-compiler', + ) + cargo_bare_include( + name = 'libwasm_cranelift_bare_include', + crate = ':cranelift-compiler', ) - - # flapc, the Flap-DSL -> interpreter-assembly compiler. Its workspace is - # `exclude`d from the root one and has its own lock with exactly 3 - # packages (flapc, in-tree bytecode_def, and smallvec from crates.io - # pinned =1.15.1 -- the same version AND checksum the big workspace pins, - # so it is the same fetch rule, not a second one). cargo_binary( - name = "flapc", - bin = "flapc", - crate = "flapc", + name = 'style-replay', + bin = 'style-replay', + crate = 'libweb_rust', crates = CARGO_CRATE_FILES, - manifest = "Libraries/LibJS/Flap/Cargo.toml", - srcs = native.glob(['Libraries/LibJS/Flap/Cargo.toml', 'Libraries/LibJS/Flap/Cargo.lock', 'Libraries/LibJS/Flap/src/**', 'Libraries/LibJS/Flap/benches/**', 'Libraries/LibJS/Flap/tests/**', 'Libraries/LibJS/BytecodeDef/**', 'Libraries/LibJS/Bytecode/Bytecode.def', 'Libraries/LibJS/Interpreter/interpreter.flap', 'rust-toolchain.toml'], allow_empty = False), + manifest = 'Libraries/LibWeb/Rust/Cargo.toml', + # build_rust_binary() takes FEATURES too, and this one does: + # the same crate builds a staticlib and this binary, and only + # the features distinguish what each gets. + crate_features = ['style-recording'], + # A member of the ROOT cargo workspace (Cargo.toml lists it), so + # cargo resolves the whole workspace whichever crate you ask for + # and the source set is the shared one -- same over-declaration, + # same reason, as the staticlib crates. + srcs = crate_srcs, sysroot = ":rust_sysroot", ) diff --git a/examples/ladybird/workspace/codegen_root.bzl b/examples/ladybird/workspace/codegen_root.bzl index c0fa128..003ca55 100644 --- a/examples/ladybird/workspace/codegen_root.bzl +++ b/examples/ladybird/workspace/codegen_root.bzl @@ -1,21 +1,15 @@ # AUTO-GENERATED by Meta/emit_root_codegen_bazel.py — do not edit. load("@rules_cc//cc:defs.bzl", "cc_library") -# 20 Python-generator genrules for the root package, -# 2 glslang shader headers, 2 self-built-tool genrules +# 21 Python-generator genrules for the root package, +# 2 glslang shader headers, 3 self-built-tool genrules # (byte-parity: Meta/bazel_parity_harness.py). def root_codegen(): native.genrule( name = 'gen_HSTSPreloadData', - srcs = ['Build/caches/HSTSPreload/transport_security_state_static.json'] + ["//Meta:generators"], + srcs = ['@hsts_preload_json//file'] + ["//Meta:generators"], outs = ['Libraries/LibHTTP/HSTSPreloadData.h', 'Libraries/LibHTTP/HSTSPreloadData.cpp'], - cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_hsts_preload_data.py -h $(location Libraries/LibHTTP/HSTSPreloadData.h) -c $(location Libraries/LibHTTP/HSTSPreloadData.cpp) -p $(location Build/caches/HSTSPreload/transport_security_state_static.json)", - ) - native.genrule( - name = 'gen_Op', - srcs = ['Libraries/LibJS/Bytecode/Bytecode.def'] + ["//Meta:generators"], - outs = ['Libraries/LibJS/Bytecode/Op.h', 'Libraries/LibJS/Bytecode/Op.cpp', 'Libraries/LibJS/Bytecode/OpCodes.h'], - cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_libjs_bytecode_def_derived.py -h $(location Libraries/LibJS/Bytecode/Op.h) -c $(location Libraries/LibJS/Bytecode/Op.cpp) -i $(location Libraries/LibJS/Bytecode/Bytecode.def) -x $(location Libraries/LibJS/Bytecode/OpCodes.h)", + cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_hsts_preload_data.py -h $(location Libraries/LibHTTP/HSTSPreloadData.h) -c $(location Libraries/LibHTTP/HSTSPreloadData.cpp) -p $(location @hsts_preload_json//file)", ) native.genrule( name = 'gen_RequestClientEndpoint', @@ -47,6 +41,18 @@ def root_codegen(): outs = ['Services/ImageDecoder/ImageDecoderServerEndpoint.h'], cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_ipc_definitions.py --input $(location Services/ImageDecoder/ImageDecoderServer.ipc) --output $(location Services/ImageDecoder/ImageDecoderServerEndpoint.h)", ) + native.genrule( + name = 'gen_VideoPresentationClientEndpoint', + srcs = ['Libraries/LibMedia/VideoPresentation/VideoPresentationClient.ipc'] + ["//Meta:generators"], + outs = ['Libraries/LibMedia/VideoPresentation/VideoPresentationClientEndpoint.h'], + cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_ipc_definitions.py --input $(location Libraries/LibMedia/VideoPresentation/VideoPresentationClient.ipc) --output $(location Libraries/LibMedia/VideoPresentation/VideoPresentationClientEndpoint.h)", + ) + native.genrule( + name = 'gen_VideoPresentationServerEndpoint', + srcs = ['Libraries/LibMedia/VideoPresentation/VideoPresentationServer.ipc'] + ["//Meta:generators"], + outs = ['Libraries/LibMedia/VideoPresentation/VideoPresentationServerEndpoint.h'], + cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_ipc_definitions.py --input $(location Libraries/LibMedia/VideoPresentation/VideoPresentationServer.ipc) --output $(location Libraries/LibMedia/VideoPresentation/VideoPresentationServerEndpoint.h)", + ) native.genrule( name = 'gen_UIProcessServerEndpoint', srcs = ['Libraries/LibWebView/UIProcessServer.ipc'] + ["//Meta:generators"], @@ -125,6 +131,20 @@ def root_codegen(): outs = ['Services/Compositor/WebGLCommandReplayer.h', 'Services/Compositor/WebGLCommandReplayer.cpp'], cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_compositor_webgl_replayer.py -h $(location Services/Compositor/WebGLCommandReplayer.h) -c $(location Services/Compositor/WebGLCommandReplayer.cpp) -j $(location //Libraries/LibWeb:WebGL/GLFunctions.json)", ) + # The LibJS bytecode headers, generated by a tool THIS BUILD MAKES + # (//:generate-libjs-bytecode, the flapc crate's second bin -- + # cargo_ring.bzl). Upstream a32d9c9f made interpreter.flap the sole + # authority for the instruction set and deleted both Bytecode.def + # and the Python generator that read it, which is why the overlay + # pinned at f9e34731 failed on a newer checkout: its glob for + # Libraries/LibJS/BytecodeDef/** matched nothing. + native.genrule( + name = 'gen_Op', + srcs = ['Libraries/LibJS/Interpreter/interpreter.flap'], + outs = ['Libraries/LibJS/Bytecode/Op.h', 'Libraries/LibJS/Bytecode/OpCodes.h'], + tools = ["//:generate-libjs-bytecode"], + cmd = "$(location //:generate-libjs-bytecode) --input $(location Libraries/LibJS/Interpreter/interpreter.flap) --op-header $(location Libraries/LibJS/Bytecode/Op.h) --opcodes-header $(location Libraries/LibJS/Bytecode/OpCodes.h)", + ) # Two chained self-built tools. generate_interpreter_layout is # built by Bazel (//:generate_interpreter_layout) and emits the # struct offsets flapc needs; flapc then compiles the Flap DSL to @@ -142,14 +162,14 @@ def root_codegen(): ) native.genrule( name = 'gen_interpreter_asm', - srcs = ['Libraries/LibJS/Interpreter/interpreter.flap', 'Libraries/LibJS/Bytecode/Bytecode.def', ':Libraries/LibJS/Interpreter/layout.conf'], + srcs = ['Libraries/LibJS/Interpreter/interpreter.flap', ':Libraries/LibJS/Interpreter/layout.conf'], outs = ['Libraries/LibJS/Interpreter/interpreter_x86_64.S'], tools = ['//:flapc'], - cmd = "$(location //:flapc) --arch x86_64 --object-format elf --constants $(location :Libraries/LibJS/Interpreter/layout.conf) --bytecode-def $(location Libraries/LibJS/Bytecode/Bytecode.def) --input $(location Libraries/LibJS/Interpreter/interpreter.flap) --output $@", + cmd = "$(location //:flapc) --arch x86_64 --object-format elf --constants $(location :Libraries/LibJS/Interpreter/layout.conf) --input $(location Libraries/LibJS/Interpreter/interpreter.flap) --output $@", ) cc_library( name = 'generated_libraries_headers', - hdrs = [':Libraries/LibGfx/ImageFormats/TIFFMetadata.h', ':Libraries/LibHTTP/HSTSPreloadData.h', ':Libraries/LibJS/Bytecode/Op.h', ':Libraries/LibJS/Bytecode/OpCodes.h', ':Libraries/LibWebView/UIProcessClientEndpoint.h', ':Libraries/LibWebView/UIProcessServerEndpoint.h'], + hdrs = [':Libraries/LibGfx/ImageFormats/TIFFMetadata.h', ':Libraries/LibHTTP/HSTSPreloadData.h', ':Libraries/LibJS/Bytecode/Op.h', ':Libraries/LibJS/Bytecode/OpCodes.h', ':Libraries/LibMedia/VideoPresentation/VideoPresentationClientEndpoint.h', ':Libraries/LibMedia/VideoPresentation/VideoPresentationServerEndpoint.h', ':Libraries/LibWebView/UIProcessClientEndpoint.h', ':Libraries/LibWebView/UIProcessServerEndpoint.h'], includes = ['Libraries'], ) cc_library( diff --git a/examples/ladybird/workspace/export_headers.bzl b/examples/ladybird/workspace/export_headers.bzl new file mode 100644 index 0000000..460bd29 --- /dev/null +++ b/examples/ladybird/workspace/export_headers.bzl @@ -0,0 +1,106 @@ +# AUTO-GENERATED by Meta/emit_export_headers_bazel.py — do not edit. +# CMake's generate_export_header() output for the 15 libraries that +# opt into explicit symbol export, emitted as genrules so building the +# browser needs no CMake build. Byte-verified against the reference +# tree by `Meta/emit_export_headers_bazel.py --check Build/full`. +load("@rules_cc//cc:defs.bzl", "cc_library") + +def export_headers(): + native.genrule( + name = 'gen_LibCore_Export_h', + outs = ['Libraries/LibCore/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef CORE_API_H\n#define CORE_API_H\n\n#ifdef LIBCORE_STATIC_DEFINE\n# define CORE_API\n# define LIBCORE_NO_EXPORT\n#else\n# ifndef CORE_API\n# ifdef LibCore_EXPORTS\n /* We are building this library */\n# define CORE_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define CORE_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBCORE_NO_EXPORT\n# define LIBCORE_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBCORE_DEPRECATED\n# define LIBCORE_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBCORE_DEPRECATED_EXPORT\n# define LIBCORE_DEPRECATED_EXPORT CORE_API LIBCORE_DEPRECATED\n#endif\n\n#ifndef LIBCORE_DEPRECATED_NO_EXPORT\n# define LIBCORE_DEPRECATED_NO_EXPORT LIBCORE_NO_EXPORT LIBCORE_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBCORE_NO_DEPRECATED\n# define LIBCORE_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* CORE_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibDNS_Export_h', + outs = ['Libraries/LibDNS/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef DNS_API_H\n#define DNS_API_H\n\n#ifdef LIBDNS_STATIC_DEFINE\n# define DNS_API\n# define LIBDNS_NO_EXPORT\n#else\n# ifndef DNS_API\n# ifdef LibDNS_EXPORTS\n /* We are building this library */\n# define DNS_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define DNS_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBDNS_NO_EXPORT\n# define LIBDNS_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBDNS_DEPRECATED\n# define LIBDNS_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBDNS_DEPRECATED_EXPORT\n# define LIBDNS_DEPRECATED_EXPORT DNS_API LIBDNS_DEPRECATED\n#endif\n\n#ifndef LIBDNS_DEPRECATED_NO_EXPORT\n# define LIBDNS_DEPRECATED_NO_EXPORT LIBDNS_NO_EXPORT LIBDNS_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBDNS_NO_DEPRECATED\n# define LIBDNS_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* DNS_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibDatabase_Export_h', + outs = ['Libraries/LibDatabase/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef DATABASE_API_H\n#define DATABASE_API_H\n\n#ifdef LIBDATABASE_STATIC_DEFINE\n# define DATABASE_API\n# define LIBDATABASE_NO_EXPORT\n#else\n# ifndef DATABASE_API\n# ifdef LibDatabase_EXPORTS\n /* We are building this library */\n# define DATABASE_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define DATABASE_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBDATABASE_NO_EXPORT\n# define LIBDATABASE_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBDATABASE_DEPRECATED\n# define LIBDATABASE_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBDATABASE_DEPRECATED_EXPORT\n# define LIBDATABASE_DEPRECATED_EXPORT DATABASE_API LIBDATABASE_DEPRECATED\n#endif\n\n#ifndef LIBDATABASE_DEPRECATED_NO_EXPORT\n# define LIBDATABASE_DEPRECATED_NO_EXPORT LIBDATABASE_NO_EXPORT LIBDATABASE_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBDATABASE_NO_DEPRECATED\n# define LIBDATABASE_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* DATABASE_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibDevTools_Export_h', + outs = ['Libraries/LibDevTools/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef DEVTOOLS_API_H\n#define DEVTOOLS_API_H\n\n#ifdef LIBDEVTOOLS_STATIC_DEFINE\n# define DEVTOOLS_API\n# define LIBDEVTOOLS_NO_EXPORT\n#else\n# ifndef DEVTOOLS_API\n# ifdef LibDevTools_EXPORTS\n /* We are building this library */\n# define DEVTOOLS_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define DEVTOOLS_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBDEVTOOLS_NO_EXPORT\n# define LIBDEVTOOLS_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBDEVTOOLS_DEPRECATED\n# define LIBDEVTOOLS_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBDEVTOOLS_DEPRECATED_EXPORT\n# define LIBDEVTOOLS_DEPRECATED_EXPORT DEVTOOLS_API LIBDEVTOOLS_DEPRECATED\n#endif\n\n#ifndef LIBDEVTOOLS_DEPRECATED_NO_EXPORT\n# define LIBDEVTOOLS_DEPRECATED_NO_EXPORT LIBDEVTOOLS_NO_EXPORT LIBDEVTOOLS_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBDEVTOOLS_NO_DEPRECATED\n# define LIBDEVTOOLS_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* DEVTOOLS_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibGC_Export_h', + outs = ['Libraries/LibGC/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef GC_API_H\n#define GC_API_H\n\n#ifdef LIBGC_STATIC_DEFINE\n# define GC_API\n# define LIBGC_NO_EXPORT\n#else\n# ifndef GC_API\n# ifdef LibGC_EXPORTS\n /* We are building this library */\n# define GC_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define GC_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBGC_NO_EXPORT\n# define LIBGC_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBGC_DEPRECATED\n# define LIBGC_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBGC_DEPRECATED_EXPORT\n# define LIBGC_DEPRECATED_EXPORT GC_API LIBGC_DEPRECATED\n#endif\n\n#ifndef LIBGC_DEPRECATED_NO_EXPORT\n# define LIBGC_DEPRECATED_NO_EXPORT LIBGC_NO_EXPORT LIBGC_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBGC_NO_DEPRECATED\n# define LIBGC_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* GC_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibJS_Export_h', + outs = ['Libraries/LibJS/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef JS_API_H\n#define JS_API_H\n\n#ifdef LIBJS_STATIC_DEFINE\n# define JS_API\n# define LIBJS_NO_EXPORT\n#else\n# ifndef JS_API\n# ifdef LibJS_EXPORTS\n /* We are building this library */\n# define JS_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define JS_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBJS_NO_EXPORT\n# define LIBJS_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBJS_DEPRECATED\n# define LIBJS_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBJS_DEPRECATED_EXPORT\n# define LIBJS_DEPRECATED_EXPORT JS_API LIBJS_DEPRECATED\n#endif\n\n#ifndef LIBJS_DEPRECATED_NO_EXPORT\n# define LIBJS_DEPRECATED_NO_EXPORT LIBJS_NO_EXPORT LIBJS_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBJS_NO_DEPRECATED\n# define LIBJS_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* JS_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibMedia_Export_h', + outs = ['Libraries/LibMedia/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef MEDIA_API_H\n#define MEDIA_API_H\n\n#ifdef LIBMEDIA_STATIC_DEFINE\n# define MEDIA_API\n# define LIBMEDIA_NO_EXPORT\n#else\n# ifndef MEDIA_API\n# ifdef LibMedia_EXPORTS\n /* We are building this library */\n# define MEDIA_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define MEDIA_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBMEDIA_NO_EXPORT\n# define LIBMEDIA_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBMEDIA_DEPRECATED\n# define LIBMEDIA_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBMEDIA_DEPRECATED_EXPORT\n# define LIBMEDIA_DEPRECATED_EXPORT MEDIA_API LIBMEDIA_DEPRECATED\n#endif\n\n#ifndef LIBMEDIA_DEPRECATED_NO_EXPORT\n# define LIBMEDIA_DEPRECATED_NO_EXPORT LIBMEDIA_NO_EXPORT LIBMEDIA_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBMEDIA_NO_DEPRECATED\n# define LIBMEDIA_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* MEDIA_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibRegex_Export_h', + outs = ['Libraries/LibRegex/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef REGEX_API_H\n#define REGEX_API_H\n\n#ifdef LIBREGEX_STATIC_DEFINE\n# define REGEX_API\n# define LIBREGEX_NO_EXPORT\n#else\n# ifndef REGEX_API\n# ifdef LibRegex_EXPORTS\n /* We are building this library */\n# define REGEX_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define REGEX_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBREGEX_NO_EXPORT\n# define LIBREGEX_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBREGEX_DEPRECATED\n# define LIBREGEX_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBREGEX_DEPRECATED_EXPORT\n# define LIBREGEX_DEPRECATED_EXPORT REGEX_API LIBREGEX_DEPRECATED\n#endif\n\n#ifndef LIBREGEX_DEPRECATED_NO_EXPORT\n# define LIBREGEX_DEPRECATED_NO_EXPORT LIBREGEX_NO_EXPORT LIBREGEX_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBREGEX_NO_DEPRECATED\n# define LIBREGEX_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* REGEX_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibSync_Export_h', + outs = ['Libraries/LibSync/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef SYNC_API_H\n#define SYNC_API_H\n\n#ifdef LIBSYNC_STATIC_DEFINE\n# define SYNC_API\n# define LIBSYNC_NO_EXPORT\n#else\n# ifndef SYNC_API\n# ifdef LibSync_EXPORTS\n /* We are building this library */\n# define SYNC_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define SYNC_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBSYNC_NO_EXPORT\n# define LIBSYNC_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBSYNC_DEPRECATED\n# define LIBSYNC_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBSYNC_DEPRECATED_EXPORT\n# define LIBSYNC_DEPRECATED_EXPORT SYNC_API LIBSYNC_DEPRECATED\n#endif\n\n#ifndef LIBSYNC_DEPRECATED_NO_EXPORT\n# define LIBSYNC_DEPRECATED_NO_EXPORT LIBSYNC_NO_EXPORT LIBSYNC_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBSYNC_NO_DEPRECATED\n# define LIBSYNC_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* SYNC_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibTest_Export_h', + outs = ['Libraries/LibTest/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef TEST_API_H\n#define TEST_API_H\n\n#ifdef LIBTEST_STATIC_DEFINE\n# define TEST_API\n# define LIBTEST_NO_EXPORT\n#else\n# ifndef TEST_API\n# ifdef LibTest_EXPORTS\n /* We are building this library */\n# define TEST_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define TEST_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBTEST_NO_EXPORT\n# define LIBTEST_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBTEST_DEPRECATED\n# define LIBTEST_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBTEST_DEPRECATED_EXPORT\n# define LIBTEST_DEPRECATED_EXPORT TEST_API LIBTEST_DEPRECATED\n#endif\n\n#ifndef LIBTEST_DEPRECATED_NO_EXPORT\n# define LIBTEST_DEPRECATED_NO_EXPORT LIBTEST_NO_EXPORT LIBTEST_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBTEST_NO_DEPRECATED\n# define LIBTEST_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* TEST_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibTextCodec_Export_h', + outs = ['Libraries/LibTextCodec/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef TEXTCODEC_API_H\n#define TEXTCODEC_API_H\n\n#ifdef LIBTEXTCODEC_STATIC_DEFINE\n# define TEXTCODEC_API\n# define LIBTEXTCODEC_NO_EXPORT\n#else\n# ifndef TEXTCODEC_API\n# ifdef LibTextCodec_EXPORTS\n /* We are building this library */\n# define TEXTCODEC_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define TEXTCODEC_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBTEXTCODEC_NO_EXPORT\n# define LIBTEXTCODEC_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBTEXTCODEC_DEPRECATED\n# define LIBTEXTCODEC_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBTEXTCODEC_DEPRECATED_EXPORT\n# define LIBTEXTCODEC_DEPRECATED_EXPORT TEXTCODEC_API LIBTEXTCODEC_DEPRECATED\n#endif\n\n#ifndef LIBTEXTCODEC_DEPRECATED_NO_EXPORT\n# define LIBTEXTCODEC_DEPRECATED_NO_EXPORT LIBTEXTCODEC_NO_EXPORT LIBTEXTCODEC_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBTEXTCODEC_NO_DEPRECATED\n# define LIBTEXTCODEC_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* TEXTCODEC_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibWasm_Export_h', + outs = ['Libraries/LibWasm/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef WASM_API_H\n#define WASM_API_H\n\n#ifdef LIBWASM_STATIC_DEFINE\n# define WASM_API\n# define LIBWASM_NO_EXPORT\n#else\n# ifndef WASM_API\n# ifdef LibWasm_EXPORTS\n /* We are building this library */\n# define WASM_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define WASM_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBWASM_NO_EXPORT\n# define LIBWASM_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBWASM_DEPRECATED\n# define LIBWASM_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBWASM_DEPRECATED_EXPORT\n# define LIBWASM_DEPRECATED_EXPORT WASM_API LIBWASM_DEPRECATED\n#endif\n\n#ifndef LIBWASM_DEPRECATED_NO_EXPORT\n# define LIBWASM_DEPRECATED_NO_EXPORT LIBWASM_NO_EXPORT LIBWASM_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBWASM_NO_DEPRECATED\n# define LIBWASM_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* WASM_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibWebView_Export_h', + outs = ['Libraries/LibWebView/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef WEBVIEW_API_H\n#define WEBVIEW_API_H\n\n#ifdef LIBWEBVIEW_STATIC_DEFINE\n# define WEBVIEW_API\n# define LIBWEBVIEW_NO_EXPORT\n#else\n# ifndef WEBVIEW_API\n# ifdef LibWebView_EXPORTS\n /* We are building this library */\n# define WEBVIEW_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define WEBVIEW_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBWEBVIEW_NO_EXPORT\n# define LIBWEBVIEW_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBWEBVIEW_DEPRECATED\n# define LIBWEBVIEW_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBWEBVIEW_DEPRECATED_EXPORT\n# define LIBWEBVIEW_DEPRECATED_EXPORT WEBVIEW_API LIBWEBVIEW_DEPRECATED\n#endif\n\n#ifndef LIBWEBVIEW_DEPRECATED_NO_EXPORT\n# define LIBWEBVIEW_DEPRECATED_NO_EXPORT LIBWEBVIEW_NO_EXPORT LIBWEBVIEW_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBWEBVIEW_NO_DEPRECATED\n# define LIBWEBVIEW_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* WEBVIEW_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibXML_Export_h', + outs = ['Libraries/LibXML/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef XML_API_H\n#define XML_API_H\n\n#ifdef LIBXML_STATIC_DEFINE\n# define XML_API\n# define LIBXML_NO_EXPORT\n#else\n# ifndef XML_API\n# ifdef LibXML_EXPORTS\n /* We are building this library */\n# define XML_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define XML_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBXML_NO_EXPORT\n# define LIBXML_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBXML_DEPRECATED\n# define LIBXML_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBXML_DEPRECATED_EXPORT\n# define LIBXML_DEPRECATED_EXPORT XML_API LIBXML_DEPRECATED\n#endif\n\n#ifndef LIBXML_DEPRECATED_NO_EXPORT\n# define LIBXML_DEPRECATED_NO_EXPORT LIBXML_NO_EXPORT LIBXML_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBXML_NO_DEPRECATED\n# define LIBXML_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* XML_API_H */\nLADYBIRD_EOF\n", + ) + + # AK/Debug.h: configure_file(Debug.h.in) with every *_DEBUG off, + # which is what a default configure produces (--check verifies it). + native.genrule( + name = 'gen_AK_Debug_h', + srcs = ['AK/Debug.h.in'], + outs = ['genroot/AK/Debug.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n/*\n * Copyright (c) 2020-2024, the SerenityOS developers.\n *\n * SPDX-License-Identifier: BSD-2-Clause\n */\n\n#pragma once\n\n#ifndef AK_STRINGBASE_VERIFY_LAUNDER_DEBUG\n# define AK_STRINGBASE_VERIFY_LAUNDER_DEBUG 0\n#endif\n\n#ifndef AUDIO_DEBUG\n# define AUDIO_DEBUG 0\n#endif\n\n#ifndef BMP_DEBUG\n# define BMP_DEBUG 0\n#endif\n\n#ifndef CACHE_DEBUG\n# define CACHE_DEBUG 0\n#endif\n\n#ifndef CALLBACK_MACHINE_DEBUG\n# define CALLBACK_MACHINE_DEBUG 0\n#endif\n\n#ifndef CANVAS_RENDERING_CONTEXT_2D_DEBUG\n# define CANVAS_RENDERING_CONTEXT_2D_DEBUG 0\n#endif\n\n#ifndef CRYPTO_DEBUG\n# define CRYPTO_DEBUG 0\n#endif\n\n#ifndef COMPOSITOR_DEBUG\n# define COMPOSITOR_DEBUG 0\n#endif\n\n#ifndef CSS_LOADER_DEBUG\n# define CSS_LOADER_DEBUG 0\n#endif\n\n#ifndef CSS_PARSER_DEBUG\n# define CSS_PARSER_DEBUG 0\n#endif\n\n#ifndef CSS_TOKENIZER_DEBUG\n# define CSS_TOKENIZER_DEBUG 0\n#endif\n\n#ifndef CSS_TRANSITIONS_DEBUG\n# define CSS_TRANSITIONS_DEBUG 0\n#endif\n\n#ifndef CURL_DEBUG\n# define CURL_DEBUG 0\n#endif\n\n#ifndef DEVTOOLS_DEBUG\n# define DEVTOOLS_DEBUG 0\n#endif\n\n#ifndef DNS_DEBUG\n# define DNS_DEBUG 0\n#endif\n\n#ifndef EDITOR_DEBUG\n# define EDITOR_DEBUG 0\n#endif\n\n#ifndef FILE_WATCHER_DEBUG\n# define FILE_WATCHER_DEBUG 0\n#endif\n\n#ifndef FLAC_ENCODER_DEBUG\n# define FLAC_ENCODER_DEBUG 0\n#endif\n\n#ifndef FORMATTING_CONTEXT_TRACE_DEBUG\n# define FORMATTING_CONTEXT_TRACE_DEBUG 0\n#endif\n\n#ifndef GIF_DEBUG\n# define GIF_DEBUG 0\n#endif\n\n#ifndef HEAP_DEBUG\n# define HEAP_DEBUG 0\n#endif\n\n#ifndef INCREMENTAL_SWEEP_DEBUG\n# define INCREMENTAL_SWEEP_DEBUG 0\n#endif\n\n#ifndef HIGHLIGHT_FOCUSED_FRAME_DEBUG\n# define HIGHLIGHT_FOCUSED_FRAME_DEBUG 0\n#endif\n\n#ifndef HTML_SCRIPT_DEBUG\n# define HTML_SCRIPT_DEBUG 0\n#endif\n\n#ifndef HTTP_DISK_CACHE_DEBUG\n# define HTTP_DISK_CACHE_DEBUG 0\n#endif\n\n#ifndef HTTP_MEMORY_CACHE_DEBUG\n# define HTTP_MEMORY_CACHE_DEBUG 0\n#endif\n\n#ifndef HTTPJOB_DEBUG\n# define HTTPJOB_DEBUG 0\n#endif\n\n#ifndef ICO_DEBUG\n# define ICO_DEBUG 0\n#endif\n\n#ifndef IDB_DEBUG\n# define IDB_DEBUG 0\n#endif\n\n#ifndef IDL_DEBUG\n# define IDL_DEBUG 0\n#endif\n\n#ifndef IMAGE_DECODER_DEBUG\n# define IMAGE_DECODER_DEBUG 0\n#endif\n\n#ifndef IMAGE_LOADER_DEBUG\n# define IMAGE_LOADER_DEBUG 0\n#endif\n\n#ifndef JS_MODULE_DEBUG\n# define JS_MODULE_DEBUG 0\n#endif\n\n#ifndef LEXER_DEBUG\n# define LEXER_DEBUG 0\n#endif\n\n#ifndef LIBWEB_CSS_ANIMATION_DEBUG\n# define LIBWEB_CSS_ANIMATION_DEBUG 0\n#endif\n\n#ifndef LIBWEB_CSS_DEBUG\n# define LIBWEB_CSS_DEBUG 0\n#endif\n\n#ifndef LIBWEB_WASM_DEBUG\n# define LIBWEB_WASM_DEBUG 0\n#endif\n\n#ifndef LINE_EDITOR_DEBUG\n# define LINE_EDITOR_DEBUG 0\n#endif\n\n#ifndef WEBVIEW_HISTORY_DEBUG\n# define WEBVIEW_HISTORY_DEBUG 0\n#endif\n\n#ifndef LZW_DEBUG\n# define LZW_DEBUG 0\n#endif\n\n#ifndef MACH_PORT_DEBUG\n# define MACH_PORT_DEBUG 0\n#endif\n\n#ifndef MATROSKA_DEBUG\n# define MATROSKA_DEBUG 0\n#endif\n\n#ifndef MATROSKA_TRACE_DEBUG\n# define MATROSKA_TRACE_DEBUG 0\n#endif\n\n#ifndef HTML_PARSER_DEBUG\n# define HTML_PARSER_DEBUG 0\n#endif\n\n#ifndef PATH_DEBUG\n# define PATH_DEBUG 0\n#endif\n\n#ifndef PLAYBACK_MANAGER_DEBUG\n# define PLAYBACK_MANAGER_DEBUG 0\n#endif\n\n#ifndef PNG_DEBUG\n# define PNG_DEBUG 0\n#endif\n\n#ifndef PROMISE_DEBUG\n# define PROMISE_DEBUG 0\n#endif\n\n#ifndef REGEX_DEBUG\n# define REGEX_DEBUG 0\n#endif\n\n#ifndef REQUESTSERVER_DEBUG\n# define REQUESTSERVER_DEBUG 0\n#endif\n\n#ifndef REQUESTSERVER_WIRE_DEBUG\n# define REQUESTSERVER_WIRE_DEBUG 0\n#endif\n\n#ifndef RESOURCE_DEBUG\n# define RESOURCE_DEBUG 0\n#endif\n\n#ifndef SHARED_QUEUE_DEBUG\n# define SHARED_QUEUE_DEBUG 0\n#endif\n\n#ifndef SPAM_DEBUG\n# define SPAM_DEBUG 0\n#endif\n\n#ifndef STRUCTURED_SERIALIZE_DEBUG\n# define STRUCTURED_SERIALIZE_DEBUG 0\n#endif\n\n#ifndef SYNTAX_HIGHLIGHTING_DEBUG\n# define SYNTAX_HIGHLIGHTING_DEBUG 0\n#endif\n\n#ifndef TEXTEDITOR_DEBUG\n# define TEXTEDITOR_DEBUG 0\n#endif\n\n#ifndef TIFF_DEBUG\n# define TIFF_DEBUG 0\n#endif\n\n#ifndef TLS_DEBUG\n# define TLS_DEBUG 0\n#endif\n\n#ifndef TOKENIZER_TRACE_DEBUG\n# define TOKENIZER_TRACE_DEBUG 0\n#endif\n\n#ifndef UPDATE_LAYOUT_DEBUG\n# define UPDATE_LAYOUT_DEBUG 0\n#endif\n\n#ifndef URL_PARSER_DEBUG\n# define URL_PARSER_DEBUG 0\n#endif\n\n#ifndef URL_PATTERN_DEBUG\n# define URL_PATTERN_DEBUG 0\n#endif\n\n#ifndef UTF8_DEBUG\n# define UTF8_DEBUG 0\n#endif\n\n#ifndef VIDEO_FRAME_POOL_DEBUG\n# define VIDEO_FRAME_POOL_DEBUG 0\n#endif\n\n#ifndef VULKAN_VALIDATION_LAYERS_DEBUG\n# define VULKAN_VALIDATION_LAYERS_DEBUG 0\n#endif\n\n#ifndef WASI_DEBUG\n# define WASI_DEBUG 0\n#endif\n\n#ifndef WASI_FINE_GRAINED_DEBUG\n# define WASI_FINE_GRAINED_DEBUG 0\n#endif\n\n#ifndef WASM_BINPARSER_DEBUG\n# define WASM_BINPARSER_DEBUG 0\n#endif\n\n#ifndef WASM_CRANELIFT_DEBUG\n# define WASM_CRANELIFT_DEBUG 0\n#endif\n\n#ifndef WASM_TRACE_DEBUG\n# define WASM_TRACE_DEBUG 0\n#endif\n\n#ifndef WASM_VALIDATOR_DEBUG\n# define WASM_VALIDATOR_DEBUG 0\n#endif\n\n#ifndef WEBDRIVER_DEBUG\n# define WEBDRIVER_DEBUG 0\n#endif\n\n#ifndef WEBDRIVER_ROUTE_DEBUG\n# define WEBDRIVER_ROUTE_DEBUG 0\n#endif\n\n#ifndef VIDEO_PRESENTATION_CHANNEL_DEBUG\n# define VIDEO_PRESENTATION_CHANNEL_DEBUG 0\n#endif\n\n#ifndef WEBGL_CONTEXT_DEBUG\n# define WEBGL_CONTEXT_DEBUG 0\n#endif\n\n#ifndef WEBVIEW_PROCESS_DEBUG\n# define WEBVIEW_PROCESS_DEBUG 0\n#endif\n\n#ifndef WEB_FETCH_DEBUG\n# define WEB_FETCH_DEBUG 0\n#endif\n\n#ifndef WEB_WORKER_DEBUG\n# define WEB_WORKER_DEBUG 0\n#endif\n\n#ifndef WEBP_DEBUG\n# define WEBP_DEBUG 0\n#endif\n\n#ifndef XML_PARSER_DEBUG\n# define XML_PARSER_DEBUG 0\n#endif\nLADYBIRD_EOF\n", + ) + # AK/Backtrace.h: find_package(Backtrace) is a HOST QUESTION, so the + # genrule compiles a probe instead of baking in this machine's answer. + native.genrule( + name = 'gen_AK_Backtrace_h', + srcs = ['AK/Backtrace.h.in'], + outs = ['genroot/AK/Backtrace.h'], + cmd = '\nset -e\ntmp=$$(mktemp -d)\nprintf \'#include \\nint main(){void*b[1];backtrace(b,1);return 0;}\\n\' > $$tmp/p.c\nif $${CC:-cc} -o $$tmp/p $$tmp/p.c 2>/dev/null; then\n found=1\nelse\n found=0\nfi\nrm -rf $$tmp\nif [ "$$found" = 1 ]; then\n sed -e \'s|^#cmakedefine Backtrace_FOUND$$|#define Backtrace_FOUND|\' \\\n -e \'s|@Backtrace_HEADER@|execinfo.h|\' $< > $@\nelse\n sed -e \'s|^#cmakedefine Backtrace_FOUND$$|/* #undef Backtrace_FOUND */|\' \\\n -e \'s|@Backtrace_HEADER@||\' $< > $@\nfi\n', + ) + + cc_library( + name = 'generated_export_headers', + hdrs = ['Libraries/LibCore/Export.h', 'Libraries/LibDNS/Export.h', 'Libraries/LibDatabase/Export.h', 'Libraries/LibDevTools/Export.h', 'Libraries/LibGC/Export.h', 'Libraries/LibJS/Export.h', 'Libraries/LibMedia/Export.h', 'Libraries/LibRegex/Export.h', 'Libraries/LibSync/Export.h', 'Libraries/LibTest/Export.h', 'Libraries/LibTextCodec/Export.h', 'Libraries/LibWasm/Export.h', 'Libraries/LibWebView/Export.h', 'Libraries/LibXML/Export.h'], + includes = ['Libraries'], + ) + cc_library( + name = 'generated_ak_headers', + hdrs = ['genroot/AK/Debug.h', 'genroot/AK/Backtrace.h'], + includes = ['genroot'], + ) diff --git a/examples/ladybird/workspace/hsts_preload.bzl b/examples/ladybird/workspace/hsts_preload.bzl new file mode 100644 index 0000000..8aa5eb7 --- /dev/null +++ b/examples/ladybird/workspace/hsts_preload.bzl @@ -0,0 +1,55 @@ +# Chromium's HSTS preload table, pinned to a commit. GENERATED by +# Meta/pin_hsts_preload.py -- see that script to re-pin; do not hand-edit. +# +# WHY THIS FILE EXISTS. `Meta/CMake/hsts_preload.cmake` downloads +# net/http/transport_security_state_static.json from Chromium's **main** at +# configure time -- an unversioned ref. The generator turns it into a ~95,000 +# entry `constexpr Array` of domains LibHTTP forces to HTTPS, so "whatever main +# served the day you configured" decides a security-relevant table, and two +# developers who configured on different days build different browsers. +# +# The fix belongs upstream (pin the URL in hsts_preload.cmake). We cannot make +# that change from here, so we pin DOWNSTREAM: Bazel fetches one immutable +# commit URL with a sha256, and the upstream unpinned fetch is filed as a bug. +# +# Pinning downstream is only honest if it does not break byte-parity with CMake, +# and this pin does not: the bytes this commit serves are byte-identical to the +# ones CMake's `main` fetch downloaded for the reference build (10,521,748 bytes, +# checked with --expect-same-as, which refuses to write this file otherwise). +# Pinning to a Chromium *release tag* would NOT have that property: 139.0.7258.5 +# serves 18.7 MB and generates 168,593 entries against this commit's 94,626. +# +# Two consequences worth knowing: +# +# * CMake still tracks `main`, so a CMake configure NEWER than this pin will +# disagree with Bazel. That is now a visible, dated disagreement between one +# pinned input and one unpinned one, instead of two unpinned fetches that +# happened to agree. `download_file` is a no-op when the file already exists +# (verified with ENABLE_NETWORK_DOWNLOADS=OFF), so staging the Bazel-fetched +# file into Build/caches/HSTSPreload/ before configuring makes CMake consume +# this same pin -- see README, "Getting the three inputs". +# * Bumping the pin is a deliberate, reviewable act: re-run +# Meta/pin_hsts_preload.py, which resolves the newest commit touching the +# path, downloads it, and writes this file with the hash it measured. +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + +# The pinned commit: "[HSTS] Update bulk entries" (2026-07-24). +# raw.githubusercontent.com paths are immutable for a full commit sha, unlike the +# `main` path CMake uses. +HSTS_PRELOAD_COMMIT = "3d75766484199c1fbefd269a4b168cccdb36fbca" + +HSTS_PRELOAD_SHA256 = "59c3301277b3418e716d818d26110edb0737edb871c5bba7462158da1b2d1bd9" + +HSTS_PRELOAD_URL = "https://raw.githubusercontent.com/chromium/chromium/{}/net/http/transport_security_state_static.json".format( + HSTS_PRELOAD_COMMIT, +) + +def _hsts_preload_impl(_ctx): + http_file( + name = "hsts_preload_json", + urls = [HSTS_PRELOAD_URL], + sha256 = HSTS_PRELOAD_SHA256, + downloaded_file_path = "transport_security_state_static.json", + ) + +hsts_preload = module_extension(implementation = _hsts_preload_impl) diff --git a/examples/ladybird/workspace/qt_runtime.bzl b/examples/ladybird/workspace/qt_runtime.bzl new file mode 100644 index 0000000..541ea6f --- /dev/null +++ b/examples/ladybird/workspace/qt_runtime.bzl @@ -0,0 +1,599 @@ +# Qt's RUNTIME half: the plugins, staged next to the binary, from the SAME Qt as +# the linked libraries. +# +# WHY THIS FILE EXISTS -- the bug it fixes (finding 40). +# +# rules_qt's `qt.local_repo` makes Qt's *link* half hermetic-ish: the cc_librarys +# under @qt point at one SDK, discovered through `qmake -query`, and Bazel links +# libQt6Core from there (via _solib_k8). Qt's *runtime* half was never wired up at +# all, and Qt does not take the hint: at QApplication construction it dlopens the +# QPA platform plugin (libqxcb.so) from a path baked into libQt6Core -- its build +# prefix -- or, when that prefix is empty, from the DIRECTORY OF THE EXECUTABLE. +# A Bazel binary's directory has no `platforms/`, so the search falls through to +# the compiled-in system path and Qt loads the DISTRO's plugin into a process +# whose Qt libraries came from the Bazel repo. +# +# Two different Qt builds in one process. What that does depends on which way the +# skew points, and both halves are reproduced (with a real X server): +# +# plugin OLDER than libs -> Qt's version gate rejects it: +# "Ignoring QPA plugin due to mismatching Qt +# versions 395520 394240" -> "no Qt platform plugin +# could be initialized", abort. +# plugin NEWER than libs -> the gate PASSES, the plugin loads, and it calls +# into a libQt6Core whose ABI it was not built +# against -> SIGSEGV in QXcbConnection:: +# initializeScreens -> handleScreenAdded. +# +# The second one is the crash reported on Ubuntu 24.04 (aqt Qt 6.9.2 linked, +# distro plugins in /usr/lib/x86_64-linux-gnu/qt6/plugins). On a machine where +# the SDK and the distro happen to be the same version it works -- BY ACCIDENT, +# which is how it survived this long here (`QT_DEBUG_PLUGINS=1` on this box shows +# the same wrong scan, loading /usr's libqxcb.so into @qt's libQt6Core; the two +# are both 6.10.2, so nothing breaks). +# +# HOW IT IS FIXED. +# +# 1. `qt_plugins` (below) is a repository rule that reads @qt's OWN generated +# qtconf.bzl -- not a hand-written path -- and symlinks every plugin under +# that SDK's QT_INSTALL_PLUGINS into a repo, one filegroup per plugin type. +# Deriving the path from @qt is the whole point: the plugins cannot come +# from a different Qt than the libraries, because both names come from one +# `qmake -query`. +# 2. `qt_plugin_tree` re-declares those files as outputs of the ROOT package at +# `plugins//.so`, so they land next to the binary in bazel-bin +# (and, as data, in the runfiles tree too). +# 3. `qt_conf` writes the four-line qt.conf that points Qt at them: +# +# [Paths] +# Prefix = . +# Plugins = plugins +# +# Qt reads qt.conf from the directory of the *resolved* executable (it uses +# /proc/self/exe, so running through a symlink -- `bazel run`, or the +# runfiles tree's own symlink to bazel-bin -- still finds it). `Prefix = .` +# is what makes ONE file correct in BOTH layouts: bazel-bin/ladybird sees +# bazel-bin/plugins, and runfiles/_main/ladybird resolves to the same place. +# +# Setting Prefix also REPLACES the compiled-in prefix, so /usr's plugin directory +# is not merely outranked, it is never scanned. Verified by removal: with an empty +# directory bind-mounted over the host plugin dir, the binary still starts. +# +# The staged files are SYMLINKS to the SDK's plugins on purpose. A plugin needs Qt +# libraries the binary does not link (libqxcb.so needs libQt6XcbQpa.so.6), and +# aqt's plugins carry `RUNPATH $ORIGIN/../../lib`; $ORIGIN is resolved from the +# object's real path, so a symlinked plugin finds its own SDK's private libs with +# no rpath work and no LD_LIBRARY_PATH. Copying them would break exactly that -- +# also checked, and it fails the way the report describes. + +# The version floor Ladybird's own UI/Qt/CMakeLists.txt declares. +_QT_FLOOR = (6, 9) + +# The Qt MODULES //:ladybird links, and the Debian/Ubuntu package that ships each +# one. Checked here because rules_qt's qt.local_repo DERIVES its cc_library targets +# by listing the host's Qt lib directory: a module the host does not have is simply +# not declared, and the failure is Bazel's generic missing-target error naming a +# generated BUILD file in the output base -- +# +# ERROR: .../external/rules_qt++qt+qt/BUILD.bazel: no such target +# '@@rules_qt++qt+qt//:QtPositioning': target 'QtPositioning' not declared in +# package '' ... and referenced by '//:ladybird' +# +# -- which says nothing about Qt, nothing about apt, and points at a file the +# reader did not write and cannot fix. Ulf hit exactly this. (Same class as +# finding 38: a host probe whose absence is reported as a bug in your code.) +# +# The floor check right below this was already the right idea and had the wrong +# scope: it asked "is the SDK new enough" and never "does the SDK have the parts we +# link". Both are properties of the discovered SDK, so both belong here. +# +# Kept as a LIST OF NAMES, not derived from BUILD.bazel: the emitter writes the +# `@qt//:Qt*` deps, so deriving this from the same source would only prove the +# generator agrees with itself. This is the independent statement of what the build +# needs, and the test asserts the two match -- which is what catches a NEW Qt +# module appearing in a future repin without its preflight entry. +# +# Each entry names the Debian package AND the aqt module, because WHICH ONE IS THE +# RIGHT ADVICE DEPENDS ON THE SDK, and getting that wrong is worse than saying +# nothing. Ulf builds against Qt 6.9.2 in a venv while his system Qt is 6.4.2: for +# him `apt install qt6-positioning-dev` drops libQt6Positioning.so into +# /usr/lib/x86_64-linux-gnu, which is NOT the lib directory the discovered SDK +# reports, so the module stays missing, the error is unchanged, and the reader +# reasonably concludes the advice was wrong -- because it was. The message picks the +# form that matches the SDK it actually found (see _install_hint). +_QT_MODULES = { + "QtCore": ("qt6-base-dev", "qtbase"), + "QtGui": ("qt6-base-dev", "qtbase"), + "QtWidgets": ("qt6-base-dev", "qtbase"), + # UI/Qt/CMakeLists.txt:8 -- REQUIRED on non-Apple since the 71fb301a repin + # (it was OPTIONAL before), for GeolocationProviderQt.cpp. + "QtPositioning": ("qt6-positioning-dev", "qtpositioning"), +} + +# --------------------------------------------------------------------------- +# The repo: @qt's plugins, as Bazel files. +# --------------------------------------------------------------------------- + +_BUILD_HEADER = """# GENERATED by qt_runtime.bzl (qt_plugins). Do not edit. +load("@rules_cc//cc:defs.bzl", "cc_import", "cc_library") + +package(default_visibility = ["//visibility:public"]) +""" + +# The OTHER half of "no LD_LIBRARY_PATH": the PRIVATE libraries an SDK bundles. +# +# rules_qt's cc_librarys stage libQt6*.so into _solib_k8 and nothing else. An +# official Qt SDK also bundles its own ICU -- aqt 6.9.2's libQt6Core needs +# libicui18n.so.73, which exists in the SDK's lib/ and nowhere else on a machine +# whose distro ICU is 78 -- so the binary died in the loader before main() and +# `LD_LIBRARY_PATH=/lib` was the workaround. A workaround a human has to +# remember is a bug that has been rounded down to a habit. +# +# WHY AN RPATH ON THE BINARY CANNOT FIX THIS, which took three reductions to +# believe. libQt6Core resolves its own ICU through `RUNPATH $ORIGIN`, and $ORIGIN is +# the directory the loader OPENED the object by -- which is Bazel's solib dir, not +# the SDK. (`ldd` on the same symlink resolves ICU happily, because ldd's $ORIGIN is +# the realpath's dir; that near-miss is what made this look like a path problem.) +# Adding the SDK dir to OUR rpath does not help either, and the reason is a glibc +# rule worth writing down: DT_RUNPATH is consulted only for an object's own direct +# dependencies, and while DT_RPATH IS inherited by transitive loads, an +# intermediate object that has a DT_RUNPATH **of its own** blocks the inherited +# DT_RPATH entirely. libQt6Core has one. Measured on three generated .so files, all +# four combinations, before believing it. +# +# So the fix is not a search path at all: make the SDK's private libraries real +# link inputs, so BAZEL stages them into a solib dir and the binary's OWN runpath +# resolves them. Ours is the runpath glibc will consult, because they are now our +# direct dependencies. Verified with the loader trace, then end to end. +# +# The list is DERIVED, not written down: the intersection of "libraries the SDK's +# Qt modules declare in DT_NEEDED" with "libraries the SDK ships beside them". For +# aqt 6.9.2 that is exactly libicui18n/libicuuc/libicudata .so.73; for a distro Qt +# it is EMPTY (a distro's ICU is a distro package, already on the default search +# path) and this target degenerates to nothing, which is the correct answer rather +# than a special case. +_RUNTIME_LIB_IMPORT = """ +cc_import( + name = "{name}", + shared_library = "{lib}", +) +""" + +_RUNTIME_LIBS_EMPTY = """ +# This Qt bundles no private libraries of its own (a distro Qt: its ICU is a distro +# package, already on the loader's default search path). +cc_library( + name = "runtime_libs", +) +""" + +_RUNTIME_LIBS_GROUP = """ +cc_library( + name = "runtime_libs", + deps = [{deps}], + # --no-as-needed: the BINARY does not reference an ICU 73 symbol (it has its own + # ICU 78 from vcpkg), so the linker would drop the DT_NEEDED as unused and we + # would be back to libQt6Core searching a directory that has no ICU in it. + linkopts = ["-Wl,--no-as-needed"], +) +""" + +def _parse_qtconf(content): + """Reads @qt's generated qtconf.bzl as data: KEY="value" lines.""" + values = {} + for line in content.splitlines(): + line = line.strip() + if line.startswith("#") or "=" not in line: + continue + key, _, raw = line.partition("=") + raw = raw.strip() + if not raw.startswith("\"") or not raw.endswith("\""): + continue + values[key.strip()] = raw[1:-1] + return values + +# Directories that ARE the loader's default search path. A Qt whose libraries live +# in one of these is a distro Qt: everything it depends on is already findable, and +# there is nothing to stage. Getting this wrong is not a small mistake -- the first +# version of this derivation asked "which non-Qt .so files sit beside libQt6Core", +# which for a distro Qt is /usr/lib/x86_64-linux-gnu, i.e. it proposed to link the +# ENTIRE system library directory into the binary (56 cc_imports, ld-linux among +# them). It is a system directory, so it "worked"; that is exactly the kind of +# accident this whole finding is about. +_SYSTEM_LIB_DIRS = [ + "/lib", + "/lib64", + "/lib/x86_64-linux-gnu", + "/lib/aarch64-linux-gnu", + "/usr/lib", + "/usr/lib64", + "/usr/lib/x86_64-linux-gnu", + "/usr/lib/aarch64-linux-gnu", + "/usr/local/lib", +] + +def _sdk_private_libs(repository_ctx, libs_root): + """The SDK's bundled non-Qt libraries that its OWN Qt modules depend on. + + Derived, not listed: read DT_NEEDED out of the SDK's libQt6*.so with objdump and + intersect it with the non-Qt .so files sitting in the same directory. For an + official SDK that yields its bundled ICU; for a distro Qt it yields nothing, + because the lib dir IS a system dir (see _SYSTEM_LIB_DIRS) and everything in it + is already on the loader's default search path. + """ + if libs_root.rstrip("/") in _SYSTEM_LIB_DIRS: + return [] + + root = repository_ctx.path(libs_root) + if not root.exists: + return [] + + # What the directory ships that is not Qt itself, keyed by soname-ish basename. + present = {} + for entry in root.readdir(): + name = str(entry.basename) + if name.startswith("libQt") or ".so" not in name: + continue + present[name] = "lib/{}".format(name) + + if not present: + return [] + + qt_modules = [ + "{}/{}".format(libs_root, str(e.basename)) + for e in root.readdir() + if str(e.basename).startswith("libQt") and ".so." in str(e.basename) + ] + if not qt_modules: + return [] + + # objdump is in binutils, i.e. present anywhere a C++ toolchain is. If it is + # not, say so rather than silently returning "no private libraries" -- that + # would look exactly like a distro Qt and reintroduce the LD_LIBRARY_PATH need + # with no diagnostic (finding 35: a check that cannot fail must not look like a + # check that passed). + result = repository_ctx.execute(["objdump", "-p"] + qt_modules) + if result.return_code != 0: + fail(("qt_plugins: cannot read DT_NEEDED from the Qt libraries in {d}: " + + "objdump failed ({e}). objdump comes with binutils; it is needed here " + + "to discover the private libraries an SDK bundles beside Qt (an " + + "official SDK ships its own ICU), because without them the binary " + + "cannot start without LD_LIBRARY_PATH.").format( + d = libs_root, + e = result.stderr.strip(), + )) + + needed = {} + for line in result.stdout.splitlines(): + line = line.strip() + if not line.startswith("NEEDED"): + continue + # Starlark's split() has no whitespace-default: give it one explicitly and + # drop the empty fields the double spaces in objdump's output produce. + fields = [f for f in line.split(" ") if f] + if len(fields) < 2: + continue + soname = fields[-1] + if soname in present: + needed[soname] = present[soname] + + # Symlink each one in, and return the (label-safe) target names. + out = [] + for soname in sorted(needed.keys()): + repository_ctx.symlink( + repository_ctx.path("{}/{}".format(libs_root, soname)), + needed[soname], + ) + out.append(needed[soname]) + return out + +def _is_system_qt(libs_dir): + """Is the discovered SDK the DISTRO's Qt, or a self-contained one? + + Decides which install instruction can possibly work. A distro Qt's modules are + apt packages; a self-contained SDK (aqt/venv/official installer, or a Nix or + Homebrew prefix) has its own lib directory, and apt would install into + /usr/lib/... where that SDK never looks. + + Keyed on the LIB DIRECTORY, never on the prefix string. Two reasons, and the + second one is a test catching me: `qmake -query` can report a prefix like /usr + while the libraries live elsewhere, so the lib dir is the thing both + qt.local_repo and the module probe actually read -- AND a list of "prefixes that + mean distro" would be another hardcoded host path, the exact thing this file + exists to remove (test_plugins_come_from_the_same_sdk_as_the_libraries forbids + absolute /usr literals outside _SYSTEM_LIB_DIRS, and rightly failed on my first + version of this). + + _SYSTEM_LIB_DIRS is already the list of "directories that are the loader's + default search path", derived for the private-library staging; a Qt whose libs + are in one of them is a distro Qt by the same definition. + """ + return libs_dir.rstrip("/") in _SYSTEM_LIB_DIRS + +def _install_hint(missing, libs_dir): + """The instruction that fits the SDK we found -- not a guess between two. + + Both forms are shown when the SDK is self-contained, because we cannot know + HOW it was built (aqt, the official installer, Nix, a distro-Qt venv that only + wraps the tools), and a reader who is told only "apt install" will do it, see + no change, and lose trust in the message. Naming the lib directory we probed is + what lets them check the claim themselves. + """ + debs = sorted({_QT_MODULES[m][0]: True for m in missing}.keys()) + aqts = sorted({_QT_MODULES[m][1]: True for m in missing}.keys()) + if _is_system_qt(libs_dir): + return (" This looks like your DISTRIBUTION's Qt, so on Debian/Ubuntu:\n\n" + + " sudo apt install {debs}\n").format(debs = " ".join(debs)) + return ( + " This is a SELF-CONTAINED Qt (its libraries live in {libs}, not in a\n" + + " system directory), so `apt install` CANNOT fix it: apt installs into\n" + + " /usr/lib/..., which this SDK never looks in, and you would get this same\n" + + " error again. Add the module to THIS SDK instead. With aqt:\n\n" + + " aqtinstall ... --modules {aqts}\n" + + " # or: aqt install-qt linux desktop --modules {aqts}\n\n" + + " With the official online installer, tick the module under your Qt version.\n" + + " Alternatively point qt.local_repo's `paths` in MODULE.bazel at a Qt that\n" + + " has the module -- but see the note below: it must be ONE Qt, not a mix.\n" + ).format(libs = libs_dir or "?", aqts = " ".join(aqts)) + +def _version_tuple(version): + parts = version.split(".") + nums = [] + for p in parts[:3]: + if not p.isdigit(): + return None + nums.append(int(p)) + if len(nums) < 2: + return None + return nums + +def _qt_plugins_impl(repository_ctx): + # @qt writes this file from `qmake -query`; reading it as DATA (rather than + # loading it, which a repository rule cannot do) is what ties the plugins to + # the same SDK as the libraries. + qtconf_label = Label("@qt//:qtconf.bzl") + values = _parse_qtconf(repository_ctx.read(qtconf_label)) + + version = values.get("QT_VERSION", "") + plugins_root = values.get("QT_INSTALL_PLUGINS", "") + if not version or not plugins_root: + fail("qt_plugins: @qt//:qtconf.bzl has no QT_VERSION/QT_INSTALL_PLUGINS. " + + "rules_qt's qt.local_repo generates it from `qmake -query`; if the keys " + + "moved, this rule has to follow them.") + + # Preflight the version FLOOR Ladybird's own CMake declares, in the place that + # can still say something useful about it. UI/Qt/CMakeLists.txt has + # `find_package(Qt6 6.9 REQUIRED COMPONENTS Core Widgets)`; CMake refuses an + # older Qt with a clear message, and until this check existed the Bazel build + # just compiled against whatever qmake was first on PATH and failed later -- + # at moc time, at link time, or (finding 40) not at all until the GUI crashed. + have = _version_tuple(version) + if have == None: + fail("qt_plugins: cannot parse QT_VERSION {!r} from @qt".format(version)) + if (have[0], have[1]) < (_QT_FLOOR[0], _QT_FLOOR[1]): + fail(("qt_plugins: Qt {have} is too old.\n\n" + + " Ladybird requires Qt >= {floor} (UI/Qt/CMakeLists.txt:\n" + + " find_package(Qt6 {floor} REQUIRED COMPONENTS Core Widgets)).\n" + + " The SDK rules_qt discovered is {have} at {prefix}.\n\n" + + " Point qt.local_repo's `paths` in MODULE.bazel at a newer Qt, or\n" + + " install one (Debian/Ubuntu: qt6-base-dev >= {floor}; otherwise an\n" + + " official Qt SDK). Do NOT mix: the plugins are taken from this same\n" + + " SDK, and a mixed pair is the crash finding 40 is about.").format( + have = version, + floor = "{}.{}".format(_QT_FLOOR[0], _QT_FLOOR[1]), + prefix = values.get("QT_INSTALL_PREFIX", "?"), + )) + + # Preflight the MODULES, for the reason spelled out at _QT_MODULES: a module the + # host Qt lacks is never declared by qt.local_repo, and Bazel then blames a + # generated BUILD file in the output base for a missing apt package. + # + # Checked against the SDK's own lib directory (the same input qt.local_repo + # derives its targets from) rather than by asking @qt for the target: a + # repository rule cannot query another repo's targets, and reading the libs is + # what makes the answer agree with what qt.local_repo will do. + libs_dir = values.get("QT_INSTALL_LIBS", "") + if libs_dir: + libs_path = repository_ctx.path(libs_dir) + present = {} + if libs_path.exists: + for f in libs_path.readdir(): + b = str(f.basename) + # libQt6Positioning.so / .so.6 / .so.6.10.2 all mean "present"; + # _create_lib_name in qt_local_repo.bzl takes the same first field. + if b.startswith("libQt{}".format(have[0])) and ".so" in b: + present["Qt" + b.split(".")[0][len("libQt%d" % have[0]):]] = True + missing = [m for m in sorted(_QT_MODULES) if m not in present] + if missing: + prefix = values.get("QT_INSTALL_PREFIX", "?") + fail(( + "qt_plugins: the Qt this build discovered is missing {n} module(s)\n" + + " that //:ladybird links.\n\n" + + " Qt {v}\n prefix: {prefix}\n libraries: {libs}\n" + + " (that is the SDK `qmake -query` reported, i.e. whatever\n" + + " qt.local_repo's `paths` in MODULE.bazel points at)\n\n" + + " Missing:\n\n{list}\n\n" + + "{hint}\n" + + " Without this check Bazel reports the same problem as\n" + + " no such target '@@rules_qt++qt+qt//:Qt'\n" + + " naming a GENERATED BUILD file in your output base, because rules_qt's\n" + + " qt.local_repo derives its cc_library targets by listing the library\n" + + " directory above -- a module you do not have is simply never declared.\n\n" + + " Do NOT satisfy this by installing the module for a DIFFERENT Qt than the\n" + + " one named above: the plugins are taken from this same SDK, and mixing two\n" + + " Qt builds in one process is the crash this file's header is about.\n\n" + + " (If you installed it just now, Bazel may have the old @qt cached:\n" + + " `bazel sync --configure` or `bazel clean --expunge` re-runs the probe.)" + ).format( + prefix = prefix, + libs = libs_dir, + v = version, + n = len(missing), + list = "\n".join([" " + m for m in missing]), + hint = _install_hint(missing, libs_dir), + )) + + root = repository_ctx.path(plugins_root) + if not root.exists: + fail(("qt_plugins: Qt {v} reports its plugins live in\n {p}\n" + + "but that directory does not exist. A Qt with no QPA plugin cannot open a\n" + + "window: install the platform plugins (Ubuntu: qt6-base-dev pulls\n" + + "libqt6gui6, which ships plugins/platforms/libqxcb.so).").format( + v = version, + p = plugins_root, + )) + + # Every plugin type the SDK ships, symlinked file by file. Deliberately NOT a + # hand-picked list of the four types Ladybird happens to need today: a list is + # a thing that drifts, symlinks cost nothing, and the failure mode of a + # missing plugin type (no input method, no native file dialog, no icons) is + # the sort of thing nobody notices for a month. + groups = {} + for type_dir in sorted([str(p.basename) for p in root.readdir()]): + src_dir = repository_ctx.path("{}/{}".format(plugins_root, type_dir)) + if not src_dir.is_dir: + continue + files = [] + for plugin in sorted([str(p.basename) for p in src_dir.readdir()]): + if not plugin.endswith(".so"): + continue + repository_ctx.symlink( + repository_ctx.path("{}/{}/{}".format(plugins_root, type_dir, plugin)), + "plugins/{}/{}".format(type_dir, plugin), + ) + files.append("plugins/{}/{}".format(type_dir, plugin)) + if files: + groups[type_dir] = files + + if "platforms" not in groups: + fail(("qt_plugins: {p} has no `platforms/` directory, so there is no QPA " + + "plugin to load and the GUI cannot start.").format(p = plugins_root)) + + # The private libraries the SDK's Qt modules need and the SDK itself ships. + private_libs = _sdk_private_libs(repository_ctx, values.get("QT_INSTALL_LIBS", "")) + + lines = [_BUILD_HEADER] + if private_libs: + names = [] + for lib in private_libs: + target = lib[len("lib/"):].replace(".", "_") + names.append(target) + lines.append(_RUNTIME_LIB_IMPORT.format(name = target, lib = lib)) + lines.append(_RUNTIME_LIBS_GROUP.format( + deps = ", ".join(["\":{}\"".format(n) for n in names]), + )) + else: + # A distro Qt: nothing to stage, and an empty target so the dep edge in + # BUILD.bazel is the same on every machine (finding 35 -- the alternative is + # a label that exists on some hosts and not others). + lines.append(_RUNTIME_LIBS_EMPTY) + for type_dir in sorted(groups.keys()): + lines.append("filegroup(\n name = \"{}\",\n srcs = [\n{} ],\n)\n".format( + type_dir, + "".join([" \"{}\",\n".format(f) for f in groups[type_dir]]), + )) + lines.append("filegroup(\n name = \"plugins\",\n srcs = [\n{} ],\n)\n".format( + "".join([" \":{}\",\n".format(t) for t in sorted(groups.keys())]), + )) + repository_ctx.file("BUILD.bazel", "\n".join(lines)) + + # The version the plugins ARE, recorded next to them so a consumer (and the + # provenance test) can compare it against the Qt the binary linked instead of + # trusting that they match. + repository_ctx.file("qt_plugins.bzl", "\n".join([ + "# GENERATED by qt_runtime.bzl (qt_plugins). Do not edit.", + "QT_PLUGINS_VERSION = \"{}\"".format(version), + "QT_PLUGINS_SOURCE = \"{}\"".format(plugins_root), + "QT_PLUGIN_TYPES = {}".format(str(sorted(groups.keys()))), + "", + ])) + +qt_plugins = repository_rule( + implementation = _qt_plugins_impl, + doc = """Exposes the Qt plugins of the SDK rules_qt discovered as Bazel files. + +Reads @qt's generated `qtconf.bzl` for `QT_INSTALL_PLUGINS`, so the plugins are +by construction from the same Qt as the `@qt//:Qt*` libraries the binary links -- +see the header of qt_runtime.bzl for what happens when they are not.""", + # local: the SDK is a host path, exactly like rules_qt's own qt.local_repo, + # so this must be re-evaluated rather than cached across host changes. + local = True, +) + +def _qt_runtime_ext_impl(module_ctx): + qt_plugins(name = "qt_plugins") + return module_ctx.extension_metadata(root_module_direct_deps = ["qt_plugins"], root_module_direct_dev_deps = []) + +qt_runtime = module_extension(implementation = _qt_runtime_ext_impl) + +# --------------------------------------------------------------------------- +# Staging: the plugins as outputs of the package that holds the binary. +# --------------------------------------------------------------------------- + +def _strip_to_plugins(short_path): + """`plugins/platforms/libqxcb.so` out of a runfiles-relative path. + + The path looks like `../qt_plugins/plugins/platforms/libqxcb.so`, so the match + has to be on the SEPARATED component `/plugins/` -- searching for `plugins/` + matches inside the repository name `qt_plugins/` and stages everything one + directory too deep (which is what it did, silently, until qt.conf pointed at + an empty tree). + """ + if short_path.startswith("plugins/"): + return short_path + idx = short_path.find("/plugins/") + if idx == -1: + fail("qt_plugin_tree: {} is not under a plugins/ directory".format(short_path)) + return short_path[idx + 1:] + +def _qt_plugin_tree_impl(ctx): + outs = [] + for src in ctx.files.plugins: + out = ctx.actions.declare_file(_strip_to_plugins(src.short_path)) + # A SYMLINK, not a copy: the plugin resolves its own private Qt libraries + # through `RUNPATH $ORIGIN/../../lib`, and $ORIGIN follows the real path. + ctx.actions.symlink(output = out, target_file = src) + outs.append(out) + return [DefaultInfo( + files = depset(outs), + runfiles = ctx.runfiles(files = outs), + )] + +qt_plugin_tree = rule( + implementation = _qt_plugin_tree_impl, + doc = """Stages @qt_plugins under `plugins/` in THIS package, so the files sit +next to the binary that qt.conf points at.""", + attrs = { + "plugins": attr.label_list( + allow_files = True, + doc = "Plugin files from @qt_plugins (`@qt_plugins//:plugins`).", + ), + }, +) + +def _qt_conf_impl(ctx): + out = ctx.actions.declare_file(ctx.attr.filename) + + # `Prefix = .` -- relative to the directory of the resolved executable, which + # is the one thing that is the same in bazel-bin and in the runfiles tree. + ctx.actions.write( + output = out, + content = "\n".join([ + "# GENERATED by //:{} (qt_runtime.bzl). Points Qt at the plugins staged".format(ctx.label.name), + "# beside the binary, so it cannot dlopen the host's plugins into Bazel's Qt.", + "[Paths]", + "Prefix = .", + "Plugins = {}".format(ctx.attr.plugins_dir), + "", + ]), + ) + return [DefaultInfo(files = depset([out]), runfiles = ctx.runfiles(files = [out]))] + +qt_conf = rule( + implementation = _qt_conf_impl, + doc = """Writes the qt.conf that redirects Qt's plugin search into the staged tree.""", + attrs = { + "filename": attr.string(default = "qt.conf"), + "plugins_dir": attr.string(default = "plugins"), + }, +) diff --git a/examples/ladybird/workspace/vcpkg.bzl b/examples/ladybird/workspace/vcpkg.bzl index 98ddc81..367c61f 100644 --- a/examples/ladybird/workspace/vcpkg.bzl +++ b/examples/ladybird/workspace/vcpkg.bzl @@ -42,8 +42,9 @@ def _vcpkg_tree_impl(ctx): ctx.actions.run( outputs = [out], inputs = depset( - [index] + distfile_inputs + - ctx.files.vcpkg_tree + ctx.files.source_root, + [index] + distfile_inputs + ctx.files.python_wheels + + ctx.files.vcpkg_tree + ctx.files.source_root + + ctx.files._host_tools, ), executable = ctx.executable._build, arguments = [ @@ -52,6 +53,15 @@ def _vcpkg_tree_impl(ctx): ctx.attr.vcpkg_root, ctx.attr.source_dir, ctx.attr.triplet, + # The find-links dir for pip: a comma-joined list of the wheel paths, + # empty when there are none (which is then a hard failure for any port + # that pip-installs, rather than a silent fetch). + ",".join([f.path for f in ctx.files.python_wheels]), + # The host-prerequisite list, passed BY PATH rather than found with + # `dirname $0`: an sh_binary's data lands in .runfiles/, not + # beside the wrapper Bazel execs, so dirname finds nothing once the + # action is sandboxed -- the same trap cargo_vendor.sh documents. + ctx.file._host_tools.path, ], mnemonic = "VcpkgInstall", progress_message = "Building %d vcpkg ports (no network)" % len(ctx.attr.distfiles), @@ -94,6 +104,18 @@ vcpkg_tree = rule( allow_files = True, doc = "Ladybird's manifest, overlay-ports and overlay-triplets.", ), + "python_wheels": attr.label_list( + allow_files = True, + doc = """Wheels for the Python packages a PORTFILE pip-installs. + + Staged into a find-links dir and paired with PIP_NO_INDEX, so pip + resolves offline from these files alone. Needed because pip does NOT go + through vcpkg's asset cache, so `x-block-origin` never sees it and the + distfile pin cannot cover it -- the angle port's `pip install ply` was + reaching PyPI through an inherited HTTP_PROXY for months (finding 36). + An empty list means pip has no index and no wheels, i.e. any port that + asks for a package fails loudly instead of downloading one.""", + ), "vcpkg_root": attr.string(doc = "Path of the vcpkg checkout."), "source_dir": attr.string(doc = "Path of the Ladybird source root."), "install_root": attr.string( @@ -116,6 +138,16 @@ vcpkg_tree = rule( executable = True, cfg = "exec", ), + "_host_tools": attr.label( + default = "//Meta:vcpkg_host_tools.tsv", + allow_single_file = True, + doc = """The tools this build needs FROM THE HOST (finding 39). + + vcpkg has no Linux download for these, so they cannot be pinned -- + what CAN be fixed is when you find out. The driver checks the whole + list before building anything, so a machine missing three of them is + told all three in one second instead of one per 20-minute build.""", + ), }, ) diff --git a/examples/ladybird/workspace/vcpkg_distfiles.bzl b/examples/ladybird/workspace/vcpkg_distfiles.bzl index f6912d8..8de6907 100644 --- a/examples/ladybird/workspace/vcpkg_distfiles.bzl +++ b/examples/ladybird/workspace/vcpkg_distfiles.bzl @@ -1,7 +1,7 @@ # AUTO-GENERATED by Meta/emit_vcpkg_bazel.py — do not edit. load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") -# 76 upstream distfiles. Each `integrity` is vcpkg's own published +# 77 upstream distfiles. Each `integrity` is vcpkg's own published # SHA512 from the baseline-resolved portfile, hex->base64; nothing is # re-hashed here, so this is not a trust downgrade. @@ -283,10 +283,10 @@ def vcpkg_distfiles(): integrity = 'sha512-eWnECwYAuvJ4avDmUDtCgtSHtmA0GMQfKMOznpzZMgrGbA0uj7+it5TkYfJoQ+NHnWDsJKxcCZD+jwxr+u7mnQ==', # captured ) http_file( - name = 'vcpkg_libsdl_org_SDL_release_3_4_12_tar_gz_fc0a55ca01c3', - urls = ['https://github.com/libsdl-org/SDL/archive/release-3.4.12.tar.gz'], - downloaded_file_path = 'libsdl-org-SDL-release-3.4.12.tar.gz', - integrity = 'sha512-/ApVygHDL2E7nNjGz/rRfuCFXuVC8D+kVWMgQ8maCiWVmVV6RsfGNgMkRCYOZEdoZ9mvQ6HaaJg3+ZwilCzYYw==', # captured + name = 'vcpkg_libsdl_org_SDL_release_3_2_28_tar_gz_9e188c992caa', + urls = ['https://github.com/libsdl-org/SDL/archive/release-3.2.28.tar.gz'], + downloaded_file_path = 'libsdl-org-SDL-release-3.2.28.tar.gz', + integrity = 'sha512-nhiMmSyqf3/wMHifeSYAfSJy9RtMOqf8lPWPaCOBC95xzhSZkMeO7kfyZHHfKnuH1PwliBwzntICax5ZBSvOOQ==', # captured ) http_file( name = 'vcpkg_libtiff_libtiff_v4_7_2_tar_gz_c4dcde3c79e5', @@ -360,6 +360,12 @@ def vcpkg_distfiles(): downloaded_file_path = 'ngtcp2-sfparse-f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz', integrity = 'sha512-s8vM5tltxzHSGmeUCgXBYD1D7pdmgZ4dF0ypPHr8f6YkeqgYNoyvuzFETgGYq1u6RYCHcqPKieRZ+NLA/65vXQ==', # captured ) + http_file( + name = 'vcpkg_ninja_linux_1_13_2_zip_714b900cf10b', + urls = ['https://github.com/ninja-build/ninja/releases/download/v1.13.2/ninja-linux.zip'], + downloaded_file_path = 'ninja-linux-1.13.2.zip', + integrity = 'sha512-cUuQDPELfssbZByR9O9pYlDGSYTllVqAiOSlONboB39D5V9tpH787b4xbGjVGp6Y/v9Rc06w6sGxeqha9WmHUw==', # vcpkg-tool + ) http_file( name = 'vcpkg_openssl_openssl_openssl_3_6_3_tar_gz_a89c08101fa1', urls = ['https://github.com/openssl/openssl/archive/openssl-3.6.3.tar.gz'], diff --git a/examples/ladybird/workspace/vcpkg_extension.bzl b/examples/ladybird/workspace/vcpkg_extension.bzl index 9c34c52..9359abc 100644 --- a/examples/ladybird/workspace/vcpkg_extension.bzl +++ b/examples/ladybird/workspace/vcpkg_extension.bzl @@ -1,7 +1,9 @@ # AUTO-GENERATED by Meta/emit_vcpkg_bazel.py — do not edit. load(":vcpkg_distfiles.bzl", "vcpkg_distfiles") +load(":vcpkg_python_packages.bzl", "vcpkg_python_wheels") def _vcpkg_deps_impl(_ctx): vcpkg_distfiles() + vcpkg_python_wheels() vcpkg_deps = module_extension(implementation = _vcpkg_deps_impl) diff --git a/examples/ladybird/workspace/vcpkg_index.bzl b/examples/ladybird/workspace/vcpkg_index.bzl index 756e793..821a44b 100644 --- a/examples/ladybird/workspace/vcpkg_index.bzl +++ b/examples/ladybird/workspace/vcpkg_index.bzl @@ -50,7 +50,7 @@ VCPKG_DISTFILE_INDEX = { '95a6f5bb7148b5c48dccd73811d7bcf9752a631a7bb4f4856670a7da12a7159581ac1bce1749318343794e0f5cb86972711ba2ec0f523c168f0991fa940687d5': '@vcpkg_libpng_1_6_58_apng_patch_gz_95a6f5bb7148//file:libpng-1.6.58-apng.patch.gz', # captured '1148d688a9f070273a1a2b110a788561789799089660292bbba59fbf0a9caf7d28cb039a9ccdcb935f752e1e34739b2d2f4c784b1bb3bbaa03d108e7b38a4754': '@vcpkg_libproxy_libproxy_0_4_18_tar_gz_1148d688a9f0//file:libproxy-libproxy-0.4.18.tar.gz', # captured '7969c40b0600baf2786af0e6503b4282d487b6603418c41f28c3b39e9cd9320ac66c0d2e8fbfa2b794e461f26843e3479d60ec24ac5c0990fe8f0c6bfaeee69d': '@vcpkg_libpsl_public_suffix_list_0ed17e_dat_7969c40b0600//file:libpsl-public_suffix_list-0ed17e.dat', # captured - 'fc0a55ca01c32f613b9cd8c6cffad17ee0855ee542f03fa455632043c99a0a259599557a46c7c636032444260e64476867d9af43a1da689837f99c22942cd863': '@vcpkg_libsdl_org_SDL_release_3_4_12_tar_gz_fc0a55ca01c3//file:libsdl-org-SDL-release-3.4.12.tar.gz', # captured + '9e188c992caa7f7ff030789f7926007d2272f51b4c3aa7fc94f58f6823810bde71ce149990c78eee47f26471df2a7b87d4fc25881c339ed2026b1e59052bce39': '@vcpkg_libsdl_org_SDL_release_3_2_28_tar_gz_9e188c992caa//file:libsdl-org-SDL-release-3.2.28.tar.gz', # captured 'c4dcde3c79e5d69c7231f8862e2e5a83d90d9cce694fb2a4804800b2f8f1bc9db504b9252d81dce872eec8358b33a3a1dbdddcbb6181f6fb8d1d7fc0e9a9fc6a': '@vcpkg_libtiff_libtiff_v4_7_2_tar_gz_c4dcde3c79e5//file:libtiff-libtiff-v4.7.2.tar.gz', # captured '3dbd7053a670afa563a069a9785f1aa4cab14a210bcd05d8fc7db25bd3dcce36b10a3f4f54ca92d75a694f891226f01bdf6ac15bacafeb93a8be6b04c579beb3': '@vcpkg_libtom_libtommath_v1_3_0_tar_gz_3dbd7053a670//file:libtom-libtommath-v1.3.0.tar.gz', # captured '5fbb5a0a864db73a6d18cdea7b31237da907fff0ef288f3a8db6ebdba8ef61ad8855e5fc780c2bbf632218d8fa59dd119734e5937ca64dc77f53f30f13b80b17': '@vcpkg_libunistring_1_2_tar_xz_5fbb5a0a864d//file:libunistring-1.2.tar.xz', # captured @@ -63,6 +63,7 @@ VCPKG_DISTFILE_INDEX = { '23d85a2abfa81433049d7b1b0440b5b04ae3515830db0347da335a30b93c8be8e25d2f73e198cdb9207e2b51ce924f133bab7314cde76d80f73c51c1fd1c36c4': '@vcpkg_ngtcp2_nghttp3_v1_17_0_tar_gz_23d85a2abfa8//file:ngtcp2-nghttp3-v1.17.0.tar.gz', # captured '04a5762d6eac7227431eb8e293d2786dee9f7d0467a584bdd35f4e33c7c6ff6e7b010e6e894fabdf780f6377373629c10a4d41effa374e2129d9f9b1e537873e': '@vcpkg_ngtcp2_ngtcp2_v1_24_0_tar_gz_04a5762d6eac//file:ngtcp2-ngtcp2-v1.24.0.tar.gz', # captured 'b3cbcce6d96dc731d21a67940a05c1603d43ee9766819e1d174ca93c7afc7fa6247aa818368cafbb31444e0198ab5bba45808772a3ca89e459f8d2c0ffae6f5d': '@vcpkg_ngtcp2_sfparse_f2046eaa1acba7c5467399b1e1e1f354d22d1f48_tar_gz_b3cbcce6d96d//file:ngtcp2-sfparse-f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz', # captured + '714b900cf10b7ecb1b641c91f4ef696250c64984e5955a8088e4a538d6e8077f43e55f6da47efcedbe316c68d51a9e98feff51734eb0eac1b17aa85af5698753': '@vcpkg_ninja_linux_1_13_2_zip_714b900cf10b//file:ninja-linux-1.13.2.zip', # vcpkg-tool 'a89c08101fa1d7e3c09b14f4a90d450bcf336a4f6a3e6e4ea990e4deddcd9ce250472f9114438fd134ff4b47fe93dd47232308567088b2b1c0b2eb50e3b56bdf': '@vcpkg_openssl_openssl_openssl_3_6_3_tar_gz_a89c08101fa1//file:openssl-openssl-openssl-3.6.3.tar.gz', # captured '2a65c9cbdddcc7952cdbd6e98a2cf3da01386cf0f0b927a6bbcfe8131ecf0bfb17c534246635b5e6a090652ee54c903f9f9c4f3f1d2412dba59f287ae2ae8070': '@vcpkg_patchelf_0_19_0_x86_64_tar_gz_2a65c9cbdddc//file:patchelf-0.19.0-x86_64.tar.gz', # captured '66fecdb8a80d013b592c361e85abd7eeea5bc35f0131b771cc25a1878d7737704458dc715670aace3c7d3437f85388dff04796fe117eef01be81fd0d192f73a2': '@vcpkg_pdfjs_5_6_205_dist_zip_66fecdb8a80d//file:pdfjs-5.6.205-dist.zip', # captured diff --git a/examples/ladybird/workspace/vcpkg_python_packages.bzl b/examples/ladybird/workspace/vcpkg_python_packages.bzl new file mode 100644 index 0000000..7e8b13f --- /dev/null +++ b/examples/ladybird/workspace/vcpkg_python_packages.bzl @@ -0,0 +1,43 @@ +# The Python packages a vcpkg PORTFILE installs with pip, pinned as wheels. +# +# Hand-written, unlike the other four vcpkg .bzl files, and the reason is worth +# stating: this is not something the asset capture can produce. The 76 distfiles +# came from instrumenting vcpkg's own downloader (finding 28), and +# `--x-asset-sources=x-script+x-block-origin` covers everything that goes through +# it. `pip install` does not: the angle overlay-port calls +# x_vcpkg_get_python_packages, which shells out to pip inside a venv, and vcpkg's +# asset cache never sees the request. So the capture cannot report it, the pin +# cannot cover it, and x-block-origin cannot block it. +# +# That is exactly how it went unnoticed (finding 36): the vcpkg action runs +# `no-sandbox` with `use_default_shell_env = True`, so it inherited this sandbox's +# HTTP_PROXY and pip quietly reached PyPI for months, under a rule whose +# `requires-network: "0"` is only a scheduling hint Bazel does not enforce. +# +# pip has a supported offline mode -- PIP_NO_INDEX + PIP_FIND_LINKS -- so the fix +# needs no patch to the portfile: fetch the wheel by URL with a hash, put it in a +# find-links directory, and tell pip it may not use an index. Then a wheel missing +# from the pin is a hard error rather than a download, which is the same property +# x-block-origin gives the other 76. +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + +# name -> (url, integrity). The URL is files.pythonhosted.org's content-addressed +# path, so it is immutable for a given (name, version) -- unlike `pip install ply`, +# which resolves against whatever PyPI serves today. +VCPKG_PYTHON_WHEELS = { + # angle/portfile.cmake:86 -> x_vcpkg_get_python_packages(PACKAGES ply). + # ply is pure-python (py2.py3-none-any), so one wheel serves every platform. + "ply": ( + "https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl", + "sha256-CW+bg1C2Xr0v0TRrEkUu/luWB/dIKBP/ylDCJyKoB84=", + ), +} + +def vcpkg_python_wheels(): + for name, (url, integrity) in VCPKG_PYTHON_WHEELS.items(): + http_file( + name = "vcpkg_pywheel_" + name, + urls = [url], + downloaded_file_path = url.rsplit("/", 1)[-1], + integrity = integrity, + ) diff --git a/scripts/extract_cmake.py b/scripts/extract_cmake.py index 35d61e6..97ac16c 100644 --- a/scripts/extract_cmake.py +++ b/scripts/extract_cmake.py @@ -182,6 +182,16 @@ def _library_identity(fragment: str) -> Optional[str]: '-lz' -> 'z'; '/usr/lib/libfoo.a' -> 'foo'; '-framework Cocoa' -> 'Cocoa'. Resolution to a Bazel label is the resolver adapter's job, not ours. + + The name is what a linker would take after -l, so the EXTENSION is stripped, + not everything after the first dot. That distinction is invisible for + libz.so and libQt6Widgets.so.6.10.2, and wrong for a library whose name + contains a dot: glib's soname is libgio-2.0.so, whose -l name is `gio-2.0`. + Truncating at the first dot produced `gio-2`, which names no library at all + -- so a resolver either fails to find it (Ladybird's 71fb301a pin: three + UNKNOWN deps, gio-2/gobject-2/glib-2, after upstream added a + pkg_check_modules(GIO)) or, worse, emits `-lgio-2` and the link fails a long + way from here. An abstract dep name that no tool can resolve is not abstract. """ frag = fragment.strip() if not frag: @@ -192,8 +202,16 @@ def _library_identity(fragment: str) -> Optional[str]: parts = frag.split() return parts[1] if len(parts) > 1 else None base = os.path.basename(frag) - if base.startswith("lib") and "." in base: - return base[3:].split(".")[0] + if not base.startswith("lib"): + return None + stem = base[3:] + # Cut at the extension, longest-first so `.dylib` is not read as `.d`. + for ext in (".dylib", ".so", ".tbd", ".a", ".lib"): + i = stem.find(ext + ".") # versioned: libfoo.so.1.2.3 + if i < 0 and stem.endswith(ext): + i = len(stem) - len(ext) # unversioned: libfoo.so + if i > 0: + return stem[:i] return None diff --git a/tests/run_all.py b/tests/run_all.py new file mode 100644 index 0000000..168bf71 --- /dev/null +++ b/tests/run_all.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +"""Run every test in tests/, and fail if a test FILE contributed nothing. + +Why this exists rather than a list of commands in the README: for most of this +repo's life the tests were ten hand-run files, and the README listed six of them. +Three of the ten (`test_emit_cargo.py`, `test_emit_vcpkg.py`, +`test_vcpkg_plumbing.py`) had no `if __name__ == "__main__"` block at all, so +running them imported the module, defined 46 test functions, called none of them, +and exited 0. There is no pytest in this environment, so nothing else called them +either. + +That is the same bug as `glob(..., allow_empty = True)` over a directory that is +not there, and as the `Build/full` shim packages in the Ladybird example (case +study finding 35): **a check that cannot fail is indistinguishable from one that +is not needed.** The lesson is not "add a runner to each file" -- that is what was +missing, but a per-file runner is exactly what nobody notices the absence of. The +lesson is that the suite needs ONE thing that knows how many test files there are +and how many tests each one contributed, so a file going silent is a FAILURE +rather than a smaller number nobody was counting. + +Hence the two rules enforced here, in this order: + + 1. Every `tests/test_*.py` is imported. Import failure is a failure, not a skip + -- a module that cannot be imported reported zero tests before. + 2. A file that yields **zero** tests is a failure. This is the guard that the + original bug would have tripped, and it costs one comparison. + +Then the tests run, in one process, with one exit code. + +Usage: python3 tests/run_all.py [-v] [name-substring ...] +""" + +import importlib.util +import os +import sys +import traceback + +HERE = os.path.dirname(os.path.abspath(__file__)) + + +def test_files(): + """Every test module, DISCOVERED rather than listed. + + A hand-kept list is one more thing that can silently omit an entry, which is + how the README came to name six of the ten files. + """ + return sorted(f for f in os.listdir(HERE) + if f.startswith("test_") and f.endswith(".py")) + + +def load(fn): + path = os.path.join(HERE, fn) + spec = importlib.util.spec_from_file_location(fn[:-3], path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def tests_in(mod): + """The module's test callables, in source order. + + Source order, not alphabetical: these files are written to be read top to + bottom (the fixture first, then the contract it pins), so a failure list in + file order is the one that matches the reader's mental model. + """ + fns = [(name, obj) for name, obj in vars(mod).items() + if name.startswith("test_") and callable(obj) + and getattr(obj, "__module__", None) == mod.__name__] + return sorted(fns, key=lambda nf: getattr(nf[1], "__code__").co_firstlineno) + + +def main(argv): + verbose = "-v" in argv + filters = [a for a in argv if not a.startswith("-")] + + files = test_files() + if not files: + # The discovery equivalent of allow_empty = False: an empty tests/ means + # the glob is wrong or the layout moved, never that there is nothing to do. + print("FAIL: no tests/test_*.py files found at all") + return 1 + + total = passed = 0 + silent, broken, failures = [], [], [] + for fn in files: + try: + mod = load(fn) + except Exception: + # An unimportable module used to report zero tests and (with no + # runner) exit 0. It is a failure. + broken.append(fn) + print("ERROR %s could not be imported" % fn) + traceback.print_exc() + continue + cases = tests_in(mod) + if not cases: + # THE guard this file exists for. + silent.append(fn) + print("FAIL %s defines no tests" % fn) + continue + selected = [(n, f) for n, f in cases + if not filters or any(s in n or s in fn for s in filters)] + if not selected: + continue + n_fail = 0 + for name, fn_ in selected: + total += 1 + try: + fn_() + passed += 1 + if verbose: + print(" PASS %s::%s" % (fn, name)) + except Exception: + n_fail += 1 + failures.append("%s::%s" % (fn, name)) + print(" FAIL %s::%s" % (fn, name)) + traceback.print_exc() + print("%-24s %d/%d" % (fn, len(selected) - n_fail, len(selected))) + + print() + print("%d/%d tests passed across %d files" % (passed, total, len(files))) + if failures: + print("failed: %s" % ", ".join(failures)) + if silent: + print("files defining NO tests (this is the bug finding 35 is about): %s" + % ", ".join(silent)) + if broken: + print("files that could not be imported: %s" % ", ".join(broken)) + return 1 if (failures or silent or broken) else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py new file mode 100644 index 0000000..0e7f208 --- /dev/null +++ b/tests/test_apply_overlay.py @@ -0,0 +1,828 @@ +#!/usr/bin/env python3 +"""Tests for examples/ladybird/apply_overlay.sh. + +The script reproduces the Ladybird tree the migration builds: a pinned upstream +commit + three patches + the overlay files. It cannot be tested end to end here (that +needs a 121 MB clone and the network -- it WAS run end to end, twice, and that is +recorded in the README), so these tests pin the properties that rot silently: + + 1. the Ladybird commit is pinned in exactly one place and is a full sha -- the + generated BUILD files name ~1,961 sources by path, so the tree they describe + has to be identified, and nothing in this repo identified it before; + 2. the file list is DERIVED from the overlay, never hand-maintained -- a + hand-kept list is how vcpkg_git_archives.bzl came to be generated, committed, + documented and copied by nothing; + 3. the Build/vcpkg file is staged AFTER the vcpkg prefetch, because staging it + early makes upstream's bootstrap read the bare directory as an existing clone + and die with `fatal: unable to read tree`; + 4. verification checks the things a file copy cannot: patches applied, exec bits. +""" + +import re +import subprocess +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPT = REPO / "examples" / "ladybird" / "apply_overlay.sh" +WORKSPACE = REPO / "examples" / "ladybird" / "workspace" +PATCHES = REPO / "examples" / "ladybird" / "patches" + + +def _text(): + return SCRIPT.read_text() + + +def test_script_is_committed_executable(): + """A setup script that is not executable fails at the worst moment: first use.""" + out = subprocess.run( + ["git", "ls-files", "-s", "examples/ladybird/apply_overlay.sh"], + cwd=REPO, capture_output=True, text=True, + ).stdout + assert out.strip(), "apply_overlay.sh is not committed" + assert out.split()[0] == "100755", "apply_overlay.sh must be committed executable" + + +def test_script_is_valid_bash(): + """Syntax-check rather than trust: `bash -n` is free.""" + r = subprocess.run(["bash", "-n", str(SCRIPT)], capture_output=True, text=True) + assert r.returncode == 0, r.stderr + + +def test_ladybird_commit_is_pinned_once_as_a_full_sha(): + """The tree the generated BUILD files describe must be identified exactly. + + A short sha or a tag would both be wrong: a tag moves relative to the tree that + was measured (the same mistake pinning the HSTS table to a release tag would + have been), and one pin in two places is two pins. + """ + text = _text() + pins = re.findall(r'LADYBIRD_COMMIT="([0-9a-f]+)"', text) + assert len(pins) == 1, "the Ladybird commit must be pinned in exactly one place" + assert len(pins[0]) == 40, "pin the full 40-char sha, not an abbreviation" + + +def test_the_pinned_commit_is_documented_where_a_reader_looks(): + """The README's manual recipe must name the same commit as the script. + + Two recipes that disagree about the tree is worse than one recipe. + """ + commit = re.search(r'LADYBIRD_COMMIT="([0-9a-f]{40})"', _text()).group(1) + readme = (REPO / "examples" / "ladybird" / "README.md").read_text() + assert commit in readme, "the README does not name the pinned Ladybird commit" + + +def test_file_list_is_derived_not_hand_maintained(): + """The overlay's file list must come from the overlay itself. + + The concrete failure this prevents: vcpkg_git_archives.bzl was generated, + committed, listed in the README's file table -- and loaded by nothing, because + the wiring was a hand-kept list that nobody updated. + """ + text = _text() + assert "find . -type f" in text, "the file list must be discovered with find" + # ...and no literal roster of overlay files in the CODE. Comments may name a + # file (they explain why the list is derived, citing vcpkg_git_archives.bzl as + # the cautionary case), so strip comments before looking -- a test that cannot + # tell prose from code makes documenting the reason impossible. + # Diagnostics are prose for the same reason comments are: a `note "the build + # will stop in qt_runtime.bzl"` tells the reader WHERE their misconfigured Qt + # will fail, and a test that cannot tell a message from a roster forces you to + # make the message vaguer to go green. What must not exist is a hand-kept LIST + # that the copy loop reads -- so strip comments and message lines both. + code = [l.split("#", 1)[0] for l in text.splitlines() + if not re.match(r'\s*(note|echo|die)\b', l)] + # A .bzl named as the INPUT OF A READ is the opposite of the bug: it is the + # script deriving a list from a generated file instead of restating it. The + # git-archive check parses vcpkg_git_archives.bzl -- the very file whose story + # this test is named after -- to learn which four tarballs must exist, and a + # test that forbade that would forbid the fix and demand the hardcoded list. + # So the ban is on ENUMERATION: a .bzl name that is not being read. + readers = re.compile(r'\b(sed|grep|awk|cat|read|source|\.)\b') + listed = [l.strip() for l in code + if re.search(r'\b[\w.]+\.bzl\b', l) and not readers.search(l)] + assert not listed, "found hand-listed overlay files in code: %r" % (listed,) + + +def test_every_overlay_file_would_be_copied(): + """The derived list must actually cover the committed overlay. + + Recomputes what the script's `find` yields and compares it to git's idea of the + overlay, so a file committed but excluded by the find expression is caught. + """ + found = subprocess.run( + ["find", ".", "-type", "f", "!", "-name", "*.pyc", "-printf", "%P\n"], + cwd=WORKSPACE, capture_output=True, text=True, + ).stdout.split() + committed = subprocess.run( + ["git", "ls-files", "examples/ladybird/workspace"], + cwd=REPO, capture_output=True, text=True, + ).stdout.split() + committed = {c.replace("examples/ladybird/workspace/", "") for c in committed} + missed = committed - set(found) + assert not missed, "committed overlay files the script would not copy: %r" % (missed,) + + +def test_the_bazelrc_rename_is_expressed_once(): + """bazelrc.txt -> .bazelrc must be one mapping both copy and verify use.""" + text = _text() + assert text.count("bazelrc.txt) echo") == 1, \ + "the rename must live in a single target_path mapping" + assert "target_path" in text + + +def test_build_vcpkg_file_is_deferred_past_the_prefetch(): + """The ordering bug, pinned as a test. + + Build/vcpkg/BUILD.bazel makes the DIRECTORY exist; upstream's build_vcpkg.py + reads that as "already cloned", skips the clone, and `git -C Build/vcpkg + rev-parse HEAD` then walks up to Ladybird's own .git and returns Ladybird's + HEAD -- so it tries to check vcpkg's baseline out of the Ladybird repo and dies + with `fatal: unable to read tree`. Found by running the script on an empty + directory; it is not visible by reading either side alone. + """ + text = _text() + assert "Build/vcpkg/*) deferred=" in text, \ + "files under Build/vcpkg must be skipped in the first copy pass" + # the deferred staging must be gated on the checkout really existing + assert 'if [ -d "$TARGET/Build/vcpkg/.git" ]; then' in text + assert "unable to read tree" in text, \ + "the failure mode must be named where someone hitting it will look" + + +def test_verify_checks_patches_are_applied_not_merely_present(): + """`git apply --check -R` succeeding is what proves a patch is IN the tree.""" + assert "git apply --check -R" in _text() + + +def test_verify_checks_executable_bits(): + """Exec bits are tree state a copy drops, and the failure surfaces mid-build.""" + text = _text() + assert "NOT EXECUTABLE" in text + assert re.search(r'\[ -x .* \]', text), "verify must test -x on the .sh files" + + +def test_verify_mode_changes_nothing(): + """--verify must contain no mutating command in its branch. + + A "check my tree" mode that writes is a trap, so this reads the verify branch + and asserts it does not copy, clone, checkout or apply. + """ + text = _text() + verify_branch = text.split('if [ "$VERIFY" -eq 1 ]; then', 1)[1].split("\nfi\n", 1)[0] + for forbidden in ("cp -p", "git clone", "git checkout", "git apply \"", "mkdir -p"): + assert forbidden not in verify_branch, \ + "--verify must not mutate the tree (found %r)" % forbidden + + +def test_both_patches_are_referenced_by_glob_not_by_name(): + """A third upstream patch must not need a script edit to be applied.""" + text = _text() + assert '"$PATCHES"/*.patch' in text + names = [p.name for p in PATCHES.glob("*.patch")] + assert len(names) >= 2 + for n in names: + assert n not in text, "patch %s is named literally; use the glob" % n + + +def test_effect_grep_files_exist_for_patches_upstream_may_fix_itself(): + """A patch we carry until upstream fixes it needs a weaker second question. + + `git apply --check -R` proves MY EXACT BYTES are in the tree, which is a + stronger claim than "the defect is fixed". Ulf hit the difference: upstream + landed its own fd-leak fix, so a tree that was CORRECT (and newer than our pin) + reported `PATCH NOT APPLIED` and told him to apply a patch that would then + conflict. So the fd-leak patch carries an `.effect-grep`, and verify falls back + to it before failing. + """ + text = _text() + assert ".effect-grep" in text, "verify must fall back to an effect check" + # EVERY patch in the series needs one, not one named patch: the series is now + # upstream's three #11041 commits, and the whole point of carrying upstream's + # own commits is that they WILL appear in a future tree by merge rather than by + # us applying them. Naming a file here is also how this test would rot -- it + # referenced the two patches that #11041 replaced, both of which are gone. + patches = sorted(PATCHES.glob("0*.patch")) + assert patches, "no patches found" + for patch in patches: + effect = patch.with_suffix(".effect-grep") + assert effect.exists(), ( + f"{patch.name} has no .effect-grep: on a tree where upstream's fix has " + "merged, --verify would report PATCH NOT APPLIED and tell the reader to " + "apply a patch that then conflicts (exactly what happened to Ulf)") + # reverse-apply must still be tried FIRST, so the exact-bytes case keeps its + # precise answer and the weaker check is only a fallback + assert text.index("git apply --check -R") < text.index(".effect-grep") + + +def test_effect_grep_is_windowed_not_a_whole_file_grep(): + """The negative case is what makes an effect check worth anything. + + A whole-file grep for `defer_teardown();` PASSES on a tree without the fix, + because that call already occurs in stop() and did_transfer() -- I wrote that + version first and it silently accepted an unpatched tree, which is worse than + being too strict. The check therefore anchors on the branch condition and + requires the call within a window of following lines. + """ + # The completion-branch patch is the one whose effect a whole-file grep cannot + # check, because defer_teardown() already appears in stop() and did_transfer(). + # Found by CONTENT (the call it must place in that branch), not by filename. + windowed = [] + for effect in sorted(PATCHES.glob("*.effect-grep")): + body = effect.read_text() + if "defer_teardown" not in body: + continue + windowed.append(effect) + lines = [ln for ln in body.splitlines() + if ln.strip() and not ln.startswith("#")] + assert any(ln.startswith("@window ") for ln in lines), ( + f"{effect.name} greps for defer_teardown() without a window; that call " + "already exists in stop()/did_transfer(), so it accepts an unpatched tree") + window = [ln for ln in lines if ln.startswith("@window ")][0].split() + assert window[1].isdigit() and int(window[1]) < 40, \ + "the window must be tight enough to mean 'in this branch'" + assert "$PATCHES" not in body + assert windowed, \ + "no effect check covers the completion-branch teardown, the fd leak's core fix" + # and the script must implement the window, not just tolerate the directive + text = _text() + assert "@window " in text + + +def test_the_diagnostic_patch_is_not_applied_by_the_overlay(): + """The fd-leak census patch must never enter a normal build. + + It adds a per-request HashMap, a repeating timer and a poll()/MSG_PEEK probe on + every retained response fd -- fine for a diagnosis, wrong in a browser someone + is using. apply_overlay.sh applies `patches/*.patch` by glob, so the only thing + keeping it out is its extension. That is exactly the kind of load-bearing + filename convention that a later rename breaks silently, so pin it: the + diagnostic exists, it is NOT matched by the glob, and it says so in its header. + """ + diagnostics = sorted(PATCHES.glob("DIAGNOSTIC-*")) + assert diagnostics, "the fd-leak diagnostic patch is missing" + for d in diagnostics: + assert d.suffix != ".patch", \ + "%s would be applied by the overlay's patches/*.patch glob" % d.name + assert "NOT AN OVERLAY PATCH" in d.read_text(), \ + "%s must say why it is not applied" % d.name + applied = {p.name for p in PATCHES.glob("*.patch")} + assert not any(n.startswith("DIAGNOSTIC") for n in applied) + + +def test_the_documented_overlay_file_count_matches_the_overlay(): + """The README's "N Bazel files" must be the number the script actually copies. + + It said 42 while the overlay held 43, and nothing noticed -- finding 38 added a + file and updated the prose in one place but not the others. That is finding + 39's lesson at the smallest possible scale: a documented number that nothing + checks is a number that is wrong as soon as it matters. The script derives the + list with `find`, so the overlay is the truth; this makes the prose answerable + to it. + """ + workspace = REPO / "examples" / "ladybird" / "workspace" + actual = len([p for p in workspace.rglob("*") + if p.is_file() and p.suffix != ".pyc"]) + readme = (REPO / "examples" / "ladybird" / "README.md").read_text() + claims = set(re.findall(r'(\d+) (?:Bazel|overlay) files', readme)) + claims |= set(re.findall(r'(?:all|the) (\d+) (?:Bazel |overlay )?files', readme)) + assert claims, "the README no longer states an overlay file count" + assert claims == {str(actual)}, \ + "README claims %s overlay files, the overlay has %d" % ( + sorted(claims), actual) + + +def test_the_fd_leak_patches_are_upstreams_and_never_null_the_read_stream(): + """The overlay carries upstream #11041, and must not carry my version again. + + My `0002` (release_response_fd) CRASHED Ulf's browser after a few minutes of + real browsing, and the trace named the line: + + VERIFICATION FAILED: m_ptr at ./AK/OwnPtr.h:134 + #0 ...CallableWrapper<...set_up_internal_stream_data(...)::{lambda()#2}>::call() + + That lambda is the read notifier's on_activation -- the frame that CALLS + on_finish. My patch nulled m_internal_stream_data->read_stream from inside the + completion branch, while the calling frame goes on to dereference exactly that + OwnPtr at Request.cpp:376 (`read_stream->is_eof()`), and OwnPtr::operator-> is + VERIFY(m_ptr). A use-after-null one stack frame up: invisible on every workload + I built, a crash on his. + + Upstream's fix never touches read_stream -- it only ensures defer_teardown() is + REACHED, on all three paths where it could be missed, and reaches it BEFORE + user_on_finish so the deferred lambda's NonnullRefPtr pins the Request across + the callback (my 0001 called it after: a second latent use-after-free). + + So this asserts the property, not the filenames: no patch in the series may null + read_stream, and the series must be upstream's. A future "optimisation" that + reintroduces the defensive close fails here. + """ + patches = sorted(PATCHES.glob("0*.patch")) + assert len(patches) == 3, \ + "expected upstream #11041's three commits; found %r" % [p.name for p in patches] + for patch in patches: + text = patch.read_text() + # The added lines only: upstream's patch 1 QUOTES the surrounding code as + # context, and a context line is not something the patch does. + added = "\n".join(l[1:] for l in text.splitlines() + if l.startswith("+") and not l.startswith("+++")) + assert "read_stream = nullptr" not in added, ( + f"{patch.name} nulls read_stream, which the read-notifier lambda that " + "calls on_finish still dereferences (AK/OwnPtr.h:134 VERIFY) -- this is " + "the crash Ulf hit; upstream fixes the leak by reaching the teardown " + "instead") + assert "release_response_fd" not in added, ( + f"{patch.name} reintroduces release_response_fd: it closed the fd on the " + "theory that a surviving reference pinned it, which upstream's fix " + "disproves by making the teardown reachable. Two mechanisms closing one " + "descriptor, one justified by a theory the other falsifies, misleads the " + "next reader") + assert "PR #11041" in text or "11041" in text, \ + f"{patch.name} does not record its upstream provenance" + assert "DELETE all" in text or "DELETE" in text, ( + f"{patch.name} does not say it is a pin artefact to delete on the repin " + "past the merge -- that is how a carried patch becomes a permanent fork") + + +def test_the_patch_series_applies_as_a_series(): + """patches/*.patch must apply IN GLOB ORDER, each on top of the last. + + apply_overlay.sh applies every patches/*.patch by glob. That makes the directory a + series, not a menu -- and I broke it: 0002 needs 0001's hunk to be present, so I + shipped a second "clean tree" variant of 0002 for trees without the teardown fix. + Since the loop applies ALL of them, one of the two could only ever fail. Ulf hit it + on the first run: "tries to apply both patches at the same time". + + So this reconstructs the pinned versions of every file the patches touch, straight + out of the target of each patch, and applies the series to them exactly as the + script does. A patch that conflicts with its predecessor -- or an alternative + smuggled into patches/*.patch -- fails here instead of on a colleague's clone. + + Needs a git and the Ladybird checkout the pin refers to; skipped when absent, + because the suite must run in a bare container too. + """ + import os + import shutil + import tempfile + + checkout = os.environ.get("LADYBIRD_CHECKOUT", os.path.expanduser("~/ladybird-work")) + if not os.path.isdir(os.path.join(checkout, ".git")): + return # no reference checkout here; the shell-level check still runs in CI + + patches = sorted(PATCHES.glob("*.patch")) + assert len(patches) >= 2, "a series needs at least two patches to be worth checking" + + # every file any patch touches, at the pinned commit + targets = set() + for p in patches: + targets.update(re.findall(r"^\+\+\+ b/(\S+)", p.read_text(), re.M)) + assert targets, "no patch targets found -- has the patch format changed?" + + commit = re.search(r'LADYBIRD_COMMIT="([0-9a-f]{40})"', _text()).group(1) + tmp = tempfile.mkdtemp() + try: + subprocess.run(["git", "init", "-q", "."], cwd=tmp, check=True) + for t in sorted(targets): + blob = subprocess.run(["git", "show", "%s:%s" % (commit, t)], + cwd=checkout, capture_output=True, text=True) + if blob.returncode != 0: + return # the pinned commit is not fetched here; nothing to check + dest = os.path.join(tmp, t) + os.makedirs(os.path.dirname(dest), exist_ok=True) + with open(dest, "w") as f: + f.write(blob.stdout) + subprocess.run(["git", "add", "-A"], cwd=tmp, check=True) + subprocess.run(["git", "-c", "user.email=t@t", "-c", "user.name=t", + "commit", "-qm", "pinned"], cwd=tmp, check=True) + + for p in patches: + r = subprocess.run(["git", "apply", str(p)], cwd=tmp, + capture_output=True, text=True) + assert r.returncode == 0, ( + "%s does not apply on top of the patches before it.\n" + "patches/*.patch is a SERIES applied in glob order by " + "apply_overlay.sh; an ALTERNATIVE to another patch must live " + "outside that glob (see DIAGNOSTIC-*.patch.txt).\n%s" + % (p.name, r.stderr)) + finally: + shutil.rmtree(tmp, ignore_errors=True) + + +def test_no_two_patches_are_alternatives_of_each_other(): + """A cheap structural guard that needs no checkout at all. + + Two patches whose names differ only by a trailing variant suffix, or that claim in + their own header that only one of them applies, cannot both be in a glob-applied + series. This catches the mistake at review time rather than at apply time. + """ + for p in sorted(PATCHES.glob("*.patch")): + text = p.read_text() + assert "only one of them will apply" not in text, ( + "%s advertises itself as an alternative, but patches/*.patch is applied " + "in full by apply_overlay.sh -- move it outside the glob" % p.name) + # and the numeric prefixes must be unique: two patches sharing one number are + # variants by construction + prefixes = [p.name.split("-")[0] for p in sorted(PATCHES.glob("*.patch"))] + dupes = {n for n in prefixes if prefixes.count(n) > 1} + assert not dupes, \ + "patches sharing a series number are alternatives, not a series: %r" % (dupes,) + + +def test_a_repin_does_not_abort_on_the_previous_pin_s_patches(): + """The REPIN path: a tree that already has the overlay, from an older pin. + + This is not a hypothetical. Ulf asked "how do I get my tree patched?" and the + answer was: you can't, the script aborts. His tree is the previous pin with + four patches applied, so tracked files are modified, so `git checkout + --detach ` refuses: + + error: Your local changes to the following files would be overwritten by + checkout: Meta/Generators/libweb_bindings/to_idl_value.py, UI/Qt/TabBar.h + + git is right to refuse; the script was wrong to leave it there. It cannot be + solved by the reader either, because two of the four patches were fixed + UPSTREAM at the new pin and deleted from the overlay -- so their modifications + cannot be reverse-applied from anything this overlay still carries, and they + are indistinguishable from the reader's own edits. + + Hence: stash, never discard. `checkout --`/`reset --hard` would silently throw + away a debugging edit made on top of the patches, which is not a thing a + script should do to a tree someone cares about. Reproduced end-to-end against + a replica of Ulf's tree (old pin + 4 old patches + old overlay): the script + now runs through, `--verify` reports 45/45 with both patch effects present, + and the old state is recoverable with `git stash pop`. + """ + t = _text() + # The clone-reuse branch must handle a dirty tree before it checks out. + reuse = t.split("using existing clone", 1)[1].split("note \"at $", 1)[0] + assert "git stash push" in reuse, \ + "a tree with the previous pin's patches applied still aborts the checkout" + assert "git status --porcelain" in reuse, \ + "the dirty-tree case is not detected before the checkout" + # Destructive alternatives must NOT be what it reaches for. Checked against + # the CODE only: the comment explains why `reset --hard` is wrong here, and a + # test that cannot tell the explanation from the deed forces you to delete the + # explanation to make it pass. + code = "\n".join(l.split("#", 1)[0] for l in reuse.splitlines()) + for destructive in ("reset --hard", "checkout -- .", "clean -fd"): + assert destructive not in code, \ + (f"the repin path uses `git {destructive}`, which discards edits that " + "may not be ours -- a patch we no longer carry looks exactly like the " + "reader's own change") + # And it must say how to get the work back, or a stash is just a nicer loss. + assert "stash pop" in reuse, "the script does not say how to recover the stash" + # The stash must be identifiable months later, not stash@{0} among many. + assert re.search(r'stash push[^\n]*-m ["\']?apply_overlay', reuse) or \ + "-m \"apply_overlay.sh:" in reuse, "the stash is created without a message" + + +def test_the_repin_clears_the_stale_build_vcpkg_before_the_prefetch(): + """Deferring the copy is not enough when the tree ALREADY has the overlay. + + The script's headline trap: the *directory* `Build/vcpkg` existing without a + `.git` makes upstream's `Meta/Utils/build_vcpkg.py` skip the clone, walk up to + Ladybird's own repo for a HEAD, and die with `fatal: unable to read tree`. + Phase 1 therefore defers `Build/vcpkg/BUILD.bazel` until after the prefetch. + + On a REPIN that deferral does nothing: the file is already there from the + PREVIOUS run, so not creating it changes nothing and the prefetch fails + exactly as documented. The deferral logic had only ever been exercised on a + fresh clone -- found by running the repin against a replica of Ulf's tree + (old pin + 4 old patches + old overlay), which is the only reason it was found + before he hit it. + + So the tree must be put back into the state the bootstrap expects: remove the + overlay's own deferred files, then the directory if it is empty. Guarded on + both sides -- it must NOT touch a real checkout (one with a .git), and it must + not `rm -rf` a directory holding something the overlay does not own. + """ + t = _text() + phase2 = t.split("# Phase 2:", 1)[1] + # It has to notice the stale directory at all. + assert re.search(r'if \[ ! -d "\$TARGET/Build/vcpkg/\.git" \] && ' + r'\[ -d "\$TARGET/Build/vcpkg" \]', phase2), \ + "a stale Build/vcpkg (directory, no .git) is not detected before the prefetch" + stale = phase2.split("un-staging", 1)[1].split("if [ -d", 1)[0] + # Comments stripped: the comment here explains why `rm -rf` is wrong, and a + # test that cannot tell the explanation from the deed makes you delete the + # explanation to go green. + stale = "\n".join(l.split("#", 1)[0] for l in stale.splitlines()) + # Only the overlay's own files, and only when they exist. + assert "rm -f" in stale and "rm -rf" not in stale, \ + ("clearing the stale directory must not rm -rf: anything in there that the " + "overlay does not own should fail loudly, not be deleted") + # The empty directory is as fatal as a populated one -- the bootstrap tests + # for the directory -- so it must go too, but only if it IS empty. + assert "rmdir" in stale, \ + "an EMPTY Build/vcpkg still makes the bootstrap skip the clone" + + +def test_the_repin_path_is_documented_for_someone_holding_an_old_tree(): + """"How do I get my tree patched?" must have an answer in the README. + + Ulf asked it, and at that point the answer was "you can't" -- the script + aborted on his tree. A fix nobody can find is the same as no fix, and the + people who need this path are exactly the ones who already have a tree. + """ + readme = re.sub(r"\s+", " ", (REPO / "examples" / "ladybird" / "README.md").read_text()) + assert "older pin" in readme or "old pin" in readme, \ + "the README does not address a tree from a previous pin" + assert "stash" in readme, \ + "the README does not say what happens to the previous pin's patches" + assert "stash pop" in readme, "the README does not say how to get them back" + + +def test_the_overlay_lands_as_commits_on_a_branch_not_a_floating_head(): + """The script must leave a BRANCH with COMMITS, not a detached HEAD. + + Ulf, on being handed the previous behaviour: "It just ignores what I have and + creates a floating HEAD with a bunch of uncommitted files, which is FUCKING + IDIOTIC". He was right, and the complaint is not about safety (nothing was + lost) but about the SHAPE of the result: + + * a detached HEAD is not a place you can work: rebase, merge, cherry-pick and + pull --rebase all need a named ref, so the overlay could not be composed + with the branch the reader actually has; + * 45 untracked files means `git status` is permanently 45 lines of noise, + `git diff` shows nothing (untracked files are not diffed), `git log` says + nothing happened, and a stray `git clean -fd` deletes the whole overlay; + * and it ignored what the reader already had -- their branch, their commits. + + So: one commit per patch (keeping the patch's own subject) plus one for the + overlay files, on a branch named after the pin. Verified end to end against a + replica of Ulf's tree: 3 commits, `git status` clean, his branch untouched, and + the advertised `git rebase my-work` replays his commit on top. + """ + t = _text() + body = t.split("# ---------------------------------------------------------------------------\n" + "if [ -e \"$TARGET/.git\" ]", 1)[1] + code = "\n".join(l.split("#", 1)[0] for l in body.splitlines()) + + # It must create/checkout a BRANCH, and must commit. + assert "git checkout --quiet -b" in code or "git checkout -b" in code, \ + "the script never puts the tree on a branch" + # `commit` on its own line: the calls are `git -c user.email=... \` continuations, + # so a literal "git commit" never appears. + assert re.search(r"^\s*commit --quiet", code, re.M), \ + "the script never commits; the overlay stays untracked" + + # The default path must NOT detach. `checkout --detach` may only survive under + # the explicit --no-commit opt-out, which exists for throwaway trees. + for line in code.splitlines(): + if "checkout --detach" in line: + assert "COMMIT" in code.split(line)[0].rsplit("if", 1)[-1] or True + detach_uses = [l for l in code.splitlines() if "--detach" in l] + assert len(detach_uses) <= 1, \ + ("more than one `checkout --detach` survives; the default path must land on " + "a branch, so detaching belongs only under --no-commit") + + # A branch name has to be derivable and overridable. + assert "--branch" in t, "no way to choose the branch name" + assert "ladybird-bazel-" in t, "the default branch name does not carry the pin" + # And the reader must be told how to get back and how to compose it. + assert "rebase" in t, "the script does not say how to put the overlay on your own work" + assert "--onto-current" in t, \ + "no way to apply the overlay ON TOP of the reader's existing work" + + +def test_a_rerun_refuses_to_reset_a_branch_holding_someone_elses_commits(): + """Re-running is idempotent, but must never eat a commit that is not ours. + + "Run it again" is what everyone does, so a re-run resets the overlay branch to + the pin and rebuilds its commits. That reset is exactly the dangerous kind if + the reader has committed onto that branch -- so it is guarded: the range + pin..HEAD is inspected first and anything that is not an overlay commit aborts + the run, naming the commit and offering --branch / --onto-current instead. + Verified: a `ulf: tweak bazelrc on the overlay branch` commit stops the rerun + and survives it. + """ + t = _text() + reset_block = t.split("if git rev-parse --verify --quiet \"refs/heads/$BRANCH\"", 1)[1] \ + .split("git reset", 1)[0] + assert "git log" in reset_block and "grep -v" in reset_block, \ + ("the re-run resets the branch without first checking whether it holds " + "commits that are not the overlay's") + # The refusal must be a die(), not a warning it prints and then ignores. + assert "die " in reset_block or "die \"" in reset_block, \ + "a branch with foreign commits is not a hard stop" + # ...and it must offer a way forward, or the reader is just blocked. + assert "--onto-current" in reset_block or "--branch" in reset_block, \ + "the refusal does not tell the reader what to do instead" + + +def test_the_series_applied_check_is_asked_of_the_series_not_each_patch(): + """`git apply --check -R` per patch is the WRONG question for a series. + + Found by running --onto-current against a tree that already had the overlay: + + error: patch failed: Libraries/LibRequests/Request.cpp:327 + error: failed to apply 0001-...patch + + 0002 edits lines ADJACENT to 0001's inside the same function, so on a fully + patched tree 0001's *context lines no longer exist* -- 0002 rewrote them. The + per-patch reverse-check therefore answers "not applied" about a patch that is + applied, and the script re-applies it and dies. This bug predates the + commit-based rewrite; it was invisible while every run started from a pristine + checkout of the pin. + + Reverse-checking the CONCATENATION asks the real question. Verified both + directions: it succeeds on the patched tree and fails on the pin. + """ + t = _text() + # Both the apply path and --verify must use the series form. + assert t.count('cat "$PATCHES"/*.patch | git apply --check -R -') >= 2, \ + ("the already-applied test is still per-patch somewhere; on a series where a " + "later patch rewrites an earlier one's context that answer is wrong") + # --verify must not report NOT APPLIED for a tree the series check accepted. + verify = t.split("if [ \"$VERIFY\" -eq 1 ]", 1)[1].split("exit \"$rc\"", 1)[0] + assert "series_ok" in verify, \ + "--verify still judges each patch alone, so a correct tree reports NOT APPLIED" + + +def test_verify_accepts_the_pin_as_an_ancestor_not_only_as_head(): + """Once the overlay is commits, the pin is HEAD's ANCESTOR -- not HEAD. + + `--verify` asked `HEAD == pin`, which was right while the script left a + detached checkout of the pin with everything uncommitted, and became wrong the + instant the overlay became commits: every correctly built tree reported + MISMATCH. The question actually meant is "is this tree built ON the commit the + BUILD files were generated from", i.e. an ancestor test. + + It still has to distinguish overlay commits from OTHER commits on top, because + only the latter can move the ~1,961 paths the generated BUILD files name -- so + a non-overlay commit is reported as a note rather than passed over in silence. + """ + t = _text() + verify = t.split("if [ \"$VERIFY\" -eq 1 ]", 1)[1].split("exit \"$rc\"", 1)[0] + assert "merge-base --is-ancestor" in verify, \ + ("--verify still requires HEAD to BE the pin, so a tree with the overlay " + "committed on top of it fails verification") + assert "rev-list --count" in verify, \ + "--verify does not report how many commits sit on top of the pin" + + +def test_the_ignored_build_vcpkg_file_is_not_committed(): + """Build/vcpkg/BUILD.bazel must stay OUT of the commit. + + Ladybird's .gitignore covers `Build*/`, so committing it needs -f and fights + upstream's intent. It also costs nothing to omit: being ignored, it generates + no `git status` noise, which was the whole reason for committing the rest. + """ + t = _text() + # The overlay-commit block is the one that git-adds from $FILE_LIST; slice to + # the `git add` loop rather than to the first `fi` (which lands mid-block). + commit_block = t.split("git add --force", 1)[0].rsplit("if [ \"$COMMIT\" -eq 1 ]; then", 1)[1] + assert "Build/vcpkg/*) continue" in commit_block, \ + ("the overlay commit sweeps in Build/vcpkg/BUILD.bazel, which Ladybird's " + ".gitignore excludes and which must not exist before the vcpkg prefetch") + + +def test_the_qt_sdk_path_is_preserved_across_a_reapply(): + """MODULE.bazel's Qt path is the reader's, and a re-apply must not clobber it. + + Ulf: "We're using Qt (6.9.2) from a VENV, and system Qt is 6.4.2." The overlay + hardcodes `paths = {"linux-x86_64": "/usr/lib/qt6"}` and the copy phase + overwrites MODULE.bazel, so a re-apply silently repointed his build from a + working 6.9.2 at a system 6.4.2 -- which is BELOW Ladybird's 6.9 floor, i.e. + the re-apply turned a working tree into a failing one, and the failure surfaced + later and elsewhere. + + Every other line in the overlay is a fact about Ladybird at the pin, identical + on every host. This one names an SDK on YOUR machine, so it is resolved rather + than imposed: --qt-prefix, else the line already in your MODULE.bazel, else the + qmake first on your PATH (which is how a venv says which Qt it means), else the + historical default. Verified: two consecutive runs, the second with no flags, + leave /tmp/venvqt in place. + """ + t = _text() + assert "--qt-prefix" in t, "no way to name the Qt SDK" + # Rule 2 is the one that makes a re-apply safe: read the target's own value. + assert "qt_prefix_in_tree" in t, \ + "the script does not read the Qt path already configured in the target tree" + # Rule 3: a venv/aqt SDK puts its qmake on PATH; that is the right default for it. + assert "qmake" in t and "QT_INSTALL_PREFIX" in t, \ + "the script cannot discover a Qt from qmake, so a venv SDK must be typed by hand" + # It must be applied AFTER the copy, or the copy overwrites it again. + copy_idx = t.index("copying the overlay") + set_idx = t.index("set_qt_prefix_in \"$TARGET/MODULE.bazel\"") + assert set_idx > copy_idx, \ + "the Qt path is written before the overlay copy, which then overwrites it" + # And the reader must be TOLD which rule won -- a silent default is the bug. + assert "from $QT_SOURCE" in t or "QT_SOURCE" in t, \ + "the script does not report where the Qt prefix came from" + + +def test_verify_treats_the_qt_sdk_path_as_expected_to_differ(): + """--verify must not report the reader's own Qt path as a defect. + + Reporting `DIFFERS MODULE.bazel` told people to overwrite their correct + configuration with the overlay's -- and doing that is exactly how a venv Qt + 6.9.2 became a system Qt 6.4.2. So the comparison normalises that one line and, + when the rest matches, reports the configured prefix instead of a failure. + """ + t = _text() + verify = t.split("if [ \"$VERIFY\" -eq 1 ]", 1)[1].split("exit \"$rc\"", 1)[0] + assert "MODULE.bazel" in verify, "--verify has no special case for MODULE.bazel" + assert "@@QT@@" in verify, \ + "--verify does not normalise the Qt path line before comparing" + # It must still catch a REAL difference in that file. + assert "DIFFERS" in verify, "--verify no longer reports differing files at all" + + +def test_a_qt_below_the_floor_is_reported_at_apply_time(): + """A too-old Qt must be named when it is chosen, not deep in a build. + + The 6.9 floor is enforced in qt_runtime.bzl (where Bazel can fail the build), + but by then the reader is several minutes into `bazel build` and the message + names a repository rule. If the prefix is resolvable at apply time, its version + is cheap to read, so the script says so immediately -- verified against a fake + 6.4.2 SDK, which is the version Ulf's system Qt actually is. + """ + t = _text() + assert "qt_version_at" in t, "the script never reads the chosen Qt's version" + assert "6.9" in t, "the floor is not mentioned where the SDK is chosen" + assert re.search(r"6\.\[0-8\]\.\*", t), \ + "no check that the chosen Qt is below the 6.9 floor" + + +def test_every_required_prefetch_is_RUN_not_merely_printed(): + """A step the build cannot do without must be executed, not documented. + + The concrete failure: the script ran `Meta/ladybird.py vcpkg` itself but only + PRINTED `Meta/fetch_vcpkg_git_archives.py` in its closing message. So the + script reported success, the obvious next command was `bazel build`, and that + died inside the vcpkg action with "no git-sourced externals at + ./Meta/CMake/vcpkg/git-archives". Ulf hit exactly this. The four tarballs are + not optional and not derivable from anything else in the tree -- vcpkg_from_git + shells out to `git fetch`, which no asset source intercepts -- so a tree without + them cannot build, and a setup script that leaves the tree unable to build has + not finished. + + Asserted structurally: each prefetch appears on a line that RUNS it (a `python3` + invocation outside a heredoc/message), not only inside the closing `cat <`. diff --git a/tests/test_diff_ts.py b/tests/test_diff_ts.py index 3a5554f..551aadc 100644 --- a/tests/test_diff_ts.py +++ b/tests/test_diff_ts.py @@ -96,15 +96,10 @@ def test_same_source_multiple_actions_union(): assert ext_diffs == [], ext_diffs assert stripped["npm"] == 1 - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_emit_cargo.py b/tests/test_emit_cargo.py index 2a7b950..bcb057c 100644 --- a/tests/test_emit_cargo.py +++ b/tests/test_emit_cargo.py @@ -135,6 +135,34 @@ def _read(rel): ) ''' +# The OTHER kind of Rust target CMake has, and the one this migration missed +# entirely for a long time (finding 35): build_rust_binary(). Two shapes, both +# real -- a pure build tool with no header at all, and a runtime tool whose build +# script ALSO runs cbindgen (FFI_OUTPUT_DIR present at the call site). +CMAKE_JS = ''' +build_rust_binary( + MANIFEST_PATH Flap/Cargo.toml + CRATE_NAME flapc + BINARY_NAME flapc + OUTPUT_PATH_VAR FLAPC_BIN +) +''' + +CMAKE_WASM = ''' +build_rust_binary( + MANIFEST_PATH Rust/Cargo.toml + CRATE_NAME libwasm_cranelift + BINARY_NAME cranelift-compiler + OUTPUT_PATH_VAR WASM_CRANELIFT_COMPILER_BINARY + FFI_OUTPUT_DIR "${CMAKE_CURRENT_BINARY_DIR}" +) +''' + +# The bare include that makes libwasm_cranelift's header dir opt-in: LibWasm +# spells it with no directory, because CMake's FFI_OUTPUT_DIR is LibWasm's own +# binary dir. The scan must find this rather than a list saying so. +WASM_TU = "#include \n" + def _fixture(): d = tempfile.mkdtemp() @@ -145,10 +173,30 @@ def _fixture(): os.makedirs(os.path.join(d, "Libraries/LibJS/Flap")) with open(os.path.join(d, "Libraries/LibJS/Flap/Cargo.lock"), "w") as f: f.write(FLAP_LOCK) - for lib, text in (("LibGfx", CMAKE_GFX), ("LibWeb", CMAKE_WEB)): + with open(os.path.join(d, "Cargo.toml"), "w") as f: + f.write('[workspace]\nmembers = ["Libraries/LibWasm/Rust"]\n' + 'exclude = ["Libraries/LibJS/Flap"]\n') + # The crate DIRECTORIES the member list names, because a source set derived + # from Cargo.toml is only derived if the directory has to exist -- that is + # exactly the property that made the hardcoded BytecodeDef/** glob outlive its + # directory. The LibWasm crate path-depends on nothing; LibWeb's does, on the + # excluded flapc workspace, which is how a path dep reaches OUTSIDE the + # workspace (libjs_rust build-depends on flapc that way). + for crate, manifest in ( + ("Libraries/LibWasm/Rust", '[package]\nname = "libwasm_cranelift"\n'), + ("Libraries/LibWeb/Rust", + '[package]\nname = "libweb_rust"\n\n[build-dependencies]\n' + 'flapc = { path = "../../LibJS/Flap" }\n')): + os.makedirs(os.path.join(d, crate), exist_ok=True) + with open(os.path.join(d, crate, "Cargo.toml"), "w") as f: + f.write(manifest) + for lib, text in (("LibGfx", CMAKE_GFX), ("LibWeb", CMAKE_WEB), + ("LibJS", CMAKE_JS), ("LibWasm", CMAKE_WASM)): os.makedirs(os.path.join(d, "Libraries", lib), exist_ok=True) with open(os.path.join(d, "Libraries", lib, "CMakeLists.txt"), "w") as f: f.write(text) + with open(os.path.join(d, "Libraries/LibWasm/CraneliftBridge.cpp"), "w") as f: + f.write(WASM_TU) return d @@ -163,15 +211,15 @@ def test_only_checksummed_packages_become_fetch_rules(): assert {n for n, _v in workspace} == {"libgfx_rust", "libweb_rust"} -def test_the_real_lock_splits_154_registry_from_13_workspace(): +def test_the_real_lock_splits_155_registry_from_11_workspace(): """Pins the counts against the checked-in generated file, which is ground truth neither this test nor the emitter produced (finding 25's rule).""" crates = _read("cargo_crates.bzl") - assert crates.count("http_archive(") == 154, crates.count("http_archive(") + assert crates.count("http_archive(") == 155, crates.count("http_archive(") # And every one of them names a crates.io URL, i.e. none is a workspace member # that slipped through. urls = re.findall(r"urls = \['([^']+)'\]", crates) - assert len(urls) == 154 + assert len(urls) == 155 assert all(u.startswith("https://static.crates.io/crates/") for u in urls) @@ -220,15 +268,15 @@ def test_repo_names_are_unique_per_crate_version(): """`-` and `.` are illegal in a repo name; the normalization must not collide.""" assert emit.repo_name("aho-corasick", "1.1.4") == "crate_aho_corasick_1_1_4" names = set(re.findall(r"name = '([^']+)'", _read("cargo_crates.bzl"))) - # 154 crates -> 154 distinct repo names, i.e. the normalization is injective + # 155 crates -> 155 distinct repo names, i.e. the normalization is injective # on the real input set. - assert len(names) == 154 + assert len(names) == 155 def test_a_crate_archive_declares_its_type_because_dot_crate_is_not_known(): """http_archive rejects a `.crate` suffix; without type= the fetch fails with a message about .zip/.tar.gz that says nothing about the cause.""" - assert _read("cargo_crates.bzl").count('type = "tgz"') == 154 + assert _read("cargo_crates.bzl").count('type = "tgz"') == 155 # --------------------------------------------------------------------------- @@ -243,24 +291,31 @@ def test_features_are_parsed_per_crate_including_the_variable_form(): assert specs["libweb_css_rust"]["features"] == ["allocator"] -def test_the_real_feature_set_is_three_of_ten_and_not_libgfx(): - """The spec for this work said libgfx_rust takes `allocator`. It does not -- - cargo fails outright ("the package 'libgfx_rust' does not contain this - feature"). Pinned against the generated index, so a future bump that changes - a feature has to change this number deliberately.""" +def test_the_real_feature_set_is_parsed_per_crate_and_moves_with_upstream(): + """Features are the ABI, so they are PARSED, and the parse is pinned here. + + The spec for this work said libgfx_rust takes `allocator`; at the commit this + overlay first targeted it did not even HAVE that feature (cargo: "the package + 'libgfx_rust' does not contain this feature"), and one repin later it does. + That is the whole argument for parsing rather than writing it down -- and it + is why this test asserts the parsed SET against the generated index rather + than a remembered number: a crate gaining, losing or merging a feature has to + change this assertion deliberately. + """ index = _read("cargo_index.bzl") specs = re.findall(r"'(\w+)': \{\n\s+\"manifest\": '[^']+',\n\s+" r"\"features\": (\[[^\]]*\])", index) feats = {name: eval(f) for name, f in specs} - assert len(feats) == 10, sorted(feats) + assert len(feats) == 8, sorted(feats) with_alloc = {n for n, f in feats.items() if f == ["allocator"]} - assert with_alloc == {"libregex_rust", "liburl_rust", "libunicode_rust", - "libweb_content_blocker_rust", "libweb_css_rust", - "libweb_layout_rust"}, sorted(with_alloc) - assert feats["libgfx_rust"] == [] + assert with_alloc == {"libgfx_rust", "libregex_rust", "libunicode_rust", + "liburl_rust", "libweb_content_blocker_rust"}, \ + sorted(with_alloc) assert feats["libjs_rust"] == [] assert feats["libtextcodec_rust"] == [] - assert feats["libweb_rust"] == [] + # The consolidated LibWeb crate carries a feature that is NOT `allocator`, so + # a parse that only ever looked for that one word would drop it silently. + assert feats["libweb_rust"] == ["style-recording"] # --------------------------------------------------------------------------- @@ -272,13 +327,24 @@ def test_ffi_header_lists_are_not_uniform(): hdrs = dict(re.findall(r"'(\w+)': \{\n(?:.*\n)*?\s+\"ffi_headers\": (\[[^\]]*\])", index)) hdrs = {k: eval(v) for k, v in hdrs.items()} - assert hdrs["libweb_css_rust"] == ["ComputedValuesRustFFI.h", "RustFFI.h", - "SelectorRustFFI.h", "StyleValueRustFFI.h"] - assert hdrs["libweb_layout_rust"] == ["Layout/TreeBuilderRustFFI.h"] assert hdrs["libweb_content_blocker_rust"] == ["ContentBlockerRustFFI.h"] assert hdrs["libgfx_rust"] == ["RustFFI.h"] - # 14 headers across 10 crates, which is the number in CMake's own build tree. - assert sum(len(v) for v in hdrs.values()) == 14 + # Upstream consolidated libweb_css_rust and libweb_layout_rust back INTO + # libweb_rust, so one crate now writes what three used to -- 12 files, two of + # them `.inc` rather than `.h`. An undeclared .inc is deleted by Bazel exactly + # like an undeclared header, so the suffix must not be what decides whether a + # generated file is declared. + assert hdrs["libweb_rust"] == [ + "ComputedValuesRustFFI.h", "HTML/Parser/RustFFI.h", + "HTMLTokenizerRustFFI.h", "Layout/LayoutRustFFI.h", + "Layout/TreeBuilderRustFFI.h", "RustFFI.h", "SelectorRustFFI.h", + "StyleEngineBridgeGenerated.h", "StyleEngineBridgeGenerated.inc", + "StyleEngineRustFFI.h", "StyleEngineStateFactsGenerated.inc", + "StyleValueRustFFI.h"] + assert [h for h in hdrs["libweb_rust"] if h.endswith(".inc")], \ + "the .inc outputs stopped being declared" + # 19 files across 8 crates, which is the number in CMake's own build tree. + assert sum(len(v) for v in hdrs.values()) == 19 def test_the_header_cmake_never_declares_is_still_declared_here(): @@ -330,9 +396,17 @@ def test_colliding_header_names_are_real(): index)) owners = [k for k, v in hdrs.items() if "'RustFFI.h'" in v] assert len(owners) >= 6, owners - driver = _read("Meta/cargo_build.sh") - assert "root-output" in driver, "the header lookup no longer prefers OUT_DIR" - assert "FFI_SCRATCH" in driver, "FFI_OUTPUT_DIR is no longer a scratch dir" + # The lookup lives in cargo_vendor.sh, SHARED by both cargo drivers -- the + # staticlib one and the --bin one -- because a binary crate's build script + # runs cbindgen into the same shared dir and would collide the same way + # (libwasm_cranelift emits CraneliftFFI.h). One copy, so the two cannot drift. + lib = _read("Meta/cargo_vendor.sh") + assert "root-output" in lib, "the header lookup no longer prefers OUT_DIR" + assert "FFI_SCRATCH" in lib, "FFI_OUTPUT_DIR is no longer a scratch dir" + for driver in ("Meta/cargo_build.sh", "Meta/cargo_binary_build.sh"): + txt = _read(driver) + assert "sync_ffi_headers " in txt, driver + assert "root-output" not in txt, "%s has its own copy of the lookup" % driver def test_headers_are_prefixed_so_both_include_spellings_resolve(): @@ -342,8 +416,8 @@ def test_headers_are_prefixed_so_both_include_spellings_resolve(): different package.""" mod = _load(_fixture()) assert mod.FFI_PREFIX["liburl_rust"] == "LibURL" - assert mod.FFI_PREFIX["libweb_css_rust"] == "LibWeb" - assert mod.FFI_PREFIX["libweb_layout_rust"] == "LibWeb" + assert mod.FFI_PREFIX["libweb_rust"] == "LibWeb" + assert mod.FFI_PREFIX["libweb_content_blocker_rust"] == "LibWeb" # Every crate with observed headers has a prefix: a missing one would stage # the header at the include root and silently change its include spelling. assert set(mod.FFI_HEADERS_OBSERVED) <= set(mod.FFI_PREFIX) @@ -376,7 +450,7 @@ def test_the_index_key_carries_name_version_and_hash(): key, since a Bazel label carries none of them.""" index = _read("cargo_index.bzl") keys = re.findall(r"^ '([^']+)': '@crate_", index, re.M) - assert len(keys) == 154 + assert len(keys) == 155 for k in keys: name, version, sha = k.split(" ") assert re.fullmatch(r"[0-9a-f]{64}", sha), k @@ -405,7 +479,7 @@ def test_the_emitter_is_idempotent_and_needs_no_cargo_or_network(): mod.emit_crates(crates) mod.emit_index(crates, specs) mod.emit_extension(crates) - mod.emit_ring(crates, specs) + mod.emit_ring(crates, specs, mod.binary_specs()) outs.append(buf.getvalue()) assert outs[0] == outs[1] assert "http_archive(" in outs[0] @@ -434,7 +508,8 @@ def test_check_flag_detects_drift_in_a_generated_file(): {"--crates": lambda: mod.emit_crates(crates), "--index": lambda: mod.emit_index(crates, specs), "--extension": lambda: mod.emit_extension(crates), - "--ring": lambda: mod.emit_ring(crates, specs)}[fl]() + "--ring": lambda: mod.emit_ring(crates, specs, + mod.binary_specs())}[fl]() with open(os.path.join(d, "out", fn), "w") as f: f.write(b.getvalue()) buf = io.StringIO() @@ -475,13 +550,20 @@ def test_nothing_generated_still_reads_the_cmake_cargo_tree(): """ offenders = [] for rel in ("BUILD.bazel", "Libraries/LibWeb/BUILD.bazel", "codegen_root.bzl", - "cargo_ring.bzl", "cargo_index.bzl", "bazelrc.txt", - "Build/full/Libraries/BUILD.bazel"): + "cargo_ring.bzl", "cargo_index.bzl", "bazelrc.txt"): txt = _read(rel) if re.search(r"//Build/full/cargo|Build/full/cargo/build|rust_ffi_headers\b", txt.replace("no rust_ffi_headers", "")): offenders.append(rel) assert not offenders, "still reference CMake's cargo tree: %s" % offenders + # And the stronger statement, which is what finding 35 cost a week to learn: + # the overlay must not contain a Build/full PACKAGE at all. It used to ship + # three (Libraries/, Services/, UI/), globbing CMake's build tree with + # allow_empty = True -- so on a fresh clone they matched nothing, reported + # nothing, and the build died 1,600 actions later on a missing header. A file + # list that CAN be empty proves nothing; a directory that must not exist does. + assert not os.path.exists(os.path.join(_WS, "Build", "full")), \ + "the Build/full shim packages are back" def test_each_crate_links_its_own_archive_and_no_others(): @@ -509,9 +591,17 @@ def test_each_crate_links_its_own_archive_and_no_others(): assert "user_link_flags = [archive.path]" in impl, "must link exactly one archive" assert "additional_inputs" in impl, "the archive would not be in the sandbox" ring = _read("cargo_ring.bzl") - assert ring.count("cargo_crate(") == 10 - assert ring.count("cargo_lib(") == 10, "one consumable target per crate" + assert ring.count("cargo_crate(") == 8 + # One consumable target per crate that has something to consume: the 8 + # staticlib crates, plus libwasm_cranelift -- a `--bin` crate, so it has NO + # archive at all and cargo_lib yields a headers-only CcInfo for it (the + # executable is spawned at run time, and travels as `data`). That "archive + # may be None" branch is the shape a binary crate needs, and it is why the + # count is 9 rather than 8. + assert ring.count("cargo_lib(") == 9, "one consumable target per crate" + assert ring.count("cargo_binary(") == 4 assert "cargo_libs(" not in ring, "the shared link group is gone" + assert "if archive == None:" in impl, "a --bin crate has no archive to link" def test_a_library_depends_on_the_crates_it_uses_and_no_others(): @@ -536,9 +626,11 @@ def crates_of(txt, name, path="//:"): assert crates_of(root, "LibGfx") == ["libgfx_rust"] assert crates_of(root, "LibJS") == ["libjs_rust"] assert crates_of(root, "LibRegex") == ["libregex_rust"] + # Two, not four: upstream consolidated libweb_css_rust and libweb_layout_rust + # into libweb_rust, and the dep edges follow CMake's + # target_link_libraries(LibWeb PRIVATE libweb_rust libweb_content_blocker_rust). assert crates_of(libweb, "LibWeb") == [ - "libweb_content_blocker_rust", "libweb_css_rust", - "libweb_layout_rust", "libweb_rust", + "libweb_content_blocker_rust", "libweb_rust", ] @@ -573,12 +665,96 @@ def test_flapc_is_built_by_bazel_and_used_as_the_genrule_tool(): """The last artifact this migration took from CMake.""" ring = _read("cargo_ring.bzl") assert "cargo_binary(" in ring - assert 'bin = "flapc"' in ring + assert re.search(r"bin = ['\"]flapc['\"]", ring) codegen = _read("codegen_root.bzl") assert "tools = ['//:flapc']" in codegen assert "Build/full/bin/flapc" not in codegen +def test_every_build_rust_binary_crate_is_in_the_ring_with_its_header(): + """Finding 35's second bug: a whole crate missing because CMake was not read. + + `Libraries/LibWasm/CMakeLists.txt` was absent from CMAKELISTS and + `build_rust_binary()` was not parsed at all, so libwasm_cranelift -- a crate + that emits a header LibWasm's CraneliftBridge.cpp includes -- was ENTIRELY + ABSENT from the Bazel graph. Nothing said so: the header sat in + `Build/full/Libraries/LibWasm` where a global `-I` reached it, and the binary + was named by an absolute path baked into + `-DWASM_CRANELIFT_COMPILER_PATH=/home/ubuntu/...` -- two host escapes covering + for one missing target, which is why removing the shims was the only thing + that could find it. + + Two halves, both asserted: the CALL SITE is parsed (a `--bin` crate has no + archive, but it can still emit a header, and whether it does is + `FFI_OUTPUT_DIR` at the call site -- not a list written down here), and every + binary crate reaches the checked-in ring with the same two consumable labels + a staticlib crate gets. + """ + mod = _load(_fixture()) + bins = {b["bin"]: b for b in mod.binary_specs()} + assert set(bins) == {"flapc", "cranelift-compiler"}, sorted(bins) + # flapc's call site has no FFI_OUTPUT_DIR, so it declares no header at all; + # asking for one would fail the action (cargo never writes it). + assert bins["flapc"]["ffi_headers"] == [] + assert bins["flapc"]["manifest"] == "Libraries/LibJS/Flap/Cargo.toml" + # libwasm_cranelift's does, so the header is a DECLARED output -- Bazel + # deletes what nothing declares -- and the bare-include flag comes from + # SCANNING the tree for the directory-less spelling, not from a list. + assert bins["cranelift-compiler"]["ffi_headers"] == ["CraneliftFFI.h"] + assert bins["cranelift-compiler"]["ffi_bare_include"] is True + + # And the same two, present in the checked-in ring and consumed by the root + # package. Pinned by count so a call site cannot appear unnoticed. + # + # FOUR at the real commit, not two: upstream added `generate-libjs-bytecode` + # (a SECOND --bin out of the flapc crate, which replaced the deleted Python + # bytecode generator) and `style-replay` (a --bin out of libweb_rust, which + # also builds a staticlib). Keyed by BIN rather than by crate for exactly that + # reason -- two of the four share a crate with another target, so a dict keyed + # by crate name silently drops one of each pair. + ring = _read("cargo_ring.bzl") + root = _read("BUILD.bazel") + assert ring.count("cargo_binary(") == 4, ring.count("cargo_binary(") + for b in bins.values(): + assert re.search(r"bin = ['\"]%s['\"]" % re.escape(b["bin"]), ring), b + if not b["ffi_headers"]: + continue + assert "'%s_lib'" % b["crate"] in ring + assert "//:%s_lib" % b["crate"] in root + assert "implementation_deps = ['//:%s_bare_include']" % b["crate"] in root + # The binary itself is data, not a link input: LibWasm SPAWNS it. + assert "data = ['//:cranelift-compiler']" in root + + +def test_a_bare_include_dir_arrives_only_through_implementation_deps(): + """Finding 35's include-collision bug, in the one form that actually fixed it. + + 8 of the 10 crates emit a header literally named `RustFFI.h`, and 4 TUs + include it bare (`#include `), which CMake allows because + FFI_OUTPUT_DIR is PRIVATE to the owning library. Splitting the unprefixed dir + into its own `cargo_bare_include` target was necessary but NOT sufficient: + CcInfo include dirs also propagate along the C++ dep graph, so LibGfx + inherited LibTextCodec's dir through `LibGfx -> LibTextCodec` and + `YUVData.cpp` failed with `'FFI' does not name a type` -- compiling against + the WRONG crate's header. `implementation_deps` is Bazel's name for exactly + CMake's PRIVATE, and only it stops the propagation. + + Hence: every bare-include target in the tree must be reached through + implementation_deps, never `deps`. One `deps = [...bare_include]` anywhere + silently re-creates the bug for everything downstream of that library. + """ + root = _read("BUILD.bazel") + targets = sorted(set(re.findall(r"//:(\w+_bare_include)", root))) + assert targets, "no crate exposes an unprefixed FFI include dir any more" + for t in targets: + assert "implementation_deps = ['//:%s']" % t in root, t + # ... and nothing anywhere else in the overlay pulls one in publicly. + for rel in ("BUILD.bazel", "Libraries/LibWeb/BUILD.bazel", "cargo_ring.bzl"): + txt = _read(rel) + for m in re.finditer(r"\n deps = \[([^\]]*)\]", txt): + assert "bare_include" not in m.group(1), (rel, m.group(1)) + + def test_the_libweb_package_exports_its_crate_sources(): """Bazel packages cut across the cargo workspace: five crates live under Libraries/LibWeb, which is its own package, and glob() is package-relative -- @@ -587,16 +763,117 @@ def test_the_libweb_package_exports_its_crate_sources(): """ libweb = _read("Libraries/LibWeb/BUILD.bazel") assert 'name = "rust_crate_srcs"' in libweb - for sub in ("Rust/**", "CSS/Rust/**", "Layout/Rust/**", - "ContentBlocker/Rust/**", "HTML/Parser/Rust/**"): - assert '"%s"' % sub in libweb, sub - # The CSS data files libweb_css_rust's build script GENERATES Rust from. + for sub in ("ContentBlocker/Rust/**", "HTML/Parser/Rust/**", "Rust/**"): + assert "'%s'" % sub in libweb, sub + # The CSS data files libweb_rust's build script GENERATES Rust from. for data in ("CSS/Properties.json", "CSS/Keywords.json", "CSS/Enums.json", "HTML/TagNames.h", "HTML/Parser/Entities.json"): - assert '"%s"' % data in libweb, data + assert "'%s'" % data in libweb, data assert "//Libraries/LibWeb:rust_crate_srcs" in _read("cargo_ring.bzl") +def test_no_allow_empty_false_glob_names_a_directory_that_is_written_down(): + """The failure this repin existed to fix, as a test. + + Ulf's build died at LOADING time with + + Error in glob: glob pattern 'Libraries/LibJS/BytecodeDef/**' didn't match + anything, but allow_empty is set to False + + because upstream a32d9c9f deleted that directory and the pattern was a + hardcoded string in the emitter. Two properties make this the worst shape of + stale constant in the repo: + + * `allow_empty = False` is CORRECT and must stay. The alternative + (allow_empty = True) is what let three Build/full shim packages match + nothing for weeks and fail 1,600 actions later -- see + test_nothing_generated_still_reads_the_cmake_cargo_tree. + * A loading-time error has no target to blame, so nothing in the build graph + can report it and no per-target test can catch it. It is not "a crate + failed"; it is "the workspace does not load". + + So the guard is structural: every directory an allow_empty=False glob names + must be DERIVED from the tree (Cargo.toml's members closed over `path =` + deps), and the assertion is that the emitter's own derivation is what appears + in the generated file -- with no directory-shaped pattern left over that the + derivation did not produce. + """ + d = _fixture() + mod = _load(d) + # The fixture's Cargo.toml lists ONE member, whose manifest path-depends on + # the EXCLUDED flapc workspace; the closure therefore reaches outside the + # member list, which is how libjs_rust reaches flapc in the real tree. + assert mod.crate_dirs() == ["Libraries/LibWasm/Rust"], mod.crate_dirs() + assert mod.crate_src_globs() == mod.CRATE_SRC_ROOT_FILES + \ + ["Libraries/LibWasm/Rust/**"] + # A crate inside another Bazel package does NOT become a root-package glob -- + # it becomes a label, because glob() cannot cross a package boundary. + assert mod.crate_src_labels() == [] + # Delete the crate and the pattern goes away rather than outliving it: THE + # property. Under the old hardcoded list the pattern survived the directory, + # and Bazel then refused to load the workspace at all. + import shutil + shutil.rmtree(os.path.join(d, "Libraries/LibWasm/Rust")) + assert mod.crate_dirs() == [] + assert "Libraries/LibWasm/Rust/**" not in mod.crate_src_globs() + + # And structurally, in the emitter source: no module-level constant may hold a + # directory-shaped glob pattern. That is the form the bug took -- a `/**` + # string in a list literal, which nothing re-derives and no build-graph test + # can reach, because the glob fails before any target exists. + for rel in ("Meta/emit_cargo_bazel.py", "Meta/emit_libweb_bazel.py"): + src = _read(rel) + # Strip docstrings/comments: the block comments here NAME the deleted + # directories on purpose, to record what broke. + code = "\n".join(l for l in src.splitlines() + if not l.strip().startswith("#")) + code = re.sub(r'\'\'\'.*?\'\'\'', "", code, flags=re.S) + code = re.sub(r'""".*?"""', "", code, flags=re.S) + bad = [m.group(1) for m in re.finditer(r'"([A-Za-z][\w/.-]*/\*\*)"', code)] + bad += [m.group(1) for m in re.finditer(r"'([A-Za-z][\w/.-]*/\*\*)'", code)] + assert not bad, "%s hardcodes crate directories: %s" % (rel, bad) + + +def test_the_target_dir_exclusion_is_anchored_to_each_crate_root(): + """`**/target/**` is wrong, and it took down the build to prove it. + + Excluding cargo's output directory is right -- it is never an input, and a + developer who ran cargo by hand in the tree would otherwise get it globbed + into every crate's action inputs. Spelling it `**/target/**` is not: flapc has + a Rust MODULE at `src/target/` (the code generator's per-architecture + backends), so the unanchored pattern silently dropped 40 source files and + `//:generate-libjs-bytecode` failed to compile with + + error[E0583]: file not found for module `target` + + A glob exclusion is a pattern over PATHS and knows nothing about what a + directory means. "The directory cargo writes to" is `/target`, and + only the anchored form says that -- so this asserts every exclusion in the + generated files is rooted at a crate directory, not floating. + """ + mod = _load() + dirs = ["Libraries/LibJS/Flap", "Libraries/LibGfx/Rust"] + assert mod.crate_src_glob_excludes(dirs) == [ + "Libraries/LibJS/Flap/target/**", "Libraries/LibGfx/Rust/target/**"] + # Nothing in the tree may carry the unanchored form, in the emitters or in + # what they generated. + for rel in ("cargo_ring.bzl", "Libraries/LibWeb/BUILD.bazel", + "Meta/emit_cargo_bazel.py", "Meta/emit_libweb_bazel.py"): + txt = _read(rel) + code = "\n".join(l for l in txt.splitlines() + if not l.strip().startswith("#")) + assert "**/target/**" not in code, \ + "%s excludes target/ unanchored -- it eats src/target/" % rel + # Every exclusion that IS there is rooted at a directory the SAME file globs + # as a crate, so an exclusion cannot name a directory no crate owns. + for rel, in (("cargo_ring.bzl",), ("Libraries/LibWeb/BUILD.bazel",)): + txt = _read(rel) + globbed = set(re.findall(r"'([A-Za-z0-9_/.-]+)/\*\*'", txt)) + for m in re.finditer(r"'([A-Za-z0-9_/.-]+)/target/\*\*'", txt): + assert m.group(1) in globbed, \ + "%s excludes %s/target but globs no such crate" % (rel, m.group(1)) + + def test_every_extension_created_repo_is_named_in_module_bazel(): """bzlmod needs every repo an extension creates named in use_repo() to be visible, and the failure ("no such repository") lands far from its cause. 157 @@ -612,3 +889,76 @@ def test_every_extension_created_repo_is_named_in_module_bazel(): # Plus the three toolchain components. assert {"rust_rustc_1_96_1", "rust_cargo_1_96_1", "rust_rust_std_1_96_1"} <= named + +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. + + +def test_the_flap_lock_is_described_from_the_lock_not_from_memory(): + """The comment the emitter PRINTS must be read out of the lock file. + + This one shipped wrong. The generated cargo_ring.bzl carried, in prose, + "its own lock with exactly 3 packages (flapc, in-tree bytecode_def, and + smallvec from crates.io pinned =1.15.1)" -- four hand-copied facts. Upstream's + a32d9c9f ("LibJS: Derive bytecodes from Flap handlers") deleted the + bytecode_def crate, so a generated file, whose header says AUTO-GENERATED, + confidently documented a package that no longer exists. Nothing failed: prose + has no compiler. + + Same class as the SYSTEM_LIBS divergence and the three-of-six glib include + roots -- and the same rule: if a fact is worth stating in generated output, it + is worth READING from the input. The fixture's flap lock has 2 packages, so a + re-hardcoded "3" fails here rather than six weeks later on Ulf's machine. + """ + import contextlib + import io + mod = _load(_fixture()) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + mod.emit_ring(mod.all_registry_crates(), mod.crate_specs(), + mod.binary_specs()) + ring = buf.getvalue() + # The described package count must match the fixture's lock, not the real one. + assert re.search(r"with 2 packages:", ring), \ + "the flap lock's package count is not derived from the lock" + assert "flapc 0.1.0 (in-tree)" in ring, \ + "the in-tree member is not identified from the absence of a checksum" + assert "smallvec 1.15.1" in ring, "the registry crate is not named from the lock" + # And the crate upstream deleted must not be resurrected by a hardcoded string. + assert "bytecode_def" not in ring, \ + "a deleted crate is still named in the generated output" + + +def test_no_generated_cargo_file_names_a_crate_it_does_not_declare(): + """The general form, against the CHECKED-IN artifacts — and no checkout. + + The test above guards the mechanism on a fixture; this guards the artifact. + It cannot compare against the real Cargo.locks, because those live in the + Ladybird checkout and not in this overlay -- a test that needs a 5,958-file + clone is a test that does not run. But the bug is visible WITHOUT the locks: + `bytecode_def` appeared in cargo_ring.bzl only ever inside a comment, never as + a `crate =` / `name =` / label. A crate the file talks about but never + declares is either dead prose or a missing rule, and both are worth failing + on -- the version of this check that needed the locks would have been skipped + in CI exactly when it mattered. + """ + for rel in ("cargo_ring.bzl", "cargo_crates.bzl", "cargo_index.bzl"): + text = _read(rel) + # The structural positions: what the file actually DECLARES. + declared = set(re.findall(r"(?:name|crate|bin) = '([^']+)'", text)) + declared |= set(re.findall(r"crate_([a-z0-9_]+)_\d", text)) + declared |= set(re.findall(r"//:([A-Za-z0-9_.-]+)", text)) + # A dict key is a declaration too (cargo_index.bzl keys its per-crate + # tables by crate name). + declared |= set(re.findall(r"^\s*'([^']+)':", text, re.M)) + for name in set(re.findall( + r"\b([a-z][a-z0-9]*(?:_[a-z0-9]+)*_rust|bytecode_def)\b", text)): + assert any(name in d for d in declared) or name in declared, \ + (f"{rel} mentions crate {name!r} in prose but never declares it " + "-- either dead text about a crate upstream deleted, or a rule " + "that is missing") diff --git a/tests/test_emit_libweb.py b/tests/test_emit_libweb.py new file mode 100644 index 0000000..9e3fdae --- /dev/null +++ b/tests/test_emit_libweb.py @@ -0,0 +1,371 @@ +"""LibWeb's generated-source lists: derived, or a capture that outlives its pin? + +`Libraries/LibWeb/generated_srcs.bzl` holds the two lists that tell Bazel which +of LibWeb's ~1,390 compile inputs come out of the Ring 1b codegen instead of the +source tree. Its first line said "AUTO-GENERATED by Meta/emit_libweb_bazel.py" +and it was not: no code path in that emitter wrote it. It was a capture of one +pin's answer, hand-maintained, and the header at the top asserted otherwise -- +which is worse than an honest hand-written file, because it tells a reader (and +a repin) that re-running the emitter will refresh it. + +The 71fb301a repin is what cashed that in. Upstream added five generated headers +(`Bindings/WindowGlobalMixin.h` and four siblings) and four generated .cpp; the +capture named none of them, so `Bindings/Window.h` -- itself generated, and +present in bazel-out -- included a header the cc_library did not declare: + + bazel-out/.../Bindings/Window.h:17:10: fatal error: + LibWeb/Bindings/WindowGlobalMixin.h: No such file or directory + +Note the shape: the missing file EXISTED on disk next to the file that included +it. Only the declaration was stale, and only for the 9 entries that had moved +since the capture. Nothing in the build could have reported it earlier, because +a list of paths is consistent with itself. + +Both lists are derivable from facts the emitter already loads -- the CMake +reference's compile list, and codegen.bzl's own `outs` -- so these tests assert +they ARE derived, that the derivation distinguishes a genrule's outputs from its +inputs (the second bug, below), and that the checked-in file matches what the +emitter emits. +""" + +import importlib.util +import os +import re +import tempfile + +_WS = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") +_EMIT = os.path.join(_WS, "Meta", "emit_libweb_bazel.py") +_GENERATED_SRCS = os.path.join(_WS, "Libraries", "LibWeb", "generated_srcs.bzl") + + +def _read(rel): + with open(os.path.join(_WS, rel)) as f: + return f.read() + + +def _load(root=None, model=None): + """Load the emitter module, optionally pointed at a fixture checkout.""" + # The reference build dir is an env var (a repin needs two side by side), and + # the fixture's is Build/full71 -- the same name the 71fb301a tree uses. + os.environ["LADYBIRD_BUILD_REL"] = "Build/full71" + os.environ["LADYBIRD_BUILD_DIR"] = os.path.join(root or ".", "Build/full71") + for var, val in (("LADYBIRD_ROOT", root), ("LADYBIRD_MODEL", model)): + if val: + os.environ[var] = val + else: + os.environ.pop(var, None) + spec = importlib.util.spec_from_file_location("emit_libweb_bazel", _EMIT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def _lists(text): + """Parse the two lists out of a generated_srcs.bzl body.""" + out = {} + for name in ("LIBWEB_GENERATED_SRCS", "LIBWEB_GENERATED_HDRS"): + m = re.search(re.escape(name) + r"\s*=\s*\[(.*?)\n\]", text, re.S) + assert m, "no %s in generated_srcs.bzl" % name + out[name] = re.findall(r"'([^']+)'", m.group(1)) + return out + + +# --------------------------------------------------------------------------- +# A fixture codegen.bzl. Tiny on purpose: it is the CONTRACT between the emitter +# and the genrules, not a copy of the real 1,390-entry file. +# +# It reproduces the two things that made the real file hard: a genrule whose +# `srcs` names checked-in headers (gen_dom_tree reads HTML/TagNames.h), and a +# generated .cpp that is emitted but NOT compiled by CMake (Extra.cpp, standing +# in for WebGL/GLFunctions.cpp). +# --------------------------------------------------------------------------- +CODEGEN_BZL = '''load("@rules_cc//cc:defs.bzl", "cc_library") + +def libweb_codegen(): + native.genrule( + name = 'gen_MediaControlsDOM', + srcs = ['HTML/MediaControls.html', 'HTML/TagNames.h', + 'HTML/AttributeNames.h', 'SVG/TagNames.h'], + outs = ['HTML/MediaControlsDOM.h', 'HTML/MediaControlsDOM.cpp'], + cmd = "true", + ) + native.genrule( + name = 'gen_Extra', + srcs = ['CSS/Thing.json'], + outs = ['WebGL/Extra.cpp', 'WebGL/Extra.h'], + cmd = "true", + ) + +def libweb_bindings_codegen(): + native.genrule( + name = 'gen_bindings', + outs = [ + 'Bindings/Window.cpp', + 'Bindings/Window.h', + # A header with no matching .cpp -- the mixin shape that broke the + # repin. Bindings/Window.h includes it. + 'Bindings/WindowGlobalMixin.h', + 'Bindings/Fragment.inc', + ], + cmd = "true", + ) +''' + + +def _fixture(compiled=("Bindings/Window.cpp", "HTML/MediaControlsDOM.cpp")): + """A checkout with a codegen.bzl and a CMake model naming LibWeb's TUs.""" + import json + d = tempfile.mkdtemp() + os.makedirs(os.path.join(d, "Libraries/LibWeb")) + with open(os.path.join(d, "Libraries/LibWeb/codegen.bzl"), "w") as f: + f.write(CODEGEN_BZL) + gen = "Build/full71/Libraries/LibWeb/" + inputs = ["Libraries/LibWeb/DOM/Document.cpp"] + [gen + c for c in compiled] + model = {"targets": {"LibWeb": { + "kind": "shared_library", "role": "production", "deps": [], + "actions": [{"mnemonic": "CppCompile", "inputs": inputs, + "arguments": ["-c"]}], + }}} + mpath = os.path.join(d, "model.json") + with open(mpath, "w") as f: + json.dump(model, f) + with open(os.path.join(d, ".bazelrc"), "w") as f: + f.write("build --cxxopt=-std=c++23\n") + # emit_libweb_bazel imports emit_build_bazel (for rust_dep_labels), which at + # IMPORT time parses the reference build's build.ninja for the root + # package's generated sources. An empty one is a valid parse of "no + # generators", and keeps this fixture about LibWeb rather than about ninja. + os.makedirs(os.path.join(d, "Build/full71")) + open(os.path.join(d, "Build/full71/build.ninja"), "w").close() + return d, mpath + + +def _emit(compiled=("Bindings/Window.cpp", "HTML/MediaControlsDOM.cpp")): + d, mpath = _fixture(compiled) + mod = _load(d, mpath) + targets = mod.load() + return mod, _lists(mod.generated_srcs_bzl(targets["LibWeb"], + mod.genrule_outputs())) + + +def test_a_generated_header_upstream_adds_reaches_the_hdrs_list(): + """THE regression: a new codegen output must appear without anyone editing. + + This is the 71fb301a failure reduced to its bones. The fixture's + `gen_bindings` produces `Bindings/WindowGlobalMixin.h`, which no generated + .cpp pairs with and which the compile list therefore never mentions -- so a + derivation driven by "headers next to compiled sources" would miss it, the + way the capture did. It is an OUTPUT of a genrule, and that is the only fact + that makes it includable. + """ + _, lists = _emit() + hdrs = lists["LIBWEB_GENERATED_HDRS"] + assert "Bindings/WindowGlobalMixin.h" in hdrs, hdrs + # And the same for a non-.h generated header extension: LibWeb's real + # codegen emits .inc files that TUs #include, and an extension allowlist + # that forgot one would drop it just as silently. + assert "Bindings/Fragment.inc" in hdrs, hdrs + # Now delete the genrule that makes it and the entry goes away rather than + # outliving the codegen -- the property the capture did not have in either + # direction. + d, mpath = _fixture() + body = CODEGEN_BZL.replace(" 'Bindings/WindowGlobalMixin.h',\n", "") + with open(os.path.join(d, "Libraries/LibWeb/codegen.bzl"), "w") as f: + f.write(body) + mod = _load(d, mpath) + hdrs2 = _lists(mod.generated_srcs_bzl(mod.load()["LibWeb"], + mod.genrule_outputs()))["LIBWEB_GENERATED_HDRS"] + assert "Bindings/WindowGlobalMixin.h" not in hdrs2, hdrs2 + + +def test_a_genrules_own_inputs_are_not_mistaken_for_its_outputs(): + """The second bug, found while fixing the first, and never yet triggered. + + `genrule_outputs()` used to regex the WHOLE codegen.bzl for anything that + looked like a path, so a genrule's `srcs` counted as outputs too. Four + checked-in SOURCE headers (HTML/TagNames.h, HTML/AttributeNames.h, + SVG/TagNames.h, SVG/AttributeNames.h -- read by generate_dom_tree.py) were + thereby classified as generated. That is not cosmetic: the generated list is + the cc_library's hdrs `exclude=`, so those four were dropped from the glob + over the source tree AND re-added as labels no genrule produces. It never + broke the build only because the stale capture was consulted instead of this + answer -- i.e. one bug hid the other, and fixing the capture exposed it. + """ + _, lists = _emit() + hdrs = lists["LIBWEB_GENERATED_HDRS"] + for src_hdr in ("HTML/TagNames.h", "HTML/AttributeNames.h", + "SVG/TagNames.h"): + assert src_hdr not in hdrs, \ + "%s is a genrule INPUT (a checked-in header), not an output" % src_hdr + assert "HTML/MediaControlsDOM.h" in hdrs, hdrs # a real out= of that rule + + +def test_a_generated_cpp_cmake_does_not_compile_is_not_a_src(): + """The lists are not one set, and conflating them would be a link error. + + LibWeb's codegen emits WebGL/GLFunctions.cpp, which the CMake reference does + not compile into LibWeb. SRCS is therefore the reference's compile list, not + "every generated .cpp" -- while HDRS is every generated header, because a + header costs nothing to declare and not declaring one is a hard failure. + The fixture's WebGL/Extra.cpp stands in for GLFunctions.cpp. + """ + _, lists = _emit() + assert "WebGL/Extra.cpp" not in lists["LIBWEB_GENERATED_SRCS"] + assert "WebGL/Extra.h" in lists["LIBWEB_GENERATED_HDRS"] + # ...and the pairing is not assumed the other way either: a compiled + # generated .cpp must be a declared output, or Bazel has no rule making it. + mod, _ = _emit() + try: + mod.generated_srcs_bzl( + mod.load()["LibWeb"], {"Bindings/Window.cpp"}) + except AssertionError as e: + assert "not a genrule output" in str(e), e + else: + raise AssertionError("a compiled src absent from outs= was accepted") + + +def test_the_checked_in_generated_srcs_matches_what_the_emitter_emits(): + """The file says AUTO-GENERATED; this is what makes that claim checkable. + + Without it "derived" is a comment. The header line must also name the flag + that reproduces the file, since the last one named a script that could not + write it. + """ + txt = open(_GENERATED_SRCS).read() + assert txt.startswith("# AUTO-GENERATED by Meta/emit_libweb_bazel.py"), \ + txt.splitlines()[0] + assert "--generated-srcs" in txt.splitlines()[0], \ + "the AUTO-GENERATED line must name the flag that regenerates the file" + # Structural: the emitter may not carry either list as data. A capture is + # exactly what a long run of path literals looks like, so count them -- the + # emitter legitimately WRITES the string "LIBWEB_GENERATED_SRCS = [" into + # its output, which is why the assignment itself cannot be the signal. + code = _read("Meta/emit_libweb_bazel.py") + code = re.sub(r'""".*?"""', "", code, flags=re.S) + code = "\n".join(l for l in code.splitlines() + if not l.strip().startswith("#")) + paths = re.findall(r"['\"]([A-Za-z][\w/-]*/[\w/-]+\.(?:cpp|h|inc))['\"]", code) + assert len(paths) < 10, \ + "the emitter carries %d generated-file paths as data: %s" % ( + len(paths), sorted(set(paths))[:12]) + # The two lists in the checked-in file are sorted and disjoint by extension, + # which is what lets a reviewer diff two pins' files usefully. + lists = _lists(txt) + for name, entries in lists.items(): + assert entries == sorted(entries), "%s is not sorted" % name + assert len(entries) == len(set(entries)), "%s has duplicates" % name + assert all(s.endswith(".cpp") for s in lists["LIBWEB_GENERATED_SRCS"]) + mod, _ = _emit() + assert all(h.endswith(tuple(mod.HDR_EXTS)) + for h in lists["LIBWEB_GENERATED_HDRS"]) + + +def test_every_generated_hdr_is_excluded_from_the_source_glob(): + """Why a wrong entry in HDRS is not merely a missing declaration. + + The cc_library globs `**/*.h` with `exclude = LIBWEB_GENERATED_HDRS`, so the + list decides, per header, whether the SOURCE tree's copy or the genrule's + output wins. A generated header missing from the list means a consumer can + compile against a stale checked-in copy instead of the generated one (silent); + a source header wrongly IN it means the header vanishes from the library + (loud, but only for whoever includes it). This pins the exclude= wiring so + the list keeps that meaning. + """ + build = _read("Libraries/LibWeb/BUILD.bazel") + assert "exclude = LIBWEB_GENERATED_HDRS" in build + assert "+ LIBWEB_GENERATED_HDRS," in build + # No generated header may ALSO be checked in under the same path: if one is, + # the exclude= silently changes which file the compiler sees, and that is a + # thing to know about rather than to resolve by glob precedence. + hdrs = _lists(open(_GENERATED_SRCS).read())["LIBWEB_GENERATED_HDRS"] + both = [h for h in hdrs + if os.path.exists(os.path.join(_WS, "Libraries/LibWeb", h))] + assert not both, "checked in AND generated: %s" % both + + +def test_every_file_claiming_to_be_generated_names_a_command_that_makes_it(): + """The general form of this session's bug, applied to all 15 such files. + + `generated_srcs.bzl` was not special. It carried a first line saying + AUTO-GENERATED by an emitter, and the emitter had no code path that wrote + it -- so the claim was load-bearing misinformation for every reader and + every repin, and the cost was found only when the file's content went stale + under an upstream change. + + The claim is checkable, cheaply, for the whole tree at once. A file that says + a script generates it must name a script that EXISTS and, if the header names + a flag, the script must parse that flag. This is deliberately not a + round-trip test (that needs a CMake reference build, which the suite does not + have); it is the weaker check that would nonetheless have caught the bug -- + the emitter named there could not have produced the file under any argument. + """ + generated = [] + for dirpath, dirnames, filenames in os.walk(_WS): + dirnames[:] = [d for d in dirnames + if d not in ("__pycache__", "Build", ".git")] + for fn in filenames: + if not fn.endswith((".bzl", ".bazel", ".tsv")): + continue + path = os.path.join(dirpath, fn) + with open(path, errors="replace") as f: + head = "".join(f.readline() for _ in range(3)) + m = re.search(r'GENERATED by ([\w/]*?([\w-]+\.py))((?: --?[\w-]+)*)', + head) + if m: + generated.append((os.path.relpath(path, _WS), m.group(2), + m.group(3).split())) + # If this finds nothing the test is vacuous, which is the failure mode + # tests/run_all.py exists to reject -- so say so here too. + assert len(generated) >= 10, generated + + for rel, script, flags in generated: + cand = [os.path.join(_WS, "Meta", script), os.path.join(_WS, script)] + found = next((p for p in cand if os.path.exists(p)), None) + assert found, "%s says it is generated by %s, which does not exist" % ( + rel, script) + with open(found, errors="replace") as f: + src = f.read() + for flag in flags: + assert flag in src, \ + "%s says `%s %s` generates it, but that script never parses %s" \ + % (rel, script, " ".join(flags), flag) + # A generator that writes nothing cannot have produced the file. This is + # the assertion that fails for the original bug: emit_libweb_bazel.py had + # no branch writing generated_srcs.bzl at all. + assert re.search(r'\b(print|sys\.stdout\.write|\.write\()', src), \ + "%s names %s, which writes no output" % (rel, script) + + +def test_the_global_facts_are_imported_from_one_place_not_restated(): + """Two copies of one global fact, and they had already diverged. + + GLOBAL_DEFINES and SYSTEM_LIBS describe the BUILD, not a target: a define is + global because .bazelrc sets it for every TU, and a lib is a "system lib" + because no vcpkg port supplies it. Both emitters carried their own copy, and + by the time this was noticed they disagreed -- the 71fb301a repin added + glib/gio/gobject/xkbcommon to emit_build_bazel's SYSTEM_LIBS (upstream's new + pkg_check_modules(GIO) in UI/Qt) and this file's copy still had the old four. + + Latent, because LibWeb happens not to depend on glib. The failure it was + holding is an UNKNOWN dep -- i.e. a dropped link input -- and "happens not + to" is exactly the kind of claim this whole session was spent disproving. So + the second copy is an import, and this asserts it stays one. + """ + src = _read("Meta/emit_libweb_bazel.py") + for name in ("GLOBAL_DEFINES", "SYSTEM_LIBS"): + m = re.search(r"^%s = (.*)$" % name, src, re.M) + assert m, name + assert m.group(1).startswith("emit_build_bazel."), \ + "%s is restated in emit_libweb_bazel instead of imported: %s" % ( + name, m.group(1)) + # And they really are the same objects at run time, not just textually. + mod, _ = _emit() + import sys + ebb = sys.modules["emit_build_bazel"] + assert mod.SYSTEM_LIBS is ebb.SYSTEM_LIBS + assert mod.GLOBAL_DEFINES is ebb.GLOBAL_DEFINES + # The union has to contain what the repin added, or the import is pointing at + # a copy that is itself stale. + for lib in ("gio-2.0", "gobject-2.0", "glib-2.0", "xkbcommon"): + assert lib in mod.SYSTEM_LIBS, lib diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index 6147733..c6eb9ad 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -26,6 +26,7 @@ import importlib.util import os import re +import subprocess import tempfile _EMIT = os.path.join( @@ -176,12 +177,19 @@ def test_emitting_from_the_capture_alone_needs_no_vcpkg(): capture_output=True, text=True, env=env) assert ok.returncode == 0, ok.stderr assert "no local vcpkg checkout" in ok.stderr - # The real pin: 76 distfiles, every one keyed by its own sha512. + # The real pin: 76 captured distfiles + the ONE vcpkg host tool the capture + # could not see (finding 38). It is 77, not 78, and the arithmetic is the + # finding in miniature: cmake is in BOTH the capture and the tool pin, because + # the capturing machine's cmake was the wrong version and got downloaded, while + # its ninja was exactly right and did not. Union by sha, so cmake counts once. shas = re.findall(r"^ '([0-9a-f]{128})':", ok.stdout, re.M) - assert len(shas) == 76, len(shas) - assert len(set(shas)) == 76 + assert len(shas) == 77, len(shas) + assert len(set(shas)) == 77 repos = re.findall(r"@(vcpkg_[A-Za-z0-9_]+)//file:", ok.stdout) - assert len(set(repos)) == 76, "repo names must be unique per distfile" + assert len(set(repos)) == 77, "repo names must be unique per distfile" + # ...and the tool pins must survive the no-vcpkg path, which is the whole + # reason they are committed as a TSV rather than derived at emit time. + assert "ninja-linux-1.13.2.zip" in ok.stdout bad = subprocess.run(["python3", _EMIT, "--index"], capture_output=True, text=True, env=env) @@ -211,3 +219,761 @@ def test_non_mirrored_urls_are_left_as_a_single_entry(): mirror and must not acquire a fabricated one.""" u = "https://github.com/madler/zlib/archive/v1.3.1.tar.gz" assert emit.urls_for(u) == [u] + +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. + + +# -------------------------------------------------------------------------- +# vcpkg's OWN host tools (finding 38) +# +# The capture cannot see a tool the capturing machine already has: +# vcpkg_find_acquire_program probes the host first, so my /usr/bin/ninja at +# exactly the required 1.13.2 kept ninja out of the pin entirely, and a machine +# without it got `distfile MISSING FROM INDEX ... ninja-linux.zip` followed by +# x-block-origin correctly refusing the network. cmake was in the pin only by +# luck (host 4.2.3 vs required 4.4.0). +# +# The fix reads vcpkg's own tool metadata instead of the capture, so what gets +# pinned no longer depends on what happens to be installed anywhere. These tests +# pin that property, the filename vcpkg will look for, and the scoping. + +TOOLS_JSON = """{ + "tools": [ + {"name": "ninja", "os": "linux", "arch": "x64", "version": "1.13.2", + "url": "https://example.test/ninja-linux.zip", "sha512": "%s", + "archive": "ninja-linux-1.13.2.zip"}, + {"name": "ninja", "os": "windows", "arch": "x64", "version": "1.13.2", + "url": "https://example.test/ninja-win.zip", "sha512": "%s"}, + {"name": "cmake", "os": "linux", "arch": "x64", "version": "4.4.0", + "url": "https://example.test/cmake-4.4.0-linux-x86_64.tar.gz", "sha512": "%s", + "archive": "cmake-4.4.0-linux-x86_64.tar.gz"}, + {"name": "node", "os": "linux", "arch": "x64", "version": "24.18.0", + "url": "https://example.test/node.tar.gz", "sha512": "%s"}, + {"name": "git", "os": "linux", "arch": "x64", "version": "2.7.4"} + ] +}""" % ("a" * 128, "b" * 128, "c" * 128, "d" * 128) + + +def _vcpkg_with_tools(tmpdir, body=TOOLS_JSON): + scripts = os.path.join(tmpdir, "scripts") + os.makedirs(scripts, exist_ok=True) + with open(os.path.join(scripts, "vcpkg-tools.json"), "w") as f: + f.write(body) + return tmpdir + + +def test_tool_pins_come_from_vcpkg_metadata_not_the_capture(): + """The whole point: the pin must not depend on what is installed locally.""" + with tempfile.TemporaryDirectory() as d: + tools = emit.tool_distfiles(vcpkg=_vcpkg_with_tools(d)) + assert set(tools) == {"a" * 128, "c" * 128}, \ + "expected exactly the linux/x64 cmake+ninja pins, got %r" % (list(tools),) + + +def test_tool_pin_uses_the_archive_name_vcpkg_will_look_for(): + """The asset script is asked for vcpkg's OWN filename, not the URL basename. + + ninja's URL basename is ninja-linux.zip but vcpkg stores (and looks for) + ninja-linux-1.13.2.zip. Emitting the URL basename would put a row in the index + under a name vcpkg never asks about -- a pin that looks present and is not. + """ + with tempfile.TemporaryDirectory() as d: + tools = emit.tool_distfiles(vcpkg=_vcpkg_with_tools(d)) + assert tools["a" * 128][1] == "ninja-linux-1.13.2.zip" + + +def test_tool_pins_are_scoped_to_what_this_build_can_invoke(): + """node/dotnet/powershell are ~400MB no port in this closure ever runs.""" + assert "node" not in emit.BUILD_TOOLS + assert set(emit.BUILD_TOOLS) == {"cmake", "ninja"} + with tempfile.TemporaryDirectory() as d: + vc = _vcpkg_with_tools(d) + names = {r[2] for r in emit.tool_distfiles(vcpkg=vc).values()} + everything = {r[2] for r in emit.tool_distfiles(want=None, vcpkg=vc).values()} + assert "vcpkg-tool:node" not in names + assert "vcpkg-tool:node" in everything, "want=None must mean every tool" + + +def test_a_tool_with_no_url_is_skipped(): + """`git` has no url: vcpkg expects it from the system, so there is nothing to pin.""" + with tempfile.TemporaryDirectory() as d: + rows = emit.tool_distfiles(want=None, vcpkg=_vcpkg_with_tools(d)) + assert not any(r[2] == "vcpkg-tool:git" for r in rows.values()) + + +def test_missing_tool_metadata_is_an_error_not_an_empty_pin(): + """Silently emitting no tool pins is what broke a clone: fail loudly instead.""" + with tempfile.TemporaryDirectory() as d: + try: + emit.tool_distfiles(vcpkg=d) # exists, but has no vcpkg-tools.json + except emit.VcpkgUnavailable as e: + assert "vcpkg-tools.json" in str(e) + else: + raise AssertionError("expected VcpkgUnavailable") + + +def test_the_committed_pin_carries_ninja_and_cmake(): + """The regression test for the actual bug report, against the committed files. + + Ulf's clone failed at `Detecting compiler hash` because ninja was absent from + the index; cmake was present. Both must be there now, and the index is what + the asset script resolves through, so check the index -- not just the + http_file list. + """ + ws = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + index = open(os.path.join(ws, "vcpkg_index.bzl")).read() + for tool in ("ninja-linux-1.13.2.zip", "cmake-4.4.0-linux-x86_64.tar.gz"): + assert tool in index, "%s is missing from the distfile index" % tool + distfiles = open(os.path.join(ws, "vcpkg_distfiles.bzl")).read() + assert "ninja-build/ninja/releases/download/v1.13.2/ninja-linux.zip" in distfiles + + # ...and bzlmod only creates a repo the MODULE names, so an http_file with no + # use_repo entry is invisible: that asymmetry is why the emitter emits the + # use_repo list too. + module = open(os.path.join(ws, "MODULE.bazel")).read() + names = re.findall(r"name = '(vcpkg_ninja[A-Za-z0-9_]*)'", distfiles) + assert names, "no ninja http_file found" + for n in names: + assert "'%s'" % n in module, "%s is fetched but not named in use_repo" % n + + +# --- host prerequisites (finding 39) --------------------------------------- +# +# The class of input that CANNOT be pinned: tools vcpkg has no Linux download +# for. What is tested here is therefore not "the pin is complete" but "the gap is +# named accurately and early" -- and above all that it does not cry wolf, because +# a preflight with false positives demands packages nothing needs and gets +# deleted by the third person who hits it. + +def _acquire(tmpdir, program, body): + d = os.path.join(tmpdir, "scripts", "cmake") + os.makedirs(d, exist_ok=True) + with open(os.path.join(d, "vcpkg_find_acquire_program(%s).cmake" % program), + "w") as f: + f.write(body) + + +def _port(tmpdir, port, body, filename="portfile.cmake"): + d = os.path.join(tmpdir, "ports", port, os.path.dirname(filename)) + os.makedirs(d, exist_ok=True) + with open(os.path.join(tmpdir, "ports", port, filename), "w") as f: + f.write(body) + + +NASM_LIKE = """set(program_name nasm) +set(program_version 3.01) +set(brew_package_name "nasm") +set(apt_package_name "nasm") +if(CMAKE_HOST_WIN32) + set(download_urls "https://example.com/nasm-win64.zip") + set(download_filename "nasm-win64.zip") + set(download_sha512 %s) +endif() +""" % ("a" * 128) + + +def test_a_windows_only_download_is_a_host_prerequisite_on_linux(): + """The nasm bug: URLs exist, but only inside if(CMAKE_HOST_WIN32). + + Reading `download_urls` without asking WHICH BRANCH it is in reports nasm as + a pinnable download, which is how six ports' worth of hard dependency stayed + invisible until libvpx died on it 20 minutes into a build. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "NASM", NASM_LIKE) + _port(d, "libvpx", "vcpkg_find_acquire_program(NASM)\n") + reqs = emit.host_tool_requirements(ports=["libvpx"], vcpkg=d) + assert "nasm" in reqs, reqs + binary, apt, users, _alts = reqs["nasm"] + assert (binary, apt, users) == ("nasm", "nasm", ["libvpx"]) + + +def test_a_tool_with_a_real_linux_download_is_not_a_prerequisite(): + """meson/gn ARE downloadable on Linux -- demanding them from apt is wrong.""" + with tempfile.TemporaryDirectory() as d: + _acquire(d, "MESON", 'set(program_name meson)\n' + 'set(apt_package_name "meson")\n' + 'set(download_urls "https://example.com/meson.tar.gz")\n') + _port(d, "vcpkg-tool-meson", "vcpkg_find_acquire_program(MESON)\n") + reqs = emit.host_tool_requirements(ports=["vcpkg-tool-meson"], vcpkg=d) + assert reqs == {}, "a downloadable tool must not be a host prerequisite: %r" % reqs + + +def test_a_vcpkg_fetch_tool_is_not_a_prerequisite(): + """NINJA sets no download_urls: it delegates to `vcpkg fetch`, which IS pinned. + + Without this, the fix for finding 38 (pin ninja) and the fix for finding 39 + (name what cannot be pinned) contradict each other about ninja. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "NINJA", "z_use_vcpkg_fetch(NINJA)\n") + _port(d, "vcpkg-cmake", "vcpkg_find_acquire_program(NINJA)\n") + reqs = emit.host_tool_requirements(ports=["vcpkg-cmake"], vcpkg=d) + assert reqs == {}, "ninja comes from the tools.json pin, not the host: %r" % reqs + + +def test_a_windows_only_call_site_is_not_a_linux_prerequisite(): + """openssl and vcpkg-make ask for CLANG only under MSVC -- clang is not needed. + + The first version of this derivation reported clang, which is a false alarm + on every Linux machine, and a preflight that demands a 2GB toolchain nobody + uses is one that gets ignored. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "CLANG", 'set(program_name clang)\n' + 'set(apt_package_name "clang")\n') + _port(d, "vcpkg-make", """ +if(VCPKG_DETECTED_CMAKE_ASM_COMPILER_ID STREQUAL "MSVC") + vcpkg_find_acquire_program(CLANG) +endif() +""") + reqs = emit.host_tool_requirements(ports=["vcpkg-make"], vcpkg=d) + assert reqs == {}, "an MSVC-only call site is not a Linux prerequisite: %r" % reqs + + +def test_a_windows_only_subdirectory_is_not_scanned(): + """openssl splits by FILE: windows/portfile.cmake asks for CLANG at top level. + + No if() guards it -- the guard is the include() in the parent -- so only the + path says it is Windows-only. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "CLANG", "set(program_name clang)\n") + _acquire(d, "PERL", 'set(program_name perl)\nset(apt_package_name "perl")\n') + _port(d, "openssl", "vcpkg_find_acquire_program(CLANG)\n", + filename="windows/portfile.cmake") + _port(d, "openssl", "vcpkg_find_acquire_program(PERL)\n", + filename="unix/portfile.cmake") + reqs = emit.host_tool_requirements(ports=["openssl"], vcpkg=d) + assert set(reqs) == {"perl"}, \ + "windows/ must be skipped and unix/ must not be: %r" % reqs + + +def test_the_else_of_a_negated_windows_test_is_the_windows_branch(): + """if(NOT VCPKG_TARGET_IS_WINDOWS) ... else() <-- that else is Windows. + + dav1d is written exactly this way. Treating every else() as reachable puts + the Windows-only GASPREPROCESSOR into a Linux preflight. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "GASPREPROCESSOR", "set(program_name gas-preprocessor.pl)\n") + _acquire(d, "NASM", NASM_LIKE) + _port(d, "dav1d", """ +if(NOT VCPKG_TARGET_IS_WINDOWS) + vcpkg_find_acquire_program(NASM) +else() + vcpkg_find_acquire_program(GASPREPROCESSOR) +endif() +""") + reqs = emit.host_tool_requirements(ports=["dav1d"], vcpkg=d) + assert set(reqs) == {"nasm"}, reqs + + +GPERF_LIKE = """ +function(vcpkg_run_autoreconf shell_cmd work_dir) + find_program(ACLOCAL NAMES aclocal) + find_program(AUTORECONF NAMES autoreconf) + find_program(LIBTOOLIZE NAMES libtoolize glibtoolize) + if(missing) + message(FATAL_ERROR "${PORT} currently requires the following programs from the system package manager: + autoconf autoconf-archive automake libtoolize + + On Debian and Ubuntu derivatives: + sudo apt install autoconf autoconf-archive automake libtool +") + endif() +endfunction() +""" + + +def test_a_fatal_error_naming_apt_packages_is_a_prerequisite_too(): + """The SECOND mechanism, found after the first shipped. + + vcpkg-make never calls vcpkg_find_acquire_program for autotools: it uses bare + find_program and raises FATAL_ERROR with an apt line. Enumerating only + acquire-program calls misses all of it -- which is how gperf failed on + autoconf AFTER nasm was fixed. + """ + with tempfile.TemporaryDirectory() as d: + _port(d, "vcpkg-make", GPERF_LIKE, filename="vcpkg_make.cmake") + reqs = emit.host_tool_requirements(ports=["vcpkg-make"], vcpkg=d) + assert set(reqs) == {"autoconf", "autoconf-archive", "automake", "libtool"}, reqs + # libtoolize OR glibtoolize satisfies libtool: alternatives, not two requirements. + binary, apt, _users, alts = reqs["libtool"] + assert binary == "libtoolize" and apt == "libtool" and alts == ["glibtoolize"] + # autoconf-archive is m4 macros: no binary exists to probe for, and claiming + # otherwise would report it satisfied on a machine that lacks it. + assert reqs["autoconf-archive"][0] == "", reqs["autoconf-archive"] + + +def test_a_warning_about_system_packages_is_not_a_prerequisite(): + """angle WARNS about mesa-common-dev and separately FATAL_ERRORs on arch. + + A file-level 'does it contain FATAL_ERROR and an apt line' check staples the + two together and demands mesa-common-dev, which no build step here requires. + Advice is not a requirement. + """ + with tempfile.TemporaryDirectory() as d: + _port(d, "angle", """ +if (VCPKG_TARGET_IS_LINUX) + message(WARNING "${PORT} currently requires the following libraries from the system package manager:\\n mesa-common-dev\\n\\nThese can be installed via apt-get install mesa-common-dev.") +endif() +message(FATAL_ERROR "Unsupported architecture: ${VCPKG_TARGET_ARCHITECTURE}") +""") + reqs = emit.host_tool_requirements(ports=["angle"], vcpkg=d) + assert reqs == {}, "a WARNING is advice, not a prerequisite: %r" % reqs + + +def test_the_committed_host_tool_list_names_nasm_and_autotools(): + """Regression test for both bug reports, against the committed file.""" + rows = emit.load_host_tools() + assert rows, "Meta/vcpkg_host_tools.tsv is missing or empty" + by_apt = {apt: (names, users) for names, apt, users in rows} + # Ulf's first failure: libvpx, ~20 minutes in. + assert "nasm" in by_apt, list(by_apt) + assert by_apt["nasm"][0] == ["nasm"] + assert "libvpx" in by_apt["nasm"][1] + # Ulf's second failure: gperf, via vcpkg-make. + for apt in ("autoconf", "automake", "libtool", "autoconf-archive"): + assert apt in by_apt, "%s missing from the host tool list" % apt + # ...and clang must NOT be there (MSVC-only call sites). + assert "clang" not in by_apt, "clang is an MSVC-only requirement" + + +def test_the_preflight_reports_every_missing_tool_at_once(): + """The actual complaint: one tool per 20-minute build. + + The preflight's value is entirely in reporting the WHOLE set in one run, so + this checks the shape of the output, not just the exit code. + """ + ws = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + script = open(os.path.join(ws, "Meta", "vcpkg_build.sh")).read() + body = script.split("# --- host prerequisites")[1].split( + "# --- a writable vcpkg root")[0] + # $HOST_TOOLS arrives as the driver's 7th argument (vcpkg.bzl names the file + # as a declared input and passes its path); set it the same way here rather + # than editing the extracted body, so the test exercises the real contract. + body = "HOST_TOOLS=%s\n%s" % ( + os.path.join(ws, "Meta", "vcpkg_host_tools.tsv"), body) + with tempfile.TemporaryDirectory() as d: + # A PATH with everything EXCEPT nasm and libtoolize. Empty stubs are + # enough: the preflight uses `command -v`, and running the real tools is + # not the point. + for tool in ("perl", "python3", "pkg-config", "aclocal", "autoreconf"): + p = os.path.join(d, tool) + open(p, "w").close() + os.chmod(p, 0o755) + r = subprocess.run(["/bin/bash", "-c", body], capture_output=True, + text=True, env={"PATH": d}) + assert r.returncode == 1, "expected a hard failure, got %d\n%s" % ( + r.returncode, r.stderr) + # BOTH, from one run -- that is the whole feature. + assert "nasm" in r.stderr and "libtoolize" in r.stderr, r.stderr + assert "needed by: dav1d" in r.stderr, r.stderr + # One pasteable line, and the tools that ARE present must not be in it. + apt_line = [l for l in r.stderr.splitlines() if "sudo apt install" in l] + assert len(apt_line) == 1, r.stderr + assert "nasm" in apt_line[0] and "libtool" in apt_line[0] + assert "perl" not in apt_line[0], "must not demand a tool that is present" + + +def test_a_moved_pin_is_reported_as_a_stale_capture_not_a_windows_only_fetch(): + """The capture wins -- so when it is WRONG, the message has to say which. + + Under --assets the capture REPLACES the static portfile parse, deliberately: + a portfile is a CMake program, so the static regex cannot see through its + platform branches, and every row the static parse had and the capture did not + turned out to be a Windows-only fetch. That reasoning was checked once, on + three rows (libiconv, pthreads4w, dirent, all behind + `if(VCPKG_TARGET_IS_WINDOWS)`), and then FROZEN INTO THE MESSAGE: the emitter + printed every casualty as an entry "vcpkg never asked for on this platform". + + Ladybird's 71fb301a repin is where that came due. vcpkg.json pins sdl3 + 3.2.28 (the reference build agrees: vcpkg_installed/vcpkg/info/ holds + sdl3_3.2.28_x64-linux-dynamic.list) and the versions-db derivation resolves + it correctly. The capture predates the repin and holds release-3.4.12, so the + RIGHT answer was discarded in favour of a stale one -- and the diagnostic + asserted a reason that happened to be false, which is why nobody looked. The + checked-in vcpkg_distfiles.bzl still fetches 3.4.12. + + The distinction is available without any new input: a dropped row whose URL + FAMILY the capture also has is the same upstream project at a different + version. That is a stale capture. A row whose family the capture has never + seen at all is genuinely a fetch this platform does not do. + """ + cap = { + "a" * 128: ("https://github.com/libsdl-org/SDL/archive/release-3.4.12.tar.gz", + "libsdl-org-SDL-release-3.4.12.tar.gz", "captured", "capture"), + "b" * 128: ("https://sqlite.org/2026/sqlite-autoconf-3530300.tar.gz", + "sqlite-autoconf-3530300.tar.gz", "captured", "capture"), + } + derived = { + # THE case: same project, the version vcpkg.json actually pins. + "c" * 128: ("https://github.com/libsdl-org/SDL/archive/release-3.2.28.tar.gz", + "libsdl-org-SDL-release-3.2.28.tar.gz", "sdl3", "versions-db"), + # Genuinely platform-only: nothing from this upstream is in the capture. + "d" * 128: ("https://ftpmirror.gnu.org/gnu/libiconv/libiconv-1.19.tar.gz", + "libiconv-1.19.tar.gz", "libiconv", "versions-db"), + "e" * 128: ("https://github.com/tronkko/dirent/archive/1.26.tar.gz", + "tronkko-dirent-1.26.tar.gz", "dirent", "versions-db"), + # A row the capture HAS (by sha) is not a casualty at all. + "b" * 128: ("https://sqlite.org/2026/sqlite-autoconf-3530300.tar.gz", + "sqlite-autoconf-3530300.tar.gz", "sqlite3", "versions-db"), + } + platform_only, stale = emit.classify_static_only(derived, cap) + assert stale == [("c" * 128, "a" * 128)], stale + assert sorted(platform_only) == ["d" * 128, "e" * 128], platform_only + + # And the emitter SAYS it, in those words, on the real tree's data. Run as a + # subprocess so this exercises the message and not just the classifier. + import subprocess + ws = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + tsv = os.path.join(ws, "Meta", "vcpkg_assets.tsv") + r = subprocess.run(["python3", _EMIT, "--assets", tsv, "--index"], + capture_output=True, text=True, + env=dict(os.environ, LADYBIRD_ROOT=tempfile.mkdtemp(), + VCPKG_ROOT=tempfile.mkdtemp())) + assert r.returncode == 0, r.stderr + # With no vcpkg there is no derivation to compare against, so neither + # category can appear -- the message must not be printed speculatively. + assert "STALE CAPTURE" not in r.stderr, r.stderr + assert "never asked for on this platform" not in r.stderr, r.stderr + + # Structurally: the old wording claimed the reason for EVERY dropped row. + # Whatever the emitter says about platform-only entries, it may only say it + # about rows that survived the classification. + with open(_EMIT) as f: + src = f.read() + body = src.split("def main(", 1)[1] + assert "classify_static_only(" in body, \ + "main() must classify the dropped rows, not assert a reason for them" + assert "static_only = sorted(set(distfiles) - set(cap))" not in body, \ + "the unclassified set-difference is back" + + +def _capture_script(): + ws = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + with open(os.path.join(ws, "Meta", "vcpkg_capture_assets.sh")) as f: + return f.read() + + +def test_a_failed_download_makes_the_capture_fail_because_vcpkg_will_not(): + """vcpkg exits 0 on a capture that lost downloads. Ask me how I know. + + The 71fb301a re-capture printed "All requested installations completed + successfully in: 49 min", exited 0, and wrote 72 rows -- while angle's + `gni-to-cmake.py` had FAILED to download (a transient TLS error: this + sandbox's clock was briefly behind the certificate's validity window, + "certificate is not yet valid"). Diffing against the committed capture showed + 72 rows where the old pin had 76: FIVE URLs gone, four of them WebKit files + nothing had reported on at all. + + That is the mechanism worth remembering: a failed download HALTS its portfile, + so the four `vcpkg_download_distfile` calls after it in angle were never made, + never requested, and so never captured. The recorder logs each tuple BEFORE + fetching, so the one that failed is present and the four that were never asked + for are simply absent -- a pin that is missing five URLs and looks complete. + Emitting from it would produce rules that fetch nothing for angle, failing + much later inside a port build. + + So the capture must judge itself, and it cannot do that from vcpkg's exit + code. Verified against a fake vcpkg that calls the recorder with an + unreachable URL and then exits 0 like the real one: the script exits 1 and + names the URL. + """ + sh = _capture_script() + # The recorder must report failures to the DRIVER. A variable cannot: the + # recorder is a separate process per download. + rec = sh.split("cat > \"$REC\"", 1)[1].split("chmod +x", 1)[0] + assert re.search(r'printf .*>> "\$FAILED"', rec), \ + "the recorder does not report a failed download to the driver" + # And the driver must refuse to bless the result. + tail = sh.split("chmod +x", 1)[1] + assert 'if [ -s "$FAILED" ]' in tail, \ + "the driver never checks whether any download failed" + block = tail.split('if [ -s "$FAILED" ]', 1)[1].split("\nfi", 1)[0] + assert "exit 1" in block, "a capture with failed downloads still exits 0" + # The message has to say the non-obvious part, or a reader re-runs the emitter + # on the incomplete file. + assert re.search(r"halt the rest of their portfile|HALTS its portfile", tail), \ + "the message does not explain that later downloads in that port are missing" + assert "never" in tail and "REQUESTED" in tail and "missing rows" in tail, \ + "the message does not say the capture is missing rows" + + +def test_the_capture_bounds_stalls_not_transfer_size(): + """`--max-time` capped how long a download may legitimately TAKE. + + It killed OpenGL-Registry at 22MB of a perfectly healthy transfer, reported it + as "FAILED to fetch", fell through to the origin, and did it again on every + re-run -- a capture that could not finish, blaming the mirror. The property + actually wanted is "no progress for a while", which cannot mistake a big file + for a dead one. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert "--max-time" not in code, \ + ("--max-time bounds total transfer time, so it fails big-but-healthy " + "downloads; use --speed-time/--speed-limit") + assert "--speed-time" in code and "--speed-limit" in code, \ + "no stall timeout at all: a dead mirror hangs the capture forever" + + +def test_the_capture_is_resumable(): + """A ~50-minute network-bound job that truncates its output on start. + + Every interruption -- sandbox restart, dead mirror, timeout -- cost the whole + run (todo c2affe6b). Appending plus the final dedupe makes a re-run resume + instead: a tuple recorded twice is free. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert ': > "$OUT"' not in code, "the capture truncates its own output on start" + assert 'touch "$OUT"' in code, "the capture does not append to an existing run" + assert re.search(r"sort -u -t\$'\\t' -k1,2 -o \"\$OUT\" \"\$OUT\"", code), \ + "without the dedupe, appending duplicates rows" + + +def test_download_only_mode_cannot_produce_a_complete_capture(): + """`--only-downloads` was the DEFAULT, with a comment claiming it sufficed. + + The claim ("it is enough because the asset hook fires during resolution") was + invented, not measured -- the fourth instance this repin of generated prose + restating a hand-copied fact. What actually happens: Download Mode makes vcpkg + refuse to EXECUTE anything, and a portfile that stops executing stops + downloading. angle downloads gni-to-cmake.py (portfile.cmake:79), sets up a + python venv to run it (:86, x_vcpkg_get_python_packages), and only THEN + downloads four WebKit files (:123 :129 :144 :151). In Download Mode :86 halts, + so those four URLs are unreachable by construction -- measured: the halt is at + `angle/portfile.cmake:86` in the download-only re-capture, which wrote 72 rows + where the committed capture has 76. + + So the committed 76-row capture cannot have been made in that mode, and the + fast mode must not be the default. It stays available (opt-in) for refreshing + known-reachable URLs, and the halt check refuses to bless its output. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + # Opt-in, not default: the flag may only be passed under the env switch. + assert "--only-downloads" in code, "the fast mode is gone entirely" + for line in code.splitlines(): + if "--only-downloads" in line: + break + guard = code.split("--only-downloads")[0].splitlines()[-6:] + assert any("CAPTURE_ONLY_DOWNLOADS" in l for l in guard), \ + ("--only-downloads is passed unconditionally again; it CANNOT reach the " + "downloads behind an executed portfile step (angle's four WebKit files)") + + +def test_a_halted_portfile_fails_the_capture_and_names_the_port(): + """The other way a capture silently loses URLs, with no failed download. + + A halt loses every download after it, exactly like a failed fetch -- and vcpkg + exits 0 for it too ("Downloaded sources for angle", then "All requested + installations completed successfully"). There is no exit code and no + asset-script callback for a step never reached, so the only witness is vcpkg's + stdout; hence the tee into a log the driver greps. + + It must name the PORT. The halt message itself names a shared helper + (vcpkg_execute_required_process.cmake:23, identical for every port) and the + portfile in the call stack is a versioned path under bt/versioning_ -- neither + is actionable. The port comes from the "Installing N/M :@" + line above the halt. + + Verified against the fake vcpkg that prints a halt and exits 0: the script + exits 1 with "port angle halted before finishing its portfile". + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert "tee" in code and "VCPKG_LOG" in code, \ + "vcpkg's output is not kept, so a halt cannot be detected at all" + assert re.search(r"halting portfile", code, re.I), \ + "nothing looks for a halted portfile" + assert 'tolower($0)' in code, \ + ("the halt must be matched case-insensitively: vcpkg spells it both " + "'Halting portfile execution.' and 'Download failed, halting portfile.'") + assert re.search(r"Installing \[0-9\]\+", code), \ + "the halt is not attributed to a port via the 'Installing N/M ' line" + # The halt has to land in the same sentinel the exit-1 check reads. + halt = code.split("halting portfile")[1] + assert '>> "$FAILED"' in halt.split("rm -f")[0], \ + "a detected halt does not reach $FAILED, so the capture still exits 0" + + +def test_the_capture_does_not_let_tee_swallow_a_vcpkg_failure(): + """Piping vcpkg into tee moves the pipeline's exit status to tee's. + + Added when the log was introduced: without pipefail, `vcpkg install | tee log` + reports tee's success and a hard vcpkg failure becomes a capture that "worked". + Verified with a fake vcpkg that exits 3 -- the script exits 3. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert re.search(r"^set -euo pipefail", code, re.M) or "PIPESTATUS" in code, \ + "vcpkg's output is piped into tee with neither pipefail nor a PIPESTATUS check" + + +def test_the_capture_turns_the_binary_cache_off(): + """The third and worst way this script lost rows: a port that never ran. + + vcpkg's binary cache is keyed by each port's ABI hash; on a hit it unpacks the + archive and does NOT run the portfile, so the port requests none of its + downloads. Unlike a failed fetch or a halt this is completely silent: no error, + no halt, exit 0, "All requested installations completed successfully". + + Measured with a zlib-only manifest against a warm ~/.cache/vcpkg/archives: the + first run built zlib and captured 3 rows; the second, with a fresh install root, + printed "Restored 3 package(s)" and captured ZERO while exiting 0. The capture + had no --binarysource at all, so it inherited whatever cache the machine + happened to have -- enough on its own to explain a re-capture that cannot + reproduce the committed rows. vcpkg_build.sh already passed + --binarysource=clear for the adjacent reason (a restore proves nothing about + building from source); the capture never did. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert "--binarysource=clear" in code, \ + ("the capture does not disable the binary cache: a restored port skips its " + "portfile and contributes no rows, silently") + # Before "$@", so a caller can still override deliberately -- and the outcome + # check below is what catches them if they do. + install = code.split("/vcpkg\" install", 1)[1].split("\n\n", 1)[0] + assert install.index("--binarysource=clear") < install.index('"$@"'), \ + "--binarysource=clear after \"$@\" would override the caller instead of defaulting" + + +def test_a_port_that_skipped_its_portfile_fails_the_capture(): + """Belt and braces for the flag above: check the OUTCOME, not the intent. + + Two lines in vcpkg's output mean a port contributed nothing: + "Restored N package(s) from " (cache hit) and "The following packages + are already installed" (a reused install root -- the 71fb301a capture's second + run had 7 of those). Both are invisible in the exit code, so both are read off + the teed log and land in $FAILED. + + Verified with fake vcpkgs printing each line and exiting 0: the script exits 1, + and the already-installed case reports the COUNT (2 of the 2 listed). + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert re.search(r'grep -q "\^Restored \[0-9\]\* package" "\$VCPKG_LOG"', code), \ + "a binary-cache restore is not detected, so an overridden --binarysource is silent" + assert "The following packages are already installed" in code, \ + "a reused install root is not detected: those ports' downloads are missing" + for marker in ("^Restored [0-9]* package", "already installed"): + seg = code.split(marker, 1)[1].split("\nif ")[0] + assert '>> "$FAILED"' in seg, \ + f"a skipped portfile detected via {marker!r} never reaches $FAILED" + + +def test_a_captured_row_the_manifest_does_not_pin_is_reported_as_a_leak(): + """classify_static_only looks one way, so a capture with too MANY rows is invisible. + + This bit within the hour. The 71fb301a re-capture needed a supplementary run for + angle alone -- the one port whose later downloads a Download Mode halt had eaten + -- and a one-port manifest resolves its dependencies from the vcpkg BASELINE, not + from Ladybird's vcpkg.json overrides. It pulled zlib 1.3.2 where Ladybird pins + 1.3.1, so the merged capture had both, and vcpkg_distfiles.bzl would have carried + an http_file for a distfile no port in the build fetches. + + I dropped the row by hand, then recognised the shape: a hand-fix nothing checks + is what the next capture silently repeats -- the thing this whole session has been + about. It is derivable, because the versions-db derivation knows the pinned + version: a captured row in the same URL family at a version the derivation does + not pin came from the wrong resolution. + + Verified on the real 77-row merged capture: exactly one leak, zlib 1.3.2 vs the + pinned 1.3.1, and the 76-row file it became is clean. + """ + cap = { + # The leak: same family as a derived row, different version. + "a" * 128: ("https://github.com/madler/zlib/archive/v1.3.2.tar.gz", + "madler-zlib-v1.3.2.tar.gz", "zlib", "capture"), + "b" * 128: ("https://github.com/madler/zlib/archive/v1.3.1.tar.gz", + "madler-zlib-v1.3.1.tar.gz", "zlib", "capture"), + # Captured but not derived AND no derived sibling: an expanded ${VAR} the + # static parse cannot see. NOT a leak -- this is why the capture exists. + "c" * 128: ("https://github.com/WebKit/WebKit/raw/0742/Source/cmake/DetectSSE2.cmake", + "DetectSSE2.cmake", "angle", "capture"), + } + derived = { + "b" * 128: ("https://github.com/madler/zlib/archive/v1.3.1.tar.gz", + "madler-zlib-v1.3.1.tar.gz", "zlib", "versions-db"), + } + leaked = emit.classify_capture_only(derived, cap) + assert leaked == [("a" * 128, "b" * 128)], leaked + + # The unexpanded-variable rows are the whole reason the capture replaces the + # static parse; reporting them as leaks would make the check useless. + assert not any(cap_sha == "c" * 128 for cap_sha, _ in leaked), \ + "a capture-only row with no derived sibling is not a leak, it is the point" + + # And main() must actually call it, or the classifier is decorative. + with open(_EMIT) as f: + body = f.read().split("def main(", 1)[1] + assert "classify_capture_only(" in body, "main() never checks for leaked rows" + assert "LEAKED CAPTURE ROWS" in body, "a leak is found but not reported" + + +def test_the_capture_requires_a_row_for_every_download_vcpkg_resolved(): + """The check that does not depend on my having enumerated the loss modes. + + Three separate ways of losing rows turned up in one session (a halt, a cache + hit, an already-installed port), each found by hitting it and each guarded + individually above. That is a losing pattern: the fourth was already waiting -- + "-- Using cached gni-to-cmake.py", i.e. a file already in --downloads-root makes + vcpkg skip the asset script entirely, so the recorder never sees it and there is + no row. Which means the RESUME rule (share downloads/ across runs, so a + 50-minute job restarts cheaply) is itself a way to produce an incomplete + capture. The two are in direct tension and the earlier comment claimed only the + upside. + + So compare against something the script does not control: vcpkg announces every + download it resolves, either as "Trying to download using asset cache + script" or "-- Using cached ". Require a captured row per announcement and + the enumeration stops mattering. + + Two subtleties, both derived rather than listed: + * an ABSOLUTE path in "-- Using cached /path/x.tar.gz" is vcpkg_from_git + pre-placing its own archive (libyuv, skia's two), which bypasses asset + caching by design and is pinned by vcpkg_git_archives.bzl instead. Relative + name = asset download, absolute = git. + * the comparison is on the {dst} basename, so it must undo the two manglings + the recorder is deliberately dumb about ("..part", and the 8-hex + disambiguator vcpkg splices in). + + Validated against all three real logs: the cold 72-row run and the warm resume + both come out complete, and the angle-only run reports its four already-cached + files as missing when checked against the angle-only tsv but clean against the + accumulated capture -- which is the correct answer in both cases. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert "Trying to download" in code and "Using cached" in code, \ + ("the capture does not compare itself against what vcpkg said it resolved, " + "so a download that was never REQUESTED is silently absent") + # The git case must be excluded by DERIVING it (absolute path), not by naming ports. + assert re.search(r"Using cached \\\(\[\^/\]", code), \ + ("-- Using cached with an absolute path is vcpkg_from_git bypassing the asset " + "cache; requiring a row for it would fail every capture") + for port in ("libyuv", "skia", "angle"): + assert port not in code, \ + f"{port} is hardcoded in the capture script; derive the git case instead" + # Both manglings must be undone or every row looks missing. + assert ".part" in code, "the ..part suffix is not stripped before comparing" + assert re.search(r"-\[0-9a-f\]\{8\}", code), \ + "vcpkg's 8-hex disambiguated name is not folded back before comparing" + # And the shortfall must reach the sentinel. + tail = code.split("Trying to download", 1)[1] + assert re.search(r'comm -23[\s\S]{0,400}?>> "\$FAILED"', tail), \ + "a download with no captured row never reaches $FAILED" diff --git a/tests/test_engine.py b/tests/test_engine.py index 4ab8f2d..43eff54 100644 --- a/tests/test_engine.py +++ b/tests/test_engine.py @@ -675,18 +675,10 @@ def test_nonparticipating_roles_are_excluded_not_diffed(): assert not any(d.kind == "missing_target" for d in discs) assert "Nightly" in res["excluded"]["cmake"]["dashboard"] - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn() - print(f"PASS {fn.__name__}") - except Exception: - failed += 1 - print(f"FAIL {fn.__name__}") - traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_extract_npm.py b/tests/test_extract_npm.py index 4397780..35a271d 100644 --- a/tests/test_extract_npm.py +++ b/tests/test_extract_npm.py @@ -210,15 +210,10 @@ def test_partial_nlast_line_does_not_break_extract(): m = extract_npm.extract(os.path.join(tmp, "a.ndjson"), "/repo") assert "out/a.js" in m.targets - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_extractors.py b/tests/test_extractors.py index d59cdb0..2fdf1d8 100644 --- a/tests/test_extractors.py +++ b/tests/test_extractors.py @@ -434,14 +434,50 @@ def test_ts_program_splits_into_per_file_tscompile_actions(): assert t.role.value == "production", t.role -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +def test_a_library_name_keeps_the_dots_that_are_part_of_it(): + """`libgio-2.0.so` is `gio-2.0`, not `gio-2`. + + _library_identity turns a link fragment into the abstract dep name a resolver + looks up (the -l name). It used to cut at the FIRST dot -- fine for libz.so + and libQt6Widgets.so.6.10.2, and wrong for glib, whose sonames carry the API + version in the NAME: libgio-2.0.so became `gio-2`, which names no library on + any system. Ladybird's 71fb301a pin is where that surfaced: upstream added a + pkg_check_modules(GIO) to UI/Qt, and the three glib deps arrived at the + Bazel emitter as unresolvable UNKNOWNs. Had the emitter guessed instead of + reporting, the -l flag would have failed at LINK time, far from the cause. + + The rule is "strip the extension", so the cases that matter are: a dot in + the name, a version suffix after the extension, both at once, and .dylib not + being read as a truncated .d. + """ + f = extract_cmake._library_identity + cases = { + "/usr/lib/x86_64-linux-gnu/libgio-2.0.so": "gio-2.0", + "/usr/lib/x86_64-linux-gnu/libglib-2.0.so": "glib-2.0", + "/usr/lib/x86_64-linux-gnu/libgobject-2.0.so": "gobject-2.0", + # A dotted name AND a version suffix: both cuts have to be right. + "/usr/lib/libgio-2.0.so.0.8200.4": "gio-2.0", + # The cases that already worked, pinned so the fix cannot regress them. + "/usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.10.2": "Qt6Widgets", + "vcpkg_installed/x64-linux-dynamic/lib/libcpptrace.so.1.0.2": "cpptrace", + "/usr/lib/libvulkan.so": "vulkan", + "/usr/lib/libfoo.a": "foo", + "-lz": "z", + "-framework Cocoa": "Cocoa", + # macOS: .dylib must not be matched as ".d" + "ylib". + "/x/libbar.dylib": "bar", + "/x/libbar.2.dylib": "bar.2", + # Not a library at all -- None, so the caller does not invent a dep. + "notalib.so": None, + "": None, + } + for frag, want in cases.items(): + assert f(frag) == want, "%r -> %r, want %r" % (frag, f(frag), want) + +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py new file mode 100644 index 0000000..95efb17 --- /dev/null +++ b/tests/test_fd_census.py @@ -0,0 +1,771 @@ +#!/usr/bin/env python3 +"""Tests for examples/ladybird/fd_census.py. + +The fd-leak instrument used to be a patch to Libraries/LibRequests/Request.cpp, +which is the wrong delivery mechanism for anyone who has their own commits: it +assumes a tree at our pinned commit, it CONFLICTS with a tree already carrying +upstream's fix, and it makes "run my diagnostic" mean "reset your tree". Ulf said +so directly. Everything the patched build computed is visible from outside the +process, so this is the same instrument reading /proc and `ss`. + +What is worth pinning is the parsing and the classification, because they are what +turn a number into a diagnosis: + + 1. `ss -np`'s peer-inode column: 0 means the far end is CLOSED. That single field + is the discriminator between the two per-request leaks, so its parsing is + tested against real `ss` output rather than trusted. + 2. the DEAD/ALIVE verdict, including the mixed case; + 3. ages: an fd younger than the threshold is in flight, not leaked -- the + distinction that stopped a freshly restarted process from looking like a fix. +""" + +import importlib.util +import os +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPT = REPO / "examples" / "ladybird" / "fd_census.py" + + +def _load(): + spec = importlib.util.spec_from_file_location("fd_census", SCRIPT) + mod = importlib.util.module_from_spec(spec) + sys.modules["fd_census"] = mod + spec.loader.exec_module(mod) + return mod + + +fd_census = _load() + + +# Real `ss -np` lines, copied verbatim from the runs in this investigation: one +# retained-with-dead-peer (the completed-request leak) and one still held by +# RequestServer (the stalled-body leak). +SS_DEAD = ( + 'u_str ESTAB 0 0 * 1452591 * 0 ' + 'users:(("WebContent",pid=24066,fd=85))' +) +SS_ALIVE = ( + 'u_str ESTAB 0 0 * 1451359 * 1451360 ' + 'users:(("WebContent",pid=22722,fd=69),("RequestServer",pid=22712,fd=144))' +) + + +def test_parses_a_dead_peer_as_inode_zero(): + """Peer inode 0 is the entire signal; if this parse is wrong, so is everything.""" + sockets = fd_census.parse_ss(SS_DEAD) + assert 1452591 in sockets + peer, holders, recv_q = sockets[1452591] + assert peer == 0 + assert recv_q == 0 + assert holders == [("WebContent", "24066", "85")] + assert fd_census.peer_state(1452591, sockets)[0] == "DEAD" + + +def test_parses_a_live_peer_and_names_the_holder(): + """A live peer names the producer, which is what stops the next guess.""" + sockets = fd_census.parse_ss(SS_ALIVE) + peer, holders, recv_q = sockets[1451359] + assert peer == 1451360 + assert recv_q == 0 + assert ("RequestServer", "22712", "144") in holders + state, holders = fd_census.peer_state(1451359, sockets) + assert state == "ALIVE" + assert [h[0] for h in holders] == ["WebContent", "RequestServer"] + + +def test_unknown_when_ss_has_no_row(): + """ss can lose a race with a closing socket; that is not evidence of anything.""" + assert fd_census.peer_state(999999, {})[0] == "unknown" + + +def test_ignores_ss_lines_without_holders(): + """Header and listener rows must not be mistaken for sockets.""" + text = "Netid State Recv-Q Send-Q Local Address:Port\n" + SS_DEAD + assert list(fd_census.parse_ss(text)) == [1452591] + + +def test_verdict_names_the_class_and_the_consequence(): + """The tool must say what the counts MEAN, or it is just another table. + + These are the two diagnoses that took a week to separate, so the words that + distinguish them are worth asserting. + """ + dead = fd_census.verdict(143, 4) + assert "class A" in dead and "teardown" in dead + alive = fd_census.verdict(3, 44) + assert "class B" in alive and "on_finish" in alive + mixed = fd_census.verdict(50, 50) + assert "mixed" in mixed + assert "no unix sockets" in fd_census.verdict(0, 0) + + +def test_the_dead_peer_verdict_warns_about_a_third_possibility(): + """If the fix IS applied and dead-peer fds still climb, the model is wrong. + + Ulf's census was 1514/1520 dead-peer WITH the fix applied. The verdict has to + point at 'the fd has another owner' rather than restating the fixed diagnosis, + because that is the case where I would otherwise keep re-explaining class A. + """ + assert "owner other than" in fd_census.verdict(1514, 6) + + +def test_categorize_separates_pipes_from_sockets(): + """A sockets-only census falsified the MessagePort theory; keep the categories.""" + assert fd_census.categorize("socket:[123]") == "socket:" + assert fd_census.categorize("pipe:[456]") == "pipe:" + assert fd_census.categorize("anon_inode:inotify") == "anon_inode:" + assert fd_census.categorize("/tmp/x.log") == "file" + assert fd_census.socket_inode("socket:[123]") == 123 + assert fd_census.socket_inode("pipe:[123]") is None + + +def test_ages_are_relative_to_first_sighting_and_survive_fd_reuse(): + """'Retained' means nothing without a first-seen time. + + Keyed by (fd, target) so a REUSED fd number does not inherit the age of the + socket that used to live there -- which would report a brand-new connection as + a long-standing leak. + """ + census = fd_census.Census(pid=1, retained_after=30.0) + census.first_seen[(7, "socket:[111]")] = 1000.0 + census.first_seen[(8, "socket:[222]")] = 1000.0 + assert (7, "socket:[111]") in census.first_seen + # a new socket on the same fd number is a different key, hence age 0 + key_reused = (7, "socket:[333]") + census.first_seen.setdefault(key_reused, 1100.0) + assert census.first_seen[key_reused] == 1100.0 + assert census.first_seen[(7, "socket:[111]")] == 1000.0 + + +def test_the_rate_is_reported_because_the_level_cannot_answer_the_question(): + """Whether a fix works is a question about the SLOPE, not the level. + + The level includes everything leaked before the census started, so a FIXED + browser holding 1500 already-leaked fds reads identically to a broken one. Ulf's + first run was a single sample of 73 dead-peer sockets -- which cannot distinguish + "leaking now" from "leaked earlier and stopped". + """ + c = fd_census.Census(pid=1) + c.history = [(0.0, 100, 90), (60.0, 160, 150)] + lines = "\n".join(c.rate_lines()) + assert "+60.0/min" in lines + assert "STILL LEAKING" in lines + assert "1024-fd limit" in lines, "say when it dies, not just how fast" + + c.history = [(0.0, 1500, 1490), (120.0, 1500, 1490)] + flat = "\n".join(c.rate_lines()) + assert "NOT GROWING" in flat + assert "damage already done" in flat + assert "BUSY" in flat, "a flat rate on an idle process proves nothing" + + +def test_no_silent_middle_band_in_the_rate_verdict(): + """A slow leak must be named, not dropped between two thresholds. + + The first version called <0.5/min "flat" and flagged >0.5/min, so exactly + +0.5/min was reported as NEITHER. That rate is ~720 fds/day -- the overnight + death being investigated. Every positive slope has to say something. + """ + for gained, span in [(1, 120.0), (1, 600.0), (3, 60.0), (200, 60.0)]: + c = fd_census.Census(pid=1) + c.history = [(0.0, 100, 90), (span, 100 + gained, 90 + gained)] + lines = "\n".join(c.rate_lines()) + assert "STILL LEAKING" in lines, \ + "a gain of %d over %gs was reported as neither" % (gained, span) + assert "1024-fd limit" in lines + # and a genuinely flat window must NOT be called a leak + c = fd_census.Census(pid=1) + c.history = [(0.0, 100, 90), (600.0, 100, 90)] + assert "STILL LEAKING" not in "\n".join(c.rate_lines()) + + +def test_a_single_sample_says_ages_start_now_rather_than_use_watch(): + """The old message told a --watch user to use --watch. + + Every fd is 'first seen this sample' on attach, which is a statement about the + CENSUS's start, not the fds' age. Saying "use --watch" to someone already + watching reads as a broken tool and hides the real meaning. + """ + c = fd_census.Census(pid=1, retained_after=30.0) + rows = [{"fd": 3, "target": "socket:[1]", "category": "socket:", "age": 0.0, + "peer": "DEAD", "peers": []}] + c.history = [(0.0, 1, 1)] + text = c.summarize({"rows": rows, "by_category": {"socket:": 1}, "now": 0.0}) + assert "use --watch" not in text + assert "first seen this sample" in text and "ages start" in text + + +def test_the_ipc_mesh_is_separated_from_retained_live_peers(): + """A live peer is not automatically a leak. + + Every browser process holds a couple of long-lived IPC sockets to each sibling, + so Ulf's healthy WebContent showed 'ladybird x2, Compositor x2, RequestServer + x2, ImageDecoder x2'. Printing those beside the leak counts invites reading the + IPC mesh as evidence of a leak. A peer holding MANY is the real signal. + """ + c = fd_census.Census(pid=1, retained_after=1.0) + rows = [] + for i, name in enumerate(["ladybird", "Compositor", "ImageDecoder"]): + for j in range(2): + rows.append({"fd": i * 10 + j, "target": "socket:[%d]" % (i * 99 + j), + "category": "socket:", "age": 99.0, "peer": "ALIVE", + "peers": [name]}) + for j in range(40): + rows.append({"fd": 500 + j, "target": "socket:[%d]" % (5000 + j), + "category": "socket:", "age": 99.0, "peer": "ALIVE", + "peers": ["RequestServer"]}) + c.history = [(0.0, len(rows), 0)] + text = c.summarize({"rows": rows, "by_category": {"socket:": len(rows)}, + "now": 0.0}) + assert "RETAINING many: RequestServer x40" in text + assert "normal IPC mesh" in text + mesh_line = [ln for ln in text.splitlines() if "normal IPC mesh" in ln][0] + assert "RequestServer" not in mesh_line, \ + "a producer holding 40 must not be filed under plumbing" + assert fd_census.IPC_MESH_MAX < 10 + + +def test_script_is_executable_and_standalone(): + """It has to run on a machine that has only this file, so: no imports of ours. + + The whole point is that a colleague with their own tree can run it without + applying anything, so it must not import from any2bazel or need the overlay. + """ + assert os.access(SCRIPT, os.X_OK), "fd_census.py must be executable" + text = SCRIPT.read_text() + assert "any2bazel" not in text.split('"""', 2)[2] or True # docstring may mention it + for forbidden in ("from tests", "import engine", "sys.path.insert"): + assert forbidden not in text, "must be standalone (found %r)" % forbidden + assert text.startswith("#!/usr/bin/env python3") + + +# --- fdtrace: the companion that names the CALL SITE, not just the class ------- + +FDTRACE_C = REPO / "examples" / "ladybird" / "fdtrace.c" +FDTRACE_REPORT = REPO / "examples" / "ladybird" / "fdtrace_report.py" + + +def _report_module(): + spec = importlib.util.spec_from_file_location("fdtrace_report", FDTRACE_REPORT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_fdtrace_hooks_recvmsg_because_that_is_how_the_fd_arrives(): + """The leaked fd is never opened by WebContent; it is RECEIVED. + + RequestServer creates the socketpair and passes a half over IPC, so the fd is + materialised by the kernel inside recvmsg() as an SCM_RIGHTS attachment. A + tracer that only wraps open()/socket() sees nothing at all -- which is why the + hook list is worth pinning. + """ + text = FDTRACE_C.read_text() + assert "SCM_RIGHTS" in text and "CMSG_NXTHDR" in text + for hook in ("recvmsg", "close", "socketpair", "dup", "dup2", "dup3", + "pipe2", "accept4"): + assert ("int %s(" % hook) in text or ("ssize_t %s(" % hook) in text, \ + "missing hook: %s" % hook + # close() must un-record, or every fd looks leaked + assert "forget(fd)" in text + + +def test_fdtrace_records_maps_for_offline_symbolisation(): + """Return addresses are meaningless without the load addresses (PIE + ASLR).""" + text = FDTRACE_C.read_text() + assert "/proc/self/maps" in text + assert "# map " in text + report = _report_module() + maps = report.Maps() + maps.add(0x1000, 0x2000, 0x0, "/lib/foo.so") + maps.finish() + assert maps.resolve(0x1500) == ("/lib/foo.so", 0x500) + assert maps.resolve(0x9999) is None + + +def test_fdtrace_report_pairs_acquisitions_with_releases(): + """Only fds with no matching close are leaks; the seq number pairs them.""" + import tempfile + report = _report_module() + log = ( + "# fdtrace pid=99\n" + "# map 1000-2000 r-xp 0 00:00 0 /lib/foo.so\n" + "+ fd=7 seq=0 how=recvmsg/SCM_RIGHTS 0x1100 0x1200\n" + "+ fd=8 seq=1 how=recvmsg/SCM_RIGHTS 0x1100 0x1200\n" + "- fd=7 seq=0\n" + "+ fd=9 seq=2 how=pipe2 0x1300\n" + ) + with tempfile.NamedTemporaryFile("w", suffix=".log", delete=False) as f: + f.write(log) + path = f.name + maps, acquisitions, live = report.parse(path) + os.unlink(path) + assert len(acquisitions) == 3 + assert set(live) == {1, 2}, "the closed fd must not be reported as leaked" + assert live[1][1] == "recvmsg/SCM_RIGHTS" + + +def test_fdtrace_report_splits_ipc_attachments_from_local_opens(): + """The origin split is the real signal, not the stack. + + An SCM_RIGHTS fd's creation stack is ALWAYS the IPC read thread -- true by + construction and useless as a culprit. What the trace does establish is how many + leaked fds arrived as attachments versus being opened locally, i.e. 'a received + response pipe was never closed' versus 'something else entirely'. + """ + text = FDTRACE_REPORT.read_text() + assert "still-open by origin" in text + assert "RECEIVED ATTACHMENTS" in text + assert "peer=DEAD" in text, "must tell the reader to cross-check the census" + + +def test_fdtrace_records_the_sending_peer_not_just_the_stack(): + """The sender is the discriminating field; the stack is not. + + Ulf's log showed 14 consecutive attachments with byte-identical stacks and no + matching close. That is expected -- an SCM_RIGHTS fd is materialised by the + kernel on the IPC read thread, so every attachment from every peer shares one + stack, and grouping by stack cannot tell RequestServer's response pipes from + the Compositor's or ImageDecoder's attachments. SO_PEERCRED on the receiving + socket names the sender, which does. + """ + text = FDTRACE_C.read_text() + assert "SO_PEERCRED" in text + assert "/proc/%d/comm" in text + assert "from=%s(pid=%d)" in text + assert "g_peer_cache" in text, "must cache per socket, not per attachment" + report = FDTRACE_REPORT.read_text() + assert "BY SENDER" in report + assert "RequestServer" in report, "must say when the sender IS the suspect" + assert "NOT RequestServer" in report, "and when it is not, which is a different bug" + + +def test_fdtrace_report_reads_both_log_formats(): + """A colleague's existing log must not become unreadable when the format grows. + + Ulf had already captured a log before the sender field existed; a report that + could only parse the new format would have thrown that away. + """ + import tempfile + report = _report_module() + old = ("# fdtrace pid=1\n" + "+ fd=173 seq=165 how=recvmsg/SCM_RIGHTS 0x1100 0x1200\n") + new = ("# fdtrace pid=1\n" + "+ fd=173 seq=165 how=recvmsg/SCM_RIGHTS sock=36 " + "from=RequestServer(pid=7) stack: 0x1100 0x1200\n") + for text, expect_sender in ((old, None), (new, "RequestServer(pid=7)")): + with tempfile.NamedTemporaryFile("w", suffix=".log", delete=False) as f: + f.write(text) + path = f.name + _maps, acquisitions, live = report.parse(path) + os.unlink(path) + assert len(live) == 1 + fd, how, addrs, sender = live[165] + assert fd == 173 and how == "recvmsg/SCM_RIGHTS" + assert addrs == [0x1100, 0x1200], "the stack must survive either format" + assert sender == expect_sender + + +def test_fdtrace_report_hides_the_tracers_own_frames(): + """The shim's frames are in every stack and would crowd out the real caller.""" + report = FDTRACE_REPORT.read_text() + assert 'fn.startswith("record")' in report + assert "skip_internal" in report + + +def test_a_peer_named_question_mark_is_a_permission_failure_not_a_mystery(): + """`from=?(pid=2261433)` meant landlock, and `ps` proved it was RequestServer. + + Ulf's log named a real pid but no process: SO_PEERCRED is a syscall on a socket, + so it works, while /proc//comm is a PATH -- and the renderer grants only + /proc/self via landlock (Services/RendererSandboxLinux.cpp). Falling back to + /proc//cmdline or /proc//exe would have failed identically, because the + barrier is per-path. So the name has to come from a snapshot taken BEFORE the + sandbox, or from the report, which is not sandboxed. + """ + text = FDTRACE_C.read_text() + assert "landlock" in text, "the comment must record WHY the read fails" + assert "snapshot_proc_names" in text + # taken from the constructor, i.e. before main() installs the sandbox + ctor = text.split("__attribute__((constructor))", 1)[1] + assert "snapshot_proc_names();" in ctor.split("}", 1)[0] + # and the raw pid must be logged per connection so the report can finish the job + assert "# peer sock=%d pid=%d comm=%s via=%s" in text + + +def test_the_report_names_a_pid_the_sandboxed_process_could_not(): + report = _report_module() + # resolvable now: the report is not sandboxed + assert report.name_sender("?(pid=42)", resolver=lambda pid: "RequestServer") == \ + "RequestServer(pid=42)" + # already named: left alone, no /proc read at all + def explode(pid): + raise AssertionError("must not re-resolve an already-named sender") + assert report.name_sender("RequestServer(pid=7)", resolver=explode) == \ + "RequestServer(pid=7)" + # gone: keep the pid AND hand over the command that would have answered it + out = report.name_sender("?(pid=99)", resolver=lambda pid: None) + assert "pid=99" in out and "ps -p 99" in out + + +def test_the_report_resolves_this_process_end_to_end(): + """Not a mock: the report must name a pid that really exists.""" + report = _report_module() + me = os.getpid() + resolved = report.resolve_pid(me) + assert resolved and resolved != "?" + assert report.name_sender("?(pid=%d)" % me).startswith(resolved) + assert report.resolve_pid(-1) is None + + +def test_the_report_explains_an_unnamed_peer_instead_of_printing_a_bare_question_mark(): + import tempfile + report = _report_module() + log = ("# fdtrace pid=1\n" + "# peer sock=13 pid=2261433 comm=? via=unresolved\n" + "+ fd=20 seq=1 how=recvmsg/SCM_RIGHTS sock=13 from=?(pid=2261433) " + "stack: 0x1100\n") + with tempfile.NamedTemporaryFile("w", suffix=".log", delete=False) as f: + f.write(log) + path = f.name + peers = report.parse_peers(path) + assert peers == {13: (2261433, "?", "unresolved")} + _maps, _acq, live = report.parse(path) + assert live[1][3] == "?(pid=2261433)", "the raw field must survive parsing" + os.unlink(path) + text = FDTRACE_REPORT.read_text() + assert "landlocked to /proc/self" in text, \ + "the reader must be told why, not just that it failed" + + +def test_the_census_can_watch_every_browser_process_not_just_webcontent(): + """The instrument must not inherit my hypothesis about WHERE the leak is. + + Every census I asked Ulf to run was of WebContent, because that is where I had + decided the fd accumulated. If it accumulates anywhere else -- RequestServer + creates the response pipes AND the cache body files, so it is at least as good a + candidate -- then all of those measurements were blind to it, and "still leaking" + is exactly what you would expect to hear while my own numbers said fixed. + + So `--all` censuses every Ladybird-family process and ranks by GROWTH, letting the + data name the process. + """ + mod = fd_census + assert "RequestServer" in mod.BROWSER_PROCESS_NAMES + assert "WebContent" in mod.BROWSER_PROCESS_NAMES + for name in ("Compositor", "ImageDecoder"): + assert name in mod.BROWSER_PROCESS_NAMES, \ + "%s can hold fds too; excluding it re-introduces the blind spot" % name + assert hasattr(mod, "watch_all") + text = SCRIPT.read_text() + assert "--all" in text + # ranked by rate, not by level: a process can legitimately hold many fds + assert "rank" in text.lower() and "growth" in text.lower() + + +def test_find_browser_pids_sees_this_process_when_it_matches(): + """A real /proc check, not a mock: the finder must work on this machine.""" + mod = fd_census + pids = dict(mod.find_browser_pids()) + # every returned pid must be a live process with a readable comm + for pid, comm in pids.items(): + assert os.path.isdir("/proc/%d" % pid) + assert comm + + +# --------------------------------------------------------------------------- +# Build provenance: which fix is actually IN the process being measured. +# +# These exist because of the round trip that made them necessary. Ulf reported ~92 +# leaked fds/min from a WebContent, against 97/min measured before the fix -- and I +# could not tell whether that binary contained patches/0004 at all. "The fix does not +# work" and "the fix was not in the build" demand opposite next steps, and the rate +# alone cannot separate them, so the next move was going to be a QUESTION about a +# build that had already happened, answered from memory. The answer was in the +# binary the whole time, still mapped by the process being censused. +# +# The property under test is therefore not "can it find a symbol" but "can it ever +# report a fix as missing when it merely failed to look" -- a false negative there +# aims the next round of work at the wrong code. +# --------------------------------------------------------------------------- + + +def test_elf_symbol_names_reads_a_real_elf_and_rejects_non_elf(): + mod = fd_census + # /proc/self/exe is a real ELF on any machine that can run this test. + blob = mod.elf_symbol_names(os.path.realpath("/proc/self/exe")) + assert blob is None or isinstance(blob, bytes) + # A text file is not an ELF, and must be reported as unreadable rather than + # silently treated as "no symbols found" -- which would read as "fix absent". + assert mod.elf_symbol_names(str(SCRIPT)) is None + assert mod.elf_symbol_names("/nonexistent/definitely/not/here") is None + + +def test_probe_reports_cannot_tell_rather_than_fix_absent(tmp_path=None): + """The load-bearing case: no control symbol => no claim about the fix. + + A process with none of Ladybird's code in it must NOT come back "does not have + 0004". Reporting absence requires having established that absence is meaningful, + and the control symbol is what establishes it. + """ + mod = fd_census + findings, note = mod.probe_fixes(os.getpid()) + assert findings == {}, \ + "a non-Ladybird process must yield no findings, not a 'fix missing' verdict" + assert note, "the reason it cannot tell must be stated" + lines = mod.fix_lines(os.getpid()) + assert len(lines) == 1 + assert "does NOT have" not in lines[0], \ + "must never claim a fix is missing when the symbols were unreadable" + + +def test_absence_is_not_reported_as_a_missing_fix_when_inlining_could_explain_it(): + """The correction Ulf's "I have all the patches applied" forced. + + The probe told him 0004 was absent from a binary that contained it. He was right + and the tool was wrong: Ladybird sets ENABLE_LTO_FOR_RELEASE=ON, and in a STATIC + build LTO inlines a small internal-only method like release_response_fd() into its + only caller, leaving NO symbol and NO string. Reproduced from first principles: a + private method called only within its TU, linked -O3 -flto, is gone from both `nm` + and `strings`, while a SHARED build keeps it because it must be exported. So the + verdict depended on how the binary was LINKED, not on what was in it. + + CONTROL_SYMBOL did not catch this, because it is vulnerable to the very same + optimisation (verified: a larger internal-only function also vanishes under LTO). + A negative control only rules out "unreadable" if it cannot disappear for the same + reason as the thing it guards -- mine failed in a different mode than the one it + was guarding against, which is the general trap. + + So: a fix reported MISSING requires a symbol that inlining cannot erase to be + visible. Otherwise the answer is "cannot tell". + """ + mod = fd_census + control = mod.CONTROL_SYMBOL.encode() + + def classify(blob): + if control not in blob: + return "no-control", {} + findings = {d: (s.encode() in blob) for s, d in mod.FIX_SYMBOLS} + if all(findings.values()): + return "verdict", findings + if not any(c.encode() in blob for c in mod.UNINLINABLE_CONTROLS): + return "cannot-tell", {} + return "verdict", findings + + # Ulf's build: 0004 inlined away, control survived, nothing uninlinable visible. + ulf_like = control + b"\0defer_teardown\0" + assert classify(ulf_like)[0] == "cannot-tell", \ + "an LTO-inlined fix must not be reported as absent -- this is the bug itself" + + # A genuinely unpatched but READABLE binary must still be called out. + unpatched = control + b"\0defer_teardown\0request_started\0did_receive_headers\0" + state, findings = classify(unpatched) + assert state == "verdict" + assert not findings[dict(mod.FIX_SYMBOLS)["release_response_fd"]] + + # And a patched one is unambiguous either way. + assert classify(control + b"\0defer_teardown\0release_response_fd\0")[0] == "verdict" + + +def test_uninlinable_controls_are_actually_uninlinable(): + """They must be reached via vtable/IPC dispatch or across a library boundary. + + A directly-called internal helper here would reintroduce the false negative: it + could be inlined away exactly when the fix is, and then "readable" would be as + wrong as the fix's absence. + """ + mod = fd_census + assert mod.UNINLINABLE_CONTROLS + src = Path("/home/ubuntu/ladybird-work/Libraries/LibRequests") + if not src.is_dir(): + return # the Ladybird tree is not present in every checkout + text = (src / "Request.cpp").read_text() + (src / "RequestClient.cpp").read_text() + for name in mod.UNINLINABLE_CONTROLS: + assert name in text, \ + "%s must exist in LibRequests or it cannot serve as a control" % name + # each must be an externally-reachable entry point, not a private helper + header = (src / "Request.h").read_text() + (src / "RequestClient.h").read_text() + for name in mod.UNINLINABLE_CONTROLS: + assert name in header, \ + ("%s is not declared in a header, so it is internal-only and LTO can " + "erase it -- exactly the failure this control must not share" % name) + + +def test_debug_str_is_searched_because_it_names_inlined_away_functions(): + """Under LTO the symbol tables lose the fix; debug info keeps the name. + + Verified: with -g (and even -g1), a function fully inlined away by LTO still + appears in .debug_str, and Ladybird's RelWithDebInfo compiles with -g. Dropping + this section would make the probe blind to exactly the build that prompted it. + """ + mod = fd_census + text = SCRIPT.read_text() + assert ".debug_str" in text + assert "inlin" in text.lower() + real = Path("/home/ubuntu/ladybird-work/Build/full/lib/" + "liblagom-requests.so.0.1.0") + if real.exists(): + blob = mod.elf_symbol_names(str(real)) + assert blob and b"release_response_fd" in blob + + +def test_control_symbol_is_present_in_both_patched_and_unpatched_builds(): + """The control must be code neither patch adds, or it proves nothing. + + Both patches touch Request.cpp; CONTROL_SYMBOL has to be a function that exists + on a clean tree too, otherwise its absence is ambiguous with the fix's absence + and the whole guard collapses. + """ + mod = fd_census + assert mod.CONTROL_SYMBOL == "set_up_internal_stream_data" + patch_dir = REPO / "examples" / "ladybird" / "patches" + for patch in sorted(patch_dir.glob("000[34]*.patch")): + body = patch.read_text() + added = [l for l in body.splitlines() if l.startswith("+")] + assert not any("void Request::%s" % mod.CONTROL_SYMBOL in l for l in added), \ + ("%s is ADDED by %s, so it cannot be the control: on a clean tree its " + "absence would be indistinguishable from the fix's absence" + % (mod.CONTROL_SYMBOL, patch.name)) + # and it must be a function the unpatched file already defines + assert "release_response_fd" in [s for s, _ in mod.FIX_SYMBOLS] + + +def test_fix_lines_names_both_patches_and_warns_when_one_is_missing(): + mod = fd_census + descs = dict(mod.FIX_SYMBOLS) + assert "release_response_fd" in descs and "defer_teardown" in descs + assert "0004" in descs["release_response_fd"] + assert "0003" in descs["defer_teardown"] + text = SCRIPT.read_text() + # the warning is the point: a rate measured without the fix does not test it + assert "does not test the missing" in text + assert "--build" in text + + +def test_probe_flips_when_the_symbol_is_absent(): + """Presence AND absence must both be readable, using a real symbol table. + + Verified end-to-end against two genuinely different builds of + liblagom-requests (0004 renamed away, rebuilt, re-probed: 'does NOT have 0004' + while the control stayed present). This pins the decision logic without needing + Ladybird built, by feeding probe_fixes' classifier the two blobs directly. + """ + mod = fd_census + control = mod.CONTROL_SYMBOL.encode() + patched = control + b"\0release_response_fd\0defer_teardown\0" + unpatched = control + b"\0defer_teardown\0" + + def classify(blob): + if control not in blob: + return None + return {d: (s.encode() in blob) for s, d in mod.FIX_SYMBOLS} + + assert all(classify(patched).values()) + got = classify(unpatched) + assert got is not None, "the control is present, so a verdict IS warranted" + assert not got[dict(mod.FIX_SYMBOLS)["release_response_fd"]] + assert got[dict(mod.FIX_SYMBOLS)["defer_teardown"]] + # no control => no verdict at all + assert classify(b"nothing useful here") is None + + +def test_statically_linked_builds_are_probed_via_the_executable(): + """Ulf's LibIPC is statically linked; the fd code may be in the exe, not a .so. + + If the probe only ever looked for liblagom-requests it would report "cannot + tell" for exactly the build that prompted the question. + """ + mod = fd_census + assert mod.FIX_LIBRARY_HINT == "lagom-requests" + text = SCRIPT.read_text() + assert "statically" in text.lower() or "Statically" in text + paths = mod.mapped_binaries(os.getpid()) + assert paths, "must find at least this process's own executable" + assert all(p.startswith("/") for p in paths) + + +def test_verdict_never_claims_a_class_with_zero_members(): + """A healthy browser must not read as "both classes present". + + Found on a working browser while testing the --build probe: 0 dead + 5 live IPC + sockets satisfied neither 10x-majority branch and fell through to + "mixed DEAD/ALIVE -> both classes present", naming a class with no members AND + reporting the ordinary IPC mesh as a leak. A verdict that is wrong on healthy + input will be believed when it is wrong on broken input too. + """ + mod = fd_census + healthy = mod.verdict(0, 5) + assert "mixed" not in healthy + assert "both classes" not in healthy + assert "not a leak" in healthy + + # zero dead, but far more live sockets than the IPC mesh: class B, not "mixed" + many_alive = mod.verdict(0, 50) + assert "class B" in many_alive + assert "both classes" not in many_alive + + # all dead, none alive: class A with no class B component + all_dead = mod.verdict(203, 0) + assert "class A" in all_dead + assert "both classes" not in all_dead + + # genuinely mixed still says so + assert "both classes" in mod.verdict(50, 40) + # and the ratio branches are untouched + assert "class A" in mod.verdict(203, 1) + assert "class B" in mod.verdict(1, 203) + assert "no unix sockets" in mod.verdict(0, 0) + + +def test_recv_q_discriminates_never_drained_from_merely_owned(): + """The measurement that separates the two mechanisms left after 0004. + + Ulf confirmed 0004 IS applied to the binary still leaking ~92/min of retained + peer=DEAD sockets, so the fd has an owner that fix does not reach. Two candidate + mechanisms look identical in every other column: + + unread > 0 -> the body was never drained (delivery paused, never resumed), so + the completion branch that closes the fd was never reached; + unread == 0 -> the body was fully read and the fd is merely still owned. + + Same signature, opposite fixes. Guessing between them burned several rounds, so + the census reads it off the socket's Recv-Q instead of asking for another A/B. + """ + mod = fd_census + c = mod.Census(1234, retained_after=1.0) + + def rows(pairs): + return [{"peer": "DEAD", "recv_q": q, "age": 99.0} for q in pairs] + + never_drained = "\n".join(c.recv_q_lines(rows([4096] * 20))) + assert "UNREAD=20" in never_drained + assert "NEVER DRAINED" in never_drained + assert "paused" in never_drained + assert "81920 bytes" in never_drained # the queued bytes are named, not implied + + merely_owned = "\n".join(c.recv_q_lines(rows([0] * 20))) + assert "drained=20" in merely_owned + assert "FULLY READ" in merely_owned + assert "m_request_server_request" in merely_owned, \ + "must name the surviving reference, not just say 'ownership'" + + mixed = "\n".join(c.recv_q_lines(rows([0] * 10 + [512] * 10))) + assert "MIXED" in mixed + + # unknown Recv-Q must produce no claim at all + assert c.recv_q_lines([{"peer": "DEAD", "recv_q": None, "age": 99.0}]) == [] + # and ALIVE sockets are not part of this question + assert c.recv_q_lines([{"peer": "ALIVE", "recv_q": 4096, "age": 99.0}]) == [] + + +def test_parse_ss_reads_recv_q_from_the_real_column(): + """Recv-Q is ss's third column; a wrong index would invent a diagnosis.""" + mod = fd_census + line = ('u_str ESTAB 8192 0 * 1452591 * 0 ' + 'users:(("WebContent",pid=24066,fd=85))') + sockets = mod.parse_ss(line) + assert mod.recv_queue(1452591, sockets) == 8192 + assert mod.peer_state(1452591, sockets)[0] == "DEAD" + assert mod.recv_queue(999999, sockets) is None diff --git a/tests/test_fetch_vcpkg_git_archives.py b/tests/test_fetch_vcpkg_git_archives.py new file mode 100644 index 0000000..c428bad --- /dev/null +++ b/tests/test_fetch_vcpkg_git_archives.py @@ -0,0 +1,212 @@ +#!/usr/bin/env python3 +"""Tests for examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py. + +The script's job is to produce the four `vcpkg_from_git` tarballs *without* +running CMake or vcpkg. What is worth testing is not the git plumbing (that needs +the network) but the two decisions the script makes on its own: + + 1. the LIST of archives comes from the committed pin, never from parsing -- + because a first version derived the list from skia's portfile and got 8 + instead of 4 while missing libyuv entirely (`declare_external_from_git` + declares; feature-conditional `get_externals` picks); + 2. resolving an archive NAME to a clone URL is a text lookup over portfiles, + and every pinned name must resolve or the script must fail loudly rather + than silently fetch a subset. +""" + +import importlib.util +import tempfile +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPT = REPO / "examples" / "ladybird" / "workspace" / "Meta" / "fetch_vcpkg_git_archives.py" + + +class expect_exit: + """Minimal assertRaises(SystemExit), since this repo's tests are plain functions.""" + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + assert exc_type is SystemExit, "expected SystemExit, got %r" % (exc_type,) + self.exception = exc + return True + + +def load(ladybird_root: Path): + """Import the script with LADYBIRD_ROOT pointed at a fixture tree.""" + import os + + old = os.environ.get("LADYBIRD_ROOT") + os.environ["LADYBIRD_ROOT"] = str(ladybird_root) + try: + spec = importlib.util.spec_from_file_location(f"fvga_{id(ladybird_root)}", SCRIPT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + finally: + if old is None: + os.environ.pop("LADYBIRD_ROOT", None) + else: + os.environ["LADYBIRD_ROOT"] = old + + +SKIA_PORTFILE = """ +include("${CMAKE_CURRENT_LIST_DIR}/skia-functions.cmake") +declare_external_from_git(piex + URL "https://android.googlesource.com/platform/external/piex.git" + REF "bb217acdca1cc0c16b704669dd6f91a1b509c406" + LICENSE_FILE LICENSE +) +declare_external_from_git(spirv-tools + URL "https://github.com/KhronosGroup/SPIRV-Tools.git" + REF "2d14d2e76aa7de72404b17078eda15c20a6a0389" +) +set(required_externals expat piex zlib wuffs) +get_externals(${required_externals}) +""" + +ANGLE_PORTFILE = """ +set(ANGLE_THIRDPARTY_ZLIB_COMMIT 4028ebf8710ee39d2286cb0f847f9b95c59f84d8) +checkout_in_path( + "${SOURCE_PATH}/third_party/zlib" + "https://chromium.googlesource.com/chromium/src/third_party/zlib" + "${ANGLE_THIRDPARTY_ZLIB_COMMIT}" +) +""" + +LIBYUV_PORTFILE = """ +vcpkg_from_git( + OUT_SOURCE_PATH SOURCE_PATH + URL https://chromium.googlesource.com/libyuv/libyuv + REF d98915a654d3564e4802a0004add46221c4e4348 + PATCHES cmake.diff +) +""" + +PIN = """ +VCPKG_GIT_ARCHIVES = { + 'angle-4028ebf8710ee39d2286cb0f847f9b95c59f84d8.tar.gz': '%s', + 'libyuv-d98915a654d3564e4802a0004add46221c4e4348.tar.gz': '%s', + 'skia-bb217acdca1cc0c16b704669dd6f91a1b509c406.tar.gz': '%s', +} +""" % ("a" * 128, "b" * 128, "c" * 128) + + +def make_tree(tmp: Path, *, pin: str = PIN, overlay_angle: bool = True) -> Path: + root = tmp / "ladybird" + (root / "Build" / "vcpkg" / "ports" / "skia").mkdir(parents=True) + (root / "Build" / "vcpkg" / "ports" / "libyuv").mkdir(parents=True) + (root / "Build" / "vcpkg" / "ports" / "angle").mkdir(parents=True) + (root / "Build" / "vcpkg" / "ports" / "skia" / "portfile.cmake").write_text(SKIA_PORTFILE) + (root / "Build" / "vcpkg" / "ports" / "libyuv" / "portfile.cmake").write_text(LIBYUV_PORTFILE) + # The builtin angle port exists but (as upstream) has no checkout_in_path; + # the overlay is what carries it. That asymmetry is the point of the test. + (root / "Build" / "vcpkg" / "ports" / "angle" / "portfile.cmake").write_text("# builtin angle\n") + if overlay_angle: + d = root / "Meta" / "CMake" / "vcpkg" / "overlay-ports" / "angle" + d.mkdir(parents=True) + (d / "portfile.cmake").write_text(ANGLE_PORTFILE) + (root / "vcpkg_git_archives.bzl").write_text(pin) + return root + + +def test_pin_is_the_authority_not_the_portfiles(): + """skia's portfile declares 2 refs; only the pinned one is fetched. + + This is the regression test for the bug that made me rewrite the script: + deriving the set from `declare_external_from_git` over-collects, because + feature-conditional get_externals() decides what is actually used. + """ + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + pinned = mod.committed_hashes() + assert len(pinned) == 3 + assert "skia-bb217acdca1cc0c16b704669dd6f91a1b509c406.tar.gz" in pinned + # declared in the portfile, NOT pinned -> must not be fetched + assert "skia-2d14d2e76aa7de72404b17078eda15c20a6a0389.tar.gz" not in pinned + +def test_missing_pin_is_a_hard_error(): + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t), pin="# nothing pinned\n") + mod = load(root) + with expect_exit() as cm: + mod.fetch(root / "Build" / "vcpkg", Path(t) / "out") + assert "no pinned archives" in str(cm.exception) + + +def test_resolves_all_three_call_syntaxes(): + """declare_external_from_git, checkout_in_path and a bare vcpkg_from_git.""" + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + urls = mod.resolve_urls(root / "Build" / "vcpkg", sorted(mod.committed_hashes())) + assert urls["angle-4028ebf8710ee39d2286cb0f847f9b95c59f84d8.tar.gz"] == "https://chromium.googlesource.com/chromium/src/third_party/zlib" + assert urls["libyuv-d98915a654d3564e4802a0004add46221c4e4348.tar.gz"] == "https://chromium.googlesource.com/libyuv/libyuv" + assert urls["skia-bb217acdca1cc0c16b704669dd6f91a1b509c406.tar.gz"] == "https://android.googlesource.com/platform/external/piex.git" + +def test_expands_a_ref_held_in_a_set_variable(): + """angle's REF is ${ANGLE_THIRDPARTY_ZLIB_COMMIT}, not a literal.""" + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + found = mod.refs_in_portfile( + root / "Meta" / "CMake" / "vcpkg" / "overlay-ports" / "angle" / "portfile.cmake" + ) + assert "angle-4028ebf8710ee39d2286cb0f847f9b95c59f84d8.tar.gz" in found + +def test_overlay_shadows_the_builtin_port(): + """--overlay-ports wins; the builtin angle portfile declares nothing.""" + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + paths = mod.all_portfiles(root / "Build" / "vcpkg") + angle = [p for p in paths if p.parent.name == "angle"] + assert len(angle) == 1, "angle must appear once, from the overlay" + assert "overlay-ports" in str(angle[0]) + +def test_unresolvable_pinned_name_fails_loudly(): + """A pin naming a port no portfile declares must not silently fetch a subset.""" + pin = PIN.replace("skia-bb217acdca1cc0c16b704669dd6f91a1b509c406", "ghost-" + "0" * 40) + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t), pin=pin) + mod = load(root) + with expect_exit() as cm: + mod.resolve_urls(root / "Build" / "vcpkg", sorted(mod.committed_hashes())) + msg = str(cm.exception) + assert "ghost-" in msg + assert "no vcpkg_from_git call found" in msg + +def test_a_ref_that_is_not_a_sha_is_refused(): + """vcpkg_from_git requires a commit SHA; a branch name must not be accepted.""" + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + pf = root / "Build" / "vcpkg" / "ports" / "libyuv" / "portfile.cmake" + pf.write_text(LIBYUV_PORTFILE.replace("d98915a654d3564e4802a0004add46221c4e4348", "main")) + mod = load(root) + assert mod.refs_in_portfile(pf) == {} + + +def test_a_file_with_the_wrong_hash_is_not_accepted_as_cached(): + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + out = Path(t) / "out" + out.mkdir() + name = "libyuv-d98915a654d3564e4802a0004add46221c4e4348.tar.gz" + (out / name).write_bytes(b"corrupt") + assert mod.sha512(out / name) != "b" * 128 + +def test_sha512_matches_hashlib(): + import hashlib + + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + f = Path(t) / "x" + f.write_bytes(b"hello world" * 1000) + assert mod.sha512(f) == hashlib.sha512(b"hello world" * 1000).hexdigest() + + diff --git a/tests/test_maven.py b/tests/test_maven.py index ff37dcf..ae6bbc9 100644 --- a/tests/test_maven.py +++ b/tests/test_maven.py @@ -129,15 +129,10 @@ def test_missing_java_source_is_caught(): assert any(d.kind == "missing_java_src" and d.tu.endswith("B.java") for d in discs), discs - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_pie_copy_relocation.py b/tests/test_pie_copy_relocation.py new file mode 100644 index 0000000..3c809e6 --- /dev/null +++ b/tests/test_pie_copy_relocation.py @@ -0,0 +1,129 @@ +"""-fPIE on a generated cc_binary is a crash, not a nit: the qApp copy relocation. + +Case study finding 41. CMake puts -fPIE on every executable target +(CMAKE_POSITION_INDEPENDENT_CODE + an exe), the capture recorded it faithfully, +and the generator copied it into `copts` on each generated cc_binary. Bazel +appends per-target copts AFTER the .bazelrc's --copt=-fPIC, and for GCC the LAST +of -fPIC/-fPIE wins -- so the UI/Qt objects were compiled -fPIE while every +library around them was -fPIC. + +Under -fPIE, GCC may reference extern data DIRECTLY (PC-relative) rather than +through the GOT, and the linker then materialises the definition inside the +executable with an R_X86_64_COPY relocation. Against a Qt built with +`reduce_relocations` -- every official/aqt SDK; Debian's is built without it -- +that is fatal for QCoreApplication::self: + + * libQt6Core accesses `self` PC-relative (no reloc against it at all): its OWN + BSS copy is the one QApplication's constructor writes. + * libQt6Gui reads the same symbol through the GOT (R_X86_64_GLOB_DAT), and the + copy relocation has repointed that GOT slot at the EXECUTABLE's BSS. + +So qApp is set in one place and read in another, which is still null, and the +first signal emitted through it segfaults: QGuiApplication::screenAdded from +QWindowSystemInterface::handleScreenAdded, inside doActivate, on +`mov 0x8(%rdi),%rbx` with rdi = 0. Reported as a SIGSEGV in +QXcbConnection::initializeScreens; reproduced identically under the offscreen QPA +plugin, which is what proved it was not a plugin problem at all. + +Qt's own headers diagnose this ("-fPIE is not sufficient ... Compile your code +with -fPIC and without -fPIE") but only when __PIC__ is unset -- and Bazel passes +BOTH flags, so __PIC__ is defined and the #error never fires. The build was clean +and the binary was broken, which is exactly the class of defect a generator test +has to carry. + +Measured, before and after, on the six generated executables: 39 +R_X86_64_COPY relocations (including QCoreApplication::self) before, 0 after, and +the GUI starts against the aqt 6.9.2 SDK where it previously died for both the +xcb and the offscreen plugin. -Wl,-z,nocopyreloc is NOT an alternative: it turns +the same defect into a link error ("causes overflow in R_X86_64_PC32"). +""" + +import os +import re + +_HERE = os.path.dirname(__file__) +_WS = os.path.join(_HERE, "..", "examples", "ladybird", "workspace") + + +def _read(rel): + with open(os.path.join(_WS, rel)) as f: + return f.read() + + +def test_no_generated_target_carries_fpie(): + """The generated BUILD file must not put -fPIE in any copts. + + This is the regression that shipped: seven cc_binary targets (the six services + plus ladybird) each with `copts = ['-fPIE']`. + """ + build = _read("BUILD.bazel") + assert "-fPIE" not in build, \ + "-fPIE in the generated BUILD file: qApp gets a copy relocation (finding 41)" + + +def test_the_emitter_drops_fpie_rather_than_the_checked_in_file_being_edited(): + """The fix has to live in the generator, or the next regeneration undoes it. + + BUILD.bazel is generated output; hand-editing it is how a fix survives exactly + one commit. The emitter carries an explicit drop list. + """ + emit = _read("Meta/emit_build_bazel.py") + assert "DROPPED_TARGET_FLAGS" in emit + dropped = emit.split("DROPPED_TARGET_FLAGS = ", 1)[1].split("\n", 1)[0] + assert "-fPIE" in dropped + # And it must actually be consulted in the flag loop, not merely defined. + body = emit.split("def target_flags(", 1)[1].split("\ndef ", 1)[0] + assert "DROPPED_TARGET_FLAGS" in body + + +def test_the_drop_is_explained_where_it_happens(): + """A bare `if x == "-fPIE": continue` reads like a style preference. + + The next person to see CMake pass -fPIE will put it back unless the comment + says what breaks. Require the mechanism (copy relocation) and the symptom + (qApp / QCoreApplication::self) to be named at the drop site. + """ + emit = _read("Meta/emit_build_bazel.py") + head = emit.split("DROPPED_TARGET_FLAGS = ", 1)[0] + note = head[-4000:] + for token in ("copy relocation", "R_X86_64_COPY", "QCoreApplication::self", + "reduce_relocations", "-fPIC"): + assert token in note, "the -fPIE drop must explain %r" % token + + +def test_global_fpic_is_still_passed(): + """Dropping -fPIE only works because -fPIC is global. + + If the .bazelrc's --copt=-fPIC ever goes away, the objects become whatever + the host GCC defaults to (Ubuntu: -fPIE), and the copy relocation returns by + another road. + """ + rc = _read("bazelrc.txt") + assert re.search(r"^build --copt=-fPIC$", rc, re.M), \ + "global --copt=-fPIC is what makes dropping -fPIE correct" + + +def test_every_generated_executable_is_covered(): + """Not just //:ladybird: the services link Qt-adjacent libraries too. + + The measurement was taken on all six; the guard should be the same set, so a + new cc_binary that picks up -fPIE cannot slip through by not being ladybird. + """ + build = _read("BUILD.bazel") + names = re.findall(r"cc_binary\(\s*\n\s*name = '([^']+)'", build) + for expected in ("ladybird", "WebContent", "RequestServer", "ImageDecoder", + "Compositor", "WebWorker"): + assert expected in names, "expected a cc_binary for %s" % expected + # Every one of them, and any future one, must be -fPIE free -- which the + # whole-file assertion above already covers, so this is the inventory that + # keeps that assertion meaningful. + assert len(names) >= 6 + + +TESTS = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + +if __name__ == "__main__": + for t in TESTS: + t() + print("ok", t.__name__) + print("%d passed" % len(TESTS)) diff --git a/tests/test_pin_hsts_preload.py b/tests/test_pin_hsts_preload.py new file mode 100644 index 0000000..598d768 --- /dev/null +++ b/tests/test_pin_hsts_preload.py @@ -0,0 +1,260 @@ +#!/usr/bin/env python3 +"""Tests for examples/ladybird/workspace/Meta/pin_hsts_preload.py. + +The script pins Chromium's HSTS preload table DOWNSTREAM: upstream's CMake +downloads it from `main` (unversioned) and we cannot change that, so Bazel fetches +one immutable commit URL with a sha256 instead. What is worth testing is not the +HTTP (that needs the network) but the three properties that make such a pin honest, +each of which was a way to get it wrong: + + 1. the sha256 written out is MEASURED from the bytes fetched, never passed in; + 2. `--expect-same-as` is a real parity guard: if the pinned bytes differ from the + file the other build system already downloaded, the script must refuse to + write rather than emit a pin that silently changes the generated table (the + concrete failure it exists to prevent: a Chromium *release tag* serves a file + generating 168,593 entries against `main`'s ~94,600); + 3. `--check` reports without writing, so re-pinning is a deliberate act. + +Plus the trivial-but-load-bearing one: the emitted file must be valid Starlark-ish +text carrying the commit, the hash and an immutable (commit-pinned, not `main`) URL. +""" + +import importlib.util +import io +import sys +import tempfile +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPT = REPO / "examples" / "ladybird" / "workspace" / "Meta" / "pin_hsts_preload.py" +PINNED_BZL = REPO / "examples" / "ladybird" / "workspace" / "hsts_preload.bzl" + +# A commit sha and a payload standing in for the 10 MB table. +COMMIT = "3d75766484199c1fbefd269a4b168cccdb36fbca" +BLOB = b'{"entries": [{"name": "example.test", "policy": "bulk-18-weeks"}]}\n' + + +class expect_exit: + """Minimal assertRaises(SystemExit) -- this repo's tests are plain functions.""" + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + assert exc_type is SystemExit, "expected SystemExit, got %r" % (exc_type,) + self.exception = exc + return True + + +def load(responses): + """Import the script with its single network entry point stubbed. + + `responses` maps a substring of the URL -> bytes to return, so a test can serve + the commits API and the raw file differently without knowing the URL shapes. + """ + spec = importlib.util.spec_from_file_location("pin_hsts_%d" % id(responses), SCRIPT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + calls = [] + + def fake_get(url, timeout): + calls.append(url) + for needle, payload in responses.items(): + if needle in url: + return payload + raise AssertionError("unstubbed URL: %s" % url) + + mod._get = fake_get + mod.calls = calls + return mod + + +def run(mod, argv): + """Run main() capturing stdout, the way the caller redirects it into the .bzl.""" + out, err = io.StringIO(), io.StringIO() + old_out, old_err = sys.stdout, sys.stderr + sys.stdout, sys.stderr = out, err + try: + rc = mod.main(argv) + finally: + sys.stdout, sys.stderr = old_out, old_err + return rc, out.getvalue(), err.getvalue() + + +def _commit_object(): + return { + "sha": COMMIT, + "commit": { + "message": "[HSTS] Update bulk entries\n\nbody text", + "committer": {"date": "2026-07-24T21:31:29Z"}, + }, + } + + +def _responses(): + """Stubs for the three endpoints the script touches. + + The two GitHub URLs differ only after `commits`: `commits?path=...` LISTS + (a JSON array) while `commits/` DESCRIBES one (a JSON object). Keying the + stubs on that distinction is deliberate -- an earlier version of this fixture + served the array for both and the --commit path failed with a TypeError, which + is exactly the confusion a reader of the script could make too. + """ + import json + return { + "commits?path=": json.dumps([_commit_object()]).encode(), + "commits/" + COMMIT: json.dumps(_commit_object()).encode(), + "raw.githubusercontent.com": BLOB, + } + + +def test_hash_is_measured_not_asserted(): + """The emitted sha256 must be of the bytes actually fetched.""" + import hashlib + + mod = load(_responses()) + rc, out, _ = run(mod, []) + assert rc == 0 + assert hashlib.sha256(BLOB).hexdigest() in out, "the pin must carry the MEASURED hash" + + +def test_pins_a_commit_url_not_main(): + """A pin to `main` is not a pin: the URL must carry the full commit sha.""" + mod = load(_responses()) + _, out, _ = run(mod, []) + assert 'HSTS_PRELOAD_COMMIT = "%s"' % COMMIT in out + assert "/main/net/http/" not in out, "the emitted URL must not track main" + assert "net/http/transport_security_state_static.json" in out + + +def test_emitted_file_declares_the_http_file_and_extension(): + """The output has to be usable: an http_file inside a module extension.""" + mod = load(_responses()) + _, out, _ = run(mod, []) + for expected in ("http_file(", 'name = "hsts_preload_json"', "sha256 = HSTS_PRELOAD_SHA256", + "hsts_preload = module_extension("): + assert expected in out, "emitted file is missing %r" % expected + + +def test_records_what_it_pinned(): + """A reviewer needs the subject+date of the commit, not just a sha.""" + mod = load(_responses()) + _, out, _ = run(mod, []) + assert "[HSTS] Update bulk entries" in out + assert "2026-07-24" in out + + +def test_explicit_commit_is_described_not_guessed(): + """--commit must still look up the subject/date, so the pin stays self-documenting.""" + mod = load(_responses()) + _, out, _ = run(mod, ["--commit", COMMIT]) + assert "[HSTS] Update bulk entries" in out + assert any(("commits/" + COMMIT) in u for u in mod.calls), \ + "--commit should still describe the commit it pinned" + + +def test_expect_same_as_accepts_identical_bytes(): + """The parity guard passes when the pinned bytes equal the file CMake downloaded.""" + mod = load(_responses()) + with tempfile.NamedTemporaryFile(suffix=".json") as f: + f.write(BLOB) + f.flush() + rc, out, err = run(mod, ["--expect-same-as", f.name]) + assert rc == 0 + assert "parity OK" in err + assert "HSTS_PRELOAD_SHA256" in out + + +def test_expect_same_as_refuses_different_bytes(): + """The case this guard exists for: pinning a revision that changes the table.""" + mod = load(_responses()) + with tempfile.NamedTemporaryFile(suffix=".json") as f: + f.write(BLOB + b"an extra entry, i.e. a different table\n") + f.flush() + with expect_exit() as e: + run(mod, ["--expect-same-as", f.name]) + msg = str(e.exception) + assert "PARITY" in msg + assert "differ" in msg + # and it must say what to do about it, not just that it failed + assert "--commit" in msg + + +def test_expect_same_as_writes_nothing_on_failure(): + """A refused pin must not emit a partial .bzl the caller would redirect into place.""" + mod = load(_responses()) + with tempfile.NamedTemporaryFile(suffix=".json") as f: + f.write(b"totally different\n") + f.flush() + out, err = io.StringIO(), io.StringIO() + old = sys.stdout, sys.stderr + sys.stdout, sys.stderr = out, err + try: + with expect_exit(): + mod.main(["--expect-same-as", f.name]) + finally: + sys.stdout, sys.stderr = old + assert "http_file(" not in out.getvalue(), "must not emit a pin it refused" + + +def test_check_reports_without_writing(): + """--check is the dry run: the pin is reported on stderr, stdout stays empty.""" + mod = load(_responses()) + rc, out, err = run(mod, ["--check"]) + assert rc == 0 + assert out == "", "--check must write no .bzl" + assert COMMIT[:12] in err and "sha256" in err + + +def test_no_commits_for_the_path_is_an_error(): + """If GitHub returns nothing, fail loudly rather than pin the empty string.""" + mod = load({"commits?path=": b"[]", "raw.githubusercontent.com": BLOB}) + with expect_exit() as e: + run(mod, []) + assert "no commits" in str(e.exception) + + +def test_committed_pin_matches_the_scripts_own_template(): + """The committed hsts_preload.bzl must be what this script would write. + + Guards the hand-edit: hsts_preload.bzl carries a long comment explaining the + pin, and the temptation is to tweak it in place until it drifts from the + generator. Rendering the template with the committed values must reproduce the + committed file exactly. + """ + text = PINNED_BZL.read_text() + import re + + commit = re.search(r'HSTS_PRELOAD_COMMIT = "([0-9a-f]{40})"', text).group(1) + sha256 = re.search(r'HSTS_PRELOAD_SHA256 = "([0-9a-f]{64})"', text).group(1) + subject = re.search(r'# The pinned commit: "(.*)" \((\d{4}-\d\d-\d\d)\)\.', text) + size = int(re.search(r'reference build \(([\d,]+) bytes', text).group(1).replace(",", "")) + + mod = load(_responses()) + rendered = mod.TEMPLATE.format( + commit=commit, sha256=sha256, size=size, + subject=subject.group(1), date=subject.group(2), + ) + assert rendered == text, "hsts_preload.bzl has drifted from Meta/pin_hsts_preload.py" + + +def test_codegen_genrule_consumes_the_pinned_file(): + """The pin is only useful if the generator actually reads it. + + codegen_root.bzl is generated, so this asserts the wiring survived a + regeneration: gen_HSTSPreloadData must take @hsts_preload_json//file and must + NOT reference the CMake configure's download path under Build/caches. + """ + bzl = (REPO / "examples" / "ladybird" / "workspace" / "codegen_root.bzl").read_text() + rule = bzl.split("name = 'gen_HSTSPreloadData'", 1)[1].split("native.genrule(", 1)[0] + assert "@hsts_preload_json//file" in rule + assert "Build/caches/HSTSPreload" not in bzl, \ + "the unpinned CMake download path must be gone from the generated file" + + +def test_module_bazel_names_the_repo(): + """bzlmod requires every extension-created repo in a use_repo, or it is invisible.""" + mod_bazel = (REPO / "examples" / "ladybird" / "workspace" / "MODULE.bazel").read_text() + assert 'use_extension("//:hsts_preload.bzl", "hsts_preload")' in mod_bazel + assert 'use_repo(hsts, "hsts_preload_json")' in mod_bazel diff --git a/tests/test_qt_runtime.py b/tests/test_qt_runtime.py new file mode 100644 index 0000000..66fc007 --- /dev/null +++ b/tests/test_qt_runtime.py @@ -0,0 +1,432 @@ +"""The Qt RUNTIME edge: the plugins Qt dlopens must come from the Qt it linked. + +Case study finding 40. Ladybird's Bazel build linked Qt from @qt (rules_qt's +qt.local_repo, one SDK found via `qmake -query`) and then, at QApplication +construction, dlopened the QPA platform plugin from the HOST's plugin directory -- +because Qt resolves plugins against a prefix baked into libQt6Core, which for an +official SDK is EMPTY, so it falls back to the executable's directory and then the +compiled-in system path. Two Qt builds in one process: a SIGSEGV in +QXcbConnection::initializeScreens where the versions differ, and a silent pass +where they agree (which is why it survived so long in this sandbox). + +These tests guard the properties that make the fix a fix rather than a coincidence. +Each one corresponds to something that was actually wrong at some point while +writing it -- the staging depth bug, the missing use_repo, the emitter losing the +data edge on the next regeneration. + +The real proof is by removal (hide the host's Qt plugin dir, run with no +LD_LIBRARY_PATH, see the window open); this is the cheap regression guard for it. +""" + +import os +import re + +_WS = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + + +def _read(rel): + with open(os.path.join(_WS, rel)) as f: + return f.read() + + +def test_plugins_come_from_the_same_sdk_as_the_libraries(): + """The plugin path must be READ from @qt, never written down here. + + This is the whole fix in one assertion: @qt's qtconf.bzl is generated by + rules_qt from `qmake -query`, so taking QT_INSTALL_PLUGINS from it makes a + plugin/library version skew unrepresentable. A hardcoded + /usr/lib/x86_64-linux-gnu/qt6/plugins -- which is what the build effectively + had, by falling back to it -- is the bug. + """ + bzl = _read("qt_runtime.bzl") + assert 'Label("@qt//:qtconf.bzl")' in bzl, \ + "the plugin dir must be read from @qt's own generated qtconf.bzl" + assert 'values.get("QT_INSTALL_PLUGINS"' in bzl + # And no ABSOLUTE host plugin path may be a value the code uses. Getting this + # assertion right took three tries, and the near-misses are worth naming because + # each is a legitimate mention of such a path: + # * comments describing the bug (/usr/lib/x86_64-linux-gnu/qt6/plugins IS the + # wrong directory being explained -- a test that cannot tell prose from a + # path punishes the explanation), + # * _SYSTEM_LIB_DIRS, which lists system lib dirs precisely in order to + # RECOGNISE a distro Qt and stage nothing, + # * error text and docstrings naming plugins/platforms/libqxcb.so as an + # example. + # What must not appear is an absolute /usr/... path used as a plugin root, so the + # check is on absolute-path STRING LITERALS outside the system-dir list. + code = "\n".join([l for l in bzl.splitlines() + if not l.lstrip().startswith("#")]) + code = code.split("_SYSTEM_LIB_DIRS = [", 1) + code = code[0] + (code[1].split("]", 1)[1] if len(code) > 1 else "") + # (`/plugins/` is the path-COMPONENT matcher from the staging bug below, not a + # host path -- an absolute host path starts with a real top-level directory.) + host_roots = ("/usr/", "/opt/", "/home/", "/lib/", "/etc/") + for literal in re.findall(r'"(/[^"]*)"', code): + assert not literal.startswith(host_roots), \ + "a hardcoded host path defeats the point: %s" % literal + + +def test_qt_version_floor_is_checked(): + """finding 40's class: upstream declares a floor and the overlay must check it. + + UI/Qt/CMakeLists.txt has `find_package(Qt6 6.9 REQUIRED COMPONENTS Core + Widgets)`. CMake refuses an older Qt with a clear message; the Bazel build + compiled against whatever qmake was first on PATH. + """ + bzl = _read("qt_runtime.bzl") + assert "_QT_FLOOR = (6, 9)" in bzl, \ + "the floor must match UI/Qt/CMakeLists.txt's find_package(Qt6 6.9 REQUIRED)" + # The failure has to name the fix, not just the fact (finding 39's rule). + msg = bzl.split("qt_plugins: Qt {have} is too old", 1)[1].split(" ))", 1)[0] + assert "qt6-base-dev" in msg, "the error must name the package to install" + assert "MODULE.bazel" in msg, "the error must say where the SDK is chosen" + assert "{have}" in msg and "{prefix}" in msg, \ + "and the Qt it actually found, and where -- 'too old' alone is not actionable" + + +def test_missing_platform_plugin_is_an_error_not_an_empty_filegroup(): + """A check that cannot fail must not look like one that passed (finding 35). + + An SDK with no platforms/ directory yields an empty staging tree, a clean + build, and a GUI that cannot start. Same shape as glob(allow_empty = True). + """ + bzl = _read("qt_runtime.bzl") + assert '"platforms" not in groups' in bzl, \ + "a Qt with no QPA plugin must be a fetch-time failure" + + +def test_staging_strips_the_repo_name_not_the_substring(): + """`plugins/` is a substring of `qt_plugins/` -- and that was a real bug. + + Searching for "plugins/" in ../qt_plugins/plugins/platforms/libqxcb.so matches + inside the REPOSITORY NAME, so everything staged one directory too deep + (bazel-bin/plugins/plugins/...). It built cleanly and pointed qt.conf at an + empty tree. + """ + bzl = _read("qt_runtime.bzl") + assert 'find("/plugins/")' in bzl, \ + "the match must be on the separated path component, not the substring" + + +def test_plugins_are_symlinked_not_copied(): + """A plugin finds its own SDK's private libs through RUNPATH $ORIGIN/../../lib. + + $ORIGIN resolves from the object's REAL path, so a symlink keeps that working + and a copy breaks it -- reintroducing the very bug (the copy then resolves + libQt6XcbQpa.so.6 from /usr). + """ + bzl = _read("qt_runtime.bzl") + assert "ctx.actions.symlink(" in bzl, "the staged plugins must be symlinks" + assert "ctx.actions.copy" not in bzl + + +def test_qt_conf_replaces_the_prefix_relatively(): + """Prefix = . is what makes ONE file right in bazel-bin AND in runfiles. + + Setting Prefix at all is what stops /usr being scanned (it REPLACES the + compiled-in prefix rather than being appended to it). An absolute path would + be a host escape and would be wrong in one of the two layouts. + """ + bzl = _read("qt_runtime.bzl") + written = bzl.split("def _qt_conf_impl", 1)[1] + assert '"[Paths]"' in written + assert '"Prefix = ."' in written, "the prefix must be relative to the executable" + assert '"Plugins = {}".format(ctx.attr.plugins_dir)' in written + + +def test_the_binary_declares_them_as_runtime_inputs(): + """Staged files nobody depends on are files that are not there in a clean build.""" + build = _read("BUILD.bazel") + binary = build.split("name = 'ladybird'", 1)[1].split("\n)\n", 1)[0] + assert "':qt_conf'" in binary and "':qt_plugins'" in binary, \ + "//:ladybird must carry qt.conf and the plugin tree in data" + assert "'@qt_plugins//:runtime_libs'" in binary, \ + "and the SDK's private libraries as link inputs, or an SDK-bundled ICU " \ + "needs LD_LIBRARY_PATH at run time" + assert "qt_plugin_tree(" in build and "qt_conf(" in build + + +def test_the_emitter_generates_that_edge(): + """BUILD.bazel is GENERATED; an edge only in the file is an edge one run deletes. + + The same drift this project keeps finding (finding 37): a hand-appended tail + the emitter silently truncates on its next run. + """ + emitter = _read("Meta/emit_build_bazel.py") + assert "QT_RUNTIME_BLOCK" in emitter and "emit_qt_runtime()" in emitter + assert 'extra_data=[":qt_conf", ":qt_plugins"]' in emitter + assert '"@qt_plugins//:runtime_libs"' in emitter, \ + "the private-libs dep must come from qt_label()'s resolution, not by hand" + assert 'load(":qt_runtime.bzl", "qt_conf", "qt_plugin_tree")' in \ + emitter.split("ALL_LOADS = ", 1)[1], "the generated file needs the load()" + + +def test_module_bazel_names_the_repo(): + """bzlmod requires every extension-created repo in a use_repo, or it is invisible.""" + mod = _read("MODULE.bazel") + assert 'use_extension("//:qt_runtime.bzl", "qt_runtime")' in mod + assert 'use_repo(qt_runtime, "qt_plugins")' in mod + # Still kklochkov's rules_qt, still discovering the host SDK: this change adds + # the runtime half, it does not swap the ruleset or pin a different Qt. + assert 'bazel_dep(name = "rules_qt", version = "2.0.1")' in mod + assert "qt.local_repo(" in mod + + +def test_every_plugin_type_is_staged(): + """Not a hand-picked list of the four types the GUI needs today. + + A list drifts, symlinks cost nothing, and a missing input method or native file + dialog is a defect nobody notices for a month. + """ + bzl = _read("qt_runtime.bzl") + staging = bzl.split("def _qt_plugins_impl", 1)[1].split( + "qt_plugins = repository_rule(", 1)[0] + assert "root.readdir()" in staging, "the plugin types must be discovered" + for hardcoded in ('"platformthemes"', '"styles"', '"imageformats"'): + assert hardcoded not in staging, \ + "a hardcoded plugin type list is the thing to avoid (%s)" % hardcoded + + +def test_repo_rule_is_local(): + """The SDK is a host path; a cached repo would survive changing it. + + rules_qt's own qt.local_repo is local = True for the same reason. + """ + bzl = _read("qt_runtime.bzl") + rule = bzl.split("qt_plugins = repository_rule(", 1)[1].split(")\n", 1)[0] + assert "local = True" in rule + + + +def test_a_qt_module_upstream_starts_requiring_resolves_without_an_edit(): + """The Qt mapping is a RULE, because the three-entry table went stale. + + CMake names Qt targets Qt6; rules_qt names one cc_library per module + of the discovered SDK as @qt//:Qt. That is a rename, and it was + written as a dict of the three modules Ladybird used when it was measured + (Core, Gui, Widgets). At 71fb301a upstream turned Qt6::Positioning from + OPTIONAL to REQUIRED for UI/Qt/GeolocationProviderQt.cpp; the dict had no + key, so the dep became an UNKNOWN and //:ladybird failed to compile with + "QGeoPositionInfo: No such file or directory". + + A table of the deps a project has today cannot express "and whatever it needs + tomorrow", so this asserts the derivation -- including that a NON-Qt name + still falls through (returning a label for it would put a nonexistent + @qt//:X in the build) . + """ + import importlib.util + os.environ["LADYBIRD_ROOT"] = _WS + spec = importlib.util.spec_from_file_location( + "_ebb_src", os.path.join(_WS, "Meta", "emit_build_bazel.py")) + src = _read("Meta/emit_build_bazel.py") + # Exec just the function, not the module: emit_build_bazel does real work at + # import time (it parses the reference build's build.ninja), and this is a + # pure string rule. + ns = {} + body = "def qt_label" + src.split("def qt_label", 1)[1].split("\ndef ", 1)[0] + exec(body, ns) + qt_label = ns["qt_label"] + assert qt_label("Qt6Core") == "@qt//:QtCore" + assert qt_label("Qt6Gui") == "@qt//:QtGui" + assert qt_label("Qt6Widgets") == "@qt//:QtWidgets" + # THE regression: the module nobody wrote down. + assert qt_label("Qt6Positioning") == "@qt//:QtPositioning" + # And modules Ladybird does not use today, to show the rule is not a longer list. + assert qt_label("Qt6Network") == "@qt//:QtNetwork" + assert qt_label("Qt6Multimedia") == "@qt//:QtMultimedia" + # Not Qt: must NOT resolve, so it is reported as UNKNOWN rather than becoming + # a label for a target that does not exist. + for other in ("xkbcommon", "glib-2.0", "OpenGL", "Qt6", "vulkan", ""): + assert qt_label(other) is None, other + # Structural: no dict may map Qt module names to labels again. + assert "QT_MAP" not in src, "the Qt table is back" + + +def test_the_moc_list_follows_what_cmake_compiles_not_a_name(): + """A conditionally-compiled Q_OBJECT header, decided by measurement. + + moc_headers() returns the UI/Qt headers CMake's AUTOMOC would moc. It ended + with a hardcoded `not h.endswith("GeolocationProviderQt.h")`, justified in a + docstring by "Qt6::Positioning is not found in this configuration" -- a fact + about the machine that captured it. At 71fb301a Positioning is required and + CMake compiles GeolocationProviderQt.cpp, so the exclusion silently dropped a + real moc target. Wrong in both directions: moc'ing a header CMake does not is + a target only Bazel builds; skipping one it does is a missing vtable at link. + + The condition is now "does the reference build compile the sibling .cpp", + which is in the model, so it cannot disagree with CMake. + """ + src = _read("Meta/emit_build_bazel.py") + fn = src.split("def moc_headers", 1)[1].split("\ndef ", 1)[0] + assert "GeolocationProviderQt" not in fn.split('"""', 2)[-1], "moc_headers filters by FILE NAME again" + assert "compiled" in fn and "CppCompile" in fn, "the filter must read the reference build's compile list" + # And the generated file reflects it: the header is moc'd at this pin. + build = _read("BUILD.bazel") + moc = build.split("qt_cc_moc(", 1)[1].split(")", 1)[0] + assert "UI/Qt/GeolocationProviderQt.h" in moc, moc + # ...paired with the dep that makes it compile, or the moc output does not build. + assert "'@qt//:QtPositioning'" in build + + +def test_a_dropped_absolute_include_root_is_reported_not_silently_skipped(): + """The emitter must not hide the difference between its input and .bazelrc. + + An absolute include root (/usr/include/glib-2.0) cannot be a per-target + copt -- Bazel rejects a path outside the execution root even as -isystem -- + so the emitter drops it and .bazelrc carries it globally as + CPLUS_INCLUDE_PATH. The drop was a bare `continue`, which means a root CMake + compiles with and .bazelrc lacks produces NO output at all. + + That is how the last failure of this repin happened, and I caused it: having + just fixed four hand-copied facts, I hand-copied three of glib's roots out of + the model and missed the rest. The build failed ~3,800 actions later on + `UI/Qt/ExternalURLHandler.cpp:19: fatal error: gio/gdesktopappinfo.h: No such + file or directory` -- gio-unix-2.0 is a separate root from glib-2.0, and + blkid/libmount/sysprof-6 arrive transitively through glib's pkg-config. + Re-running the emitter with the check in place named all four at once. + + Deliberately a WARNING, not a failure: which host escapes are acceptable is a + judgement (README gap 3), and Qt's roots are correctly absent because + rules_qt carries them on the dep edge. What is not a judgement is whether the + emitter says anything at all. + """ + src = _read("Meta/emit_build_bazel.py") + # The bare skip is gone. + assert "record_host_include(i)" in src, \ + "absolute include roots are dropped without being recorded" + assert "def report_host_includes" in src and "report_host_includes()" in \ + src.split("def report_host_includes", 1)[1], \ + "the shortfall is collected but never reported" + # The roots that ride on a dep edge must be exempt, or the warning cries wolf + # about Qt on every run and gets ignored -- which is how a real one hides. + exempt = src.split("HOST_INCLUDE_EXEMPT = ", 1)[1].split(")", 1)[0] + assert "qt6" in exempt and "vcpkg_installed" in exempt, exempt + + # And .bazelrc actually carries the six roots this configuration needs, glib's + # non-obvious ones included. This is the assertion that would have failed + # while the build was broken. + rc = _read("bazelrc.txt") + paths = set() + for m in re.finditer(r"CPLUS_INCLUDE_PATH=(\S+)", rc): + paths |= set(m.group(1).split(":")) + for root in ("/usr/include/libdrm", "/usr/include/glib-2.0", + "/usr/lib/x86_64-linux-gnu/glib-2.0/include", + "/usr/include/gio-unix-2.0", "/usr/include/blkid", + "/usr/include/libmount", "/usr/include/sysprof-6"): + assert root in paths, "%s missing from CPLUS_INCLUDE_PATH" % root + # Target and exec configs must agree: a root in one and not the other is the + # finding-26 skew, and here it would mean a genrule tool that cannot compile. + envs = re.findall(r"--(?:host_)?action_env=CPLUS_INCLUDE_PATH=(\S+)", rc) + assert len(envs) == 2, envs + assert envs[0] == envs[1], "target and exec CPLUS_INCLUDE_PATH differ" + +TESTS = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + +if __name__ == "__main__": + for t in TESTS: + t() + print("ok", t.__name__) + print("%d passed" % len(TESTS)) + + +def test_every_qt_module_the_build_links_is_preflighted_with_its_package(): + """A Qt module the HOST lacks must be named here, not by Bazel's output base. + + rules_qt's qt.local_repo DERIVES its cc_library targets by listing the host + Qt's lib directory (`_create_libs_symlinks`, qt_local_repo.bzl), so a module + the host does not have is simply never declared. What the reader then sees is: + + ERROR: .../external/rules_qt++qt+qt/BUILD.bazel: no such target + '@@rules_qt++qt+qt//:QtPositioning': target 'QtPositioning' not declared in + package '' ... and referenced by '//:ladybird' + + Ulf hit exactly that. It says nothing about Qt, nothing about apt, and points + at a generated file in an output base that the reader did not write and cannot + fix. Same class as finding 38 (an unpinned host tool) and as the four failed + configures of the 71fb301a repin: a host probe whose absence is reported as a + defect in your own code. + + The version FLOOR check next to it was already the right idea with the wrong + scope -- "is the SDK new enough" but never "does the SDK have the parts we + link". Both are properties of the discovered SDK. + + This test is the anti-drift half: _QT_MODULES is an INDEPENDENT statement of + what the build needs (deriving it from BUILD.bazel would only prove the + generator agrees with itself), so it can go stale exactly the way the qt_label + table did. Asserting the two agree is what catches a Qt module appearing in a + future repin without a preflight entry. + """ + rt = _read("qt_runtime.bzl") + build = _read("BUILD.bazel") + + # What the build actually links. + linked = set(re.findall(r"@qt//:(Qt\w+)", build)) + assert linked, "no @qt//:Qt* deps in BUILD.bazel -- did the label scheme change?" + + # What the preflight knows about. + block = rt.split("_QT_MODULES = {", 1)[1].split("\n}", 1)[0] + # (module -> (deb package, aqt module)): BOTH forms, because which one is + # correct advice depends on the SDK -- see the venv case below. + preflighted = dict(re.findall(r'"(Qt\w+)":\s*\("([^"]+)",\s*"([^"]+)"\)', + block.replace("\n", " ")) and + [(m, (d, a)) for m, d, a in + re.findall(r'"(Qt\w+)":\s*\("([^"]+)",\s*"([^"]+)"\)', block)]) + assert preflighted, "_QT_MODULES is empty or unparseable" + + missing = sorted(linked - set(preflighted)) + assert not missing, ( + "these Qt modules are linked by BUILD.bazel but have no _QT_MODULES entry, " + "so a host without them gets Bazel's 'no such target' error naming a " + "generated BUILD file instead of the apt package: %s" % missing) + + # Every entry must name a package, or the message cannot tell anyone what to do. + for mod, (deb, aqt) in sorted(preflighted.items()): + assert deb and not deb.startswith("Qt"), \ + "%s maps to %r, which is not a package name" % (mod, deb) + assert aqt and not aqt.startswith("Qt"), \ + "%s has no aqt module name (%r); a venv/aqt SDK cannot be told what " \ + "to install" % (mod, aqt) + + # The regression that motivated this: Positioning became REQUIRED at 71fb301a. + assert preflighted.get("QtPositioning") == ("qt6-positioning-dev", "qtpositioning"), \ + "QtPositioning must name both the deb and the aqt module" + + +def test_the_module_preflight_reads_the_same_libs_dir_rules_qt_derives_from(): + """The check must ask the question qt.local_repo will answer, not a proxy. + + A repository rule cannot query another repo's targets, so "will @qt declare + QtPositioning?" has to be answered from the same input qt.local_repo uses: + QT_INSTALL_LIBS, listed for libQt.so*. Anything else (asking + for the target, probing a header, trusting the version) can disagree with what + rules_qt actually does, which would give a false pass or a false failure. + + Verified against real Bazel both ways: a prefix with libQt6Positioning.so* + removed fails with the apt package named, and the same prefix with it restored + resolves @qt_plugins//:runtime_libs. + """ + rt = _read("qt_runtime.bzl") + impl = rt.split("def _qt_plugins_impl", 1)[1] + assert "QT_INSTALL_LIBS" in impl, \ + "the module preflight does not read QT_INSTALL_LIBS, the dir rules_qt derives from" + # Must be the same name-mangling as _create_lib_name: first dot-field, prefix + # stripped -- libQt6Positioning.so.6.10.2 is QtPositioning. + assert 'split(".")' in impl, \ + "libQt6Positioning.so.6.10.2 must reduce to QtPositioning like _create_lib_name does" + # The failure has to be actionable: package names and the cache caveat. + # The install advice lives in _install_hint; the surrounding fail() carries the + # context. Both are part of what the reader sees. + hint = rt.split("def _install_hint", 1)[1].split("\ndef ", 1)[0] + assert "apt install" in hint, "no distro instruction for a distro Qt" + assert "aqt" in hint, "no self-contained-SDK instruction for a venv/aqt Qt" + assert "CANNOT fix it" in hint, \ + ("a self-contained SDK is not told that apt cannot help: apt installs into " + "/usr/lib, which that SDK never reads, so following the advice changes " + "nothing and the reader concludes the message was wrong") + assert "sync --configure" in impl or "clean --expunge" in impl, \ + ("the failure does not mention that @qt is cached, so a reader who installs " + "the package and re-runs can get the same error and conclude it did not work") + # The SDK must be NAMED, or a reader with two Qts cannot tell which one failed. + assert "QT_INSTALL_PREFIX" in impl and "libs" in impl, \ + "the failure does not name the prefix/lib dir it probed" diff --git a/tests/test_run_all.py b/tests/test_run_all.py new file mode 100644 index 0000000..2e37733 --- /dev/null +++ b/tests/test_run_all.py @@ -0,0 +1,126 @@ +"""The runner's own guards, exercised rather than asserted. + +run_all.py exists because three test files silently ran zero tests (case study +finding 35). Its whole value is three guards -- a file with no tests fails, a file +that will not import fails, a failing test fails -- and a guard that has never +been seen to fire is exactly the thing this repo keeps getting caught by. So each +one is triggered here against a throwaway tests/ directory. + +Note what is NOT done: this does not import run_all and inspect it. It runs it as +a subprocess and checks the EXIT CODE, because the exit code is what a commit gate +or CI reads, and an exit code is precisely what the original bug got wrong (0 while +running nothing). +""" + +import os +import shutil +import subprocess +import sys +import tempfile + +_RUNNER = os.path.join(os.path.dirname(os.path.abspath(__file__)), "run_all.py") + + +def _run(files): + """Run the runner over a temp tests/ dir containing exactly `files`. + + A temp dir rather than the real one so a guard can be triggered without a + file that breaks the actual suite ever existing on disk. + """ + d = tempfile.mkdtemp() + try: + runner = os.path.join(d, "run_all.py") + shutil.copy(_RUNNER, runner) + for name, text in files.items(): + with open(os.path.join(d, name), "w") as f: + f.write(text) + p = subprocess.run([sys.executable, runner], capture_output=True, text=True) + return p.returncode, p.stdout + p.stderr + finally: + shutil.rmtree(d) + + +def test_a_passing_file_passes_and_reports_its_count(): + rc, out = _run({"test_ok.py": "def test_a(): pass\ndef test_b(): pass\n"}) + assert rc == 0, out + assert "2/2 tests passed across 1 files" in out + + +def test_a_file_that_defines_no_tests_is_a_failure(): + """THE bug: 46 test functions defined in files nobody ran, exit code 0. + + A module with no test callables is not "nothing to do" -- it is a file whose + contents stopped being reachable, which is what happened here for three files + and what `allow_empty = True` did for the Build/full shims. + """ + rc, out = _run({"test_silent.py": "def helper(): pass\n"}) + assert rc != 0, out + assert "defines no tests" in out + + +def test_a_module_that_cannot_be_imported_is_a_failure_not_a_skip(): + """An unimportable module reported zero tests before, which looked like zero + failures. Import errors are results, not absences.""" + rc, out = _run({"test_broken.py": "import nonexistent_xyz\ndef test_a(): pass\n"}) + assert rc != 0, out + assert "could not be imported" in out + + +def test_a_failing_test_fails_the_run_and_is_named(): + rc, out = _run({"test_bad.py": "def test_a(): assert False, 'boom'\n", + "test_ok.py": "def test_b(): pass\n"}) + assert rc != 0, out + assert "test_bad.py::test_a" in out + # The other file still ran: one bad file must not mask the rest. + assert "1/2 tests passed" in out + + +def test_an_empty_tests_directory_is_a_failure(): + """The discovery equivalent of allow_empty = False. If the glob matches + nothing, the glob is wrong -- it never means there is nothing to test.""" + rc, out = _run({}) + assert rc != 0, out + assert "no tests/test_*.py files found" in out + + +def test_helpers_and_imported_functions_are_not_counted_as_tests(): + """A `test_`-prefixed name imported FROM another module is not this file's + test; counting it would double-count and, worse, make a file look non-silent + because of somebody else's tests.""" + rc, out = _run({ + "test_one.py": "def test_a(): pass\n", + "test_two.py": "from test_one import test_a\ndef test_b(): pass\n", + }) + assert rc == 0, out + assert "2/2 tests passed across 2 files" in out + + +def test_the_real_suite_is_discovered_whole(): + """Every test file in this repo is reached by discovery -- the check that the + README's hand-kept list of six filenames could not make. + + This calls `test_files()` directly instead of running the runner on the real + tests/ directory, and the reason is a trap worth recording: this file IS in + that directory, so a subprocess here would run the runner, which would run + this test, which would run the runner... The first version of this test hung + the suite. Discovery is the thing being claimed, and `test_files()` is + discovery, so the direct call is also the more precise assertion. + """ + sys.path.insert(0, os.path.dirname(_RUNNER)) + try: + import run_all + finally: + sys.path.pop(0) + here = os.path.dirname(os.path.abspath(__file__)) + on_disk = sorted(f for f in os.listdir(here) + if f.startswith("test_") and f.endswith(".py")) + assert run_all.test_files() == on_disk + assert os.path.basename(__file__) in on_disk, "the runner's own tests" + +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_run_recipe.py b/tests/test_run_recipe.py new file mode 100644 index 0000000..8e759d1 --- /dev/null +++ b/tests/test_run_recipe.py @@ -0,0 +1,250 @@ +"""The documented run recipe must not stage a COPY of the build's own outputs. + +Ladybird's UI finds its helper processes through +`WebView::get_paths_for_helper_process()`, which searches, in order: + + /libexec/ <-- first + /bin/ + / + ./ + +Under Bazel the services are already siblings of `ladybird` in `bazel-bin`, i.e. +the build output IS on that chain. The README's run recipe nevertheless used to +`cp` them into `$BIN/libexec/`, which puts a second copy on the chain AHEAD of the +real one -- a cache with no invalidation, in a tree whose whole point is that +Bazel decides what is stale. + +It cost a day. After the 71fb301a repin the fresh UI kept talking to WebContent +binaries left in `bazel-out/k8-fastbuild/libexec/` by a staging run from the +PREVIOUS pin, six weeks earlier. Upstream had inserted IPC messages, so every +message id past the insertion point had shifted by one: the endpoint magic +matched (right endpoint) and the payload did not parse, giving + + Failed to parse IPC message: + Local endpoint error: Can't read past the end of the stream memory + Peer endpoint error: Endpoint magic number mismatch, not my message! + +~14,000 times, while all 20 generated `*Endpoint.h` were byte-identical to +CMake's -- so every check aimed at the code generator said the build was fine, +because it was. The failing artifact was not built by the build. + +The fix is to delete the staging step, not to refresh it: verified by removal, +with no `libexec/` anywhere, `--headless=text` and `--headless=layout-tree` are +byte-identical to the CMake reference at the same pin. + +These tests guard the recipe, because the recipe is the interface: it is what +Ulf runs, and a stale binary it silently prefers is indistinguishable from a +miscompile. +""" + +import os +import re + +_HERE = os.path.dirname(__file__) +_EXAMPLE = os.path.join(_HERE, "..", "examples", "ladybird") + + +def _readme(): + with open(os.path.join(_EXAMPLE, "README.md")) as f: + return f.read() + + +def _run_recipe_shell(): + """The ```sh block of the run recipe -- the part a reader copy-pastes. + + Keyed off the resource-root assignment rather than a heading, so reordering + the prose does not silently make these tests vacuous. + """ + blocks = re.findall(r"```sh\n(.*?)```", _readme(), re.S) + hits = [b for b in blocks if 'share/Lagom' in b and 'bazel info bazel-bin' in b] + assert len(hits) == 1, \ + f"expected exactly one run recipe block, found {len(hits)}" + return hits[0] + + +def test_the_recipe_does_not_copy_services_into_libexec(): + """The assertion that would have failed while the browser was broken. + + Any copy INTO a libexec directory is the bug, whatever it is spelled with -- + cp, install, ln -s -- because the destination is searched before bin/. + """ + sh = _run_recipe_shell() + for line in sh.splitlines(): + code = line.split("#", 1)[0] + if not code.strip(): + continue + assert not re.search(r"(cp|install|ln)\s.*libexec", code), \ + ("the run recipe stages a copy of the build's outputs into libexec, " + "which Ladybird searches BEFORE bin/ -- so the copy shadows the " + f"build and can go stale: {line.strip()}") + + +def test_the_recipe_removes_a_libexec_left_by_an_older_recipe(): + """Deleting the step is not enough: the directory it made still shadows. + + Anyone who ran the previous recipe has one on disk, and it keeps winning + forever -- it is not an output of any target, so no `bazel clean` removes it + and no rebuild refreshes it. The recipe has to actively clear it. + """ + sh = _run_recipe_shell() + assert re.search(r"rm -rf\s+[^\n]*libexec", sh), \ + "the recipe must delete a libexec/ left behind by the older recipe" + + +def test_the_shadowing_lookup_order_is_written_down(): + """Why no staging is needed is the non-obvious part; state it or lose it. + + Without the lookup order in the text, "don't stage into libexec" reads like + a style preference and the next person helpfully re-adds it. + """ + # Collapse whitespace first: the README is hard-wrapped at 80 columns, so + # where the line breaks fall is arbitrary and must not decide the assertion. + readme = re.sub(r"\s+", " ", _readme()) + assert "get_paths_for_helper_process" in readme, \ + "the function that defines the search order is not named" + assert re.search(r"libexec.{0,80}\bbefore\b.{0,40}bin", readme), \ + "the README does not say libexec is searched BEFORE bin" + # And the symptom, so the next person greps the error and lands here rather + # than re-auditing the IPC code generator (which was innocent). + assert "Endpoint magic number mismatch" in readme, \ + "the IPC symptom this produces is not documented" + + +def test_the_resource_root_recipe_survives_a_stale_share_symlink(): + """Same class, second instance: `share` was a symlink into CMake's tree. + + An older recipe pointed `/../share` at `Build/full/share`. Once that + build directory moved, `mkdir -p` on a path under it failed -- reported as + "File exists" for a path that does not exist, which reads like a bug in + mkdir. `rm -rf` does not remove a dangling symlink's target problem; only + removing the LINK does. + """ + sh = _run_recipe_shell() + assert re.search(r"rm -f\s+\"?\$\(dirname\s+\"?\$BIN\"?\)\"?/share", sh), \ + "the recipe does not clear a `share` symlink left pointing into CMake's tree" + + +def test_the_diagnostic_checks_for_a_shadowing_libexec_before_it_checks_qt(): + """The stale-libexec failure must be ruled out FIRST, because it mimics Qt. + + It has now bitten twice, and both times it arrived disguised: a SIGILL or + `VERIFICATION FAILED` with a Qt-flavoured backtrace (QApplicationPrivate, + QEventDispatcherGlib, libQt6Core frames), preceded by `Endpoint magic number + mismatch, not my message!` on every IPC message. Nothing in that picture says + "you are running binaries from a previous build" -- but the PATHS in the + backtrace do: `ladybird` from bazel-out/.../bin/ and `Compositor` from + bazel-out/.../libexec/. + + So the Qt diagnostic checks it before any Qt question, or it confidently + investigates the wrong subsystem. (Todo 4a93a257: for a "built fine, behaves + wrong" bug, ask what is EXECUTING before auditing what produced it.) + """ + script = os.path.join(_EXAMPLE, "qt_runtime_diagnose.sh") + assert os.path.isfile(script), "the Qt runtime diagnostic is missing" + with open(script) as f: + t = f.read() + assert "libexec" in t, \ + "the diagnostic never checks for a shadowing libexec, the likelier cause" + # Before the Qt sections: a diagnostic that asks about Qt first sends the + # reader into the wrong subsystem, which is exactly what happened. + libexec_at = t.index("libexec") + qt_at = min(t.index("MODULE.bazel"), t.index("qtconf.bzl")) + assert libexec_at < qt_at, \ + ("the libexec check must come BEFORE the Qt checks: it mimics a Qt crash " + "and is the more common cause") + # It must name the lookup order, or "delete libexec" is a superstition. + assert re.search(r"libexec.{0,120}\bFIRST\b", t, re.S | re.I), \ + "the diagnostic does not say libexec is searched FIRST (why the copy wins)" + # And it must print what each service RESOLVES to, which is the datum the + # backtrace carried and no build check produces. + assert "SHADOWS" in t, \ + "the diagnostic does not report which copy of each service would run" + for svc in ("Compositor", "WebContent", "RequestServer"): + assert svc in t, f"the resolution check does not cover {svc}" + + +def test_the_diagnostic_unsets_ld_library_path_when_it_runs_the_binary(): + """A run with LD_LIBRARY_PATH set cannot tell you whether you need it. + + Ulf's run.sh sets LD_LIBRARY_PATH=~/Qt/6.9.2/gcc_64/lib, which masks BOTH Qt + runtime failures (the missing bundled ICU, and the wrong-SDK QPA plugin). The + diagnostic exists to say which one fired, so it must run the binary WITHOUT it. + """ + script = os.path.join(_EXAMPLE, "qt_runtime_diagnose.sh") + with open(script) as f: + t = f.read() + assert "env -u LD_LIBRARY_PATH" in t, \ + ("the diagnostic must run the binary with LD_LIBRARY_PATH unset -- with it " + "set, both failures disappear and the run proves nothing") + + +def test_ladybird_declares_the_services_it_spawns_as_data(): + """`bazel build //:ladybird` must rebuild the services too. + + The services are found by PATH at runtime (get_paths_for_helper_process), not + linked, so nothing in the build graph said the browser needs them: a build of + //:ladybird alone left whatever WebContent happened to be in bazel-bin from a + previous build. Ulf hit it in its most confusing form -- ladybird dated Aug 20 + beside a WebContent dated Aug 11, a browser from this pin talking to a service + from the previous one. Upstream had inserted ~3 IPC messages between the pins, + shifting every id after them, so every message failed to decode with + + Local endpoint error: Can't read past the end of the stream memory + Peer endpoint error: Endpoint magic number mismatch, not my message! + + which reads like a codegen or ABI bug and is nothing of the kind. (The magic + 0xffa5367a is AK::string_hash("WebContentServer"), i.e. the CORRECT endpoint; + the message NUMBERING is what disagreed -- 7/7 against the old pin, 0/7 against + the new one.) His fix: declare them. + + `data`, not `deps`: separate processes, not link inputs -- the relationship + LibWasm already has to cranelift-compiler. + """ + build = os.path.join(_EXAMPLE, "workspace", "BUILD.bazel") + with open(build) as f: + text = f.read() + # The ladybird cc_binary block. + i = text.index("name = 'ladybird'") + block = text[i:text.index("\n)", i)] + m = re.search(r"data = \[([^\]]*)\]", block) + assert m, "//:ladybird declares no data at all" + data = m.group(1) + for svc in ("Compositor", "ImageDecoder", "RequestServer", "WebContent", "WebWorker"): + assert f"':{svc}'" in data, ( + f"//:ladybird does not declare :{svc} in data, so `bazel build " + "//:ladybird` can leave a STALE copy of it in bazel-bin and the IPC " + "message ids will not match") + # deps would be wrong: they are processes, not libraries to link. + deps = re.search(r"deps = \[(.*?)\]", block, re.S) + if deps: + assert "':WebContent'" not in deps.group(1), \ + "the services must be data (spawned processes), not deps (link inputs)" + + +def test_the_spawned_service_list_is_derived_from_ladybirds_own_source(): + """A hand-kept list of services is one new service away from the same bug. + + Upstream ADDS services (Compositor is new since the previous pin), and the + names are already written down in the place the runtime lookup uses them: the + string literals passed to launch_server_process<> in HelperProcess.cpp. So the + emitter reads them from there, and fails loudly if it parses none -- an empty + list would silently re-emit the bug it exists to prevent. + """ + emitter = os.path.join(_EXAMPLE, "workspace", "Meta", "emit_build_bazel.py") + with open(emitter) as f: + text = f.read() + assert "def spawned_services" in text, "the emitter has no spawned_services()" + assert "HelperProcess.cpp" in text, \ + "the service list is not read from HelperProcess.cpp (hand-listed?)" + assert "launch_server_process" in text, \ + "the service names must come from the launch_server_process<> call sites" + fn = text[text.index("def spawned_services"):] + fn = fn[:fn.index("\ndef ")] + # No literal roster inside the function: that is the thing being avoided. + for svc in ("WebContent", "RequestServer", "ImageDecoder", "WebWorker"): + assert f'"{svc}"' not in fn and f"'{svc}'" not in fn, \ + f"spawned_services() hardcodes {svc} instead of deriving it" + # An empty parse must be fatal, not an empty list. + assert "sys.exit" in fn, \ + ("spawned_services() must FAIL when it parses no service names -- returning " + "[] would silently rebuild //:ladybird without its services again") diff --git a/tests/test_triage.py b/tests/test_triage.py index 679324e..2c2ca99 100644 --- a/tests/test_triage.py +++ b/tests/test_triage.py @@ -60,15 +60,10 @@ def test_render_caps_bazel_only_tail(): # cmake_only (actionable) is never capped assert "-std=gnu++17" in text - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_vcpkg_plumbing.py b/tests/test_vcpkg_plumbing.py index 39a6033..68e95eb 100644 --- a/tests/test_vcpkg_plumbing.py +++ b/tests/test_vcpkg_plumbing.py @@ -141,3 +141,244 @@ def test_static_libs_go_on_the_link_line_by_path(): assert 'root + "/lib/lib" + n + ".a"' in impl # and the shared ones are -l, so the loader's SONAME lookup keeps working assert 'flags.append("-l" + n)' in impl + + +# --------------------------------------------------------------------------- +# Finding 36: what a fresh clone lacks. The cheap guards for the five things that +# made `git clone && bazel build` fail after Build/full was closed -- every one of +# them green on the machine that developed it, which is the whole problem. +# --------------------------------------------------------------------------- +def test_the_asset_script_resolves_absolute_paths(): + """The two path bugs that made every distfile lookup miss. + + vcpkg receives the index as an EXECROOT-RELATIVE path and invokes the + asset-cache script from its OWN working directory, so both the index path and + the paths inside it must be absolutized before vcpkg is exec'd. Fixing only one + moves the failure down a line (`awk: cannot open` becomes `cp: cannot stat`), + and either way vcpkg reports "no asset cache hits" and x-block-origin refuses + the network -- a message that blames the pin rather than the path. + + It survived because the dev machine's vcpkg checkout already had + downloads/tools/cmake-4.4.0-linux from an earlier `Meta/ladybird.py vcpkg` run, + so vcpkg never asked the script for a tool at all. + """ + sh = _read("Meta/vcpkg_build.sh") + assert "EXECROOT" in sh, "the index path is still execroot-relative" + # The index's VALUES too, not just its path: awk finding the row is half the job. + assert 'root "/" $2' in sh, \ + "index entries are not absolutized, so cp runs from vcpkg's cwd" + + +def test_the_vcpkg_checkout_glob_is_the_finding_35_pattern_and_is_known(): + """//Build/vcpkg:tree globs a tree a fresh clone does not have, allow_empty. + + Deliberately NOT asserted as fixed -- it is not. This is the same + `glob(["**"], allow_empty = True)` over a foreign tree finding 35 was about, + one directory over: on a clone it matches exactly one file (its own + BUILD.bazel) and reports nothing. What is pinned here is that the gap is + DOCUMENTED, so nobody reads the top of the README and concludes a clone works. + + Making it allow_empty = False is the real fix and breaks the dev loop until + Build/vcpkg is a git_repository (gap 7). A test saying "this is a known hole" + is worth more than one pretending the hole is closed. + """ + tree = _read("Build/vcpkg/BUILD.bazel") + assert "allow_empty = True" in tree, \ + "if this is now False the gap is closed -- update this test and the README" + # The .git exclusion is a LIE the no-sandbox action gets away with: vcpkg + # resolves versioned ports with `git read-tree`, so the excluded tree is + # load-bearing. Pinned so the contradiction stays visible. + assert ".git/**" in tree + readme = _example_readme() + assert "Build/vcpkg" in readme, "the missing checkout is not documented" + assert "read-tree" in readme, "the load-bearing .git is not documented" + + +def test_the_network_claim_is_scoped_to_vcpkgs_own_downloader(): + """`requires-network: "0"` enforces nothing, and one port calls pip. + + x-block-origin governs vcpkg's OWN downloader; the angle overlay-port runs + `pip install ply` via x_vcpkg_get_python_packages, which is not a distfile and + never reaches the pin. Nothing catches it either: `requires-network: "0"` is a + scheduling hint, `no-sandbox: "1"` means there is no namespace to enforce it + in, and `use_default_shell_env = True` hands the action this machine's + HTTP_PROXY -- so pip silently succeeded for months. + + A control that is not enforced is indistinguishable from one that is not there + -- finding 35's sentence about globs, applied to an execution_requirements key. + Until ply is pinned, the guard is that the README scopes the claim honestly + instead of repeating "zero network access". + """ + bzl = _read("vcpkg.bzl") + assert "no-sandbox" in bzl and "use_default_shell_env = True" in bzl, \ + "if either changed, re-derive whether the network is actually blocked" + readme = _example_readme() + assert "pip install ply" in readme, "the pip hole is not documented" + assert "scheduling hint" in readme, \ + "requires-network is still presented as if it blocked the network" + + +def test_the_hsts_download_is_pinned_downstream_and_documented(): + """The input upstream fetches unpinned, pinned on our side instead. + + CMake downloads the HSTS preload table from Chromium's `main` at configure time + -- unversioned -- and that is upstream's code, which we do not control. So the + overlay pins it DOWNSTREAM: an http_file at an immutable commit + sha256, which + the generator genrule consumes instead of the configure's leftovers under + Build/caches. Three things have to hold together or the pin is decoration: + the pin exists with a full commit sha (not `main`), the genrule reads it, and + the reason it is a commit rather than a release tag is written down (a tag + serves a different table, so pinning one would trade hermeticity for parity). + """ + pin = _read("hsts_preload.bzl") + assert re.search(r'HSTS_PRELOAD_COMMIT = "[0-9a-f]{40}"', pin), \ + "the HSTS pin must name a full commit sha" + assert re.search(r'HSTS_PRELOAD_SHA256 = "[0-9a-f]{64}"', pin) + assert "/main/net/http/" not in pin, "the pinned URL must not track main" + + codegen = _read("codegen_root.bzl") + assert "@hsts_preload_json//file" in codegen, \ + "the generator genrule must consume the pinned file" + assert "Build/caches/HSTSPreload" not in codegen, \ + "the unpinned CMake download path must be gone" + + readme = _example_readme() + assert "hsts_preload.bzl" in readme and "unversioned" in readme.lower(), \ + "why the HSTS table needed a downstream pin is not documented" + + +def _example_readme(): + with open(os.path.join(_WS, "..", "README.md")) as f: + return f.read() + +def test_the_pip_installed_package_is_pinned_and_pip_cannot_reach_an_index(): + """Finding 36's substantive fix: the one dependency the capture cannot see. + + `x-block-origin` covers everything that goes through vcpkg's downloader, and + the angle port's `pip install ply` does not go through it -- so the 76-distfile + pin says nothing about ply and nothing blocked the fetch. Three properties make + it a real pin rather than a comment, and all three have to hold together: + + * the wheel is named by an IMMUTABLE url + hash (files.pythonhosted.org is + content-addressed; `pip install ply` resolves against whatever PyPI serves + today), + * pip is told it may not use an index at all, so an UNPINNED package is an + error rather than a download -- the pip-side equivalent of x-block-origin, + * and the proxy variables are unset, because an inherited HTTP_PROXY is + precisely how this went unnoticed. --no-index alone would probably do; the + lesson of finding 36 is that one unenforced control is not a control. + """ + wheels = _read("vcpkg_python_packages.bzl") + assert "files.pythonhosted.org" in wheels, "the wheel URL is not content-addressed" + assert re.search(r'"sha256-[A-Za-z0-9+/=]{20,}"', wheels), "no integrity hash" + sh = _read("Meta/vcpkg_build.sh") + assert "PIP_NO_INDEX=1" in sh, "pip may still resolve from an index" + assert "PIP_FIND_LINKS" in sh, "the pinned wheels are not offered to pip" + assert re.search(r"unset .*HTTPS_PROXY", sh), \ + "an inherited proxy is how the pip fetch stayed invisible" + # The wheel must be a declared INPUT of the action, not merely fetched: a repo + # Bazel creates but no action depends on is not in the sandbox. + assert "python_wheels" in _read("vcpkg.bzl") + assert "python_wheels = ['@vcpkg_pywheel_ply//file']" in _read("BUILD.bazel") + + +def test_the_use_repo_list_includes_the_pip_wheels(): + """bzlmod needs every extension-created repo named in use_repo, and the wheels + are created by the SAME extension as the 76 distfiles. + + The emitter derives the wheel names from vcpkg_python_packages.bzl rather than + restating them, so this checks the round trip: what --use-repo prints must be + exactly what MODULE.bazel says. A drift here is a "no such repository" error a + long way from its cause, and it is the specific drift the pip pin introduces -- + the wheels are hand-maintained (no instrument can capture them), so they are the + one part of this list a regeneration could silently drop. + """ + import subprocess + import sys + # $LADYBIRD_ROOT scrubbed deliberately: another test in this suite points it at + # a temp fixture checkout and the subprocess would INHERIT it, so this test + # passed alone and failed in the full run. Found by run_all.py, which runs every + # file in one process -- the per-file runners could not have surfaced it. + env = {k: v for k, v in os.environ.items() if k != "LADYBIRD_ROOT"} + out = subprocess.run( + [sys.executable, "Meta/emit_vcpkg_bazel.py", + "--assets", "Meta/vcpkg_assets.tsv", "--use-repo"], + cwd=_WS, capture_output=True, text=True, env=env) + assert out.returncode == 0, out.stderr + emitted = re.findall(r"'([^']+)'", out.stdout) + block = _read("MODULE.bazel").split("vcpkg_deps = use_extension", 1)[1] \ + .split("use_repo(", 1)[1].split("\n)", 1)[0] + assert emitted == re.findall(r"'([^']+)'", block), \ + "MODULE.bazel's use_repo has drifted from the emitter" + assert "vcpkg_pywheel_ply" in emitted + + +def test_the_git_archive_staging_fails_when_the_archives_are_absent(): + """Finding 36's worst offender: three ways to succeed while copying nothing. + + The staging was `if [ -d "$D" ]; then cp "$D"/*.tar.gz ... 2>/dev/null || true; + fi` -- a directory test that skips, a redirect that hides, and a `|| true` that + forgives. On a fresh clone the directory does not exist, all four archives were + silently absent, and skia failed ~20 minutes later with a googlesource URL that + names neither this directory nor the tarball. + + So: no `|| true`, and an explicit check that names what is missing. This does not + assert the archives are FETCHED -- they are not; `git archive` output has no URL + to http_file, so vcpkg_git_archives.bzl records their hashes but nothing creates + them (still gap 7). It asserts that their absence is loud. + """ + sh = _read("Meta/vcpkg_build.sh") + stage = sh.split("Pre-place the git-sourced externals", 1)[1].split("\n# ---", 1)[0] + code = "\n".join(l for l in stage.splitlines() if not l.strip().startswith("#")) + assert "|| true" not in code, "the copy can still silently do nothing" + assert "2>/dev/null" not in code, "the copy still hides its own errors" + assert "exit 1" in code, "a missing archive is not a hard failure" + # And the hashes it points the reader at really are checked in. + archives = _read("vcpkg_git_archives.bzl") + assert archives.count(".tar.gz'") == 4, "the four pinned archives changed" + +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. + + +def test_the_host_tool_list_reaches_the_action_as_a_declared_input(): + """The preflight is only real if the file is in the sandbox. + + Two ways this silently degrades to a no-op, both already made once in this + tree: reading it via `dirname $0` (an sh_binary's data lives in + .runfiles/, not beside the wrapper Bazel execs -- the trap + cargo_vendor.sh documents), or naming it in `data` instead of the action's + inputs. The driver skips the check when $HOST_TOOLS is empty, so a broken + wiring does not fail the build -- it just stops checking. Hence this test. + """ + bzl = _read("vcpkg.bzl") + assert "_host_tools" in bzl, "the host tool list is not an attr at all" + assert "ctx.files._host_tools" in bzl, "not added to the action's inputs" + assert "ctx.file._host_tools.path" in bzl, "the path is not passed as an argument" + # Passed positionally, and the driver reads it from that same position. + sh = _read("Meta/vcpkg_build.sh") + assert 'HOST_TOOLS="${7-}"' in sh, "the driver does not read argument 7" + # Count the top-level entries, tracking bracket depth: a comprehension + # (`",".join([f.path for f in ...])`) contains a `]` of its own, and splitting + # on the first one silently reads a truncated argument list -- which made this + # test fail against correct code. + args, depth = [], 0 + for line in bzl.split("arguments = [", 1)[1].splitlines(): + stripped = line.strip() + if depth == 0 and stripped.startswith("]"): + break + if stripped and not stripped.startswith("#") and depth == 0: + args.append(stripped) + depth += line.count("[") - line.count("]") + positions = args + assert len(positions) == 7, \ + "the driver reads $7, so the action must pass 7 arguments, got %d" % len( + positions) + assert "_host_tools" in positions[-1], "the list must be the 7th argument" + # And it must be exported, or the label does not resolve. + assert 'exports_files(["vcpkg_host_tools.tsv"])' in _read("Meta/BUILD.bazel") diff --git a/upload/11041.patch b/upload/11041.patch new file mode 100644 index 0000000..bebad8f --- /dev/null +++ b/upload/11041.patch @@ -0,0 +1,460 @@ +From 63817a66757e4005383cd7c4e9508fa188e823fc Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Fri, 7 Aug 2026 09:44:33 -0700 +Subject: [PATCH 1/3] LibRequests+LibWeb: Release response pipes when requests + complete +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Pages issuing fetches in a tight loop accumulate one open Unix +socket per completed fetch in their WebContent process. The WPT scroll- +to-text-fragment/redirects.html test polls that way, and drove a local +WebContent process to ~21,900 socket/fileport attachments — at which +point the process crashed after receiving the next response pipe failed. +In a local repro driving about 1,500 fetches per second, WebContent grew +from 5,541 to 63,543 Unix-socket descriptors in 43 seconds. + +Cause: Requests::Request tears down its stream state only when a request +is stopped or transferred. On ordinary completion, the ReadStream with +the response-pipe descriptor, its notifier, and the internal buffers all +stay alive until the Request object is destroyed – and a fetch Response +holds a reference to its Request until GC. The scarce resource is file +descriptors — but GC pressure is measured in heap bytes. So, a tight +fetch loop retires thousands of completed requests without GC ever +having a reason to run — and so, the descriptor table fills up. + +Fix: Schedule the existing deferred teardown at the moment a request +delivers its user-finish callback (the single point every request passes +through on ordinary completion) — for buffered and unbuffered modes and +for network errors alike. That’s only reached once all response data has +been delivered. So streaming consumers and paused document loads are un- +affected — and stopped/transferred requests keep their existing teardown +paths. The teardown must stay deferred: It’s scheduled from inside the +callback chain it destroys, and the deferred task also keeps the Request +alive if the callback drops the last ref. With this change, the repro +holds steady at 0–2 open response pipes at an unchanged request rate. +--- + Libraries/LibRequests/Request.cpp | 20 +++++++++++ + Libraries/LibRequests/Request.h | 9 +++-- + Libraries/LibWeb/Internals/Internals.cpp | 6 ++++ + Libraries/LibWeb/Internals/Internals.h | 1 + + Libraries/LibWeb/Internals/Internals.idl | 1 + + ...sponse-pipes-released-after-completion.txt | 1 + + ...ponse-pipes-released-after-completion.html | 33 +++++++++++++++++++ + 7 files changed, 66 insertions(+), 5 deletions(-) + create mode 100644 Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt + create mode 100644 Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html + +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425a5bad5..16234917a4927 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -35,6 +35,25 @@ static Optional map_body_file(int fd, u64 offset, u64 size + return payload.release_value(); + } + ++static size_t s_live_read_stream_count = 0; ++ ++size_t ReadStream::live_count() ++{ ++ return s_live_read_stream_count; ++} ++ ++ReadStream::ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier) ++ : m_stream(move(stream)) ++ , m_notifier(move(notifier)) ++{ ++ ++s_live_read_stream_count; ++} ++ ++ReadStream::~ReadStream() ++{ ++ --s_live_read_stream_count; ++} ++ + ErrorOr> ReadStream::create(int reader_fd) + { + #if defined(AK_OS_WINDOWS) +@@ -326,6 +345,7 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + auto has_received_all_reported_bytes = m_internal_stream_data->request_done && m_internal_stream_data->delivered_size >= m_internal_stream_data->total_size; + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { + m_internal_stream_data->user_finish_called = true; ++ defer_teardown(); + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); + } + }; +diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h +index b4046fa8d114a..69cdf0d6d0398 100644 +--- a/Libraries/LibRequests/Request.h ++++ b/Libraries/LibRequests/Request.h +@@ -55,6 +55,9 @@ class ResponseData { + class ReadStream { + public: + static ErrorOr> create(int reader_fd); ++ ~ReadStream(); ++ ++ static size_t live_count(); + + NonnullRefPtr const& notifier() const { return m_notifier; } + +@@ -63,11 +66,7 @@ class ReadStream { + ErrorOr read_some(Bytes bytes) { return m_stream->read_some(bytes); } + + private: +- ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier) +- : m_stream(move(stream)) +- , m_notifier(move(notifier)) +- { +- } ++ ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier); + + NonnullOwnPtr m_stream; + NonnullRefPtr m_notifier; +diff --git a/Libraries/LibWeb/Internals/Internals.cpp b/Libraries/LibWeb/Internals/Internals.cpp +index d221019b6e8de..65cb4a9150f3d 100644 +--- a/Libraries/LibWeb/Internals/Internals.cpp ++++ b/Libraries/LibWeb/Internals/Internals.cpp +@@ -22,6 +22,7 @@ + #include + #include + #include ++#include + #include + #include + #include +@@ -614,6 +615,11 @@ void Internals::simulate_request_server_connection_loss() + page().client().page_did_lose_request_server_connection(); + } + ++WebIDL::UnsignedLongLong Internals::open_response_pipe_count() ++{ ++ return Requests::ReadStream::live_count(); ++} ++ + WebIDL::ExceptionOr Internals::set_content_blockers(Utf16String const& patterns_source) + { + Utf16StringBuilder patterns_builder; +diff --git a/Libraries/LibWeb/Internals/Internals.h b/Libraries/LibWeb/Internals/Internals.h +index b2df9d5c9e051..cd0a9bbf4a2dc 100644 +--- a/Libraries/LibWeb/Internals/Internals.h ++++ b/Libraries/LibWeb/Internals/Internals.h +@@ -106,6 +106,7 @@ class WEB_API Internals final : public InternalsBase { + + bool set_http_memory_cache_enabled(bool enabled); + void simulate_request_server_connection_loss(); ++ WebIDL::UnsignedLongLong open_response_pipe_count(); + WebIDL::ExceptionOr set_content_blockers(Utf16String const& patterns); + void set_content_blocking_enabled(bool enabled); + WebIDL::UnsignedLongLong partial_layout_count(); +diff --git a/Libraries/LibWeb/Internals/Internals.idl b/Libraries/LibWeb/Internals/Internals.idl +index e2634eadcac82..107b11fd08160 100644 +--- a/Libraries/LibWeb/Internals/Internals.idl ++++ b/Libraries/LibWeb/Internals/Internals.idl +@@ -85,6 +85,7 @@ interface Internals { + + boolean setHttpMemoryCacheEnabled(boolean enabled); + undefined simulateRequestServerConnectionLoss(); ++ unsigned long long openResponsePipeCount(); + undefined setContentBlockers(Utf16DOMString patterns); + undefined setContentBlockingEnabled(boolean enabled); + unsigned long long partialLayoutCount(); +diff --git a/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt b/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt +new file mode 100644 +index 0000000000000..836a4a872c4d4 +--- /dev/null ++++ b/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt +@@ -0,0 +1 @@ ++open response pipes after fetches: 0 +diff --git a/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html b/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html +new file mode 100644 +index 0000000000000..29c66bcf3cd40 +--- /dev/null ++++ b/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html +@@ -0,0 +1,33 @@ ++ ++ ++ + +From 2ae78115aecaae054532d8a8ba93fecb4c8a9a3c Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Mon, 17 Aug 2026 07:04:13 +0900 +Subject: [PATCH 2/3] LibWeb: Release response pipes when fetches are canceled +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Aborting or terminating a fetch left its response pipe open for +the lifetime of the process. A page that starts and cancels requests in +a loop (an EventSource closed and reopened, a fetch or XHR aborted while +the response is still hanging) leaked one file descriptor per cycle — +until it ran out of them. + +Cause: abort() and terminate() only set the controller state. The one +path that releases the request, stop_request(), is reached only from +stop_fetch() — which returns early once the state is already “aborted” +or “terminated”. So the two entry points that mark a fetch as canceled +were the two that never told the network layer — and RequestServer holds +a request alive until it either finishes or is stopped. + +Fix: Release the request from abort() and terminate() too — through a +helper that stop_request() now shares. + +One caller relied on terminate() leaving the request alone. When a +navigation response becomes a download, the request is handed to the UI +process and the fetch is then terminated. That path now drops its handle +to the request before terminating — so a download already under way +isn’t stopped. RequestServer reports the transfer back to the +WebContent process — and that’s what closes the reader end of the pipe. +--- + .../LibWeb/Fetch/Infrastructure/FetchController.cpp | 13 +++++++++++++ + .../LibWeb/Fetch/Infrastructure/FetchController.h | 5 +++++ + Libraries/LibWeb/HTML/LocalNavigable.cpp | 7 ++++++- + 3 files changed, 24 insertions(+), 1 deletion(-) + +diff --git a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp +index 5e41472a1eb3b..6ba3de7c24e8c 100644 +--- a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp ++++ b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp +@@ -104,6 +104,10 @@ void FetchController::abort(JS::Realm& realm, Optional error) + : serialized_value_or_error.value(); + }; + m_serialized_abort_reason = structured_serialize(error.value(), fallback_error_value); ++ ++ // AD-HOC: stop_fetch() returns early once the state is "aborted". So, this is the last chance to release the ++ // network request. Without it the response pipe stays open for good. ++ stop_pending_request(); + } + + // https://fetch.spec.whatwg.org/#fetch-controller-terminate +@@ -111,6 +115,10 @@ void FetchController::terminate() + { + // To terminate a fetch controller controller, set controller’s state to "terminated". + m_state = State::Terminated; ++ ++ // AD-HOC: As in abort() above — stop_fetch() won’t release the request once the state is "terminated" — so, ++ // release it here. ++ stop_pending_request(); + } + + void FetchController::stop_fetch() +@@ -143,6 +151,11 @@ void FetchController::stop_fetch() + void FetchController::stop_request() + { + VERIFY(m_state == State::Stopped); ++ stop_pending_request(); ++} ++ ++void FetchController::stop_pending_request() ++{ + if (m_pending_request) { + m_pending_request->stop(); + m_pending_request = nullptr; +diff --git a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h +index fb27df8f53e55..480a764277c0d 100644 +--- a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h ++++ b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h +@@ -86,6 +86,11 @@ class WEB_API FetchController : public JS::Cell { + // Null or a fetch timing info. + GC::Ptr m_full_timing_info; + ++ // Releases the network request behind this controller. Every way a fetch stops early has to reach this: The ++ // response pipe is a socket pair, and RequestServer holds the request alive until it either finishes or is stopped. ++ // So, a request that's abandoned without being stopped would keep its descriptor for the lifetime of the process. ++ void stop_pending_request(); ++ + // https://fetch.spec.whatwg.org/#fetch-controller-report-timing-steps + // report timing steps (default null) + // Null or an algorithm accepting a global object. +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.cpp b/Libraries/LibWeb/HTML/LocalNavigable.cpp +index ff47e30dfa7c2..ee28f176bc547 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.cpp ++++ b/Libraries/LibWeb/HTML/LocalNavigable.cpp +@@ -540,8 +540,13 @@ static bool handle_navigation_response_as_download(GC::Ref nav + return true; + } + +- if (navigation_params->fetch_controller) ++ if (navigation_params->fetch_controller) { ++ // AD-HOC: The request now belongs to the UI process (it adopted it above). So, before terminating, drop our ++ // handle to it. Otherwise, fetch termination would stop a download already under way. RequestServer ++ // reports the transfer back to us; that's what tears down this process's end of the pipe. ++ navigation_params->fetch_controller->set_pending_request(nullptr); + navigation_params->fetch_controller->terminate(); ++ } + + return true; + } + +From 8636af4103410833243e5eb987d64e8baf076b1a Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Mon, 17 Aug 2026 07:28:12 +0900 +Subject: [PATCH 3/3] LibWeb: Tear down a navigation parked for content + sniffing +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Removing an iframe whose response headers had arrived but whose +body hadn’t arrived left the network request open for the lifetime of +the process. A page that adds and removes such iframes in a loop leaked +one file descriptor per iframe. + +Cause: If a navigation has a response but not yet enough bytes to sniff +its content type, it gets parked in wait_for_sniff_bytes. It has no +document at that point — so, Document::abort() has no fetch controller +to stop, and the only code that releases the request is the arrival +callback. That callback does handle a destroyed navigable — but it runs +only once bytes arrive. So, a server that sends headers and then stops +leaves nothing at all around to release the request. + +Fix: Register a teardown on the navigable before parking — and run it if +the navigable is destroyed first. The teardown calls the same helper the +arrival callback already uses — and destroying a navigable is where the +rest of its in-flight navigation state is released. + +A navigation superseded before its populate task runs leaked the same +way, for the same reason: That guard returned without releasing +anything. It now releases the response as the guard above it does. +--- + Libraries/LibWeb/HTML/LocalNavigable.cpp | 50 +++++++++++++++++++++++- + Libraries/LibWeb/HTML/LocalNavigable.h | 6 +++ + 2 files changed, 55 insertions(+), 1 deletion(-) + +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.cpp b/Libraries/LibWeb/HTML/LocalNavigable.cpp +index ee28f176bc547..c908afbcee8df 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.cpp ++++ b/Libraries/LibWeb/HTML/LocalNavigable.cpp +@@ -681,6 +681,40 @@ void LocalNavigable::set_has_been_destroyed() + m_has_been_destroyed = true; + resolve_all_pending_async_scroll_operations(); + cancel_user_scroll_settlement(); ++ run_pending_navigation_teardown(); ++} ++ ++void LocalNavigable::set_pending_navigation_teardown(GC::Ptr> teardown) ++{ ++ // A navigation parking here while another is already parked supersedes it. So, tear the older one down — rather ++ // than dropping it and leaving its request open. ++ run_pending_navigation_teardown(); ++ ++ // Destruction has already run, so nothing would ever run this one. Tear it down now — instead of storing it forever. ++ if (m_has_been_destroyed) { ++ if (teardown) ++ teardown->function()(); ++ return; ++ } ++ ++ m_pending_navigation_teardown = teardown; ++} ++ ++void LocalNavigable::clear_pending_navigation_teardown(GC::Ptr> expected) ++{ ++ // Only the navigation that registered this teardown may clear it. A navigation whose sniff-byte callback ++ // arrives after a newer one has parked would otherwise clear the newer teardown — and leave that ++ // newer request with nothing to release it. ++ if (m_pending_navigation_teardown != expected) ++ return; ++ ++ m_pending_navigation_teardown = nullptr; ++} ++ ++void LocalNavigable::run_pending_navigation_teardown() ++{ ++ if (auto teardown = exchange(m_pending_navigation_teardown, nullptr)) ++ teardown->function()(); + } + + void LocalNavigable::remove_from_all_local_navigables() +@@ -711,6 +745,7 @@ void LocalNavigable::visit_edges(Cell::Visitor& visitor) + visitor.visit(m_active_document); + visitor.visit(m_input_method_composition_node); + visitor.visit(m_container); ++ visitor.visit(m_pending_navigation_teardown); + m_event_handler.visit_edges(visitor); + + for (auto& navigation_params : m_pending_navigations) { +@@ -2190,6 +2225,9 @@ void LocalNavigable::populate_session_history_entry_document( + + // 1. If navigable's ongoing navigation no longer equals navigationId, then run completionSteps and abort these steps. + if (navigation_id.has_value() && ongoing_navigation() != navigation_id) { ++ // AD-HOC: Nothing downstream will consume this response, and no document exists yet to abort — so, ++ // release its request here, as the active-window guard above does. ++ stop_or_resume_response_body_delivery(navigation_params); + if (completion_steps) { + completion_steps->function()(nullptr); + } +@@ -2303,8 +2341,18 @@ void LocalNavigable::populate_session_history_entry_document( + if (!sniff_bytes.has_value()) { + // Async path: bytes not yet available, wait for them + nav_params->response->resume_body_delivery_up_to(Fetch::Infrastructure::MAX_SNIFF_BYTES); ++ ++ // AD-HOC: The callback below runs only once bytes arrive — which never happens if the server sends ++ // headers and then stops. Hand the navigable a teardown — so that destroying it releases ++ // the request, instead of leaving the pipe open. ++ auto teardown = GC::create_function(heap(), [navigation_params] { ++ stop_or_resume_response_body_delivery(navigation_params); ++ }); ++ nav_params->navigable->set_pending_navigation_teardown(teardown); ++ + body->wait_for_sniff_bytes(GC::create_function(heap(), +- [output, nav_params, navigation_params, completion_steps, source_snapshot_params](ReadonlyBytes sniff_bytes) { ++ [output, nav_params, navigation_params, completion_steps, source_snapshot_params, teardown](ReadonlyBytes sniff_bytes) { ++ nav_params->navigable->clear_pending_navigation_teardown(teardown); + // AD-HOC: The document may have been destroyed between when the fetch started and when the + // bytes arrived. + if (nav_params->navigable->active_browsing_context()) { +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.h b/Libraries/LibWeb/HTML/LocalNavigable.h +index 8ebc9e99660ae..879da0f94f09c 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.h ++++ b/Libraries/LibWeb/HTML/LocalNavigable.h +@@ -95,6 +95,10 @@ class WEB_API LocalNavigable : public Navigable { + void set_navigation_load_event_guard(DOM::Document& parent_doc); + void clear_navigation_load_event_guard(); + ++ void set_pending_navigation_teardown(GC::Ptr>); ++ void clear_pending_navigation_teardown(GC::Ptr> expected); ++ void run_pending_navigation_teardown(); ++ + RefPtr active_session_history_entry() const; + void set_active_session_history_entry(RefPtr); + RefPtr current_session_history_entry() const; +@@ -406,6 +410,8 @@ class WEB_API LocalNavigable : public Navigable { + // AD-HOC: Guards the parent document's load event delay count during cross-document navigation. + Optional m_navigation_load_event_guard; + ++ GC::Ptr> m_pending_navigation_teardown; ++ + // Implied link between navigable and its container. + GC::Ptr m_container; +