diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index d7356d09..288440db 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -17,6 +17,32 @@ diff, and swapping a test's transport is a change that wants its own verificatio ## Distribution +### An arm64 Linux release build, so SBCs stop building from source (2026-09-09) + +Every Linux job in `release.yml` runs on `ubuntu-latest`, which is x86-64, so the release publishes +`projectMM-linux-x64` and `projectmm_X.Y.Z_amd64.deb` and nothing for arm64. That one gap is why a +Raspberry Pi or a NanoPi has to clone and compile, and why the container image can only be amd64 +(the image in PR #98 installs the released `.deb`, so an arm64 image needs an arm64 `.deb` first). + +GitHub offers arm64 Linux runners for public repositories (`ubuntu-24.04-arm`), so this is a second +job rather than cross-compilation. Unverified against this repo: whether `package_desktop.py` runs +there unmodified, and whether the runner is available on this plan. Check both before promising it. + +Shipping it collapses three problems into one fix: the SBC route becomes `apt install`, the +container can publish a multi-arch manifest (one tag, Docker picks per host), and +[installing-on-linux.md](../tutorials/installing-on-linux.md) loses its build-from-source branch. + +### A flashable SD image with projectMM already on it (robwomp, 2026-09-08) + +Suggested on Discord while bringing up a NanoPi R28S: most Pi users want to write an `.img` to a +card and be running, not to install a toolchain. Armbian's own build tooling supports exactly this +(`armbian/os` carries per-application "extensions", OMV being a small worked example), so the image +is a spin of a maintained distribution plus our package rather than a distribution to maintain. + +Wants the arm64 `.deb` above first: with it the extension is roughly "install this package, enable +this service", which is the shape those extensions already have. Without it, the image would have to +carry a source build, which is the thing it exists to avoid. + ### OTA upload refuses a normal client: the body must arrive within ~50 ms (2026-09-02) `POST /api/firmware/upload` answers `400 {"error":"incomplete request body"}` to an ordinary @@ -1142,6 +1168,27 @@ module's tick goes from ~615 us to ~658 us, about 40 us (7%) against 3.3 ms of t Stability confirmed over a soak with HLS streaming: zero crashes, zero corrupt packets. **Reported upstream:** [esp-idf#19025](https://github.com/espressif/esp-idf/issues/19025) + +**Root cause found upstream (2026-09-09), and it is not the one above.** Espressif reproduced it +deterministically and found two separate problems. The "Cause" paragraph above, which names the +unguarded HWLP save path, was our reading of `portasm.S` and it was wrong; it stays as the record of +what we thought. What is actually happening: + +1. **esp-dsp violates the P4's hardware-loop constraints.** The loop start (`esp.lp.setup`) and the + last loop instruction must be 4-byte aligned, the body needs enough 32-bit instructions, and the + last instruction must not be a coprocessor (FPU/PIE) op. Some kernels are misaligned, and an + interrupt landing exactly on that boundary corrupts the loop counter. That is why it is + timing-dependent (it hid for a 20-minute soak on 2026-09-08) and why `CONFIG_DSP_ANSI` cured it: + the C kernels never enter the misaligned assembly. Fix: an esp-dsp release with `.balign 4`. +2. **The IDF erratum workaround was gated on a misnamed macro** (`ESP32P4_REV_MIN_FULL` without + the `CONFIG_` prefix), so it was always on for every revision. Being corrected to rev < v3.0 only, + with a positive test (preemption inside a live hardware loop) and negative tests (interrupt at a + misaligned boundary). + +**To do when the esp-dsp release lands:** bump the managed component, drop the `CONFIG_DSP_ANSI` +line for good, and re-run the workload that found it (continuous FFT + the H.264/HLS encoder as a +second task, sustained HTTP load) on the rev v1 board. We have the reproducing workload and told +them we would report back; that soak is the report. (2026-08-28), which names the unguarded save path. Espressif already had the symptom on file from another reporter: [esp-dsp#119](https://github.com/espressif/esp-dsp/issues/119) hits the same fault at the same instruction on IDF v5.5-beta1 and settles on the same `CONFIG_DSP_ANSI=y` @@ -1308,7 +1355,11 @@ We build for hardware we cannot see. Which chips people actually run, how big th - **Which features are switched on.** The modules present and enabled: audio, MoonLive, MQTT, Art-Net or E1.31 send and receive, panel cards, Ethernet versus WiFi. This is the half that answers "is anyone actually using this", which is what decides whether a feature earns its maintenance. - **A country**, derived at the server from the address the request arrives from, so the map is by region rather than by installation. -**What it must never carry**, and this list is a constraint on the design rather than a note on it: device name, IP or MAC address, WiFi credentials, MQTT passwords, any free-text field a user typed, the contents of a layout or a script, and anything that lets two reports be recognized as the same installation. There is no device identifier, which also means no de-duplication: a machine reporting twice counts twice, and that inaccuracy is the price of not being able to track anyone. It is the right trade. +**What it must never carry**, and this list is a constraint on the design rather than a note on it: device name, IP or MAC address, WiFi credentials, MQTT passwords, any free-text field a user typed, and the contents of a layout or a script. + +**The identifier: a random value per install, dropped after seven days** (settled 2026-09-08). Not derived from the MAC or from anything else about the hardware, so it cannot be reversed to a device even by whoever holds it. It exists to answer one question the aggregates cannot: how many DEVICES, rather than how many reports, and therefore how many installs are still on an old version. The server keeps per-report rows for seven days, de-duplicates, then keeps only the aggregate and drops the id. + +Two options were weighed and rejected. **No identifier at all** was the original text here: simplest to promise, but it makes every per-device question unanswerable, and "how many are still on the old version" is one of the questions this whole entry exists to answer. **A hashed MAC** is what WLED shipped first (`sha1("WLEDUSAGE" + MAC)`) and it buys exactly one thing more than a random id, given reports are one-time: linking one device's reports across months. That is the tracking capability itself, it survives a factory reset and a reflash, an open-source salt over a MAC space that size is rainbow-tablable, and a stable pseudonymous identifier is personal data under GDPR rather than anonymous. WLED's own shipped version moved off it to an anonymous id, which is the same conclusion from people who had it running. **Consent.** Opt-in, from a prompt shown after a fresh install or an upgrade, with a decline that is as easy to click as the accept and is remembered. One report per install or upgrade, never a heartbeat. A user who declines transmits nothing at all, rather than transmitting a "declined" record. @@ -1316,6 +1367,95 @@ We build for hardware we cannot see. Which chips people actually run, how big th **Open questions for whoever picks this up.** Where the server runs and who administers it, since it is the first piece of infrastructure this project would own rather than borrow from GitHub. Whether the dashboard is public, which we would want it to be for the same reason the source is. Whether the desktop reports at all or only devices, given the desktop is the half we currently know nothing about. And what happens to a report from a version whose fields have since changed, because a schema that cannot be read a year later answers nothing. +**WLED ships this, and the dashboard is public**: [usage.wled.me](https://usage.wled.me). Worth reading before building anything, because it answers two of the questions above by example and disagrees with this entry on a third. + +What it shows: distributions by version, chip, matrix, flash size, PSRAM, release, LED count and filesystem usage; upgrade versus install events over six months, split by chip; which LED features, peripherals, integrations, usermods and bus types are in use; and device count by country. That list is close to what is proposed above, which is reassuring about the shape. + +How it is collected ([PR #5116](https://github.com/wled/WLED/pull/5116), merged 2025-11-27, superseding an earlier [#4342](https://github.com/wled/WLED/pull/4342)): a one-time POST to `usage.wled.me/api/usage/upgrade` when a persisted version file shows the firmware changed, behind a startup prompt offering Yes / Not Now / Never Ask, and suppressed entirely in AP mode. The server is open source ([netmindz/WLED_usage](https://github.com/netmindz/WLED_usage)), which is how they answer "is the deployed code the published code". + +**Where they differ from this entry, and it is the interesting part.** WLED sends a device id: the first design hashed the MAC (`sha1("WLEDUSAGE" + MAC)`, "unique but not reversible"), the shipped one an anonymous `deviceId`. Either way the point is de-duplication, and it is what lets their dashboard say how many DEVICES rather than how many reports. This entry deliberately refuses that, and pays for it in accuracy. Their earlier PR also discussed a retention split (per-device data for 7 days, aggregates longer), which is a middle position between the two: keep an identifier only long enough to de-duplicate, then drop it. Worth considering rather than assuming the strictest option is automatically right. + +### The smallest honest version + +Sketched 2026-09-08. Not started; the blocker is the server, not the firmware. + +**The firmware side is small, because the payload already exists.** Every field is a control or a +module name the device publishes today: `chip`, `cpu`, `flash`, `psram`, `sdk`, `firmware` and +`deviceModel` are SystemModule controls; `version` and the previous version are FirmwareUpdateModule +controls; the enabled modules are the state tree's own top level; the light setup is the Layouts +grid and the Drivers children. So this is a serializer over facts already in memory, not new +instrumentation. Budget it as one module of a few hundred lines, plus the consent prompt. + +**When it sends.** Once, when a persisted version file shows the firmware changed, which is the +shape WLED converged on after starting from a UDP spray every second. Never a heartbeat. Suppressed +in AP mode, where there is no internet and a device is usually mid-provisioning. + +**Consent.** A prompt on the first boot after an install or upgrade: Yes / Not Now / Never, with +Never remembered. A decline sends nothing at all, not even a "declined" record. The prompt says what +is in the report in one sentence, and links the privacy policy. + +**Ask WLED to host it, before building a server at all.** Investigated 2026-09-08, and their server +is closer to a fit than expected. + +[netmindz/WLED_usage](https://github.com/netmindz/WLED_usage) is Kotlin on Spring Boot with a MySQL +schema under Flyway migrations, a Docker compose deploy, and one endpoint: +`POST /api/usage/upgrade`, no auth, with the country derived server-side from an `X-Country-Code` +header. The dashboard is a single static `index.html` against a `/api/stats` controller. + +**It is already multi-project.** The `device` table carries a `repo` column (added 2025-12), there +is a `RepoHistory` entity, and EVERY stats query is written `(:repo IS NULL OR repo = :repo)`. So a +second project reporting under its own repo name is a supported case rather than a change they would +have to make, and our figures would not be mixed into theirs. + +**We adopt their schema unchanged, and drop the one field that does not fit** (settled 2026-09-08). +chip, version, previousVersion, releaseName, ledCount, isMatrix, flashSize, psramSize/Present, +fsUsed/Total, busCount and busTypes mean the same thing in both projects, so those figures are +directly comparable and projectMM can sit in a pooled overview rather than being a special case. +Our own vocabulary rides the free-form lists: ledFeatures, peripherals, integrations, and `usermods` +for the enabled-module names. That works because migration V2026040301 (2026-04) replaced twenty-odd +fixed boolean feature columns with three comma-separated lists, aggregated by counting whatever +values appear: adding projectMM's names costs their server nothing. + +The layout dimensions this entry originally wanted (width x height x depth, since we are 2D and 3D) +have no equivalent there, and are DROPPED rather than added. `ledCount` plus `isMatrix` answers most +of what they were for, and being identical to a schema someone else maintains is worth more than one +field. If a real question later needs the third dimension, a list value is the place for it. + +**Their payload is close to what this entry wants.** `UpgradeEventRequest` already carries +deviceId, version, previousVersion, releaseName, chip, ledCount, isMatrix, bootloaderSHA256, brand, +product, flashSize, partitionSizes, psramSize, psramPresent, repo, fsUsed, fsTotal, busCount, +busTypes, ledFeatures, peripherals, integrations, usermods. Every field we listed above maps onto one +of those except the layout dimensions, and `usermods` is the natural home for "which modules are +enabled". Sending it means shaping our report to their names, which is a small price for not owning +a server. + +**What to settle with them before relying on it.** The repository has NO LICENCE file, so +strictly nobody may reuse it, and it was last pushed 2026-05-31, so it is quiet rather than dead: +both are conversations rather than blockers, but they are conversations to have first. Then the real +questions: are they willing to take another project's reports at all, who administers the box and +what happens to our data if that person stops, does the public dashboard gain a repo selector or +would we render our own from their API, and does the seven-day identifier retention this entry +commits to match what their server actually does (their earlier PR discussed it; the shipped schema +keeps a `device` row keyed by id, which suggests it does not). + +If the answer is yes, this feature loses its blocker entirely and becomes a firmware change plus a +conversation. If it is no, the plan below stands. + +**The server is the whole cost, and it is a standing commitment rather than a feature.** It is the +first infrastructure this project would own rather than borrow from GitHub, and it needs an +administrator, a domain, TLS, a retention job that actually runs, and a public dashboard. WLED +publishes their server ([netmindz/WLED_usage](https://github.com/netmindz/WLED_usage)), which is how +they answer "is the deployed code the published code": whatever runs here should be public for the +same reason the firmware is. Until someone owns that, this feature cannot ship honestly, and that is +the reason it is still in the backlog rather than in a branch. + +**Order of work, and the first two are worth doing whether or not the rest ever lands.** Write the +privacy policy revision FIRST, since it is the promise everything else has to match, and the current +page says plainly that nothing of the kind exists. Then the report BUILDER as a pure function over +the state tree, with a unit test asserting that the forbidden fields cannot appear in its output: +that test is the design constraint made executable, and it is worth having even if nothing ever +sends. Only then the consent prompt, the one-time trigger, and last the server. + ## A driven GPIO the Pins map never sees: bus padding, and a hidden clockPin **Found:** 2026-08-21, on MM-S31, after a bench session that started as "the LED panel stopped working" and cost hours chasing a firmware regression that did not exist. @@ -1445,3 +1585,140 @@ the rows should be. Until then a list is user-populated, which is the honest behavior: the device knows the pin, the user knows what the button should do. + +## P4 WiFi cascade into a dead co-processor link aborts the board (2026-09-08) + +Bench, MHC-WLED ESP32-P4 shield on `esp32p4rev1-eth-wifi`, no Ethernet cable: the NetworkModule +cascades to WiFi, and the board reboots every ~22 s with `task_wdt: main (CPU 0)` while `IDLE0` +runs. The main task is not spinning, it is BLOCKED: on the P4 every `esp_wifi_*` call is forwarded +over SDIO to the ESP32-C6 by `esp_wifi_remote`, and this shield's C6 never completes the ESP-Hosted +handshake (`E H_API: ESP-Hosted link not yet up` at boot), so `esp_wifi_init` and the calls after it +each wait out their timeout, on the render thread, past the 5 s watchdog. The wait is +esp_hosted's `transport_drv.c` slave-ready loop: 200 ms polls, a slave reset every 50 of them (the +`Reset slave using GPIO[54]` lines at 10 s and 21 s), up to MAX_RETRY_TRANSPORT_ACTIVE = 100 +polls, so worst case ~20.0 s in the caller's task, which is ours. Not the HWLOOP/FFT +erratum: audio was ruled out by the watchdog text itself (a blocked main task with idle running, +not a spinning core). Control experiment, same image on the bench P4 (.139, same shield model, +esp_hosted host 2.12.13): its SDIO card init succeeds (`Card init success, TRANSPORT_RX_ACTIVE`), +it never associates either, but the main task keeps ticking, it falls back to its own AP, and it +rejoins Ethernet when the cable returns. The new shield never prints a card-init success: its C6 +does not answer the bus (no slave firmware, or one that does not match the 2.12.x host). + +Two defects are ours, whatever the C6 carries: + +1. **The cascade does not consult the link.** `coprocessorWifi()` in `platform_esp32.cpp` already + asks the C6 for its firmware version, bounded to two attempts, exactly to detect an absent or + incompatible slave. The WiFi cascade never asks; it walks straight into `esp_wifi_init`. Gate the + cascade on that answer (or on `esp_hosted` reporting the link up) and degrade to "no network, + C6 not answering" as a Network status. +2. **A failing cascade must not abort.** Robustness says degrade visibly, never crash. Even with the + gate, a link that dies later would hit the same watchdog: the forwarded calls need to run off the + render thread, or with a timeout shorter than the watchdog, so the worst case is a status line. + +Practical today: the shield runs the eth-only image without WiFi, or its C6 gets the ESP-Hosted slave +firmware flashed (the `ships: false` note on the variant in `build_esp32.py` says why that is not +yet reproducible). The Improv script's eth-only rule and the catalog's two firmwares for the shield +are already in place. + +**Everything tried on 2026-09-08 was reverted; both P4s now run `esp32p4rev1-eth` and are stable.** +The tree is back to its pre-attempt state except for the catalog, which now lists BOTH P4 firmwares +for the MHC-WLED shield so the installer offers the WiFi variant at all (it listed only the eth one, +which is why the WiFi image could not be picked). What the day established, so the next attempt does +not repeat it: + +- **The C6 link is INTERMITTENT on this shield, not simply dead.** After the vendor C6-update tool + ran (it never completed an OTA: it looped on "Not able to connect with ESP-Hosted slave device"), + the link came up and the board reached a DHCP lease on WiFi, first attempt, no retries. A power + cycle later it was back to `sdmmc_send_cmd returned 0x107` and a reboot loop. Any future fix has + to survive a cold boot, not one lucky session. +- **A patient STA retry (MoonLight's 5 s cadence, ~2 min budget) was implemented and reverted.** It + never fired on either board: both connected on the first attempt when the link worked, and when it + did not the board died before the render loop. Keeping untested robustness was not worth the + surface. The reasoning still holds and is worth redoing WITH a repro: MoonLight retries forever + and never tears the radio down, and its own comment warns that toggling WiFi on a co-processor + board forces costly esp_hosted reinit cycles, which is exactly what our AP fallback does. +- **Espressif does not endorse retrying into readiness.** Their esp-hosted troubleshooting puts + "not able to connect with slave" and SDIO 0x107 down to wiring, pull-ups, signal integrity, or a + host/slave VERSION MISMATCH: "use the same version for master and slave". Our host is 2.12.13; the + shield's C6 is on the factory build the vendor tool calls 0.0.6 (target 2.0.17). Updating the C6 + needs Method 2 (direct USB/UART to the C6), since Method 1 needs the very link that is broken. + +**A fail-fast gate was tried on the bench (2026-09-08) and REVERTED.** The idea was right and both +signals were wrong. `platform::wifiHardwareReady()` gated the STA and AP init, and it STOPPED the +reboot loop dead: the shield ran 80 s with 0 reboots and 59 ticks, logging "WiFi co-processor link +not up" instead of aborting. But on the bench P4 (.139), whose link demonstrably works, the same +predicate read FALSE at its 26 s cascade and refused WiFi on a healthy board. Two signals were +tried, neither is usable as a readiness test: + +- **`ESP_HOSTED_EVENT_TRANSPORT_UP`**, subscribed on the default event loop (both lazily and from + `ensureNetifInit`, i.e. before esp_hosted's task posts it). Never observed arriving. esp_hosted + posts through its own `g_h.funcs->_h_event_post` indirection; where that lands was not chased. +- **`esp_hosted_get_coprocessor_fwversion` returning a non-zero version.** Also false on .139, which + matches what `coprocessorWifi()` already records: on a live link this RPC times out rather than + answering, which is why that function is bounded to two attempts. + +**Not a regression: the RELEASED v4.0.0 image fails the same way on this shield** (bench, same day). +Flashed from the web installer after an erase, it logs `App version: v4.0.0` and then: + +``` +W (13438) H_SDIO_DRV: Reset slave using GPIO[54] +W (13438) gpio: conflict found for GPIO[54] +E (14988) sdmmc_io: sdmmc_io_rw_extended: sdmmc_send_cmd returned 0x107 (ESP_ERR_TIMEOUT) +E (14988) H_SDIO_DRV: failed to read registers +``` + +and reboots without ever reaching the render loop. The same v4.0.0 image serves WiFi on the bench +P4 (.139) with no SDIO error and no GPIO 54 conflict, and no projectMM config on any P4 entry +references GPIO 54 (esp_hosted drives it as the slave reset). The two boards differ in silicon +revision: this shield is chip rev **v1.0**, .139 is **v1.3**. So the C6 side of this shield does not +come up, which is a board/slave-firmware matter rather than anything in our WiFi path, and the +firmware's job is only to degrade rather than reboot. + +**The product owner reports this same shield ran WiFi under MoonLight on IDF 5.5**, which makes a +dead C6 unlikely and points at host-side SDIO configuration on 6.1. What was checked (2026-09-08): + +- The SDIO data pins are NOT board-preset dependent: they come from the SLOT choice + (`ESP_HOSTED_SDIO_SLOT_1`, fixed silicon pins), so swapping `ESP_HOSTED_P4_DEV_BOARD_*` presets + would not move them. The preset mostly moves SPI pins, which we do not use. +- `ESP_HOSTED_SDIO_GPIO_RESET_SLAVE` defaults to **54 on any P4** regardless of preset, so the + `gpio: conflict found for GPIO[54]` line is esp_hosted resetting the slave twice, not a wrong pin + from our config. No projectMM P4 entry references 54. + +**Our SDIO configuration is not the difference.** Diffed against a known-good local reference +(`ewowi/FlowFields/sdkconfig.esp32-p4`, an IDF 5.5-era P4 build), every hosted setting is IDENTICAL: +pins (CLK 18, CMD 19, D0-D3 14-17), `GPIO_RESET_SLAVE` 54, `RESET_ACTIVE_HIGH=y`, `CLOCK_FREQ_KHZ` +40000, `SLOT_1`, `4_BIT_BUS`, `RESET_DELAY_MS` 1500, `RX_STREAMING_MODE`. So reset polarity, reset +pin and SDIO clock are all ruled out as differences, and the identical released v4.0.0 image works +on .139 and not on this shield. Same firmware, same config, two boards, two outcomes: what remains +is on the board side (C6 slave firmware, power/strapping, or the SDIO traces on this revision), and +the only projectMM work left is the degrade-instead-of-reboot fix above. + +What esp_hosted's private `is_transport_tx_ready()` reports is the signal the slave-ready loop +itself polls, but it lives in a PRIVATE include dir (`host/drivers/transport`, not in the +component's `pub_include`), so reaching it means either adding that dir to our include path or +asking upstream to export a readiness getter. That is the next thing to try, and it needs the +.139-still-associates control run alongside the shield-stops-rebooting one: the fix is only right +when BOTH hold. + +## The persisted `firmware` variant survives a flash to a different variant (2026-09-08) + +`SystemModule` writes the compile-time `kFirmwareName` into its `firmware` control at +`defineControls()`, and its own comment states the intent: "written from kFirmwareName on every boot +rather than read from the file: the compile-time constant is the truth". But the control is +`addText`, which is PERSISTED, and the config load runs after `defineControls()`, so a saved value +from a previous image overwrites the compile-time one. + +Bench (MHC-WLED P4 shield, flashed from `esp32p4rev1-eth-wifi` to `esp32p4rev1-eth`): the Firmware +card correctly reported `esp32p4rev1-eth` (it reads the running image), while `/api/modules/System` +still reported `esp32p4rev1-eth-wifi` from the old config. The two disagreed on the same board, and +the stale one is the field an outside reader trusts. + +Why it matters beyond cosmetics: the comment explains this value exists so **MoonBase** can narrow +the recovery image list to one variant. A stale value points a recovering board at the wrong image, +which is the failure that list exists to prevent (picking an `esp32s3-n16r8` build for a Zero +installs a flash layout the board does not have). + +The fix has to keep the value readable by another image (that is why it is persisted at all) while +making the compile-time constant win: re-assert `kFirmwareName` after the config load rather than +only at `defineControls()`, and pin it with a test that loads a config naming a DIFFERENT variant +and checks the control still reads the compiled one. diff --git a/docs/backlog/backlog-light.md b/docs/backlog/backlog-light.md index 1da07234..abea1d29 100644 --- a/docs/backlog/backlog-light.md +++ b/docs/backlog/backlog-light.md @@ -41,8 +41,19 @@ What it does NOT address: on the classic ESP32 the DMA half compiles to nothing, moves the channel's interrupt off core 0 (the root cause found on the Dig-Next-2, fixed by creating the channel from core 1). The two are complementary, one driver with the right answer per chip: DMA where the silicon has it, the core-1 refill where it does not. Adopt his DMA and -callback path, keep the core hop, and drop the classic-only `txInFlight_` guard where his busy -flag covers it. Study, do not copy: write it against the seam as it stands, credit the branch. +callback path and keep the core hop. Study, do not copy: write it against the seam as it stands, +credit the branch. + +Re-read against the wire-byte driver (2026-09-09), which changed two of the assumptions above. The +driver no longer holds pre-expanded symbols at all: it ships the correction's wire bytes and the +peripheral expands them (IDF bytes encoder on the DMA chips, the level-5 refill on classic), so the +frame buffer is ~3 bytes per light rather than 96. So (a) his `mem_block_symbols = 1024` is now the +only symbol memory in play and is cheap, and (b) `rmt_transmit` already takes the bytes, so the DMA +path needs no buffer change, only `flags.with_dma` plus the callback. The `txInFlight_` guard STAYS +whatever happens: the peripheral streams straight out of the driver's frame buffer, so a rebuild +that frees it mid-frame still tears; his busy flag would replace the blocking wait, not that guard. +The win left on the table is the blocking `rmtWs2812Wait` on S3/P4, which today costs the tick the +whole wire time of the longest strand. ### A script's setControl rebuilds a control subtree on every write (2026-09-06) @@ -218,6 +229,40 @@ The bandwidth arithmetic (datasheet-derived): DMA demand = bus-bytes × pclk. Di **Conclusion — this does not open a new path; the proper ring fix already is the path.** The internal-RAM footprint of the ring is NOT set by light count: the ring transposes from a PSRAM-resident source into a small fixed internal buffer pool, so PSRAM is never on the DMA's read path at all. The 240-light wall is the `kRingBufs=16` no-reuse stopgap (the wrap read-while-write race), NOT the ring's design — and "more buffers" is a confirmed dead end. The shipped ring (above) holds internal RAM constant at arbitrary light count, which is exactly the "unlimited lights/strand" the PSRAM-hybrid idea was reaching for — obtained the correct way, at the mandatory shift clock, without PSRAM on the read path. **Action: none — the ring shipped; the "lower shift pclk + PSRAM whole-frame" hybrid is closed as physically blocked and should not be re-attempted.** (If the mechanism is ever contested, the one bench measurement worth doing is registering GDMA underrun/FIFO-empty counters at 26.67 MHz whole-frame-PSRAM to distinguish underrun from latency — but it would not change the conclusion.) +### RmtLedDriver's out-of-memory status has no test, and cannot have one yet (2026-09-09) + +`frameUnusable_` reports a frame buffer the driver could not get (or, on classic, one that landed in +PSRAM where the refill cannot read it), and two guards keep that error from being overwritten by the +resting "driving N of N" status. All of it shipped unpinned, and not by oversight: the desktop +`allocInternal` is a flat `trackedAlloc` that never refuses and ignores `setTestMaxAllocBlock`, and +`ptrIsPsram` is hard-coded false. A test written against that would pass whatever the driver did. +`unit_JsonSink_overflow` shows the trap: it grows until a 64-bit host might refuse and accepts "no +refusal" as a pass, so it tests nothing on a machine with RAM. + +**What it takes:** a cap inside the desktop `allocInternal` that `setTestMaxAllocBlock` (or a new +`setTestInternalCap`) drives, so a test can make the frame allocation fail deterministically, plus a +`setTestPsramFrom(ptr)` seam so the PSRAM branch is reachable off-target. Then three cases: the +status appears, `prepare()` does not replace it with "driving N of N", and `reinit()` does not clear +it (the bug the bench found: the error retracted microseconds after it was set). A platform change +with its own review, which is why it is here and not in the change that shipped the status. + +### The ring's memory readout counts one buffer, not the pool (CodeRabbit, 2026-09-09) + +`ParallelLedDriver::driverHeapBytes()` derives its DMA figure from `peripheral_->busCapacity()`, +which is `st->cap`: the size of ONE buffer. It then adds a second when `busBuffer(1)` exists, which +is right for the double-buffered path and wrong for the ring, where `createRingState` allocates +`ringBufs` slices (up to 30 at the 48x256 geometry) plus the shared zero-pad. So a ring board +under-reports its DMA memory by roughly the buffer count, on the card that exists to make exactly +that number visible. + +Not a correctness bug: the memory is spent either way, and nothing sizes an allocation from this +value. It is a reporting bug in the one readout a user consults before adding lights. + +**What it takes:** an aggregate size on the peripheral (the ring knows `ringBufs * bufBytes + pad` +at creation) rather than arithmetic in the driver, which cannot see the pool. Wants a ring-mode +accounting test alongside, since the existing ones only cover the non-ring path. Touching +`createRingState` means a bench pass on the wall, which is why this is its own change. + ### MoonI80 ring — boot / first-frame-after-rebuild trips a transient give-up status (2026-07-16) A cosmetic residual left after the rebuild-wedge fix (below): on boot, and for a beat after any shift-ring rebuild, the driver shows **"output stalled"** even though `wireUs` reports live completions — the *first* frame after a fresh ring build occasionally misses its completion window and trips the dead-frame give-up before the ring settles, so the stale error latches until the next interaction clears it. It is NOT the old wedge (that stayed dead until reboot; this self-clears on any control edit and the ring is genuinely driving underneath). Two clean fixes to weigh: (a) don't count the very first frame after a rebuild toward `deadFrames_` (give the fresh ring one grace frame), or (b) have the give-up retry re-derive status the moment `wireUs` shows a real completion. Low priority — the LEDs drive correctly; only the status string is briefly wrong. @@ -447,7 +492,7 @@ where upscaling has the least to offer. **The fix when it earns its place:** iterate the OUTPUT rows rather than the input lights, so writes are sequential: for each output row, walk its source row once and emit `scale` copies of -each light's colour, then `memcpy` that finished row to the remaining `scale - 1` rows of the +each light's color, then `memcpy` that finished row to the remaining `scale - 1` rows of the block. Same output, one pass through the destination in address order. ### Sprite follow-ups (draw::sprite + FlyingToasters shipped; spec + plan in the plans archive) @@ -489,16 +534,26 @@ a codec. So the three things a Tab5 could be are separate pieces of work, and on ### Multi-card walls — does a daisy chain work today? (open, ask before building) -The ColorLight format has **no card addressing**: the destination MAC is a fixed constant and every card filters on it, so every card on a segment shows the same image. A user with six cards on a switch observed exactly that. +The ColorLight format has **no card addressing in the PIXEL path**: the destination MAC is a fixed +constant and every card filters on it, so every card on a segment shows the same image. A user with +six cards on a switch observed exactly that. + +The DISCOVERY path does distinguish them. A discovery reply (0x08) carries a controller number at +payload offset 0x62, and the acknowledgement echoes it plus one, which is how a sender tells several +cards apart. Documented by a reader of [Harald Kubota's protocol +write-up](https://hkubota.wordpress.com/2022/01/31/winter-project-colorlight-5a-75b-protocol/) and +confirmed by its author. That is an identity, not a destination: it does not let a sender aim pixel +data at one card, so the same-image behavior above stands. It is what a per-card brightness or +color-temperature feature below would key on. -The industry-standard answer is **daisy-chaining** — a sending card's ports each drive a chain, and each card takes its region by position in the chain. That user works around it with per-card VLANs and a managed switch instead, which he built for throughput and for per-card colour-temperature grouping across mixed panel batches; he described it as his own solution, not a standard. +The industry-standard answer is **daisy-chaining** — a sending card's ports each drive a chain, and each card takes its region by position in the chain. That user works around it with per-card VLANs and a managed switch instead, which he built for throughput and for per-card color-temperature grouping across mixed panel batches; he described it as his own solution, not a standard. **Establish first whether a daisy chain already works with projectMM** (one contact has a 96K daisy-chained rig). If the cards self-assign by chain position, the standard multi-card case is already solved and nothing is needed. Only if it does not work is there a feature here, and it should follow the daisy-chain standard rather than the VLAN workaround. 802.1Q tagging is technically a clean fit for a raw-L2 sender (the tag is part of the Ethernet header, the switch strips it before the card, so card firmware is unaffected), but it serves one bespoke architecture. ### Smaller asks from the same thread - **Read the wall layout from the ColorLight cards.** The cards can report their configuration and at least one user's own tool already does it; it would remove the manual layout step. -- **Per-card colour temperature and brightness**, via the ColorLight sync-packet bytes, grouped by sync group — used to colour-match mixed panel batches live. +- **Per-card color temperature and brightness**, via the ColorLight sync-packet bytes, grouped by sync group — used to color-match mixed panel batches live. - **Docker image**, asked for by a user tracking updates in an IoT system. The Linux binary and `.deb` already ship, so this is packaging rather than new capability. ## Sensors and audio-reactive input @@ -685,7 +740,6 @@ The LED-driver increments **shipped**: increment 1 (RMT/WS2812B single-strand on **Build and prove chunking on the unshifted path first** (Parlio 4,096 → 16,384 is the measurable win, on proven code), then let shift mode inherit it — that is a sequencing rule about *where to de-risk the mechanism*, **not** a claim that the expander is optional. It is not: it is the only route to 100 fps at this scale without spending 48+ GPIOs. Correct WS2812 inter-chunk timing is the one hard constraint: the lines must idle LOW for < 300 µs between chunks or the strand latches mid-frame. The driver already rejects an over-limit frame with a loud status. Measured detail: [performance.md § Multi-pin](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid). - **`rmtWs2812Show` fuller error handling** (deferred from PR #17 / 🐇 CodeRabbit). The shipped path has a finite `rmt_tx_wait_all_done` timeout (1 s) so a wedged DMA can't hang the render tick forever, and a dropped frame self-heals (the driver re-encodes the whole frame next tick). The fuller version — `rmt_transmit` return check, `rmt_tx_stop` to cancel an in-flight transfer on timeout, `show()` returning failure so `loop()` won't reuse `symbols_` mid-transmit — belongs with the **core-1 driver-task** work, since that task owns the buffer lifetime and in-flight state the cancel logic needs. -- **Surface RMT symbol-buffer alloc failure as a status** (bench-found 2026-07-12, [multi-pin driving results](../performance.md#multi-pin-led-driving-all-three-peripherals-128128-grid)). `resizeSymbols()` sizes for the driver's `count` window, so on a classic ESP32 (~90 KB heap) a whole-grid window (`count=0` on a 128×128 grid ≈ 1.5 MB) fails to allocate: `symbols_` stays null, `tick()` bails at its `!symbols_` guard, and the strip goes **dark with no status** — the user sees nothing lit and no error. The fix mirrors the Parlio over-limit guard (already loud): when the symbol alloc returns null, set a clear "not enough memory — reduce lights or use start/count" status instead of silently idling. Small, robustness-principle work; pairs with the fuller RMT error handling above. - **Auto-derived DMA buffer count** (7 / 30 / 75 per [analysis §7.4](../history/leddriver-analysis-top-down.md)), **16-bit pipeline + dither** ([§7.3](../history/leddriver-analysis-top-down.md)), **shift-register expander stubs** ([§7.5](../history/leddriver-analysis-top-down.md)). - **IR RX live-reconfigure recovery — unconfirmed, park until it recurs** (bench 2026-07-13, SE16). IR reception on the SE16 (`IrService` pin 5) went dead mid-session and only a **hard reset** brought it back; a warm/API path did not. **Ruled out:** not hardware (hard reset fixed it, receiver+switch+wiring fine), not LED-count (IR survives the full 16384-light / 8 fps load — a received code still toggled a control at max load), not a regression from the i80 commit (`platform_esp32_ir.cpp` untouched, the 1250 ns glitch-filter fix intact). **Prime suspect (unproven):** the session's live pin churn — including transiently setting the i80 `clockPin` to **5, which IS the IR pin** — left GPIO 5 routed to the wrong peripheral, and the RMT-RX channel (a pin-keyed static behind `platform::irStop`/`ensureChannel`) didn't re-acquire cleanly on the next `irRead`; only a full GPIO re-init (hard reset) cleared it. This may be pure test artifact (nothing in a *normal* user flow points two live modules at GPIO 5). **To conclude:** from a fresh hard reset (IR working), in isolation set i80 `clockPin=5` then restore `clockPin=8` and check whether IR dies and whether it self-recovers *without* a hard reset — self-recovers → no bug (test artifact); stays dead → a real live-reconfigure gap in the IR channel re-acquire worth fixing (per *No reboot to apply a configuration change*). Small robustness/repro work; do it only if IR breaks again in real use. - **Moving-head preview = peer interpreter.** When moving heads land, the previewer must interpret channel semantics (pan/tilt/RGBW-at-arbitrary-indices) to render a moving fixture — the same light-preset model physical drivers use, interpreted to screen. This is *why* the increments named the abstraction "interpret the preset" rather than "apply correction / opt out": so Preview becomes a full peer here without a rename. Its own design plan when moving-head support starts. @@ -867,3 +921,33 @@ index on load, so a stored selection survives a re-sort. `paletteScript` already file name for the editor, so the value exists; what is missing is using it as the authority when the list changes. The alternative, appending new scripts rather than sorting them, keeps indices stable but makes the picker unreadable as the list grows, which is the trade the sort was chosen over. + +## Classic-board memory: the wins are system-level, not per-module (2026-09-09) + +The Dig-Octa on a clean boot with RmtLed has **88,284 bytes free internal, 86,016 largest block**. +Under load and after driver swaps that largest block fell to 24,576, so fragmentation matters as +much as the total. + +What the modules hold is NOT where the RAM went. The whole tree reports **9,342 bytes** of +`dynamicBytes` (Layer 3,586, Preview 2,560, RmtLed 1,536, Drivers 1,536). So per-module trimming has +almost nothing left to win, and the remaining ~230 KB of internal RAM in use is WiFi, lwIP, FreeRTOS +task stacks and the binary's static data. + +Where to look, in order of likely return: + +- **Task stack sizes.** Every task allocates its stack from internal RAM at creation, and the + defaults are generous. The Tasks module already reports them, so the measurement exists. +- **lwIP pool sizes** (`CONFIG_LWIP_*`): TCP PCBs, pbuf counts, and the send/receive windows are all + sdkconfig knobs, sized for a general-purpose device rather than a controller with a handful of + connections. +- **WiFi buffer counts** (`CONFIG_ESP32_WIFI_*_BUFFER_NUM`): the static RX/TX buffer pools are the + single largest tunable block on a classic board. +- **Fragmentation, not just totals.** A 24 KB largest block with 50 KB free is a placement problem; + allocating the long-lived buffers early and together is what fixes that. + +Each is an sdkconfig change measurable on the bench in minutes (`freeInternal` and `maxBlock` on a +clean boot), and each risks a different failure: too few WiFi buffers drops packets under load, too +small a stack overflows under a rare path. So measure one at a time on a board that is actually +serving the UI, not idle. The payoff is real: at 88 KB free a classic board is one large allocation +away from trouble, which is what drove both driver decisions on 2026-09-09. + diff --git a/docs/history/plans/Plan-20260908 - Stream the WebSocket state instead of buffering it (attempted, reverted).md b/docs/history/plans/Plan-20260908 - Stream the WebSocket state instead of buffering it (attempted, reverted).md new file mode 100644 index 00000000..59d29afa --- /dev/null +++ b/docs/history/plans/Plan-20260908 - Stream the WebSocket state instead of buffering it (attempted, reverted).md @@ -0,0 +1,191 @@ +# Plan: stream the WebSocket state instead of buffering it whole + +> **Reverted the same day.** The design was built and verified on the bench (the Dig-Octa received +> its 44,539-byte state as 6 fragments, byte-identical to `/api/state`, where it had been truncating +> at 30,719), and then backed out: the chain still holds the WHOLE document, as slices, for the +> duration of the drain. On a 65 KB board a UI refresh (a second `/ws` client mid-drain, the `/wsp` +> preview, plus `/api/types` and `/api/state` fetches) took the heap under what lwIP needs and the +> board crashed. The bench check had been one raw client on one socket, which proved the framing and +> not the memory budget. +> +> The product owner's call was to reconsider the architecture instead: **snapshot over HTTP, deltas +> over WebSocket**. `GET /api/state` already streams through a 1 KB socket-mode sink with no document +> in RAM, and the value patches already exist; what is missing is the UI fetching the snapshot on WS +> open and a small `{"resync":true}` on a structural change. That deletes the full-state-over-WS path +> rather than shrinking it. Tracked in [backlog-core.md](../../backlog/backlog-core.md). +> +> Kept from this work: `JsonSink` now FLAGS a refused heap grow instead of truncating silently +> (`unit_JsonSink_overflow`), which is the bug that made a cut document indistinguishable from a +> whole one. + +## The problem, measured + +A classic ESP32 cannot build its own state document. `buildStateJson` serializes the whole module +tree into one heap buffer, and on the bench (2026-09-08) that document is **44,261 bytes** on the +QuinLED Dig-Octa and **42,365** on the Olimex. The buffer grows by doubling, and old and new are both +live across the `memcpy`, so reaching 64 KB asks for roughly 96 KB of CONTIGUOUS heap. The Dig-Octa +has 57 KB as its largest block. + +The failure was silent and total. `JsonSink::append` dropped on a refused allocation and returned +without a flag, `buildStateJson` finished "successfully" holding a truncated document, and +`startBufferedTextSend` wrote a correct WebSocket header declaring it complete. The browser parsed +it, threw, and dropped every module past the cut: **effect cards simply vanished**, with nothing +logged on the device or in the console. Both classic boards shipped a cut document, the Dig-Octa +losing 11,226 bytes and the Olimex 8,101. + +Two fixes already landed and neither is sufficient: + +- `JsonSink::append` now sets `overflowed_`, so a truncated document is detectable. +- `ensureHeap` backs off in quarters when a doubling is refused, so growth lands on a size the heap + can serve rather than giving up. That moved the Dig-Octa's cut from 16,383 to 30,719 bytes. It + still does not reach 44,261, and no back-off can: grow-and-copy needs old plus new at once. + +One dead end is worth recording. Refusing to send an overflowed document (returning early) FROZE THE +WHOLE UI: `fullResyncPending_` stayed set, the else-branch that pushes value patches was never +reached, and the header stopped updating fps and free heap. A partial tree that keeps updating beats +a whole one that never arrives. That refusal was replaced by a warning. + +**Ethernet dissolves it and is not the answer.** A board on Ethernet never starts the WiFi stack and +keeps ~50 KB more heap, which is why the Olimex (Ethernet) looks healthy and the Dig-Octa (WiFi STA) +does not. Most users are on WiFi, so the fix has to work there. + +## What the current design buys, and must keep + +The buffer is not laziness. Three constraints hold it up, and a replacement has to satisfy all three: + +1. **The render loop must never block.** `drainStateSend` writes what the socket takes on `tick20ms` + and leaves the rest, bounded by `drainChunkBytes()` (a fraction of the largest free block, floor + 2048, ceiling 65536). +2. **Clients drain at their own pace.** `stateSend_.sent[MAX_WS_CLIENTS]` is a per-client cursor over + the same body; a slow client lags without holding the others. +3. **A message stays atomic per client.** While a state send is active, patch and WLED pushes to + `/ws` are skipped so nothing interleaves inside one WebSocket message. + +## Approach: fragment the message, keep one modest buffer + +WebSocket messages may be split across frames: a first frame carrying the real opcode with FIN +clear, then continuation frames (opcode 0x0), the last with FIN set. The browser reassembles them +into one `message` event, so **the client needs no change at all**. + +The state is then produced and sent in slices of a few KB. The peak allocation becomes one slice +rather than the whole document, which removes the contiguous-block requirement entirely. + +The one hard question is that `buildStateJson` is a single forward walk of the module tree: it cannot +be resumed from an arbitrary byte offset. Three ways to reconcile that with per-client cursors, and +the plan takes the third: + +- **A. Serialize per client, straight to the socket.** `JsonSink` already has socket mode. Simplest + to write, but it re-walks the whole tree once per client and blocks on a slow socket, breaking + constraint 1. +- **B. Resumable serialization.** Teach `buildStateJson` to stop at a slice boundary and resume from + a saved position in the tree. No large buffer at all, but it means a serialization cursor + (module index, control index, partial-value state) that has to stay correct while the tree can + change underneath it. The most invasive option and the easiest to get subtly wrong. +- **C. Slice-at-a-time with a shared buffer (chosen).** Serialize into a fixed slice buffer, send + that slice as one fragment to every client, and only then produce the next. Peak memory is one + slice; the tree is walked once; no per-client re-serialization. The cost is that the slowest client + paces the others, which is acceptable: the state frame is rare (connect and structural change) and + a slow client already stalls behind its own cursor today. + +## As built (2026-09-08) + +The word "ring" in option C2 was misleading, and the design that landed is simpler. The constraint +was never TOTAL memory (the Dig-Octa has 62 KB free for a 44 KB document) but CONTIGUOUS memory (a +49 KB largest block against a doubling buffer that needs ~96 KB). So the document is serialized ONCE, +at push time, into a **chain of small slices**, each its own allocation of `drainChunkBytes()` bytes, +which any fragmented heap can serve. Nothing pauses the walk and nothing waits for a client. + +- **`JsonSink` slice mode** (`JsonSink.h`): a caller-owned slice plus a `FlushFn`; a full slice is + handed over and reused, `finish()` flushes the tail, a failed flush sets `overflowed()`. +- **`WsSliceChain`** (`WsSliceChain.h`, header-only like `JsonSink`): the flush target. Each slice + carries `kHdrMax` bytes of frame-header room in front of its payload, so `finalize(opcode)` stamps + the fragment headers in place (first: opcode with FIN clear; middle: 0x00; last: 0x80; a single + slice: 0x81, byte-identical to the old framing) and every frame is one contiguous span. A per-client + `Cursor` walks the chain; slices are shared, cursors are not. `writeWsFrameHeader` moved here. +- **`HttpServerModule`**: the push allocates one staging slice, serializes through it into + `stateSend_.chain`, frees the staging slice, and arms `startStateChainSend()`. `drainStateSend` + moves bytes from `remaining()` and `advance()`s; the chain is freed once every live client is done. + A refused allocation mid-document clears the chain, warns, and clears `fullResyncPending_` so the + value patches keep flowing (the early-return that froze the whole UI is the dead end above). + +**Tests**: `unit_JsonSink_slices` (4 cases) and `unit_WsSliceChain` (6 cases: the exact frame +sequence, the single-slice framing unchanged, byte-for-byte reassembly at socket writes of 1, 7, 333 +and 100000 bytes, eight clients at eight speeds on one chain, a refused allocation refusing to +finalize, clear-and-reuse). Both control-checked: sabotaging the tail flush or the last frame's FIN +fails exactly the test that pins it. 1,910 unit tests green. + +**Bench** (Dig-Octa .181, WiFi, 65 KB free, 59 KB largest block, the board that lost its effect +cards): a raw WebSocket client received the state as **6 frames** (opcode 1 with FIN clear, four +continuations, FIN on the last: 5 x 7,935 + 4,864 bytes) and reassembled **44,539 bytes, +byte-identical to `/api/state`**, with both effects present. The first flash sent nothing: the +document holds formatted fragments over 256 bytes, and `appendf`'s long-fragment branch flagged +overflow in slice mode instead of flushing, so every chain was dropped. Fixed and pinned +(`unit_JsonSink_slices`, the long-fragment case). + +## Steps + +### 1. A slice-sized sink +`JsonSink` gains a mode that fills a caller-owned fixed buffer and, when full, hands it to a callback +before continuing. Fixed-buffer mode already exists and flags overflow instead of growing; this is +that path plus a flush hook, so no new buffer strategy is introduced. Slice size comes from +`drainChunkBytes()`, so a tight board takes small slices and a roomy one takes large. + +### 2. Fragmented framing +`writeWsFrameHeader` already encodes any length; it needs the FIN bit and opcode as parameters +rather than always `0x81`. First slice: opcode 0x1, FIN clear. Middle: opcode 0x0, FIN clear. Last: +opcode 0x0, FIN set. A single-slice document keeps today's exact framing (0x81), so the common small +case is unchanged. + +### 3. Slice-aware StateSend +`stateSend_` holds one slice plus its header, with the per-client cursor over THAT slice. When every +live client has drained it, the next slice is produced and the cursors reset. Add a +`slicesRemaining` / `lastSlice` flag so the drain knows when the message is complete and only then +clears `fullResyncPending_` and rebaselines the leaf hashes. + +### 4. Failure paths +A client that dies mid-message is dropped as today. A client that connects DURING a send must not +receive a half message: it waits for the next full state (`fullResyncPending_` is set on connect +anyway). If a slice cannot be allocated at all, warn and keep the patch stream alive, which is the +lesson from the refusal that froze the UI. + +## Tests + +- `unit_JsonSink_slices`: a document larger than the slice buffer emits the expected sequence of + slices, and reassembling them byte-for-byte equals the same document built in buffer mode. +- `unit_HttpServer_fragmented_state`: the frame sequence for a multi-slice document is + `0x01 FIN=0`, `0x00 FIN=0`, ..., `0x00 FIN=1`, and a single-slice document is exactly `0x81 FIN=1` + (today's framing, so small devices are untouched). +- A regression test for the original bug: a state document that exceeds any single allocation still + arrives complete and parseable. +- Scenario: the existing WS state scenario re-run, with the observation block showing the tick cost + did not regress. + +## Verification + +Desktop first. Then on the bench, both classic boards, since they bracket the problem: + +1. **Dig-Octa (.181, WiFi, 57 KB max block, 44 KB state)**: the whole tree arrives, every effect card + renders including the MoonLive one that vanished, and the header keeps updating. +2. **Olimex (.210, Ethernet, 102 KB max block)**: unchanged behavior, no regression on a board that + was already comfortable. +3. A raw WebSocket client asserts the reassembled document is valid JSON and matches `/api/state` + byte-for-byte. +4. Watch the tick cost on both: slicing runs on `tick20ms` and must not lengthen the render tick. + +## Risks + +- **A half-sent tree is worse than a truncated one.** If the last fragment is lost the browser holds + an incomplete message forever. The FIN bookkeeping is the load-bearing part and the test above + pins it directly. +- **The tree can change between slices.** A module added or removed mid-message would produce a + document that is internally inconsistent. Mitigation: a structural change during a send sets + `fullResyncPending_` again, so the next full state supersedes it, exactly as today. +- **Slower clients now pace each other.** Accepted, and reversible: per-client slice buffers would + undo it at the cost of the memory this plan exists to save. + +## Out of scope, worth noting + +`LightPresets` alone is **13,086 bytes, 30% of the document**: one `list` control holding 13 built-in +presets that never change at runtime. Fetching it on demand rather than pushing it in every state +would cut the document by a third and help every board. Real, separate, and not a substitute for +streaming, since 31 KB still would not fit on a tight board. diff --git a/docs/metrics/repo-health.json b/docs/metrics/repo-health.json index 8fc21bd1..a73e6f4a 100644 --- a/docs/metrics/repo-health.json +++ b/docs/metrics/repo-health.json @@ -1,47 +1,49 @@ { - "commit": "52e03cbe", + "commit": "b2a5bce6", "flash": { - "esp32s3-n16r8": 2100032, - "desktop": 1909016, - "esp32": 2059664, - "esp32p4rev1-eth": 1979232, - "esp32p4rev1-eth-wifi": 2019392, + "esp32s3-n16r8": 2103024, + "desktop": 1910456, + "esp32": 2060272, + "esp32p4rev1-eth": 1997440, + "esp32p4rev1-eth-wifi": 2284640, "esp32s3-n8r8": 2087168, "esp32s31": 2348592, - "esp32-16mb": 1809472, + "esp32-16mb": 2060368, "esp32-eth": 1397456, "esp32-wrover": 1843760, "qemu": 1383648, "esp32p4rev3-eth": 1643760, "esp32s3-zero": 2024192, - "esp32-pico": 2071584 + "esp32-pico": 2107168 }, "measured": { - "esp32p4rev1-eth": "2026-09-08", + "esp32p4rev1-eth": "2026-09-09", "esp32s31": "2026-09-06", - "esp32": "2026-09-08", - "esp32-pico": "2026-09-06", - "esp32s3-n16r8": "2026-09-08", - "desktop": "2026-09-08", + "esp32": "2026-09-09", + "esp32-pico": "2026-09-09", + "esp32s3-n16r8": "2026-09-09", + "desktop": "2026-09-09", "esp32s3-n8r8": "2026-09-08", - "esp32s3-zero": "2026-09-08" + "esp32s3-zero": "2026-09-08", + "esp32-16mb": "2026-09-09", + "esp32p4rev1-eth-wifi": "2026-09-08" }, "perf": { "desktop": { - "tick_us": 140, - "fps": 7142, + "tick_us": 147, + "fps": 6802, "scenario_p50": { "Layer_base_pipeline": { - "p50": 71, - "p95": 197, + "p50": 69, + "p95": 139, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "Layer_memory_1to1": { "p50": 5, - "p95": 40, + "p95": 11, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" } } }, @@ -52,10 +54,10 @@ "scenario_matrix": { "MoonModule_control_change": { "desktop-macos": { - "p50": 143, - "p95": 248, + "p50": 125, + "p95": 246, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "esp32-eth-wifi": { "p50": 89895, @@ -168,10 +170,10 @@ }, "Audio_mutation": { "desktop-macos": { - "p50": 26, - "p95": 70, + "p50": 23, + "p95": 61, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 40, @@ -194,18 +196,18 @@ }, "Aurora_fps": { "desktop-macos": { - "p50": 1518, - "p95": 1828, - "n": 26, - "last": "2026-09-07" + "p50": 1556, + "p95": 2013, + "n": 32, + "last": "2026-09-09" } }, "Driver_mutation": { "desktop-macos": { "p50": 20, - "p95": 88, + "p95": 29, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 42, @@ -228,10 +230,10 @@ }, "Effects_composition": { "desktop-macos": { - "p50": 148, - "p95": 768, + "p50": 147, + "p95": 248, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 549, @@ -242,26 +244,26 @@ }, "Fields_polar_lut": { "desktop-macos": { - "p50": 1239, + "p50": 1261, "p95": 1686, - "n": 27, - "last": "2026-09-07" + "n": 32, + "last": "2026-09-09" } }, "Fluid_solver": { "desktop-macos": { - "p50": 222, - "p95": 259, - "n": 20, - "last": "2026-09-07" + "p50": 226, + "p95": 265, + "n": 30, + "last": "2026-09-09" } }, "GridBlacks_blackpixel": { "desktop-macos": { "p50": 2, - "p95": 16, + "p95": 3, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "esp32s3-n16r8": { "p50": 267, @@ -284,10 +286,10 @@ }, "GridLayout_resize": { "desktop-macos": { - "p50": 127, - "p95": 311, + "p50": 122, + "p95": 189, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "esp32-eth-wifi": { "p50": 82231, @@ -328,10 +330,10 @@ }, "Layer_base_pipeline": { "desktop-macos": { - "p50": 71, - "p95": 197, + "p50": 69, + "p95": 139, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 118, @@ -343,9 +345,9 @@ "Layer_memory_1to1": { "desktop-macos": { "p50": 5, - "p95": 40, + "p95": 11, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 1, @@ -356,10 +358,10 @@ }, "Layouts_mutation": { "desktop-macos": { - "p50": 97, - "p95": 248, + "p50": 94, + "p95": 169, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 111, @@ -410,8 +412,8 @@ "desktop-macos": { "p50": 6, "p95": 8, - "n": 16, - "last": "2026-09-08" + "n": 26, + "last": "2026-09-09" }, "esp32s3-n16r8": { "p50": 8255, @@ -458,10 +460,10 @@ "last": "2026-08-20" }, "desktop-macos": { - "p50": 6, - "p95": 21, + "p50": 5, + "p95": 11, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 1, @@ -473,9 +475,9 @@ "MultiplyModifier_memory_lut": { "desktop-macos": { "p50": 3, - "p95": 19, + "p95": 4, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 3, @@ -486,10 +488,10 @@ }, "MultiplyModifier_pipeline": { "desktop-macos": { - "p50": 126, - "p95": 283, + "p50": 121, + "p95": 172, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 225, @@ -500,18 +502,18 @@ }, "Trails_ladder": { "desktop-macos": { - "p50": 360, + "p50": 364, "p95": 442, - "n": 21, - "last": "2026-09-07" + "n": 31, + "last": "2026-09-09" } }, "modifier_chain": { "desktop-macos": { - "p50": 44, - "p95": 112, + "p50": 43, + "p95": 48, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 69, @@ -528,10 +530,10 @@ }, "modifier_swap": { "desktop-macos": { - "p50": 24, - "p95": 87, + "p50": 22, + "p95": 31, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "esp32-eth": { "p50": 1010, @@ -566,10 +568,10 @@ }, "perf_full": { "desktop-macos": { - "p50": 279, - "p95": 1114, + "p50": 265, + "p95": 402, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "esp32s3-n16r8": { "p50": 16915, @@ -598,10 +600,10 @@ }, "perf_light": { "desktop-macos": { - "p50": 17, - "p95": 61, + "p50": 16, + "p95": 24, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "esp32s3-n16r8": { "p50": 2485, @@ -642,10 +644,10 @@ "last": "2026-07-25" }, "desktop-macos": { - "p50": 296, - "p95": 881, + "p50": 272, + "p95": 813, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "desktop-windows": { "p50": 649, @@ -669,9 +671,9 @@ }, "desktop-macos": { "p50": 4, - "p95": 14, + "p95": 6, "n": 32, - "last": "2026-09-07" + "last": "2026-09-09" }, "esp32p4rev1-eth": { "p50": 217, @@ -695,54 +697,54 @@ } }, "loc": { - "core": 26039, - "light": 35627, - "platform": 18503, - "ui": 10621, - "test": 57455, - "moondeck": 22710 + "core": 26218, + "light": 35665, + "platform": 18705, + "ui": 10659, + "test": 57853, + "moondeck": 22906 }, "comments": { "core": { - "lines": 10452, - "ratio": 0.433 + "lines": 10544, + "ratio": 0.434 }, "light": { - "lines": 13532, + "lines": 13554, "ratio": 0.417 }, "platform": { - "lines": 6500, - "ratio": 0.385 + "lines": 6608, + "ratio": 0.387 }, "ui": { - "lines": 3164, - "ratio": 0.314 + "lines": 3190, + "ratio": 0.316 }, "test": { - "lines": 10784, + "lines": 10859, "ratio": 0.215 }, "moondeck": { - "lines": 3690, + "lines": 3723, "ratio": 0.186 } }, "tests": { - "cases": 1992, + "cases": 2004, "scenarios": 27 }, "docs": { - "md_files": 220, - "md_lines": 36832, - "plans_files": 116, - "backlog_lines": 6794, + "md_files": 222, + "md_lines": 37624, + "plans_files": 117, + "backlog_lines": 7155, "lessons_lines": 705, "claude_md_lines": 259 }, "complexity": { - "functions": 3498, - "over_threshold": 249, - "worst_ccn": 108 + "functions": 3514, + "over_threshold": 251, + "worst_ccn": 128 } } diff --git a/docs/metrics/repo-health.md b/docs/metrics/repo-health.md index 07314a18..6d0ecf66 100644 --- a/docs/metrics/repo-health.md +++ b/docs/metrics/repo-health.md @@ -1,6 +1,6 @@ # Repo health -Measured at `52e03cbe`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** +Measured at `b2a5bce6`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** Current state only; the trend is this file's git history (`git log -p docs/metrics/repo-health.md`). Nothing here fails a build: the numbers make growth visible, the judgment stays human. @@ -8,19 +8,19 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Target | Flash | Capacity | Used | Built | |---|---:|---:|---:|:--:| -| desktop | 1,864 KB | - | - | yes | -| esp32 | 2,011 KB (+3 KB) ⚠ | 2,496 KB | 81% | yes | -| esp32-16mb | 1,767 KB | - | - | carried (age?) | +| desktop | 1,866 KB (+0 KB) ⚠ | - | - | yes | +| esp32 | 2,012 KB | 2,496 KB | 81% | yes | +| esp32-16mb | 2,012 KB (+0 KB) ⚠ | 4,096 KB | 49% | yes | | esp32-eth | 1,365 KB | - | - | carried (age?) | -| esp32-pico | 2,023 KB | 3,072 KB | 66% | carried 2d | +| esp32-pico | 2,058 KB | 3,072 KB | 67% | yes | | esp32-wrover | 1,801 KB | - | - | carried (age?) | -| esp32p4rev1-eth | 1,933 KB | 4,096 KB | 47% | yes | -| esp32p4rev1-eth-wifi | 1,972 KB | - | - | carried (age?) | +| esp32p4rev1-eth | 1,951 KB (+0 KB) ⚠ | 4,096 KB | 48% | yes | +| esp32p4rev1-eth-wifi | 2,231 KB | 4,096 KB | 54% | carried 1d | | esp32p4rev3-eth | 1,605 KB | - | - | carried (age?) | -| esp32s3-n16r8 | 2,051 KB | 4,096 KB | 50% | yes | -| esp32s3-n8r8 | 2,038 KB | 3,072 KB | 66% | yes | -| esp32s3-zero | 1,977 KB | 2,496 KB | 79% | yes | -| esp32s31 | 2,294 KB | 4,096 KB | 56% | carried 2d | +| esp32s3-n16r8 | 2,054 KB (+0 KB) ⚠ | 4,096 KB | 50% | yes | +| esp32s3-n8r8 | 2,038 KB | 3,072 KB | 66% | carried 1d | +| esp32s3-zero | 1,977 KB | 2,496 KB | 79% | carried 1d | +| esp32s31 | 2,294 KB | 4,096 KB | 56% | carried 3d | | qemu | 1,351 KB | - | - | carried (age?) | `Built: yes` was measured this run. `carried (age?)` was not rebuilt either and predates this record, so its age is unknown: it dates itself on the next build. `carried Nd` was NOT rebuilt and its number is N days old, so an absent delta says nothing about the change. **STALE** marks a carry older than 7 days: the number has gone unchecked long enough that growth will surface later as one jump, blamed on whichever commit happens to rebuild that target. `Used` is against the app slot in the firmware's own partition table. @@ -29,39 +29,39 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Target | Tick | FPS | |---|---:|---:| -| desktop | 140 µs (−489 µs) ✓ | 7,142 (+5,553) ✓ | +| desktop | 147 µs (+7 µs) ⚠ | 6,802 (−340) ⚠ | | esp32 | 8,354 µs | 119 | ### Scenario tick by target (p50 of each sample window) | Scenario | desktop-macos | desktop-windows | esp32 | esp32s3-n16r8 | esp32p4rev1-eth | esp32s31 | esp32-eth | esp32-eth-wifi | unknown | |---|---|---|---|---|---|---|---|---|---| -| Audio_mutation | 26 | 40 ? | 13,152 | 47 ? | - | - | - | - | - | -| Aurora_fps | 1,518 | - | - | - | - | - | - | - | - | +| Audio_mutation | 23 | 40 ? | 13,152 | 47 ? | - | - | - | - | - | +| Aurora_fps | 1,556 (+22) ⚠ | - | - | - | - | - | - | - | - | | Driver_mutation | 20 | 42 ? | 12,812 | 39 ? | - | - | - | - | - | -| Effects_composition | 148 | 549 ? | - | - | - | - | - | - | - | -| Fields_polar_lut | 1,239 | - | - | - | - | - | - | - | - | -| Fluid_solver | 222 | - | - | - | - | - | - | - | - | +| Effects_composition | 147 | 549 ? | - | - | - | - | - | - | - | +| Fields_polar_lut | 1,261 (+21) ⚠ | - | - | - | - | - | - | - | - | +| Fluid_solver | 226 (+2) ⚠ | - | - | - | - | - | - | - | - | | GridBlacks_blackpixel | 2 | 8 ? | 269 ? | 267 ? | - | - | - | - | - | -| GridLayout_resize | 127 | 219 ? | 1,352 ? | 1,011 ? | 1,143 ? | - | 95,771 ? | 82,231 ? | - | -| Layer_base_pipeline | 71 | 118 ? | - | - | - | - | - | - | - | +| GridLayout_resize | 122 | 219 ? | 1,352 ? | 1,011 ? | 1,143 ? | - | 95,771 ? | 82,231 ? | - | +| Layer_base_pipeline | 69 | 118 ? | - | - | - | - | - | - | - | | Layer_memory_1to1 | 5 | 1 ? | - | - | - | - | - | - | - | -| Layouts_mutation | 97 | 111 ? | 13,692 | 45 ? | - | - | 27 ? | - | - | +| Layouts_mutation | 94 | 111 ? | 13,692 | 45 ? | - | - | 27 ? | - | - | | MoonLiveEffect_controls | 11 ? | - | 12,901 | 4,624 ? | - | - | - | - | - | | MoonLiveEffect_livescript | 6 | - | 13,433 ? | 8,255 ? | 11,336 ? | - | - | - | - | -| MoonLive_pipeline | 6 | 1 ? | 9,604 ? | 3,278 ? | - | 11,398 ? | - | - | 4,393 ? | -| MoonModule_control_change | 143 | 262 ? | 212 ? | 166 ? | 165 ? | - | 111,731 ? | 89,895 ? | - | +| MoonLive_pipeline | 5 | 1 ? | 9,604 ? | 3,278 ? | - | 11,398 ? | - | - | 4,393 ? | +| MoonModule_control_change | 125 | 262 ? | 212 ? | 166 ? | 165 ? | - | 111,731 ? | 89,895 ? | - | | MqttModule_haDiscovery_toggle | 3 ? | - | 36 ? | 36 ? | - | - | - | - | - | | MultiplyModifier_memory_lut | 3 | 3 ? | - | - | - | - | - | - | - | -| MultiplyModifier_pipeline | 126 | 225 ? | - | - | - | - | - | - | - | +| MultiplyModifier_pipeline | 121 | 225 ? | - | - | - | - | - | - | - | | NetworkModule_eth_reconfigure | - | - | 1,169 ? | 97,843 ? | - | - | - | - | - | | NetworkModule_mdns_toggle | 13 ? | - | 36 ? | 36 ? | 21 ? | - | 109,767 ? | 93,963 ? | - | -| Trails_ladder | 360 | - | - | - | - | - | - | - | - | -| modifier_chain | 44 | 69 ? | 13,337 | - | - | - | - | - | - | -| modifier_swap | 24 | 41 ? | 12,250 | 354 ? | 362 ? | - | 1,010 ? | - | - | -| perf_full | 279 | 592 ? | 10,392 | 16,915 ? | 17,433 ? | - | - | - | - | -| perf_light | 17 | 35 ? | 2,183 | 2,485 ? | 2,038 ? | - | - | - | - | -| peripheral_grid_sweep | 296 | 649 ? | 6,991 ? | - | 11,495 ? | 12,273 ? | - | - | - | +| Trails_ladder | 364 (+2) ⚠ | - | - | - | - | - | - | - | - | +| modifier_chain | 43 (−1) ✓ | 69 ? | 13,337 | - | - | - | - | - | - | +| modifier_swap | 22 | 41 ? | 12,250 | 354 ? | 362 ? | - | 1,010 ? | - | - | +| perf_full | 265 | 592 ? | 10,392 | 16,915 ? | 17,433 ? | - | - | - | - | +| perf_light | 16 | 35 ? | 2,183 | 2,485 ? | 2,038 ? | - | - | - | - | +| peripheral_grid_sweep | 272 | 649 ? | 6,991 ? | - | 11,495 ? | 12,273 ? | - | - | - | | peripheral_switch | 4 | 9 ? | 437 | 46 ? | 217 ? | - | - | - | - | Microseconds. `?` marks a cell backed by fewer than 4 samples, which is a first impression rather than a percentile; several are months old and were captured during a network reconfigure, so they read as whole milliseconds. `-` means that target has never run that scenario. @@ -72,8 +72,8 @@ Microseconds. `?` marks a cell backed by fewer than 4 samples, which is a first | Scenario | p50 | p95 | n | |---|---:|---:|---:| -| Layer_base_pipeline | 71 µs | 197 µs | 32 | -| Layer_memory_1to1 | 5 µs | 40 µs | 32 | +| Layer_base_pipeline | 69 µs | 139 µs | 32 | +| Layer_memory_1to1 | 5 µs | 11 µs | 32 | These build a bare pipeline with no optional modules, so a change here is a change in the pipeline itself rather than in what was measured. A new module belongs in an advanced scenario, which keeps its own numbers. @@ -81,36 +81,36 @@ These build a bare pipeline with no optional modules, so a change here is a chan | Area | Lines | Comments | Comment share | |---|---:|---:|---:| -| core | 26,039 (+20) ⚠ | 10,452 | 43.3 % | -| light | 35,627 (+1) ⚠ | 13,532 | 41.7 % | -| platform | 18,503 (+25) ⚠ | 6,500 | 38.5 % (+0.1 %) ⚠ | -| ui | 10,621 (+40) ⚠ | 3,164 | 31.4 % (+0.1 %) ⚠ | -| test | 57,455 (+53) ⚠ | 10,784 | 21.5 % | -| moondeck | 22,710 (+9) ⚠ | 3,690 | 18.6 % | +| core | 26,218 (+19) ⚠ | 10,544 | 43.4 % | +| light | 35,665 | 13,554 | 41.7 % | +| platform | 18,705 | 6,608 | 38.7 % | +| ui | 10,659 | 3,190 | 31.6 % | +| test | 57,853 (+32) ⚠ | 10,859 | 21.5 % | +| moondeck | 22,906 | 3,723 | 18.6 % | ## Tests | Kind | Count | |---|---:| -| unit cases | 1,992 (+2) ✓ | +| unit cases | 2,004 (+2) ✓ | | scenarios | 27 | ## Complexity | Metric | Value | |---|---:| -| functions | 3,498 (+1) ✓ | -| over threshold | 249 | -| worst CCN | 108 | +| functions | 3,514 | +| over threshold | 251 (+1) ⚠ | +| worst CCN | 128 | ## Documentation | Metric | Value | |---|---:| -| markdown files | 220 (+1) ⚠ | -| markdown lines | 36,832 (+1,940) ⚠ | -| plan files | 116 (+1) ⚠ | -| backlog lines | 6,794 | +| markdown files | 222 | +| markdown lines | 37,624 (+16) ⚠ | +| plan files | 117 | +| backlog lines | 7,155 (+16) ⚠ | | lessons lines | 705 | | CLAUDE.md lines | 259 | diff --git a/docs/moonmodules/light/drivers.md b/docs/moonmodules/light/drivers.md index fc36ee70..063fafde 100644 --- a/docs/moonmodules/light/drivers.md +++ b/docs/moonmodules/light/drivers.md @@ -134,6 +134,8 @@ No IP is involved — no address, no port, no DHCP — so the driver works on a Origin: ColorLight 5A-75 documented byte layout. Inspired by [FPP](https://github.com/FalconChristmas/fpp) (Falcon Player), the show player that drives these cards from a Raspberry Pi: seeing an FPP rig feed a wall of panels is what prompted this driver, since a board already rendering those frames can send them itself and remove the host from the installation. FPP is also the reference point for what good looks like here, sustaining 50 fps. +Protocol references: [FPP's ColorLight-5a-75.cpp](https://github.com/FalconChristmas/fpp/blob/master/src/channeloutput/ColorLight-5a-75.cpp) is the implementation this driver's byte layout agrees with, and Harald Kubota's [5A-75B protocol write-up](https://hkubota.wordpress.com/2022/01/31/winter-project-colorlight-5a-75b-protocol/) documents the same wire format independently, including the brightness and color-temperature bytes and the discovery exchange. Read it with its comments: a reader supplied the controller-number field that makes multiple cards on one segment distinguishable, and the article's own MAC pair is printed the other way round from FPP's (destination `11:22:33:44:55:66`, source `22:22:33:44:55:66`, which is what this driver sends and what the cards filter on). Its lineage runs back to the [original mplayer-colorlight reverse engineering](http://www.mylifesucks.de/oss/mplayer-colorlight/). + [Tests](../../tests/unit-tests.md#panelcarddriver) Detail: [technical](moxygen/PanelCardDriver.md) diff --git a/docs/performance.md b/docs/performance.md index 831e39ee..0689adea 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -400,7 +400,7 @@ Each parallel LED driver run on real hardware at a 128×128 = 16384-light grid, **LOLIN D32 (classic ESP32-WROOM) usable LED GPIOs:** `4,13,14,18,19,21,22,23,25,26,27,32,33` plus `16,17` (free on WROOM — they're the PSRAM bus only on WROVER). Avoid straps `0,2,12,15`, the onboard LED on `5`, and battery-sense on `35`; input-only `34–39` can't drive an LED. (Chip-level set: [gpio-usage.md](reference/gpio-usage.md).) -**Diagnostic used:** RMT `tickTimeUs > 1000` = actively encoding (LEDs on); a tiny ~30 µs tick = the symbol alloc failed and `tick()` bailed (dark). `dynamicBytes` is not reported for RMT (plain-heap symbol buffer), so it always reads 0. +**Diagnostic used:** RMT `tickTimeUs > 1000` = actively encoding (LEDs on); a tiny ~30 µs tick = the symbol alloc failed and `tick()` bailed (dark). `dynamicBytes` for RMT is the frame buffer (`driverHeapBytes()` returns `frameCap_`: outChannels bytes per light). The **acceptance floors** these establish for the parallel backends: RMT **8×256 = 2048** (verified above); the parallel-I2S (classic i80) driver **16×256 = 4096** (verified 2026-07-13); the virtual (shift-register) driver **48×256 = 12288** — each backend must clear its floor on real hardware. diff --git a/docs/reference/mhc-wled-esp32-p4-shield.md b/docs/reference/mhc-wled-esp32-p4-shield.md index e9a9188e..386d8764 100644 --- a/docs/reference/mhc-wled-esp32-p4-shield.md +++ b/docs/reference/mhc-wled-esp32-p4-shield.md @@ -22,17 +22,19 @@ The output/RS-485 terminals, left to right, with the P4 GPIO each carries: ### 12x outputs — level-shifted, single-ended (LED data) -The LED-data outputs. Each terminal is `O` on the silkscreen; a level shifter drives the 5 V strand from the P4's 3.3 V. The catalog wires the Parallel LED driver (peripheral `Parlio`) to the first eight (`21,20,25,5,22,23,24,27`). +The LED-data outputs. Each terminal is `O` on the silkscreen; a level shifter drives the 5 V strand from the P4's 3.3 V. The catalog wires the Parallel LED driver (peripheral `Parlio`) to the **first eight terminals in physical order**: `21,20,25,5,7,23,8,27`. + +**The first eight terminals are not the eight lowest-numbered outputs.** Positions 5 and 7 carry GPIO **7** and **8**, not 22 and 24. An earlier version of this table listed `O22`/`O24` in those positions and the catalog followed it, so a strip on terminals 5 and 7 stayed dark while GPIO 22/24 emitted on their RS-485 terminals instead (bench 2026-09-08: two panels of eight unlit, fixed by moving lanes 4 and 6 to GPIO 7/8). | Terminal | GPIO | Note | |---|---|---| -| O21 O20 O25 O5 O22 O23 O24 O27 | 21 20 25 5 22 23 24 27 | LED lanes (Parallel LED, peripheral `Parlio`, default) | -| O7 / O8 | 7 / 8 | also the I²C bus (SDA 7 / SCL 8, catalog I2cScan) | +| O21 O20 O25 O5 O7 O23 O8 O27 | 21 20 25 5 7 23 8 27 | LED lanes (Parallel LED, peripheral `Parlio`, default), in terminal order | +| O7 / O8 | 7 / 8 | ALSO the I²C bus (SDA 7 / SCL 8). Driving them as LED lanes means no I²C on this shield, which is why the catalog entry carries no I2cScan module. To get I²C back: move those two strands to `O22`/`O24` **and** set the driver's `pins` to `21,20,25,5,22,23,24,27` to match. Rewiring alone leaves the lanes still driving 7/8. | | O3 | 3 | also on RS-485 (`A-3-B`) — see note below | | O4 | 4 | also on RS-485 (`A-4-B`) — see note below | | GND | — | ground for the output block | -> **GPIO 3, 4, 22, 24 each appear TWICE** — once here (level-shifted single-ended output, `O`) and once in the RS-485 block (`A--B`). It's the *same* P4 GPIO fanned out to two output forms: driving the pin lights up **both** its `O` terminal and its `A--B` transceiver at once. Wire to whichever form you need. GPIO 21/20/25/5/23/27 have **only** the level-shifted path (no RS-485), which is why the LED-driver default uses those + 22/24 for strips and leaves 3/4 free. +> **GPIO 3, 4, 22, 24 each appear TWICE**: once here (level-shifted single-ended output, `O`) and once in the RS-485 block (`A--B`). It's the *same* P4 GPIO fanned out to two output forms: driving the pin lights up **both** its `O` terminal and its `A--B` transceiver at once. Wire to whichever form you need. GPIO 21/20/25/5/23/27 have **only** the level-shifted path (no RS-485). The LED-driver default uses those six plus **7 and 8** (the first eight terminals in physical order, per the table above), leaving 3/4/22/24 for RS-485. Moving the two strands off 7/8 to `O22`/`O24` is what frees I²C, and it needs the driver's `pins` changed to match: the terminals are wired, the lane list is not. ### 4x RS-485 — differential A/B pairs (range extender + DMX) diff --git a/docs/tutorials/installing-on-linux.md b/docs/tutorials/installing-on-linux.md new file mode 100644 index 00000000..5ccd1f53 --- /dev/null +++ b/docs/tutorials/installing-on-linux.md @@ -0,0 +1,236 @@ +# Running projectMM on a Linux machine + +projectMM runs as an ordinary Linux application: the same effect pipeline, web UI and network drivers as a board, with a real CPU behind them. A small always-on machine makes a good installation controller, whether that is a server, a Raspberry Pi, or a NanoPi. + +This page is about **deploying** to such a machine. Developing on one is [building.md](../building.md), which covers the build itself and is referenced rather than repeated here. + +> Windows, with screenshots: [Installing projectMM on a desktop](installing-to-desktop.md). Flashing a board: [Install & first light](../gettingstarted.md). + +--- + +## Which route applies to your machine + +The fork in the road is the CPU, so check it first: + +```sh +uname -m +``` + +`x86_64` is an Intel or AMD machine. `aarch64` is an arm64 board: a Raspberry Pi, a NanoPi, most single-board computers. + +| Your machine | Route | +|---|---| +| `x86_64` PC, server or VM | Install the released package | +| `aarch64` board (Pi, NanoPi, other SBC) | Build from source | + +The released Linux binaries are x86-64 only, so an arm64 board builds from source. The recipe below +is the whole of it, and the result is the identical program. + +Either route assumes a **Debian-based** system (Debian, Ubuntu, Raspberry Pi OS, Armbian), which is +what nearly every SBC image is. Another distribution works too; the package names in step 5 are the +part you would translate. + +> **x64 and amd64 are the same thing**, two names for Intel/AMD 64-bit. The distinction that matters is amd64 against **arm64**, which are genuinely different machine code. + +--- + +## x86-64: install the package + +The releases page carries `projectmm_X.Y.Z_amd64.deb` for Debian, Ubuntu and Raspberry Pi OS on Intel hardware: + +```sh +sudo apt install ./projectmm_X.Y.Z_amd64.deb +projectMM +``` + +That puts it on your `PATH`. There is also a `.tar.gz` to unpack anywhere. Both are listed in the [README](https://github.com/MoonModules/projectMM#readme). + +Open `http://:8080` and the UI is there. + +--- + +## arm64: build from source + +The same recipe on a Raspberry Pi and a NanoPi, and on most other Debian-family boards. Allow an +hour the first time, most of it waiting. + +### 1. Write an OS image to the SD card + +**Take any Debian-based image**, and the rest of this page works unchanged. Raspberry Pi OS, Armbian +and most vendor images are all Debian underneath, so they share `apt`, the same package names, and +systemd. That is the only thing this recipe depends on, which is why it is the requirement rather +than a particular distribution. + +Two ways to get one: + +- **The vendor image.** For a Raspberry Pi that is [Raspberry Pi OS](https://www.raspberrypi.com/software/), and take the **Lite** build: the desktop one carries a lot a controller never uses, and leaves less memory for the build. For a NanoPi it is the FriendlyELEC image linked from that board's wiki page. +- **[Armbian](https://www.armbian.com/)**, which is the better answer as soon as you have more than one kind of board: Raspberry Pi OS is for the Pi alone, while Armbian covers Rockchip and Amlogic boards too, from one project and with more regular updates than most vendor images. Check its [board list](https://www.armbian.com/download/) first, since coverage varies and a board can be in development without a released image. + +**What to avoid is a router firmware.** FriendlyELEC ships **FriendlyWrt** (OpenWrt-based) alongside +Debian, Ubuntu and Buildroot for the NanoPi R28S, all from [their wiki](https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R28S#Flashing_the_OS_to_the_microSD_card). +OpenWrt uses a different package manager and none of the steps below apply to it. Prefer a newer +Debian release where the vendor offers one, for the longer-supported kernel. + +Write it with [Raspberry Pi Imager](https://www.raspberrypi.com/software/) or [balenaEtcher](https://etcher.balena.io/), both of which take the compressed download directly. + +**On a Raspberry Pi, use Imager's settings dialog before writing.** Current Raspberry Pi OS ships with no default user and **SSH switched off**, so a card written without it boots to a machine you cannot log in to remotely. The dialog sets the username and password, the hostname, your WiFi credentials, and enables SSH. Doing it here saves needing a keyboard and monitor later. + +### 2. First boot + +Put the card in, plug in the network cable, then power. **Give it 10 to 20 minutes**: a first boot resizes the filesystem and generates host keys, and the board may reboot itself while doing so. It is not stuck. + +Then find it on the network. Any of these works: + +```sh +ping raspberrypi.local # or NanoPi-R28S.local +arp -a # everything the network has seen +``` + +Your router's client list is the reliable fallback when mDNS is not resolving. + +### 3. Log in + +```sh +ssh pi@NanoPi-R28S # FriendlyELEC Debian: user pi, password pi +ssh @.local # Raspberry Pi OS: the user you set in Imager +``` + +On the FriendlyELEC Debian image the hostname is the hardware model, so `NanoPi-R28S` resolves without a `.local` suffix, and the root account is disabled (`sudo passwd root` if you ever want it). Credentials differ per image and the board's own wiki is the authority. **Change a default password immediately:** + +```sh +passwd +``` + +If it is not on the network yet, attach a keyboard and monitor, or a USB serial adapter, and configure it there: + +```sh +sudo nmtui # a menu for WiFi and static addresses +ip ad # what addresses the board actually has +``` + +### 4. Bring the system up to date + +```sh +sudo apt update +sudo apt upgrade -y +``` + +On a fresh image this can take a while. Worth doing before building, so the compiler and libraries you build against are the ones you keep. + +### 5. Install the prerequisites + +```sh +sudo apt install -y python3-pip cmake build-essential git +pip install uv --break-system-packages +``` + +`--break-system-packages` looks alarming and is routine: Debian 12 and later mark the system Python +as externally managed, and this flag is how a user-level tool installs anyway. It affects pip's own +environment, not the system. + +If `uv` is not found afterwards, it landed in `~/.local/bin`, which is not always on the path: + +```sh +echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc +``` + +### 6. Build and run + +```sh +git clone https://github.com/MoonModules/projectMM.git +cd projectMM +uv run moondeck/build/build_desktop.py +uv run moondeck/run/run_desktop.py +``` + +The build takes a few minutes on a Pi 4 or a NanoPi, longer on older boards. Everything about the +build itself, including how to run the tests, is in [building.md](../building.md). + +`run_desktop.py` detaches, so the program outlives the ssh session. Open `http://:8080` and +the UI is there. + +**That is a running system.** The rest of this page makes it survive a reboot and covers the two +boards; none of it is needed to start building shows. + +> **A board with 1 GB of RAM or less can run out of memory while compiling.** The symptom is the +> compiler being killed rather than an error you can read. Either add swap +> (edit `/etc/dphys-swapfile` to raise `CONF_SWAPSIZE`, then `sudo dphys-swapfile swapoff && +> sudo dphys-swapfile setup && sudo dphys-swapfile swapon`: `setup` is what regenerates the +> file at the new size, editing alone changes nothing), or build with fewer parallel jobs. + +## Keeping it running after a reboot + +So far projectMM stops when the board restarts. A permanent installation wants it back by itself. Give it a systemd unit at `/etc/systemd/system/projectmm.service`: + +```ini +[Unit] +Description=projectMM +After=network-online.target +Wants=network-online.target + +[Service] +ExecStart=/home/pi/projectMM/build/projectMM +Restart=always +RestartSec=5 +User=pi + +[Install] +WantedBy=multi-user.target +``` + +Then: + +```sh +sudo systemctl enable --now projectmm +systemctl status projectmm +``` + +`Restart=always` covers a crash as well as a reboot. Adjust `User` and the path to match where you built it. + +## Shutting down + +**Do not pull the power.** An SD card interrupted mid-write can corrupt the filesystem, and the board then does not come back: + +```sh +sudo shutdown now # or: sudo reboot +``` + +projectMM itself writes to disk rarely (settings on change, not per frame), so an SD card is a fine home for it. The risk is the operating system's writes, not ours. + +--- + +## Board notes + +### Raspberry Pi + +A Pi 4 or 5 has ample headroom for the render pipeline. Prefer the **Lite** image: a controller has +no use for a desktop, and it leaves more memory for the build. + +### NanoPi R28S + +A small metal-cased board with **two Gigabit ethernet ports**, so it can sit between a house network +and a lighting network. The case is the heatsink, so it runs without a fan. It boots from SD; no +eMMC needed. + +- **Images**: FriendlyELEC offers FriendlyWrt, Debian, Ubuntu and Buildroot. **Take Debian**: FriendlyWrt is router firmware, with a different package manager and none of this recipe. Armbian has RK3528 work in progress but publishes no R28S image at the time of writing. +- **Finding the image** is the fiddly part. The [wiki's flashing section](https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R28S#Flashing_the_OS_to_the_microSD_card) links a Google Drive; the file you want is under **`01_Official images/01_SD card images`**, a `.gz` you can hand to Imager or Etcher without extracting. The other directories are for installing to eMMC, which you do not need. Use an **8 GB card or larger**. +- The wiki's own instructions assume Windows and `win32diskimager`. Imager or Etcher do the same job on macOS and Linux. +- **It has 1 GB of RAM**, which is enough to run projectMM comfortably and tight for compiling it. If the build is killed, add swap as described in step 6. +- **Configure the second port** with `sudo nmtui`; `ip ad` shows what each picked up. + +--- + +## Containers + +A container image is [in development](https://github.com/MoonModules/projectMM/pull/98), not yet +released. When it lands it will run a full instance on anything that runs Docker. + +One thing to know before reaching for it on a board: **a container does not emulate a CPU.** It shares the host kernel and runs native instructions, so an amd64 image needs an amd64 host. Docker Desktop on Apple Silicon is the exception, bundling emulation so an amd64 image runs (slowly) on an arm64 Mac. An arm64 SBC has no such emulator, so on a Pi or a NanoPi, build from source as above. + +--- + +## Where to go next + +- [Install & first light](../gettingstarted.md): the same program on an ESP32. +- [How projectMM works](how-projectmm-works.md): layouts, layers, effects and drivers. +- [building.md](../building.md): building, testing and packaging in depth. diff --git a/esp32/sdkconfig.defaults.esp32p4rev1-eth b/esp32/sdkconfig.defaults.esp32p4rev1-eth index 33b843b1..43a80033 100644 --- a/esp32/sdkconfig.defaults.esp32p4rev1-eth +++ b/esp32/sdkconfig.defaults.esp32p4rev1-eth @@ -61,15 +61,34 @@ CONFIG_ETH_DMA_TX_BUFFER_NUM=10 CONFIG_ESP_SYSTEM_MEMPROT_FEATURE=n CONFIG_HEAP_HAS_EXEC_HEAP=y -# esp-dsp's PORTABLE C kernels instead of its hand-written P4 assembly. The assembly FFT uses the -# P4's hardware-loop instruction (esp.lp.setup), and that unit carries a documented erratum - -# Espressif's own soc_caps: SOC_CPU_HAS_HWLOOP_STATE_BUG, "HWLOOP state doesn't go to DIRTY after -# executing the last instruction of a loop". FreeRTOS saves those registers lazily, keyed on that -# DIRTY flag, so a task switched out of the FFT has its loop state silently dropped and faults on -# resume (bench: a crash loop the moment HLS added a second task, Illegal instruction / Load -# access fault inside dsps_fft2r_fc32_arp4.S). IDF patches the erratum on the coprocessor RESTORE -# paths but not on SAVE. Measured cost of the portable kernels: audio tick ~615 us -> ~658 us. -# A WORKAROUND, not a fix: esp-idf#19025 (ours, naming the unguarded save path) and esp-dsp#119 -# (the same fault from another reporter, on IDF v5.5-beta1). Remove once upstream answers. -# Lives in the board fragment because every P4 image (rev1/rev3, eth/eth-wifi) layers on it. -CONFIG_DSP_ANSI=y +# esp-dsp's OPTIMIZED P4 assembly kernels: the default, and what we ship. The line below forces +# the portable C kernels instead and is currently OFF. +# +# Why it exists at all. The assembly FFT uses the P4's hardware-loop instruction (esp.lp.setup), +# and that unit carries a documented erratum - Espressif's own soc_caps: +# SOC_CPU_HAS_HWLOOP_STATE_BUG, "HWLOOP state doesn't go to DIRTY after executing the last +# instruction of a loop". FreeRTOS saves those registers lazily, keyed on that DIRTY flag, so a +# task switched out of the FFT has its loop state silently dropped and faults on resume (bench: +# a crash loop the moment HLS added a second task, Illegal instruction / Load access fault inside +# dsps_fft2r_fc32_arp4.S). IDF patches the erratum on the coprocessor RESTORE paths but not on +# SAVE. Cost of the portable kernels: audio tick ~615 us -> ~658 us. +# Tracked in esp-idf#19025 (ours, naming the unguarded save path) and esp-dsp#119 (the same fault +# from another reporter, on IDF v5.5-beta1). +# +# Off since 2026-09-08, and the ROOT CAUSE is now known (esp-idf#19025, Espressif's diagnosis of +# 2026-09-09, reproduced deterministically by them). It is not the save path named above; that was +# our guess and it was wrong. Two separate things: +# 1. esp-dsp: some assembly kernels violate the P4's hardware-loop constraints (the loop start +# and last instruction must be 4-byte aligned, and the last instruction must not be a +# coprocessor op). An interrupt landing exactly on a misaligned loop boundary corrupts the +# loop counter: WDT, illegal instruction, or a load fault. Timing-dependent, which is why it +# hid for a 20-minute soak, and why the ANSI kernels "fixed" it: they never enter that assembly. +# 2. ESP-IDF: the context-save erratum workaround was gated on a misnamed config macro, so it was +# always on regardless of silicon revision. Being corrected to apply only on rev < v3.0. +# Both fixes are upstream (an esp-dsp release, an IDF fix with regression tests). Until they land the +# optimized kernels are the desired config and we run them; re-enable this line if the fault +# returns. When the esp-dsp release ships, re-run the FFT + HLS soak against it (backlog-core). +# Ruled OUT as the cause of the P4 shield's 22 s reboot loop the same day: putting it back changed +# nothing (3 reboots in 70 s against 4), which is what a blocked network cascade looks like, not an +# FFT fault. Lives in the board fragment because every P4 image (rev1/rev3, eth/eth-wifi) layers on it. +# CONFIG_DSP_ANSI=y diff --git a/mkdocs.yml b/mkdocs.yml index eee832a9..6a2f4b70 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -134,6 +134,7 @@ nav: - Tutorials: - How projectMM works: tutorials/how-projectmm-works.md - Installing projectMM on a desktop: tutorials/installing-to-desktop.md + - Running projectMM on a Linux machine: tutorials/installing-on-linux.md - Driving LED panels with a receiving card: tutorials/panel-cards.md - Driving projectMM from a phone or tablet: tutorials/control-surface.md - Making beautiful effects: tutorials/generative-effects.md diff --git a/moonbase/main/moonbase_main.cpp b/moonbase/main/moonbase_main.cpp index f16ec381..7bd2c0e2 100644 --- a/moonbase/main/moonbase_main.cpp +++ b/moonbase/main/moonbase_main.cpp @@ -59,6 +59,10 @@ constexpr FsCandidate kFsCandidates[] = { }; constexpr const char* kFsMountPoint = "/fs"; constexpr const char* kNetworkConfig = "/fs/.config/NetworkModule.json"; +// The application persists its build variant here, which is the one fact this image cannot know +// about the board it is on: MoonBase is chip-specific and variant-agnostic, so without it the +// release picker can only offer every firmware for the chip and ask a user in recovery to choose. +constexpr const char* kSystemConfig = "/fs/.config/SystemModule.json"; // The AP fallback address matches the application's (NetworkModule uses 4.3.2.1), so a user who // has provisioned this device before sees the same address in both firmwares. @@ -154,6 +158,23 @@ struct { // (the catalog pins e.g. 8 dBm for them), and a brownout during recovery is the worst time. int txPowerDbm_ = 0; +// The application's build variant ("esp32s3-zero"), or empty when the app has never run here. +// Empty is the honest answer for a freshly flashed or wiped device, and the page falls back to +// offering every firmware for the chip rather than pretending to know which one this board takes. +char g_appVariant[24] = {}; + +void loadAppVariant() { + FILE* f = std::fopen(kSystemConfig, "r"); + if (!f) return; + // Same bounded prefix read as the credentials above, and for the same reason: the file carries + // every child module's config behind the identity keys this image needs. + char buf[1024]; + const size_t got = std::fread(buf, 1, sizeof(buf) - 1, f); + buf[got] = '\0'; + std::fclose(f); + jsonFindString(buf, "firmware", g_appVariant, sizeof(g_appVariant)); +} + // Read the stored WiFi credentials and Ethernet wiring, if there are any. Absent, unreadable // or empty all mean the same thing to the caller: fall through the cascade. void loadCredentials() { @@ -189,6 +210,10 @@ void loadCredentials() { jsonFindBool(buf, "ethClockExtIn", ðCfg_.clockExtIn); jsonFindInt(buf, "txPowerSetting", &txPowerDbm_); } + // BEFORE THE UNMOUNT. This function owns the only window in which the volume is mounted: it + // registers the partition above and unregisters it here, so anything that reads a file has to + // do it now. Reading the variant after this call returned "no file" for exactly that reason. + loadAppVariant(); esp_vfs_littlefs_unregister(label); } @@ -348,6 +373,14 @@ bool wifiAccessPoint() { // --------------------------------------------------------------------------------------------- // The one page MoonBase serves. Inline and tiny: no filesystem read, no compression, no assets. +// The chip this image was built for, as the release assets spell it: `firmware-esp32s3-zero-...` +// begins with the IDF target. MoonBase is chip-specific and variant-agnostic, so this is the most +// it can know about the board, and it is exactly enough to filter a release's asset list down to +// the firmwares that could run here. +#ifndef MOONBASE_CHIP +#define MOONBASE_CHIP CONFIG_IDF_TARGET +#endif + const char kPage[] = "" "MoonBase" @@ -380,6 +413,14 @@ const char kPage[] = "
Firmware files: github.com/MoonModules/projectMM/releases " "(the firmware-...bin matching this board)" + // FROM A RELEASE, without typing a URL. The browser fetches the release list from GitHub + // itself (api.github.com sends access-control-allow-origin: *), so THIS image gains no network + // code at all: it still only receives a URL, which is what it already accepts. That matters + // because a device in recovery is the one that most needs an easy install and the one least + // able to offer the application's own picker. + "
From a release
" + " " + "
" "
From a URL
" "
" "
Back to the app
Boot the installed firmware without changing it." @@ -428,17 +469,73 @@ const char kPage[] = "if(p.status==404){clearInterval(t);S('done, the app is starting...');" "setTimeout(()=>location.reload(),3000);}else{S(await p.text());}}" "catch(_){S('restarting...');}},2000);}" + // The release list, filtered to assets this CHIP can run. MoonBase is chip-specific and + // variant-agnostic (one image serves every variant of a chip), so it cannot know which variant + // the board runs: it offers the ones that fit and lets the user pick, which is the same choice + // the application's picker presents. + "const CHIP='" MOONBASE_CHIP "';let RELS=[],VAR='';" + // The board's own variant, when the application has run here and persisted it. With it the + // list is the ONE firmware this board takes, as the application's picker shows; without it, + // every firmware for the chip, because guessing which of three flash layouts a board has is + // how a user in recovery installs the wrong one. + "fetch('/api/variant').then(r=>r.text()).then(t=>{VAR=t.trim();fillFw();}).catch(()=>{});" + "function fwList(i){const r=RELS[i];if(!r)return [];" + "return (r.assets||[]).map(a=>a.name).filter(n=>/^firmware-.+\\.bin$/.test(n)" + "&&!/-(bootloader|partition-table|ota-data|slot0)\\.bin$/.test(n)" + // The chip must match to a BOUNDARY: "esp32s31" starts with "esp32s3" and is different + // silicon, so a plain prefix test offered an S31 image on an S3 board. Most assets spell the + // chip then a hyphen, whether a variant follows ("esp32s3-zero-v...") or the version does + // ("esp32-v..."), so requiring that hyphen is the rule. + // + // The P4 is the exception: CONFIG_IDF_TARGET is "esp32p4" while every asset carries the + // SILICON REVISION in the same token ("esp32p4rev1-eth-v..."), so the character after the chip + // is a digit-bearing "rev", not a hyphen. Requiring the hyphen alone left a P4 in MoonBase with + // an EMPTY firmware list, which is the one place a user has no other way to install. Accept + // "rev" as an alternative boundary: it keeps the s3/s31 separation (nothing spells + // "esp32s3rev") while matching every P4 asset we publish. + "&&n.slice(9).startsWith(CHIP)" + "&&(n.slice(9+CHIP.length).startsWith('-')||/^rev\\d/.test(n.slice(9+CHIP.length)))" + "&&(!VAR||n.slice(9).startsWith(VAR+'-')));}" + "function fillFw(){const f=document.getElementById('fw');f.innerHTML='';" + "const l=fwList(document.getElementById('rel').selectedIndex);" + "for(const n of l){const o=document.createElement('option');o.textContent=n;f.appendChild(o);}" + "document.getElementById('rs').textContent=l.length?'':'no firmware for this chip in that release';}" + "fetch('https://api.github.com/repos/MoonModules/projectMM/releases?per_page=10')" + ".then(r=>r.json()).then(j=>{RELS=j;const s=document.getElementById('rel');" + "for(const r of RELS){const o=document.createElement('option');" + "o.textContent=(r.name||r.tag_name)+(r.prerelease?' (pre)':'');s.appendChild(o);}" + "s.onchange=fillFw;fillFw();})" + ".catch(()=>{document.getElementById('rs').textContent=" + "'could not reach github: use a URL or a file below';});" + // Installing a release is installing its URL: one path, so the vetting, the progress and the + // retry all behave identically however the URL was chosen. + "async function rl(){const r=RELS[document.getElementById('rel').selectedIndex];" + "const n=document.getElementById('fw').value;if(!r||!n)return;" + "const a=(r.assets||[]).find(x=>x.name===n);if(!a)return;" + "document.getElementById('u').value=a.browser_download_url;url();}" "async function url(){const u=document.getElementById('u').value;if(!u)return;" "const r=await fetch('/api/firmware/url',{method:'POST',body:u});S(await r.text());if(r.ok)W();}" // Prefill the URL field with the last install source (RAM-held), so Install doubles as // retry: the escape after a cancel or failure wiped the app slot. "fetch('/api/firmware/last-url').then(r=>r.text()).then(u=>{if(u)document.getElementById('u').value=u;})" ".catch(()=>{});" + // RELOAD WHEN THE APP ANSWERS, not after a fixed wait. Eight seconds was a guess that an + // S3-Zero misses, so the page reloaded while the device was still booting and showed a failed + // page the user then had to refresh by hand. /moonbase is the identity probe: MoonBase answers + // it and the app 404s it, so a 404 means the application is up and serving. "async function ba(){const r=await fetch('/api/firmware/boot-app',{method:'POST'});S(await r.text());" - "if(r.ok)setTimeout(()=>location.reload(),8000);}" + "if(!r.ok)return;S('booting the app...');" + "for(let i=0;i<60;i++){await new Promise(f=>setTimeout(f,1000));" + "try{const p=await fetch('/moonbase',{cache:'no-store'});" + "if(p.status==404){location.reload();return;}}catch(e){}}" + "S('the app is not answering: it may not be installed');}" "async function cx(){S(await (await fetch('/api/firmware/cancel',{method:'POST'})).text());}" ""; +// The application's build variant ("esp32s3-zero"), or empty when the app has never run here. +// Empty is the honest answer for a freshly flashed or wiped device, and the page falls back to +// offering every firmware for the chip rather than pretending to know. + // The application slot. From the factory partition esp_ota_get_next_update_partition returns the // first OTA slot, which is the one we want and is never the one we are running from. // @@ -809,6 +906,10 @@ void serveOne(int sock) { } else { sendResponse(sock, "200 OK", "text/plain", "nothing to cancel"); } + } else if (std::strncmp(head, "GET /api/variant", 16) == 0) { + // The application's build variant, read from its config at boot. Empty when the app has + // never run here, which the page treats as "offer every firmware for the chip". + sendResponse(sock, "200 OK", "text/plain", g_appVariant); } else if (std::strncmp(head, "GET /api/version", 16) == 0) { // This image's version, from the app descriptor IDF puts in every binary (PROJECT_VER, // set by build_moonbase to the same string the application reports). Its own route @@ -879,7 +980,7 @@ extern "C" void app_main() { esp_event_loop_create_default(); esp_event_handler_instance_register(IP_EVENT, ESP_EVENT_ANY_ID, &onGotIp, nullptr, nullptr); - loadCredentials(); + loadCredentials(); // also reads the app's build variant, inside its mount window // The cascade: Ethernet where the config wires it (its DHCP window overlaps the WiFi // join since the GOT_IP bit is shared), then WiFi STA with the stored credentials, then diff --git a/moondeck/MoonDeck.md b/moondeck/MoonDeck.md index 2cbd3c62..573bcf3c 100644 --- a/moondeck/MoonDeck.md +++ b/moondeck/MoonDeck.md @@ -122,7 +122,7 @@ uv run moondeck/run/preview_installer.py Long-running — MoonDeck shows **Stop** while the server is up. Two modes, picked automatically: - **Render-only.** When no `build/esp32-*/projectMM.bin` is present, the picker populates against the real GitHub Releases API and dropdowns work, but clicking **Install** fails because the local server has no `releases/` tree. Useful for iterating on HTML / CSS / JS without burning a build. Equivalent to "Recipe A" in [mooninstaller/README.md](../mooninstaller/README.md). -- **Flash-ready.** When at least one ESP32 build exists, the script additionally stages every `build/esp32-*/projectMM.bin` it finds into `releases/local-dev/` and generates matching Pages-relative manifests via the same `generate_manifest.py` the release workflow uses. The picker shows `local-dev` as the newest tag; clicking **Install** flashes a USB-connected ESP32 and hands off to the repository's custom orchestrator UI (Improv-Serial provisioning + SET_DEVICE_MODEL + control fan-out, all in `install-orchestrator.js` — not ESP Web Tools). End-to-end, same code paths as the public installer. This is the developer's test ground for the install flow before deploying to GitHub Pages: Web Serial works on `http://localhost` without the secure-origin requirement that gates the public site. +- **Flash-ready.** When at least one ESP32 build exists, the script additionally stages every `build/esp32-*/projectMM.bin` it finds into `releases/local-dev/` and generates matching Pages-relative manifests via the same `generate_manifest.py` the release workflow uses. The picker shows `local-dev` as the newest tag; clicking **Install** flashes a USB-connected ESP32 and hands off to the repository's custom orchestrator UI (Improv-Serial provisioning + the APPLY_OP config push of the device model's modules and controls, all in `install-orchestrator.js`, not ESP Web Tools). End-to-end, same code paths as the public installer. This is the developer's test ground for the install flow before deploying to GitHub Pages: Web Serial works on `http://localhost` without the secure-origin requirement that gates the public site. Add `?nocache=1` to the URL to bypass the picker's 5-minute sessionStorage cache while editing. @@ -1076,7 +1076,7 @@ Push WiFi credentials to a running projectMM device over USB-serial. Uses the [I **One-click flow**: pick the device's port in MoonDeck, hit **Improv WiFi**. The script reads SSID + password from the **active network's WiFi block in `moondeck/moondeck.json`** (the one shown in the network bar at the top of the sidebar). If that block is empty, it falls back to detecting the host machine's currently-joined WiFi (macOS Keychain / Linux NetworkManager / Windows `netsh`). The device replies with its new URL when STA comes up — typically 5-10 s end to end. -**Device-model dropdown (pre-association injection)**: pick your device model next to the Firmware dropdown and the flow forwards `--device-model` — the script then resolves the deviceModel's `deviceModels.json` settings and pushes the TX-power cap over the `SET_TX_POWER` vendor RPC **before** the credentials, plus `SET_DEVICE_MODEL` after success. This matters for brown-out-prone weak-powered device models (cap 8 dBm): at full TX power they fail their very first WiFi association, so the cap can't wait for the post-online HTTP injection. Leave the dropdown on "(any model)" for device models without special settings. +**Device-model dropdown (pre-association injection)**: pick your device model next to the Firmware dropdown and the flow forwards `--device-model`: the script then resolves the deviceModel's `deviceModels.json` settings and pushes the TX-power cap over the `SET_TX_POWER` vendor RPC **before** the credentials, then applies the entry's modules and controls over serial as `APPLY_OP` ops (the same push the web installer does; the model name is one of those controls, `System.deviceModel`). One-way on boards whose LED pins include GPIO 1/3: once the driver claims the UART pins the board can no longer receive over serial, so a provisioned QuinLED board is reconfigured from its web UI, not by re-running this. This matters for brown-out-prone weak-powered device models (cap 8 dBm): at full TX power they fail their very first WiFi association, so the cap can't wait for the post-online HTTP injection. Leave the dropdown on "(any model)" for device models without special settings. ```bash # Equivalent CLI for a weak-powered board (cap resolved from deviceModels.json): diff --git a/moondeck/build/improv_provision.py b/moondeck/build/improv_provision.py index 92f3defa..0725bb15 100644 --- a/moondeck/build/improv_provision.py +++ b/moondeck/build/improv_provision.py @@ -54,6 +54,7 @@ TYPE_RPC = 0x03 TYPE_RPC_RESPONSE = 0x04 CMD_WIFI_SETTINGS = 0x01 +CMD_GET_CURRENT_STATE = 0x02 def checksum(buf: bytes) -> int: @@ -203,6 +204,146 @@ def self_test() -> int: return 0 +# --- Device-model config push: "Improv = REST over serial" ----------------------------- +# +# The device applies a catalog entry through ONE vendor RPC, APPLY_OP (0xFC), carrying one +# REST operation as JSON per frame sequence: the same {"op":"add"|"set"|"clearChildren"} shapes +# the HTTP API takes. The browser installer plans the op sequence in mooninstaller/config-ops.js +# and sends it open-loop (Web Serial cannot read the ack while it holds the writer). This is a +# faithful port of that planner, and the sender is CLOSED-loop: the device acks every frame +# with an RPC_RESPONSE, or answers ERROR 0x82 while its single op buffer is still busy, which +# means "send that frame again". So a slow tick never drops an op here. +# +# Until 2026-09-08 this script sent a vendor RPC that no firmware has ever handled, then +# printed "pushed": three boards were provisioned with WiFi and nothing else while the script +# claimed success. The op sequence below is what the browser was doing all along. + +IMPROV_CMD_APPLY_OP = 0xFC +IMPROV_ERROR_INVALID_OP = 0x82 # device: op buffer busy, retry this frame +APPLY_OP_CHUNK_MAX = 128 - 3 # kImprovMaxPayload minus the [cmd][seq][last] header + + +def is_eth_only(entry) -> bool: + """An entry whose firmware is an Ethernet-only build (`-eth`, not `-eth-wifi`) has WiFi + compiled out and no WIFI_SETTINGS RPC to answer: provisioning is plugging the cable in. + Mirrors install.js (`ethOnly = /-eth$/.test(firmware)`) so the two front ends agree.""" + fws = entry.get("firmwares") if isinstance(entry, dict) else None + return bool(fws) and all(isinstance(f, str) and f.endswith("-eth") for f in fws) + + +def _is_addable(m) -> bool: + """A module the entry ADDS: a non-empty id, a parent to add it under, and a type.""" + return (isinstance(m, dict) + and isinstance(m.get("id"), str) and bool(m.get("id")) + and isinstance(m.get("parent_id"), str) and bool(m.get("parent_id")) + and bool(m.get("type"))) + + +def plan_config_ops(entry) -> list: + """The ordered APPLY_OP sequence for a catalog entry: mirrors config-ops.js exactly. + + clearChildren pre-pass (every parent the entry adds into, plus any container flagged + replaceChildren, unless that parent is itself added fresh), then per module an add, then its + control sets. The clear pass is what makes a re-provision converge on a NON-erased device: + add is idempotent on id, so without it a stale module lingers and a structural change never + lands. + + Catalog order is preserved, and it matters on boards whose LED pins include GPIO 1/3 + (UART0): the op that sets those pins ends serial reception, so every op after it is lost. + Such an entry lists its LED driver LAST. (Bench 2026-09-08, QuinLED Dig-Uno / Dig-Quad.) + """ + ops = [] + modules = entry.get("modules") if isinstance(entry, dict) else None + modules = modules if isinstance(modules, list) else [] + added_ids = {m["id"] for m in modules if _is_addable(m)} + clear_parents = [] # insertion-ordered, deduped + for m in modules: + if not isinstance(m, dict): + continue + if m.get("replaceChildren") and isinstance(m.get("id"), str) and m["id"]: + if m["id"] not in clear_parents: + clear_parents.append(m["id"]) + if _is_addable(m) and m["parent_id"] not in clear_parents: + clear_parents.append(m["parent_id"]) + for parent in clear_parents: + if parent in added_ids: + continue + ops.append({"op": "clearChildren", "parent": parent}) + for m in modules: + if not isinstance(m, dict) or not isinstance(m.get("id"), str) or m["id"] == "": + continue + if _is_addable(m): + ops.append({"op": "add", "type": m["type"], "id": m["id"], "parent": m["parent_id"]}) + controls = m.get("controls") + if isinstance(controls, dict): + for control, value in controls.items(): + ops.append({"op": "set", "module": m["id"], "control": control, "value": value}) + return ops + + +def encode_apply_op_frames(op: dict) -> list: + """One op -> the Improv RPC frames carrying it: [0xFC][seq][last][chunk], chunk <= 125 B.""" + import json + body = json.dumps(op, separators=(",", ":")).encode("utf-8") + chunks = [body[i:i + APPLY_OP_CHUNK_MAX] for i in range(0, len(body), APPLY_OP_CHUNK_MAX)] or [b""] + frames = [] + for seq, chunk in enumerate(chunks): + last = 1 if seq == len(chunks) - 1 else 0 + frames.append(build_frame(TYPE_RPC, bytes([IMPROV_CMD_APPLY_OP, seq, last]) + chunk)) + return frames + + +def send_apply_op(ser, op: dict, ack_timeout: float = 2.0, retries: int = 20) -> bool: + """Send one op closed-loop: every frame must be acked; a busy device (ERROR 0x82) gets the + same frame again after a short pause. Returns False when a frame is refused for any other + reason or never acked.""" + for frame in encode_apply_op_frames(op): + for attempt in range(retries): + ser.write(frame) + ser.flush() + # Wait for THIS frame's verdict. The device also volunteers CURRENT_STATE frames + # (it is still announcing PROVISIONED right after the credentials), and those are + # not a reply to us: skip them rather than reading one as a refusal, which is how + # the very first op of every push failed on the bench (2026-09-08). + deadline = time.monotonic() + ack_timeout + verdict = None + while verdict is None and time.monotonic() < deadline: + reply = parse_frame(ser, deadline) + if reply is None: + break + msg_type, body = reply + if msg_type == TYPE_RPC_RESPONSE: + verdict = "ack" + elif msg_type == TYPE_ERROR_STATE: + verdict = "busy" if (body and body[0] == IMPROV_ERROR_INVALID_OP) else "refused" + # anything else (CURRENT_STATE, ...) is chatter: keep waiting + if verdict == "ack": + break # next frame + if verdict == "busy": + time.sleep(0.15) # the previous op is still being applied + continue + return False # refused, or no ack within the timeout + else: + return False # busy for the whole retry budget + return True + + +def apply_device_model(ser, entry: dict, name: str) -> int: + """Push a catalog entry as APPLY_OP ops; returns the number of ops that FAILED.""" + ops = plan_config_ops(entry) + ser.reset_input_buffer() # stale state frames from provisioning are not acks + failed = 0 + for op in ops: + label = {"clearChildren": lambda o: f"clearChildren {o['parent']}", + "add": lambda o: f"add {o['type']} as {o['id']} under {o['parent']}", + "set": lambda o: f"set {o['module']}.{o['control']} = {o['value']!r}"}[op["op"]](op) + ok = send_apply_op(ser, op) + print(f" {'ok ' if ok else 'FAIL'} {label}") + failed += 0 if ok else 1 + print(f"==> applied deviceModel {name!r}: {len(ops) - failed}/{len(ops)} ops ok") + return failed + + def main() -> int: ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) ap.add_argument("--self-test", action="store_true", @@ -222,9 +363,9 @@ def main() -> int: ap.add_argument("--device-model", dest="device_model", default=None, metavar="NAME", help="deviceModel name from mooninstaller/deviceModels.json (e.g. " "'ESP32-S3 N16R8 Dev'). Resolves the deviceModel's TX-power cap " - "(controls.Network.txPowerSetting) automatically and " - "pushes the name via SET_DEVICE_MODEL after " - "provisioning — the same injection the web installer " + "(controls.Network.txPowerSetting) automatically and, after " + "provisioning, applies the entry's modules and controls over " + "serial as APPLY_OP ops: the same config push the web installer " "does. An explicit --tx-power overrides the lookup.") ap.add_argument("--tx-power", type=int, default=None, metavar="DBM", help="Send the SET_TX_POWER vendor RPC (0..21 whole dBm) " @@ -304,6 +445,8 @@ def main() -> int: args.tx_power = cap print(f"==> deviceModel {args.device_model!r}: TX-power cap {cap} dBm from deviceModels.json") + eth_only = bool(args.device_model) and is_eth_only(entry) + try: ser = serial.Serial(args.port, baudrate=115200, timeout=0.1) except serial.SerialException as e: @@ -315,7 +458,7 @@ def main() -> int: print(f"ERROR: --tx-power {args.tx_power} out of range 0..21", file=sys.stderr) return 2 # SET_TX_POWER vendor RPC (0xFD): [cmd][data_len=1][dBm]. Mirrors - # SET_DEVICE_MODEL's framing; the firmware persists + applies it before the + # The firmware persists + applies it before the # association the credentials below will trigger. The 2.5 s pause lets # the module's 1 Hz consumer pick the cap up first. print(f"==> sending SET_TX_POWER {args.tx_power} dBm to {args.port}") @@ -328,6 +471,51 @@ def main() -> int: file=sys.stderr) time.sleep(2.5) + # Ask before telling: a device that is ALREADY provisioned answers GET_CURRENT_STATE with + # PROVISIONED and never re-announces it for a second WIFI_SETTINGS, so a re-provision used + # to sit out the whole timeout and skip the config push that needs no provisioning at all. + # (The push works on any reachable device with the port open, the same as the browser's.) + ser.reset_input_buffer() + ser.write(build_frame(TYPE_RPC, bytes([CMD_GET_CURRENT_STATE, 0]))) + ser.flush() + already_provisioned = False + probe_answered = False + probe_deadline = time.monotonic() + 3.0 + while time.monotonic() < probe_deadline: + reply = parse_frame(ser, probe_deadline) + if reply is None: + break + probe_answered = True + if reply[0] == TYPE_CURRENT_STATE and reply[1] and reply[1][0] == 4: + already_provisioned = True + break + if not probe_answered: + # No Improv listener heard us. The common cause on QuinLED boards is not a fault: their + # catalog pins include GPIO 1/3 (UART0), so once the LED driver claims them a provisioned + # board can no longer RECEIVE over this port, and neither a re-provision nor a config push + # can reach it (use the device's web UI or the HTTP push instead). The credentials are + # still sent below in case this is a fresh board on firmware that ignores the probe. + print("==> warning: no answer to GET_CURRENT_STATE on this port (already provisioned with " + "the LED driver on the UART pins?); continuing, expect a timeout if so", + file=sys.stderr) + if eth_only and not already_provisioned: + print("==> Ethernet-only firmware: no WiFi to provision, connect the cable; pushing the config") + already_provisioned = True # same path: the push needs only the open port + if already_provisioned: + print("==> device reports PROVISIONED already: keeping its WiFi credentials" if not eth_only + else "==> applying the catalog entry over serial") + if args.device_model: + print(f"==> applying deviceModel {args.device_model!r} over serial (APPLY_OP)") + failed = apply_device_model(ser, entry, args.device_model) + ser.close() + if failed: + print(f"ERROR: {failed} op(s) not applied; the device config is incomplete", + file=sys.stderr) + return 1 + else: + ser.close() + return 0 + print(f"==> sending WIFI_SETTINGS to {args.port} (SSID: {args.ssid!r})") payload = build_wifi_settings_payload(args.ssid, args.password) frame = build_frame(TYPE_RPC, payload) @@ -369,15 +557,15 @@ def main() -> int: url = urls[0] if urls else "(no URL reported)" print(f"==> provisioned: {url}") if args.device_model: - # SET_DEVICE_MODEL vendor RPC (0xFE): [cmd][1+len][len][name] — - # the same post-provision push the web installer does, so - # the device persists its physical-board identity. - name = args.device_model.encode("utf-8") - ser.write(build_frame(TYPE_RPC, - bytes([0xFE, 1 + len(name), len(name)]) + name)) - ser.flush() - time.sleep(0.5) # let the device's serial task consume it - print(f"==> pushed SET_DEVICE_MODEL {args.device_model!r}") + # The catalog entry's modules + controls, over serial as APPLY_OP ops (the + # deviceModel name is just one of those controls: System.deviceModel). + print(f"==> applying deviceModel {args.device_model!r} over serial (APPLY_OP)") + failed = apply_device_model(ser, entry, args.device_model) + if failed: + print(f"ERROR: {failed} op(s) not applied; the device is provisioned but " + f"its config is incomplete", file=sys.stderr) + ser.close() + return 1 ser.close() return 0 diff --git a/moondeck/moondeck.py b/moondeck/moondeck.py index b261e459..d2d839bd 100644 --- a/moondeck/moondeck.py +++ b/moondeck/moondeck.py @@ -1128,9 +1128,17 @@ def _read_usb_ports() -> dict: return {} import re try: - out = subprocess.run( + # BYTES, then a lenient decode. `ioreg -l` dumps every property in the registry, including + # raw device data that is not text at all, so a strict UTF-8 decode raises + # UnicodeDecodeError on whatever happens to be attached: on the bench it fired on every + # /api/ports request while boards were connected, and since UnicodeDecodeError is neither + # OSError nor SubprocessError it escaped this handler and 500'd the request, leaving + # MoonDeck's port dropdown empty with boards plugged in. The parse below only ever reads + # ASCII keys, so replacing the undecodable bytes costs nothing and keeps the listing. + raw = subprocess.run( ["ioreg", "-l", "-w0"], - capture_output=True, text=True, timeout=5).stdout + capture_output=True, timeout=5).stdout + out = raw.decode("utf-8", "replace") except (OSError, subprocess.SubprocessError): return {} # The IORegistry is a tree: a USB device node holds the descriptor diff --git a/mooninstaller/deviceModels.json b/mooninstaller/deviceModels.json index bedef4ac..20743691 100644 --- a/mooninstaller/deviceModels.json +++ b/mooninstaller/deviceModels.json @@ -159,12 +159,20 @@ "deviceModel": "QuinLED Dig-2-Go" } }, + { + "type": "Drivers", + "id": "Drivers", + "controls": { + "relayPins": "12" + } + }, { "type": "RmtLedDriver", "id": "RmtLed", "parent_id": "Drivers", "controls": { - "pins": "16" + "pins": "16", + "lightPreset": 4 } } ] @@ -317,14 +325,6 @@ "deviceModel": "QuinLED Dig-Octa 32-8L" } }, - { - "type": "RmtLedDriver", - "id": "RmtLed", - "parent_id": "Drivers", - "controls": { - "pins": "0,1,2,3,4,5,12,13" - } - }, { "type": "NetworkModule", "id": "Network", @@ -337,8 +337,17 @@ "ethMdioGpio": 18, "ethRstGpio": -1 } + }, + { + "type": "RmtLedDriver", + "id": "RmtLed", + "parent_id": "Drivers", + "controls": { + "pins": "0,1,2,3,4,5,12,13" + } } - ] + ], + "flashBaud": 460800 }, { "name": "Serg UniShield V5", @@ -885,17 +894,6 @@ "height": 8 } }, - { - "type": "ParallelLedDriver", - "id": "ParallelLed", - "parent_id": "Drivers", - "controls": { - "peripheral": "LCD-IDF", - "pins": "47,21,14,9,8,16,15,7,1,2,42,41,40,39,38,48", - "clockPin": 19, - "dcPin": 20 - } - }, { "type": "NetworkModule", "id": "Network", @@ -916,6 +914,17 @@ "controls": { "pin": 4 } + }, + { + "type": "ParallelLedDriver", + "id": "ParallelLed", + "parent_id": "Drivers", + "controls": { + "peripheral": "LCD-IDF", + "pins": "47,21,14,9,8,16,15,7,1,2,42,41,40,39,38,48", + "clockPin": 19, + "dcPin": 20 + } } ] }, @@ -944,17 +953,6 @@ "deviceModel": "SE 16 V1" } }, - { - "type": "ParallelLedDriver", - "id": "ParallelLed", - "parent_id": "Drivers", - "controls": { - "peripheral": "LCD-IDF", - "pins": "47,48,21,38,14,39,13,40,12,41,11,42,10,2,3,1", - "clockPin": 16, - "dcPin": 17 - } - }, { "type": "NetworkModule", "id": "Network", @@ -974,6 +972,17 @@ "controls": { "pin": 5 } + }, + { + "type": "ParallelLedDriver", + "id": "ParallelLed", + "parent_id": "Drivers", + "controls": { + "peripheral": "LCD-IDF", + "pins": "47,48,21,38,14,39,13,40,12,41,11,42,10,2,3,1", + "clockPin": 16, + "dcPin": 17 + } } ] }, @@ -1040,7 +1049,8 @@ "name": "MHC-WLED ESP32-P4 shield", "chip": "ESP32-P4", "firmwares": [ - "esp32p4rev1-eth" + "esp32p4rev1-eth", + "esp32p4rev1-eth-wifi" ], "image": "assets/deviceModels/mhc-wled-esp32-p4-shield.jpg", "url": "https://shop.myhome-control.de/en/ABC-WLED-ESP32-P4-shield/HW10027", @@ -1081,7 +1091,7 @@ "parent_id": "Drivers", "controls": { "peripheral": "Parlio", - "pins": "21,20,25,5,22,23,24,27" + "pins": "21,20,25,5,7,23,8,27" } }, { @@ -1096,14 +1106,6 @@ "ethClockGpio": 50, "ethClockExtIn": true } - }, - { - "type": "I2cScanModule", - "id": "I2cScan", - "controls": { - "sda": 7, - "scl": 8 - } } ] }, diff --git a/mooninstaller/install-orchestrator.js b/mooninstaller/install-orchestrator.js index 8f1cb562..e28f146f 100644 --- a/mooninstaller/install-orchestrator.js +++ b/mooninstaller/install-orchestrator.js @@ -448,6 +448,27 @@ async function releaseDetected() { try { await port.close(); } catch (_) { /* already closed */ } } +/// Overall flash progress across ALL images, 0-100. +/// +/// esptool-js reports per FILE: `written`/`total` restart at zero for each image, and an install +/// writes several (bootloader, partition table, ota data, app, and MoonBase where the layout has +/// one). Plotting that raw ran the bar 0-100% once per image, so a user watched it reach 100% and +/// start again (bench 2026-09-08, a Shelly on the public installer). Weighting each file by its +/// size against the whole write makes the bar cross the modal exactly once. +/// +/// `sizes` is the byte length of each image, in the order esptool-js writes them. +export function flashPercent(sizes, idx, written, total) { + const all = Array.isArray(sizes) ? sizes.map(n => (typeof n === "number" && n > 0 ? n : 0)) : []; + const grand = all.reduce((a, b) => a + b, 0); + if (grand <= 0) return 0; + const done = all.slice(0, Math.max(0, idx)).reduce((a, b) => a + b, 0); + // `written`/`total` are this file's own bytes; fall back to zero for this file when total is + // missing, so a bad tick can never push the bar backwards past what is already written. + const here = total > 0 ? (written / total) * (all[idx] || 0) : 0; + const pct = Math.round(100 * (done + here) / grand); + return pct < 0 ? 0 : pct > 100 ? 100 : pct; +} + export const installer = { /** * Drive the full install flow: request port, flash via esptool-js, @@ -759,7 +780,8 @@ export const installer = { flashSize: "keep", compress: true, reportProgress: (idx, written, total) => { - const pct = total > 0 ? Math.round(100 * written / total) : 0; + const pct = flashPercent(fileArray.map(f => (f.data && f.data.length) || 0), + idx, written, total); // Don't bump lastStage on every progress tick — keep it as // "flash" set just above; intermediate ticks are detail only. onProgress("flash", { pct, fileIdx: idx }); @@ -805,16 +827,31 @@ export const installer = { // // Some USB-serial chips (rare CH340 silicon revisions, mis-driven // adapters) don't survive the close+reopen cleanly — the OS handle - // ends up stale and port.open() throws. Catch that and prompt for - // a fresh requestPort(); the browser's permission grant from the - // earlier requestPort means it surfaces a picker but no auth - // dialog. User picks the same physical port; we get a fresh - // SerialPort handle. Slightly worse UX (extra click) than the - // transparent reopen, but never silently fails. + // ends up stale and port.open() throws. Catch that and get a fresh + // SerialPort handle. + // + // requestPort() needs a USER GESTURE, and by this point there is none: the + // flash took tens of seconds, so the click that opened the modal is long + // expired and Chrome refuses with "Must be handling a user gesture to show a + // permission request" (bench 2026-09-08, a Shelly on the public installer). + // The browser's earlier permission grant does not help: it covers ACCESS to + // the port, not the right to show the picker. So ask the user to click first, + // exactly as the wrong-port path above does, and call requestPort() inside + // that click. Without the callback (an older host page) there is nothing to + // click, so report the real reason rather than throwing a browser message + // that reads like a bug in the installer. try { await port.open({ baudRate: 115200 }); } catch (openErr) { - if (onLog) onLog(`[orchestrator] port.open() failed (${openErr.message}); falling back to requestPort()`); + if (onLog) onLog(`[orchestrator] port.open() failed (${openErr.message}); asking for a fresh port`); + if (!uiWaitForPortRetry) { + throw new Error( + "the serial port did not survive the flash and the page cannot re-prompt for it; " + + "unplug and replug the device, then install again"); + } + trackProgress("wrong-port-retry"); + await uiWaitForPortRetry(); + trackProgress("request-port"); port = await navigator.serial.requestPort({}); await port.open({ baudRate: 115200 }); } diff --git a/src/core/HttpServerModule.cpp b/src/core/HttpServerModule.cpp index 919277d6..8c184a00 100644 --- a/src/core/HttpServerModule.cpp +++ b/src/core/HttpServerModule.cpp @@ -1549,13 +1549,21 @@ void HttpServerModule::serveSystem(platform::TcpConnection& conn) { // maxBlock = internal-only (maxInternalAllocBlock): the all-memory // variant reports ~8 MB on PSRAM boards and is meaningless as a // pressure signal. Same rationale as main.cpp's tick log line. + // `allocated` / `allocPeak` are what the system TOOK, which is the figure that means the same + // thing on a board and on a laptop: free heap does not, since a desktop has as much as it wants + // and reports 0. It is what makes a memory change measurable without hardware. sink.appendf( - "{\"fps\":%u,\"tickTimeUs\":%u,\"freeHeap\":%u,\"freeInternal\":%u,\"maxBlock\":%u,\"uptime\":%u,\"modules\":[", + "{\"fps\":%u,\"tickTimeUs\":%u,\"freeHeap\":%u,\"freeInternal\":%u,\"maxBlock\":%u,\"maxExec\":%u," + "\"allocated\":%u,\"allocPeak\":%u,\"allocBlocks\":%u,\"uptime\":%u,\"modules\":[", static_cast(scheduler_ ? scheduler_->fps() : 0), static_cast(scheduler_ ? scheduler_->tickTimeUs() : 0), static_cast(platform::freeHeap()), static_cast(platform::freeInternalHeap()), static_cast(platform::maxInternalAllocBlock()), + static_cast(platform::maxExecAllocBlock()), + static_cast(platform::allocatedBytes()), + static_cast(platform::allocatedPeak()), + static_cast(platform::allocatedCount()), static_cast(scheduler_ ? scheduler_->elapsed() / 1000 : 0)); // Per-module timing (walk tree recursively) @@ -2970,6 +2978,16 @@ void HttpServerModule::pushStateToWebSockets() { if (stateSend_.active) return; JsonSink sink; buildStateJson(sink); + // A sink that ran out of heap holds a TRUNCATED document, and the frame header would declare + // it complete: the browser parses it, throws, and drops every module past the cut. Send it + // anyway, but CLEAR the resync flag first and say so. Returning early here instead looks + // safer and is worse: fullResyncPending_ stays set, the else-branch that pushes value + // patches is never reached, and the whole UI freezes (no fps, no heap, no live values) on a + // board where the state simply does not fit. A partial tree that keeps updating beats a + // whole one that never arrives. (Bench 2026-09-08, both classic boards.) + if (sink.overflowed()) { + setStatus("state too large for free memory: some modules may not show", Severity::Warning); + } const size_t len = sink.size(); char* owned = sink.detach(); // move ownership to the sender (frees on drain-complete) if (owned && startBufferedTextSend(owned, len)) { diff --git a/src/core/JsonSink.h b/src/core/JsonSink.h index e2b91bda..b10b68ad 100644 --- a/src/core/JsonSink.h +++ b/src/core/JsonSink.h @@ -47,6 +47,7 @@ class JsonSink { if (fixed_ && fixedCap_ > 0) fixed_[0] = '\0'; } + ~JsonSink() { if (heap_) platform::free(heap_); } JsonSink(const JsonSink&) = delete; @@ -66,7 +67,11 @@ class JsonSink { fixed_[fixedLen_++] = *s++; fixed_[fixedLen_] = '\0'; } else { - if (!ensureHeap(heapLen_ + 1)) return; // out of memory: drop + // Out of memory. Flag it exactly as the fixed-buffer path above does: a caller + // that ships the buffer anyway sends a TRUNCATED document, and a truncated JSON + // frame is indistinguishable from a whole one at the far end (the browser parses + // it, throws, and drops the tail: the module cards past the cut simply vanish). + if (!ensureHeap(heapLen_ + 1)) { overflowed_ = true; return; } heap_[heapLen_++] = *s++; } } @@ -225,17 +230,36 @@ class JsonSink { } // Grow the heap buffer to hold at least `need` bytes plus a null terminator. + /// + /// Doubling is the right default (amortized O(1) appends), but the old and new buffers are both + /// live across the memcpy, so growing 16 KB to 32 KB asks for ~48 KB of CONTIGUOUS heap at once. + /// A classic ESP32 serving a 40 KB state document has the free bytes and not the block: the + /// allocation failed, every later append dropped, and the device shipped a truncated frame that + /// looked complete. So a refused doubling steps down toward the minimum rather than giving up: + /// slower to grow, and it fits where doubling cannot. (Measured on the bench 2026-09-08: both + /// classic boards cut their state at a power of two, losing 8-11 KB of the tree.) bool ensureHeap(size_t need) { if (need + 1 <= heapCap_) return true; - size_t newCap = heapCap_ == 0 ? 2048 : heapCap_ * 2; - while (newCap < need + 1) newCap *= 2; - char* grown = static_cast(platform::alloc(newCap)); - if (!grown) return false; - if (heap_) { std::memcpy(grown, heap_, heapLen_); platform::free(heap_); } - heap_ = grown; - heapCap_ = newCap; - heap_[heapLen_] = 0; - return true; + size_t want = heapCap_ == 0 ? 2048 : heapCap_ * 2; + while (want < need + 1) want *= 2; + // Step down in QUARTERS of the current capacity rather than to `need + 1`. Backing off to the + // bare minimum serves this one append and then grows again on the next character, which is + // O(n^2) copying and thrashes exactly the fragmented heap that refused the doubling. A + // quarter still leaves useful headroom, so the next grow is thousands of appends away. + const size_t step = heapCap_ / 4 > 4096 ? heapCap_ / 4 : 4096; + const size_t floorCap = need + 1 > step ? need + 1 : step; + for (;;) { + if (char* grown = static_cast(platform::alloc(want))) { + if (heap_) { std::memcpy(grown, heap_, heapLen_); platform::free(heap_); } + heap_ = grown; + heapCap_ = want; + heap_[heapLen_] = 0; + return true; + } + if (want <= floorCap) return false; // even a useful minimum is refused: genuinely out + const size_t next = want - step; + want = next < floorCap ? floorCap : next; + } } platform::TcpConnection* conn_ = nullptr; // socket mode when non-null diff --git a/src/core/SystemModule.h b/src/core/SystemModule.h index 3d4c1877..36f573c4 100644 --- a/src/core/SystemModule.h +++ b/src/core/SystemModule.h @@ -3,6 +3,7 @@ #include "core/MoonModule.h" #include "core/Scheduler.h" #include "core/FilesystemModule.h" // setDeviceModel() arms the debounced save (noteDirty) +#include "core/build_info.h" // kFirmwareName: the variant persisted for MoonBase #include "platform/platform.h" #include @@ -136,6 +137,17 @@ class SystemModule : public MoonModule { /// Changing verbosity does not reshape any derived state, so this is onControlChanged, not prepare. void onControlChanged(const char* controlName) override { if (std::strcmp(controlName, "logLevel") == 0) applyLogLevel(); + // `firmware` is persisted Text (so MoonBase can read it from the config file), and the + // config load writes the PREVIOUS image's value back over the compile-time one that + // defineControls set: a board flashed from eth-wifi to eth-only kept reporting eth-wifi, + // and that field is what steers MoonBase's recovery list toward the right flash layout. + // The compile-time constant is the truth, so re-assert it whenever the control is written + // and mark it dirty, which persists the correction rather than the stale value. + if (std::strcmp(controlName, "firmware") == 0 + && std::strcmp(firmwareVariant_, kFirmwareName) != 0) { + std::snprintf(firmwareVariant_, sizeof(firmwareVariant_), "%s", kFirmwareName); + markDirty(); + } } void defineControls() override { @@ -158,6 +170,20 @@ class SystemModule : public MoonModule { // the UI (pushed, never user-typed); bound as Text — not ReadOnly — because Text is // auto-persisted and the readonly flag is only a UI-render hint. controls_.addText("deviceModel", deviceModel_, sizeof(deviceModel_), validateDeviceModel); + + // firmware: the build variant this image is (`esp32s3-zero`), written from kFirmwareName + // on every boot rather than read from the file: the compile-time constant is the truth, + // and persisting it only puts it somewhere ANOTHER IMAGE can read. + // + // That reader is MoonBase. It is chip-specific but variant-agnostic (one image serves + // every variant of a chip), so on its own it can only offer every firmware for the chip + // and ask a user in recovery to pick the right one, where picking an esp32s3-n16r8 for a + // Zero installs a flash layout the board does not have. Reading this file narrows the list + // to one, the same way the application's own picker does. Text, not ReadOnly, for the + // reason deviceModel gives above: Text is what gets persisted. + std::snprintf(firmwareVariant_, sizeof(firmwareVariant_), "%s", kFirmwareName); + controls_.addText("firmware", firmwareVariant_, sizeof(firmwareVariant_)); + controls_.setHidden(controls_.count() - 1, true); // FirmwareUpdateModule's card shows it controls_.setReadOnly(controls_.count() - 1, true); // Dynamic (updated every second) @@ -349,6 +375,7 @@ class SystemModule : public MoonModule { // entry ("Olimex ESP32-Gateway Rev G" = 26) with headroom; the Improv RPC handler // caps str_len against this size dynamically. char deviceModel_[32] = {}; + char firmwareVariant_[24] = {}; ///< the build variant, persisted for MoonBase to read // Dynamic (updated in tick1s) char uptimeStr_[16] = {}; diff --git a/src/core/moonlive/MoonLive.cpp b/src/core/moonlive/MoonLive.cpp index d36fa3be..2f69e263 100644 --- a/src/core/moonlive/MoonLive.cpp +++ b/src/core/moonlive/MoonLive.cpp @@ -1,4 +1,5 @@ #include "core/moonlive/MoonLive.h" +#include #include "core/moonlive/MoonLiveCompiler.h" #include "platform/platform.h" @@ -113,7 +114,22 @@ bool MoonLive::compile(const char* source, const BuiltinTable& table, const SysV // the compiler's convention and everyone else's, rather than in each reader. if (!cr.ok) { freeCode(); - error_ = cr.error; + // The allocator's refusal is the one error worth the numbers behind it (which guard, and the + // budget it saw): they are what turned "codegen failed" into a diagnosis on the bench. One + // FILE-static line, formatted on this cold path only: not a buffer per engine (there are + // several), not one per result (a temporary; a pointer into it dangled). Two modules failing + // this way at once would share the line, which is a diagnostic corner, not a data path. + if (cr.error == kSpillRefused) { + // 160: the literal is 63 bytes and each of six uint8 fields can print three digits, which + // GCC's -Wformat-truncation proved 112 could clip (clang says nothing, the CI gap). + static char detail[160]; + const SpillDetail& d = spillDetail(); + std::snprintf(detail, sizeof(detail), "%s (guard %u, avail %u, temps %u, vregs %u, slots %u, spilled %u)", + kSpillRefused, d.guard, d.avail, d.temps, d.vregs, d.slots, d.spilled); + error_ = detail; + } else { + error_ = cr.error; + } errorPos_ = cr.errorCol > 0 ? static_cast(cr.errorCol - 1) : 0; hasErrorPos_ = true; return false; diff --git a/src/core/moonlive/MoonLiveCompiler.cpp b/src/core/moonlive/MoonLiveCompiler.cpp index a7af4cea..daa40146 100644 --- a/src/core/moonlive/MoonLiveCompiler.cpp +++ b/src/core/moonlive/MoonLiveCompiler.cpp @@ -1949,13 +1949,14 @@ CompileResult compileSource(const char* source, const BuiltinTable& table, if (!source) { r.error = "no source"; return r; } if (!out || cap == 0) { r.error = "no code buffer"; return r; } - // Size the op array to THIS script before parsing. The bound is per-TOKEN rather than - // per-construct: no token the lexer can produce lowers to more than a handful of ops (the - // densest is a call argument — evaluate, then the Call itself), so counting tokens and - // multiplying is an over-estimate that cannot undershoot. Over-estimating costs a few unused - // entries on a cold path; undershooting would fail a script that fits, so the direction of the - // error is the whole point. push() still refuses past `cap`, so a wrong estimate degrades with - // a diagnostic rather than corrupting memory. + // Size the op array to THIS script before parsing, from its token count times kIrOpsPerToken. + // That constant is EMPIRICAL, not a per-construct bound: measured across every shipped script + // the ratio is 0.75 ops per token and never above 0.85, and the reservation is the largest + // single block a compile makes on a classic ESP32, so it is sized to what scripts need rather + // than to a worst case that failed real scripts (see kIrOpsPerToken). A script denser than the + // constant is refused by push() with "script too large" rather than corrupting memory, which is + // the graceful direction; the cost of a too-small constant is a refusal, of a too-large one a + // compile that cannot get its memory at all. const uint32_t tokens = countTokens(source); IrProgram ir; // +8 covers a program's fixed overhead (the prologue/epilogue ops a tiny script still needs) @@ -1978,7 +1979,21 @@ CompileResult compileSource(const char* source, const BuiltinTable& table, // what a device would execute without flashing one. The front end is identical either way, which // is the point — the seam is one function pointer, not a second copy of the compiler. size_t len = lower ? lower(ir, out, cap, squeeze) : lowerToBytes(ir, out, cap, squeeze); - if (len == 0) { r.error = kCodegenFailed; return r; } + // len == 0 with a null buffer means an allocation failed upstream, not that the program is + // unlowerable: report the cause a user can act on (free memory) rather than one they cannot. + if (len == 0) { + switch (lowerRefusal()) { + // The allocator's own numbers are in spillDetail() (thread_local, outlives this result), so + // the error is a literal and the caller formats the detail if it wants it: nothing here + // owns a buffer the result would carry out of scope. + case LowerRefusal::Spill: r.error = kSpillRefused; break; + case LowerRefusal::NullCall: r.error = "codegen failed: a builtin has no function on this target"; break; + case LowerRefusal::AsmOverflow: r.error = "codegen failed: assembler overflow (branch range, slot, or immediate)"; break; + case LowerRefusal::OverCap: r.error = "codegen failed: code larger than its buffer"; break; + default: r.error = kCodegenFailed; break; + } + return r; + } r.ok = true; r.len = len; // Surface the declared controls so the binding can create real MoonModule controls. diff --git a/src/core/moonlive/MoonLiveCompiler.h b/src/core/moonlive/MoonLiveCompiler.h index 41ba933a..94cf6f10 100644 --- a/src/core/moonlive/MoonLiveCompiler.h +++ b/src/core/moonlive/MoonLiveCompiler.h @@ -25,6 +25,7 @@ namespace mm::moonlive { /// Named so a test can distinguish "this host has no JIT" from "this script is wrong" without /// matching on prose. inline constexpr const char* kCodegenFailed = "codegen failed (unsupported on this target, or too large)"; +inline constexpr const char* kSpillRefused = "codegen failed: too many live values for this chip's registers"; // Result of compiling source: on success, ok==true and the bytes are in out[0..len). On diff --git a/src/core/moonlive/MoonLiveIr.h b/src/core/moonlive/MoonLiveIr.h index 7da38fcd..591f83bc 100644 --- a/src/core/moonlive/MoonLiveIr.h +++ b/src/core/moonlive/MoonLiveIr.h @@ -40,10 +40,21 @@ static constexpr uint8_t kMaxVRegs = 32; // a diagnostic instead of asking for an allocation that would exhaust a small device's heap. static constexpr uint16_t kMaxIrOps = 4096; -// Ops a single source token can lower to, worst case. The compiler sizes its op array by counting -// tokens and multiplying — an over-estimate by construction, which is the safe direction: a few -// unused entries on a cold path, versus refusing a script that would have fit. -static constexpr uint16_t kIrOpsPerToken = 4; +// Ops a source token lowers to. The compiler sizes its op array by counting tokens and multiplying, +// so this is a ceiling on the RESERVATION, and it is the number that decides whether a script +// compiles on a classic ESP32 at all. +// +// It was 4, on the reasoning that an over-estimate is the safe direction ("a few unused entries on +// a cold path"). Measured across every shipped script it is 0.75 ops per token, never above 0.85, +// so 4 reserved five to six times the IR a script actually builds: for the largest scripts that was +// ~1,900 ops at 32 bytes each, a 61 KB single-block request for 10 KB of IR, made while the staging +// buffer and the spill pass's second array are also live. On a heap whose largest free block is +// 65 KB it failed, and the compile reported "codegen failed" for scripts that lower fine on the +// host (bench Dig-Octa 2026-09-09: six of 33 shipped scripts). A reservation that refuses a script +// which fits is the wrong direction to be conservative in. 1 keeps a real margin over the measured +// 0.85, and a script that somehow exceeds it still fails cleanly: IrProgram::push refuses past +// cap, which the parser reports as "script too large" rather than writing past the array. +static constexpr uint16_t kIrOpsPerToken = 1; // The op set — neutral. Three-address form: dst plus up to three source operands. (Counted // Control flow arrived with the script-level `for`, which is what the note here anticipated: the diff --git a/src/core/moonlive/MoonLiveSpill.cpp b/src/core/moonlive/MoonLiveSpill.cpp index 2bf682d4..402eaffa 100644 --- a/src/core/moonlive/MoonLiveSpill.cpp +++ b/src/core/moonlive/MoonLiveSpill.cpp @@ -1,4 +1,5 @@ #include "core/moonlive/MoonLiveSpill.h" +#include "core/moonlive/moonlive_emit.h" // Linear-scan register allocation with spilling to the call frame. See MoonLiveSpill.h for why this // lives in core rather than in each backend. @@ -143,19 +144,20 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { // the ONE frame — so a spill numbered from zero would land on a loop counter. Start above them, // and report the total the prologue must reserve. slotsUsed = ir.localSlots; - if (!ir.ops) return false; + if (!ir.ops) { spillDetail().guard = 1; return false; }; + spillDetail() = SpillDetail{}; // every field honest for whichever guard fires, including 1-3 const uint8_t avail = budget.allocatable(); // The front end's own variables have to fit the frame whether or not anything spills — it hands // out slot indices without knowing the target, and a slot the backend cannot address would be // encoded as a truncated offset writing over something else. Checked BEFORE the early return // below, or a program that needs no spilling skips the check entirely. - if (ir.localSlots > kMaxLocals || ir.localSlots > budget.slots) return false; + if (ir.localSlots > kMaxLocals || ir.localSlots > budget.slots) { spillDetail().guard = 2; return false; }; // Already fits: leave the program byte-identical. A script that never needed the allocator must // not pay a renumbering for its existence — and this is the path every shipped script takes. if (ir.vregsUsed <= avail) return true; - if (ir.vregsUsed > kMaxVRegs) return false; + if (ir.vregsUsed > kMaxVRegs) { spillDetail().guard = 3; return false; }; // The fixed ABI vregs (buf, nLights, cpl, t, ctrls) arrive in machine registers the host chose // and every backend indexes them directly, so they can be neither renumbered nor spilled. They @@ -173,11 +175,13 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { // reading `buf` or `t` needs no temp for it. Everything else may end up in a slot and therefore // may need somewhere to land, so count the distinct non-ABI sources of the widest op present. // - // Note the shape this leaves on Xtensa: 10 registers - 1 inline scratch - 5 fixed ABI vregs - // leaves 4, and a setRGB reads exactly 4 distinct operands — so a looped effect lands on - // keepable == 0 and is refused. The pass is right to refuse (it has nothing to allocate with); - // what is wrong is that FIVE registers are reserved for host arguments that a script reads - // rarely. Freeing those is register-promotion work, deliberately out of scope for this step. + // Note the shape this leaves on Xtensa: 10 registers minus 3 inline scratch leaves 7, and the + // ABI vregs are no longer subtracted (see the note below the loop), so a widest-op reservation + // of 4 leaves 3 keepable, which every shipped script fits (pinned by the Xtensa codegen test + // that compiles all of them at the device's own budget). A "codegen failed" for one of those on + // a classic ESP32 is therefore NOT this pass: on 2026-09-09 it was the assembler's heap buffer + // failing to allocate on a fragmented board, misreported as a codegen fault. That allocation is + // gone (the assembler now emits into the caller's buffer) and the report says "no memory". uint8_t reloadTemps = 0; for (uint16_t i = 0; i < ir.count; i++) { VReg s[4]; @@ -196,7 +200,8 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { // they hold a register only for the parking store itself — which runs before any temp exists, so // sharing those registers afterwards is not a conflict. Reserving five here was holding space for // values that had already moved out, and on a ten-register target that was the entire budget. - if (avail <= reloadTemps) return false; + { auto& d = spillDetail(); d.guard = 0; d.avail = avail; d.temps = reloadTemps; d.vregs = ir.vregsUsed; d.slots = ir.localSlots; } + if (avail <= reloadTemps) { spillDetail().guard = 4; return false; }; const uint8_t keepable = static_cast(avail - reloadTemps); // --- 1. Find the loops, innermost first ---------------------------------------------------- @@ -213,10 +218,10 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { for (uint16_t i = 0; i < ir.count; i++) { const IrInst& in = ir.ops[i]; if (in.op != IrOp::BranchNe) continue; - if (in.imm < 0 || in.imm >= kIrLabels) return false; // an unbindable label: refuse + if (in.imm < 0 || in.imm >= kIrLabels) { spillDetail().guard = 5; return false; }; // an unbindable label: refuse const int32_t tgt = labelAt[in.imm]; if (tgt < 0 || static_cast(tgt) > i) continue; // forward branch — not a loop - if (loopCount >= kIrLabels) return false; + if (loopCount >= kIrLabels) { spillDetail().guard = 6; return false; }; loops[loopCount++] = {static_cast(tgt), i}; } // Proper nesting is what makes "innermost first" meaningful and what the extension rule below @@ -228,7 +233,7 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { const bool disjoint = loops[x].back < loops[y].header || loops[y].back < loops[x].header; const bool xInY = loops[y].header <= loops[x].header && loops[x].back <= loops[y].back; const bool yInX = loops[x].header <= loops[y].header && loops[y].back <= loops[x].back; - if (!disjoint && !xInY && !yInX) return false; + if (!disjoint && !xInY && !yInX) { spillDetail().guard = 7; return false; }; } } @@ -313,7 +318,7 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { // nActive >= keepable >= 1 here: the `avail <= reloadTemps` guard above makes keepable at // least one, and this branch is only reached when nActive is not below it. Stated because // the index below would read active[-1] if that invariant ever moved. - if (nActive == 0) return false; + if (nActive == 0) { spillDetail().guard = 8; return false; }; const VReg furthest = active[nActive - 1]; if (iv[furthest].end > iv[cur].end) { iv[furthest].spilled = true; @@ -331,7 +336,8 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { // arguments (hostArgSlot), which are stored once at entry and reloaded wherever a script reads // buf/nLights/cpl/t/ctrls. Allowing a spill into that range would overwrite them — budget.slots // is the frame's whole capacity, of which only the bottom kMaxLocals are assignable. - if (nSpilled > kMaxLocals || nSpilled > budget.slots) return false; + spillDetail().spilled = nSpilled; + if (nSpilled > kMaxLocals || nSpilled > budget.slots) { spillDetail().guard = 9; return false; }; // --- 4. Compact the survivors --------------------------------------------------------------- // The kept temps take the register numbers directly above the fixed ABI vregs, so the rewritten @@ -352,17 +358,61 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { // --- 5. Rewrite ----------------------------------------------------------------------------- // Into a SECOND program: a Reload has to be inserted before the op that reads a spilled value and // a Spill after the op that defines one, and a right-sized array has no room to shift into. + // NOTHING spilled: the program already fits the register file, so every vreg keeps a register + // and the rewrite below would copy the program op for op into a second array only to swap it + // back. Skip it. That is the common case (32 of the 33 shipped scripts) and it makes those + // compiles cost no allocation here at all, which on a classic ESP32 is the difference between + // holding two IR arrays at the peak and holding one. The compacted register numbering still + // has to be applied, since a kept vreg's `assigned` may differ from its original index. + if (nSpilled == ir.localSlots) { + for (uint16_t i = 0; i < ir.count; i++) { + IrInst& in = ir.ops[i]; + VReg src[4]; + const uint8_t n = sourcesOf(in, src); + auto keep = [&](VReg v) -> VReg { return (v < kMaxVRegs && iv[v].live) ? iv[v].assigned : v; }; + if (n > 0) in.a = keep(src[0]); + if (n > 1) in.b = keep(src[1]); + if (n > 2) in.c = keep(src[2]); + if (n > 3) in.d = keep(src[3]); + if (writesDst(in) && in.dst < kMaxVRegs) in.dst = keep(in.dst); + } + ir.vregsUsed = newHighWater; + slotsUsed = nSpilled; + return true; + } + IrProgram out; - // Worst case per op: four Reloads, the op, one Spill. Over-estimating costs a cold-path - // allocation; under-estimating would fail a script that fits, so the direction is deliberate. - const uint32_t want = static_cast(ir.count) * (kMaxReloadTemps + 2); - if (want > kMaxIrOps) return false; - if (!out.reserve(static_cast(want))) return false; + // Sized to what the rewrite below will ACTUALLY emit, counted in a dry pass over the same rules: + // one Reload per distinct spilled source of an op, the op, one Spill per spilled destination. + // It used to reserve the worst case, six ops per input op, and that was the ceiling on a classic + // ESP32: for a 284-op script it asked for ~41 KB in one block while the first IR array, the + // staging buffer and the assembler were all still live, so on a heap whose largest free block + // had fragmented to 65 KB the allocation failed and the compile reported "codegen failed" for a + // script that spills nothing at all (bench 2026-09-09, plasma/balls/nebula/aurora). The exact + // count is a fraction of that for every shipped script, and it is also the honest number: an + // estimate that fails a script which fits is the wrong direction to be conservative in. + uint32_t want = 0; + for (uint16_t i = 0; i < ir.count; i++) { + const IrInst& in = ir.ops[i]; + VReg src[4]; + const uint8_t n = sourcesOf(in, src); + for (uint8_t s = 0; s < n; s++) { + const VReg v = src[s]; + if (v >= kMaxVRegs || !iv[v].spilled) continue; + bool already = false; + for (uint8_t q = 0; q < s; q++) if (src[q] == v) { already = true; break; } + if (!already) want++; // a Reload + } + want++; // the op itself + if (writesDst(in) && in.dst < kMaxVRegs && iv[in.dst].spilled) want++; // a Spill + } + if (want > kMaxIrOps) { spillDetail().guard = 10; return false; }; + if (!out.reserve(static_cast(want))) { spillDetail().guard = 11; return false; }; auto emit = [&](const IrInst& in) { // push() also re-validates every vreg against kMaxVRegs, so a rewrite that named a register // outside the budget fails the compile here instead of reaching a backend's register map. - if (!out.push(in)) return false; + if (!out.push(in)) { spillDetail().guard = 12; return false; }; return true; }; @@ -408,7 +458,7 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { rl.op = IrOp::Reload; rl.dst = tempOf[s]; rl.imm = iv[v].slot; - if (!emit(rl)) return false; + if (!emit(rl)) { spillDetail().guard = 13; return false; }; } // Rewrite the operands in place: a spilled one now names its temp, a kept one its compacted @@ -430,13 +480,13 @@ bool spillToBudget(IrProgram& ir, const RegBudget& budget, uint8_t& slotsUsed) { // instruction, so the aliasing is the ordinary `add d, d, b` every ISA here defines. in.dst = dstSpilled ? firstTemp : (in.dst < kMaxVRegs ? iv[in.dst].assigned : in.dst); } - if (!emit(in)) return false; + if (!emit(in)) { spillDetail().guard = 14; return false; }; if (dstSpilled) { IrInst sp{}; sp.op = IrOp::Spill; sp.a = firstTemp; sp.imm = dstSlot; - if (!emit(sp)) return false; + if (!emit(sp)) { spillDetail().guard = 15; return false; }; } } diff --git a/src/core/moonlive/moonlive_emit.h b/src/core/moonlive/moonlive_emit.h index da0b0a67..2457cf02 100644 --- a/src/core/moonlive/moonlive_emit.h +++ b/src/core/moonlive/moonlive_emit.h @@ -70,6 +70,21 @@ struct IrProgram; // src/core/moonlive/MoonLiveIr.h // know about an ISA, and the reason the allocator is written once instead of three times. Each // backend fills this in from its own map and hands it to spillToBudget (MoonLiveSpill.h); nothing // ISA-specific crosses in the other direction. +// WHY the last lowering returned 0. A lowering has four distinct ways to refuse and they used to +// share one return value, so a failure on the device read "codegen failed (unsupported on this +// target, or too large)" whether the register allocator gave up, the assembler overflowed, or the +// code outgrew its buffer. Compiling the same script on the host through the same emitter succeeded, +// which left only the device able to say which, and it could not. Static rather than threaded +// through LowerFn: the seam has three backends and a test double, and the compile is single-threaded +// per call, so one byte read straight after the call is the whole contract. +enum class LowerRefusal : uint8_t { None, Spill, NullCall, AsmOverflow, OverCap }; +inline LowerRefusal& lowerRefusal() { thread_local LowerRefusal r = LowerRefusal::None; return r; } +// The register allocator's own refusal detail: which of its guards fired, and the budget it saw. +// Written by spillToBudget, read by compileSource into the error string, so a device can say +// "avail 7, temps 3, guard 5" instead of one message for six different causes. +struct SpillDetail { uint8_t guard = 0, avail = 0, temps = 0, vregs = 0, slots = 0, spilled = 0; }; +inline SpillDetail& spillDetail() { thread_local SpillDetail d; return d; } + struct RegBudget { uint8_t regs = 0; // machine registers the vreg map exposes (kRegCount) uint8_t reserved = 0; // registers the backend keeps for the inline ops this program contains diff --git a/src/core/moonlive/moonlive_lower.h b/src/core/moonlive/moonlive_lower.h index dc811dec..0b0759ff 100644 --- a/src/core/moonlive/moonlive_lower.h +++ b/src/core/moonlive/moonlive_lower.h @@ -78,7 +78,8 @@ size_t lowerWith(IrProgram& ir, uint8_t* out, size_t cap, const RegBudget* squee // instruction. constexpr uint8_t kSharedScratch = 2; const uint8_t scratchTotal = kSharedScratch + 1; - if (!out || cap == 0) return 0; + lowerRefusal() = LowerRefusal::None; + if (!out || cap == 0) return 0; // unreachable from compileSource, which refuses this first // Run the register allocator before lowering. It leaves a program that already fits untouched, // and rewrites one that does not into Spill/Reload against this backend's frame, replacing the @@ -91,16 +92,17 @@ size_t lowerWith(IrProgram& ir, uint8_t* out, size_t cap, const RegBudget* squee // lowering then overwrites, miscompiling exactly the squeezed programs the seam exists to prove. const RegBudget budget = squeeze ? RegBudget{squeeze->regs, scratchTotal, squeeze->slots} : RegBudget{regCount, scratchTotal, A::kMaxSpillSlots}; - if (!spillToBudget(ir, budget, slots)) return 0; + if (!spillToBudget(ir, budget, slots)) { lowerRefusal() = LowerRefusal::Spill; return 0; } // sAddr FIRST: it is the one StoreElem also uses, and a store-only program reserves a single // scratch, so the shared one has to be the lowest index or it would name an unreserved register. const RegId sAddr = static_cast(ir.vregsUsed); // per-channel address (both ops) const RegId sCtr = static_cast(ir.vregsUsed + 1); // FillElems loop counter - // Size the assembler's buffer to the CALLER's: `cap` is what the staging buffer holds, so the - // two can never disagree about how much a script may emit (they were separately constant, and - // a script that fit one overflowed the other). - A a(cap); + // Emit INTO the caller's buffer. `out` and `cap` are the staging buffer and its size, so the + // assembler and the caller cannot disagree about how much a script may emit (they were + // separately constant once, and a script that fit one overflowed the other), and there is no + // second full-size allocation to fail: see the borrowing constructor for what that cost. + A a(out, cap); using LabelId = decltype(a.newLabel()); // The LAST reserved scratch index, derived from scratchTotal rather than hard-coded: the `+1` // in scratchTotal above IS this register, so the reservation and the use cannot drift apart. @@ -402,7 +404,7 @@ size_t lowerWith(IrProgram& ir, uint8_t* out, size_t cap, const RegBudget* squee // the FRAME SLOT its arguments start at. Hand the host their address and their // count: nothing is held in a register across the call, and how many arguments a // builtin takes stops being a property of this instruction. - if (!op.callFn) return 0; + if (!op.callFn) { lowerRefusal() = LowerRefusal::NullCall; return 0; } const RegId argPtr = static_cast(ir.vregsUsed); a.slotAddr(argPtr, static_cast(op.imm)); const RegId argN = static_cast(ir.vregsUsed + 1); @@ -463,9 +465,9 @@ size_t lowerWith(IrProgram& ir, uint8_t* out, size_t cap, const RegBudget* squee if (ir.fnCount > 0) closeFn(static_cast(ir.fnCount - 1)); else a.epilogue(); a.finalize(); - if (a.overflowed() || a.size() > cap) return 0; - std::memcpy(out, a.bytes(), a.size()); - return a.size(); + if (a.overflowed()) { lowerRefusal() = LowerRefusal::AsmOverflow; return 0; } + if (a.size() > cap) { lowerRefusal() = LowerRefusal::OverCap; return 0; } + return a.size(); // already in `out`: the assembler emitted there } } // namespace mm::moonlive diff --git a/src/light/ColorLight5A75Packet.h b/src/light/ColorLight5A75Packet.h index ca58d364..4f33149d 100644 --- a/src/light/ColorLight5A75Packet.h +++ b/src/light/ColorLight5A75Packet.h @@ -73,8 +73,14 @@ constexpr size_t COLORLIGHT_ROW_PREFIX = COLORLIGHT_DATA_OFFSET + COLORLIGHT_ROW constexpr uint16_t COLORLIGHT_MAX_PIXELS_PER_PACKET = 497; constexpr uint8_t COLORLIGHT_BYTES_PER_PIXEL = 3; -// Largest frame built: 21 + 497×3 = 1512 bytes. Above the 1500-byte MTU on purpose — these are raw +// Largest frame built: 21 + 497x3 = 1512 bytes. Above the 1500-byte MTU on purpose: these are raw // L2 frames on a dedicated link to dumb receivers, never IP packets a router would fragment. +// +// FPP packs rows the same way. Harald Kubota's write-up sends 128 pixels per packet instead (391 +// bytes), which stays inside the MTU: both work against the cards, and the larger packet is fewer +// frames for the same wall. Worth knowing if a wall ever goes dark through a SWITCH that will not +// pass a 1512-byte frame, since nothing in the path reports that: the card simply never sees a row +// and its activity LED stays still, which looks exactly like a transmit path that is not running. constexpr size_t COLORLIGHT_MAX_FRAME = COLORLIGHT_ROW_PREFIX + COLORLIGHT_MAX_PIXELS_PER_PACKET * COLORLIGHT_BYTES_PER_PIXEL; // 1512 diff --git a/src/light/drivers/ParallelLedDriver.h b/src/light/drivers/ParallelLedDriver.h index 0b0ff867..95ac3481 100644 --- a/src/light/drivers/ParallelLedDriver.h +++ b/src/light/drivers/ParallelLedDriver.h @@ -75,6 +75,11 @@ class ParallelLedDriver : public DriverBase { if (p) p->attach(this); } + /// Push the heap total to the module readout. Production calls this from the alloc/free sites + /// (publishHeapBytes); a test that drives a mock peripheral's busInit directly has no such site, + /// so it asks for the recompute here and then asserts the PUBLIC dynamicBytes() the card shows. + void publishHeapBytesForTest() { publishHeapBytes(); } + // --- Peripheral backend registry --- // The set of peripheral backends compiled into THIS build. Each backend header self-registers its // factory + label at static-init, via an `inline const bool kXxxPeripheralRegistered = @@ -169,9 +174,10 @@ class ParallelLedDriver : public DriverBase { /// SIMULTANEOUSLY, fed consecutive slices of this driver's window. A token may be a single pin or an /// inclusive range ("20-23" → 20,21,22,23), mixing freely ("20-22,35,38-40"). Shared control shape with /// RmtLedDriver (parsers in PinList.h). Defaults live on the derived (chip-specific safe pins), - /// so the derived sets them after construction; the base just declares them. i80 needs exactly - /// 8 OR 16 real pins (a partial bus is rejected — a sub-16 board parks unused lanes + WR/DC on - /// spare GPIOs); Parlio runs on 1..16. Sized for 16 two-digit GPIOs + separators. + /// so the derived sets them after construction; the base just declares them. Both backends take + /// 1..16 pins: the i80 BUS WIDTH rounds to 8 or 16 (powerOfTwoBus) and parks the unused lanes plus + /// WR/DC on spare GPIOs, which is invisible to the pin list; Parlio's width is the pin count, so + /// nothing rounds. Sized for 16 two-digit GPIOs + separators. char pins[64] = ""; /// Comma-separated lights-per-lane; the unassigned remainder splits evenly over the remaining /// lanes. **A lane is a STRAND, not a pin** — which only differ through the expander: direct @@ -713,8 +719,8 @@ class ParallelLedDriver : public DriverBase { void tick1s() MM_NONBLOCKING override { if (!peripheral_) return; // A bus that lost a shared peripheral to another module comes back on its own once that - // module lets go. On the classic ESP32 the i80 bus and a PDM microphone both need I2S0, so - // whichever asks second is refused; without this the loser stayed dark until the user + // module lets go. On the classic ESP32 the LED bus is an I2S peripheral and drives from + // instance 1, so whichever asks second is refused; without this the loser stayed dark until the user // happened to edit a control, which is a reboot-to-apply in all but name (architecture.md, // live reconfiguration). Gated tightly, because this runs on the render thread: only while // the driver WANTS the bus and does not hold it, and only when the backend says the thing @@ -1378,10 +1384,29 @@ class ParallelLedDriver : public DriverBase { size_t snapshotCap_ = 0; // allocated capacity, grows to fit the window /// This driver's heap = the base scratch + the streaming snapshot (the ring's immutable frame copy, - /// the biggest single driver buffer at ~36 KB). Summed for the per-module memory readout — see - /// DriverBase::driverHeapBytes. The DMA ring buffers are platform-owned (not driver heap), so they - /// are not counted here. - size_t driverHeapBytes() const override { return DriverBase::driverHeapBytes() + snapshotCap_; } + /// the biggest single driver buffer at ~36 KB) + the peripheral's DMA buffers. Summed for the + /// per-module memory readout (see DriverBase::driverHeapBytes). + /// + /// The DMA buffers are allocated by the PLATFORM rather than by this driver, and they used to be + /// left out on that ownership argument. That made the readout lie about the thing a user actually + /// decides on: the i80 frame is sized by the BUS WIDTH (8 or 16 lanes) and not by the pins in use, + /// so a one-lane board pays the same ~50 KB as an eight-lane one. On the bench (2026-09-08) a + /// QuinLED Dig-2-Go showed 512 bytes for this driver against RmtLedDriver's 32 KB while actually + /// costing 49 KB MORE free heap, and nothing on the card said so. Ownership is the wrong question + /// for a memory readout: what the user needs is what choosing this driver costs. + size_t driverHeapBytes() const override { + size_t dma = 0; + if (peripheral_) { + const size_t cap = peripheral_->busCapacity(); + // Count the buffers that EXIST, not the ones `doubleBuffer` asks for: the control + // defaults on, a peripheral may refuse the second allocation (or not support it at + // all), and a readout that trusts the request over the allocation reports memory the + // board never spent. + dma = cap; + if (peripheral_->busBuffer(1)) dma += cap; + } + return DriverBase::driverHeapBytes() + snapshotCap_ + dma; + } // The snapshot copy's inputs, set before copyRange runs. The snapshot is serial (on the ring's core-1 // tick); only the ring PRIME still forks to the core-0 helper (busTransmitRing), so no cross-core copy // bounds live here. diff --git a/src/light/drivers/ParallelSlots.h b/src/light/drivers/ParallelSlots.h index 3715b74a..d7a9215d 100644 --- a/src/light/drivers/ParallelSlots.h +++ b/src/light/drivers/ParallelSlots.h @@ -41,7 +41,7 @@ namespace mm { /// Bus bit L = the L-th entry of the driver's `pins` list (D0 = first pin). /// Bits go MSB-first per byte; channel order (GRB, …) is already applied by /// Correction before the encode, so the encoder is order-agnostic (same -/// contract as encodeWs2812Symbols). +/// contract the RMT driver's wire bytes follow). /// /// The data slot is an 8×8 BIT-MATRIX TRANSPOSE: 8 lane bytes (rows) → 8 bus /// bytes (one per data bit, the columns), byte b bit L = lane L's bit b. This diff --git a/src/light/drivers/RmtLedDriver.h b/src/light/drivers/RmtLedDriver.h index 80498f46..cd5b3106 100644 --- a/src/light/drivers/RmtLedDriver.h +++ b/src/light/drivers/RmtLedDriver.h @@ -4,7 +4,7 @@ #include "light/drivers/LedDriverConfig.h" #include "light/drivers/PinList.h" // parsePinList / assignCounts (shared with MultiPinLedDriver) -#include "light/drivers/RmtSymbol.h" // encodeWs2812Symbols (host-testable) +#include "light/drivers/RmtSymbol.h" // makeRmtSymbol (the bit shapes the peripheral expands with) #include "platform/platform.h" namespace mm { @@ -15,7 +15,7 @@ namespace mm { /// buffer (8-bit, GRB). The default LED driver for classic-ESP32 and S3 board entries, and the /// readable EXAMPLE future LED drivers copy: a sibling of NetworkSendDriver (same DriverBase hooks, /// same per-light `correction_.apply()` guard, same once-allocated owned buffer sized off the hot -/// path); only the emit differs — this fuses the correction + WS2812 symbol-encode into one pass +/// path); only the emit differs: this fuses the correction into the wire-byte frame into one pass /// (the encode is `RmtSymbol.h`, host-tested) then hands per-pin slices to the platform. /// /// **Wire contract — [WS2812B](https://cdn-shop.adafruit.com/datasheets/WS2812B.pdf):** 1-wire NRZ @@ -218,7 +218,7 @@ class RmtLedDriver : public DriverBase { /// Parse the config and (re)init the RMT channels. Lifecycle has two /// deliberately-separate concerns, so the buffer half stays host-testable and a /// hardware-only guard can never strand it: - /// - SYMBOL BUFFER (plain heap): resizeSymbols() / freeSymbols(), run on + /// - FRAME BUFFER (plain heap): resizeFrame() / freeFrame(), run on /// every platform. /// - RMT CHANNELS (hardware): reinit() / deinitAll(), RMT-targets-only /// (if constexpr). @@ -234,7 +234,7 @@ class RmtLedDriver : public DriverBase { /// fail/config-error state (DriverBase::release()). void release() override { deinitAll(); - freeSymbols(); + freeFrame(); DriverBase::release(); // frees the correction scratch, clears failBuf_ + configErr_ } @@ -243,7 +243,7 @@ class RmtLedDriver : public DriverBase { /// only calls this when effectively-enabled and routes to release() (release) otherwise, so the /// channels + buffer free when the driver, or a parent, is disabled. void prepare() override { - // Drain first. resizeSymbols() may free the symbol buffer and reinit() deletes the + // Drain first. resizeFrame() may free the symbol buffer and reinit() deletes the // channel, and a prepare arrives from a control change, which can land mid-frame: the // peripheral is then still reading those symbols. Bounded, because a wedged transfer must // not block a config change forever; past the deadline the rebuild proceeds, which is the @@ -251,14 +251,14 @@ class RmtLedDriver : public DriverBase { if (txInFlight_) { for (uint8_t attempt = 0; attempt < 4 && txInFlight_; attempt++) txInFlight_ = !waitForPins(); - // Still busy after every attempt: the peripheral is reading symbols_ right now, so + // Still busy after every attempt: the peripheral is reading frame_ right now, so // rebuilding would free the buffer under it, which is the corruption this drain exists // to prevent. Defer instead. tick() re-waits and the config applies on a later prepare; // the alternative, rebuilding anyway, trades a delayed config change for a torn frame. if (txInFlight_) return; } parseConfig(); - resizeSymbols(); + resizeFrame(); reinit(); // Re-assert the resting "driving N of M lights" status after the full build. parseConfig sets it // too, but only when a buffer is already wired (txLightCount_ > 0); on the boot path setup()'s @@ -269,14 +269,18 @@ class RmtLedDriver : public DriverBase { // touching configErr_/configWarn_, so the `!warn` rule alone would overwrite that error with a // false "driving N lights" while tick() bails and the strand stays dark. Only assert the resting // status when the channels actually came up. - if (inited_ && !configErr_ && !configWarn_ && txLightCount_ > 0) + // frameUnusable_ joins inited_ here for the same reason: resizeFrame reports the + // no-transmit case at Severity::Error without touching configErr_/configWarn_, so the + // `!warn` rule alone would replace it with a false "driving N lights" while the strip sits + // frozen. That false-OK status is precisely what made issue #94 so hard to place. + if (inited_ && !frameUnusable_ && !configErr_ && !configWarn_ && txLightCount_ > 0) setDrivingInfo(txLightCount_, winLen_, correction_.outChannels); } /// Preset toggle (RGB↔RGBW) changes outChannels without a structural rebuild — /// the per-pin symbol offsets scale with outChannels, so re-derive them too. Skipped /// while (effectively) disabled (would re-alloc the symbol buffer a disabled driver released). - void onCorrectionChanged() override { if (!effectivelyEnabled()) return; parseConfig(); resizeSymbols(); } + void onCorrectionChanged() override { if (!effectivelyEnabled()) return; parseConfig(); resizeFrame(); } /// Point the driver at the source frame buffer; re-parse (counts derive from its light count) /// and resize the symbol buffer to match. The resize is skipped while (effectively) disabled @@ -284,7 +288,7 @@ class RmtLedDriver : public DriverBase { void setSourceBuffer(Buffer* buf) override { sourceBuffer_ = buf; parseConfig(); // counts derive from the buffer's light count - if (effectivelyEnabled()) resizeSymbols(); + if (effectivelyEnabled()) resizeFrame(); } /// Per-tick output: fuse the correction and WS2812 symbol-encode in one pass @@ -302,7 +306,7 @@ class RmtLedDriver : public DriverBase { // Encode within this driver's window only. winLen_ is the slice length; // txLightCount_ (Σ pinCounts_) is what the pins clock out — n is the min, // so a window smaller than the configured pin total never reads past it. - // A frame still on the wire OWNS symbols_: the RMT copy encoder streams straight out of it, + // A frame still on the wire OWNS frame_: the peripheral expands its bytes straight out of it, // so re-encoding now rewrites bytes the peripheral is mid-way through clocking. That is not // a dropped frame, it is a corrupted one, and it shows as a handful of lights in a color // the effect never drew. Only a timed-out wait leaves this set, so the normal path never @@ -310,7 +314,7 @@ class RmtLedDriver : public DriverBase { // encodes cleanly. Bench: this is what remained after the memory-block and interrupt // priority work, and it is independent of light count, which is what ruled those out. if (txInFlight_) { - txInFlight_ = !waitForPins(); // still busy: leave symbols_ alone for another tick + txInFlight_ = !waitForPins(); // still busy: leave frame_ alone for another tick if (txInFlight_) return; } @@ -319,24 +323,19 @@ class RmtLedDriver : public DriverBase { // Same defensive guard ArtNet uses: skip rather than overrun if the // symbol buffer is stale (e.g. correction swapped without a resize). if (n == 0 || outCh == 0 || pinCount_ == 0 - || !symbols_ || symbolCap_ < symbolsFor(n, outCh) - || !wire_ || wireCap_ < outCh) return; // wire_ sized to outChannels — skip if not ready + || !frame_ || frameCap_ < frameBytesFor(n, outCh)) return; // buffer not ready // Fused single pass: correct one light into wire bytes, encode those // bytes straight into the symbol buffer. No second sweep over encoded // data, no per-light heap. const uint8_t* src = sourceBuffer_->data(); const uint8_t srcCh = sourceBuffer_->channelsPerLight(); - const uint16_t t0h = nsToTicks(cfg_.t0h_ns); - const uint16_t t1h = nsToTicks(cfg_.t1h_ns); - const uint16_t period = nsToTicks(cfg_.period_ns); - size_t s = 0; + // Correct each light straight into the WIRE-BYTE frame. The bit expansion that used to + // happen here (one 32-bit symbol per data bit, 96 bytes per RGB light) now happens on the + // way to the peripheral, so this buffer is outCh bytes per light: 3 KB at 1024 lights + // where the symbol form wanted 96 KB and fell back to unusable PSRAM (issue #94). for (nrOfLightsType i = 0; i < n; i++) { - // Read the windowed light: this driver's slice starts at winStart_. wire_ is sized to - // outChannels off the hot path (resizeSymbols), so apply() can't overflow it. - correction_.apply(src + (winStart_ + i) * srcCh, wire_, srcCh); - encodeWs2812Symbols(wire_, outCh, t0h, t1h, period, symbols_ + s); - s += static_cast(outCh) * 8; + correction_.apply(src + (winStart_ + i) * srcCh, frame_ + static_cast(i) * outCh, srcCh); } // Start every pin's slice before waiting on any — the channels clock out // concurrently, so the tick is charged the longest strand, not the sum. @@ -350,16 +349,16 @@ class RmtLedDriver : public DriverBase { // Normally Σ pinCounts_ == n, but if the buffer shrank since the last parseConfig (a grid // resize lands a tick before the config re-parse) n can be below Σ pinCounts_ — cap each // pin at the encoded boundary so it never clocks out stale symbols past what we wrote. - const size_t wordsPerLight = static_cast(outCh) * 8; + const size_t bytesPerLight = static_cast(outCh); bool started[kMaxPins] = {}; for (uint8_t i = 0; i < pinCount_; i++) { - const nrOfLightsType pinStart = static_cast(pinOffsets_[i] / wordsPerLight); + const nrOfLightsType pinStart = static_cast(pinOffsets_[i] / bytesPerLight); if (pinStart >= n) break; // contiguous: this pin and all later ones are past the encoded lights const nrOfLightsType pinLights = (pinStart + pinCounts_[i] > n) ? static_cast(n - pinStart) : pinCounts_[i]; if (pinLights == 0) continue; - started[i] = platform::rmtWs2812Transmit(rmt_[i], symbols_ + pinOffsets_[i], - static_cast(pinLights) * wordsPerLight); + started[i] = platform::rmtWs2812Transmit(rmt_[i], frame_ + pinOffsets_[i], + static_cast(pinLights) * bytesPerLight); } for (uint8_t i = 0; i < pinCount_; i++) started_[i] = started[i]; txInFlight_ = !waitForPins(); @@ -379,20 +378,20 @@ class RmtLedDriver : public DriverBase { return allDone; } - /// Test-only accessors. symbolBuffer/symbolCapacity mirror ArtNet's + /// Test-only accessors. frameBuffer/frameCapacity mirror ArtNet's /// correctedBuffer() and let unit tests pin the buffer-lifecycle invariants a - /// hardware bug already taught us; pinCount/pinLightCount/pinSymbolOffsetWords + /// hardware bug already taught us; pinCount/pinLightCount/pinFrameOffsetBytes /// pin the multi-pin slice arithmetic (unit_RmtLedDriver_pins.cpp). Not part /// of any runtime API. - const uint32_t* symbolBuffer() const { return symbols_; } - /// Words allocated in the symbol buffer. Test-only. - size_t symbolCapacity() const { return symbolCap_; } + const uint8_t* frameBuffer() const { return frame_; } + /// Bytes allocated in the symbol buffer. Test-only. + size_t frameCapacity() const { return frameCap_; } /// Number of parsed output pins (0 = idle). Test-only. uint8_t pinCount() const { return pinCount_; } /// Lights on pin `i` (0 if out of range). Test-only. nrOfLightsType pinLightCount(uint8_t i) const { return i < pinCount_ ? pinCounts_[i] : 0; } - /// Word offset of pin `i`'s slice in the symbol buffer (0 if out of range). Test-only. - size_t pinSymbolOffsetWords(uint8_t i) const { return i < pinCount_ ? pinOffsets_[i] : 0; } + /// Byte offset of pin `i`'s slice in the symbol buffer (0 if out of range). Test-only. + size_t pinFrameOffsetBytes(uint8_t i) const { return i < pinCount_ ? pinOffsets_[i] : 0; } private: // Source frame. The output correction (channel order + white + brightness) lives on @@ -412,23 +411,17 @@ class RmtLedDriver : public DriverBase { platform::RmtWs2812Handle rmt_[kMaxPins]; uint16_t pinList_[kMaxPins] = {}; // parsed pins, list order nrOfLightsType pinCounts_[kMaxPins] = {}; // lights per pin (slice lengths) - size_t pinOffsets_[kMaxPins] = {}; // slice start in symbols_, words + size_t pinOffsets_[kMaxPins] = {}; // slice start in frame_, bytes nrOfLightsType txLightCount_ = 0; // Σ pinCounts_ — lights actually transmitted/encoded nrOfLightsType winStart_ = 0; // first source-buffer light this driver reads (the window) nrOfLightsType winLen_ = 0; // window length (lights), clamped to the buffer uint8_t pinCount_ = 0; // 0 = idle (parse error / no pins) bool inited_ = false; // all-or-nothing across the pins bool started_[kMaxPins] = {}; // which pins have a transmit still to be waited on - bool txInFlight_ = false; // a frame is still clocking out of symbols_ - uint32_t* symbols_ = nullptr; // owned; one word per WS2812 data bit - size_t symbolCap_ = 0; // words allocated - // Per-light scratch for correction_.apply(): `outChannels` bytes, one light at a time. Heap, sized - // to the channel count (no fixed cap — a light may carry any number of channels, RGB=3, RGBW=4, - // RGBCCT=5, or an N-channel fixture; the only limit is memory). Allocated off the hot path in - // resizeSymbols(), reused every tick (tick() never allocates). A fixed stack array here overflowed - // for >4-channel corrections and corrupted the stack — the SE16 bootloop, 2026-07-13. - // (wire_ / wireCap_ live on DriverBase — the grow-only scratch lifecycle is shared with - // ParallelLedDriver; this driver sizes it to ONE light's outChannels.) + bool frameUnusable_ = false; // frame buffer missing, or in PSRAM the refill cannot read + bool txInFlight_ = false; // a frame is still clocking out of frame_ + uint8_t* frame_ = nullptr; // owned; the wire bytes for the whole frame (outChannels per light) + size_t frameCap_ = 0; // bytes allocated // The parse-error literal currently shown in the status slot (nullptr when // configErr_, failBuf_, kFailBufLen and the clearConfigErr/clearFailBuf/ @@ -446,14 +439,28 @@ class RmtLedDriver : public DriverBase { : kMaxPins; } - static size_t symbolsFor(nrOfLightsType lights, uint8_t channels) { - return static_cast(lights) * channels * 8; + static size_t frameBytesFor(nrOfLightsType lights, uint8_t channels) { + return static_cast(lights) * channels; } // Convert a ns duration to RMT ticks using the resolution the platform // granted. Falls back to the requested clock when not inited (host/desktop). - uint16_t nsToTicks(uint32_t ns) const MM_NONBLOCKING { - uint32_t hz = inited_ ? platform::rmtWs2812Resolution(rmt_[0]) : kResolutionHz; + /// Hand the peripheral the two symbols a data bit expands to. The bit expansion now happens on + /// the way out (the IDF bytes encoder, or the level-5 refill), so these shapes are the whole of + /// what the wire timing means: the `timing` control changes them live, between frames. + bool pushBitTiming(uint8_t i) { + const uint16_t t0h = nsToTicks(cfg_.t0h_ns, i); + const uint16_t t1h = nsToTicks(cfg_.t1h_ns, i); + const uint16_t period = nsToTicks(cfg_.period_ns, i); + return platform::rmtWs2812SetBitTiming(rmt_[i], + makeRmtSymbol(t0h, 1, static_cast(period - t0h), 0), + makeRmtSymbol(t1h, 1, static_cast(period - t1h), 0)); + } + + /// Ticks for `ns` on channel `i`. Every channel is inited at kResolutionHz today so they agree, + /// but the granted rate is the channel's, so a mismatch would not be papered over by channel 0. + uint16_t nsToTicks(uint32_t ns, uint8_t i = 0) const MM_NONBLOCKING { + uint32_t hz = platform::rmtWs2812Resolution(rmt_[i]); // 0 before init: fall through if (hz == 0) hz = kResolutionHz; return static_cast((static_cast(ns) * hz) / 1'000'000'000ull); } @@ -466,7 +473,7 @@ class RmtLedDriver : public DriverBase { // clears it. Off the hot path. /// Turn the `timing` selection into the wire timing the encoder reads. Called from /// parseConfig, so every path that rebuilds the driver picks it up: the numbers are read per - /// frame from cfg_, so a change takes effect on the next frame with no channel reinit (the RMT + /// frame from cfg_, so a change reaches the peripheral through pushBitTiming, which reinit() runs: `timing` is in affectsPrepare, so a change is a rebuild, not a per-frame read /// tick clock is unchanged, only how many ticks each bit lasts). void applyTiming() { switch (timing) { @@ -514,7 +521,7 @@ class RmtLedDriver : public DriverBase { txLightCount_ = 0; for (uint8_t i = 0; i < pinCount_; i++) { pinOffsets_[i] = off; - off += static_cast(pinCounts_[i]) * outCh * 8; + off += static_cast(pinCounts_[i]) * outCh; // BYTES: frame_ holds wire bytes txLightCount_ = static_cast(txLightCount_ + pinCounts_[i]); } clearConfigErr(); @@ -534,7 +541,7 @@ class RmtLedDriver : public DriverBase { // (Re)allocate the symbol buffer for the current source + correction. Off the // hot path. Grows only — keeps a big-enough existing allocation. - void resizeSymbols() { + void resizeFrame() { if (!sourceBuffer_) return; // Size for the lights this driver actually CLOCKS OUT, not the whole window. The window (start, // count) can be far larger than the pins encode: `ledsPerPin` (or fewer pins than the window has @@ -542,7 +549,7 @@ class RmtLedDriver : public DriverBase { // encodes that many (n = min(txLightCount_, winLen_) there). Sizing to the window instead made an // 8×8 strip on one pin (ledsPerPin 64) inside a 70×82 grid (count=all, window 5740) try to alloc // ~550 KB of symbols for lights it never encodes — the alloc failed on a small-heap classic ESP32, - // symbols_ stayed null, and tick() bailed → the strip went dark even though only 64 lights were + // frame_ stayed null, and tick() bailed, so the strip went dark even though only 64 lights were // wanted. Bound to txLightCount_ so the buffer matches the real output. Fall back to the window // when no pins are parsed yet (txLightCount_ == 0), so the buffer is ready before pins are set. nrOfLightsType winStart, win; @@ -550,40 +557,60 @@ class RmtLedDriver : public DriverBase { nrOfLightsType n = txLightCount_ > 0 ? txLightCount_ : win; if (n > win) n = win; // never exceed the window's own light count const uint8_t ch = correction_.outChannels; - if (n == 0 || ch == 0) return; + if (n == 0 || ch == 0) { frameUnusable_ = false; return; } // Per-light correction scratch: grow to `ch` bytes when the channel count grows (off the hot // path). Sized to outChannels so a >4-channel correction (RGBCCT, a fixture) can't overflow it. - ensureWire(ch); // DriverBase owns the grow-only allocate/free - const size_t need = symbolsFor(n, ch); - if (symbols_ && symbolCap_ >= need) return; - freeSymbols(); - // INTERNAL RAM, deliberately, on a chip that would otherwise put this in PSRAM. The RMT copy - // encoder runs inside the refill interrupt and reads these symbols straight into the - // peripheral's memory, so on a DMA-less classic ESP32 every refill is a read of this buffer - // under a 40-160 us deadline. From PSRAM that read goes through the 32 KB cache that WiFi - // and the render loop also churn, and one miss is a stall of microseconds: a late refill, - // a few wrong lights, at any light count and on any core. Bench: QuinLED Dig-Next-2 - // (PICO-V3-02, 2 MB PSRAM), 2026-09-05. The buffer is small (24 bytes x 4 per light: 24 KB - // at 256 lights) so internal RAM affords it; a board that cannot falls back to the general - // heap rather than to no output at all. - symbols_ = static_cast(platform::allocInternal(need * sizeof(uint32_t))); - if (!symbols_) symbols_ = static_cast(platform::alloc(need * sizeof(uint32_t))); - symbolCap_ = symbols_ ? need : 0; - publishHeapBytes(); // the symbol buffer grew — refresh the memory readout + const size_t need = frameBytesFor(n, ch); + if (frame_ && frameCap_ >= need) { frameUnusable_ = false; return; } + freeFrame(); + // INTERNAL RAM, deliberately. On the classic ESP32 the level-5 refill expands these bytes + // with the flash cache possibly off, where a PSRAM read is a fault rather than a stall; on + // the DMA chips the bytes encoder reads them under the same deadline pressure. At outCh + // bytes per light this is ~3 KB for 1024 RGB lights, so internal RAM affords it at any + // strand length a classic ESP32 can drive. (The pre-expanded symbol form this replaced + // wanted 96 bytes per light: 96 KB at 1024 lights, which internal RAM does NOT have, so it + // fell back to PSRAM and the transmit refused every frame. Issue #94.) + frame_ = static_cast(platform::allocInternal(need)); + if (!frame_) frame_ = static_cast(platform::alloc(need)); + frameCap_ = frame_ ? need : 0; + // A failed allocation still has to SAY so: tick()'s `!frame_` guard then skips the frame and + // the strip would otherwise sit frozen while the card reports "driving N of N" at a healthy + // fps, which is what made issue #94 so hard to place. + // The classic-ESP32 refill reads these bytes with the flash cache possibly off, so a + // PSRAM buffer is refused frame after frame by rmtWs2812Transmit while the card would + // otherwise report "driving N of N": the same silent freeze this change set out to remove. + // Far less reachable at 3 bytes per light than at the old 96, but the fallback still exists. + frameUnusable_ = !frame_ || platform::ptrIsPsram(frame_); + if (frameUnusable_) { + if (failBufEnsure()) { + std::snprintf(failBuf_, kFailBufLen, frame_ + ? "%u lights need %u KB of internal RAM" + : "out of memory for %u lights (%u KB)", + static_cast(n), + static_cast((need + 1023) / 1024)); + setStatus(failBuf_, Severity::Error); + } else { + setStatus(kNoFrameMemMsg, Severity::Error); + } + if (frame_) freeFrame(); // hand back memory the transmit will never read + } else if (status() == failBuf_ || status() == kNoFrameMemMsg) { + clearStatus(); // the count came back down (or the heap freed up): retract our error + } + publishHeapBytes(); // the frame buffer grew: refresh the memory readout } - void freeSymbols() { - if (symbols_) { platform::free(symbols_); symbols_ = nullptr; symbolCap_ = 0; publishHeapBytes(); } + void freeFrame() { + if (frame_) { platform::free(frame_); frame_ = nullptr; frameCap_ = 0; publishHeapBytes(); } } protected: // Matches DriverBase's visibility — a private override would silently hide the hook from any // future caller holding a DriverBase*. ParallelLedDriver keeps it protected for the same reason. - /// This driver's heap = the base scratch + the RMT symbol buffer (one word per WS2812 data bit, + /// This driver's heap = the base scratch + the RMT frame buffer (one byte per output channel per light, /// the driver's largest buffer). Summed for the per-module memory readout — see /// DriverBase::driverHeapBytes. size_t driverHeapBytes() const override { - return DriverBase::driverHeapBytes() + static_cast(symbolCap_) * sizeof(uint32_t); + return DriverBase::driverHeapBytes() + frameCap_; } private: @@ -678,6 +705,7 @@ class RmtLedDriver : public DriverBase { // --- RMT channels (hardware; RMT targets only) --- static constexpr const char* kInitFailMsg = "RMT init failed, check the pins"; + static constexpr const char* kNoFrameMemMsg = "out of memory for this many lights"; // All-or-nothing: a failing pin deinits everything and reports which pin, // so tick()'s guard stays a single bool and the user sees one clear error @@ -688,7 +716,10 @@ class RmtLedDriver : public DriverBase { if (pinCount_ == 0) return; // parse error — already in the status slot for (uint8_t i = 0; i < pinCount_; i++) { if (platform::rmtWs2812Init(rmt_[i], static_cast(pinList_[i]), - kResolutionHz, cfg_.invert)) continue; + kResolutionHz, cfg_.invert) + && pushBitTiming(i)) { // the expander needs the bit shapes before the first frame + continue; + } // Surface which pin failed instead of silently no-op'ing in tick() — // the status tells the user why output is dark (usually a bad pin), // rather than leaving them to wonder why nothing lights. @@ -704,13 +735,17 @@ class RmtLedDriver : public DriverBase { return; } inited_ = true; - // A prior init failure recovered (e.g. a pin fixed) — drop the stale error. - if (failBuf_ && status() == failBuf_) clearFailBuf(); + // A prior init failure recovered (e.g. a pin fixed): drop the stale error. NOT when + // resizeFrame left the symbol-buffer error there: reinit() runs right after it on every + // rebuild, and failBuf_ carries both messages, so an unguarded clear here retracted the + // "needs N KB internal RAM" error microseconds after it was set and the card fell back to + // a blank status while the strip stayed frozen (bench, 900 lights on a Dig-Next-2). + if (failBuf_ && status() == failBuf_ && !frameUnusable_) clearFailBuf(); if (status() == kInitFailMsg) clearStatus(); } // Releases only the RMT channels — NOT the symbol buffer (that's - // freeSymbols(), owned by release). reinit() calls this on every rebuild, + // freeFrame(), owned by release). reinit() calls this on every rebuild, // so freeing the buffer here would strand tick() — the original bug. void deinitAll() { if constexpr (platform::rmtTxChannels == 0) return; diff --git a/src/light/drivers/RmtSymbol.h b/src/light/drivers/RmtSymbol.h index 671f3766..3031cc26 100644 --- a/src/light/drivers/RmtSymbol.h +++ b/src/light/drivers/RmtSymbol.h @@ -5,9 +5,9 @@ namespace mm { -// Encode wire-ordered LED bytes into ESP32 RMT symbols — domain logic, no ESP -// header, so it is host-testable without an ESP32 (the platform owns only the -// peripheral that consumes these symbols; see platform.h rmtWs2812*). +// The ESP32 RMT bit-shape word: domain logic, no ESP header, so it is host-testable without an +// ESP32 (the platform owns only the peripheral that expands wire bytes with these shapes; see +// platform.h rmtWs2812SetBitTiming). // // RMT symbol layout (matches ESP-IDF's rmt_symbol_word_t, documented here so no // driver/rmt_*.h leaks into src/light/): one 32-bit word is two 16-bit halves, @@ -26,32 +26,4 @@ constexpr uint32_t makeRmtSymbol(uint16_t dur0, uint8_t lvl0, | (static_cast(lvl1 & 1) << 31); } -// Encode one light's already-wire-ordered bytes (`channels` of them — brightness, -// GRB reorder and any RGBW white have ALREADY been applied by Correction) into -// `channels * 8` RMT symbols at `out`, MSB-first within each byte. Each data bit -// becomes one symbol: HIGH for t1hTicks (a 1) or t0hTicks (a 0), then LOW for -// (periodTicks - that high time). Durations are in RMT ticks (the caller converts -// ns→ticks from the peripheral's granted resolution). `out` must hold at least -// channels*8 words. -// -// Header-only inline (light domain is header-only; see coding-standards.md). The -// host encoder test asserts this contract: GRB order via the corrected input, -// MSB-first, exact high/low ticks per bit. Implemented in Phase C. -inline void encodeWs2812Symbols(const uint8_t* wire, uint8_t channels, - uint16_t t0hTicks, uint16_t t1hTicks, - uint16_t periodTicks, uint32_t* out) { - const uint16_t t0Low = static_cast(periodTicks - t0hTicks); - const uint16_t t1Low = static_cast(periodTicks - t1hTicks); - // Pre-build the two possible symbols once; each data bit picks one. - const uint32_t sym0 = makeRmtSymbol(t0hTicks, 1, t0Low, 0); - const uint32_t sym1 = makeRmtSymbol(t1hTicks, 1, t1Low, 0); - size_t s = 0; - for (uint8_t ch = 0; ch < channels; ch++) { - const uint8_t byte = wire[ch]; - for (int bit = 7; bit >= 0; bit--) { // MSB-first within the byte - out[s++] = (byte & (1u << bit)) ? sym1 : sym0; - } - } -} - } // namespace mm diff --git a/src/platform/desktop/moonlive_asm_host.h b/src/platform/desktop/moonlive_asm_host.h index 51f5a602..c2e2627c 100644 --- a/src/platform/desktop/moonlive_asm_host.h +++ b/src/platform/desktop/moonlive_asm_host.h @@ -40,15 +40,27 @@ class HostAssembler { // Named here because each backend's Reg is its own enum, sized to its own file. using RegType = Reg; - // Owns buf_ (see below). Freed here, copying deleted — an emitter that was copied - // would double-free the buffer it emits into. - ~HostAssembler() { platform::free(buf_); } + // Frees buf_ only when this emitter OWNS it (the one-argument form below). The borrowing form + // emits straight into a buffer the caller already holds, so there is nothing to free. Copying + // deleted: a copied owner would double-free. + ~HostAssembler() { if (owned_) platform::free(buf_); } /// `cap` is the code buffer's size, chosen per SCRIPT by the caller (codeCapFor) rather - /// than a shared constant — the backends differ by up to 1.9x on identical source, so one + /// than a shared constant: the backends differ by up to 1.9x on identical source, so one /// number cannot fit them all. Defaults to the sanity bound for callers that emit a fixed - /// blob (emitFill) and have no token count to size from. + /// blob (emitFill) and have no token count to size from. This form ALLOCATES; the unit tests + /// that probe single instructions use it. explicit HostAssembler(size_t cap = kCodeCap) - : kCap(cap), buf_(static_cast(platform::alloc(cap))) {} + : kCap(cap), buf_(static_cast(platform::alloc(cap))), owned_(true) {} + /// Emit straight into `out`, the caller's staging buffer, rather than into a twin of it. The + /// lowering used to allocate a second full-size buffer here and memcpy into `out` at the end, + /// so every compile held TWO copies of its code cap on the heap. On a classic ESP32 whose + /// largest free block had fragmented to 24 KB that second allocation failed, and because the + /// failure surfaced through emit() as overflow_, the compile reported "codegen failed + /// (unsupported on this target, or too large)" for a script that compiles fine (the shipped + /// noise.mle, bench 2026-09-09). Borrowing removes that allocation and halves the compile's + /// transient heap; when `out` is null the emitter reports through overflowed() as before. + HostAssembler(uint8_t* out, size_t cap) + : kCap(cap), buf_(out), owned_(false) {} HostAssembler(const HostAssembler&) = delete; HostAssembler& operator=(const HostAssembler&) = delete; @@ -174,8 +186,10 @@ class HostAssembler { // member put 2 KB on the compile chain's stack — on top of the staging buffer and the parser // frames. On a classic ESP32 that overflowed the task and faulted inside _xt_context_save // (the plan named this: "buf_[kCap] inside the assembler, itself a stack local"). The buffer is - // scratch that ends in a memcpy to the caller's output, so nothing outlives the object. + // the caller's own staging buffer in the borrowing form (nothing to copy out), or a private + // one in the owning form; either way nothing outlives the object. uint8_t* buf_; + bool owned_; // true = we allocated buf_ and free it; false = the caller's buffer size_t len_ = 0; bool overflow_ = false; // Frame size in bytes, 0 when no prologue was emitted. arm64's epilogue reads it, so its diff --git a/src/platform/desktop/platform_desktop.cpp b/src/platform/desktop/platform_desktop.cpp index 7d8f0501..cc344552 100644 --- a/src/platform/desktop/platform_desktop.cpp +++ b/src/platform/desktop/platform_desktop.cpp @@ -201,18 +201,60 @@ uint32_t micros() MM_NONBLOCKING { #pragma clang diagnostic pop #endif +// WHAT THIS PROCESS HAS DELIBERATELY ALLOCATED, in bytes. Not a heap figure: a desktop has as much +// memory as it wants, and freeHeap() keeps reporting 0 because three call sites read that 0 as +// "unlimited" and switch off gates that only mean something on a device (polar.h's LUT budget, +// MappingLUT's paging fallback). +// +// What it IS good for is the DELTA. Every buffer the system takes on purpose (layer buffers, +// mapping LUTs, script arenas, driver rings) comes through alloc/allocInternal, so adding or +// removing a module moves this number by exactly what that module costs, on a laptop, in a second, +// with no board attached. The process's own RSS cannot answer that: the allocator, the JIT and the +// HTTP buffers move it too, and a 100 KB layer would be lost in the noise. +// +// The REQUESTED size is recorded rather than malloc's rounded one (malloc_size reports 1024 for a +// 1000-byte ask), so a reported delta is the number the caller asked for. +std::atomic g_allocatedBytes{0}; +std::atomic g_allocatedPeak{0}; +std::atomic g_allocCount{0}; + +namespace { +// Requested size, kept immediately before the block handed out. 16 bytes rather than 8 so the +// returned pointer keeps the alignment malloc promised for any type. +constexpr size_t kAllocHeader = 16; + +void* trackedAlloc(size_t bytes) { + void* raw = std::malloc(bytes + kAllocHeader); + if (!raw) return nullptr; + *static_cast(raw) = bytes; + const size_t now = g_allocatedBytes.fetch_add(bytes, std::memory_order_relaxed) + bytes; + // Peak is advisory, so a lost race between two threads costs a slightly low high-water mark + // rather than anything a caller depends on. + if (now > g_allocatedPeak.load(std::memory_order_relaxed)) + g_allocatedPeak.store(now, std::memory_order_relaxed); + g_allocCount.fetch_add(1, std::memory_order_relaxed); + return static_cast(raw) + kAllocHeader; +} +} // namespace + void* alloc(size_t bytes) { - return std::malloc(bytes); + return trackedAlloc(bytes); } bool ptrIsPsram(const void* /*p*/) { return false; } // desktop has no PSRAM void* allocInternal(size_t bytes) { - return std::malloc(bytes); // desktop has one flat RAM — internal == ordinary + return trackedAlloc(bytes); // desktop has one flat RAM: internal == ordinary } void free(void* ptr) { - std::free(ptr); + if (!ptr) return; + void* raw = static_cast(ptr) - kAllocHeader; + g_allocatedBytes.fetch_sub(*static_cast(raw), std::memory_order_relaxed); + // Decremented, so the count is LIVE blocks and not allocations-ever. Without this it only + // climbed, which reads as a leak on any device left running. + g_allocCount.fetch_sub(1, std::memory_order_relaxed); + std::free(raw); } // Executable memory for MoonLive's emitted code. macOS on Apple Silicon enforces W^X @@ -302,6 +344,10 @@ void pauseLoop() { lastWake = std::chrono::steady_clock::now(); } +size_t allocatedBytes() { return g_allocatedBytes.load(std::memory_order_relaxed); } +size_t allocatedPeak() { return g_allocatedPeak.load(std::memory_order_relaxed); } +uint32_t allocatedCount() { return g_allocCount.load(std::memory_order_relaxed); } + size_t freeHeap() { return 0; // Not meaningful on desktop (0 = unlimited) } @@ -325,6 +371,10 @@ size_t maxInternalAllocBlock() { return 0; // Not meaningful on desktop (0 = unlimited) } +size_t maxExecAllocBlock() { + return 0; // no distinct executable pool: pages are mapped per allocation +} + // No RTOS on desktop — the TasksModule shows only its MoonModule cost table here. // Test seam: a unit test can inject a canned task snapshot + render-task name so TasksModule's // row/detail JSON + the nesting predicate are exercised on the host (no RTOS here otherwise). Empty @@ -1384,7 +1434,7 @@ int wifiStaRssi() { return 0; } void wifiStaBssid(uint8_t out[6]) { std::memset(out, 0, 6); } int wifiStaChannel() { return 0; } -bool wifiApInit(const char* /*apName*/, const char* /*ip*/) { return false; } +bool wifiApInit(const char* /*apName*/, const char* /*ip*/) { return false; } // no AP on a host bool wifiApConnected() { return false; } void wifiApStop() {} uint32_t wifiApClientCount() { return 0; } @@ -1955,11 +2005,14 @@ bool rmtWs2812Init(RmtWs2812Handle& h, uint8_t /*gpio*/, uint32_t resolutionHz, uint32_t rmtWs2812Resolution(const RmtWs2812Handle& h) MM_NONBLOCKING { return h.impl ? static_cast(h.impl)->resolutionHz : 0; } -bool rmtWs2812Transmit(RmtWs2812Handle& h, const uint32_t* symbols, - size_t symbolCount) { - if (!h.impl || !symbols || symbolCount == 0) return false; +bool rmtWs2812Transmit(RmtWs2812Handle& h, const uint8_t* wire, size_t byteCount) { + if (!h.impl || !wire || byteCount == 0) return false; return true; } + +bool rmtWs2812SetBitTiming(RmtWs2812Handle& h, uint32_t /*sym0*/, uint32_t /*sym1*/) { + return h.impl != nullptr; // no peripheral to program off-target +} bool rmtWs2812Wait(RmtWs2812Handle& /*h*/, uint32_t /*timeoutMs*/) { return true; } void rmtWs2812Deinit(RmtWs2812Handle& h) { delete static_cast(h.impl); diff --git a/src/platform/esp32/moonlive_asm_riscv.h b/src/platform/esp32/moonlive_asm_riscv.h index 63c5d17e..30214c63 100644 --- a/src/platform/esp32/moonlive_asm_riscv.h +++ b/src/platform/esp32/moonlive_asm_riscv.h @@ -38,15 +38,27 @@ class RiscvAssembler { // Named here because each backend's Reg is its own enum, sized to its own file. using RegType = Reg; - // Owns buf_ (see below). Freed here, copying deleted — an emitter that was copied - // would double-free the buffer it emits into. - ~RiscvAssembler() { platform::free(buf_); } + // Frees buf_ only when this emitter OWNS it (the one-argument form below). The borrowing form + // emits straight into a buffer the caller already holds, so there is nothing to free. Copying + // deleted: a copied owner would double-free. + ~RiscvAssembler() { if (owned_) platform::free(buf_); } /// `cap` is the code buffer's size, chosen per SCRIPT by the caller (codeCapFor) rather - /// than a shared constant — the backends differ by up to 1.9x on identical source, so one + /// than a shared constant: the backends differ by up to 1.9x on identical source, so one /// number cannot fit them all. Defaults to the sanity bound for callers that emit a fixed - /// blob (emitFill) and have no token count to size from. + /// blob (emitFill) and have no token count to size from. This form ALLOCATES; the unit tests + /// that probe single instructions use it. explicit RiscvAssembler(size_t cap = kCodeCap) - : kCap(cap), buf_(static_cast(platform::alloc(cap))) {} + : kCap(cap), buf_(static_cast(platform::alloc(cap))), owned_(true) {} + /// Emit straight into `out`, the caller's staging buffer, rather than into a twin of it. The + /// lowering used to allocate a second full-size buffer here and memcpy into `out` at the end, + /// so every compile held TWO copies of its code cap on the heap. On a classic ESP32 whose + /// largest free block had fragmented to 24 KB that second allocation failed, and because the + /// failure surfaced through emit() as overflow_, the compile reported "codegen failed + /// (unsupported on this target, or too large)" for a script that compiles fine (the shipped + /// noise.mle, bench 2026-09-09). Borrowing removes that allocation and halves the compile's + /// transient heap; when `out` is null the emitter reports through overflowed() as before. + RiscvAssembler(uint8_t* out, size_t cap) + : kCap(cap), buf_(out), owned_(false) {} RiscvAssembler(const RiscvAssembler&) = delete; RiscvAssembler& operator=(const RiscvAssembler&) = delete; @@ -135,8 +147,10 @@ class RiscvAssembler { // member put 2 KB on the compile chain's stack — on top of the staging buffer and the parser // frames. On a classic ESP32 that overflowed the task and faulted inside _xt_context_save // (the plan named this: "buf_[kCap] inside the assembler, itself a stack local"). The buffer is - // scratch that ends in a memcpy to the caller's output, so nothing outlives the object. + // the caller's own staging buffer in the borrowing form (nothing to copy out), or a private + // one in the owning form; either way nothing outlives the object. uint8_t* buf_; + bool owned_; // true = we allocated buf_ and free it; false = the caller's buffer size_t len_ = 0; bool overflow_ = false; // Frame size in bytes, 0 when no prologue was emitted. epilogue() reads it, so the teardown can diff --git a/src/platform/esp32/moonlive_asm_xtensa.h b/src/platform/esp32/moonlive_asm_xtensa.h index fbbdc68c..1a2cbcec 100644 --- a/src/platform/esp32/moonlive_asm_xtensa.h +++ b/src/platform/esp32/moonlive_asm_xtensa.h @@ -38,15 +38,27 @@ class XtensaAssembler { // Named here because each backend's Reg is its own enum, sized to its own file. using RegType = Reg; - // Owns buf_ (see below). Freed here, copying deleted — an emitter that was copied - // would double-free the buffer it emits into. - ~XtensaAssembler() { platform::free(buf_); } + // Frees buf_ only when this emitter OWNS it (the one-argument form below). The borrowing form + // emits straight into a buffer the caller already holds, so there is nothing to free. Copying + // deleted: a copied owner would double-free. + ~XtensaAssembler() { if (owned_) platform::free(buf_); } /// `cap` is the code buffer's size, chosen per SCRIPT by the caller (codeCapFor) rather - /// than a shared constant — the backends differ by up to 1.9x on identical source, so one + /// than a shared constant: the backends differ by up to 1.9x on identical source, so one /// number cannot fit them all. Defaults to the sanity bound for callers that emit a fixed - /// blob (emitFill) and have no token count to size from. + /// blob (emitFill) and have no token count to size from. This form ALLOCATES; the unit tests + /// that probe single instructions use it. explicit XtensaAssembler(size_t cap = kCodeCap) - : kCap(cap), buf_(static_cast(platform::alloc(cap))) {} + : kCap(cap), buf_(static_cast(platform::alloc(cap))), owned_(true) {} + /// Emit straight into `out`, the caller's staging buffer, rather than into a twin of it. The + /// lowering used to allocate a second full-size buffer here and memcpy into `out` at the end, + /// so every compile held TWO copies of its code cap on the heap. On a classic ESP32 whose + /// largest free block had fragmented to 24 KB that second allocation failed, and because the + /// failure surfaced through emit() as overflow_, the compile reported "codegen failed + /// (unsupported on this target, or too large)" for a script that compiles fine (the shipped + /// noise.mle, bench 2026-09-09). Borrowing removes that allocation and halves the compile's + /// transient heap; when `out` is null the emitter reports through overflowed() as before. + XtensaAssembler(uint8_t* out, size_t cap) + : kCap(cap), buf_(out), owned_(false) {} XtensaAssembler(const XtensaAssembler&) = delete; XtensaAssembler& operator=(const XtensaAssembler&) = delete; @@ -153,8 +165,10 @@ class XtensaAssembler { // member put 2 KB on the compile chain's stack — on top of the staging buffer and the parser // frames. On a classic ESP32 that overflowed the task and faulted inside _xt_context_save // (the plan named this: "buf_[kCap] inside the assembler, itself a stack local"). The buffer is - // scratch that ends in a memcpy to the caller's output, so nothing outlives the object. + // the caller's own staging buffer in the borrowing form (nothing to copy out), or a private + // one in the owning form; either way nothing outlives the object. uint8_t* buf_; + bool owned_; // true = we allocated buf_ and free it; false = the caller's buffer size_t len_ = 0; bool overflow_ = false; diff --git a/src/platform/esp32/platform_config.h b/src/platform/esp32/platform_config.h index d3687b2c..fc5a1fa8 100644 --- a/src/platform/esp32/platform_config.h +++ b/src/platform/esp32/platform_config.h @@ -430,7 +430,7 @@ constexpr bool hasOta = true; // Improv-serial is the device's serial RPC channel (UART0 + native USB-Serial-JTAG): // the WiFi-provisioning RPCs (WIFI_SETTINGS, GET_WIFI_NETWORKS) AND the vendor RPCs -// (SET_DEVICE_MODEL, SET_TX_POWER, APPLY_OP: "Improv = REST over serial"). The +// (SET_TX_POWER, APPLY_OP: "Improv = REST over serial"). The // transport is always available on ESP32, so the listener runs everywhere: including // Ethernet-only builds (`--firmware esp32-eth*`), where the WiFi-only RPCs are compiled // out (the `esp_wifi_*` calls aren't linked) but the vendor RPCs still work, so the web diff --git a/src/platform/esp32/platform_esp32.cpp b/src/platform/esp32/platform_esp32.cpp index bc286606..cc3ec993 100644 --- a/src/platform/esp32/platform_esp32.cpp +++ b/src/platform/esp32/platform_esp32.cpp @@ -216,6 +216,28 @@ void reboot() { esp_restart(); } +// The same three numbers the desktop counts by hand, from the allocator that already tracks them. +// A scenario reads one metric on both platforms: how much the system has taken, its high-water +// mark, and how many blocks are live. USED rather than free, so the figure means the same thing on +// a board with 320 KB and a laptop with gigabytes. +size_t allocatedBytes() { + multi_heap_info_t info = {}; + heap_caps_get_info(&info, MALLOC_CAP_8BIT); + return info.total_allocated_bytes; +} +size_t allocatedPeak() { + // The minimum-ever free, expressed as a peak used: IDF tracks the low-water mark of free heap, + // which is the same fact from the other side. + const size_t total = heap_caps_get_total_size(MALLOC_CAP_8BIT); + const size_t minFree = heap_caps_get_minimum_free_size(MALLOC_CAP_8BIT); + return total > minFree ? total - minFree : 0; +} +uint32_t allocatedCount() { + multi_heap_info_t info = {}; + heap_caps_get_info(&info, MALLOC_CAP_8BIT); + return static_cast(info.allocated_blocks); +} + size_t freeHeap() { return heap_caps_get_free_size(MALLOC_CAP_8BIT); } @@ -245,6 +267,14 @@ size_t maxInternalAllocBlock() { return heap_caps_get_largest_free_block(MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT); } +size_t maxExecAllocBlock() { + // The pool allocExec draws from. On a classic ESP32 that is IRAM, tens of KB rather than the + // hundreds the data heap has, and it is not visible in freeInternalHeap: a script whose compiled + // code does not fit reported "codegen failed" with 80 KB of DRAM free, which is what made this + // worth reporting rather than inferring. + return heap_caps_get_largest_free_block(MALLOC_CAP_EXEC | MALLOC_CAP_32BIT); +} + size_t totalHeap() { return heap_caps_get_total_size(MALLOC_CAP_8BIT); } diff --git a/src/platform/esp32/platform_esp32_rmt.cpp b/src/platform/esp32/platform_esp32_rmt.cpp index bb415a8a..66d516bb 100644 --- a/src/platform/esp32/platform_esp32_rmt.cpp +++ b/src/platform/esp32/platform_esp32_rmt.cpp @@ -1,15 +1,15 @@ -// RMT WS2812 LED output — the peripheral half of the LED driver. +// RMT WS2812 LED output: the peripheral half of the LED driver. // -// The driver (src/light/drivers/RmtLedDriver.h) does all the domain work: -// applies Correction and encodes each pixel into RMT symbols. This file owns -// only the peripheral — channel setup, the copy-encoder that streams the -// pre-built symbols, transmit + wait, and the RX side the on-device loopback -// test uses. No domain logic here. +// The driver (src/light/drivers/RmtLedDriver.h) does all the domain work: it applies Correction and +// hands us the WIRE BYTES, the finished per-channel values the strip expects. This file owns only +// the peripheral: channel setup, the bit expansion, transmit + wait, and the RX side the on-device +// loopback test uses. No domain logic here. // -// Pre-encoded path: the driver hands us a flat array of WS2812 symbols already -// in rmt_symbol_word_t layout (our makeRmtSymbol() in RmtSymbol.h packs exactly -// that 32-bit format), so the TX path uses a *copy* encoder — it just DMAs the -// bytes out, no per-call symbol generation. +// Wire-byte path: each byte becomes eight symbols on the way to the peripheral, MSB-first, using +// the two bit shapes rmtWs2812SetBitTiming programs (makeRmtSymbol in RmtSymbol.h packs that 32-bit +// format). The IDF bytes encoder does it where RMT has DMA; the classic ESP32's level-5 refill does +// it inline in rmtHiFill. So the caller keeps 3-4 bytes per light rather than 32 bytes per byte of +// it: a long strand no longer outgrows the internal RAM the refill is restricted to. #include "platform/platform.h" @@ -54,7 +54,7 @@ // critical section masks, and that is what let a refill arrive late (see rmt_hi_vector.S). The // RMT interrupt source on core 1 is rerouted to vector 26 (level 5, refused by esp_intr_alloc as // "special", so routed by hand), and this code plays each frame ping-pong out of the channel's -// memory, one half-block per threshold interrupt, straight from the driver's symbol buffer. +// memory, one half-block per threshold interrupt, straight from the driver's frame buffer. // // At file scope, outside every namespace: the assembly bridge calls rmtHiIsr by its C name, and // RMTMEM is the linker's symbol, so both need external C linkage, which an anonymous namespace @@ -66,8 +66,10 @@ // opens on both cores. // --------------------------------------------------------------------------------------------- struct RmtHiChannel { - const uint32_t* cur = nullptr; // next symbol to copy in - const uint32_t* end = nullptr; // one past the last + const uint8_t* cur = nullptr; // next WIRE BYTE to expand + const uint8_t* end = nullptr; // one past the last + uint32_t sym0 = 0; // the symbol a 0 bit expands to + uint32_t sym1 = 0; // ... and a 1 bit uint16_t half = 0; // symbols per half-block (the threshold) uint16_t offset = 0; // where the next half goes: 0 or `half` volatile bool busy = false; // a frame is on the wire @@ -88,7 +90,20 @@ static void IRAM_ATTR rmtHiFill(uint8_t ch) { RmtHiChannel& c = s_hi[ch]; volatile uint32_t* dst = &RMTMEM.chan[ch].data32[c.offset]; uint32_t n = c.half; - while (n && c.cur != c.end) { *dst++ = *c.cur++; n--; } + // Expand WIRE BYTES to symbols here, MSB-first, rather than copying symbols a caller + // pre-expanded. Eight symbols per byte, so the resident buffer is the 3-4 bytes per light the + // correction already produces instead of 8 words (32 bytes) per byte of it: 3 KB for 1024 + // lights where the pre-expanded form wanted 96 KB. That buffer has to be internal RAM (this + // runs with the flash cache possibly off), and 96 KB of internal RAM is what a classic ESP32 + // does not have, so above ~800 lights the pre-expanded form fell back to PSRAM and the + // transmit refused every frame: issue #94's frozen strip. The work per half-block is a shift + // and a select per bit, well inside the ~40 us deadline. + const uint32_t s0 = c.sym0, s1 = c.sym1; + while (n >= 8 && c.cur != c.end) { + uint8_t data = *c.cur++; + for (uint8_t bit = 0; bit < 8; bit++) { *dst++ = (data & 0x80u) ? s1 : s0; data = static_cast(data << 1); } + n -= 8; + } if (n) *dst = 0; // end marker inside this half c.offset = static_cast(c.offset ? 0 : c.half); } @@ -161,6 +176,9 @@ struct RmtTxState { rmt_channel_handle_t channel = nullptr; rmt_encoder_handle_t encoder = nullptr; uint32_t resolutionHz = 0; + bool timed = false; // rmtWs2812SetBitTiming succeeded: a transmit before it would clock all-zero symbols + uint32_t sym0 = 0, sym1 = 0; // bit shapes, set live by rmtWs2812SetBitTiming (every chip: + // the bytes encoder takes them, and so does the level-5 refill) #if CONFIG_IDF_TARGET_ESP32 uint8_t channelId = 0xFF; // the peripheral channel the IDF gave us, read back from the GPIO matrix uint16_t blockSymbols = 0; // symbols the channel's memory holds (64 per block) @@ -216,8 +234,15 @@ void rmtInitOnThisCore(void* arg) { txCfg.mem_block_symbols = SOC_RMT_MEM_WORDS_PER_CHANNEL; if (rmt_new_tx_channel(&txCfg, &st->channel) != ESP_OK) { job->ok = false; return; } - rmt_copy_encoder_config_t copyCfg = {}; - if (rmt_new_copy_encoder(©Cfg, &st->encoder) != ESP_OK) { + // A BYTES encoder, the IDF's own WS2812-shaped one: it expands each wire byte to eight symbols + // as it feeds the peripheral, so the caller keeps only the 3-4 bytes per light the correction + // produces. The copy encoder this replaces required the caller to pre-expand every bit into a + // 32-bit symbol first, 96 bytes per RGB light, which is the buffer that outgrew internal RAM. + // The bit timings are placeholders: the driver's `timing` control is live, so + // rmtWs2812SetBitTiming rewrites them (rmt_bytes_encoder_update_config) before each frame. + rmt_bytes_encoder_config_t bytesCfg = {}; + bytesCfg.flags.msb_first = 1; // WS2812 clocks the most significant bit first + if (rmt_new_bytes_encoder(&bytesCfg, &st->encoder) != ESP_OK) { rmt_del_channel(st->channel); st->channel = nullptr; job->ok = false; return; } @@ -272,20 +297,44 @@ uint32_t rmtWs2812Resolution(const RmtWs2812Handle& h) MM_NONBLOCKING { return st ? st->resolutionHz : 0; } -bool rmtWs2812Transmit(RmtWs2812Handle& h, const uint32_t* symbols, size_t symbolCount) { +bool rmtWs2812SetBitTiming(RmtWs2812Handle& h, uint32_t sym0, uint32_t sym1) { auto* st = static_cast(h.impl); - if (!st || !symbols || symbolCount == 0) return false; + if (!st || !st->encoder) return false; + // The `timing` control is live (400 kHz WS2811, 800 kHz, custom ns), so the encoder's bit + // shapes are rewritten rather than fixed at init. + rmt_bytes_encoder_config_t cfg = {}; + static_assert(sizeof(rmt_symbol_word_t) == sizeof(uint32_t), "symbol word is one 32-bit word"); + std::memcpy(&cfg.bit0, &sym0, sizeof(uint32_t)); + std::memcpy(&cfg.bit1, &sym1, sizeof(uint32_t)); + cfg.flags.msb_first = 1; + // Commit the shapes only once the encoder took them, so a transmit can never run on stale or + // zero symbols: `timed` is what rmtWs2812Transmit checks. + if (rmt_bytes_encoder_update_config(st->encoder, &cfg) != ESP_OK) return false; + st->sym0 = sym0; st->sym1 = sym1; st->timed = true; + return true; +} + +bool rmtWs2812Transmit(RmtWs2812Handle& h, const uint8_t* wire, size_t byteCount) { + auto* st = static_cast(h.impl); + if (!st || !wire || byteCount == 0 || !st->timed) return false; // no bit shapes yet: refuse, not garbage #if CONFIG_IDF_TARGET_ESP32 if (st->channelId != 0xFF) { - // The level-5 path. The symbol buffer must be internal RAM: the refill runs with the - // flash cache possibly off, and a PSRAM read there is a fault, not a stall. The driver - // allocates internal-first; this is the guard for the fallback case. - if (!esp_ptr_internal(symbols)) return false; + // The level-5 path expands the bytes itself (rmtHiFill). Those bytes must be internal RAM: + // the refill runs with the flash cache possibly off, where a PSRAM read is a fault rather + // than a stall. At 3-4 bytes per light that is a few KB even for a long strand, so unlike + // the pre-expanded symbol form this does not outgrow internal RAM: issue #94. + if (!esp_ptr_internal(wire)) return false; RmtHiChannel& c = s_hi[st->channelId]; if (c.busy) return false; const uint8_t ch = st->channelId; - c.cur = symbols; c.end = symbols + symbolCount; + c.cur = wire; c.end = wire + byteCount; + c.sym0 = st->sym0; c.sym1 = st->sym1; + // The expander consumes whole BYTES (8 symbols each), so a half-block that is not a + // multiple of 8 would leave 1..7 symbols unfilled and end the frame early with no + // diagnostic. True for every chip today; asserted so a mem_block_symbols change says so. + static_assert(SOC_RMT_MEM_WORDS_PER_CHANNEL % 16 == 0, + "half-block must be a multiple of 8 symbols: rmtHiFill expands whole bytes"); c.half = st->blockSymbols / 2; c.offset = 0; c.busy = true; rmt_ll_tx_reset_pointer(&RMT, ch); @@ -300,13 +349,11 @@ bool rmtWs2812Transmit(RmtWs2812Handle& h, const uint32_t* symbols, size_t symbo rmt_transmit_config_t txCfg = {}; txCfg.loop_count = 0; // single shot, no hardware loop - // Our symbols are already rmt_symbol_word_t-shaped; the copy encoder takes a - // byte size. This only *starts* the transfer — channels started back-to-back - // clock out concurrently, which is what makes a multi-pin frame cost the - // longest strand instead of the sum. The caller pairs this with - // rmtWs2812Wait and owns the inter-frame latch after the last wait. - return rmt_transmit(st->channel, st->encoder, symbols, - symbolCount * sizeof(uint32_t), &txCfg) == ESP_OK; + // The bytes encoder expands each byte to eight symbols as it feeds the peripheral, so the wire + // bytes go straight out. This only *starts* the transfer: channels started back-to-back clock + // out concurrently, which is what makes a multi-pin frame cost the longest strand instead of + // the sum. The caller pairs this with rmtWs2812Wait and owns the inter-frame latch. + return rmt_transmit(st->channel, st->encoder, wire, byteCount, &txCfg) == ESP_OK; } bool rmtWs2812Wait(RmtWs2812Handle& h, uint32_t timeoutMs) { @@ -321,12 +368,12 @@ bool rmtWs2812Wait(RmtWs2812Handle& h, uint32_t timeoutMs) { // classic ESP32, rmt_disable() while a transmission is still active triggers an // interrupt-WDT panic (espressif/esp-idf#17692, classic-only — S3/C6/P4 are // unaffected). A panic is a worse failure than a dropped frame, so we leave the - // stuck transfer alone. It self-heals safely: the next tick re-encodes symbols_ + // stuck transfer alone. It self-heals safely: the next tick re-encodes the frame buffer // and calls rmt_transmit again; if the channel is still busy, rmt_transmit // returns an error, rmtWs2812Transmit returns false, and RmtLedDriver::tick() // skips waiting on that channel (its started[] guard) — no crash, no corruption. // The RESULT is what the caller needs: a timeout leaves the frame in flight, and re-encoding - // into symbols_ next tick would rewrite bytes the peripheral is still clocking out. That is a + // into the frame buffer next tick would rewrite bytes the peripheral is still clocking out. That is a // silent corruption rather than a dropped frame, and it shows on the strip as a few lights in // the wrong color, independent of light count. #if CONFIG_IDF_TARGET_ESP32 @@ -334,9 +381,19 @@ bool rmtWs2812Wait(RmtWs2812Handle& h, uint32_t timeoutMs) { // TX_DONE clears `busy` from the level-5 handler. Polled with a yield, not a semaphore: // the handler runs where no RTOS call is allowed, so it cannot signal one. const int64_t deadline = esp_timer_get_time() + static_cast(timeoutMs) * 1000; + // SPIN first, yield only if the frame is genuinely long. `vTaskDelay(1)` sleeps a whole + // scheduler tick, 10 ms at CONFIG_FREERTOS_HZ=100, so a frame that clocks out in 240 us + // still cost 10 ms: the tick measured a FLAT ~9,600 us on a Dig-Octa whether it drove 8 + // lights or 256, on one lane or eight, which is the scheduler and not the wire (bench + // 2026-09-09). A WS2812 frame is bounded and short (1.25 us per bit: 1.9 ms for 64 lights, + // 7.7 ms for 256), so busy-waiting to about one tick and only then sleeping keeps the CPU + // for the case that is over in microseconds while still yielding on a long strand rather + // than burning a core. + const int64_t spinUntil = esp_timer_get_time() + 10000; // ~1 scheduler tick while (s_hi[st->channelId].busy) { - if (esp_timer_get_time() > deadline) return false; - vTaskDelay(1); + const int64_t now = esp_timer_get_time(); + if (now > deadline) return false; + if (now > spinUntil) vTaskDelay(1); // long frame: hand the core back } return true; } @@ -703,14 +760,11 @@ RmtLoopbackResult rmtWs2812Loopback(uint8_t txGpio, uint8_t rxGpio) { const uint32_t sym1 = static_cast(kT1H) | (1u << 15) | (static_cast(kPeriod - kT1H) << 16); constexpr size_t kBits = 24; - uint32_t txSymbols[kBits]; - size_t s = 0; - for (int b = 0; b < 3; b++) - for (int bit = 7; bit >= 0; bit--) - txSymbols[s++] = (r.sent[b] & (1u << bit)) ? sym1 : sym0; + const uint8_t txWire[3] = { r.sent[0], r.sent[1], r.sent[2] }; RmtWs2812Handle tx; if (!rmtWs2812Init(tx, txGpio, kLoopbackResHz, /*invert=*/false)) return r; + rmtWs2812SetBitTiming(tx, sym0, sym1); // RX must be listening while we transmit; run the (blocking) capture in a task // and resend the short frame until the receiver latches one or we give up. @@ -727,7 +781,7 @@ RmtLoopbackResult rmtWs2812Loopback(uint8_t txGpio, uint8_t rxGpio) { if (xTaskCreate(rxTask, "rmtlb", 4096, &cap, 5, nullptr) == pdPASS) { vTaskDelay(pdMS_TO_TICKS(50)); for (int i = 0; i < 50 && !cap.done; i++) { - rmtWs2812Transmit(tx, txSymbols, kBits); + rmtWs2812Transmit(tx, txWire, sizeof(txWire)); rmtWs2812Wait(tx, 1000); ets_delay_us(300); // inter-frame latch vTaskDelay(pdMS_TO_TICKS(10)); @@ -782,33 +836,31 @@ RmtLoopbackResult rmtWs2812LoopbackFrame(uint8_t txGpio, uint8_t rxGpio, #endif const size_t kBits = static_cast(lights) * bitsPerLight; - // One real frame's worth of symbols, DMA-capable internal RAM (the same - // place the driver's own symbol buffer lives). Off the hot path — this is - // a control-driven self-test. - auto* txSymbols = static_cast(heap_caps_malloc( - kBits * sizeof(uint32_t), MALLOC_CAP_DMA | MALLOC_CAP_INTERNAL)); + // One real frame's worth of WIRE BYTES, DMA-capable internal RAM (the same place the driver's + // own frame buffer lives). Off the hot path: a control-driven self-test. + const size_t txBytes = static_cast(lights) * channels; + auto* txWire = static_cast(heap_caps_malloc( + txBytes, MALLOC_CAP_DMA | MALLOC_CAP_INTERNAL)); const size_t capMax = kBits + 16; auto* rxSymbols = static_cast(heap_caps_aligned_alloc( 64, capMax * sizeof(uint32_t), MALLOC_CAP_DMA | MALLOC_CAP_INTERNAL)); - if (!txSymbols || !rxSymbols) { - heap_caps_free(txSymbols); + if (!txWire || !rxSymbols) { + heap_caps_free(txWire); heap_caps_free(rxSymbols); return r; } size_t s = 0; for (uint16_t light = 0; light < lights; light++) - for (uint8_t ch = 0; ch < channels; ch++) { - const uint8_t byte = ch < 3 ? r.sent[ch] : 0x00; - for (int bit = 7; bit >= 0; bit--) - txSymbols[s++] = (byte & (1u << bit)) ? sym1 : sym0; - } + for (uint8_t ch = 0; ch < channels; ch++) + txWire[s++] = ch < 3 ? r.sent[ch] : 0x00; RmtWs2812Handle tx; if (!rmtWs2812Init(tx, txGpio, kLoopbackResHz, /*invert=*/false)) { - heap_caps_free(txSymbols); + heap_caps_free(txWire); heap_caps_free(rxSymbols); return r; } + rmtWs2812SetBitTiming(tx, sym0, sym1); struct Cap { uint8_t rxGpio; uint32_t* buf; size_t max; @@ -825,7 +877,7 @@ RmtLoopbackResult rmtWs2812LoopbackFrame(uint8_t txGpio, uint8_t rxGpio, // Back-to-back frames, the render loop's cadence. The capture latches // one whole frame; we keep resending so it can't miss the window. for (int i = 0; i < 100 && !cap.done; i++) { - rmtWs2812Transmit(tx, txSymbols, kBits); + rmtWs2812Transmit(tx, txWire, txBytes); rmtWs2812Wait(tx, 1000); ets_delay_us(300); // inter-frame WS2812 latch } @@ -856,7 +908,7 @@ RmtLoopbackResult rmtWs2812LoopbackFrame(uint8_t txGpio, uint8_t rxGpio, r.got[b / 8] = static_cast((r.got[b / 8] << 1) | bit); } } - heap_caps_free(txSymbols); + heap_caps_free(txWire); heap_caps_free(rxSymbols); return r; } diff --git a/src/platform/platform.h b/src/platform/platform.h index 8eac7f09..4eb93dbe 100644 --- a/src/platform/platform.h +++ b/src/platform/platform.h @@ -74,6 +74,19 @@ void free(void* ptr); // Free with the ordinary free(). void* allocInternal(size_t bytes); +/// How many bytes this process has deliberately allocated through alloc/allocInternal, and the +/// high-water mark. NOT a heap figure: freeHeap() stays 0 on desktop because callers read that as +/// "unlimited" and switch off gates that only mean something on a device. +/// +/// The point is the DELTA. Every buffer the system takes on purpose comes through this seam, so +/// adding or removing a module moves these by exactly that module's cost, measurable on a laptop +/// without a board. On ESP32 they report the same thing from the real heap, so a scenario reads one +/// number on both. `count` is the number of live blocks, which separates "one buffer got bigger" +/// from "something is allocating per frame". +size_t allocatedBytes(); +size_t allocatedPeak(); +uint32_t allocatedCount(); + // True when the pointer resolves to external (PSRAM) memory: the standard residency probe (IDF's // esp_ptr_external_ram). Diagnostic companion to allocInternal's internal-first-PSRAM-fallback pattern: // the caller of that pattern cannot otherwise tell which way an allocation landed, and for buffers an @@ -130,6 +143,12 @@ size_t freeInternalHeap(); // internal RAM only (for stack/HTTP/WiFi reserve ch size_t maxAllocBlock(); // largest contiguous block (any memory type: incl PSRAM) size_t maxInternalAllocBlock(); // largest contiguous block in INTERNAL RAM only +// Largest contiguous block of EXECUTABLE memory (IRAM on an ESP32). A separate, much smaller pool +// than the data heap above: a MoonLive script's compiled code is allocated from it, so this is what +// bounds how large a script may be, and nothing else reports it. Zero where the platform has no +// distinct executable pool (desktop maps pages on demand). +size_t maxExecAllocBlock(); + // --- RTOS task introspection (TasksModule) -------------------------------------------------- // A fixed-size, allocation-free snapshot of the OS tasks, filled by the platform layer so no // FreeRTOS type escapes src/platform/ (the platform-boundary rule). ESP32 fills it from @@ -999,12 +1018,18 @@ bool rmtWs2812Init(RmtWs2812Handle& h, uint8_t gpio, uint32_t resolutionHz, bool // The driver converts its ns timings to ticks with this. 0 if not initialized. uint32_t rmtWs2812Resolution(const RmtWs2812Handle& h) MM_NONBLOCKING; -// Start transmitting `symbolCount` pre-encoded WS2812 RMT symbols and return -// immediately: channels started back-to-back clock out concurrently. Pair with -// rmtWs2812Wait; the caller owns the inter-frame latch (delayUs) after the last -// wait. The symbol buffer must stay valid until the wait returns. Returns false -// when the channel isn't initialized (and on targets without RMT). -bool rmtWs2812Transmit(RmtWs2812Handle& h, const uint32_t* symbols, size_t symbolCount); +// Transmit one frame as WIRE BYTES (the corrected, channel-ordered bytes the strip expects). +// Each byte is expanded to eight symbols on the way to the peripheral, MSB-first, using the bit +// shapes set by rmtWs2812SetBitTiming: the IDF's bytes encoder does it where RMT has DMA, and the +// classic ESP32's level-5 refill does it inline. So the caller's resident buffer is 3-4 bytes per +// light rather than 32 bytes per byte of that (96 per RGB light), which is what let a long strand +// outgrow internal RAM and silently stop transmitting. On the classic ESP32 the bytes must be in +// internal RAM (the refill can run with the flash cache off); a few KB, so this is not a limit. +bool rmtWs2812Transmit(RmtWs2812Handle& h, const uint8_t* wire, size_t byteCount); + +// Set the symbols a 0 and a 1 bit expand to. Live: the driver's `timing` control (400 kHz WS2811, +// 800 kHz, custom) rewrites these between frames. +bool rmtWs2812SetBitTiming(RmtWs2812Handle& h, uint32_t sym0, uint32_t sym1); // Block until the channel's in-flight transmission finishes, bounded by // `timeoutMs` so a wedged peripheral can't hang the render tick forever: a diff --git a/src/ui/app.js b/src/ui/app.js index 32fd7f4f..557a7072 100644 --- a/src/ui/app.js +++ b/src/ui/app.js @@ -672,7 +672,7 @@ async function addModule(type, parentName, id) { // Bring a module's card into view and focus its first control (added via the "+" flow). function focusModule(name) { - const card = document.querySelector(`.card[data-module="${cssEscape(name)}"]`); + const card = queryByName(`.card[data-module="${cssEscape(name)}"]`, "data-module", name); if (!card) return; // A child card can wrap its controls in a collapsed
(.card-controls-collapse): open it // FIRST so the card is at its expanded height, THEN scroll: scrolling a still-collapsed card lands @@ -1151,7 +1151,7 @@ function applyTabDot(tab, mod) { // WS value patch deliberately never re-runs: so without this, a fault (or an enable/disable) on a background // tab would stay invisible until the next full render. (The UI has two render paths; a rule must live in both.) function updateTabDot(mod) { - const tab = document.querySelector(`.tab[data-tab-mid="${cssEscape(mod.name)}"]`); + const tab = queryByName(`.tab[data-tab-mid="${cssEscape(mod.name)}"]`, "data-tab-mid", mod.name); if (!tab) return; applyTabDot(tab, mod); tab.classList.toggle("tab--disabled", mod.enabled === false); // grey a disabled module's tab title @@ -1738,7 +1738,7 @@ function createCard(mod, depth) { // instead of waiting ~1s for the server's full-state round-trip. (updateTabDot still syncs it on the // patch path, idempotently, so this just makes the on/off button the immediate driver.) The tab // lives in the parent's strip, found by the same data-tab-mid updateTabDot uses. - const tabEl = document.querySelector(`.tab[data-tab-mid="${cssEscape(mod.name)}"]`); + const tabEl = queryByName(`.tab[data-tab-mid="${cssEscape(mod.name)}"]`, "data-tab-mid", mod.name); if (tabEl) tabEl.classList.toggle("tab--disabled", !on); }; setEnabledUi(mod.enabled === undefined ? true : !!mod.enabled); @@ -2812,502 +2812,7 @@ function createControl(moduleName, moduleType, ctrl) { row.appendChild(input); break; } - case "filepath": { - // A file NAME plus an editor for that file's contents. The value travels through - // /api/control like any text control; the BODY never does (it cannot: only /api/file - // may exceed the request buffer), so the pane below reads and writes it directly. - // - // `dir` and `ext` come from the module that declared the control, so nothing here knows - // what kind of file this is. - const dir = ctrl.dir || ""; - const ext = ctrl.ext || ""; - // No `dir` means the name IS the path: joinFsPath("", n) would return "/n" and point - // at the filesystem root instead of the file the module named. - const pathOf = (n) => (n ? (dir ? joinFsPath(dir, n) : n) : ""); - // Where a script actually IS, which is not always `dir`: a factory script sits in the - // catalog's directory until an edit forks it into the user's. The device resolves the - // same way (user copy first), so the editor has to look in both or it would open an - // empty box for a script that is plainly listed. - const scriptPathOf = async (n) => { - if (!n || !dir) return pathOf(n); - const local = joinFsPath(dir, n); - if (!mlGroupForExt(ext)) return local; - try { - const here = await fmFetchDir(dir).catch(() => []); - if (here.some(e => !e.isDir && e.name === n)) return local; - const cat = await mlFetchCatalog(); - return joinFsPath(cat.dir, n); - } catch (_) { return local; } - }; - - const stack = document.createElement("div"); - stack.className = "control-fileedit-stack"; - row.appendChild(stack); - - const bar = document.createElement("div"); - bar.className = "fileedit-bar"; - - // A select-SHAPED button, opening the shared picker. It reads as a select (the current - // name, then the ⌄ affordance) and behaves as one, but the list it opens is the same - // widget the module picker uses: search, emoji chips, keyboard, one row per script. - // A native does (`value`, `options`, `disabled`, and a - // `change` event), so everything around it (the fork/share/delete labels, the editor - // load, the download-on-pick) is unchanged and unaware. - const picker = document.createElement("button"); - picker.type = "button"; - picker.className = "control-select fileedit-pick"; - picker.dataset.mid = moduleName; - picker.dataset.key = ctrl.name; - // A READ-ONLY filepath names the file something else chose, so it must not offer a - // second way to choose: the Drivers palette editor is the case, where `palette` owns the - // selection and this pane only edits what that selection resolved to. Two selectors for - // one value is how they end up disagreeing. - if (ctrl.readonly) { picker.disabled = true; picker.classList.add("is-readonly"); } - // The options, as data. `fillPicker` appends option elements exactly as it did to the - // 's own change handler, which already downloads a - // remote script, updates the delete label and loads the editor. One path for - // both ways of choosing. - commit: (name) => { - picker.value = name; - picker.dispatchEvent(new Event("change")); - }, - }); - }; - picker.addEventListener("click", openScriptPicker); - - popBtn.addEventListener("click", async () => { - if (!picker.value) return; - // Flush unsaved edits first: the modal loads the file from the device, so opening - // it on a dirty pane would show stale bytes and then save them back over the edit. - // save() RESOLVES on a failed write (it reports, it does not throw), so the flush is - // only trustworthy if the pane came clean: opening anyway would discard the edit. - await editor.save(); - if (editor.isDirty()) { alert("Not opening: this script still has unsaved changes."); return; } - const p = await scriptPathOf(picker.value); - await openFileEditor(p, undefined, moduleName); - await editor.load(p); - }); - - picker.addEventListener("change", async () => { - // Same reason as the modal above: switching files discards the edit otherwise, and - // a save that failed leaves the pane dirty while resolving normally. - await editor.save(); - if (editor.isDirty()) { - alert("Not switching: this script still has unsaved changes."); - picker.value = String(ctrl.value ?? ""); - return; - } - const chosen = picker.value; - // A factory script the device does not hold yet: download it BEFORE selecting it, - // so the module never points at a file that is not there. A failure reports and - // puts the picker back, rather than leaving the card pointing at nothing. - if (remote.includes(chosen)) { - const previous = String(ctrl.value ?? ""); - picker.disabled = true; - try { - await mlDownloadScript(chosen, mlGroupForExt(ext)); - } catch (e) { - picker.disabled = false; - alert("could not download " + chosen + ": " + (e && e.message ? e.message : e)); - picker.value = previous; - return; - } - picker.disabled = false; - await fillPicker(); // it is local now, so it loses its marker - picker.value = chosen; - } - refreshDelLabel(); - dragTs[key] = Date.now(); - sendControl(moduleName, ctrl.name, chosen); - editor.load(await scriptPathOf(chosen)); - }); - - newBtn.addEventListener("click", async () => { - let name = (prompt("New file name in " + dir + ":") || "").trim(); - if (!name) return; - if (ext && !name.endsWith(ext)) name += ext; // a name without its extension is a typo - const r = await fmCreateFile(dir, name, ctrl.tmpl || ""); - if (!r.ok) { alert("create file failed: " + r.message); return; } - await fillPicker(); - picker.value = name; - dragTs[key] = Date.now(); - sendControl(moduleName, ctrl.name, name); - await editor.load(pathOf(name)); - editor.textarea.focus(); - }); - - // Two clicks to delete, the same arm-then-confirm the File Manager uses for its own - // delete: destructive next to frequent is how people lose work. - armPressTwice(delBtn, async () => { - const victim = picker.value; - if (!victim) return; - const wasFork = forks.has(victim); - try { - // A fork is deleted from the USER directory, which is the whole revert: the - // factory copy underneath is what resolves afterwards. Anything else is deleted - // where it actually sits, because a downloaded factory script has no user copy - // and a DELETE on /moonlive/ would report a failure for a file that was - // never there. - const target = wasFork ? pathOf(victim) : await scriptPathOf(victim); - const res = await fetch("/api/dir?path=" + encodeURIComponent(target), - { method: "DELETE" }); - if (!res.ok) throw new Error(await errorMessage(res)); - } catch (err) { - alert((wasFork ? "revert failed: " : "delete failed: ") + err.message); - return; - } - await fillPicker(); - if (wasFork) { - // The factory script is what resolves now, so the module keeps running: stay on - // it rather than unsetting the control, which is the whole point of a revert. - picker.value = victim; - refreshDelLabel(); - dragTs[key] = Date.now(); - sendControl(moduleName, ctrl.name, victim); - await editor.load(await scriptPathOf(victim)); - return; - } - picker.value = ""; - refreshDelLabel(); - dragTs[key] = Date.now(); - sendControl(moduleName, ctrl.name, ""); - await editor.load(""); - }, { armedText: "✓", armedTitle: "Click again to confirm" }); - - // The editor mounted on the USER path above, which is right for a script the user - // wrote and wrong for a factory one that has never been edited. Resolving needs the - // catalog, so it cannot happen during the synchronous mount: re-point it once the - // listing is in, and only when it actually resolves elsewhere. - fillPicker().then(async () => { - const cur = String(ctrl.value ?? ""); - if (!cur || !mlGroupForExt(ext)) return; - const real = await scriptPathOf(cur); - if (real !== pathOf(cur)) await editor.load(real); - }); - break; - } + case "filepath": return buildFilePathControl(row, label, key, moduleName, ctrl); case "password": { // ctrl.value arrives XOR-obfuscated + base64-encoded (see // HttpServerModule PASSWORD_XOR_KEY). Decode it so the input holds @@ -3379,186 +2884,7 @@ function createControl(moduleName, moduleType, ctrl) { appendResetButton(row, moduleName, ctrl, def, () => { sel.value = def; }); break; } - case "palette": { - // A color-palette dropdown where EVERY option shows its own gradient: so the colors - // are visible before selecting, not just after. A native ) - const enabledEl = document.querySelector(`button.module-enabled[data-mid="${cssEscape(mod.name)}"]`); + const enabledEl = queryByName(`button.module-enabled[data-mid="${cssEscape(mod.name)}"]`, "data-mid", mod.name); if (enabledEl) { const ts = dragTs[mod.name + ":enabled"] || 0; if (Date.now() - ts > 1000) { @@ -5049,7 +4375,7 @@ function updateValues() { setText(enabledEl, "\u23FB"); enabledEl.classList.toggle("module-enabled--off", !on); enabledEl.setAttribute("aria-pressed", on ? "true" : "false"); - const cardEl = document.querySelector(`.card[data-module="${cssEscape(mod.name)}"]`); + const cardEl = queryByName(`.card[data-module="${cssEscape(mod.name)}"]`, "data-module", mod.name); if (cardEl) cardEl.classList.toggle("card--disabled", !on); } } @@ -5081,7 +4407,7 @@ function allModules() { // card's child-module block / install-picker mount, never converge, and re-fire // every WS tick: a render loop that wedges the UI). function syncVisibleControls(mod) { - const card = document.querySelector(`.card[data-module="${cssEscape(mod.name)}"]`); + const card = queryByName(`.card[data-module="${cssEscape(mod.name)}"]`, "data-module", mod.name); if (!card) return false; // The controls host is THIS card's own collapse wrapper: must be a DIRECT // child (`:scope >`), not any descendant: a container card (e.g. Effects) nests @@ -5176,7 +4502,11 @@ function updateModuleControls(mod) { if (syncVisibleControls(mod)) return; // re-rendered: values are fresh, skip patch for (const ctrl of mod.controls) { - const mid = cssEscape(mod.name); + // RAW, not escaped: every selector below escapes it itself, and queryByName compares this + // against the data-mid ATTRIBUTE, which the DOM returns unescaped. Escaping here made the + // comparison fail for any module name that needs escaping (a quote or a backslash), so the + // live patch silently found nothing and that card stopped updating. + const mid = mod.name; const k = cssEscape(ctrl.name); const dragKey = mod.name + ":" + ctrl.name; const ts = dragTs[dragKey] || 0; @@ -5195,7 +4525,7 @@ function updateModuleControls(mod) { case "int16": case "int32": case "pin": { // pin is a plain number input (no slider sibling); patches the same way - const input = document.querySelector(`input[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); // While the demo sweep animates a control, leave it alone: the sweep restores the // real value when it ends, and the next patch after that lands normally. if (input && surfaceDemoRunning() && @@ -5209,23 +4539,27 @@ function updateModuleControls(mod) { break; } case "bool": { - const input = document.querySelector(`input[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (input && input.checked !== !!ctrl.value) input.checked = !!ctrl.value; break; } case "text": { - const input = document.querySelector(`input[type="text"][data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input[type="text"][data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (input && input.value !== (ctrl.value ?? "")) input.value = ctrl.value ?? ""; break; } case "textarea": { - const input = document.querySelector(`textarea[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`textarea[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); // Don't clobber the box while the user is typing in it. if (input && document.activeElement !== input && input.value !== (ctrl.value ?? "")) input.value = ctrl.value ?? ""; break; } case "filepath": { - const sel = document.querySelector(`select.fileedit-pick[data-mid="${mid}"][data-key="${k}"]`); + // A BUTTON, not a select: buildFilePathControl replaced the native select with a + // painted picker that keeps its value in `_value` behind a `value` property, so the + // assignment below still works. Matching `select` here found nothing, and a script + // changed from another client (or by a preset) never repainted this picker. + const sel = queryByName(`button.fileedit-pick[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); // Don't clobber the choice while it is focused, and don't reload the pane under // someone who is typing in it: the value is only pushed back when it really moved. if (sel && document.activeElement !== sel && sel.value !== (ctrl.value ?? "")) { @@ -5235,13 +4569,13 @@ function updateModuleControls(mod) { } case "password": { // The peek button flips the input to type="text", so match either. - const input = document.querySelector(`input[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); const decoded = decodePassword(ctrl.value); if (input && input.value !== decoded) input.value = decoded; break; } case "select": { - const sel = document.querySelector(`select[data-mid="${mid}"][data-key="${k}"]`); + const sel = queryByName(`select[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); // Never overwrite a select the user currently has OPEN (popup // showing) or focused. data-open is set on pointerdown/focus and // cleared on change/blur: more reliable than document.activeElement, @@ -5272,7 +4606,7 @@ function updateModuleControls(mod) { case "palette": { // Custom dropdown: patch the trigger (swatch + name) and the selected row, but not // while the user has the list open (data-open === "true"). - const wrap = document.querySelector(`.palette-control[data-mid="${mid}"][data-key="${k}"]`); + const wrap = queryByName(`.palette-control[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (wrap && wrap.dataset.open !== "true" && Number(wrap.dataset.value) !== Number(ctrl.value)) { wrap.dataset.value = ctrl.value; const cols = ((ctrl.options || [])[ctrl.value] || {}).colors || ""; @@ -5290,18 +4624,18 @@ function updateModuleControls(mod) { case "display": { // A url-valued display is an , not a (see renderControl), so this path // has to know both shapes or the link goes stale on the next push. - const link = document.querySelector(`a.control-url[data-mid="${mid}"][data-key="${k}"]`); + const link = queryByName(`a.control-url[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (link) { setUrlDisplay(link, ctrl.value); break; } // The display strip is a segment renderer, not a span: it has to be patched through // its own hook or the surface would freeze at whatever it showed on first render. - const strip = document.querySelector(`.seg16[data-mid="${mid}"][data-key="${k}"]`); + const strip = queryByName(`.seg16[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (strip && strip._setText) { strip._setText(ctrl.value ?? ""); break; } - const span = document.querySelector(`span.display[data-mid="${mid}"][data-key="${k}"]`); + const span = queryByName(`span.display[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (span) setText(span, String(ctrl.value ?? "")); break; } case "display-int": { - const span = document.querySelector(`span.display[data-mid="${mid}"][data-key="${k}"]`); + const span = queryByName(`span.display[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (span) { // Re-cache the unit in case the device changed it (it // shouldn't, but the WS path is the authority). @@ -5312,27 +4646,27 @@ function updateModuleControls(mod) { } case "ipv4": { // Guarded by the shared userActive check above (same as text). - const input = document.querySelector(`input.ipv4-input[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input.ipv4-input[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (input && input.value !== (ctrl.value ?? "")) input.value = ctrl.value ?? ""; break; } case "time": { - const span = document.querySelector(`span.display[data-mid="${mid}"][data-key="${k}"]`); + const span = queryByName(`span.display[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (span) setText(span, fmtTime(ctrl.value ?? 0)); break; } case "progress": { - const bar = document.querySelector(`progress[data-mid="${mid}"][data-key="${k}"]`); + const bar = queryByName(`progress[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (bar) { bar.value = ctrl.value ?? 0; bar.max = ctrl.total ?? 100; } - const lbl = document.querySelector(`span.control-value[data-mid="${mid}"][data-key="${k}.label"]`); + const lbl = queryByName(`span.control-value[data-mid="${cssEscape(mid)}"][data-key="${k}.label"]`, "data-mid", mid); if (lbl) setText(lbl, fmtProgressLabel(ctrl)); break; } case "list": { - const list = document.querySelector(`div.list-control[data-mid="${mid}"][data-key="${k}"]`); + const list = queryByName(`div.list-control[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (!list) break; const rows = Array.isArray(ctrl.value) ? ctrl.value : []; const details = Array.isArray(ctrl.detail) ? ctrl.detail : []; @@ -5399,7 +4733,7 @@ function updateModuleControls(mod) { // control itself wins over the type-level ones from /api/types; see defaultFor. const def = defaultFor(mod.type, ctrl.name, ctrl); if (def !== undefined && def !== null) { - const btn = document.querySelector(`button.reset-btn[data-mid="${mid}"][data-key="${k}.reset"]`); + const btn = queryByName(`button.reset-btn[data-mid="${cssEscape(mid)}"][data-key="${k}.reset"]`, "data-mid", mid); if (btn) { const eq = controlValuesEqual(ctrl, def); btn.classList.toggle("active", !eq); @@ -5426,6 +4760,23 @@ function cssEscape(s) { return String(s).replace(/(["\\])/g, "\\$1"); } +// Find the one element whose `attr` EQUALS `value`, case included. +// +// CSS attribute selectors match values case-INSENSITIVELY in an HTML document, while the firmware +// compares module names with strcmp. So `lines` (a MoonLive effect) and `Lines` (a LinesEffect) are +// two different modules that every `[data-module="..."]` lookup confuses: querySelector returns +// whichever sits first in the DOM, and on the bench (2026-09-08) a Layer holding both rendered no +// effect cards at all. Selectors Level 4 has a case-sensitivity flag for exactly this +// (`[data-module="x" s]`) and Chrome does not support it: it throws SyntaxError, which takes out +// every card on the page. So the match is narrowed here instead, which works everywhere. +function queryByName(selector, attr, value) { + const want = String(value); + for (const el of document.querySelectorAll(selector)) { + if (el.getAttribute(attr) === want) return el; + } + return null; +} + // --------------------------------------------------------------------------- // 6. Type picker // --------------------------------------------------------------------------- @@ -7622,7 +6973,8 @@ async function openFileEditor(relPath, expectedSize, moduleName) { // and a compile failure that happened before the modal opened would otherwise show unmarked // until the module recompiles. The row is the card's, so it holds the same text either way. if (moduleName) { - const row = document.querySelector(`[data-status-mid="${cssEscape(moduleName)}"] .status-value`); + const row = (queryByName(`[data-status-mid="${cssEscape(moduleName)}"]`, "data-status-mid", moduleName) + || {}).querySelector?.(".status-value"); if (row) ed.markError(row.textContent); } dlg.showModal(); @@ -7644,3 +6996,690 @@ async function openFileEditor(relPath, expectedSize, moduleName) { // --------------------------------------------------------------------------- document.addEventListener("DOMContentLoaded", init); + + +/// The palette editor: a gradient strip whose stops are dragged, added and removed, plus the +/// pickers behind them. Lifted out of createControl's switch, where its 180 lines and the +/// filepath case's 496 made a dispatch table read as an implementation. +/// +/// Same shape as buildKnob / buildListEntries / buildListPads above: one control type, one +/// function, taking exactly what it needs from the row createControl already built. +function buildPaletteControl(row, key, def, moduleName, ctrl) { + // A color-palette dropdown where EVERY option shows its own gradient: so the colors + // are visible before selecting, not just after. A native can only render plain text, so a script's emoji and dimension had + // nowhere to go. + // + // It presents the SAME surface a ; they are never rendered, they are the list the modal is built from. + picker.options = []; + picker.appendChild = (o) => { picker.options.push(o); return o; }; + const paintPicker = () => { + const cur = picker.options.find(o => o.value === picker._value); + picker.textContent = cur ? cur.textContent : "(none)"; + const caret = document.createElement("span"); + caret.className = "fileedit-pick-caret"; + caret.textContent = "\u2304"; // ⌄, the select affordance + picker.append(caret); + }; + Object.defineProperty(picker, "value", { + get: () => picker._value ?? "", + set: (v) => { picker._value = String(v ?? ""); paintPicker(); }, + }); + picker._value = ""; + // innerHTML = "" is how fillPicker clears the list; keep that meaning. + Object.defineProperty(picker, "innerHTML", { + set: (v) => { if (v === "") { picker.options = []; picker.replaceChildren(); } }, + get: () => "", + }); + // Which scripts this picker can offer that are not on the device yet. Names only: + // picking one downloads it. Empty for a filepath control that is not a script picker. + let remote = []; + // Local names that also exist in the catalog: a user edit shadowing a factory script. + let forks = new Set(); + // Every name the catalog ships for this role, whether or not it is on the device. + let catalogNames = new Set(); + // Names in the USER's directory: written or edited here, so worth proposing upstream. + let localNames = new Set(); + const fillPicker = async () => { + picker.innerHTML = ""; + const none = document.createElement("option"); + none.value = ""; none.textContent = "(none)"; + picker.appendChild(none); + let names = []; + if (dir) { + try { + const entries = await fmFetchDir(dir); + names = entries.filter(e => !e.isDir && (!ext || e.name.endsWith(ext))) + .map(e => e.name); + } catch (_) { /* an unreachable directory leaves just "none" */ } + } + // The user's OWN files, before the factory listing is merged in below: a name here + // is something they wrote or edited, which is what the share button offers. + localNames = new Set(names); + // A script picker also lists the FACTORY directory, where downloads land. A name in + // both is the user's edit shadowing the factory copy, which is what the device + // resolves too, so it appears once. + const group = mlGroupForExt(ext); + let cat = null; + if (group) { + try { + cat = await mlFetchCatalog(); + const factory = await fmFetchDir(cat.dir, true).catch(() => []); + for (const e of factory) + if (!e.isDir && e.name.endsWith(ext) && !names.includes(e.name)) + names.push(e.name); + } catch (_) { /* no catalog: the picker still lists what is here */ } + } + names.sort(); + // A local name that ALSO exists in the catalog is a fork: the user edited a factory + // script, so their copy shadows one that can be restored. Deleting it is a revert, + // not a loss, and the delete button says so. + // localNames, NOT names: by here `names` also carries the factory listing, so a + // script that was downloaded and never touched counted as a fork. It showed the + // revert arrow for an edit that does not exist, and reverting it deleted a + // /moonlive path with nothing at it. + forks = cat ? new Set(((cat[group] || {}).names || []).filter(n => localNames.has(n))) + : new Set(); + // Everything the catalog offers that is not here yet, listed after the local ones + // so a user's own scripts stay at the top of the list. + remote = cat ? ((cat[group] || {}).names || []).filter(n => !names.includes(n)) : []; + // Every name the library ships for this role, downloaded or not: what the share + // button uses to tell a user's own script from one of ours. + catalogNames = new Set(cat ? ((cat[group] || {}).names || []) : []); + + // The current value may name a file the listing does not have (deleted underneath, + // or a directory that could not be read). Keep it selectable so the card still + // shows what the module is pointing at, rather than silently appearing unset. + const cur = String(ctrl.value ?? ""); + if (cur && !names.includes(cur) && !remote.includes(cur)) names.unshift(cur); + // What the catalog says each factory script is: its dimension and its own emoji, + // read from the script's `int dimensions()` / `string tags()` at build time. So a + // row reads like the module picker's rows do, BEFORE the script is downloaded. A + // script the catalog does not carry (the user's own) simply has no prefix. + const decl = (n) => { + const g = cat && cat[group]; + if (!g || !g.names) return ""; + const i = g.names.indexOf(n); + if (i < 0) return ""; + const marks = []; + if (g.tags && g.tags[i]) marks.push(g.tags[i]); + if (g.dim && DIM_EMOJI[g.dim[i]]) marks.push(DIM_EMOJI[g.dim[i]]); + return marks.length ? marks.join("") + " " : ""; + }; + for (const n of names) { + const o = document.createElement("option"); + o.value = n; o.textContent = decl(n) + n; + picker.appendChild(o); + } + // Marked, because picking one costs a download and can fail. One list rather than + // two groups: to the user it is one library, and where a script happens to live is + // the device's business. + for (const n of remote) { + const o = document.createElement("option"); + o.value = n; + o.textContent = "\u2601 " + decl(n) + n; // cloud: not on this device yet + picker.appendChild(o); + } + picker.value = cur; + refreshDelLabel(); + }; + + // Save sits with the other file actions rather than in a row of its own: the card is + // already narrow, and the dot on it is what marks unsaved work. + const saveBtn = document.createElement("button"); + saveBtn.className = "card-btn fm-editor-save fileedit-glyph-lg"; + // U+2398, the ISO "store" symbol. Not an arrow: ↥ and ↧ already mean upload and + // download here, and ⤓ downloads a file in the tree, so an arrow would read as + // "fetch this" on a button that writes. Not ✓ either, which is the ARMED DELETE + // state one button along. + saveBtn.textContent = "⎘"; + saveBtn.title = "Save (or click away, or Ctrl/Cmd+S)"; + + // The same modal the File Manager opens from a file row: one editor, reached two ways, + // so a script that needs room gets the full-size box without a second implementation. + const popBtn = document.createElement("button"); + popBtn.className = "card-btn fileedit-glyph-lg"; + popBtn.textContent = "⤢"; // expand, the usual glyph for a bigger view + popBtn.title = "Open in a larger window"; + + // No second status line: the module's own `status` control already reports what the + // save produced ("2036 B", or the parse error), and it is the authoritative one because + // the DEVICE writes it. A browser-side copy said the same thing in different words and + // could only ever disagree. What the browser knows and the device cannot (unsaved work, + // a save in flight, a failed write) rides the Save button instead: its dot, its + // disabled state, and its tooltip. + const statusEl = document.createElement("span"); + statusEl.hidden = true; + + const newBtn = document.createElement("button"); + newBtn.className = "card-btn"; + newBtn.textContent = "+"; // the same + the module tree adds with + newBtn.title = "New script"; + const delBtn = document.createElement("button"); + delBtn.className = "card-btn card-btn-del"; + delBtn.textContent = "×"; // the card's own delete, red on the symbol + delBtn.title = "Delete this script"; + // The SAME button reverts a factory script, because it is the same operation: the + // editor only ever saves to the user directory, so an edited factory script is a second + // file shadowing the first, and removing it brings the original back. Saying "delete" + // there would misdescribe it, and a second button would make one act look like two. + const shareBtn = document.createElement("button"); + shareBtn.className = "card-btn"; + shareBtn.textContent = "\u2197"; // north-east arrow: it leaves for somewhere else + shareBtn.title = "Propose this script for the shared library"; + + function refreshDelLabel() { + const isFork = forks.has(picker.value); + delBtn.textContent = isFork ? "\u21ba" : "\u00d7"; // undo arrow, or the delete cross + delBtn.title = isFork + ? "Revert to the shipped version (discards your changes)" + : "Delete this script"; + delBtn.classList.toggle("card-btn-del", !isFork); + // Offered for anything the user WROTE, which is a script of their own or a fork of + // a shipped one: both are a change worth sending back, and the flow differs only in + // which GitHub URL it opens. NOT offered for an untouched factory copy, where the + // file on the device is byte-identical to the one in the repo and a pull request + // would propose no change at all. + const known = catalogNames.has(picker.value); + const edited = localNames.has(picker.value); // it sits in the USER directory + shareBtn.hidden = !picker.value || !mlGroupForExt(ext) || !edited; + shareBtn.title = known + ? "Propose your changes to the shared library" + : "Propose this script for the shared library"; + } + // Share: open a pull request adding this script to the library. + // + // GitHub's "new file" URL takes the path and the contents as query parameters and opens + // its editor pre-filled, forking the repo on the user's behalf when they propose it. So + // a script someone wrote on their own device reaches the library with one click and no + // API, no token and nothing stored here. + // + // Only for scripts a user WROTE: a factory script is already in the library, and a fork + // of one would open a PR that recreates a file that exists. + shareBtn.addEventListener("click", async () => { + const name = picker.value; + const group = mlGroupForExt(ext); + if (!name || !group) return; + await editor.save(); // propose what is on screen, not the last save + // A save that failed leaves the pane dirty, and the read below would then fetch the + // PREVIOUS text from the device: the user would be proposing something other than + // what they are looking at, which is the one outcome worth refusing outright. + if (editor.isDirty()) { + alert("Save the script first: it still has unsaved changes."); + return; + } + let text = ""; + try { + const res = await fetch("/api/file?path=" + encodeURIComponent(await scriptPathOf(name))); + if (!res.ok) throw new Error(await errorMessage(res)); + text = await res.text(); + } catch (err) { alert("could not read the script: " + err.message); return; } + + const cat = await mlFetchCatalog().catch(() => null); + const folder = cat && cat[group] ? cat[group].folder : group; + // A name the library already ships is an EDIT of that file; anything else is a new + // one. GitHub has a flow for each, and both fork on the user's behalf when they + // propose the change, so neither needs write access to this repo. + const repoPath = "moonlive/" + folder + "/" + name; + const url = catalogNames.has(name) + ? "https://github.com/MoonModules/projectMM/edit/main/" + repoPath + + "?value=" + encodeURIComponent(text) + : "https://github.com/MoonModules/projectMM/new/main" + + "?filename=" + encodeURIComponent(repoPath) + + "&value=" + encodeURIComponent(text); + // The script rides in the query string, and browsers stop honoring a URL somewhere + // past ~8 KB. Every shipped script is under 2.5 KB so this is headroom rather than a + // real limit, but a long one would otherwise open a truncated editor and look fine. + if (url.length > 7000) { + await navigator.clipboard.writeText(text).catch(() => {}); + alert("This script is too long to send through a link.\n\n" + + "It has been copied to your clipboard: open\n" + + "github.com/MoonModules/projectMM, add a file under moonlive/" + folder + + "/ and paste it there."); + return; + } + window.open(url, "_blank", "noopener"); + }); + + bar.appendChild(picker); + const tools = document.createElement("div"); + tools.className = "fileedit-tools"; + tools.appendChild(saveBtn); + tools.appendChild(popBtn); + if (dir) { tools.appendChild(newBtn); tools.appendChild(shareBtn); tools.appendChild(delBtn); } + bar.appendChild(tools); + stack.appendChild(bar); + + const pane = document.createElement("div"); + pane.className = "control-fileedit"; + stack.appendChild(pane); + + // Saving re-derives on the device: a written file asks the tree to re-prepare, so the + // module recompiles or reloads on its own. The browser sends nothing extra. + const editor = fmMountEditor(pane, pathOf(ctrl.value), { + sizeKey: key, + // The status this module is ALREADY reporting, so a card built while its script is + // broken shows the marked line straight away rather than waiting for a recompile. + // The EDITOR applies it once the file has loaded: marking at construction would + // convert the offset against an empty textarea and put every error on line 1. + initialStatus: (findModule(moduleName) || {}).status || "", + saveButton: saveBtn, + statusEl, + // Editing a factory script FORKS it: the read came from the library directory, but + // the write goes to the user's, so the shipped copy stays untouched and the new one + // shadows it. Without this an edit overwrote the library copy and there was nothing + // left to revert to. + savePath: (readPath) => { + if (!dir) return readPath; + const base = readPath.slice(readPath.lastIndexOf("/") + 1); + return base ? joinFsPath(dir, base) : readPath; + }, + // A save may have just created the fork, so what the picker thinks is local is out + // of date: re-read it, which is also what turns the delete button into revert. + onSaved: (written) => { + if (!mlGroupForExt(ext)) return; + if (!written.startsWith(dir + "/")) return; + const sel = picker.value; + fillPicker().then(() => { picker.value = sel; refreshDelLabel(); }); + }, + }); + mlEditorAdd(moduleName, editor); + // Apply the status the card is ALREADY showing: registration only catches the next one, + // so a card rendered while its script is broken would report the error with no line + // marked until something recompiled. The modal does the same on open. + { + const row = (queryByName(`[data-status-mid="${cssEscape(moduleName)}"]`, "data-status-mid", moduleName) + || {}).querySelector?.(".status-value"); + if (row) editor.markError(row.textContent); + } + + // Re-read after the modal closes: it edits the same file through the same endpoints, so + // whatever it saved is what this pane should now show. + // One row per script, shaped like a type so the shared picker can render it: the name is + // what the control stores, the dimension and emoji come from the catalog, and the role + // is what the picker prints on the right. + const openScriptPicker = async () => { + if (picker.disabled) return; + const group = mlGroupForExt(ext); + const cat = group ? await mlFetchCatalog().catch(() => null) : null; + const g = (cat && cat[group]) || {}; + const roleWord = group ? group.replace(/s$/, "") : "file"; + const seen = new Set(); + const items = []; + const add = (n, remoteFlag) => { + if (seen.has(n)) return; + seen.add(n); + const i = (g.names || []).indexOf(n); + items.push({ + name: n, + // The cloud marks a script the device does not hold yet: picking it costs a + // download, which is worth knowing before choosing. + displayName: (remoteFlag ? "\u2601 " : "") + n, + role: roleWord, + tags: i >= 0 && g.tags ? (g.tags[i] || "") : "", + dim: i >= 0 && g.dim ? g.dim[i] : 0, + }); + }; + for (const o of picker.options) if (o.value) add(o.value, remote.includes(o.value)); + for (const n of (g.names || [])) add(n, !localNames.has(n) && remote.includes(n)); + if (!items.length) return; + // Anchored to the field itself: the picker opens under it as a modal, sized and + // placed by openPicker rather than by whatever element it hangs from. + openPicker(picker, { + items, + actionLabel: "use", + currentType: picker.value, + // Route through the