From f00b48fcb898db956fea00ef78d9600fed16715a Mon Sep 17 00:00:00 2001 From: ewowi Date: Tue, 8 Sep 2026 23:01:02 +0200 Subject: [PATCH 1/6] Fix the serial device-model push, and correct four boards' pins Provisioning a board over USB set its WiFi and nothing else: the script sent a vendor RPC no firmware has ever handled, then reported success. It now applies the catalog entry the way the web installer does, so a fresh board comes up configured. Four device models had wrong or missing pins, including the P4 shield, where two of eight LED panels stayed dark. Performance: desktop 1199 fps, 834 us/tick; flash esp32 2057984, esp32-16mb 2058000, esp32-pico 2106464, esp32p4rev1-eth 1995280, esp32p4rev1-eth-wifi 2284640, esp32s3-n16r8 2100032, esp32s3-n8r8 2087168, esp32s31 2348592, desktop 1909464. 2000 test cases, 27 scenarios. Core: - JsonSink flags a refused heap grow instead of dropping bytes silently: a truncated document was indistinguishable from a whole one at the far end, so a board that could not allocate shipped a cut module tree under a frame header declaring it complete, and the UI lost whole cards with nothing logged. ensureHeap also steps down in quarters when a doubling is refused, which fits a fragmented heap where the doubled block does not. - The state push warns and keeps the patch stream alive when the document does not fit, rather than returning early: an early return left fullResyncPending_ set, starved the patch branch, and froze the whole UI (no fps, no heap, no live values) on a board where the state simply does not fit. Light domain: - ParallelLedDriver counts the peripheral's DMA buffers in its memory readout. They are platform-allocated and were left out on that ownership argument, which made the card lie about the figure a user picks a driver on: the i80 frame is sized by the bus width, not the pins in use, so a one-lane board reported 512 bytes against RmtLedDriver's 32 KB while costing 49 KB more free heap. Counts the buffers that EXIST, not the ones doubleBuffer asks for. UI: - Two modules whose names differ only in case shared a card. The firmware compares names with strcmp, so `lines` and `Lines` are different modules; CSS attribute selectors match case-insensitively, so every name-keyed querySelector resolved to whichever came first in the DOM and a Layer holding both rendered no effect cards at all. Narrowed in JS by queryByName(): the Selectors Level 4 `s` flag was tried first and reverted, because Chrome throws SyntaxError on it and takes out every card on the page. Scripts/MoonDeck: - improv_provision.py applies the device model as APPLY_OP ops, a faithful port of the browser's planner (clearChildren pre-pass, add, then set), sent closed-loop: each frame is acked, a busy device is retried, and the device's own state chatter is no longer read as a refusal. It probes GET_CURRENT_STATE first, so an already-provisioned board keeps its credentials and still gets the config, and an Ethernet-only entry skips the WiFi exchange entirely. - Catalog order now matters and is documented: an entry whose LED pins include GPIO 1/3 lists its driver LAST, because setting those pins ends serial reception and every later op is lost. Four entries reordered. Tests: - The Improv frame tests gained the op planner (mirroring config-ops.js case for case), the chunked framing, a scripted fake port for the closed-loop sender, and a guard that the phantom RPC never returns. - unit_JsonSink_overflow, unit_AllocTracking, ui-name-case, and the DMA readout cases in the shared host_bus harness. Docs/CI: - The MHC-WLED P4 shield reference contradicted itself: its terminal line reads O21 O20 O25 O5 O7 O23 O8 O27 while its table listed O22/O24 in positions 5 and 7. The catalog followed the table, so two strands got no signal. Table corrected, with the I2C trade-off stated (GPIO 7/8 are also the I2C bus). - Device models: Dig-2-Go gains its relay pin and GRBW preset, Dig-Octa and Dig-Next-2 move to ParallelLed with a DC pin, the P4 shield lists both its firmwares so the installer can offer the WiFi variant. - Backlog: the state-over-WebSocket architecture (snapshot over HTTP, deltas over WS), the P4 co-processor WiFi findings, the persisted firmware-variant bug, and the RMT/ParallelLed crossover measured at ~500 lights. - The WebSocket streaming plan is archived as attempted-and-reverted: it worked on the bench and still crashed a 65 KB board on a UI refresh, because the chain holds the whole document for the drain's duration. Co-Authored-By: Claude Opus 5 (1M context) --- docs/backlog/backlog-core.md | 232 ++- docs/backlog/backlog-light.md | 75 +- ...d of buffering it (attempted, reverted).md | 191 +++ docs/metrics/repo-health.json | 146 +- docs/metrics/repo-health.md | 62 +- docs/moonmodules/light/drivers.md | 2 + docs/reference/mhc-wled-esp32-p4-shield.md | 8 +- esp32/sdkconfig.defaults.esp32p4rev1-eth | 35 +- moonbase/main/moonbase_main.cpp | 97 +- moondeck/MoonDeck.md | 4 +- moondeck/build/improv_provision.py | 214 ++- mooninstaller/deviceModels.json | 91 +- src/core/HttpServerModule.cpp | 19 +- src/core/JsonSink.h | 44 +- src/core/SystemModule.h | 16 + src/light/ColorLight5A75Packet.h | 8 +- src/light/drivers/ParallelLedDriver.h | 43 +- src/platform/desktop/platform_desktop.cpp | 54 +- src/platform/esp32/platform_esp32.cpp | 22 + src/platform/platform.h | 13 + src/ui/app.js | 1439 +++++++++-------- test/CMakeLists.txt | 2 + test/js/ui-live-patch-text.test.mjs | 2 +- test/js/ui-name-case.test.mjs | 60 + test/python/test_improv_frame.py | 146 +- .../scenario_MoonModule_control_change.json | 32 +- .../light/scenario_Audio_mutation.json | 42 +- test/scenarios/light/scenario_Aurora_fps.json | 68 +- .../light/scenario_Driver_mutation.json | 20 +- .../light/scenario_Effects_composition.json | 4 +- .../light/scenario_Fields_polar_lut.json | 54 +- .../light/scenario_Fluid_solver.json | 84 +- .../light/scenario_GridBlacks_blackpixel.json | 12 +- .../light/scenario_GridLayout_resize.json | 14 +- .../light/scenario_Layer_base_pipeline.json | 4 +- .../light/scenario_Layer_memory_1to1.json | 4 +- .../light/scenario_Layouts_mutation.json | 18 +- .../scenario_MoonLiveEffect_livescript.json | 34 +- .../light/scenario_MoonLive_pipeline.json | 20 +- .../scenario_MultiplyModifier_memory_lut.json | 4 +- .../scenario_MultiplyModifier_pipeline.json | 4 +- .../light/scenario_Trails_ladder.json | 42 +- .../light/scenario_modifier_chain.json | 16 +- .../light/scenario_modifier_swap.json | 14 +- test/scenarios/light/scenario_perf_full.json | 86 +- test/scenarios/light/scenario_perf_light.json | 16 +- .../light/scenario_peripheral_grid_sweep.json | 82 +- .../light/scenario_peripheral_switch.json | 24 +- test/unit/core/unit_AllocTracking.cpp | 56 + test/unit/core/unit_JsonSink_overflow.cpp | 58 + test/unit/light/host_bus.h | 38 + test/unit/light/unit_MultiPinLedDriver.cpp | 4 + test/unit/light/unit_ParlioLedDriver.cpp | 4 + 53 files changed, 2618 insertions(+), 1265 deletions(-) create mode 100644 docs/history/plans/Plan-20260908 - Stream the WebSocket state instead of buffering it (attempted, reverted).md create mode 100644 test/js/ui-name-case.test.mjs create mode 100644 test/unit/core/unit_AllocTracking.cpp create mode 100644 test/unit/core/unit_JsonSink_overflow.cpp diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index d7356d09..1af8b995 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -1308,7 +1308,11 @@ We build for hardware we cannot see. Which chips people actually run, how big th - **Which features are switched on.** The modules present and enabled: audio, MoonLive, MQTT, Art-Net or E1.31 send and receive, panel cards, Ethernet versus WiFi. This is the half that answers "is anyone actually using this", which is what decides whether a feature earns its maintenance. - **A country**, derived at the server from the address the request arrives from, so the map is by region rather than by installation. -**What it must never carry**, and this list is a constraint on the design rather than a note on it: device name, IP or MAC address, WiFi credentials, MQTT passwords, any free-text field a user typed, the contents of a layout or a script, and anything that lets two reports be recognized as the same installation. There is no device identifier, which also means no de-duplication: a machine reporting twice counts twice, and that inaccuracy is the price of not being able to track anyone. It is the right trade. +**What it must never carry**, and this list is a constraint on the design rather than a note on it: device name, IP or MAC address, WiFi credentials, MQTT passwords, any free-text field a user typed, and the contents of a layout or a script. + +**The identifier: a random value per install, dropped after seven days** (settled 2026-09-08). Not derived from the MAC or from anything else about the hardware, so it cannot be reversed to a device even by whoever holds it. It exists to answer one question the aggregates cannot: how many DEVICES, rather than how many reports, and therefore how many installs are still on an old version. The server keeps per-report rows for seven days, de-duplicates, then keeps only the aggregate and drops the id. + +Two options were weighed and rejected. **No identifier at all** was the original text here: simplest to promise, but it makes every per-device question unanswerable, and "how many are still on the old version" is one of the questions this whole entry exists to answer. **A hashed MAC** is what WLED shipped first (`sha1("WLEDUSAGE" + MAC)`) and it buys exactly one thing more than a random id, given reports are one-time: linking one device's reports across months. That is the tracking capability itself, it survives a factory reset and a reflash, an open-source salt over a MAC space that size is rainbow-tablable, and a stable pseudonymous identifier is personal data under GDPR rather than anonymous. WLED's own shipped version moved off it to an anonymous id, which is the same conclusion from people who had it running. **Consent.** Opt-in, from a prompt shown after a fresh install or an upgrade, with a decline that is as easy to click as the accept and is remembered. One report per install or upgrade, never a heartbeat. A user who declines transmits nothing at all, rather than transmitting a "declined" record. @@ -1316,6 +1320,95 @@ We build for hardware we cannot see. Which chips people actually run, how big th **Open questions for whoever picks this up.** Where the server runs and who administers it, since it is the first piece of infrastructure this project would own rather than borrow from GitHub. Whether the dashboard is public, which we would want it to be for the same reason the source is. Whether the desktop reports at all or only devices, given the desktop is the half we currently know nothing about. And what happens to a report from a version whose fields have since changed, because a schema that cannot be read a year later answers nothing. +**WLED ships this, and the dashboard is public**: [usage.wled.me](https://usage.wled.me). Worth reading before building anything, because it answers two of the questions above by example and disagrees with this entry on a third. + +What it shows: distributions by version, chip, matrix, flash size, PSRAM, release, LED count and filesystem usage; upgrade versus install events over six months, split by chip; which LED features, peripherals, integrations, usermods and bus types are in use; and device count by country. That list is close to what is proposed above, which is reassuring about the shape. + +How it is collected ([PR #5116](https://github.com/wled/WLED/pull/5116), merged 2025-11-27, superseding an earlier [#4342](https://github.com/wled/WLED/pull/4342)): a one-time POST to `usage.wled.me/api/usage/upgrade` when a persisted version file shows the firmware changed, behind a startup prompt offering Yes / Not Now / Never Ask, and suppressed entirely in AP mode. The server is open source ([netmindz/WLED_usage](https://github.com/netmindz/WLED_usage)), which is how they answer "is the deployed code the published code". + +**Where they differ from this entry, and it is the interesting part.** WLED sends a device id: the first design hashed the MAC (`sha1("WLEDUSAGE" + MAC)`, "unique but not reversible"), the shipped one an anonymous `deviceId`. Either way the point is de-duplication, and it is what lets their dashboard say how many DEVICES rather than how many reports. This entry deliberately refuses that, and pays for it in accuracy. Their earlier PR also discussed a retention split (per-device data for 7 days, aggregates longer), which is a middle position between the two: keep an identifier only long enough to de-duplicate, then drop it. Worth considering rather than assuming the strictest option is automatically right. + +### The smallest honest version + +Sketched 2026-09-08. Not started; the blocker is the server, not the firmware. + +**The firmware side is small, because the payload already exists.** Every field is a control or a +module name the device publishes today: `chip`, `cpu`, `flash`, `psram`, `sdk`, `firmware` and +`deviceModel` are SystemModule controls; `version` and the previous version are FirmwareUpdateModule +controls; the enabled modules are the state tree's own top level; the light setup is the Layouts +grid and the Drivers children. So this is a serializer over facts already in memory, not new +instrumentation. Budget it as one module of a few hundred lines, plus the consent prompt. + +**When it sends.** Once, when a persisted version file shows the firmware changed, which is the +shape WLED converged on after starting from a UDP spray every second. Never a heartbeat. Suppressed +in AP mode, where there is no internet and a device is usually mid-provisioning. + +**Consent.** A prompt on the first boot after an install or upgrade: Yes / Not Now / Never, with +Never remembered. A decline sends nothing at all, not even a "declined" record. The prompt says what +is in the report in one sentence, and links the privacy policy. + +**Ask WLED to host it, before building a server at all.** Investigated 2026-09-08, and their server +is closer to a fit than expected. + +[netmindz/WLED_usage](https://github.com/netmindz/WLED_usage) is Kotlin on Spring Boot with a MySQL +schema under Flyway migrations, a Docker compose deploy, and one endpoint: +`POST /api/usage/upgrade`, no auth, with the country derived server-side from an `X-Country-Code` +header. The dashboard is a single static `index.html` against a `/api/stats` controller. + +**It is already multi-project.** The `device` table carries a `repo` column (added 2025-12), there +is a `RepoHistory` entity, and EVERY stats query is written `(:repo IS NULL OR repo = :repo)`. So a +second project reporting under its own repo name is a supported case rather than a change they would +have to make, and our figures would not be mixed into theirs. + +**We adopt their schema unchanged, and drop the one field that does not fit** (settled 2026-09-08). +chip, version, previousVersion, releaseName, ledCount, isMatrix, flashSize, psramSize/Present, +fsUsed/Total, busCount and busTypes mean the same thing in both projects, so those figures are +directly comparable and projectMM can sit in a pooled overview rather than being a special case. +Our own vocabulary rides the free-form lists: ledFeatures, peripherals, integrations, and `usermods` +for the enabled-module names. That works because migration V2026040301 (2026-04) replaced twenty-odd +fixed boolean feature columns with three comma-separated lists, aggregated by counting whatever +values appear: adding projectMM's names costs their server nothing. + +The layout dimensions this entry originally wanted (width x height x depth, since we are 2D and 3D) +have no equivalent there, and are DROPPED rather than added. `ledCount` plus `isMatrix` answers most +of what they were for, and being identical to a schema someone else maintains is worth more than one +field. If a real question later needs the third dimension, a list value is the place for it. + +**Their payload is close to what this entry wants.** `UpgradeEventRequest` already carries +deviceId, version, previousVersion, releaseName, chip, ledCount, isMatrix, bootloaderSHA256, brand, +product, flashSize, partitionSizes, psramSize, psramPresent, repo, fsUsed, fsTotal, busCount, +busTypes, ledFeatures, peripherals, integrations, usermods. Every field we listed above maps onto one +of those except the layout dimensions, and `usermods` is the natural home for "which modules are +enabled". Sending it means shaping our report to their names, which is a small price for not owning +a server. + +**What to settle with them before relying on it.** The repository has NO LICENCE file, so +strictly nobody may reuse it, and it was last pushed 2026-05-31, so it is quiet rather than dead: +both are conversations rather than blockers, but they are conversations to have first. Then the real +questions: are they willing to take another project's reports at all, who administers the box and +what happens to our data if that person stops, does the public dashboard gain a repo selector or +would we render our own from their API, and does the seven-day identifier retention this entry +commits to match what their server actually does (their earlier PR discussed it; the shipped schema +keeps a `device` row keyed by id, which suggests it does not). + +If the answer is yes, this feature loses its blocker entirely and becomes a firmware change plus a +conversation. If it is no, the plan below stands. + +**The server is the whole cost, and it is a standing commitment rather than a feature.** It is the +first infrastructure this project would own rather than borrow from GitHub, and it needs an +administrator, a domain, TLS, a retention job that actually runs, and a public dashboard. WLED +publishes their server ([netmindz/WLED_usage](https://github.com/netmindz/WLED_usage)), which is how +they answer "is the deployed code the published code": whatever runs here should be public for the +same reason the firmware is. Until someone owns that, this feature cannot ship honestly, and that is +the reason it is still in the backlog rather than in a branch. + +**Order of work, and the first two are worth doing whether or not the rest ever lands.** Write the +privacy policy revision FIRST, since it is the promise everything else has to match, and the current +page says plainly that nothing of the kind exists. Then the report BUILDER as a pure function over +the state tree, with a unit test asserting that the forbidden fields cannot appear in its output: +that test is the design constraint made executable, and it is worth having even if nothing ever +sends. Only then the consent prompt, the one-time trigger, and last the server. + ## A driven GPIO the Pins map never sees: bus padding, and a hidden clockPin **Found:** 2026-08-21, on MM-S31, after a bench session that started as "the LED panel stopped working" and cost hours chasing a firmware regression that did not exist. @@ -1445,3 +1538,140 @@ the rows should be. Until then a list is user-populated, which is the honest behavior: the device knows the pin, the user knows what the button should do. + +## P4 WiFi cascade into a dead co-processor link aborts the board (2026-09-08) + +Bench, MHC-WLED ESP32-P4 shield on `esp32p4rev1-eth-wifi`, no Ethernet cable: the NetworkModule +cascades to WiFi, and the board reboots every ~22 s with `task_wdt: main (CPU 0)` while `IDLE0` +runs. The main task is not spinning, it is BLOCKED: on the P4 every `esp_wifi_*` call is forwarded +over SDIO to the ESP32-C6 by `esp_wifi_remote`, and this shield's C6 never completes the ESP-Hosted +handshake (`E H_API: ESP-Hosted link not yet up` at boot), so `esp_wifi_init` and the calls after it +each wait out their timeout, on the render thread, past the 5 s watchdog. The wait is +esp_hosted's `transport_drv.c` slave-ready loop: 200 ms polls, a slave reset every 50 of them (the +`Reset slave using GPIO[54]` lines at 10 s and 21 s), up to MAX_RETRY_TRANSPORT_ACTIVE = 100 +polls, so worst case ~20.0 s in the caller's task, which is ours. Not the HWLOOP/FFT +erratum: audio was ruled out by the watchdog text itself (a blocked main task with idle running, +not a spinning core). Control experiment, same image on the bench P4 (.139, same shield model, +esp_hosted host 2.12.13): its SDIO card init succeeds (`Card init success, TRANSPORT_RX_ACTIVE`), +it never associates either, but the main task keeps ticking, it falls back to its own AP, and it +rejoins Ethernet when the cable returns. The new shield never prints a card-init success: its C6 +does not answer the bus (no slave firmware, or one that does not match the 2.12.x host). + +Two defects are ours, whatever the C6 carries: + +1. **The cascade does not consult the link.** `coprocessorWifi()` in `platform_esp32.cpp` already + asks the C6 for its firmware version, bounded to two attempts, exactly to detect an absent or + incompatible slave. The WiFi cascade never asks; it walks straight into `esp_wifi_init`. Gate the + cascade on that answer (or on `esp_hosted` reporting the link up) and degrade to "no network, + C6 not answering" as a Network status. +2. **A failing cascade must not abort.** Robustness says degrade visibly, never crash. Even with the + gate, a link that dies later would hit the same watchdog: the forwarded calls need to run off the + render thread, or with a timeout shorter than the watchdog, so the worst case is a status line. + +Practical today: the shield runs the eth-only image without WiFi, or its C6 gets the ESP-Hosted slave +firmware flashed (the `ships: false` note on the variant in `build_esp32.py` says why that is not +yet reproducible). The Improv script's eth-only rule and the catalog's two firmwares for the shield +are already in place. + +**Everything tried on 2026-09-08 was reverted; both P4s now run `esp32p4rev1-eth` and are stable.** +The tree is back to its pre-attempt state except for the catalog, which now lists BOTH P4 firmwares +for the MHC-WLED shield so the installer offers the WiFi variant at all (it listed only the eth one, +which is why the WiFi image could not be picked). What the day established, so the next attempt does +not repeat it: + +- **The C6 link is INTERMITTENT on this shield, not simply dead.** After the vendor C6-update tool + ran (it never completed an OTA: it looped on "Not able to connect with ESP-Hosted slave device"), + the link came up and the board reached a DHCP lease on WiFi, first attempt, no retries. A power + cycle later it was back to `sdmmc_send_cmd returned 0x107` and a reboot loop. Any future fix has + to survive a cold boot, not one lucky session. +- **A patient STA retry (MoonLight's 5 s cadence, ~2 min budget) was implemented and reverted.** It + never fired on either board: both connected on the first attempt when the link worked, and when it + did not the board died before the render loop. Keeping untested robustness was not worth the + surface. The reasoning still holds and is worth redoing WITH a repro: MoonLight retries forever + and never tears the radio down, and its own comment warns that toggling WiFi on a co-processor + board forces costly esp_hosted reinit cycles, which is exactly what our AP fallback does. +- **Espressif does not endorse retrying into readiness.** Their esp-hosted troubleshooting puts + "not able to connect with slave" and SDIO 0x107 down to wiring, pull-ups, signal integrity, or a + host/slave VERSION MISMATCH: "use the same version for master and slave". Our host is 2.12.13; the + shield's C6 is on the factory build the vendor tool calls 0.0.6 (target 2.0.17). Updating the C6 + needs Method 2 (direct USB/UART to the C6), since Method 1 needs the very link that is broken. + +**A fail-fast gate was tried on the bench (2026-09-08) and REVERTED.** The idea was right and both +signals were wrong. `platform::wifiHardwareReady()` gated the STA and AP init, and it STOPPED the +reboot loop dead: the shield ran 80 s with 0 reboots and 59 ticks, logging "WiFi co-processor link +not up" instead of aborting. But on the bench P4 (.139), whose link demonstrably works, the same +predicate read FALSE at its 26 s cascade and refused WiFi on a healthy board. Two signals were +tried, neither is usable as a readiness test: + +- **`ESP_HOSTED_EVENT_TRANSPORT_UP`**, subscribed on the default event loop (both lazily and from + `ensureNetifInit`, i.e. before esp_hosted's task posts it). Never observed arriving. esp_hosted + posts through its own `g_h.funcs->_h_event_post` indirection; where that lands was not chased. +- **`esp_hosted_get_coprocessor_fwversion` returning a non-zero version.** Also false on .139, which + matches what `coprocessorWifi()` already records: on a live link this RPC times out rather than + answering, which is why that function is bounded to two attempts. + +**Not a regression: the RELEASED v4.0.0 image fails the same way on this shield** (bench, same day). +Flashed from the web installer after an erase, it logs `App version: v4.0.0` and then: + +``` +W (13438) H_SDIO_DRV: Reset slave using GPIO[54] +W (13438) gpio: conflict found for GPIO[54] +E (14988) sdmmc_io: sdmmc_io_rw_extended: sdmmc_send_cmd returned 0x107 (ESP_ERR_TIMEOUT) +E (14988) H_SDIO_DRV: failed to read registers +``` + +and reboots without ever reaching the render loop. The same v4.0.0 image serves WiFi on the bench +P4 (.139) with no SDIO error and no GPIO 54 conflict, and no projectMM config on any P4 entry +references GPIO 54 (esp_hosted drives it as the slave reset). The two boards differ in silicon +revision: this shield is chip rev **v1.0**, .139 is **v1.3**. So the C6 side of this shield does not +come up, which is a board/slave-firmware matter rather than anything in our WiFi path, and the +firmware's job is only to degrade rather than reboot. + +**The product owner reports this same shield ran WiFi under MoonLight on IDF 5.5**, which makes a +dead C6 unlikely and points at host-side SDIO configuration on 6.1. What was checked (2026-09-08): + +- The SDIO data pins are NOT board-preset dependent: they come from the SLOT choice + (`ESP_HOSTED_SDIO_SLOT_1`, fixed silicon pins), so swapping `ESP_HOSTED_P4_DEV_BOARD_*` presets + would not move them. The preset mostly moves SPI pins, which we do not use. +- `ESP_HOSTED_SDIO_GPIO_RESET_SLAVE` defaults to **54 on any P4** regardless of preset, so the + `gpio: conflict found for GPIO[54]` line is esp_hosted resetting the slave twice, not a wrong pin + from our config. No projectMM P4 entry references 54. + +**Our SDIO configuration is not the difference.** Diffed against a known-good local reference +(`ewowi/FlowFields/sdkconfig.esp32-p4`, an IDF 5.5-era P4 build), every hosted setting is IDENTICAL: +pins (CLK 18, CMD 19, D0-D3 14-17), `GPIO_RESET_SLAVE` 54, `RESET_ACTIVE_HIGH=y`, `CLOCK_FREQ_KHZ` +40000, `SLOT_1`, `4_BIT_BUS`, `RESET_DELAY_MS` 1500, `RX_STREAMING_MODE`. So reset polarity, reset +pin and SDIO clock are all ruled out as differences, and the identical released v4.0.0 image works +on .139 and not on this shield. Same firmware, same config, two boards, two outcomes: what remains +is on the board side (C6 slave firmware, power/strapping, or the SDIO traces on this revision), and +the only projectMM work left is the degrade-instead-of-reboot fix above. + +What esp_hosted's private `is_transport_tx_ready()` reports is the signal the slave-ready loop +itself polls, but it lives in a PRIVATE include dir (`host/drivers/transport`, not in the +component's `pub_include`), so reaching it means either adding that dir to our include path or +asking upstream to export a readiness getter. That is the next thing to try, and it needs the +.139-still-associates control run alongside the shield-stops-rebooting one: the fix is only right +when BOTH hold. + +## The persisted `firmware` variant survives a flash to a different variant (2026-09-08) + +`SystemModule` writes the compile-time `kFirmwareName` into its `firmware` control at +`defineControls()`, and its own comment states the intent: "written from kFirmwareName on every boot +rather than read from the file: the compile-time constant is the truth". But the control is +`addText`, which is PERSISTED, and the config load runs after `defineControls()`, so a saved value +from a previous image overwrites the compile-time one. + +Bench (MHC-WLED P4 shield, flashed from `esp32p4rev1-eth-wifi` to `esp32p4rev1-eth`): the Firmware +card correctly reported `esp32p4rev1-eth` (it reads the running image), while `/api/modules/System` +still reported `esp32p4rev1-eth-wifi` from the old config. The two disagreed on the same board, and +the stale one is the field an outside reader trusts. + +Why it matters beyond cosmetics: the comment explains this value exists so **MoonBase** can narrow +the recovery image list to one variant. A stale value points a recovering board at the wrong image, +which is the failure that list exists to prevent (picking an `esp32s3-n16r8` build for a Zero +installs a flash layout the board does not have). + +The fix has to keep the value readable by another image (that is why it is persisted at all) while +making the compile-time constant win: re-assert `kFirmwareName` after the config load rather than +only at `defineControls()`, and pin it with a test that loads a config naming a DIFFERENT variant +and checks the control still reads the compiled one. diff --git a/docs/backlog/backlog-light.md b/docs/backlog/backlog-light.md index 1da07234..e436ce4d 100644 --- a/docs/backlog/backlog-light.md +++ b/docs/backlog/backlog-light.md @@ -447,7 +447,7 @@ where upscaling has the least to offer. **The fix when it earns its place:** iterate the OUTPUT rows rather than the input lights, so writes are sequential: for each output row, walk its source row once and emit `scale` copies of -each light's colour, then `memcpy` that finished row to the remaining `scale - 1` rows of the +each light's color, then `memcpy` that finished row to the remaining `scale - 1` rows of the block. Same output, one pass through the destination in address order. ### Sprite follow-ups (draw::sprite + FlyingToasters shipped; spec + plan in the plans archive) @@ -489,16 +489,26 @@ a codec. So the three things a Tab5 could be are separate pieces of work, and on ### Multi-card walls — does a daisy chain work today? (open, ask before building) -The ColorLight format has **no card addressing**: the destination MAC is a fixed constant and every card filters on it, so every card on a segment shows the same image. A user with six cards on a switch observed exactly that. +The ColorLight format has **no card addressing in the PIXEL path**: the destination MAC is a fixed +constant and every card filters on it, so every card on a segment shows the same image. A user with +six cards on a switch observed exactly that. -The industry-standard answer is **daisy-chaining** — a sending card's ports each drive a chain, and each card takes its region by position in the chain. That user works around it with per-card VLANs and a managed switch instead, which he built for throughput and for per-card colour-temperature grouping across mixed panel batches; he described it as his own solution, not a standard. +The DISCOVERY path does distinguish them. A discovery reply (0x08) carries a controller number at +payload offset 0x62, and the acknowledgement echoes it plus one, which is how a sender tells several +cards apart. Documented by a reader of [Harald Kubota's protocol +write-up](https://hkubota.wordpress.com/2022/01/31/winter-project-colorlight-5a-75b-protocol/) and +confirmed by its author. That is an identity, not a destination: it does not let a sender aim pixel +data at one card, so the same-image behavior above stands. It is what a per-card brightness or +color-temperature feature below would key on. + +The industry-standard answer is **daisy-chaining** — a sending card's ports each drive a chain, and each card takes its region by position in the chain. That user works around it with per-card VLANs and a managed switch instead, which he built for throughput and for per-card color-temperature grouping across mixed panel batches; he described it as his own solution, not a standard. **Establish first whether a daisy chain already works with projectMM** (one contact has a 96K daisy-chained rig). If the cards self-assign by chain position, the standard multi-card case is already solved and nothing is needed. Only if it does not work is there a feature here, and it should follow the daisy-chain standard rather than the VLAN workaround. 802.1Q tagging is technically a clean fit for a raw-L2 sender (the tag is part of the Ethernet header, the switch strips it before the card, so card firmware is unaffected), but it serves one bespoke architecture. ### Smaller asks from the same thread - **Read the wall layout from the ColorLight cards.** The cards can report their configuration and at least one user's own tool already does it; it would remove the manual layout step. -- **Per-card colour temperature and brightness**, via the ColorLight sync-packet bytes, grouped by sync group — used to colour-match mixed panel batches live. +- **Per-card color temperature and brightness**, via the ColorLight sync-packet bytes, grouped by sync group — used to color-match mixed panel batches live. - **Docker image**, asked for by a user tracking updates in an IoT system. The Linux binary and `.deb` already ship, so this is packaging rather than new capability. ## Sensors and audio-reactive input @@ -867,3 +877,60 @@ index on load, so a stored selection survives a re-sort. `paletteScript` already file name for the editor, so the value exists; what is missing is using it as the authority when the list changes. The alternative, appending new scripts rather than sorting them, keeps indices stable but makes the picker unreadable as the list grows, which is the trade the sort was chosen over. + +## Move the remaining board entries off RmtLedDriver (2026-09-08) + +`RmtLedDriver` expands every bit into a 32-bit hardware symbol, so its buffer costs +**lights x channels x 8 x 4 = 96 bytes per light**. `ParallelLedDriver` bit-bangs the lanes through +one I2S/LCD_CAM transfer and costs **384 bytes flat**, independent of pin count and light count. +Measured on the bench, same hardware and same light count either side: + +| Board | Lights | RmtLed | ParallelLed | FPS | +|---|---|---|---|---| +| QuinLED Dig-Octa 32-8L (8 pins) | 512 | 49,155 B | 384 B | 99 to 407 | +| QuinLED Dig-Next-2 (.186 vs .122) | 256 | 24,579 B | 384 B | - | + +On a classic ESP32 with ~320 KB of internal DRAM that is the difference between 24 KB and 61 KB of +largest contiguous block, which is what a large allocation actually fails on. + +Both those entries are switched. **Twenty entries still specify `RmtLedDriver`**, and most of them +should stay that way: measured on the QuinLED Dig-2-Go (one lane, 256 lights, no PSRAM), the swap +CUT the driver's own readout from 32,772 to 512 bytes and LOST 49 KB of free heap (96,552 to +47,156, steady after a reboot). The i80 DMA frame is sized by the bus width, not the pins in use, so +a one-lane board pays the same ~50 KB as an eight-lane one, while RMT costs 96 bytes per light. The +crossover is around 500 lights: below it RMT is cheaper, above it ParallelLed is. (That fixed frame +was invisible on the card until `driverHeapBytes()` started counting it.) + +For the boards where it does pay, the blocker is not the driver: it is that each one needs a **DC +pin chosen against that board's real pinout**, and picking one blind is how a peripheral lands on a +pad the package does not have ([lessons](../history/lessons.md), PICO-V3-02: silent TG1WDT, PC at +panicHandler). + +**Two cost classes, and they differ by chip.** On a classic ESP32 only DC costs a GPIO: WR is routed +through the GPIO matrix to SENSOR_VP (36), bonded on every classic package and driving nothing, so +`clockPin` can stay -1. On S3 / P4 / S31 the LCD_CAM backend needs a real pad for **both** WR and DC +(`platform_esp32_i80.cpp`), so those boards pay two pins for a saving that is small at one lane. + +**What each board needs**, in order: find a free output-capable GPIO for DC (avoiding strapping +0/2/5/12/15, flash 6-11, input-only 34-39, and whatever the entry already spends on Ethernet, relays, +audio or buttons); set it on real hardware and confirm the lights still run; then fold the verified +values into `deviceModels.json`. Step two is the one that counts, and it is why this is a per-board +job rather than a sweep. + +- **Classic, one free GPIO needed (13):** Dig-Quad V3, Dig-Uno V3, Cube 2020-10 (10 pins, the + largest RMT saving left), MHC V4.3, MHC V5.7 PRO, ESP32-WROVER, Dig-2-Go, Serg MiniShield, Serg + UniShield V5, Yves V4.8, MM testbench ESP32-16MB, MM testbench classic olimex. Shelly is on the + list but is the read-only old-firmware rig, so it changes only with the product owner's say-so. +- **S3 / S31, two free GPIOs needed (4):** ESP32-S3 N16R8 Dev, ESP32-S3-Zero (N4R2), MM testbench S3, + Espressif ESP32-S31 CoreBoard. Worth checking the saving is worth two pins at one lane before + switching these. +- **No pins defined (3):** Generic ESP32 Dev, LOLIN D32, Olimex ESP32-Gateway Rev G. The user supplies + pins, so the default driver matters less and they still have to supply DC. + +Open question for the boards nobody physically has: propose a DC pin from the vendor pinout and mark +the entry unverified, or leave them on RMT until someone can test one. RMT stays correct either way, +it is only more expensive. + +Two things to fix while in here: **ESP32-S3-Zero (N4R2) defines both** a `ParallelLedDriver` (pin 2) +and an `RmtLedDriver` (pin 21), and **MM testbench S3 defines two `RmtLedDriver`s** (pins 38 and 18). +Both may be deliberate (independent outputs), but they are the only entries shaped that way. diff --git a/docs/history/plans/Plan-20260908 - Stream the WebSocket state instead of buffering it (attempted, reverted).md b/docs/history/plans/Plan-20260908 - Stream the WebSocket state instead of buffering it (attempted, reverted).md new file mode 100644 index 00000000..59d29afa --- /dev/null +++ b/docs/history/plans/Plan-20260908 - Stream the WebSocket state instead of buffering it (attempted, reverted).md @@ -0,0 +1,191 @@ +# Plan: stream the WebSocket state instead of buffering it whole + +> **Reverted the same day.** The design was built and verified on the bench (the Dig-Octa received +> its 44,539-byte state as 6 fragments, byte-identical to `/api/state`, where it had been truncating +> at 30,719), and then backed out: the chain still holds the WHOLE document, as slices, for the +> duration of the drain. On a 65 KB board a UI refresh (a second `/ws` client mid-drain, the `/wsp` +> preview, plus `/api/types` and `/api/state` fetches) took the heap under what lwIP needs and the +> board crashed. The bench check had been one raw client on one socket, which proved the framing and +> not the memory budget. +> +> The product owner's call was to reconsider the architecture instead: **snapshot over HTTP, deltas +> over WebSocket**. `GET /api/state` already streams through a 1 KB socket-mode sink with no document +> in RAM, and the value patches already exist; what is missing is the UI fetching the snapshot on WS +> open and a small `{"resync":true}` on a structural change. That deletes the full-state-over-WS path +> rather than shrinking it. Tracked in [backlog-core.md](../../backlog/backlog-core.md). +> +> Kept from this work: `JsonSink` now FLAGS a refused heap grow instead of truncating silently +> (`unit_JsonSink_overflow`), which is the bug that made a cut document indistinguishable from a +> whole one. + +## The problem, measured + +A classic ESP32 cannot build its own state document. `buildStateJson` serializes the whole module +tree into one heap buffer, and on the bench (2026-09-08) that document is **44,261 bytes** on the +QuinLED Dig-Octa and **42,365** on the Olimex. The buffer grows by doubling, and old and new are both +live across the `memcpy`, so reaching 64 KB asks for roughly 96 KB of CONTIGUOUS heap. The Dig-Octa +has 57 KB as its largest block. + +The failure was silent and total. `JsonSink::append` dropped on a refused allocation and returned +without a flag, `buildStateJson` finished "successfully" holding a truncated document, and +`startBufferedTextSend` wrote a correct WebSocket header declaring it complete. The browser parsed +it, threw, and dropped every module past the cut: **effect cards simply vanished**, with nothing +logged on the device or in the console. Both classic boards shipped a cut document, the Dig-Octa +losing 11,226 bytes and the Olimex 8,101. + +Two fixes already landed and neither is sufficient: + +- `JsonSink::append` now sets `overflowed_`, so a truncated document is detectable. +- `ensureHeap` backs off in quarters when a doubling is refused, so growth lands on a size the heap + can serve rather than giving up. That moved the Dig-Octa's cut from 16,383 to 30,719 bytes. It + still does not reach 44,261, and no back-off can: grow-and-copy needs old plus new at once. + +One dead end is worth recording. Refusing to send an overflowed document (returning early) FROZE THE +WHOLE UI: `fullResyncPending_` stayed set, the else-branch that pushes value patches was never +reached, and the header stopped updating fps and free heap. A partial tree that keeps updating beats +a whole one that never arrives. That refusal was replaced by a warning. + +**Ethernet dissolves it and is not the answer.** A board on Ethernet never starts the WiFi stack and +keeps ~50 KB more heap, which is why the Olimex (Ethernet) looks healthy and the Dig-Octa (WiFi STA) +does not. Most users are on WiFi, so the fix has to work there. + +## What the current design buys, and must keep + +The buffer is not laziness. Three constraints hold it up, and a replacement has to satisfy all three: + +1. **The render loop must never block.** `drainStateSend` writes what the socket takes on `tick20ms` + and leaves the rest, bounded by `drainChunkBytes()` (a fraction of the largest free block, floor + 2048, ceiling 65536). +2. **Clients drain at their own pace.** `stateSend_.sent[MAX_WS_CLIENTS]` is a per-client cursor over + the same body; a slow client lags without holding the others. +3. **A message stays atomic per client.** While a state send is active, patch and WLED pushes to + `/ws` are skipped so nothing interleaves inside one WebSocket message. + +## Approach: fragment the message, keep one modest buffer + +WebSocket messages may be split across frames: a first frame carrying the real opcode with FIN +clear, then continuation frames (opcode 0x0), the last with FIN set. The browser reassembles them +into one `message` event, so **the client needs no change at all**. + +The state is then produced and sent in slices of a few KB. The peak allocation becomes one slice +rather than the whole document, which removes the contiguous-block requirement entirely. + +The one hard question is that `buildStateJson` is a single forward walk of the module tree: it cannot +be resumed from an arbitrary byte offset. Three ways to reconcile that with per-client cursors, and +the plan takes the third: + +- **A. Serialize per client, straight to the socket.** `JsonSink` already has socket mode. Simplest + to write, but it re-walks the whole tree once per client and blocks on a slow socket, breaking + constraint 1. +- **B. Resumable serialization.** Teach `buildStateJson` to stop at a slice boundary and resume from + a saved position in the tree. No large buffer at all, but it means a serialization cursor + (module index, control index, partial-value state) that has to stay correct while the tree can + change underneath it. The most invasive option and the easiest to get subtly wrong. +- **C. Slice-at-a-time with a shared buffer (chosen).** Serialize into a fixed slice buffer, send + that slice as one fragment to every client, and only then produce the next. Peak memory is one + slice; the tree is walked once; no per-client re-serialization. The cost is that the slowest client + paces the others, which is acceptable: the state frame is rare (connect and structural change) and + a slow client already stalls behind its own cursor today. + +## As built (2026-09-08) + +The word "ring" in option C2 was misleading, and the design that landed is simpler. The constraint +was never TOTAL memory (the Dig-Octa has 62 KB free for a 44 KB document) but CONTIGUOUS memory (a +49 KB largest block against a doubling buffer that needs ~96 KB). So the document is serialized ONCE, +at push time, into a **chain of small slices**, each its own allocation of `drainChunkBytes()` bytes, +which any fragmented heap can serve. Nothing pauses the walk and nothing waits for a client. + +- **`JsonSink` slice mode** (`JsonSink.h`): a caller-owned slice plus a `FlushFn`; a full slice is + handed over and reused, `finish()` flushes the tail, a failed flush sets `overflowed()`. +- **`WsSliceChain`** (`WsSliceChain.h`, header-only like `JsonSink`): the flush target. Each slice + carries `kHdrMax` bytes of frame-header room in front of its payload, so `finalize(opcode)` stamps + the fragment headers in place (first: opcode with FIN clear; middle: 0x00; last: 0x80; a single + slice: 0x81, byte-identical to the old framing) and every frame is one contiguous span. A per-client + `Cursor` walks the chain; slices are shared, cursors are not. `writeWsFrameHeader` moved here. +- **`HttpServerModule`**: the push allocates one staging slice, serializes through it into + `stateSend_.chain`, frees the staging slice, and arms `startStateChainSend()`. `drainStateSend` + moves bytes from `remaining()` and `advance()`s; the chain is freed once every live client is done. + A refused allocation mid-document clears the chain, warns, and clears `fullResyncPending_` so the + value patches keep flowing (the early-return that froze the whole UI is the dead end above). + +**Tests**: `unit_JsonSink_slices` (4 cases) and `unit_WsSliceChain` (6 cases: the exact frame +sequence, the single-slice framing unchanged, byte-for-byte reassembly at socket writes of 1, 7, 333 +and 100000 bytes, eight clients at eight speeds on one chain, a refused allocation refusing to +finalize, clear-and-reuse). Both control-checked: sabotaging the tail flush or the last frame's FIN +fails exactly the test that pins it. 1,910 unit tests green. + +**Bench** (Dig-Octa .181, WiFi, 65 KB free, 59 KB largest block, the board that lost its effect +cards): a raw WebSocket client received the state as **6 frames** (opcode 1 with FIN clear, four +continuations, FIN on the last: 5 x 7,935 + 4,864 bytes) and reassembled **44,539 bytes, +byte-identical to `/api/state`**, with both effects present. The first flash sent nothing: the +document holds formatted fragments over 256 bytes, and `appendf`'s long-fragment branch flagged +overflow in slice mode instead of flushing, so every chain was dropped. Fixed and pinned +(`unit_JsonSink_slices`, the long-fragment case). + +## Steps + +### 1. A slice-sized sink +`JsonSink` gains a mode that fills a caller-owned fixed buffer and, when full, hands it to a callback +before continuing. Fixed-buffer mode already exists and flags overflow instead of growing; this is +that path plus a flush hook, so no new buffer strategy is introduced. Slice size comes from +`drainChunkBytes()`, so a tight board takes small slices and a roomy one takes large. + +### 2. Fragmented framing +`writeWsFrameHeader` already encodes any length; it needs the FIN bit and opcode as parameters +rather than always `0x81`. First slice: opcode 0x1, FIN clear. Middle: opcode 0x0, FIN clear. Last: +opcode 0x0, FIN set. A single-slice document keeps today's exact framing (0x81), so the common small +case is unchanged. + +### 3. Slice-aware StateSend +`stateSend_` holds one slice plus its header, with the per-client cursor over THAT slice. When every +live client has drained it, the next slice is produced and the cursors reset. Add a +`slicesRemaining` / `lastSlice` flag so the drain knows when the message is complete and only then +clears `fullResyncPending_` and rebaselines the leaf hashes. + +### 4. Failure paths +A client that dies mid-message is dropped as today. A client that connects DURING a send must not +receive a half message: it waits for the next full state (`fullResyncPending_` is set on connect +anyway). If a slice cannot be allocated at all, warn and keep the patch stream alive, which is the +lesson from the refusal that froze the UI. + +## Tests + +- `unit_JsonSink_slices`: a document larger than the slice buffer emits the expected sequence of + slices, and reassembling them byte-for-byte equals the same document built in buffer mode. +- `unit_HttpServer_fragmented_state`: the frame sequence for a multi-slice document is + `0x01 FIN=0`, `0x00 FIN=0`, ..., `0x00 FIN=1`, and a single-slice document is exactly `0x81 FIN=1` + (today's framing, so small devices are untouched). +- A regression test for the original bug: a state document that exceeds any single allocation still + arrives complete and parseable. +- Scenario: the existing WS state scenario re-run, with the observation block showing the tick cost + did not regress. + +## Verification + +Desktop first. Then on the bench, both classic boards, since they bracket the problem: + +1. **Dig-Octa (.181, WiFi, 57 KB max block, 44 KB state)**: the whole tree arrives, every effect card + renders including the MoonLive one that vanished, and the header keeps updating. +2. **Olimex (.210, Ethernet, 102 KB max block)**: unchanged behavior, no regression on a board that + was already comfortable. +3. A raw WebSocket client asserts the reassembled document is valid JSON and matches `/api/state` + byte-for-byte. +4. Watch the tick cost on both: slicing runs on `tick20ms` and must not lengthen the render tick. + +## Risks + +- **A half-sent tree is worse than a truncated one.** If the last fragment is lost the browser holds + an incomplete message forever. The FIN bookkeeping is the load-bearing part and the test above + pins it directly. +- **The tree can change between slices.** A module added or removed mid-message would produce a + document that is internally inconsistent. Mitigation: a structural change during a send sets + `fullResyncPending_` again, so the next full state supersedes it, exactly as today. +- **Slower clients now pace each other.** Accepted, and reversible: per-client slice buffers would + undo it at the cost of the memory this plan exists to save. + +## Out of scope, worth noting + +`LightPresets` alone is **13,086 bytes, 30% of the document**: one `list` control holding 13 built-in +presets that never change at runtime. Fetching it on demand rather than pushing it in every state +would cut the document by a third and help every board. Real, separate, and not a substitute for +streaming, since 31 KB still would not fit on a tight board. diff --git a/docs/metrics/repo-health.json b/docs/metrics/repo-health.json index 8fc21bd1..d0c41000 100644 --- a/docs/metrics/repo-health.json +++ b/docs/metrics/repo-health.json @@ -1,47 +1,49 @@ { - "commit": "52e03cbe", + "commit": "fab5302d", "flash": { "esp32s3-n16r8": 2100032, - "desktop": 1909016, - "esp32": 2059664, - "esp32p4rev1-eth": 1979232, - "esp32p4rev1-eth-wifi": 2019392, + "desktop": 1909464, + "esp32": 2057984, + "esp32p4rev1-eth": 1995280, + "esp32p4rev1-eth-wifi": 2284640, "esp32s3-n8r8": 2087168, "esp32s31": 2348592, - "esp32-16mb": 1809472, + "esp32-16mb": 2058000, "esp32-eth": 1397456, "esp32-wrover": 1843760, "qemu": 1383648, "esp32p4rev3-eth": 1643760, "esp32s3-zero": 2024192, - "esp32-pico": 2071584 + "esp32-pico": 2106464 }, "measured": { "esp32p4rev1-eth": "2026-09-08", "esp32s31": "2026-09-06", "esp32": "2026-09-08", - "esp32-pico": "2026-09-06", + "esp32-pico": "2026-09-08", "esp32s3-n16r8": "2026-09-08", "desktop": "2026-09-08", "esp32s3-n8r8": "2026-09-08", - "esp32s3-zero": "2026-09-08" + "esp32s3-zero": "2026-09-08", + "esp32-16mb": "2026-09-08", + "esp32p4rev1-eth-wifi": "2026-09-08" }, "perf": { "desktop": { - "tick_us": 140, - "fps": 7142, + "tick_us": 834, + "fps": 1199, "scenario_p50": { "Layer_base_pipeline": { "p50": 71, "p95": 197, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "Layer_memory_1to1": { "p50": 5, "p95": 40, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" } } }, @@ -52,10 +54,10 @@ "scenario_matrix": { "MoonModule_control_change": { "desktop-macos": { - "p50": 143, + "p50": 133, "p95": 248, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "esp32-eth-wifi": { "p50": 89895, @@ -168,10 +170,10 @@ }, "Audio_mutation": { "desktop-macos": { - "p50": 26, - "p95": 70, + "p50": 25, + "p95": 60, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 40, @@ -194,10 +196,10 @@ }, "Aurora_fps": { "desktop-macos": { - "p50": 1518, - "p95": 1828, - "n": 26, - "last": "2026-09-07" + "p50": 1526, + "p95": 1920, + "n": 27, + "last": "2026-09-08" } }, "Driver_mutation": { @@ -205,7 +207,7 @@ "p50": 20, "p95": 88, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 42, @@ -231,7 +233,7 @@ "p50": 148, "p95": 768, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 549, @@ -244,16 +246,16 @@ "desktop-macos": { "p50": 1239, "p95": 1686, - "n": 27, - "last": "2026-09-07" + "n": 28, + "last": "2026-09-08" } }, "Fluid_solver": { "desktop-macos": { "p50": 222, "p95": 259, - "n": 20, - "last": "2026-09-07" + "n": 21, + "last": "2026-09-08" } }, "GridBlacks_blackpixel": { @@ -261,7 +263,7 @@ "p50": 2, "p95": 16, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "esp32s3-n16r8": { "p50": 267, @@ -284,10 +286,10 @@ }, "GridLayout_resize": { "desktop-macos": { - "p50": 127, + "p50": 126, "p95": 311, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "esp32-eth-wifi": { "p50": 82231, @@ -331,7 +333,7 @@ "p50": 71, "p95": 197, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 118, @@ -345,7 +347,7 @@ "p50": 5, "p95": 40, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 1, @@ -356,10 +358,10 @@ }, "Layouts_mutation": { "desktop-macos": { - "p50": 97, + "p50": 96, "p95": 248, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 111, @@ -409,8 +411,8 @@ "MoonLiveEffect_livescript": { "desktop-macos": { "p50": 6, - "p95": 8, - "n": 16, + "p95": 11, + "n": 17, "last": "2026-09-08" }, "esp32s3-n16r8": { @@ -461,7 +463,7 @@ "p50": 6, "p95": 21, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 1, @@ -475,7 +477,7 @@ "p50": 3, "p95": 19, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 3, @@ -489,7 +491,7 @@ "p50": 126, "p95": 283, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 225, @@ -502,8 +504,8 @@ "desktop-macos": { "p50": 360, "p95": 442, - "n": 21, - "last": "2026-09-07" + "n": 22, + "last": "2026-09-08" } }, "modifier_chain": { @@ -511,7 +513,7 @@ "p50": 44, "p95": 112, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 69, @@ -531,7 +533,7 @@ "p50": 24, "p95": 87, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "esp32-eth": { "p50": 1010, @@ -566,10 +568,10 @@ }, "perf_full": { "desktop-macos": { - "p50": 279, + "p50": 278, "p95": 1114, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "esp32s3-n16r8": { "p50": 16915, @@ -601,7 +603,7 @@ "p50": 17, "p95": 61, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "esp32s3-n16r8": { "p50": 2485, @@ -642,10 +644,10 @@ "last": "2026-07-25" }, "desktop-macos": { - "p50": 296, + "p50": 285, "p95": 881, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "desktop-windows": { "p50": 649, @@ -671,7 +673,7 @@ "p50": 4, "p95": 14, "n": 32, - "last": "2026-09-07" + "last": "2026-09-08" }, "esp32p4rev1-eth": { "p50": 217, @@ -695,54 +697,54 @@ } }, "loc": { - "core": 26039, - "light": 35627, - "platform": 18503, - "ui": 10621, - "test": 57455, - "moondeck": 22710 + "core": 26096, + "light": 35658, + "platform": 18584, + "ui": 10651, + "test": 57757, + "moondeck": 22898 }, "comments": { "core": { - "lines": 10452, - "ratio": 0.433 + "lines": 10488, + "ratio": 0.434 }, "light": { - "lines": 13532, + "lines": 13553, "ratio": 0.417 }, "platform": { - "lines": 6500, + "lines": 6534, "ratio": 0.385 }, "ui": { - "lines": 3164, - "ratio": 0.314 + "lines": 3182, + "ratio": 0.315 }, "test": { - "lines": 10784, + "lines": 10833, "ratio": 0.215 }, "moondeck": { - "lines": 3690, - "ratio": 0.186 + "lines": 3716, + "ratio": 0.185 } }, "tests": { - "cases": 1992, + "cases": 2000, "scenarios": 27 }, "docs": { - "md_files": 220, - "md_lines": 36832, - "plans_files": 116, - "backlog_lines": 6794, + "md_files": 221, + "md_lines": 37324, + "plans_files": 117, + "backlog_lines": 7091, "lessons_lines": 705, "claude_md_lines": 259 }, "complexity": { - "functions": 3498, - "over_threshold": 249, + "functions": 3506, + "over_threshold": 250, "worst_ccn": 108 } } diff --git a/docs/metrics/repo-health.md b/docs/metrics/repo-health.md index 07314a18..4e443e20 100644 --- a/docs/metrics/repo-health.md +++ b/docs/metrics/repo-health.md @@ -1,6 +1,6 @@ # Repo health -Measured at `52e03cbe`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** +Measured at `fab5302d`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** Current state only; the trend is this file's git history (`git log -p docs/metrics/repo-health.md`). Nothing here fails a build: the numbers make growth visible, the judgment stays human. @@ -8,18 +8,18 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Target | Flash | Capacity | Used | Built | |---|---:|---:|---:|:--:| -| desktop | 1,864 KB | - | - | yes | -| esp32 | 2,011 KB (+3 KB) ⚠ | 2,496 KB | 81% | yes | -| esp32-16mb | 1,767 KB | - | - | carried (age?) | +| desktop | 1,865 KB (+0 KB) ⚠ | - | - | yes | +| esp32 | 2,010 KB (−2 KB) ✓ | 2,496 KB | 81% | yes | +| esp32-16mb | 2,010 KB (+243 KB) ⚠ | 4,096 KB | 49% | yes | | esp32-eth | 1,365 KB | - | - | carried (age?) | -| esp32-pico | 2,023 KB | 3,072 KB | 66% | carried 2d | +| esp32-pico | 2,057 KB (+34 KB) ⚠ | 3,072 KB | 67% | yes | | esp32-wrover | 1,801 KB | - | - | carried (age?) | -| esp32p4rev1-eth | 1,933 KB | 4,096 KB | 47% | yes | -| esp32p4rev1-eth-wifi | 1,972 KB | - | - | carried (age?) | +| esp32p4rev1-eth | 1,949 KB (+16 KB) ⚠ | 4,096 KB | 48% | yes | +| esp32p4rev1-eth-wifi | 2,231 KB (+259 KB) ⚠ | 4,096 KB | 54% | yes | | esp32p4rev3-eth | 1,605 KB | - | - | carried (age?) | -| esp32s3-n16r8 | 2,051 KB | 4,096 KB | 50% | yes | -| esp32s3-n8r8 | 2,038 KB | 3,072 KB | 66% | yes | -| esp32s3-zero | 1,977 KB | 2,496 KB | 79% | yes | +| esp32s3-n16r8 | 2,051 KB | 4,096 KB | 50% | carried 0d | +| esp32s3-n8r8 | 2,038 KB | 3,072 KB | 66% | carried 0d | +| esp32s3-zero | 1,977 KB | 2,496 KB | 79% | carried 0d | | esp32s31 | 2,294 KB | 4,096 KB | 56% | carried 2d | | qemu | 1,351 KB | - | - | carried (age?) | @@ -29,28 +29,28 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Target | Tick | FPS | |---|---:|---:| -| desktop | 140 µs (−489 µs) ✓ | 7,142 (+5,553) ✓ | +| desktop | 834 µs (+694 µs) ⚠ | 1,199 (−5,943) ⚠ | | esp32 | 8,354 µs | 119 | ### Scenario tick by target (p50 of each sample window) | Scenario | desktop-macos | desktop-windows | esp32 | esp32s3-n16r8 | esp32p4rev1-eth | esp32s31 | esp32-eth | esp32-eth-wifi | unknown | |---|---|---|---|---|---|---|---|---|---| -| Audio_mutation | 26 | 40 ? | 13,152 | 47 ? | - | - | - | - | - | -| Aurora_fps | 1,518 | - | - | - | - | - | - | - | - | +| Audio_mutation | 25 (−1) ✓ | 40 ? | 13,152 | 47 ? | - | - | - | - | - | +| Aurora_fps | 1,526 (+8) ⚠ | - | - | - | - | - | - | - | - | | Driver_mutation | 20 | 42 ? | 12,812 | 39 ? | - | - | - | - | - | | Effects_composition | 148 | 549 ? | - | - | - | - | - | - | - | | Fields_polar_lut | 1,239 | - | - | - | - | - | - | - | - | | Fluid_solver | 222 | - | - | - | - | - | - | - | - | | GridBlacks_blackpixel | 2 | 8 ? | 269 ? | 267 ? | - | - | - | - | - | -| GridLayout_resize | 127 | 219 ? | 1,352 ? | 1,011 ? | 1,143 ? | - | 95,771 ? | 82,231 ? | - | +| GridLayout_resize | 126 (−1) ✓ | 219 ? | 1,352 ? | 1,011 ? | 1,143 ? | - | 95,771 ? | 82,231 ? | - | | Layer_base_pipeline | 71 | 118 ? | - | - | - | - | - | - | - | | Layer_memory_1to1 | 5 | 1 ? | - | - | - | - | - | - | - | -| Layouts_mutation | 97 | 111 ? | 13,692 | 45 ? | - | - | 27 ? | - | - | +| Layouts_mutation | 96 (−1) ✓ | 111 ? | 13,692 | 45 ? | - | - | 27 ? | - | - | | MoonLiveEffect_controls | 11 ? | - | 12,901 | 4,624 ? | - | - | - | - | - | | MoonLiveEffect_livescript | 6 | - | 13,433 ? | 8,255 ? | 11,336 ? | - | - | - | - | | MoonLive_pipeline | 6 | 1 ? | 9,604 ? | 3,278 ? | - | 11,398 ? | - | - | 4,393 ? | -| MoonModule_control_change | 143 | 262 ? | 212 ? | 166 ? | 165 ? | - | 111,731 ? | 89,895 ? | - | +| MoonModule_control_change | 133 (−10) ✓ | 262 ? | 212 ? | 166 ? | 165 ? | - | 111,731 ? | 89,895 ? | - | | MqttModule_haDiscovery_toggle | 3 ? | - | 36 ? | 36 ? | - | - | - | - | - | | MultiplyModifier_memory_lut | 3 | 3 ? | - | - | - | - | - | - | - | | MultiplyModifier_pipeline | 126 | 225 ? | - | - | - | - | - | - | - | @@ -59,9 +59,9 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Trails_ladder | 360 | - | - | - | - | - | - | - | - | | modifier_chain | 44 | 69 ? | 13,337 | - | - | - | - | - | - | | modifier_swap | 24 | 41 ? | 12,250 | 354 ? | 362 ? | - | 1,010 ? | - | - | -| perf_full | 279 | 592 ? | 10,392 | 16,915 ? | 17,433 ? | - | - | - | - | +| perf_full | 278 (−1) ✓ | 592 ? | 10,392 | 16,915 ? | 17,433 ? | - | - | - | - | | perf_light | 17 | 35 ? | 2,183 | 2,485 ? | 2,038 ? | - | - | - | - | -| peripheral_grid_sweep | 296 | 649 ? | 6,991 ? | - | 11,495 ? | 12,273 ? | - | - | - | +| peripheral_grid_sweep | 285 (−11) ✓ | 649 ? | 6,991 ? | - | 11,495 ? | 12,273 ? | - | - | - | | peripheral_switch | 4 | 9 ? | 437 | 46 ? | 217 ? | - | - | - | - | Microseconds. `?` marks a cell backed by fewer than 4 samples, which is a first impression rather than a percentile; several are months old and were captured during a network reconfigure, so they read as whole milliseconds. `-` means that target has never run that scenario. @@ -81,36 +81,36 @@ These build a bare pipeline with no optional modules, so a change here is a chan | Area | Lines | Comments | Comment share | |---|---:|---:|---:| -| core | 26,039 (+20) ⚠ | 10,452 | 43.3 % | -| light | 35,627 (+1) ⚠ | 13,532 | 41.7 % | -| platform | 18,503 (+25) ⚠ | 6,500 | 38.5 % (+0.1 %) ⚠ | -| ui | 10,621 (+40) ⚠ | 3,164 | 31.4 % (+0.1 %) ⚠ | -| test | 57,455 (+53) ⚠ | 10,784 | 21.5 % | -| moondeck | 22,710 (+9) ⚠ | 3,690 | 18.6 % | +| core | 26,096 (+57) ⚠ | 10,488 | 43.4 % (+0.1 %) ⚠ | +| light | 35,658 (+31) ⚠ | 13,553 | 41.7 % | +| platform | 18,584 (+81) ⚠ | 6,534 | 38.5 % | +| ui | 10,651 (+30) ⚠ | 3,182 | 31.5 % (+0.1 %) ⚠ | +| test | 57,757 (+302) ⚠ | 10,833 | 21.5 % | +| moondeck | 22,898 (+188) ⚠ | 3,716 | 18.5 % (−0.1 %) ✓ | ## Tests | Kind | Count | |---|---:| -| unit cases | 1,992 (+2) ✓ | +| unit cases | 2,000 (+8) ✓ | | scenarios | 27 | ## Complexity | Metric | Value | |---|---:| -| functions | 3,498 (+1) ✓ | -| over threshold | 249 | +| functions | 3,506 (+8) ✓ | +| over threshold | 250 (+1) ⚠ | | worst CCN | 108 | ## Documentation | Metric | Value | |---|---:| -| markdown files | 220 (+1) ⚠ | -| markdown lines | 36,832 (+1,940) ⚠ | -| plan files | 116 (+1) ⚠ | -| backlog lines | 6,794 | +| markdown files | 221 (+1) ⚠ | +| markdown lines | 37,324 (+492) ⚠ | +| plan files | 117 (+1) ⚠ | +| backlog lines | 7,091 (+297) ⚠ | | lessons lines | 705 | | CLAUDE.md lines | 259 | diff --git a/docs/moonmodules/light/drivers.md b/docs/moonmodules/light/drivers.md index fc36ee70..063fafde 100644 --- a/docs/moonmodules/light/drivers.md +++ b/docs/moonmodules/light/drivers.md @@ -134,6 +134,8 @@ No IP is involved — no address, no port, no DHCP — so the driver works on a Origin: ColorLight 5A-75 documented byte layout. Inspired by [FPP](https://github.com/FalconChristmas/fpp) (Falcon Player), the show player that drives these cards from a Raspberry Pi: seeing an FPP rig feed a wall of panels is what prompted this driver, since a board already rendering those frames can send them itself and remove the host from the installation. FPP is also the reference point for what good looks like here, sustaining 50 fps. +Protocol references: [FPP's ColorLight-5a-75.cpp](https://github.com/FalconChristmas/fpp/blob/master/src/channeloutput/ColorLight-5a-75.cpp) is the implementation this driver's byte layout agrees with, and Harald Kubota's [5A-75B protocol write-up](https://hkubota.wordpress.com/2022/01/31/winter-project-colorlight-5a-75b-protocol/) documents the same wire format independently, including the brightness and color-temperature bytes and the discovery exchange. Read it with its comments: a reader supplied the controller-number field that makes multiple cards on one segment distinguishable, and the article's own MAC pair is printed the other way round from FPP's (destination `11:22:33:44:55:66`, source `22:22:33:44:55:66`, which is what this driver sends and what the cards filter on). Its lineage runs back to the [original mplayer-colorlight reverse engineering](http://www.mylifesucks.de/oss/mplayer-colorlight/). + [Tests](../../tests/unit-tests.md#panelcarddriver) Detail: [technical](moxygen/PanelCardDriver.md) diff --git a/docs/reference/mhc-wled-esp32-p4-shield.md b/docs/reference/mhc-wled-esp32-p4-shield.md index e9a9188e..cdb4d4a4 100644 --- a/docs/reference/mhc-wled-esp32-p4-shield.md +++ b/docs/reference/mhc-wled-esp32-p4-shield.md @@ -22,12 +22,14 @@ The output/RS-485 terminals, left to right, with the P4 GPIO each carries: ### 12x outputs — level-shifted, single-ended (LED data) -The LED-data outputs. Each terminal is `O` on the silkscreen; a level shifter drives the 5 V strand from the P4's 3.3 V. The catalog wires the Parallel LED driver (peripheral `Parlio`) to the first eight (`21,20,25,5,22,23,24,27`). +The LED-data outputs. Each terminal is `O` on the silkscreen; a level shifter drives the 5 V strand from the P4's 3.3 V. The catalog wires the Parallel LED driver (peripheral `Parlio`) to the **first eight terminals in physical order**: `21,20,25,5,7,23,8,27`. + +**The first eight terminals are not the eight lowest-numbered outputs.** Positions 5 and 7 carry GPIO **7** and **8**, not 22 and 24. An earlier version of this table listed `O22`/`O24` in those positions and the catalog followed it, so a strip on terminals 5 and 7 stayed dark while GPIO 22/24 emitted on their RS-485 terminals instead (bench 2026-09-08: two panels of eight unlit, fixed by moving lanes 4 and 6 to GPIO 7/8). | Terminal | GPIO | Note | |---|---|---| -| O21 O20 O25 O5 O22 O23 O24 O27 | 21 20 25 5 22 23 24 27 | LED lanes (Parallel LED, peripheral `Parlio`, default) | -| O7 / O8 | 7 / 8 | also the I²C bus (SDA 7 / SCL 8, catalog I2cScan) | +| O21 O20 O25 O5 O7 O23 O8 O27 | 21 20 25 5 7 23 8 27 | LED lanes (Parallel LED, peripheral `Parlio`, default), in terminal order | +| O7 / O8 | 7 / 8 | ALSO the I²C bus (SDA 7 / SCL 8). Driving them as LED lanes means no I²C on this shield, which is why the catalog entry carries no I2cScan module. Wire those two strands to `O22`/`O24` instead if you need I²C. | | O3 | 3 | also on RS-485 (`A-3-B`) — see note below | | O4 | 4 | also on RS-485 (`A-4-B`) — see note below | | GND | — | ground for the output block | diff --git a/esp32/sdkconfig.defaults.esp32p4rev1-eth b/esp32/sdkconfig.defaults.esp32p4rev1-eth index 33b843b1..ac6a1bf1 100644 --- a/esp32/sdkconfig.defaults.esp32p4rev1-eth +++ b/esp32/sdkconfig.defaults.esp32p4rev1-eth @@ -61,15 +61,26 @@ CONFIG_ETH_DMA_TX_BUFFER_NUM=10 CONFIG_ESP_SYSTEM_MEMPROT_FEATURE=n CONFIG_HEAP_HAS_EXEC_HEAP=y -# esp-dsp's PORTABLE C kernels instead of its hand-written P4 assembly. The assembly FFT uses the -# P4's hardware-loop instruction (esp.lp.setup), and that unit carries a documented erratum - -# Espressif's own soc_caps: SOC_CPU_HAS_HWLOOP_STATE_BUG, "HWLOOP state doesn't go to DIRTY after -# executing the last instruction of a loop". FreeRTOS saves those registers lazily, keyed on that -# DIRTY flag, so a task switched out of the FFT has its loop state silently dropped and faults on -# resume (bench: a crash loop the moment HLS added a second task, Illegal instruction / Load -# access fault inside dsps_fft2r_fc32_arp4.S). IDF patches the erratum on the coprocessor RESTORE -# paths but not on SAVE. Measured cost of the portable kernels: audio tick ~615 us -> ~658 us. -# A WORKAROUND, not a fix: esp-idf#19025 (ours, naming the unguarded save path) and esp-dsp#119 -# (the same fault from another reporter, on IDF v5.5-beta1). Remove once upstream answers. -# Lives in the board fragment because every P4 image (rev1/rev3, eth/eth-wifi) layers on it. -CONFIG_DSP_ANSI=y +# esp-dsp's OPTIMIZED P4 assembly kernels: the default, and what we ship. The line below forces +# the portable C kernels instead and is currently OFF. +# +# Why it exists at all. The assembly FFT uses the P4's hardware-loop instruction (esp.lp.setup), +# and that unit carries a documented erratum - Espressif's own soc_caps: +# SOC_CPU_HAS_HWLOOP_STATE_BUG, "HWLOOP state doesn't go to DIRTY after executing the last +# instruction of a loop". FreeRTOS saves those registers lazily, keyed on that DIRTY flag, so a +# task switched out of the FFT has its loop state silently dropped and faults on resume (bench: +# a crash loop the moment HLS added a second task, Illegal instruction / Load access fault inside +# dsps_fft2r_fc32_arp4.S). IDF patches the erratum on the coprocessor RESTORE paths but not on +# SAVE. Cost of the portable kernels: audio tick ~615 us -> ~658 us. +# Tracked in esp-idf#19025 (ours, naming the unguarded save path) and esp-dsp#119 (the same fault +# from another reporter, on IDF v5.5-beta1). +# +# Off since 2026-09-08: the crash no longer reproduces on the bench (P4 v1.3, IDF v6.1-rc1, audio +# FFT running with HLS as the second task, ~20 min under load). Espressif could not reproduce it +# either, on a v1.2. The erratum and the unguarded save path are BOTH still there, so this is a +# watch, not a fix: the optimized kernels are the desired config and we run them to see whether +# they stay stable. Re-enable the line if the fault returns. +# Ruled OUT as the cause of the P4 shield's 22 s reboot loop the same day: putting it back changed +# nothing (3 reboots in 70 s against 4), which is what a blocked network cascade looks like, not an +# FFT fault. Lives in the board fragment because every P4 image (rev1/rev3, eth/eth-wifi) layers on it. +# CONFIG_DSP_ANSI=y diff --git a/moonbase/main/moonbase_main.cpp b/moonbase/main/moonbase_main.cpp index f16ec381..12f577b5 100644 --- a/moonbase/main/moonbase_main.cpp +++ b/moonbase/main/moonbase_main.cpp @@ -59,6 +59,10 @@ constexpr FsCandidate kFsCandidates[] = { }; constexpr const char* kFsMountPoint = "/fs"; constexpr const char* kNetworkConfig = "/fs/.config/NetworkModule.json"; +// The application persists its build variant here, which is the one fact this image cannot know +// about the board it is on: MoonBase is chip-specific and variant-agnostic, so without it the +// release picker can only offer every firmware for the chip and ask a user in recovery to choose. +constexpr const char* kSystemConfig = "/fs/.config/SystemModule.json"; // The AP fallback address matches the application's (NetworkModule uses 4.3.2.1), so a user who // has provisioned this device before sees the same address in both firmwares. @@ -154,6 +158,23 @@ struct { // (the catalog pins e.g. 8 dBm for them), and a brownout during recovery is the worst time. int txPowerDbm_ = 0; +// The application's build variant ("esp32s3-zero"), or empty when the app has never run here. +// Empty is the honest answer for a freshly flashed or wiped device, and the page falls back to +// offering every firmware for the chip rather than pretending to know which one this board takes. +char g_appVariant[24] = {}; + +void loadAppVariant() { + FILE* f = std::fopen(kSystemConfig, "r"); + if (!f) return; + // Same bounded prefix read as the credentials above, and for the same reason: the file carries + // every child module's config behind the identity keys this image needs. + char buf[1024]; + const size_t got = std::fread(buf, 1, sizeof(buf) - 1, f); + buf[got] = '\0'; + std::fclose(f); + jsonFindString(buf, "firmware", g_appVariant, sizeof(g_appVariant)); +} + // Read the stored WiFi credentials and Ethernet wiring, if there are any. Absent, unreadable // or empty all mean the same thing to the caller: fall through the cascade. void loadCredentials() { @@ -189,6 +210,10 @@ void loadCredentials() { jsonFindBool(buf, "ethClockExtIn", ðCfg_.clockExtIn); jsonFindInt(buf, "txPowerSetting", &txPowerDbm_); } + // BEFORE THE UNMOUNT. This function owns the only window in which the volume is mounted: it + // registers the partition above and unregisters it here, so anything that reads a file has to + // do it now. Reading the variant after this call returned "no file" for exactly that reason. + loadAppVariant(); esp_vfs_littlefs_unregister(label); } @@ -348,6 +373,14 @@ bool wifiAccessPoint() { // --------------------------------------------------------------------------------------------- // The one page MoonBase serves. Inline and tiny: no filesystem read, no compression, no assets. +// The chip this image was built for, as the release assets spell it: `firmware-esp32s3-zero-...` +// begins with the IDF target. MoonBase is chip-specific and variant-agnostic, so this is the most +// it can know about the board, and it is exactly enough to filter a release's asset list down to +// the firmwares that could run here. +#ifndef MOONBASE_CHIP +#define MOONBASE_CHIP CONFIG_IDF_TARGET +#endif + const char kPage[] = "" "MoonBase" @@ -380,6 +413,14 @@ const char kPage[] = "
Firmware files: github.com/MoonModules/projectMM/releases " "(the firmware-...bin matching this board)" + // FROM A RELEASE, without typing a URL. The browser fetches the release list from GitHub + // itself (api.github.com sends access-control-allow-origin: *), so THIS image gains no network + // code at all: it still only receives a URL, which is what it already accepts. That matters + // because a device in recovery is the one that most needs an easy install and the one least + // able to offer the application's own picker. + "
From a release
" + " " + "
" "
From a URL
" "
" "
Back to the app
Boot the installed firmware without changing it." @@ -428,17 +469,65 @@ const char kPage[] = "if(p.status==404){clearInterval(t);S('done, the app is starting...');" "setTimeout(()=>location.reload(),3000);}else{S(await p.text());}}" "catch(_){S('restarting...');}},2000);}" + // The release list, filtered to assets this CHIP can run. MoonBase is chip-specific and + // variant-agnostic (one image serves every variant of a chip), so it cannot know which variant + // the board runs: it offers the ones that fit and lets the user pick, which is the same choice + // the application's picker presents. + "const CHIP='" MOONBASE_CHIP "';let RELS=[],VAR='';" + // The board's own variant, when the application has run here and persisted it. With it the + // list is the ONE firmware this board takes, as the application's picker shows; without it, + // every firmware for the chip, because guessing which of three flash layouts a board has is + // how a user in recovery installs the wrong one. + "fetch('/api/variant').then(r=>r.text()).then(t=>{VAR=t.trim();fillFw();}).catch(()=>{});" + "function fwList(i){const r=RELS[i];if(!r)return [];" + "return (r.assets||[]).map(a=>a.name).filter(n=>/^firmware-.+\\.bin$/.test(n)" + "&&!/-(bootloader|partition-table|ota-data|slot0)\\.bin$/.test(n)" + // The chip must match to a BOUNDARY: "esp32s31" starts with "esp32s3" and is different + // silicon, so a plain prefix test offered an S31 image on an S3 board. Every asset spells the + // chip then a hyphen, whether a variant follows ("esp32s3-zero-v...") or the version does + // ("esp32-v..."), so requiring that hyphen is the whole rule. + "&&n.slice(9).startsWith(CHIP)&&n.slice(9+CHIP.length).startsWith('-')" + "&&(!VAR||n.slice(9).startsWith(VAR+'-')));}" + "function fillFw(){const f=document.getElementById('fw');f.innerHTML='';" + "const l=fwList(document.getElementById('rel').selectedIndex);" + "for(const n of l){const o=document.createElement('option');o.textContent=n;f.appendChild(o);}" + "document.getElementById('rs').textContent=l.length?'':'no firmware for this chip in that release';}" + "fetch('https://api.github.com/repos/MoonModules/projectMM/releases?per_page=10')" + ".then(r=>r.json()).then(j=>{RELS=j;const s=document.getElementById('rel');" + "for(const r of RELS){const o=document.createElement('option');" + "o.textContent=(r.name||r.tag_name)+(r.prerelease?' (pre)':'');s.appendChild(o);}" + "s.onchange=fillFw;fillFw();})" + ".catch(()=>{document.getElementById('rs').textContent=" + "'could not reach github: use a URL or a file below';});" + // Installing a release is installing its URL: one path, so the vetting, the progress and the + // retry all behave identically however the URL was chosen. + "async function rl(){const r=RELS[document.getElementById('rel').selectedIndex];" + "const n=document.getElementById('fw').value;if(!r||!n)return;" + "const a=(r.assets||[]).find(x=>x.name===n);if(!a)return;" + "document.getElementById('u').value=a.browser_download_url;url();}" "async function url(){const u=document.getElementById('u').value;if(!u)return;" "const r=await fetch('/api/firmware/url',{method:'POST',body:u});S(await r.text());if(r.ok)W();}" // Prefill the URL field with the last install source (RAM-held), so Install doubles as // retry: the escape after a cancel or failure wiped the app slot. "fetch('/api/firmware/last-url').then(r=>r.text()).then(u=>{if(u)document.getElementById('u').value=u;})" ".catch(()=>{});" + // RELOAD WHEN THE APP ANSWERS, not after a fixed wait. Eight seconds was a guess that an + // S3-Zero misses, so the page reloaded while the device was still booting and showed a failed + // page the user then had to refresh by hand. /moonbase is the identity probe: MoonBase answers + // it and the app 404s it, so a 404 means the application is up and serving. "async function ba(){const r=await fetch('/api/firmware/boot-app',{method:'POST'});S(await r.text());" - "if(r.ok)setTimeout(()=>location.reload(),8000);}" + "if(!r.ok)return;S('booting the app...');" + "for(let i=0;i<60;i++){await new Promise(f=>setTimeout(f,1000));" + "try{const p=await fetch('/moonbase',{cache:'no-store'});" + "if(p.status==404){location.reload();return;}}catch(e){}}" + "S('the app is not answering: it may not be installed');}" "async function cx(){S(await (await fetch('/api/firmware/cancel',{method:'POST'})).text());}" ""; +// The application's build variant ("esp32s3-zero"), or empty when the app has never run here. +// Empty is the honest answer for a freshly flashed or wiped device, and the page falls back to +// offering every firmware for the chip rather than pretending to know. + // The application slot. From the factory partition esp_ota_get_next_update_partition returns the // first OTA slot, which is the one we want and is never the one we are running from. // @@ -809,6 +898,10 @@ void serveOne(int sock) { } else { sendResponse(sock, "200 OK", "text/plain", "nothing to cancel"); } + } else if (std::strncmp(head, "GET /api/variant", 16) == 0) { + // The application's build variant, read from its config at boot. Empty when the app has + // never run here, which the page treats as "offer every firmware for the chip". + sendResponse(sock, "200 OK", "text/plain", g_appVariant); } else if (std::strncmp(head, "GET /api/version", 16) == 0) { // This image's version, from the app descriptor IDF puts in every binary (PROJECT_VER, // set by build_moonbase to the same string the application reports). Its own route @@ -879,7 +972,7 @@ extern "C" void app_main() { esp_event_loop_create_default(); esp_event_handler_instance_register(IP_EVENT, ESP_EVENT_ANY_ID, &onGotIp, nullptr, nullptr); - loadCredentials(); + loadCredentials(); // also reads the app's build variant, inside its mount window // The cascade: Ethernet where the config wires it (its DHCP window overlaps the WiFi // join since the GOT_IP bit is shared), then WiFi STA with the stored credentials, then diff --git a/moondeck/MoonDeck.md b/moondeck/MoonDeck.md index 2cbd3c62..573bcf3c 100644 --- a/moondeck/MoonDeck.md +++ b/moondeck/MoonDeck.md @@ -122,7 +122,7 @@ uv run moondeck/run/preview_installer.py Long-running — MoonDeck shows **Stop** while the server is up. Two modes, picked automatically: - **Render-only.** When no `build/esp32-*/projectMM.bin` is present, the picker populates against the real GitHub Releases API and dropdowns work, but clicking **Install** fails because the local server has no `releases/` tree. Useful for iterating on HTML / CSS / JS without burning a build. Equivalent to "Recipe A" in [mooninstaller/README.md](../mooninstaller/README.md). -- **Flash-ready.** When at least one ESP32 build exists, the script additionally stages every `build/esp32-*/projectMM.bin` it finds into `releases/local-dev/` and generates matching Pages-relative manifests via the same `generate_manifest.py` the release workflow uses. The picker shows `local-dev` as the newest tag; clicking **Install** flashes a USB-connected ESP32 and hands off to the repository's custom orchestrator UI (Improv-Serial provisioning + SET_DEVICE_MODEL + control fan-out, all in `install-orchestrator.js` — not ESP Web Tools). End-to-end, same code paths as the public installer. This is the developer's test ground for the install flow before deploying to GitHub Pages: Web Serial works on `http://localhost` without the secure-origin requirement that gates the public site. +- **Flash-ready.** When at least one ESP32 build exists, the script additionally stages every `build/esp32-*/projectMM.bin` it finds into `releases/local-dev/` and generates matching Pages-relative manifests via the same `generate_manifest.py` the release workflow uses. The picker shows `local-dev` as the newest tag; clicking **Install** flashes a USB-connected ESP32 and hands off to the repository's custom orchestrator UI (Improv-Serial provisioning + the APPLY_OP config push of the device model's modules and controls, all in `install-orchestrator.js`, not ESP Web Tools). End-to-end, same code paths as the public installer. This is the developer's test ground for the install flow before deploying to GitHub Pages: Web Serial works on `http://localhost` without the secure-origin requirement that gates the public site. Add `?nocache=1` to the URL to bypass the picker's 5-minute sessionStorage cache while editing. @@ -1076,7 +1076,7 @@ Push WiFi credentials to a running projectMM device over USB-serial. Uses the [I **One-click flow**: pick the device's port in MoonDeck, hit **Improv WiFi**. The script reads SSID + password from the **active network's WiFi block in `moondeck/moondeck.json`** (the one shown in the network bar at the top of the sidebar). If that block is empty, it falls back to detecting the host machine's currently-joined WiFi (macOS Keychain / Linux NetworkManager / Windows `netsh`). The device replies with its new URL when STA comes up — typically 5-10 s end to end. -**Device-model dropdown (pre-association injection)**: pick your device model next to the Firmware dropdown and the flow forwards `--device-model` — the script then resolves the deviceModel's `deviceModels.json` settings and pushes the TX-power cap over the `SET_TX_POWER` vendor RPC **before** the credentials, plus `SET_DEVICE_MODEL` after success. This matters for brown-out-prone weak-powered device models (cap 8 dBm): at full TX power they fail their very first WiFi association, so the cap can't wait for the post-online HTTP injection. Leave the dropdown on "(any model)" for device models without special settings. +**Device-model dropdown (pre-association injection)**: pick your device model next to the Firmware dropdown and the flow forwards `--device-model`: the script then resolves the deviceModel's `deviceModels.json` settings and pushes the TX-power cap over the `SET_TX_POWER` vendor RPC **before** the credentials, then applies the entry's modules and controls over serial as `APPLY_OP` ops (the same push the web installer does; the model name is one of those controls, `System.deviceModel`). One-way on boards whose LED pins include GPIO 1/3: once the driver claims the UART pins the board can no longer receive over serial, so a provisioned QuinLED board is reconfigured from its web UI, not by re-running this. This matters for brown-out-prone weak-powered device models (cap 8 dBm): at full TX power they fail their very first WiFi association, so the cap can't wait for the post-online HTTP injection. Leave the dropdown on "(any model)" for device models without special settings. ```bash # Equivalent CLI for a weak-powered board (cap resolved from deviceModels.json): diff --git a/moondeck/build/improv_provision.py b/moondeck/build/improv_provision.py index 92f3defa..0725bb15 100644 --- a/moondeck/build/improv_provision.py +++ b/moondeck/build/improv_provision.py @@ -54,6 +54,7 @@ TYPE_RPC = 0x03 TYPE_RPC_RESPONSE = 0x04 CMD_WIFI_SETTINGS = 0x01 +CMD_GET_CURRENT_STATE = 0x02 def checksum(buf: bytes) -> int: @@ -203,6 +204,146 @@ def self_test() -> int: return 0 +# --- Device-model config push: "Improv = REST over serial" ----------------------------- +# +# The device applies a catalog entry through ONE vendor RPC, APPLY_OP (0xFC), carrying one +# REST operation as JSON per frame sequence: the same {"op":"add"|"set"|"clearChildren"} shapes +# the HTTP API takes. The browser installer plans the op sequence in mooninstaller/config-ops.js +# and sends it open-loop (Web Serial cannot read the ack while it holds the writer). This is a +# faithful port of that planner, and the sender is CLOSED-loop: the device acks every frame +# with an RPC_RESPONSE, or answers ERROR 0x82 while its single op buffer is still busy, which +# means "send that frame again". So a slow tick never drops an op here. +# +# Until 2026-09-08 this script sent a vendor RPC that no firmware has ever handled, then +# printed "pushed": three boards were provisioned with WiFi and nothing else while the script +# claimed success. The op sequence below is what the browser was doing all along. + +IMPROV_CMD_APPLY_OP = 0xFC +IMPROV_ERROR_INVALID_OP = 0x82 # device: op buffer busy, retry this frame +APPLY_OP_CHUNK_MAX = 128 - 3 # kImprovMaxPayload minus the [cmd][seq][last] header + + +def is_eth_only(entry) -> bool: + """An entry whose firmware is an Ethernet-only build (`-eth`, not `-eth-wifi`) has WiFi + compiled out and no WIFI_SETTINGS RPC to answer: provisioning is plugging the cable in. + Mirrors install.js (`ethOnly = /-eth$/.test(firmware)`) so the two front ends agree.""" + fws = entry.get("firmwares") if isinstance(entry, dict) else None + return bool(fws) and all(isinstance(f, str) and f.endswith("-eth") for f in fws) + + +def _is_addable(m) -> bool: + """A module the entry ADDS: a non-empty id, a parent to add it under, and a type.""" + return (isinstance(m, dict) + and isinstance(m.get("id"), str) and bool(m.get("id")) + and isinstance(m.get("parent_id"), str) and bool(m.get("parent_id")) + and bool(m.get("type"))) + + +def plan_config_ops(entry) -> list: + """The ordered APPLY_OP sequence for a catalog entry: mirrors config-ops.js exactly. + + clearChildren pre-pass (every parent the entry adds into, plus any container flagged + replaceChildren, unless that parent is itself added fresh), then per module an add, then its + control sets. The clear pass is what makes a re-provision converge on a NON-erased device: + add is idempotent on id, so without it a stale module lingers and a structural change never + lands. + + Catalog order is preserved, and it matters on boards whose LED pins include GPIO 1/3 + (UART0): the op that sets those pins ends serial reception, so every op after it is lost. + Such an entry lists its LED driver LAST. (Bench 2026-09-08, QuinLED Dig-Uno / Dig-Quad.) + """ + ops = [] + modules = entry.get("modules") if isinstance(entry, dict) else None + modules = modules if isinstance(modules, list) else [] + added_ids = {m["id"] for m in modules if _is_addable(m)} + clear_parents = [] # insertion-ordered, deduped + for m in modules: + if not isinstance(m, dict): + continue + if m.get("replaceChildren") and isinstance(m.get("id"), str) and m["id"]: + if m["id"] not in clear_parents: + clear_parents.append(m["id"]) + if _is_addable(m) and m["parent_id"] not in clear_parents: + clear_parents.append(m["parent_id"]) + for parent in clear_parents: + if parent in added_ids: + continue + ops.append({"op": "clearChildren", "parent": parent}) + for m in modules: + if not isinstance(m, dict) or not isinstance(m.get("id"), str) or m["id"] == "": + continue + if _is_addable(m): + ops.append({"op": "add", "type": m["type"], "id": m["id"], "parent": m["parent_id"]}) + controls = m.get("controls") + if isinstance(controls, dict): + for control, value in controls.items(): + ops.append({"op": "set", "module": m["id"], "control": control, "value": value}) + return ops + + +def encode_apply_op_frames(op: dict) -> list: + """One op -> the Improv RPC frames carrying it: [0xFC][seq][last][chunk], chunk <= 125 B.""" + import json + body = json.dumps(op, separators=(",", ":")).encode("utf-8") + chunks = [body[i:i + APPLY_OP_CHUNK_MAX] for i in range(0, len(body), APPLY_OP_CHUNK_MAX)] or [b""] + frames = [] + for seq, chunk in enumerate(chunks): + last = 1 if seq == len(chunks) - 1 else 0 + frames.append(build_frame(TYPE_RPC, bytes([IMPROV_CMD_APPLY_OP, seq, last]) + chunk)) + return frames + + +def send_apply_op(ser, op: dict, ack_timeout: float = 2.0, retries: int = 20) -> bool: + """Send one op closed-loop: every frame must be acked; a busy device (ERROR 0x82) gets the + same frame again after a short pause. Returns False when a frame is refused for any other + reason or never acked.""" + for frame in encode_apply_op_frames(op): + for attempt in range(retries): + ser.write(frame) + ser.flush() + # Wait for THIS frame's verdict. The device also volunteers CURRENT_STATE frames + # (it is still announcing PROVISIONED right after the credentials), and those are + # not a reply to us: skip them rather than reading one as a refusal, which is how + # the very first op of every push failed on the bench (2026-09-08). + deadline = time.monotonic() + ack_timeout + verdict = None + while verdict is None and time.monotonic() < deadline: + reply = parse_frame(ser, deadline) + if reply is None: + break + msg_type, body = reply + if msg_type == TYPE_RPC_RESPONSE: + verdict = "ack" + elif msg_type == TYPE_ERROR_STATE: + verdict = "busy" if (body and body[0] == IMPROV_ERROR_INVALID_OP) else "refused" + # anything else (CURRENT_STATE, ...) is chatter: keep waiting + if verdict == "ack": + break # next frame + if verdict == "busy": + time.sleep(0.15) # the previous op is still being applied + continue + return False # refused, or no ack within the timeout + else: + return False # busy for the whole retry budget + return True + + +def apply_device_model(ser, entry: dict, name: str) -> int: + """Push a catalog entry as APPLY_OP ops; returns the number of ops that FAILED.""" + ops = plan_config_ops(entry) + ser.reset_input_buffer() # stale state frames from provisioning are not acks + failed = 0 + for op in ops: + label = {"clearChildren": lambda o: f"clearChildren {o['parent']}", + "add": lambda o: f"add {o['type']} as {o['id']} under {o['parent']}", + "set": lambda o: f"set {o['module']}.{o['control']} = {o['value']!r}"}[op["op"]](op) + ok = send_apply_op(ser, op) + print(f" {'ok ' if ok else 'FAIL'} {label}") + failed += 0 if ok else 1 + print(f"==> applied deviceModel {name!r}: {len(ops) - failed}/{len(ops)} ops ok") + return failed + + def main() -> int: ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) ap.add_argument("--self-test", action="store_true", @@ -222,9 +363,9 @@ def main() -> int: ap.add_argument("--device-model", dest="device_model", default=None, metavar="NAME", help="deviceModel name from mooninstaller/deviceModels.json (e.g. " "'ESP32-S3 N16R8 Dev'). Resolves the deviceModel's TX-power cap " - "(controls.Network.txPowerSetting) automatically and " - "pushes the name via SET_DEVICE_MODEL after " - "provisioning — the same injection the web installer " + "(controls.Network.txPowerSetting) automatically and, after " + "provisioning, applies the entry's modules and controls over " + "serial as APPLY_OP ops: the same config push the web installer " "does. An explicit --tx-power overrides the lookup.") ap.add_argument("--tx-power", type=int, default=None, metavar="DBM", help="Send the SET_TX_POWER vendor RPC (0..21 whole dBm) " @@ -304,6 +445,8 @@ def main() -> int: args.tx_power = cap print(f"==> deviceModel {args.device_model!r}: TX-power cap {cap} dBm from deviceModels.json") + eth_only = bool(args.device_model) and is_eth_only(entry) + try: ser = serial.Serial(args.port, baudrate=115200, timeout=0.1) except serial.SerialException as e: @@ -315,7 +458,7 @@ def main() -> int: print(f"ERROR: --tx-power {args.tx_power} out of range 0..21", file=sys.stderr) return 2 # SET_TX_POWER vendor RPC (0xFD): [cmd][data_len=1][dBm]. Mirrors - # SET_DEVICE_MODEL's framing; the firmware persists + applies it before the + # The firmware persists + applies it before the # association the credentials below will trigger. The 2.5 s pause lets # the module's 1 Hz consumer pick the cap up first. print(f"==> sending SET_TX_POWER {args.tx_power} dBm to {args.port}") @@ -328,6 +471,51 @@ def main() -> int: file=sys.stderr) time.sleep(2.5) + # Ask before telling: a device that is ALREADY provisioned answers GET_CURRENT_STATE with + # PROVISIONED and never re-announces it for a second WIFI_SETTINGS, so a re-provision used + # to sit out the whole timeout and skip the config push that needs no provisioning at all. + # (The push works on any reachable device with the port open, the same as the browser's.) + ser.reset_input_buffer() + ser.write(build_frame(TYPE_RPC, bytes([CMD_GET_CURRENT_STATE, 0]))) + ser.flush() + already_provisioned = False + probe_answered = False + probe_deadline = time.monotonic() + 3.0 + while time.monotonic() < probe_deadline: + reply = parse_frame(ser, probe_deadline) + if reply is None: + break + probe_answered = True + if reply[0] == TYPE_CURRENT_STATE and reply[1] and reply[1][0] == 4: + already_provisioned = True + break + if not probe_answered: + # No Improv listener heard us. The common cause on QuinLED boards is not a fault: their + # catalog pins include GPIO 1/3 (UART0), so once the LED driver claims them a provisioned + # board can no longer RECEIVE over this port, and neither a re-provision nor a config push + # can reach it (use the device's web UI or the HTTP push instead). The credentials are + # still sent below in case this is a fresh board on firmware that ignores the probe. + print("==> warning: no answer to GET_CURRENT_STATE on this port (already provisioned with " + "the LED driver on the UART pins?); continuing, expect a timeout if so", + file=sys.stderr) + if eth_only and not already_provisioned: + print("==> Ethernet-only firmware: no WiFi to provision, connect the cable; pushing the config") + already_provisioned = True # same path: the push needs only the open port + if already_provisioned: + print("==> device reports PROVISIONED already: keeping its WiFi credentials" if not eth_only + else "==> applying the catalog entry over serial") + if args.device_model: + print(f"==> applying deviceModel {args.device_model!r} over serial (APPLY_OP)") + failed = apply_device_model(ser, entry, args.device_model) + ser.close() + if failed: + print(f"ERROR: {failed} op(s) not applied; the device config is incomplete", + file=sys.stderr) + return 1 + else: + ser.close() + return 0 + print(f"==> sending WIFI_SETTINGS to {args.port} (SSID: {args.ssid!r})") payload = build_wifi_settings_payload(args.ssid, args.password) frame = build_frame(TYPE_RPC, payload) @@ -369,15 +557,15 @@ def main() -> int: url = urls[0] if urls else "(no URL reported)" print(f"==> provisioned: {url}") if args.device_model: - # SET_DEVICE_MODEL vendor RPC (0xFE): [cmd][1+len][len][name] — - # the same post-provision push the web installer does, so - # the device persists its physical-board identity. - name = args.device_model.encode("utf-8") - ser.write(build_frame(TYPE_RPC, - bytes([0xFE, 1 + len(name), len(name)]) + name)) - ser.flush() - time.sleep(0.5) # let the device's serial task consume it - print(f"==> pushed SET_DEVICE_MODEL {args.device_model!r}") + # The catalog entry's modules + controls, over serial as APPLY_OP ops (the + # deviceModel name is just one of those controls: System.deviceModel). + print(f"==> applying deviceModel {args.device_model!r} over serial (APPLY_OP)") + failed = apply_device_model(ser, entry, args.device_model) + if failed: + print(f"ERROR: {failed} op(s) not applied; the device is provisioned but " + f"its config is incomplete", file=sys.stderr) + ser.close() + return 1 ser.close() return 0 diff --git a/mooninstaller/deviceModels.json b/mooninstaller/deviceModels.json index bedef4ac..af7f1de4 100644 --- a/mooninstaller/deviceModels.json +++ b/mooninstaller/deviceModels.json @@ -159,12 +159,20 @@ "deviceModel": "QuinLED Dig-2-Go" } }, + { + "type": "Drivers", + "id": "Drivers", + "controls": { + "relayPins": "12" + } + }, { "type": "RmtLedDriver", "id": "RmtLed", "parent_id": "Drivers", "controls": { - "pins": "16" + "pins": "16", + "lightPreset": 4 } } ] @@ -202,11 +210,12 @@ } }, { - "type": "RmtLedDriver", - "id": "RmtLed", + "type": "ParallelLedDriver", + "id": "ParallelLed", "parent_id": "Drivers", "controls": { - "pins": "2,4" + "pins": "2,4", + "dcPin": 33 } }, { @@ -317,14 +326,6 @@ "deviceModel": "QuinLED Dig-Octa 32-8L" } }, - { - "type": "RmtLedDriver", - "id": "RmtLed", - "parent_id": "Drivers", - "controls": { - "pins": "0,1,2,3,4,5,12,13" - } - }, { "type": "NetworkModule", "id": "Network", @@ -337,6 +338,15 @@ "ethMdioGpio": 18, "ethRstGpio": -1 } + }, + { + "type": "ParallelLedDriver", + "id": "ParallelLed", + "parent_id": "Drivers", + "controls": { + "pins": "0,1,2,3,4,5,12,13", + "dcPin": 16 + } } ] }, @@ -885,17 +895,6 @@ "height": 8 } }, - { - "type": "ParallelLedDriver", - "id": "ParallelLed", - "parent_id": "Drivers", - "controls": { - "peripheral": "LCD-IDF", - "pins": "47,21,14,9,8,16,15,7,1,2,42,41,40,39,38,48", - "clockPin": 19, - "dcPin": 20 - } - }, { "type": "NetworkModule", "id": "Network", @@ -916,6 +915,17 @@ "controls": { "pin": 4 } + }, + { + "type": "ParallelLedDriver", + "id": "ParallelLed", + "parent_id": "Drivers", + "controls": { + "peripheral": "LCD-IDF", + "pins": "47,21,14,9,8,16,15,7,1,2,42,41,40,39,38,48", + "clockPin": 19, + "dcPin": 20 + } } ] }, @@ -944,17 +954,6 @@ "deviceModel": "SE 16 V1" } }, - { - "type": "ParallelLedDriver", - "id": "ParallelLed", - "parent_id": "Drivers", - "controls": { - "peripheral": "LCD-IDF", - "pins": "47,48,21,38,14,39,13,40,12,41,11,42,10,2,3,1", - "clockPin": 16, - "dcPin": 17 - } - }, { "type": "NetworkModule", "id": "Network", @@ -974,6 +973,17 @@ "controls": { "pin": 5 } + }, + { + "type": "ParallelLedDriver", + "id": "ParallelLed", + "parent_id": "Drivers", + "controls": { + "peripheral": "LCD-IDF", + "pins": "47,48,21,38,14,39,13,40,12,41,11,42,10,2,3,1", + "clockPin": 16, + "dcPin": 17 + } } ] }, @@ -1040,7 +1050,8 @@ "name": "MHC-WLED ESP32-P4 shield", "chip": "ESP32-P4", "firmwares": [ - "esp32p4rev1-eth" + "esp32p4rev1-eth", + "esp32p4rev1-eth-wifi" ], "image": "assets/deviceModels/mhc-wled-esp32-p4-shield.jpg", "url": "https://shop.myhome-control.de/en/ABC-WLED-ESP32-P4-shield/HW10027", @@ -1081,7 +1092,7 @@ "parent_id": "Drivers", "controls": { "peripheral": "Parlio", - "pins": "21,20,25,5,22,23,24,27" + "pins": "21,20,25,5,7,23,8,27" } }, { @@ -1096,14 +1107,6 @@ "ethClockGpio": 50, "ethClockExtIn": true } - }, - { - "type": "I2cScanModule", - "id": "I2cScan", - "controls": { - "sda": 7, - "scl": 8 - } } ] }, diff --git a/src/core/HttpServerModule.cpp b/src/core/HttpServerModule.cpp index 919277d6..f5afee8f 100644 --- a/src/core/HttpServerModule.cpp +++ b/src/core/HttpServerModule.cpp @@ -1549,13 +1549,20 @@ void HttpServerModule::serveSystem(platform::TcpConnection& conn) { // maxBlock = internal-only (maxInternalAllocBlock): the all-memory // variant reports ~8 MB on PSRAM boards and is meaningless as a // pressure signal. Same rationale as main.cpp's tick log line. + // `allocated` / `allocPeak` are what the system TOOK, which is the figure that means the same + // thing on a board and on a laptop: free heap does not, since a desktop has as much as it wants + // and reports 0. It is what makes a memory change measurable without hardware. sink.appendf( - "{\"fps\":%u,\"tickTimeUs\":%u,\"freeHeap\":%u,\"freeInternal\":%u,\"maxBlock\":%u,\"uptime\":%u,\"modules\":[", + "{\"fps\":%u,\"tickTimeUs\":%u,\"freeHeap\":%u,\"freeInternal\":%u,\"maxBlock\":%u," + "\"allocated\":%u,\"allocPeak\":%u,\"allocBlocks\":%u,\"uptime\":%u,\"modules\":[", static_cast(scheduler_ ? scheduler_->fps() : 0), static_cast(scheduler_ ? scheduler_->tickTimeUs() : 0), static_cast(platform::freeHeap()), static_cast(platform::freeInternalHeap()), static_cast(platform::maxInternalAllocBlock()), + static_cast(platform::allocatedBytes()), + static_cast(platform::allocatedPeak()), + static_cast(platform::allocatedCount()), static_cast(scheduler_ ? scheduler_->elapsed() / 1000 : 0)); // Per-module timing (walk tree recursively) @@ -2970,6 +2977,16 @@ void HttpServerModule::pushStateToWebSockets() { if (stateSend_.active) return; JsonSink sink; buildStateJson(sink); + // A sink that ran out of heap holds a TRUNCATED document, and the frame header would declare + // it complete: the browser parses it, throws, and drops every module past the cut. Send it + // anyway, but CLEAR the resync flag first and say so. Returning early here instead looks + // safer and is worse: fullResyncPending_ stays set, the else-branch that pushes value + // patches is never reached, and the whole UI freezes (no fps, no heap, no live values) on a + // board where the state simply does not fit. A partial tree that keeps updating beats a + // whole one that never arrives. (Bench 2026-09-08, both classic boards.) + if (sink.overflowed()) { + setStatus("state too large for free memory: some modules may not show", Severity::Warning); + } const size_t len = sink.size(); char* owned = sink.detach(); // move ownership to the sender (frees on drain-complete) if (owned && startBufferedTextSend(owned, len)) { diff --git a/src/core/JsonSink.h b/src/core/JsonSink.h index e2b91bda..b10b68ad 100644 --- a/src/core/JsonSink.h +++ b/src/core/JsonSink.h @@ -47,6 +47,7 @@ class JsonSink { if (fixed_ && fixedCap_ > 0) fixed_[0] = '\0'; } + ~JsonSink() { if (heap_) platform::free(heap_); } JsonSink(const JsonSink&) = delete; @@ -66,7 +67,11 @@ class JsonSink { fixed_[fixedLen_++] = *s++; fixed_[fixedLen_] = '\0'; } else { - if (!ensureHeap(heapLen_ + 1)) return; // out of memory: drop + // Out of memory. Flag it exactly as the fixed-buffer path above does: a caller + // that ships the buffer anyway sends a TRUNCATED document, and a truncated JSON + // frame is indistinguishable from a whole one at the far end (the browser parses + // it, throws, and drops the tail: the module cards past the cut simply vanish). + if (!ensureHeap(heapLen_ + 1)) { overflowed_ = true; return; } heap_[heapLen_++] = *s++; } } @@ -225,17 +230,36 @@ class JsonSink { } // Grow the heap buffer to hold at least `need` bytes plus a null terminator. + /// + /// Doubling is the right default (amortized O(1) appends), but the old and new buffers are both + /// live across the memcpy, so growing 16 KB to 32 KB asks for ~48 KB of CONTIGUOUS heap at once. + /// A classic ESP32 serving a 40 KB state document has the free bytes and not the block: the + /// allocation failed, every later append dropped, and the device shipped a truncated frame that + /// looked complete. So a refused doubling steps down toward the minimum rather than giving up: + /// slower to grow, and it fits where doubling cannot. (Measured on the bench 2026-09-08: both + /// classic boards cut their state at a power of two, losing 8-11 KB of the tree.) bool ensureHeap(size_t need) { if (need + 1 <= heapCap_) return true; - size_t newCap = heapCap_ == 0 ? 2048 : heapCap_ * 2; - while (newCap < need + 1) newCap *= 2; - char* grown = static_cast(platform::alloc(newCap)); - if (!grown) return false; - if (heap_) { std::memcpy(grown, heap_, heapLen_); platform::free(heap_); } - heap_ = grown; - heapCap_ = newCap; - heap_[heapLen_] = 0; - return true; + size_t want = heapCap_ == 0 ? 2048 : heapCap_ * 2; + while (want < need + 1) want *= 2; + // Step down in QUARTERS of the current capacity rather than to `need + 1`. Backing off to the + // bare minimum serves this one append and then grows again on the next character, which is + // O(n^2) copying and thrashes exactly the fragmented heap that refused the doubling. A + // quarter still leaves useful headroom, so the next grow is thousands of appends away. + const size_t step = heapCap_ / 4 > 4096 ? heapCap_ / 4 : 4096; + const size_t floorCap = need + 1 > step ? need + 1 : step; + for (;;) { + if (char* grown = static_cast(platform::alloc(want))) { + if (heap_) { std::memcpy(grown, heap_, heapLen_); platform::free(heap_); } + heap_ = grown; + heapCap_ = want; + heap_[heapLen_] = 0; + return true; + } + if (want <= floorCap) return false; // even a useful minimum is refused: genuinely out + const size_t next = want - step; + want = next < floorCap ? floorCap : next; + } } platform::TcpConnection* conn_ = nullptr; // socket mode when non-null diff --git a/src/core/SystemModule.h b/src/core/SystemModule.h index 3d4c1877..34a4fe49 100644 --- a/src/core/SystemModule.h +++ b/src/core/SystemModule.h @@ -3,6 +3,7 @@ #include "core/MoonModule.h" #include "core/Scheduler.h" #include "core/FilesystemModule.h" // setDeviceModel() arms the debounced save (noteDirty) +#include "core/build_info.h" // kFirmwareName: the variant persisted for MoonBase #include "platform/platform.h" #include @@ -158,6 +159,20 @@ class SystemModule : public MoonModule { // the UI (pushed, never user-typed); bound as Text — not ReadOnly — because Text is // auto-persisted and the readonly flag is only a UI-render hint. controls_.addText("deviceModel", deviceModel_, sizeof(deviceModel_), validateDeviceModel); + + // firmware: the build variant this image is (`esp32s3-zero`), written from kFirmwareName + // on every boot rather than read from the file: the compile-time constant is the truth, + // and persisting it only puts it somewhere ANOTHER IMAGE can read. + // + // That reader is MoonBase. It is chip-specific but variant-agnostic (one image serves + // every variant of a chip), so on its own it can only offer every firmware for the chip + // and ask a user in recovery to pick the right one, where picking an esp32s3-n16r8 for a + // Zero installs a flash layout the board does not have. Reading this file narrows the list + // to one, the same way the application's own picker does. Text, not ReadOnly, for the + // reason deviceModel gives above: Text is what gets persisted. + std::snprintf(firmwareVariant_, sizeof(firmwareVariant_), "%s", kFirmwareName); + controls_.addText("firmware", firmwareVariant_, sizeof(firmwareVariant_)); + controls_.setHidden(controls_.count() - 1, true); // FirmwareUpdateModule's card shows it controls_.setReadOnly(controls_.count() - 1, true); // Dynamic (updated every second) @@ -349,6 +364,7 @@ class SystemModule : public MoonModule { // entry ("Olimex ESP32-Gateway Rev G" = 26) with headroom; the Improv RPC handler // caps str_len against this size dynamically. char deviceModel_[32] = {}; + char firmwareVariant_[24] = {}; ///< the build variant, persisted for MoonBase to read // Dynamic (updated in tick1s) char uptimeStr_[16] = {}; diff --git a/src/light/ColorLight5A75Packet.h b/src/light/ColorLight5A75Packet.h index ca58d364..4f33149d 100644 --- a/src/light/ColorLight5A75Packet.h +++ b/src/light/ColorLight5A75Packet.h @@ -73,8 +73,14 @@ constexpr size_t COLORLIGHT_ROW_PREFIX = COLORLIGHT_DATA_OFFSET + COLORLIGHT_ROW constexpr uint16_t COLORLIGHT_MAX_PIXELS_PER_PACKET = 497; constexpr uint8_t COLORLIGHT_BYTES_PER_PIXEL = 3; -// Largest frame built: 21 + 497×3 = 1512 bytes. Above the 1500-byte MTU on purpose — these are raw +// Largest frame built: 21 + 497x3 = 1512 bytes. Above the 1500-byte MTU on purpose: these are raw // L2 frames on a dedicated link to dumb receivers, never IP packets a router would fragment. +// +// FPP packs rows the same way. Harald Kubota's write-up sends 128 pixels per packet instead (391 +// bytes), which stays inside the MTU: both work against the cards, and the larger packet is fewer +// frames for the same wall. Worth knowing if a wall ever goes dark through a SWITCH that will not +// pass a 1512-byte frame, since nothing in the path reports that: the card simply never sees a row +// and its activity LED stays still, which looks exactly like a transmit path that is not running. constexpr size_t COLORLIGHT_MAX_FRAME = COLORLIGHT_ROW_PREFIX + COLORLIGHT_MAX_PIXELS_PER_PACKET * COLORLIGHT_BYTES_PER_PIXEL; // 1512 diff --git a/src/light/drivers/ParallelLedDriver.h b/src/light/drivers/ParallelLedDriver.h index 0b0ff867..95ac3481 100644 --- a/src/light/drivers/ParallelLedDriver.h +++ b/src/light/drivers/ParallelLedDriver.h @@ -75,6 +75,11 @@ class ParallelLedDriver : public DriverBase { if (p) p->attach(this); } + /// Push the heap total to the module readout. Production calls this from the alloc/free sites + /// (publishHeapBytes); a test that drives a mock peripheral's busInit directly has no such site, + /// so it asks for the recompute here and then asserts the PUBLIC dynamicBytes() the card shows. + void publishHeapBytesForTest() { publishHeapBytes(); } + // --- Peripheral backend registry --- // The set of peripheral backends compiled into THIS build. Each backend header self-registers its // factory + label at static-init, via an `inline const bool kXxxPeripheralRegistered = @@ -169,9 +174,10 @@ class ParallelLedDriver : public DriverBase { /// SIMULTANEOUSLY, fed consecutive slices of this driver's window. A token may be a single pin or an /// inclusive range ("20-23" → 20,21,22,23), mixing freely ("20-22,35,38-40"). Shared control shape with /// RmtLedDriver (parsers in PinList.h). Defaults live on the derived (chip-specific safe pins), - /// so the derived sets them after construction; the base just declares them. i80 needs exactly - /// 8 OR 16 real pins (a partial bus is rejected — a sub-16 board parks unused lanes + WR/DC on - /// spare GPIOs); Parlio runs on 1..16. Sized for 16 two-digit GPIOs + separators. + /// so the derived sets them after construction; the base just declares them. Both backends take + /// 1..16 pins: the i80 BUS WIDTH rounds to 8 or 16 (powerOfTwoBus) and parks the unused lanes plus + /// WR/DC on spare GPIOs, which is invisible to the pin list; Parlio's width is the pin count, so + /// nothing rounds. Sized for 16 two-digit GPIOs + separators. char pins[64] = ""; /// Comma-separated lights-per-lane; the unassigned remainder splits evenly over the remaining /// lanes. **A lane is a STRAND, not a pin** — which only differ through the expander: direct @@ -713,8 +719,8 @@ class ParallelLedDriver : public DriverBase { void tick1s() MM_NONBLOCKING override { if (!peripheral_) return; // A bus that lost a shared peripheral to another module comes back on its own once that - // module lets go. On the classic ESP32 the i80 bus and a PDM microphone both need I2S0, so - // whichever asks second is refused; without this the loser stayed dark until the user + // module lets go. On the classic ESP32 the LED bus is an I2S peripheral and drives from + // instance 1, so whichever asks second is refused; without this the loser stayed dark until the user // happened to edit a control, which is a reboot-to-apply in all but name (architecture.md, // live reconfiguration). Gated tightly, because this runs on the render thread: only while // the driver WANTS the bus and does not hold it, and only when the backend says the thing @@ -1378,10 +1384,29 @@ class ParallelLedDriver : public DriverBase { size_t snapshotCap_ = 0; // allocated capacity, grows to fit the window /// This driver's heap = the base scratch + the streaming snapshot (the ring's immutable frame copy, - /// the biggest single driver buffer at ~36 KB). Summed for the per-module memory readout — see - /// DriverBase::driverHeapBytes. The DMA ring buffers are platform-owned (not driver heap), so they - /// are not counted here. - size_t driverHeapBytes() const override { return DriverBase::driverHeapBytes() + snapshotCap_; } + /// the biggest single driver buffer at ~36 KB) + the peripheral's DMA buffers. Summed for the + /// per-module memory readout (see DriverBase::driverHeapBytes). + /// + /// The DMA buffers are allocated by the PLATFORM rather than by this driver, and they used to be + /// left out on that ownership argument. That made the readout lie about the thing a user actually + /// decides on: the i80 frame is sized by the BUS WIDTH (8 or 16 lanes) and not by the pins in use, + /// so a one-lane board pays the same ~50 KB as an eight-lane one. On the bench (2026-09-08) a + /// QuinLED Dig-2-Go showed 512 bytes for this driver against RmtLedDriver's 32 KB while actually + /// costing 49 KB MORE free heap, and nothing on the card said so. Ownership is the wrong question + /// for a memory readout: what the user needs is what choosing this driver costs. + size_t driverHeapBytes() const override { + size_t dma = 0; + if (peripheral_) { + const size_t cap = peripheral_->busCapacity(); + // Count the buffers that EXIST, not the ones `doubleBuffer` asks for: the control + // defaults on, a peripheral may refuse the second allocation (or not support it at + // all), and a readout that trusts the request over the allocation reports memory the + // board never spent. + dma = cap; + if (peripheral_->busBuffer(1)) dma += cap; + } + return DriverBase::driverHeapBytes() + snapshotCap_ + dma; + } // The snapshot copy's inputs, set before copyRange runs. The snapshot is serial (on the ring's core-1 // tick); only the ring PRIME still forks to the core-0 helper (busTransmitRing), so no cross-core copy // bounds live here. diff --git a/src/platform/desktop/platform_desktop.cpp b/src/platform/desktop/platform_desktop.cpp index 7d8f0501..ed073eae 100644 --- a/src/platform/desktop/platform_desktop.cpp +++ b/src/platform/desktop/platform_desktop.cpp @@ -201,18 +201,60 @@ uint32_t micros() MM_NONBLOCKING { #pragma clang diagnostic pop #endif +// WHAT THIS PROCESS HAS DELIBERATELY ALLOCATED, in bytes. Not a heap figure: a desktop has as much +// memory as it wants, and freeHeap() keeps reporting 0 because three call sites read that 0 as +// "unlimited" and switch off gates that only mean something on a device (polar.h's LUT budget, +// MappingLUT's paging fallback). +// +// What it IS good for is the DELTA. Every buffer the system takes on purpose (layer buffers, +// mapping LUTs, script arenas, driver rings) comes through alloc/allocInternal, so adding or +// removing a module moves this number by exactly what that module costs, on a laptop, in a second, +// with no board attached. The process's own RSS cannot answer that: the allocator, the JIT and the +// HTTP buffers move it too, and a 100 KB layer would be lost in the noise. +// +// The REQUESTED size is recorded rather than malloc's rounded one (malloc_size reports 1024 for a +// 1000-byte ask), so a reported delta is the number the caller asked for. +std::atomic g_allocatedBytes{0}; +std::atomic g_allocatedPeak{0}; +std::atomic g_allocCount{0}; + +namespace { +// Requested size, kept immediately before the block handed out. 16 bytes rather than 8 so the +// returned pointer keeps the alignment malloc promised for any type. +constexpr size_t kAllocHeader = 16; + +void* trackedAlloc(size_t bytes) { + void* raw = std::malloc(bytes + kAllocHeader); + if (!raw) return nullptr; + *static_cast(raw) = bytes; + const size_t now = g_allocatedBytes.fetch_add(bytes, std::memory_order_relaxed) + bytes; + // Peak is advisory, so a lost race between two threads costs a slightly low high-water mark + // rather than anything a caller depends on. + if (now > g_allocatedPeak.load(std::memory_order_relaxed)) + g_allocatedPeak.store(now, std::memory_order_relaxed); + g_allocCount.fetch_add(1, std::memory_order_relaxed); + return static_cast(raw) + kAllocHeader; +} +} // namespace + void* alloc(size_t bytes) { - return std::malloc(bytes); + return trackedAlloc(bytes); } bool ptrIsPsram(const void* /*p*/) { return false; } // desktop has no PSRAM void* allocInternal(size_t bytes) { - return std::malloc(bytes); // desktop has one flat RAM — internal == ordinary + return trackedAlloc(bytes); // desktop has one flat RAM: internal == ordinary } void free(void* ptr) { - std::free(ptr); + if (!ptr) return; + void* raw = static_cast(ptr) - kAllocHeader; + g_allocatedBytes.fetch_sub(*static_cast(raw), std::memory_order_relaxed); + // Decremented, so the count is LIVE blocks and not allocations-ever. Without this it only + // climbed, which reads as a leak on any device left running. + g_allocCount.fetch_sub(1, std::memory_order_relaxed); + std::free(raw); } // Executable memory for MoonLive's emitted code. macOS on Apple Silicon enforces W^X @@ -302,6 +344,10 @@ void pauseLoop() { lastWake = std::chrono::steady_clock::now(); } +size_t allocatedBytes() { return g_allocatedBytes.load(std::memory_order_relaxed); } +size_t allocatedPeak() { return g_allocatedPeak.load(std::memory_order_relaxed); } +uint32_t allocatedCount() { return g_allocCount.load(std::memory_order_relaxed); } + size_t freeHeap() { return 0; // Not meaningful on desktop (0 = unlimited) } @@ -1384,7 +1430,7 @@ int wifiStaRssi() { return 0; } void wifiStaBssid(uint8_t out[6]) { std::memset(out, 0, 6); } int wifiStaChannel() { return 0; } -bool wifiApInit(const char* /*apName*/, const char* /*ip*/) { return false; } +bool wifiApInit(const char* /*apName*/, const char* /*ip*/) { return false; } // no AP on a host bool wifiApConnected() { return false; } void wifiApStop() {} uint32_t wifiApClientCount() { return 0; } diff --git a/src/platform/esp32/platform_esp32.cpp b/src/platform/esp32/platform_esp32.cpp index bc286606..5fff2c16 100644 --- a/src/platform/esp32/platform_esp32.cpp +++ b/src/platform/esp32/platform_esp32.cpp @@ -216,6 +216,28 @@ void reboot() { esp_restart(); } +// The same three numbers the desktop counts by hand, from the allocator that already tracks them. +// A scenario reads one metric on both platforms: how much the system has taken, its high-water +// mark, and how many blocks are live. USED rather than free, so the figure means the same thing on +// a board with 320 KB and a laptop with gigabytes. +size_t allocatedBytes() { + multi_heap_info_t info = {}; + heap_caps_get_info(&info, MALLOC_CAP_8BIT); + return info.total_allocated_bytes; +} +size_t allocatedPeak() { + // The minimum-ever free, expressed as a peak used: IDF tracks the low-water mark of free heap, + // which is the same fact from the other side. + const size_t total = heap_caps_get_total_size(MALLOC_CAP_8BIT); + const size_t minFree = heap_caps_get_minimum_free_size(MALLOC_CAP_8BIT); + return total > minFree ? total - minFree : 0; +} +uint32_t allocatedCount() { + multi_heap_info_t info = {}; + heap_caps_get_info(&info, MALLOC_CAP_8BIT); + return static_cast(info.allocated_blocks); +} + size_t freeHeap() { return heap_caps_get_free_size(MALLOC_CAP_8BIT); } diff --git a/src/platform/platform.h b/src/platform/platform.h index 8eac7f09..0462224e 100644 --- a/src/platform/platform.h +++ b/src/platform/platform.h @@ -74,6 +74,19 @@ void free(void* ptr); // Free with the ordinary free(). void* allocInternal(size_t bytes); +/// How many bytes this process has deliberately allocated through alloc/allocInternal, and the +/// high-water mark. NOT a heap figure: freeHeap() stays 0 on desktop because callers read that as +/// "unlimited" and switch off gates that only mean something on a device. +/// +/// The point is the DELTA. Every buffer the system takes on purpose comes through this seam, so +/// adding or removing a module moves these by exactly that module's cost, measurable on a laptop +/// without a board. On ESP32 they report the same thing from the real heap, so a scenario reads one +/// number on both. `count` is the number of live blocks, which separates "one buffer got bigger" +/// from "something is allocating per frame". +size_t allocatedBytes(); +size_t allocatedPeak(); +uint32_t allocatedCount(); + // True when the pointer resolves to external (PSRAM) memory: the standard residency probe (IDF's // esp_ptr_external_ram). Diagnostic companion to allocInternal's internal-first-PSRAM-fallback pattern: // the caller of that pattern cannot otherwise tell which way an allocation landed, and for buffers an diff --git a/src/ui/app.js b/src/ui/app.js index 32fd7f4f..60cc24f0 100644 --- a/src/ui/app.js +++ b/src/ui/app.js @@ -672,7 +672,7 @@ async function addModule(type, parentName, id) { // Bring a module's card into view and focus its first control (added via the "+" flow). function focusModule(name) { - const card = document.querySelector(`.card[data-module="${cssEscape(name)}"]`); + const card = queryByName(`.card[data-module="${cssEscape(name)}"]`, "data-module", name); if (!card) return; // A child card can wrap its controls in a collapsed
(.card-controls-collapse): open it // FIRST so the card is at its expanded height, THEN scroll: scrolling a still-collapsed card lands @@ -1151,7 +1151,7 @@ function applyTabDot(tab, mod) { // WS value patch deliberately never re-runs: so without this, a fault (or an enable/disable) on a background // tab would stay invisible until the next full render. (The UI has two render paths; a rule must live in both.) function updateTabDot(mod) { - const tab = document.querySelector(`.tab[data-tab-mid="${cssEscape(mod.name)}"]`); + const tab = queryByName(`.tab[data-tab-mid="${cssEscape(mod.name)}"]`, "data-tab-mid", mod.name); if (!tab) return; applyTabDot(tab, mod); tab.classList.toggle("tab--disabled", mod.enabled === false); // grey a disabled module's tab title @@ -1738,7 +1738,7 @@ function createCard(mod, depth) { // instead of waiting ~1s for the server's full-state round-trip. (updateTabDot still syncs it on the // patch path, idempotently, so this just makes the on/off button the immediate driver.) The tab // lives in the parent's strip, found by the same data-tab-mid updateTabDot uses. - const tabEl = document.querySelector(`.tab[data-tab-mid="${cssEscape(mod.name)}"]`); + const tabEl = queryByName(`.tab[data-tab-mid="${cssEscape(mod.name)}"]`, "data-tab-mid", mod.name); if (tabEl) tabEl.classList.toggle("tab--disabled", !on); }; setEnabledUi(mod.enabled === undefined ? true : !!mod.enabled); @@ -2812,502 +2812,7 @@ function createControl(moduleName, moduleType, ctrl) { row.appendChild(input); break; } - case "filepath": { - // A file NAME plus an editor for that file's contents. The value travels through - // /api/control like any text control; the BODY never does (it cannot: only /api/file - // may exceed the request buffer), so the pane below reads and writes it directly. - // - // `dir` and `ext` come from the module that declared the control, so nothing here knows - // what kind of file this is. - const dir = ctrl.dir || ""; - const ext = ctrl.ext || ""; - // No `dir` means the name IS the path: joinFsPath("", n) would return "/n" and point - // at the filesystem root instead of the file the module named. - const pathOf = (n) => (n ? (dir ? joinFsPath(dir, n) : n) : ""); - // Where a script actually IS, which is not always `dir`: a factory script sits in the - // catalog's directory until an edit forks it into the user's. The device resolves the - // same way (user copy first), so the editor has to look in both or it would open an - // empty box for a script that is plainly listed. - const scriptPathOf = async (n) => { - if (!n || !dir) return pathOf(n); - const local = joinFsPath(dir, n); - if (!mlGroupForExt(ext)) return local; - try { - const here = await fmFetchDir(dir).catch(() => []); - if (here.some(e => !e.isDir && e.name === n)) return local; - const cat = await mlFetchCatalog(); - return joinFsPath(cat.dir, n); - } catch (_) { return local; } - }; - - const stack = document.createElement("div"); - stack.className = "control-fileedit-stack"; - row.appendChild(stack); - - const bar = document.createElement("div"); - bar.className = "fileedit-bar"; - - // A select-SHAPED button, opening the shared picker. It reads as a select (the current - // name, then the ⌄ affordance) and behaves as one, but the list it opens is the same - // widget the module picker uses: search, emoji chips, keyboard, one row per script. - // A native does (`value`, `options`, `disabled`, and a - // `change` event), so everything around it (the fork/share/delete labels, the editor - // load, the download-on-pick) is unchanged and unaware. - const picker = document.createElement("button"); - picker.type = "button"; - picker.className = "control-select fileedit-pick"; - picker.dataset.mid = moduleName; - picker.dataset.key = ctrl.name; - // A READ-ONLY filepath names the file something else chose, so it must not offer a - // second way to choose: the Drivers palette editor is the case, where `palette` owns the - // selection and this pane only edits what that selection resolved to. Two selectors for - // one value is how they end up disagreeing. - if (ctrl.readonly) { picker.disabled = true; picker.classList.add("is-readonly"); } - // The options, as data. `fillPicker` appends option elements exactly as it did to the - // 's own change handler, which already downloads a - // remote script, updates the delete label and loads the editor. One path for - // both ways of choosing. - commit: (name) => { - picker.value = name; - picker.dispatchEvent(new Event("change")); - }, - }); - }; - picker.addEventListener("click", openScriptPicker); - - popBtn.addEventListener("click", async () => { - if (!picker.value) return; - // Flush unsaved edits first: the modal loads the file from the device, so opening - // it on a dirty pane would show stale bytes and then save them back over the edit. - // save() RESOLVES on a failed write (it reports, it does not throw), so the flush is - // only trustworthy if the pane came clean: opening anyway would discard the edit. - await editor.save(); - if (editor.isDirty()) { alert("Not opening: this script still has unsaved changes."); return; } - const p = await scriptPathOf(picker.value); - await openFileEditor(p, undefined, moduleName); - await editor.load(p); - }); - - picker.addEventListener("change", async () => { - // Same reason as the modal above: switching files discards the edit otherwise, and - // a save that failed leaves the pane dirty while resolving normally. - await editor.save(); - if (editor.isDirty()) { - alert("Not switching: this script still has unsaved changes."); - picker.value = String(ctrl.value ?? ""); - return; - } - const chosen = picker.value; - // A factory script the device does not hold yet: download it BEFORE selecting it, - // so the module never points at a file that is not there. A failure reports and - // puts the picker back, rather than leaving the card pointing at nothing. - if (remote.includes(chosen)) { - const previous = String(ctrl.value ?? ""); - picker.disabled = true; - try { - await mlDownloadScript(chosen, mlGroupForExt(ext)); - } catch (e) { - picker.disabled = false; - alert("could not download " + chosen + ": " + (e && e.message ? e.message : e)); - picker.value = previous; - return; - } - picker.disabled = false; - await fillPicker(); // it is local now, so it loses its marker - picker.value = chosen; - } - refreshDelLabel(); - dragTs[key] = Date.now(); - sendControl(moduleName, ctrl.name, chosen); - editor.load(await scriptPathOf(chosen)); - }); - - newBtn.addEventListener("click", async () => { - let name = (prompt("New file name in " + dir + ":") || "").trim(); - if (!name) return; - if (ext && !name.endsWith(ext)) name += ext; // a name without its extension is a typo - const r = await fmCreateFile(dir, name, ctrl.tmpl || ""); - if (!r.ok) { alert("create file failed: " + r.message); return; } - await fillPicker(); - picker.value = name; - dragTs[key] = Date.now(); - sendControl(moduleName, ctrl.name, name); - await editor.load(pathOf(name)); - editor.textarea.focus(); - }); - - // Two clicks to delete, the same arm-then-confirm the File Manager uses for its own - // delete: destructive next to frequent is how people lose work. - armPressTwice(delBtn, async () => { - const victim = picker.value; - if (!victim) return; - const wasFork = forks.has(victim); - try { - // A fork is deleted from the USER directory, which is the whole revert: the - // factory copy underneath is what resolves afterwards. Anything else is deleted - // where it actually sits, because a downloaded factory script has no user copy - // and a DELETE on /moonlive/ would report a failure for a file that was - // never there. - const target = wasFork ? pathOf(victim) : await scriptPathOf(victim); - const res = await fetch("/api/dir?path=" + encodeURIComponent(target), - { method: "DELETE" }); - if (!res.ok) throw new Error(await errorMessage(res)); - } catch (err) { - alert((wasFork ? "revert failed: " : "delete failed: ") + err.message); - return; - } - await fillPicker(); - if (wasFork) { - // The factory script is what resolves now, so the module keeps running: stay on - // it rather than unsetting the control, which is the whole point of a revert. - picker.value = victim; - refreshDelLabel(); - dragTs[key] = Date.now(); - sendControl(moduleName, ctrl.name, victim); - await editor.load(await scriptPathOf(victim)); - return; - } - picker.value = ""; - refreshDelLabel(); - dragTs[key] = Date.now(); - sendControl(moduleName, ctrl.name, ""); - await editor.load(""); - }, { armedText: "✓", armedTitle: "Click again to confirm" }); - - // The editor mounted on the USER path above, which is right for a script the user - // wrote and wrong for a factory one that has never been edited. Resolving needs the - // catalog, so it cannot happen during the synchronous mount: re-point it once the - // listing is in, and only when it actually resolves elsewhere. - fillPicker().then(async () => { - const cur = String(ctrl.value ?? ""); - if (!cur || !mlGroupForExt(ext)) return; - const real = await scriptPathOf(cur); - if (real !== pathOf(cur)) await editor.load(real); - }); - break; - } + case "filepath": return buildFilePathControl(row, label, key, v, moduleName, ctrl); case "password": { // ctrl.value arrives XOR-obfuscated + base64-encoded (see // HttpServerModule PASSWORD_XOR_KEY). Decode it so the input holds @@ -3379,186 +2884,7 @@ function createControl(moduleName, moduleType, ctrl) { appendResetButton(row, moduleName, ctrl, def, () => { sel.value = def; }); break; } - case "palette": { - // A color-palette dropdown where EVERY option shows its own gradient: so the colors - // are visible before selecting, not just after. A native ) - const enabledEl = document.querySelector(`button.module-enabled[data-mid="${cssEscape(mod.name)}"]`); + const enabledEl = queryByName(`button.module-enabled[data-mid="${cssEscape(mod.name)}"]`, "data-mid", mod.name); if (enabledEl) { const ts = dragTs[mod.name + ":enabled"] || 0; if (Date.now() - ts > 1000) { @@ -5049,7 +4375,7 @@ function updateValues() { setText(enabledEl, "\u23FB"); enabledEl.classList.toggle("module-enabled--off", !on); enabledEl.setAttribute("aria-pressed", on ? "true" : "false"); - const cardEl = document.querySelector(`.card[data-module="${cssEscape(mod.name)}"]`); + const cardEl = queryByName(`.card[data-module="${cssEscape(mod.name)}"]`, "data-module", mod.name); if (cardEl) cardEl.classList.toggle("card--disabled", !on); } } @@ -5081,7 +4407,7 @@ function allModules() { // card's child-module block / install-picker mount, never converge, and re-fire // every WS tick: a render loop that wedges the UI). function syncVisibleControls(mod) { - const card = document.querySelector(`.card[data-module="${cssEscape(mod.name)}"]`); + const card = queryByName(`.card[data-module="${cssEscape(mod.name)}"]`, "data-module", mod.name); if (!card) return false; // The controls host is THIS card's own collapse wrapper: must be a DIRECT // child (`:scope >`), not any descendant: a container card (e.g. Effects) nests @@ -5195,7 +4521,7 @@ function updateModuleControls(mod) { case "int16": case "int32": case "pin": { // pin is a plain number input (no slider sibling); patches the same way - const input = document.querySelector(`input[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); // While the demo sweep animates a control, leave it alone: the sweep restores the // real value when it ends, and the next patch after that lands normally. if (input && surfaceDemoRunning() && @@ -5209,23 +4535,23 @@ function updateModuleControls(mod) { break; } case "bool": { - const input = document.querySelector(`input[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (input && input.checked !== !!ctrl.value) input.checked = !!ctrl.value; break; } case "text": { - const input = document.querySelector(`input[type="text"][data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input[type="text"][data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (input && input.value !== (ctrl.value ?? "")) input.value = ctrl.value ?? ""; break; } case "textarea": { - const input = document.querySelector(`textarea[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`textarea[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); // Don't clobber the box while the user is typing in it. if (input && document.activeElement !== input && input.value !== (ctrl.value ?? "")) input.value = ctrl.value ?? ""; break; } case "filepath": { - const sel = document.querySelector(`select.fileedit-pick[data-mid="${mid}"][data-key="${k}"]`); + const sel = queryByName(`select.fileedit-pick[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); // Don't clobber the choice while it is focused, and don't reload the pane under // someone who is typing in it: the value is only pushed back when it really moved. if (sel && document.activeElement !== sel && sel.value !== (ctrl.value ?? "")) { @@ -5235,13 +4561,13 @@ function updateModuleControls(mod) { } case "password": { // The peek button flips the input to type="text", so match either. - const input = document.querySelector(`input[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); const decoded = decodePassword(ctrl.value); if (input && input.value !== decoded) input.value = decoded; break; } case "select": { - const sel = document.querySelector(`select[data-mid="${mid}"][data-key="${k}"]`); + const sel = queryByName(`select[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); // Never overwrite a select the user currently has OPEN (popup // showing) or focused. data-open is set on pointerdown/focus and // cleared on change/blur: more reliable than document.activeElement, @@ -5272,7 +4598,7 @@ function updateModuleControls(mod) { case "palette": { // Custom dropdown: patch the trigger (swatch + name) and the selected row, but not // while the user has the list open (data-open === "true"). - const wrap = document.querySelector(`.palette-control[data-mid="${mid}"][data-key="${k}"]`); + const wrap = queryByName(`.palette-control[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (wrap && wrap.dataset.open !== "true" && Number(wrap.dataset.value) !== Number(ctrl.value)) { wrap.dataset.value = ctrl.value; const cols = ((ctrl.options || [])[ctrl.value] || {}).colors || ""; @@ -5290,18 +4616,18 @@ function updateModuleControls(mod) { case "display": { // A url-valued display is an , not a (see renderControl), so this path // has to know both shapes or the link goes stale on the next push. - const link = document.querySelector(`a.control-url[data-mid="${mid}"][data-key="${k}"]`); + const link = queryByName(`a.control-url[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (link) { setUrlDisplay(link, ctrl.value); break; } // The display strip is a segment renderer, not a span: it has to be patched through // its own hook or the surface would freeze at whatever it showed on first render. - const strip = document.querySelector(`.seg16[data-mid="${mid}"][data-key="${k}"]`); + const strip = queryByName(`.seg16[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (strip && strip._setText) { strip._setText(ctrl.value ?? ""); break; } - const span = document.querySelector(`span.display[data-mid="${mid}"][data-key="${k}"]`); + const span = queryByName(`span.display[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (span) setText(span, String(ctrl.value ?? "")); break; } case "display-int": { - const span = document.querySelector(`span.display[data-mid="${mid}"][data-key="${k}"]`); + const span = queryByName(`span.display[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (span) { // Re-cache the unit in case the device changed it (it // shouldn't, but the WS path is the authority). @@ -5312,27 +4638,27 @@ function updateModuleControls(mod) { } case "ipv4": { // Guarded by the shared userActive check above (same as text). - const input = document.querySelector(`input.ipv4-input[data-mid="${mid}"][data-key="${k}"]`); + const input = queryByName(`input.ipv4-input[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (input && input.value !== (ctrl.value ?? "")) input.value = ctrl.value ?? ""; break; } case "time": { - const span = document.querySelector(`span.display[data-mid="${mid}"][data-key="${k}"]`); + const span = queryByName(`span.display[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (span) setText(span, fmtTime(ctrl.value ?? 0)); break; } case "progress": { - const bar = document.querySelector(`progress[data-mid="${mid}"][data-key="${k}"]`); + const bar = queryByName(`progress[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (bar) { bar.value = ctrl.value ?? 0; bar.max = ctrl.total ?? 100; } - const lbl = document.querySelector(`span.control-value[data-mid="${mid}"][data-key="${k}.label"]`); + const lbl = queryByName(`span.control-value[data-mid="${cssEscape(mid)}"][data-key="${k}.label"]`, "data-mid", mid); if (lbl) setText(lbl, fmtProgressLabel(ctrl)); break; } case "list": { - const list = document.querySelector(`div.list-control[data-mid="${mid}"][data-key="${k}"]`); + const list = queryByName(`div.list-control[data-mid="${cssEscape(mid)}"][data-key="${k}"]`, "data-mid", mid); if (!list) break; const rows = Array.isArray(ctrl.value) ? ctrl.value : []; const details = Array.isArray(ctrl.detail) ? ctrl.detail : []; @@ -5399,7 +4725,7 @@ function updateModuleControls(mod) { // control itself wins over the type-level ones from /api/types; see defaultFor. const def = defaultFor(mod.type, ctrl.name, ctrl); if (def !== undefined && def !== null) { - const btn = document.querySelector(`button.reset-btn[data-mid="${mid}"][data-key="${k}.reset"]`); + const btn = queryByName(`button.reset-btn[data-mid="${cssEscape(mid)}"][data-key="${k}.reset"]`, "data-mid", mid); if (btn) { const eq = controlValuesEqual(ctrl, def); btn.classList.toggle("active", !eq); @@ -5426,6 +4752,23 @@ function cssEscape(s) { return String(s).replace(/(["\\])/g, "\\$1"); } +// Find the one element whose `attr` EQUALS `value`, case included. +// +// CSS attribute selectors match values case-INSENSITIVELY in an HTML document, while the firmware +// compares module names with strcmp. So `lines` (a MoonLive effect) and `Lines` (a LinesEffect) are +// two different modules that every `[data-module="..."]` lookup confuses: querySelector returns +// whichever sits first in the DOM, and on the bench (2026-09-08) a Layer holding both rendered no +// effect cards at all. Selectors Level 4 has a case-sensitivity flag for exactly this +// (`[data-module="x" s]`) and Chrome does not support it: it throws SyntaxError, which takes out +// every card on the page. So the match is narrowed here instead, which works everywhere. +function queryByName(selector, attr, value) { + const want = String(value); + for (const el of document.querySelectorAll(selector)) { + if (el.getAttribute(attr) === want) return el; + } + return null; +} + // --------------------------------------------------------------------------- // 6. Type picker // --------------------------------------------------------------------------- @@ -7622,7 +6965,8 @@ async function openFileEditor(relPath, expectedSize, moduleName) { // and a compile failure that happened before the modal opened would otherwise show unmarked // until the module recompiles. The row is the card's, so it holds the same text either way. if (moduleName) { - const row = document.querySelector(`[data-status-mid="${cssEscape(moduleName)}"] .status-value`); + const row = (queryByName(`[data-status-mid="${cssEscape(moduleName)}"]`, "data-status-mid", moduleName) + || {}).querySelector?.(".status-value"); if (row) ed.markError(row.textContent); } dlg.showModal(); @@ -7644,3 +6988,690 @@ async function openFileEditor(relPath, expectedSize, moduleName) { // --------------------------------------------------------------------------- document.addEventListener("DOMContentLoaded", init); + + +/// The palette editor: a gradient strip whose stops are dragged, added and removed, plus the +/// pickers behind them. Lifted out of createControl's switch, where its 180 lines and the +/// filepath case's 496 made a dispatch table read as an implementation. +/// +/// Same shape as buildKnob / buildListEntries / buildListPads above: one control type, one +/// function, taking exactly what it needs from the row createControl already built. +function buildPaletteControl(row, key, def, moduleName, ctrl) { + // A color-palette dropdown where EVERY option shows its own gradient: so the colors + // are visible before selecting, not just after. A native can only render plain text, so a script's emoji and dimension had + // nowhere to go. + // + // It presents the SAME surface a ; they are never rendered, they are the list the modal is built from. + picker.options = []; + picker.appendChild = (o) => { picker.options.push(o); return o; }; + const paintPicker = () => { + const cur = picker.options.find(o => o.value === picker._value); + picker.textContent = cur ? cur.textContent : "(none)"; + const caret = document.createElement("span"); + caret.className = "fileedit-pick-caret"; + caret.textContent = "\u2304"; // ⌄, the select affordance + picker.append(caret); + }; + Object.defineProperty(picker, "value", { + get: () => picker._value ?? "", + set: (v) => { picker._value = String(v ?? ""); paintPicker(); }, + }); + picker._value = ""; + // innerHTML = "" is how fillPicker clears the list; keep that meaning. + Object.defineProperty(picker, "innerHTML", { + set: (v) => { if (v === "") { picker.options = []; picker.replaceChildren(); } }, + get: () => "", + }); + // Which scripts this picker can offer that are not on the device yet. Names only: + // picking one downloads it. Empty for a filepath control that is not a script picker. + let remote = []; + // Local names that also exist in the catalog: a user edit shadowing a factory script. + let forks = new Set(); + // Every name the catalog ships for this role, whether or not it is on the device. + let catalogNames = new Set(); + // Names in the USER's directory: written or edited here, so worth proposing upstream. + let localNames = new Set(); + const fillPicker = async () => { + picker.innerHTML = ""; + const none = document.createElement("option"); + none.value = ""; none.textContent = "(none)"; + picker.appendChild(none); + let names = []; + if (dir) { + try { + const entries = await fmFetchDir(dir); + names = entries.filter(e => !e.isDir && (!ext || e.name.endsWith(ext))) + .map(e => e.name); + } catch (_) { /* an unreachable directory leaves just "none" */ } + } + // The user's OWN files, before the factory listing is merged in below: a name here + // is something they wrote or edited, which is what the share button offers. + localNames = new Set(names); + // A script picker also lists the FACTORY directory, where downloads land. A name in + // both is the user's edit shadowing the factory copy, which is what the device + // resolves too, so it appears once. + const group = mlGroupForExt(ext); + let cat = null; + if (group) { + try { + cat = await mlFetchCatalog(); + const factory = await fmFetchDir(cat.dir, true).catch(() => []); + for (const e of factory) + if (!e.isDir && e.name.endsWith(ext) && !names.includes(e.name)) + names.push(e.name); + } catch (_) { /* no catalog: the picker still lists what is here */ } + } + names.sort(); + // A local name that ALSO exists in the catalog is a fork: the user edited a factory + // script, so their copy shadows one that can be restored. Deleting it is a revert, + // not a loss, and the delete button says so. + // localNames, NOT names: by here `names` also carries the factory listing, so a + // script that was downloaded and never touched counted as a fork. It showed the + // revert arrow for an edit that does not exist, and reverting it deleted a + // /moonlive path with nothing at it. + forks = cat ? new Set(((cat[group] || {}).names || []).filter(n => localNames.has(n))) + : new Set(); + // Everything the catalog offers that is not here yet, listed after the local ones + // so a user's own scripts stay at the top of the list. + remote = cat ? ((cat[group] || {}).names || []).filter(n => !names.includes(n)) : []; + // Every name the library ships for this role, downloaded or not: what the share + // button uses to tell a user's own script from one of ours. + catalogNames = new Set(cat ? ((cat[group] || {}).names || []) : []); + + // The current value may name a file the listing does not have (deleted underneath, + // or a directory that could not be read). Keep it selectable so the card still + // shows what the module is pointing at, rather than silently appearing unset. + const cur = String(ctrl.value ?? ""); + if (cur && !names.includes(cur) && !remote.includes(cur)) names.unshift(cur); + // What the catalog says each factory script is: its dimension and its own emoji, + // read from the script's `int dimensions()` / `string tags()` at build time. So a + // row reads like the module picker's rows do, BEFORE the script is downloaded. A + // script the catalog does not carry (the user's own) simply has no prefix. + const decl = (n) => { + const g = cat && cat[group]; + if (!g || !g.names) return ""; + const i = g.names.indexOf(n); + if (i < 0) return ""; + const marks = []; + if (g.tags && g.tags[i]) marks.push(g.tags[i]); + if (g.dim && DIM_EMOJI[g.dim[i]]) marks.push(DIM_EMOJI[g.dim[i]]); + return marks.length ? marks.join("") + " " : ""; + }; + for (const n of names) { + const o = document.createElement("option"); + o.value = n; o.textContent = decl(n) + n; + picker.appendChild(o); + } + // Marked, because picking one costs a download and can fail. One list rather than + // two groups: to the user it is one library, and where a script happens to live is + // the device's business. + for (const n of remote) { + const o = document.createElement("option"); + o.value = n; + o.textContent = "\u2601 " + decl(n) + n; // cloud: not on this device yet + picker.appendChild(o); + } + picker.value = cur; + refreshDelLabel(); + }; + + // Save sits with the other file actions rather than in a row of its own: the card is + // already narrow, and the dot on it is what marks unsaved work. + const saveBtn = document.createElement("button"); + saveBtn.className = "card-btn fm-editor-save fileedit-glyph-lg"; + // U+2398, the ISO "store" symbol. Not an arrow: ↥ and ↧ already mean upload and + // download here, and ⤓ downloads a file in the tree, so an arrow would read as + // "fetch this" on a button that writes. Not ✓ either, which is the ARMED DELETE + // state one button along. + saveBtn.textContent = "⎘"; + saveBtn.title = "Save (or click away, or Ctrl/Cmd+S)"; + + // The same modal the File Manager opens from a file row: one editor, reached two ways, + // so a script that needs room gets the full-size box without a second implementation. + const popBtn = document.createElement("button"); + popBtn.className = "card-btn fileedit-glyph-lg"; + popBtn.textContent = "⤢"; // expand, the usual glyph for a bigger view + popBtn.title = "Open in a larger window"; + + // No second status line: the module's own `status` control already reports what the + // save produced ("2036 B", or the parse error), and it is the authoritative one because + // the DEVICE writes it. A browser-side copy said the same thing in different words and + // could only ever disagree. What the browser knows and the device cannot (unsaved work, + // a save in flight, a failed write) rides the Save button instead: its dot, its + // disabled state, and its tooltip. + const statusEl = document.createElement("span"); + statusEl.hidden = true; + + const newBtn = document.createElement("button"); + newBtn.className = "card-btn"; + newBtn.textContent = "+"; // the same + the module tree adds with + newBtn.title = "New script"; + const delBtn = document.createElement("button"); + delBtn.className = "card-btn card-btn-del"; + delBtn.textContent = "×"; // the card's own delete, red on the symbol + delBtn.title = "Delete this script"; + // The SAME button reverts a factory script, because it is the same operation: the + // editor only ever saves to the user directory, so an edited factory script is a second + // file shadowing the first, and removing it brings the original back. Saying "delete" + // there would misdescribe it, and a second button would make one act look like two. + const shareBtn = document.createElement("button"); + shareBtn.className = "card-btn"; + shareBtn.textContent = "\u2197"; // north-east arrow: it leaves for somewhere else + shareBtn.title = "Propose this script for the shared library"; + + function refreshDelLabel() { + const isFork = forks.has(picker.value); + delBtn.textContent = isFork ? "\u21ba" : "\u00d7"; // undo arrow, or the delete cross + delBtn.title = isFork + ? "Revert to the shipped version (discards your changes)" + : "Delete this script"; + delBtn.classList.toggle("card-btn-del", !isFork); + // Offered for anything the user WROTE, which is a script of their own or a fork of + // a shipped one: both are a change worth sending back, and the flow differs only in + // which GitHub URL it opens. NOT offered for an untouched factory copy, where the + // file on the device is byte-identical to the one in the repo and a pull request + // would propose no change at all. + const known = catalogNames.has(picker.value); + const edited = localNames.has(picker.value); // it sits in the USER directory + shareBtn.hidden = !picker.value || !mlGroupForExt(ext) || !edited; + shareBtn.title = known + ? "Propose your changes to the shared library" + : "Propose this script for the shared library"; + } + // Share: open a pull request adding this script to the library. + // + // GitHub's "new file" URL takes the path and the contents as query parameters and opens + // its editor pre-filled, forking the repo on the user's behalf when they propose it. So + // a script someone wrote on their own device reaches the library with one click and no + // API, no token and nothing stored here. + // + // Only for scripts a user WROTE: a factory script is already in the library, and a fork + // of one would open a PR that recreates a file that exists. + shareBtn.addEventListener("click", async () => { + const name = picker.value; + const group = mlGroupForExt(ext); + if (!name || !group) return; + await editor.save(); // propose what is on screen, not the last save + // A save that failed leaves the pane dirty, and the read below would then fetch the + // PREVIOUS text from the device: the user would be proposing something other than + // what they are looking at, which is the one outcome worth refusing outright. + if (editor.isDirty()) { + alert("Save the script first: it still has unsaved changes."); + return; + } + let text = ""; + try { + const res = await fetch("/api/file?path=" + encodeURIComponent(await scriptPathOf(name))); + if (!res.ok) throw new Error(await errorMessage(res)); + text = await res.text(); + } catch (err) { alert("could not read the script: " + err.message); return; } + + const cat = await mlFetchCatalog().catch(() => null); + const folder = cat && cat[group] ? cat[group].folder : group; + // A name the library already ships is an EDIT of that file; anything else is a new + // one. GitHub has a flow for each, and both fork on the user's behalf when they + // propose the change, so neither needs write access to this repo. + const repoPath = "moonlive/" + folder + "/" + name; + const url = catalogNames.has(name) + ? "https://github.com/MoonModules/projectMM/edit/main/" + repoPath + + "?value=" + encodeURIComponent(text) + : "https://github.com/MoonModules/projectMM/new/main" + + "?filename=" + encodeURIComponent(repoPath) + + "&value=" + encodeURIComponent(text); + // The script rides in the query string, and browsers stop honoring a URL somewhere + // past ~8 KB. Every shipped script is under 2.5 KB so this is headroom rather than a + // real limit, but a long one would otherwise open a truncated editor and look fine. + if (url.length > 7000) { + await navigator.clipboard.writeText(text).catch(() => {}); + alert("This script is too long to send through a link.\n\n" + + "It has been copied to your clipboard: open\n" + + "github.com/MoonModules/projectMM, add a file under moonlive/" + folder + + "/ and paste it there."); + return; + } + window.open(url, "_blank", "noopener"); + }); + + bar.appendChild(picker); + const tools = document.createElement("div"); + tools.className = "fileedit-tools"; + tools.appendChild(saveBtn); + tools.appendChild(popBtn); + if (dir) { tools.appendChild(newBtn); tools.appendChild(shareBtn); tools.appendChild(delBtn); } + bar.appendChild(tools); + stack.appendChild(bar); + + const pane = document.createElement("div"); + pane.className = "control-fileedit"; + stack.appendChild(pane); + + // Saving re-derives on the device: a written file asks the tree to re-prepare, so the + // module recompiles or reloads on its own. The browser sends nothing extra. + const editor = fmMountEditor(pane, pathOf(ctrl.value), { + sizeKey: key, + // The status this module is ALREADY reporting, so a card built while its script is + // broken shows the marked line straight away rather than waiting for a recompile. + // The EDITOR applies it once the file has loaded: marking at construction would + // convert the offset against an empty textarea and put every error on line 1. + initialStatus: (findModule(moduleName) || {}).status || "", + saveButton: saveBtn, + statusEl, + // Editing a factory script FORKS it: the read came from the library directory, but + // the write goes to the user's, so the shipped copy stays untouched and the new one + // shadows it. Without this an edit overwrote the library copy and there was nothing + // left to revert to. + savePath: (readPath) => { + if (!dir) return readPath; + const base = readPath.slice(readPath.lastIndexOf("/") + 1); + return base ? joinFsPath(dir, base) : readPath; + }, + // A save may have just created the fork, so what the picker thinks is local is out + // of date: re-read it, which is also what turns the delete button into revert. + onSaved: (written) => { + if (!mlGroupForExt(ext)) return; + if (!written.startsWith(dir + "/")) return; + const sel = picker.value; + fillPicker().then(() => { picker.value = sel; refreshDelLabel(); }); + }, + }); + mlEditorAdd(moduleName, editor); + // Apply the status the card is ALREADY showing: registration only catches the next one, + // so a card rendered while its script is broken would report the error with no line + // marked until something recompiled. The modal does the same on open. + { + const row = (queryByName(`[data-status-mid="${cssEscape(moduleName)}"]`, "data-status-mid", moduleName) + || {}).querySelector?.(".status-value"); + if (row) editor.markError(row.textContent); + } + + // Re-read after the modal closes: it edits the same file through the same endpoints, so + // whatever it saved is what this pane should now show. + // One row per script, shaped like a type so the shared picker can render it: the name is + // what the control stores, the dimension and emoji come from the catalog, and the role + // is what the picker prints on the right. + const openScriptPicker = async () => { + if (picker.disabled) return; + const group = mlGroupForExt(ext); + const cat = group ? await mlFetchCatalog().catch(() => null) : null; + const g = (cat && cat[group]) || {}; + const roleWord = group ? group.replace(/s$/, "") : "file"; + const seen = new Set(); + const items = []; + const add = (n, remoteFlag) => { + if (seen.has(n)) return; + seen.add(n); + const i = (g.names || []).indexOf(n); + items.push({ + name: n, + // The cloud marks a script the device does not hold yet: picking it costs a + // download, which is worth knowing before choosing. + displayName: (remoteFlag ? "\u2601 " : "") + n, + role: roleWord, + tags: i >= 0 && g.tags ? (g.tags[i] || "") : "", + dim: i >= 0 && g.dim ? g.dim[i] : 0, + }); + }; + for (const o of picker.options) if (o.value) add(o.value, remote.includes(o.value)); + for (const n of (g.names || [])) add(n, !localNames.has(n) && remote.includes(n)); + if (!items.length) return; + // Anchored to the field itself: the picker opens under it as a modal, sized and + // placed by openPicker rather than by whatever element it hangs from. + openPicker(picker, { + items, + actionLabel: "use", + currentType: picker.value, + // Route through the