feat(clients): executionGroups on endpoints + buildoor testing-build playbook

[qu0b]

Endpoint executionGroups + buildoor testing-build playbook

Why

assertoor feeds every configured execution endpoint into spamoor's client pool, and scenarios spread their transactions across all of them. That makes it impossible to point one scenario at a single endpoint such as buildoor's tx intake (POST /rpc, from ethpandaops/buildoor's testing build source), where transactions must go to exactly one place so a block can be built from precisely that list.

What changed

• endpoints[].executionGroups: [..] (global config). The groups become spamoor's group(a,b) rpchost prefix, so a scenario with client_group: builder can address those endpoints. Endpoints without the field are unaffected.
• verifyClientGroups fails at startup when the linked spamoor did not apply a configured group entry, so isolation can never be quietly weaker than the config says.
• playbooks/dev/buildoor-testing-build.yaml: switches a buildoor to the testing build source, runs eoatx into the intake through the builder group, and requires that buildoor proposes minVerifiedBlocks blocks with minTxsPerBlock+ transactions, that its plan-check tally shows zero mismatches, zero missing blocks and zero missed payloads, and that the chain has no forks. Cleanup switches the builder back to the pool source.

Review fixes (redpandabot)

• 🔴 The isolation claim was false. spamoor seeds every client with the default group and group() only appends, and default is what every group-less selection resolves to — wallet funding and refills, contract deployments, and any scenario without client_group. So an endpoint in a named group still received that traffic, and the claim that it "only receives transactions from scenarios that ask for it" was wrong in the field docs, the config example and this description. All three now describe the additive behaviour, and point at -default as the way to actually reserve an endpoint. Group removal needs feat(client): group(-name) removes a client from a group spamoor#283; until it ships, verifyClientGroups turns a dropped removal into a startup error rather than silent under-isolation (verified locally: assertoor refuses to start with ["builder", "-default"] against the pinned v1.2.2).
• 🟡 The playbook could fail on its first poll. The tx-queue task asserted the match count together with the must-be-zero counters under failOnCheckMiss, where an unsatisfied assertion is a failure rather than a wait — so it could fail while buildoor was still tallying the block that had just been proposed. Now split: a polled wait for the count, then a fail-fast guard on the zero counters, which are cumulative and so still cover the whole run.

Found by running it

Re-running the playbook against a kurtosis devnet surfaced three more problems, all fixed here:

• spamoor's own wallet funding must reach the intake too. While buildoor runs the testing source its blocks are the only inclusion path, so a transaction sent to a normal EL sits in the public txpool forever. Funding uses the group-less default selection rather than the scenario's client_group, so with two execution endpoints it picked the plain EL and the scenario hung — verified as exactly one transaction pending in geth. The playbook now documents both correct shapes: the intake as the only execution endpoint, or the others removed from default.
• scenarioYaml refill amounts are wei. The --refill-* CLI flags take ETH and convert; the YAML fields are uint256 wei. Passing 50 funded every wallet with 50 wei, which cleared the refill threshold, so spamoor reported "no funding needed" and every transaction was then skipped for insufficient funds with empty blocks as the result.
• jq numbers are float64. An unquoted 50000000000000000000 reached spamoor as "5e+19" and failed its uint256 parse, and a configVars expression starting with [ gets a . prepended and becomes an index rather than a list.

Second review round, both fixed:

• Endpoints were paired with spamoor clients by index. InitClients appends only the options it could build, so after a failure every later endpoint got another endpoint's client — wiring transactions to the wrong node and checking the wrong endpoint's groups. This was pre-existing in the s.clients mapping and my check inherited it. The count is now verified before the pairing; a client only fails to build on a malformed rpchost, which is a config error worth stopping for.
• The documented "simple form" could not work. It showed the intake with no executionGroups while this test's clientGroup defaults to builder, and a scenario selecting a group no client is in gets no client at all, so only the group-less wallet funding would have reached the intake.
• Group names were not trimmed. spamoor runs TrimSpace on every group name while the check compared the raw config entry, so a stray space made the two disagree: an added group failed startup with a spurious "not in it", and a removal spamoor applied under a different name passed verification, leaving the endpoint in default while the config said otherwise. Both sides now normalize, with a test that runs the config through spamoor's own parser.

Final run on a minimal-preset devnet with gloas at epoch 1: all ten tasks pass, 256 tx plans checked, 256 matches, zero mismatches, missed payloads or unverifiable blocks. Re-run after the alignment fix: all ten pass again.

Endpoint example

endpoints:
  - name: buildoor-intake
    executionUrl: http://buildoor:8080/rpc
    consensusUrl: http://beacon:5052
    executionGroups: [builder, "-default"]

Documented in docs/02-global-config.md.

CI

_shared-check.yaml installed staticcheck@latest, and v0.8.x now requires Go 1.26 while the workflow runs Go 1.25.x with GOTOOLCHAIN=local, so every PR failed at the install step. This branch pins staticcheck@v0.7.0 (the last release for Go 1.25) in a separate commit; drop it if you prefer to bump the Go version instead.

https://claude.ai/code/session_01P3LSEorv3qsJrWb12ZKHNo

[redpandabot]

Summary

Adds spamoor client groups (endpoints[].executionGroups, additive, with -name removal) plus a startup-time verifyClientGroups check that fails if the linked spamoor did not apply a configured entry, and a buildoor testing-build playbook. The core feature is correct against the pinned spamoor v1.2.2 (I traced the group() parser, HasGroup, and the default-group selection in the spamoor checkout), but the playbook's recommended 'simple' endpoint setup cannot feed the shipped scenario, and the new group check inherits a fragile client/endpoint index pairing.

Issues

• 🟡 playbooks/dev/buildoor-testing-build.yaml:48 — documented 'simple form' can't work with default clientGroup: "builder" — see the thread on that line
• 🟡 pkg/txmgr/spamoor.go:49 — verifyClientGroups pairs endpoints[i] with allClients[i], which misaligns when a client fails to init — see the thread on that line
• 🟢 playbooks/dev/buildoor-testing-build.yaml:136 — zero-counter check is a single snapshot, not whole-run coverage — With failOnCheckMiss the check succeeds on its first evaluation and is never re-polled, so the comment "covers the whole run" overstates: blocks proposed after it (e.g. during the following check_consensus_forks epoch, while the background scenario keeps feeding the intake) are never tallied. A mismatch/unknown block in that final ~1 epoch would be missed.

---

_{Reviewed @ aae4eef6
"Firefox" is another name for the red panda — yes, that Firefox.}

[redpandabot]

Summary

The PR threads a new endpoints[].executionGroups config into per-endpoint spamoor client groups (group(...) rpchost prefix), guarded at startup so a group entry the linked spamoor ignored is a hard error rather than silent under-isolation (needed because the pinned spamoor v1.2.2 is append-only), plus a buildoor testing-build playbook. I traced the group plumbing end-to-end (config -> execution.ClientConfig -> spamoor parser -> verifyClientGroups) and against the cross-referenced spamoor source: the trimming matches spamoor's parser, the -default removal branch correctly fails startup against the pinned append-only parser, and the new length-check on InitClients is sound since a client is only dropped on a malformed rpchost and the pool preserves option order. The Go change and tests are correct; the one issue I found is in the playbook's coverage guarantee.

Issues

• 🟡 playbooks/dev/buildoor-testing-build.yaml:149 — The 'no mismatch' guard passes on its first poll and never checks blocks built afterwards — see the thread on that line

---

_{Reviewed @ 51518104
"Firefox" is another name for the red panda — yes, that Firefox.}