devnet-9: cut buildoor memory instead of resizing the hosts - #80
Conversation
The four buildoors are the only 16 GB hosts in a 1002x32 GB fleet and are OOM-killed 7-27 times per hour. Measured per-container, the beacon node is 62-72% of RAM on three of them and every sidecar combined is ~1-2 GiB, so the fix belongs in the beacon config, not the instance size. Buildoors carry no validator range (only builder_index), so they never attest or aggregate, yet buildoor.yaml explicitly subscribed all five clients to every attestation subnet and additionally imported all attestations on lighthouse. On a 1M-validator network that pulls ~31k attestations per slot across 64 gossip meshes into a node that has no use for any of them. Drop the subnet-subscription flags and --import-all-attestations; keep the payload-preparation flags, which are the actual builder requirement. Also override the teku heap for this group: teku.yaml sets -Xmx12g for the validator fleet's 32 GB hosts, which cannot fit on a 16 GB buildoor alongside the EL and sidecars. Claude-Session: https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6
There was a problem hiding this comment.
This PR stops buildoors from subscribing to all 64 attestation subnets (keeping every payload-preparation flag) and overrides teku's heap from -Xmx12g to -Xmx8g via the established ansible_group_priority=100 mechanism, to fit the 16 GB buildoor tier without resizing hosts. I traced the precedence, env-dict replacement (correctly re-specifies all 4 keys since the repo has no hash_behaviour=merge), flag removal (real, not a no-op), and that builder-only hosts carry no validator ranges; the change is mechanically correct and consistent with prior art in devnet-8. The main risks (unmeasured memory savings, 8g heap fit) are openly acknowledged in the description with a validation plan.
Reviewed 1 changed file(s) @ 733dfe86 — no blocking issues found.
"Cheaper to fly to Vietnam than to fix it here."
Summary
Supersedes #78, which proposed doubling the four buildoors to 32 GB. That was the wrong fix: these are not validator nodes and should not need a validator-sized host. This reduces what they consume on 16 GB instead.
Where the memory actually goes
Measured per-container with
docker statson all four hosts:(
buildoor-prysm-ethrex-1is low only because it was freshly restarted — it has 148 restarts.)Trimming sidecars buys ~1-2 GiB at most. The beacon container is the problem, so that is where this change is.
What was configured that a builder cannot use
Buildoors carry no validator range — the inventory gives them only
builder_index. They never attest and never aggregate. Butbuildoor.yamlexplicitly subscribed every client to all attestation subnets, and additionally set--import-all-attestationson lighthouse.On a 1M-validator network that is ~31,000 attestations per slot pulled across 64 gossip meshes, plus the per-mesh peer and scoring state, into a node that has no use for any of it. A builder needs head and
payload_attributesevents (beacon API SSE, not gossip) and the global bid topic.This removes the subnet-subscription flags and
--import-all-attestations, and keeps every payload-preparation flag — those are the actual builder requirement.Teku heap
teku.yamlsets-Xmx12g, sized for the validator fleet's 32 GB hosts. On a 16 GB buildoor running the EL (~1.1 GiB) and sidecars (~0.9 GiB), a 12 GiB heap plus JVM off-heap overhead does not fit —buildoor-teku-nethermind-1has the highest restart count in the fleet at 558.ansible_group_priority=100on[buildoor:vars]lets us override it for this group only; verified resolving to-Xmx8gon the buildoor while the validator fleet keeps 12g.Notes
--logging=debugis also set on the teku buildoor. Left alone here since it comes from the fleet-wideteku.yamland costs CPU/disk more than RSS.nodes.tfdocumentssize : Instance size override (provider-specific), implemented inhetzner.tfbut with no equivalent indigitalocean.tf, so the documented override is silently ignored on every devnet-9 node. That is a latent bug independent of buildoor sizing and should be its own PR.Context: #79
https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6