Skip to content

devnet-9: cut buildoor memory instead of resizing the hosts - #80

Merged
qu0b merged 1 commit into
masterfrom
qu0b/devnet-9-buildoor-memory
Sep 3, 2026
Merged

devnet-9: cut buildoor memory instead of resizing the hosts#80
qu0b merged 1 commit into
masterfrom
qu0b/devnet-9-buildoor-memory

Conversation

@qu0b

@qu0b qu0b commented Sep 3, 2026

Copy link
Copy Markdown
Member

Summary

Supersedes #78, which proposed doubling the four buildoors to 32 GB. That was the wrong fix: these are not validator nodes and should not need a validator-sized host. This reduces what they consume on 16 GB instead.

Where the memory actually goes

Measured per-container with docker stats on all four hosts:

host beacon all other containers combined
buildoor-lighthouse-geth-1 11.2 GiB (71.7%) ~1.2 GiB
buildoor-teku-nethermind-1 10.05 GiB (64.4%) ~2.0 GiB
buildoor-lodestar-ethrex-1 9.64 GiB (61.7%) ~1.4 GiB
buildoor-prysm-ethrex-1 3.63 GiB (23.2%) ~2.4 GiB

(buildoor-prysm-ethrex-1 is low only because it was freshly restarted — it has 148 restarts.)

Trimming sidecars buys ~1-2 GiB at most. The beacon container is the problem, so that is where this change is.

What was configured that a builder cannot use

Buildoors carry no validator range — the inventory gives them only builder_index. They never attest and never aggregate. But buildoor.yaml explicitly subscribed every client to all attestation subnets, and additionally set --import-all-attestations on lighthouse.

On a 1M-validator network that is ~31,000 attestations per slot pulled across 64 gossip meshes, plus the per-mesh peer and scoring state, into a node that has no use for any of it. A builder needs head and payload_attributes events (beacon API SSE, not gossip) and the global bid topic.

This removes the subnet-subscription flags and --import-all-attestations, and keeps every payload-preparation flag — those are the actual builder requirement.

Teku heap

teku.yaml sets -Xmx12g, sized for the validator fleet's 32 GB hosts. On a 16 GB buildoor running the EL (~1.1 GiB) and sidecars (~0.9 GiB), a 12 GiB heap plus JVM off-heap overhead does not fit — buildoor-teku-nethermind-1 has the highest restart count in the fleet at 558. ansible_group_priority=100 on [buildoor:vars] lets us override it for this group only; verified resolving to -Xmx8g on the buildoor while the validator fleet keeps 12g.

Notes

  • Not measured end-to-end. The reasoning is that removing 64 subnet subscriptions removes the dominant driver of the beacon working set, but the saving has not been observed on a running host. Validate on one buildoor before rolling to all four.
  • If teku still cannot fit in 8 GiB with a 4M-entry registry after unsubscribing, that is a finding worth reporting rather than something to fix by enlarging the host — nimbus does the same job in 8.19 GiB total RSS.
  • --logging=debug is also set on the teku buildoor. Left alone here since it comes from the fleet-wide teku.yaml and costs CPU/disk more than RSS.
  • One piece of devnet-9: implement the digitalocean size override and size the buildoors #78 is worth keeping and is not included here: nodes.tf documents size : Instance size override (provider-specific), implemented in hetzner.tf but with no equivalent in digitalocean.tf, so the documented override is silently ignored on every devnet-9 node. That is a latent bug independent of buildoor sizing and should be its own PR.

Context: #79

https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6

The four buildoors are the only 16 GB hosts in a 1002x32 GB fleet and are
OOM-killed 7-27 times per hour. Measured per-container, the beacon node is
62-72% of RAM on three of them and every sidecar combined is ~1-2 GiB, so the
fix belongs in the beacon config, not the instance size.

Buildoors carry no validator range (only builder_index), so they never attest
or aggregate, yet buildoor.yaml explicitly subscribed all five clients to every
attestation subnet and additionally imported all attestations on lighthouse.
On a 1M-validator network that pulls ~31k attestations per slot across 64
gossip meshes into a node that has no use for any of them.

Drop the subnet-subscription flags and --import-all-attestations; keep the
payload-preparation flags, which are the actual builder requirement.

Also override the teku heap for this group: teku.yaml sets -Xmx12g for the
validator fleet's 32 GB hosts, which cannot fit on a 16 GB buildoor alongside
the EL and sidecars.

Claude-Session: https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6

@redpandabot redpandabot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR stops buildoors from subscribing to all 64 attestation subnets (keeping every payload-preparation flag) and overrides teku's heap from -Xmx12g to -Xmx8g via the established ansible_group_priority=100 mechanism, to fit the 16 GB buildoor tier without resizing hosts. I traced the precedence, env-dict replacement (correctly re-specifies all 4 keys since the repo has no hash_behaviour=merge), flag removal (real, not a no-op), and that builder-only hosts carry no validator ranges; the change is mechanically correct and consistent with prior art in devnet-8. The main risks (unmeasured memory savings, 8g heap fit) are openly acknowledged in the description with a validation plan.


Reviewed 1 changed file(s) @ 733dfe86 — no blocking issues found.
"Cheaper to fly to Vietnam than to fix it here."

@qu0b
qu0b merged commit b759dec into master Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant