devnet-9: implement the digitalocean size override and size the buildoors

[qu0b]

Summary

Two coupled fixes. The second one is the reason the buildoors are broken; the first is a latent bug that made it impossible to fix them cleanly.

⚠️ This PR needs a human decision before apply — see Notes. No terraform was run.

Changes

1. The size override was never implemented for DigitalOcean

nodes.tf:14 documents:

#     - size            : Instance size override (provider-specific)

hetzner.tf:107-113 implements it (# Size: explicit > supernode-based default). digitalocean.tf never did — size was purely the supernode ternary at what was line 261:

size = vm.supernode ? var.digitalocean_supernode_size : var.digitalocean_fullnode_size

Every one of the 1,008 devnet-9 nodes is DigitalOcean. So a documented, per-node option was being silently ignored fleet-wide: set size on any DO node and nothing happens, with no error. This half is arguably the more valuable one — it is a footgun independent of the buildoors.

Implemented mirroring the hetzner pattern exactly, so the precedence rule is identical across providers.

2. Put the four buildoors on the 32 GB tier

nodes.tf:26-29, using the override from change 1.

The buildoor entries set no supernode key, do not match the (bootnode|mev) regex — buildoor was never added alongside the two older zero-validator classes — and carry zero validators. So they fall through every branch to digitalocean_fullnode_size = s-8vcpu-16gb, while all ~1,000 validator nodes carry supernode = true explicitly and get s-8vcpu-32gb-640gb-intel.

The deeper flaw: the fallback tier uses validators signed for as a proxy for state that must be held. On a 4M-entry registry those are unrelated — a buildoor holds the same ~560 MB genesis state as everyone else while signing for nothing.

Consequences measured on devnet-9:

host	RAM	OOM kills	beacon restarts
buildoor-teku-nethermind-1	16 GB	27.0/h	563 in 36 h
buildoor-prysm-ethrex-1	16 GB	16.9/h	148
buildoor-lighthouse-geth-1	16 GB	7.0/h	50
buildoor-lodestar-ethrex-1	16 GB	2 (wedged at slot 31 since genesis)	0

The builders have never produced a bid — not once, from genesis onward. Zero bid-submission messages exist in the entire log history. 20 slots sampled across the whole post-fork range gave self_built=13, builder_built=0, missing=7, every present block carrying builder_index=18446744073709551615 (UINT64_MAX) and value=0.

devnet-9's headline feature is EIP-7732 enshrined PBS. Its external builder path has never been exercised.

Notes

supernode = true was deliberately not used, even though it is a one-line change and matches the file's dominant convention. That flag drives two things: the size ternary, and the supernode:True droplet tag, which ansible_inventory.tmpl:10 turns into ethereum_node_cl_supernode_enabled. Setting it would resize the buildoors and flip them into full data-column custody — increasing their memory and bandwidth needs. That may or may not be desirable for a builder, but it is a consensus-behaviour change, not a sizing change, and should be decided separately.

⚠️ Operational risk — read before apply. resize_disk = true (digitalocean.tf:271, 321) means changing size is an in-place DigitalOcean resize, not a replacement: the droplet powers off, resizes, powers on. Because resize_disk = true, the disk grows permanently and the node can never be downsized again. Four hosts, requires downtime, irreversible.

Change 1 is a no-op for every existing node (the new expression returns exactly what the old ternary did when size is null), so the plan should show changes on the four buildoors only. Please confirm that against a real terraform plan before applying — I did not run one.

Testing

terraform fmt parses both files cleanly and reports no formatting changes for the lines touched. (The fmt -diff output does flag pre-existing alignment drift in the unrelated digitalocean_region_overrides block; left alone deliberately.) terraform validate and plan were not run — they need init and cloud credentials.

Full investigation: https://panda-uploads-production.devops-539.workers.dev/panda/uploads/596c58/report-buildoor.html

https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6

[qu0b]

Closing — resizing is the wrong fix here.

The buildoors are not validator nodes and shouldn't need 32 GB. The right direction is to reduce what they consume on 16 GB, not to double the host and move on. Reopening this as a memory-reduction change once I've measured where the 16 GB actually goes on these four hosts.

One piece of this PR is worth keeping regardless of the sizing decision, and I'll raise it separately: nodes.tf documents size : Instance size override (provider-specific), and it is implemented in hetzner.tf but has no equivalent in digitalocean.tf. Since every devnet-9 node is DigitalOcean, that documented override is silently ignored fleet-wide. That's a latent bug independent of buildoor sizing.

https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6

[redpandabot]

Summary

Implements the long-documented but silently-ignored per-node size override on the DigitalOcean side, mirroring the hetzner.tf pattern exactly, and uses it to move the four zero-validator buildoor droplets from the 16 GB fullnode tier to the 32 GB supernode tier. The terraform is correct: the size fallback reproduces the supernode ternary, so no existing node is unintentionally resized — only the four buildoors change. Two low-severity notes: the fix only addresses the OOM-driven restarts, and the 32 GB slug is hardcoded rather than tied to the existing variable.

Issues

• 🟡 terraform/devnet-9/digitalocean.tf — Resize may not restore all four builders; apply against a live fleet was never planned — The change doubles instance RAM, which plausibly fixes the three hosts OOM-killed 7–27x/h, but your own data shows buildoor-lodestar-ethrex-1 is wedged at slot 31 since genesis with only 2 OOM kills and 0 restarts — a failure mode host memory does not address, so at least one of the four may still never produce a bid. Separately, the PR notes no terraform was run: the apply will in-place resize (power-cycle) four live droplets and 32 GB capacity in ams3 (teku's override region) is unverified, so a plan should be reviewed against the live state before apply rather than relying on the resize to converge.
• 🟢 terraform/devnet-9/nodes.tf:27 — Hardcoded 32 GB slug duplicates var.digitalocean_supernode_size — see the thread on that line

---

_{Reviewed @ 40c5a2e6
Red pandas have a false thumb: an enlarged wrist bone evolved for gripping bamboo.}

[redpandabot]

🟢 Hardcoded 32 GB slug duplicates var.digitalocean_supernode_size

The four entries hardcode s-8vcpu-32gb-640gb-intel, which is exactly the default of var.digitalocean_supernode_size (digitalocean.tf:17) that sizes the other ~1000 nodes. If that variable is retuned, the buildoors silently stay on the old tier — the same silent-divergence footgun this PR is fixing. Consider a dedicated digitalocean_buildoor_size var or a try(var.digitalocean_supernode_size, ...) base instead of a literal.