Skip to content

Timed-out hostname lookups keep holding blocking-pool threads, with no bound on concurrent lookups #617

Open

Description

@beardthelion

Problem

A hostname lookup for an outbound request holds a Tokio blocking-pool thread until the system resolver gives up, even after the request itself has timed out. Nothing limits how many of these lookups run at once. A nameserver that never answers can therefore keep the node's blocking pool occupied long after every request that started a lookup has returned. The same pool runs other spawn_blocking work, including git smart HTTP serving and the /ipfs walks.

On main, the shared outbound client (build_http_client in crates/twigpine-node/src/lib.rs) uses reqwest's default resolver. That resolver runs getaddrinfo through tokio::task::spawn_blocking (hyper-util client/legacy/connect/dns.rs), with no deadline and no concurrency bound. The request timeout only stops the caller from waiting. #612 adds a lookup deadline through tokio::net::lookup_host, which runs on the same blocking pool, so the thread stays occupied after the deadline there too.

Reachability

Peer announce is unauthenticated, so anyone can register a peer whose hostname lives on a nameserver they control and make that nameserver never answer. Gossip pings, sync fan-out and webhook delivery then start lookups for that name on their own schedule. Each stalled lookup holds a blocking thread until the resolver gives up.

Evidence

A standalone probe (tokio 1.54, reqwest 0.12.28, the workspace lockfile) ran in a container whose only nameserver is unreachable (docker run --dns 10.255.255.1). It used a runtime capped at 4 blocking threads, made 8 concurrent lookups with a 1 s deadline, then timed one unrelated spawn_blocking(|| ()):

mode=reqwest n=8: all 8 requests returned after 1.005690656s
an unrelated spawn_blocking(|| ()) then waited 19.016606871s for a blocking thread
mode=lookup n=8: all 8 requests returned after 1.001493922s
an unrelated spawn_blocking(|| ()) then waited 39.030732157s for a blocking thread

With working DNS (control), the same spawn_blocking waited 14 µs. reqwest is main's client shape. lookup is the lookup_host plus tokio::time::timeout shape #612 uses. The default pool is 512 threads rather than 4, so this takes more concurrent stalled lookups in production, but the threads are held the same way.

Severity

P3. This is availability only. Each stalled lookup costs a resolver timeout, and the attacker has to keep enough lookups in flight to matter against a 512-thread pool.

Possible direction

Admit hostname lookups through a bounded permit pool and refuse a lookup when no permit is free. Hold the permit inside the blocking closure until the OS lookup returns, not until the deadline fires. Not prototyped here.

Related

#340 asked for the resolved-address check that #612 adds; neither scopes how many lookups run at once. #166 (closed) and #164 cover blocking-pool exhaustion through git subprocesses, which is a different trigger for the same pool.

Activity

  1. added
    sev:lowCosmetic, cleanup, or nice-to-have
    crate:nodegitlawb-node — the serving node and REST API
    kind:bugDefect fix — wrong or unsafe behavior
    subsystem:peersPeer announce, discovery, and registry
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    crate:nodegitlawb-node — the serving node and REST APIkind:bugDefect fix — wrong or unsafe behaviorsev:lowCosmetic, cleanup, or nice-to-havesubsystem:peersPeer announce, discovery, and registry

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions