fix(downloader): defer rescue timers at full concurrency instead of dropping them

[detail-app]

Detail bug report: View on Detail

Bug

In download_replicas (src/object_store/downloader/replicas.rs), the multi-replica read path bounds concurrent copies at MAX_CONCURRENT_COPIES = 3. The rescue select! arm eagerly consumed the next rescue entry with rescues.pop_front(), while the actual launch was gated after the pop by active.len() >= MAX_CONCURRENT_COPIES. When a rescue timer fired while 3 copies were already in flight, the loop hit continue and the rescue entry was permanently discarded — no code path ever re-enqueued it.

With 4+ buckets, this could drop the rescue for an untried replica, leaving it never attempted. The bug is masked under the default config (bucket_timeout = 5s < page_timeout = 10s): a copy's bucket_timeout expires before the latest rescue, freeing a slot so the reactive fallback (routing.best) compensates. It becomes available with bucket_timeout >= page_timeout (a valid, validated configuration exercised by several simulation scenarios), where all copies can stall to the page deadline before any slot frees. The bug was introduced in c7c9dbd (#139), which placed the eager pop_front() inside the rescue arm while gating launches only after the pop.

Fix

Defer the rescue instead of dropping it by adding active.len() < MAX_CONCURRENT_COPIES to the rescue select! arm's guard:

() = sleep_until(next_rescue),
    if !rescues.is_empty() && active.len() < MAX_CONCURRENT_COPIES =>
{
    (false, rescues.pop_front().map(|(_, index)| index))
},

When at full capacity, the guard is false so select! skips this arm — the rescue entry stays at the front of the queue and is re-polled on the next loop iteration once a slot frees. The next_rescue instant is derived from rescues.front() (unchanged), so the timer remains accurate. The existing tried[index] dedup guard for already-launched replicas is preserved. A #[allow(clippy::too_many_lines)] was added to download_replicas (it grew from 100 to 101 lines), consistent with existing usage elsewhere in the codebase.

Testing

• Added three unit tests in src/object_store/downloader.rs covering the previously-untested 4+ bucket rescue-at-capacity path:
  • A 4-replica case where a body error frees a slot for a healthy fourth replica (bucket_timeout == page_timeout).
  • A 5-replica chained fallback through two unhealthy copies to a healthy fifth.
  • A 4-replica case where all copies stall to the deadline, verifying a clean timeout with exactly three backend requests and no leaked active copies.
• All 42 downloader unit tests, 6 stats unit tests, and 4 simulation regression tests pass.
• Full workspace test suite passes (104 tests, including 12 Docker/RustFS integration tests).
• The full simulation campaign (all scenarios × 3 seeds, with bucket_timeout == page_timeout scenarios such as cold_regional_failover, regional_deadline, and finite_capacity) completes with no regression in completion counts or service work.
• Clippy (-D warnings), nightly rustfmt, and git diff --check all pass.

Closes #143

---

Automatic Fixes PRs can be configured here.

[greptile-apps]

PR author is not in the allowed authors list.