From 971746f1980c3d5eb91358c7f305e25f1de1ac0d Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Wed, 18 Nov 2026 16:15:04 -0600 Subject: [PATCH 1/4] fix(node): bound the write guard's lock-pool hold with a lock-hold deadline A RepoWriteGuard pins one dedicated advisory-lock pool connection for its whole life: taken at acquire_write, held across receive-pack under the full git_service_timeout, released only at release's upload tail. With the default 32-connection pool and a 600s git timeout, ~32 concurrent slow pushes deny every write on the node for up to ten minutes. Stamp one lock-hold deadline at lock-take, sized by the new TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS knob (unset: tracks git_service_timeout_secs; an explicit value above it is rejected at boot). Every phase under the lock consumes min(its own bound, the deadline's remaining budget) through the shared hold_bounded and hold_deadline clamps: the under-lock refresh, recovery prereqs, marker writes, the receive-pack subprocess, release's publish and unlock, and the mirror fetch's preflights and fetch. An expired deadline starts nothing: every clamp short-circuits rather than letting timeout(0) poll (and spawn) once, a pre-git expiry sheds as a git timeout, and an expired unlock closes the session instead of returning a lock-holding connection to the pool. A deadline-cut receive-pack keeps the lock until its process group is reaped, so the slot never frees while the write it guarded still runs. Refs #282 --- .env.example | 17 + README.md | 7 +- crates/twigpine-node/src/api/repos.rs | 478 ++++++++++++++++++-- crates/twigpine-node/src/config.rs | 131 ++++++ crates/twigpine-node/src/git/repo_store.rs | 351 ++++++++++++-- crates/twigpine-node/src/git/store.rs | 22 +- crates/twigpine-node/src/lib.rs | 6 +- crates/twigpine-node/src/upstream_mirror.rs | 130 +++++- crates/twigpine-node/tests/inv22_gates.rs | 21 +- 9 files changed, 1052 insertions(+), 111 deletions(-) diff --git a/.env.example b/.env.example index d18c7324..1d4b9d22 100644 --- a/.env.example +++ b/.env.example @@ -91,6 +91,23 @@ TWIGPINE_DB_LOCK_POOL_MAX_CONNECTIONS=40 # twice this value plus the git work between them. Read it together with the pool # size above and with TWIGPINE_GIT_SERVICE_TIMEOUT_SECS. TWIGPINE_LOCK_HELD_TRANSFER_TIMEOUT_SECS=300 +# Maximum seconds a repo write guard may hold its advisory-lock pool +# connection, measured from the moment the lock is taken. Every phase under +# the lock consumes this ONE deadline: the under-lock refresh, the caller's +# own under-lock work (for a push, the whole receive-pack subprocess, clamped +# to the lesser of TWIGPINE_GIT_SERVICE_TIMEOUT_SECS and the remaining +# budget), and the release-side upload/unlock. Without it a slow push pins a +# lock-pool slot for the SUM of every per-phase bound (~1500s worst case), so +# ~32 concurrent slow pushes deny every write on the node. Does NOT bound the +# acquire-side lock wait (TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS), post-git outcome +# bookkeeping, the ~4s the detached reaper holds the session while the git +# process group dies, or a caller's synchronous under-lock work that never +# consults the deadline (merge/issue writes still run unbounded git under +# their guards). Unset, the effective bound is TWIGPINE_GIT_SERVICE_TIMEOUT_SECS +# itself (600 by default), so it tracks the git budget up or down. Set it only +# to tighten the hold below the git budget; a value above +# TWIGPINE_GIT_SERVICE_TIMEOUT_SECS is rejected at boot. +#TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS=300 # Seconds a request waits for a pool connection before failing with 503. TWIGPINE_DB_ACQUIRE_TIMEOUT_SECS=5 # Upper bound on each startup connect+migrate attempt, in seconds. Keep it diff --git a/README.md b/README.md index 107cc23b..d8cdf2cb 100644 --- a/README.md +++ b/README.md @@ -67,8 +67,8 @@ Good today: Known limitations: -- Repository write authorization is one switch: `TWIGPINE_ENFORCE_OWNER_PUSH`, which defaults to `true`. With it off — rolling upgrades only — a valid HTTP Signature identifies a pusher but does not authorize one, so every signed caller may push everywhere. -- UCAN capabilities are consulted on the push path only. There, the chain's root issuer is anchored to the repository owner, so an owner-rooted, time-bounded delegation that names the repository grants scoped collaborator access for pushing to it, for the `git/push`, `*`, or `repo/admin` actions. A `*` *resource* is refused: a delegation must name the repository it applies to, so it cannot grow to cover repos created later. Push-only cuts the other way too: a delegation grants no read access, so on a private repository the delegate must also be a reader (`twigpine visibility set`) — the two gates a push crosses are described in [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). The rest is unchanged: there is no revocation path, `nb` constraints are refused rather than interpreted, and no other route — reads, pull requests, issues, agents — consults capabilities at all. +- Repository write authorization is one switch: `TWIGPINE_ENFORCE_OWNER_PUSH`, which defaults to `true`. With it off (rolling upgrades only) a valid HTTP Signature identifies a pusher but does not authorize one, so every signed caller may push everywhere. +- UCAN capabilities are consulted on the push path only. There, the chain's root issuer is anchored to the repository owner, so an owner-rooted, time-bounded delegation that names the repository grants scoped collaborator access for pushing to it, for the `git/push`, `*`, or `repo/admin` actions. A `*` *resource* is refused: a delegation must name the repository it applies to, so it cannot grow to cover repos created later. Push-only cuts the other way too: a delegation grants no read access, so on a private repository the delegate must also be a reader (`twigpine visibility set`); the two gates a push crosses are described in [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). The rest is unchanged: there is no revocation path, `nb` constraints are refused rather than interpreted, and no other route (reads, pull requests, issues, agents) consults capabilities at all. - Agent lifecycle revocation is not enforced by HTTP Signature authorization; do not rely on removing or revoking an agent record to block a compromised signer. - Task list/get reads (REST and GraphQL) are repository/task-gated and omit `ucan_token`. Slash-form `repo_id` rows (mirror-hosted tasks) can appear on list for the delegator or trimmed assignee, but `GET /api/v1/tasks/{id}` and GraphQL `task` fail closed for every caller because there is no local repository backing to serve. Read visibility is not a blanket data-classification boundary: IPFS-pin and Arweave-anchor listings are not repository-gated; withheld path names can be visible to a root reader; and later visibility changes cannot retract content already announced or externally anchored. - Peer writes are signed by upgraded nodes, but strict signed-peer enforcement is opt-in during rolling upgrades. @@ -427,12 +427,13 @@ Important node settings: | `TWIGPINE_BOOTSTRAP_DISABLE_SEEDS` | Disable embedded seed peers for isolated dev/test networks. | | `TWIGPINE_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. Defaults to `false` during the staged rollout below. | | `TWIGPINE_REQUIRE_SIGNATURE_NONCE` | Require a signed `nonce` on write routes. Default false; see the staged rollout note above. | -| `TWIGPINE_ENFORCE_OWNER_PUSH` | Require the pusher on `git-receive-pack` to be the repo owner, or to present an owner-rooted, time-bounded UCAN `git/push` capability naming the repository (`X-Ucan`). **Defaults to `true`.** A `did:key` signature is authentication, not authorization — anyone can mint a key and sign — so with this off every signed caller may push to every repository, private ones included. A CI or delegated key with no such delegation counts as a non-owner and cannot push while this is on. The delegation clears the push gate only; on a private repository the delegate must also be a reader. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). | +| `TWIGPINE_ENFORCE_OWNER_PUSH` | Require the pusher on `git-receive-pack` to be the repo owner, or to present an owner-rooted, time-bounded UCAN `git/push` capability naming the repository (`X-Ucan`). **Defaults to `true`.** A `did:key` signature is authentication, not authorization (anyone can mint a key and sign), so with this off every signed caller may push to every repository, private ones included. A CI or delegated key with no such delegation counts as a non-owner and cannot push while this is on. The delegation clears the push gate only; on a private repository the delegate must also be a reader. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). | | `TWIGPINE_AUTO_SYNC` | Enable automatic sync from known peers. | | `TWIGPINE_RECONCILIATION_SWEEP` | Enable the hourly durability sweep that re-pins/backstops missing objects (default `true`; disabled when no IPFS/Pinata backend is configured). Public pin repair runs against any configured backend (IPFS, Pinata, or both). Encrypted recovery repair requires local IPFS (`TWIGPINE_IPFS_API`); Pinata-only nodes reconcile public pins only. | | `TWIGPINE_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. | | `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. | | `TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. | +| `TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS` | Max seconds a repo write guard may hold its advisory-lock pool connection, measured from lock-take. One deadline covers every phase under the lock: the under-lock refresh, the caller's under-lock work (for a push, the whole `receive-pack` subprocess, clamped to the lesser of `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), and the release-side upload/unlock. It bounds the lock-pool slot's hold, so a slow push cannot starve other writers for the sum of every per-phase bound. It does not bound the acquire-side lock wait (`TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS`), post-git outcome bookkeeping, the few seconds the detached reaper holds the session while the git process group dies, or a caller's synchronous under-lock work that never consults the deadline (merge/issue writes still run unbounded git under their guards). Unset, the effective bound is `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` itself (600 by default), tracking the git budget; set it only to tighten the hold below the git budget. An explicit value above `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` is rejected at boot. | | `TWIGPINE_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `TWIGPINE_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. | | `TWIGPINE_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `TWIGPINE_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `TWIGPINE_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | diff --git a/crates/twigpine-node/src/api/repos.rs b/crates/twigpine-node/src/api/repos.rs index bfe73d24..7fe48278 100644 --- a/crates/twigpine-node/src/api/repos.rs +++ b/crates/twigpine-node/src/api/repos.rs @@ -3006,10 +3006,40 @@ pub async fn git_receive_pack( } Ok(Ok(g)) => g, }; + // The guard stamps its lock-hold deadline at lock-take inside + // acquire_write. Every phase that runs while the lock is held consumes + // the remaining budget of that one instant, recomputed at each clamp + // rather than carried as a duration that silently ages: a slow push is + // cut at the deadline, so it cannot pin its lock-pool slot for the sum + // of every per-phase bound (#282). + let lock_hold_deadline = guard.lock_hold_deadline(); + let git_service_timeout = std::time::Duration::from_secs(state.config.git_service_timeout_secs); + // `timeout(0, drive_git_child)` would still spawn git on its first poll, + // so an expired deadline sheds here rather than racing inside the bound. + // Shed as a git timeout, matching what a mid-git deadline expiry returns. + // The shed must precede `guard.path()`: taking the path declares the + // tree handed out, and a dropped guard would quarantine a tree no phase + // ever touched. + let Some(git_timeout) = + crate::git::repo_store::hold_bounded(lock_hold_deadline, git_service_timeout) + else { + if let Err(e) = state + .db + .refuse_receive_pack_before_git( + &request_id, + "write-lock hold deadline expired before git", + ) + .await + { + tracing::warn!(err = %e, request_id = %request_id, "pre-git refusal cleanup failed"); + } + return Err(AppError::Timeout( + "write-lock hold deadline expired before git could start".into(), + )); + }; let disk_path = guard.path().to_path_buf(); tracing::debug!(repo = %name, path = %disk_path.display(), "running git receive-pack"); let body_len = body.len(); - let git_timeout = std::time::Duration::from_secs(state.config.git_service_timeout_secs); // Recovery upgrade invariant (best-effort): before the first // durable intent or marker relies on reflogs/hidden refs, // idempotently enable them for new and upgraded repos. Failures @@ -3021,11 +3051,17 @@ pub async fn git_receive_pack( // Bounded through the configured git binary: the unbounded // synchronous variant would run several `git config` children with // no timeout while holding both admission permits, the lease, and - // the repo write lock. + // the repo write lock. The bound is an Instant, not a duration: the + // helper runs several children, and one shared deadline stops a slow + // sequence from spending the budget several times over. if let Err(e) = crate::git::store::verify_recovery_prereqs_bounded( &state.git_bin, &disk_path, - std::time::Duration::from_secs(10), + crate::git::repo_store::hold_deadline( + lock_hold_deadline, + std::time::Duration::from_secs(10), + ) + .unwrap_or_else(std::time::Instant::now), ) .await { @@ -3079,40 +3115,51 @@ pub async fn git_receive_pack( // marker through the bounded runners using the configured git binary: // both hold admission permits, the lease, and the write lock, so an // unbounded `hash-object` could pin them all on a hung child. - match crate::git::store::marker_value_for_bounded( - &state.git_bin, - &disk_path, - &req_row.request_bytes_hash, - std::time::Duration::from_secs(10), - ) - .await + // `hold_bounded` returns None on an expired deadline, which skips the + // call outright rather than letting `timeout(0)` spawn the child once. + if let Some(marker_value_bound) = + crate::git::repo_store::hold_bounded(lock_hold_deadline, std::time::Duration::from_secs(10)) { - Ok(marker_value) => { - if let Err(e) = crate::git::store::write_marker_bounded( - &state.git_bin, - &disk_path, - &request_id, - &marker_value, - std::time::Duration::from_secs(10), - ) - .await - { + match crate::git::store::marker_value_for_bounded( + &state.git_bin, + &disk_path, + &req_row.request_bytes_hash, + marker_value_bound, + ) + .await + { + Ok(marker_value) => { + if let Some(marker_write_bound) = crate::git::repo_store::hold_bounded( + lock_hold_deadline, + std::time::Duration::from_secs(10), + ) { + if let Err(e) = crate::git::store::write_marker_bounded( + &state.git_bin, + &disk_path, + &request_id, + &marker_value, + marker_write_bound, + ) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "bounded marker write failed; reconcile will quarantine this request" + ); + } + } + } + Err(e) => { tracing::warn!( err = %e, request_id = %request_id, repo = %name, - "bounded marker write failed; reconcile will quarantine this request" + "marker value computation failed; reconcile will quarantine this request" ); } } - Err(e) => { - tracing::warn!( - err = %e, - request_id = %request_id, - repo = %name, - "marker value computation failed; reconcile will quarantine this request" - ); - } } // P1 (reviewer-1/2 round 3): use receive_pack_raw to get the raw @@ -3134,12 +3181,33 @@ pub async fn git_receive_pack( // guard then marks children uncertain and terminalizes the parent, which // the mark_uncertain path below cannot do on a drop — it never runs, and // the detached reaper does process teardown only, no DB writes. + // Re-clamp at the call: the prereq and marker phases above spent the + // same shared deadline, and a bound stamped before them would let + // receive-pack overshoot it by whatever they took. `hold_bounded` is + // also the zero check: an expired deadline sheds as a git timeout + // instead of spawning a receive-pack that is killed on the first poll. + let Some(run_bound) = crate::git::repo_store::hold_bounded(lock_hold_deadline, git_timeout) + else { + if let Err(e) = state + .db + .refuse_receive_pack_before_git( + &request_id, + "write-lock hold deadline expired before git", + ) + .await + { + tracing::warn!(err = %e, request_id = %request_id, "pre-git refusal cleanup failed"); + } + return Err(AppError::Timeout( + "write-lock hold deadline expired before git could start".into(), + )); + }; _intent_guard.notify_git_started(); let (receive_raw, exit_ok) = match smart_http::receive_pack_raw_with_reflog( &state.git_bin, &disk_path, body, - git_timeout, + run_bound, Some(admission), Some(&reflog_action), ) @@ -19296,6 +19364,356 @@ exit 0 "a damaged ref must surface as a git error, not an empty listing" ); } + /// #282 REPRO (RED on main): a push's write guard pins one dedicated + /// lock-pool connection for the WHOLE op: taken at `acquire_write`, held + /// across `smart_http::receive_pack` under `git_service_timeout_secs` + /// (default 600), released only at `release`'s upload tail, with no + /// hold-time bound of its own. A saturated pool then starves every other + /// repo's `acquire_write`, not just the slow pusher's. + /// + /// A one-connection lock pool stands in for "the dedicated pool is full": + /// one held guard is the smallest case of the flood the issue scales to + /// pool size. The shim sleeps on `receive-pack` alone (a slow-but-legit + /// push; prereq and marker children stay real and fast), so the held + /// window is the git phase itself. The probe asserts the desired + /// behaviour (an unrelated write is served while the slow push still + /// runs) and goes RED on main, which is the defect. + #[cfg(unix)] + #[sqlx::test] + async fn repro_282_slow_push_lock_hold_starves_unrelated_writes(pool: sqlx::PgPool) { + use axum::extract::{Path, State}; + use axum::Extension; + use std::net::SocketAddr; + + let tmp = tempfile::TempDir::new().unwrap(); + let repos_dir = tmp.path().join("repos"); + std::fs::create_dir_all(&repos_dir).unwrap(); + let lock_pool = sqlx::postgres::PgPoolOptions::new() + .max_connections(1) + .connect_with((*pool.connect_options()).clone()) + .await + .expect("size-1 lock pool on the test database"); + let mut state = crate::test_support::test_state(pool.clone()).await; + state.repo_store = + crate::git::repo_store::RepoStore::for_testing(repos_dir, lock_pool.clone()) + .with_lock_hold_timeout(std::time::Duration::from_secs(2)); + state.push_limiter_trust = crate::rate_limit::TrustedProxy::None; + let shim_dir = tempfile::TempDir::new().unwrap(); + let marker = shim_dir.path().join("receive-pack-started"); + state.git_bin = write_fake_git( + shim_dir.path(), + &format!( + "#!/bin/sh\ncase \" $*\" in *receive-pack*) : > \"{}\"; sleep 15;; esac\nexec git \"$@\"\n", + marker.display() + ), + ); + + let owner_a = "z6hold282a".to_string(); + let name_a = "slowrepo"; + let owner_b = "z6hold282b".to_string(); + let name_b = "otherrepo"; + let mut repo_a_id = String::new(); + for (owner, name) in [(&owner_a, name_a), (&owner_b, name_b)] { + state + .db + .upsert_mirror_repo(owner, name, &format!("/unused/{owner}/{name}"), None, false) + .await + .unwrap(); + let rec = state.db.get_repo(owner, name).await.unwrap().unwrap(); + state + .repo_store + .init(&rec.owner_did, &rec.name) + .await + .unwrap(); + if name == name_a { + repo_a_id = rec.id.clone(); + } + } + + let state_a = state.clone(); + let owner_push = owner_a.clone(); + let push = tokio::spawn(async move { + git_receive_pack( + State(state_a), + Path((owner_push.clone(), name_a.to_string())), + Extension(crate::auth::AuthenticatedDid(format!( + "did:key:{owner_push}" + ))), + None, + crate::rate_limit::PeerAddr(Some( + "203.0.113.77:5000".parse::().unwrap(), + )), + axum::http::HeaderMap::new(), + ref_update_body(&"a".repeat(40)), + ) + .await + }); + + // The push must reach its git phase (the only lock-pool connection + // pinned) before the unrelated write is probed. + let pin_deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while lock_pool.num_idle() != 0 { + if push.is_finished() { + let r = push.await; + panic!("push returned before pinning the lock pool: {r:?}"); + } + assert!( + std::time::Instant::now() < pin_deadline, + "push never pinned the lock pool" + ); + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + } + + // The pin alone does not prove the git phase: `acquire_write` checks + // the connection out before the advisory lock is even attempted, so + // an early-dead push would free the slot without the bound ever + // firing and the probe below would pass vacuously. Wait for the + // shim's own receive-pack marker so the starved write is only probed + // while the push is genuinely sleeping inside git. + let marker_deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); + while !marker.exists() { + if push.is_finished() { + let r = push.await; + panic!("push returned before reaching its git phase: {r:?}"); + } + assert!( + std::time::Instant::now() < marker_deadline, + "push never reached git receive-pack" + ); + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + } + + // The hold bound (2s) plus process-group reap (~4s cap) frees the + // slot well inside this window; the shim's own 15s sleep is what the + // bound is racing, so a window shorter than the sleep proves the + // bound, not the sleep's end, freed the slot. + let starved = tokio::time::timeout( + std::time::Duration::from_secs(9), + state.repo_store.acquire_write(&owner_b, name_b), + ) + .await; + let served = matches!(&starved, Ok(Ok(_))); + assert!( + served, + "#282: an unrelated repo's acquire_write must be served once the \ + slow push's lock-hold deadline passes (still pending after 9s of \ + a 15s git sleep with a 2s hold bound)" + ); + let push_result = push.await.expect("push task"); + assert!( + push_result.is_err(), + "a push cut by its lock-hold deadline must fail, not report success" + ); + + // A push cut mid-git cannot prove its refs did not land, so the + // intent children must go reconcile-owned `uncertain` (or stay + // `prepared` for startup reconcile), never terminal `cancelled`. + let states: Vec<(String,)> = + sqlx::query_as("SELECT state FROM pending_ref_transitions WHERE repo_id = $1") + .bind(&repo_a_id) + .fetch_all(&pool) + .await + .expect("pending_ref_transitions rows for repo A"); + assert!( + !states.is_empty(), + "the push carried one ref update, so an intent child must exist" + ); + for (state_str,) in &states { + assert!( + state_str == crate::db::pending_state::UNCERTAIN + || state_str == crate::db::pending_state::PREPARED, + "a deadline-cut push's outcome rows must stay reconcile-owned, got {state_str}" + ); + } + } + + /// #282, the other half of the bound: it must never free the lock EARLY. + /// When the hold deadline ends a push the existing git-timeout machinery + /// tears the process group down, and the reaper-carried guard keeps the + /// advisory lock until the group is confirmed gone, the same invariant + /// `receive_pack_disconnect_holds_the_write_lock_until_the_group_is_reaped` + /// pins for the disconnect path, here driven by the deadline instead. + /// + /// A fake git whose descendant ignores SIGTERM widens the teardown window: + /// after the handler's `git_timeout` clamp fires, the group survives the + /// reaper's SIGTERM grace and dies only on the SIGKILL escalation. The lock + /// probed from an independent session must stay untakeable the whole time + /// the descendant lives, and must be takeable once it is gone. + #[cfg(unix)] + #[sqlx::test] + async fn receive_pack_lock_hold_deadline_keeps_the_lock_until_the_group_is_reaped( + pool: sqlx::PgPool, + ) { + use axum::extract::{Path, State}; + use axum::Extension; + use std::net::SocketAddr; + + let owner = "z6deadline"; + let name = "dl1"; + let repos_dir = tempfile::TempDir::new().unwrap(); + let tmp = tempfile::TempDir::new().unwrap(); + let descfile = tmp.path().join("desc.pid"); + let body = format!( + "#!/bin/sh\n\ + sh -c 'trap \"\" TERM; echo $$ > \"{}\"; i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done' &\n\ + wait\n", + descfile.display() + ); + let git_bin = write_fake_git(tmp.path(), &body); + + let mut state = crate::test_support::test_state(pool.clone()).await; + state.git_bin = git_bin; + state.repo_store = crate::git::repo_store::RepoStore::new( + repos_dir.path().to_path_buf(), + None, + crate::git::repo_store::build_lock_pool(&pool, 4, std::time::Duration::from_secs(5)), + std::time::Duration::from_secs(300), + ) + .with_lock_hold_timeout(std::time::Duration::from_secs(2)); + let mut cfg = (*state.config).clone(); + // Long enough that the service timeout is never what ends this push; + // the lock-hold deadline is. + cfg.git_service_timeout_secs = 600; + cfg.enforce_owner_push = false; + state.config = std::sync::Arc::new(cfg); + state + .db + .upsert_mirror_repo(owner, name, "/tmp/z6deadline-dl1", None, false) + .await + .unwrap(); + + use crate::git::repo_store::advisory_lock_key; + let key = advisory_lock_key(&owner.replace([':', '/'], "_"), name); + let probe = sqlx::postgres::PgPoolOptions::new() + .max_connections(2) + .connect_lazy_with((*pool.connect_options()).clone()); + + assert!( + write_lock_is_takeable(&probe, key).await, + "the write lock must be free before the push" + ); + + // Same spawn-retry shape as the disconnect test: a freshly written + // fake `git` can transiently fail to exec under fork-storm load, and a + // losing attempt's future is dropped, which cleans up after itself. + const SPAWN_ATTEMPTS: u64 = 12; + let (mut fut, desc) = { + let mut attempt = 0u64; + loop { + attempt += 1; + let _ = std::fs::remove_file(&descfile); + let mut fut = Box::pin(git_receive_pack( + State(state.clone()), + Path((owner.to_string(), name.to_string())), + Extension(crate::auth::AuthenticatedDid( + "did:key:z6deadline".to_string(), + )), + None, + crate::rate_limit::PeerAddr(Some( + "203.0.113.82:5000".parse::().unwrap(), + )), + axum::http::HeaderMap::new(), + axum::body::Bytes::from_static(b"0000"), + )); + let mut found: Option = None; + for _ in 0..500 { + let finished = + tokio::time::timeout(std::time::Duration::from_millis(10), &mut fut) + .await + .is_ok(); + if let Some(p) = std::fs::read_to_string(&descfile) + .ok() + .and_then(|s| s.trim().parse::().ok()) + { + found = Some(p); + break; + } + if finished { + break; + } + } + match found { + Some(p) => break (fut, p), + None => { + drop(fut); + assert!( + attempt < SPAWN_ATTEMPTS, + "the push never reached receive-pack after {SPAWN_ATTEMPTS} \ + attempts (persistent failure, not a transient runner miss)" + ); + tokio::time::sleep(std::time::Duration::from_millis(100 * attempt)).await; + } + } + } + }; + let _cleanup = KillOnPanic(vec![desc]); + assert!( + pid_alive(desc), + "the receive-pack group must be running when the deadline fires" + ); + + // The 2s hold deadline ends the push. On the timeout path the future + // stays pending while `reap_group_on_timeout` gives the group its + // SIGTERM grace and then SIGKILLs: the SIGTERM-ignoring descendant + // lives through that whole window, and the write lock (carried by the + // guard inside the admission bundle) must stay untakeable for all of + // it. A bound that freed early would show up here as takeable while + // the group is still on the repo. + let mut takeable_while_group_alive = false; + let mut saw_live_group_window = false; + let poll_deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); + let result = loop { + let takeable = write_lock_is_takeable(&probe, key).await; + let group_alive = pid_alive(desc); + if group_alive { + saw_live_group_window = true; + } + if takeable && group_alive { + takeable_while_group_alive = true; + } + if let Ok(done) = + tokio::time::timeout(std::time::Duration::from_millis(10), &mut fut).await + { + break done; + } + assert!( + std::time::Instant::now() < poll_deadline, + "the deadline-cut push never returned within 20s" + ); + }; + // Clean up regardless so a failure leaves no orphan behind. + // SAFETY: kill(2) takes integers only. + unsafe { + libc::kill(desc, libc::SIGKILL); + } + assert!( + result.is_err(), + "a push cut by its lock-hold deadline must fail, not report success" + ); + assert!( + saw_live_group_window, + "the SIGTERM-ignoring descendant should outlive part of teardown; \ + without that the early-free check below observed nothing" + ); + assert!( + !takeable_while_group_alive, + "the repo write lock was takeable while the deadline-cut push's \ + receive-pack group was still alive: the hold bound must never \ + free the lock early (#282)" + ); + // Postgres frees the advisory lock when it notices the closed session, + // which trails the connection drop by a scheduler's tick: poll rather + // than probe once, so a loaded database cannot make this flap. + let free_deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); + while !write_lock_is_takeable(&probe, key).await { + assert!( + std::time::Instant::now() < free_deadline, + "once the deadline-cut push's group is reaped the write lock must \ + be free for the next writer" + ); + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + } + } } #[cfg(test)] diff --git a/crates/twigpine-node/src/config.rs b/crates/twigpine-node/src/config.rs index 728ec717..4bce7b6a 100644 --- a/crates/twigpine-node/src/config.rs +++ b/crates/twigpine-node/src/config.rs @@ -459,6 +459,34 @@ pub struct Config { )] pub lock_held_transfer_timeout_secs: u64, + /// Maximum wall-clock time a `RepoWriteGuard` may hold its advisory-lock + /// pool connection, measured from the instant `pg_try_advisory_lock` + /// succeeds, in seconds. Every phase that runs while the lock is held + /// (the under-lock Tigris refresh, the caller's own under-lock work; for a + /// push, the whole `receive-pack` subprocess, clamped to the lesser of + /// `git_service_timeout_secs` and the remaining budget, and `release`'s + /// bounded upload/unlock) consumes this one deadline, so a slow or + /// stalled push cannot pin a lock-pool slot for the sum of every + /// per-phase bound (~1500s worst case before this knob existed). On + /// expiry the in-flight git op is torn down and the write fails rather + /// than publishing late. Does NOT bound the acquire-side wait for the + /// lock (`git_acquire_timeout_secs`), the post-git outcome bookkeeping on + /// the main pool, the ~2-4s the detached reaper holds the session while + /// the process group dies, or a caller's synchronous under-lock work that + /// never consults the deadline (the merge/issue write paths still run + /// unbounded git under their guards). Must be positive and at most + /// `git_service_timeout_secs` (a hold bound above the git timeout can + /// never bind before it). Unset (the default) the effective bound is + /// `git_service_timeout_secs` itself (600s shipped), so every push keeps + /// its stated budget and lowering the git timeout lowers the hold bound + /// with it; set explicitly only to tighten the hold below the git budget. + #[arg( + long, + env = "TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS", + value_parser = clap::value_parser!(u64).range(1..=GIT_SERVICE_TIMEOUT_SECS_MAX) + )] + pub write_lock_hold_timeout_secs: Option, + /// Maximum time a request waits for a pool connection before failing with /// 503, in seconds. Bounds queueing when the database is slow or down. #[arg( @@ -1083,6 +1111,16 @@ impl Config { /// lock-pool exhaustion on unrelated repositories. pub const DB_LOCK_POOL_NON_PUSH_HEADROOM: u32 = 8; + /// Effective write-lock hold bound: the explicit knob when set, else + /// `git_service_timeout_secs`, so an unset bound always tracks the push + /// budget (and a lowered git timeout lowers it too). + pub fn write_lock_hold_timeout(&self) -> std::time::Duration { + std::time::Duration::from_secs( + self.write_lock_hold_timeout_secs + .unwrap_or(self.git_service_timeout_secs), + ) + } + /// Cross-field boot validation. Single-field ranges are enforced by clap; this /// catches combinations that ship a denial-of-service under otherwise-valid /// values. Call once at startup and fail fast on `Err`. @@ -1110,6 +1148,16 @@ impl Config { Self::DB_LOCK_POOL_NON_PUSH_HEADROOM )); } + if let Some(hold) = self.write_lock_hold_timeout_secs { + if hold > self.git_service_timeout_secs { + return Err(format!( + "TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS ({}) must not exceed \ + TWIGPINE_GIT_SERVICE_TIMEOUT_SECS ({}): a hold bound above the \ + git timeout can never bind before it", + hold, self.git_service_timeout_secs + )); + } + } if self.upstream_mirror_enabled && !self.enforce_owner_push { return Err( "TWIGPINE_UPSTREAM_MIRROR_ENABLED requires TWIGPINE_ENFORCE_OWNER_PUSH=true: \ @@ -1397,6 +1445,89 @@ mod tests { ); } + #[test] + fn write_lock_hold_timeout_defaults_to_git_budget_and_rejects_zero() { + let default = Config::parse_from(["twigpine-node"]); + assert_eq!(default.write_lock_hold_timeout_secs, None); + assert_eq!( + default.write_lock_hold_timeout(), + std::time::Duration::from_secs(default.git_service_timeout_secs), + "an unset hold bound tracks the git service budget so every push keeps its stated budget" + ); + assert_eq!( + Config::parse_from(["twigpine-node", "--write-lock-hold-timeout-secs", "120"]) + .write_lock_hold_timeout_secs, + Some(120) + ); + // 0 would expire every write guard the instant its lock lands; clap + // must reject it rather than boot a node that can never write. + assert!( + Config::try_parse_from(["twigpine-node", "--write-lock-hold-timeout-secs", "0"]) + .is_err() + ); + // Above the 100-year ceiling every derived deadline can represent. + assert!(Config::try_parse_from([ + "twigpine-node", + "--write-lock-hold-timeout-secs", + "3153600001" + ]) + .is_err()); + } + + /// An unset hold bound must not turn a lowered git timeout into a boot + /// failure: the effective bound simply tracks the configured git budget + /// down with it. + #[test] + fn write_lock_hold_timeout_unset_tracks_a_lowered_git_budget() { + let mut config = Config::parse_from(["twigpine-node", "--git-service-timeout-secs", "120"]); + config + .validate() + .expect("a lowered git timeout with the hold bound unset must validate"); + assert_eq!( + config.write_lock_hold_timeout(), + std::time::Duration::from_secs(120) + ); + } + + /// `validate()` rejects a hold bound above the git service timeout: such a + /// bound can never bind before the git op's own timeout, so accepting it + /// would let an operator believe a bound exists that never fires. + #[test] + fn write_lock_hold_timeout_must_not_exceed_the_git_budget() { + // Equality is legal: the bound fires exactly at the git timeout. + Config::parse_from([ + "twigpine-node", + "--git-service-timeout-secs", + "600", + "--write-lock-hold-timeout-secs", + "600", + ]) + .validate() + .expect("hold bound equal to the git budget must validate"); + + Config::parse_from([ + "twigpine-node", + "--git-service-timeout-secs", + "600", + "--write-lock-hold-timeout-secs", + "120", + ]) + .validate() + .expect("hold bound below the git budget must validate"); + + let mut over = Config::parse_from([ + "twigpine-node", + "--git-service-timeout-secs", + "60", + "--write-lock-hold-timeout-secs", + "61", + ]); + assert!( + over.validate().is_err(), + "a hold bound above the git service timeout must be rejected at boot" + ); + } + #[test] fn require_signature_nonce_ships_off() { assert!( diff --git a/crates/twigpine-node/src/git/repo_store.rs b/crates/twigpine-node/src/git/repo_store.rs index 986aa391..29b43d31 100644 --- a/crates/twigpine-node/src/git/repo_store.rs +++ b/crates/twigpine-node/src/git/repo_store.rs @@ -13,7 +13,7 @@ use std::collections::HashSet; use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; -use std::time::Duration; +use std::time::{Duration, Instant}; use anyhow::{Context, Result}; use sqlx::postgres::PgPoolOptions; @@ -64,6 +64,12 @@ pub struct RepoStore { lock_pool: PgPool, /// Bound on any object-storage transfer that runs while the lock is HELD. lock_held_transfer_timeout: Duration, + /// Wall-clock cap on HOLDING the lock, stamped onto every guard at + /// lock-take. One deadline covers every phase under the lock, so a slow + /// caller cannot pin a lock-pool slot for the sum of every per-phase + /// bound. A field rather than a bare const so the bound is configurable + /// (`TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS`) and testable. + lock_hold_timeout: Duration, /// Wall-clock cap on WAITING for the lock. A field rather than a bare const so /// the busy path can be driven in a test without a 90s wait. lock_acquire_deadline: Duration, @@ -116,6 +122,17 @@ impl RepoStore { self } + /// Bound on how long a `RepoWriteGuard` from this store may hold its lock, + /// stamped as a deadline at lock-take. Every phase under the lock consumes + /// `min(its own bound, the deadline's remaining budget)`. + /// + /// Production: set from `Config::write_lock_hold_timeout_secs`. Tests: a + /// small value drives the expired-deadline path without a 600s wait. + pub fn with_lock_hold_timeout(mut self, timeout: Duration) -> Self { + self.lock_hold_timeout = timeout; + self + } + /// Test-only: how many write guards from this store have reached the Tigris upload /// site. See [`RepoWriteGuard::release`]. #[cfg(all(test, unix))] @@ -135,6 +152,7 @@ impl RepoStore { tigris, lock_pool, lock_held_transfer_timeout, + lock_hold_timeout: LOCK_HOLD_TIMEOUT, lock_acquire_deadline: LOCK_ACQUIRE_DEADLINE, migrated: Arc::new(Mutex::new(HashSet::new())), #[cfg(test)] @@ -648,6 +666,7 @@ impl RepoStore { conn: Some(lock_conn), tigris: self.tigris.clone(), lock_held_transfer_timeout: self.lock_held_transfer_timeout, + lock_hold_deadline: Instant::now() + self.lock_hold_timeout, // Overwritten by the refresh below with the generation actually // observed under the lock. Only reachable unset when no backend is // configured, in which case `release` publishes nothing at all. @@ -678,11 +697,17 @@ impl RepoStore { // bounding only the download would leave a mute endpoint able to hold // both indefinitely on the HEAD, and bounding them separately would make // worst-case occupancy two budgets instead of one. - let refreshed = bounded_transfer( - "acquire-refresh", - repo_name, - self.lock_held_transfer_timeout, - async { + // The refresh's own bound, clamped to what is left of this + // guard's lock-hold deadline: the two compose, they do not add. + let refresh_bound = guard.hold_bounded(self.lock_held_transfer_timeout); + let refreshed = match refresh_bound { + None => None, + Some(bound) => { + bounded_transfer( + "acquire-refresh", + repo_name, + bound, + async { // The HEAD and the download fail for epistemically DIFFERENT // reasons, so they are kept apart rather than collapsed into one // `Result`. A failed HEAD leaves us not knowing whether an archive @@ -721,8 +746,10 @@ impl RepoStore { Err(e) => Err(RefreshFailure::Unknown(e)), } }, - ) - .await; + ) + .await + } + }; match refreshed { Some(Ok(fence)) => { @@ -784,16 +811,30 @@ impl RepoStore { // the whole bound is not that: it pinned a lock-pool slot for the // full duration, and this raise-site `error!` is what keeps it // paging. Do NOT "fix" it to match the arm above. - tracing::error!( - repo = %repo_name, - owner = %owner_slug, - bound_secs = self.lock_held_transfer_timeout.as_secs(), - "under-lock tigris refresh exceeded the transfer bound, refusing the write" - ); - return Err(anyhow::Error::new(RepoUnavailable).context(format!( - "tigris refresh exceeded the {}s under-lock bound for {owner_slug}/{repo_name}", - self.lock_held_transfer_timeout.as_secs() - ))); + match refresh_bound { + Some(bound) => { + tracing::error!( + repo = %repo_name, + owner = %owner_slug, + bound_secs = bound.as_secs(), + "under-lock tigris refresh exceeded its bound, refusing the write" + ); + return Err(anyhow::Error::new(RepoUnavailable).context(format!( + "tigris refresh exceeded the {}s under-lock bound for {owner_slug}/{repo_name}", + bound.as_secs() + ))); + } + None => { + tracing::error!( + repo = %repo_name, + owner = %owner_slug, + "under-lock tigris refresh never started: the lock-hold deadline left no budget, refusing the write" + ); + return Err(anyhow::Error::new(RepoUnavailable).context(format!( + "the lock-hold deadline left no budget for the under-lock tigris refresh for {owner_slug}/{repo_name}" + ))); + } + } } } } @@ -2054,6 +2095,13 @@ pub struct RepoWriteGuard { tigris: Option, /// Bound on the release-side upload, which runs with the lock still held. lock_held_transfer_timeout: Duration, + /// The instant this guard's lock-pool hold expires, stamped at lock-take + /// from `RepoStore::lock_hold_timeout`. Every phase that runs while the + /// lock is held consumes `min(its own bound, the deadline's remaining + /// budget)` through `hold_bounded`, so one deadline covers refresh, the + /// caller's under-lock work, and release rather than letting each spend a + /// full budget of its own. + lock_hold_deadline: Instant, /// The generation of the stored archive as observed by the HEAD inside /// `acquire_write`, under the lock. `release` publishes fenced on it, so a /// PUT abandoned by an earlier writer's timeout cannot land on top of a @@ -2108,6 +2156,20 @@ pub struct RepoWriteGuard { } impl RepoWriteGuard { + /// The deadline stamped at lock-take. Exposed so a caller that keeps the + /// guard in shared state (the receive-pack reaper bundle) can clamp every + /// under-lock phase against the same instant instead of a duration that + /// silently ages. + pub(crate) fn lock_hold_deadline(&self) -> Instant { + self.lock_hold_deadline + } + + /// `own` clamped to what this guard's lock-hold deadline still funds; see + /// the module-level `hold_bounded` for the rule a `None` carries. + fn hold_bounded(&self, own: Duration) -> Option { + hold_bounded(self.lock_hold_deadline, own) + } + /// Backend pid of the session holding the lock. Test-only observable for the /// must-not-over-close check: if `release` closed the session instead of /// returning it, consecutive writes would report different pids. @@ -2124,6 +2186,25 @@ impl RepoWriteGuard { pid.0 } + /// Backend pid PLUS its start timestamp. `pg_backend_pid` alone cannot tell + /// "the same session" from "the kernel recycled the pid for the next + /// backend", which the pool-exhaustion tests hit under parallel load; a + /// recycled pid still gets a fresh `backend_start`. + #[cfg(test)] + async fn backend_identity_for_test(&mut self) -> (i32, String) { + let conn = self + .conn + .as_mut() + .expect("guard still holds its connection"); + sqlx::query_as( + "SELECT pid, backend_start::text FROM pg_stat_activity \ + WHERE pid = pg_backend_pid()", + ) + .fetch_one(&mut **conn) + .await + .expect("backend identity") + } + /// Path to the bare repo on local disk. /// /// Handing the tree out is also the DECLARATION that a write may land on it. @@ -2315,15 +2396,23 @@ impl RepoWriteGuard { // A separate, deliberately small slice rather than a share of the // already-exhausted publish budget: this runs with the lock still held, // and a store that just stalled for the whole bound must not be able to - // hold it for a second one. - match bounded_transfer( - "release-reconcile", - &self.repo_name, - RECONCILE_BOUND, - tigris.attempt_landed(&self.owner_slug, &self.repo_name, &attempt), - ) - .await - { + // hold it for a second one. An expired lock-hold deadline starts + // nothing: `hold_bounded` returns None rather than let `timeout(0)` + // poll the probe once and possibly land an answer the deadline no + // longer funds. + let reconciled = match self.hold_bounded(RECONCILE_BOUND) { + None => None, + Some(bound) => { + bounded_transfer( + "release-reconcile", + &self.repo_name, + bound, + tigris.attempt_landed(&self.owner_slug, &self.repo_name, &attempt), + ) + .await + } + }; + match reconciled { Some(Ok(true)) => { info!( repo = %self.repo_name, @@ -2382,14 +2471,24 @@ impl RepoWriteGuard { // refresh uses one for its HEAD and download together: this runs // with the lock held and a lock-pool slot pinned, so bounding // each attempt separately would double the worst-case occupancy. - match bounded_transfer( - "release-upload", - &self.repo_name, - self.lock_held_transfer_timeout, - self.publish(&tigris), - ) - .await - { + // The bound also composes with the lock-hold deadline: a guard + // whose deadline has passed starts nothing, classified through + // `resolve_unfinished_publish` as definite non-publication at + // stage Idle. The skip is explicit because `timeout(0)` would + // still poll the publish once. + let published = match self.hold_bounded(self.lock_held_transfer_timeout) { + None => None, + Some(bound) => { + bounded_transfer( + "release-upload", + &self.repo_name, + bound, + self.publish(&tigris), + ) + .await + } + }; + match published { Some(Ok(())) => {} // Both attempts were definitively refused. The raise site // already logged which and why, so this only has to carry @@ -2519,18 +2618,26 @@ impl RepoWriteGuard { if let Some(gate) = self.test_pre_unlock_gate.clone() { gate.notified().await; } - let unlock = if let Some(conn) = self.conn.as_mut() { - bounded_transfer( - "advisory-unlock", - &self.repo_name, - self.lock_held_transfer_timeout, - sqlx::query_as::<_, (bool,)>("SELECT pg_advisory_unlock($1)") - .bind(lock_key) - .fetch_one(&mut **conn), - ) - .await - } else { - None + // A lock-hold deadline already past is excluded by `hold_bounded` + // rather than left to `timeout(0)`, which would still poll the + // query once and could land it. + let unlock_bound = self.hold_bounded(self.lock_held_transfer_timeout); + let unlock = match (self.conn.as_mut(), unlock_bound) { + (Some(conn), Some(bound)) => { + bounded_transfer( + "advisory-unlock", + &self.repo_name, + bound, + sqlx::query_as::<_, (bool,)>("SELECT pg_advisory_unlock($1)") + .bind(lock_key) + .fetch_one(&mut **conn), + ) + .await + } + // No connection, or an expired deadline with nothing left to + // fund the wire call: the `None` arms below close the session, + // which frees lock and slot regardless. + _ => None, }; match unlock { Some(Ok((true,))) => { @@ -2553,11 +2660,14 @@ impl RepoWriteGuard { ); } None if self.conn.is_some() => { + // `unlock` is None on a bounded_transfer timeout AND on a + // deadline-expired skip; log the effective bound so a + // zero says which happened. warn!( repo = %self.repo_name, lock_key, - bound_secs = self.lock_held_transfer_timeout.as_secs(), - "advisory unlock exceeded its bound — closing the session so the lock-pool slot is not held longer" + bound_secs = unlock_bound.map(|b| b.as_secs()).unwrap_or(0), + "advisory unlock did not finish inside its bound; closing the session so the lock-pool slot is not held longer" ); } None => {} @@ -2740,6 +2850,29 @@ impl Drop for RepoWriteGuard { /// `infra/fly/fly.toml`; it is not, and reconciling the two is tracked separately. const LOCK_ACQUIRE_DEADLINE: Duration = Duration::from_secs(90); +/// Production `RepoStore::new` sites override this from +/// `Config::write_lock_hold_timeout_secs`; the fallback matches that knob's +/// default so a store built without it keeps the shipped bound. +const LOCK_HOLD_TIMEOUT: Duration = Duration::from_secs(600); + +/// `own` clamped to what `deadline` still funds, or `None` once nothing is +/// left. This is the one body that carries the lock-hold rule every under-lock +/// phase follows: a `tokio::time::timeout` at exactly zero polls the future +/// once before noticing (and a process runner has already spawned by then), so +/// an expired deadline must short-circuit here rather than race inside the +/// bound. Callers keep their own `None` policy: refuse, warn-and-skip, or bail. +pub(crate) fn hold_bounded(deadline: Instant, own: Duration) -> Option { + let bound = own.min(deadline.saturating_duration_since(Instant::now())); + (!bound.is_zero()).then_some(bound) +} + +/// `hold_bounded` for callees that take an `Instant` deadline rather than a +/// `Duration`: the phase's own bound as an instant, never past `deadline`, or +/// `None` once the hold deadline has passed. +pub(crate) fn hold_deadline(deadline: Instant, own: Duration) -> Option { + Instant::now().checked_add(hold_bounded(deadline, own)?) +} + /// Why an under-lock refresh did not complete, split by what it leaves us knowing. /// /// `Unknown` (the existence check failed) and `Download` (the archive is there and @@ -4081,6 +4214,7 @@ pub(crate) mod tests { conn: Some(pool.acquire().await.expect("conn")), tigris: None, lock_held_transfer_timeout: Duration::from_secs(300), + lock_hold_deadline: Instant::now() + Duration::from_secs(600), publish_fence: UploadPrecondition::Unconditional, refresh_swap_authority: None, publish_stage: Arc::new(PublishStageCell::new()), @@ -4451,6 +4585,7 @@ pub(crate) mod tests { conn: Some(lock_pool.acquire().await.unwrap()), tigris: None, lock_held_transfer_timeout: Duration::from_secs(300), + lock_hold_deadline: Instant::now() + Duration::from_secs(600), // No backend, so nothing is ever published and the fence is unread. publish_fence: UploadPrecondition::Unconditional, refresh_swap_authority: None, @@ -9239,4 +9374,122 @@ pub(crate) mod tests { mock.shutdown(); } + + /// #282: every phase under the lock consumes the guard's one lock-hold + /// deadline. A guard whose deadline has already passed must not START a + /// publish the bound cannot finish: the `None` (timed out) arm routes + /// through `resolve_unfinished_publish`, where stage `Idle` reads as a + /// definite non-publication (`UploadFailed`), and no PUT is dispatched. + /// + /// Load-bearing on the clamp: without the `hold_bounded` deadline the + /// publish would run against the mock and the counter below would move. + #[sqlx::test] + async fn release_after_lock_hold_deadline_starts_no_publish(pool: sqlx::PgPool) { + use axum::response::IntoResponse; + + let puts = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let puts_in = Arc::clone(&puts); + let app = axum::Router::new().route( + "/{*key}", + axum::routing::any(move |method: axum::http::Method| { + let puts = Arc::clone(&puts_in); + async move { + if method == axum::http::Method::PUT { + puts.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + } + axum::http::StatusCode::OK.into_response() + } + }), + ); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let endpoint = format!("http://127.0.0.1:{}", listener.local_addr().unwrap().port()); + let server = tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + + let dir = tempfile::TempDir::new().unwrap(); + let guard = RepoWriteGuard { + owner_slug: "did_key_z6MkHoldExpired".to_string(), + repo_name: "expired-hold".to_string(), + local_path: validated_repo_disk_path( + dir.path(), + "did:key:z6MkHoldExpired", + "expired-hold", + ) + .expect("test path"), + lock_key: 995_777, + conn: Some(pool.acquire().await.unwrap()), + tigris: Some(TigrisClient::for_testing_with_endpoint( + "test-bucket", + &endpoint, + )), + lock_held_transfer_timeout: Duration::from_secs(300), + // Already past: the publish's budget must clamp to zero. + lock_hold_deadline: Instant::now() - Duration::from_secs(1), + publish_fence: UploadPrecondition::Unconditional, + refresh_swap_authority: None, + publish_stage: Arc::new(PublishStageCell::new()), + tree_settled: false, + settlement_armed: Arc::new(AtomicBool::new(true)), + path_handed_out: Arc::new(AtomicBool::new(true)), + #[cfg(test)] + test_pre_unlock_gate: None, + #[cfg(test)] + upload_site_reached: Arc::new(std::sync::atomic::AtomicUsize::new(0)), + }; + + let outcome = guard.release(true).await; + assert_eq!( + outcome, + ReleaseOutcome::UploadFailed, + "an expired lock-hold deadline makes the publish a definite non-publication, got {outcome:?}" + ); + assert_eq!( + puts.load(std::sync::atomic::Ordering::SeqCst), + 0, + "no PUT may be dispatched once the lock-hold deadline has passed" + ); + server.abort(); + } + + /// #282: the unlock is also a phase under the deadline. A release entered + /// after the deadline has passed leaves no budget to await the wire, so + /// the session must close rather than return to the pool, freeing lock + /// AND slot even though `pg_advisory_unlock` never ran. + /// + /// The observable is the backend identity on a one-connection pool: a + /// returned connection is handed back as the same session; a closed one + /// forces the next acquire onto a fresh backend. The identity carries + /// `backend_start` alongside the pid because the kernel recycles pids, so + /// an equal pid alone cannot separate same-session from new-backend. + /// Load-bearing on the clamp: without it the unlock succeeds, the + /// connection returns, and the identity repeats. + #[sqlx::test] + async fn release_after_lock_hold_deadline_closes_the_session(pool: sqlx::PgPool) { + let opts = (*pool.connect_options()).clone(); + let lock_pool = no_reap_pool(&opts, 1).await; + let dir = tempfile::TempDir::new().unwrap(); + let store = RepoStore::for_testing(dir.path().to_path_buf(), lock_pool.clone()) + .with_lock_hold_timeout(Duration::from_millis(200)); + + let owner = "did:key:z6MkHoldDeadlineSessionAAAAAAAAAAAAAAAAA"; + let name = "holddeadline"; + let mut guard = store.acquire_write(owner, name).await.expect("acquire"); + let identity_before = guard.backend_identity_for_test().await; + + // Let the deadline pass, then release: the unlock gets no budget. + tokio::time::sleep(Duration::from_millis(300)).await; + let _ = guard.release(false).await; + + let mut next = store + .acquire_write(owner, name) + .await + .expect("the lock must be free once the expired session closes"); + let identity_after = next.backend_identity_for_test().await; + assert_ne!( + identity_before, identity_after, + "an expired-deadline release must close the session, not pool it" + ); + let _ = next.release(false).await; + } } diff --git a/crates/twigpine-node/src/git/store.rs b/crates/twigpine-node/src/git/store.rs index bfedeb72..c296eb0b 100644 --- a/crates/twigpine-node/src/git/store.rs +++ b/crates/twigpine-node/src/git/store.rs @@ -168,15 +168,23 @@ pub fn ensure_marker_hidden_checked(repo_path: &Path) -> Result<()> { pub async fn verify_recovery_prereqs_bounded( git_bin: &str, repo_path: &Path, - timeout: std::time::Duration, + deadline: std::time::Instant, ) -> Result<()> { use tokio::process::Command as AsyncCommand; async fn run_git( git_bin: &str, repo_path: &Path, args: &[&str], - timeout: std::time::Duration, + deadline: std::time::Instant, ) -> Result { + // Recompute per child: several `git config` calls run in sequence + // and one shared deadline must cover them all, not the same + // duration spent once per child. `hold_bounded` returns None on an + // expired deadline so the child is never spawned. + let Some(bound) = super::repo_store::hold_bounded(deadline, std::time::Duration::MAX) + else { + anyhow::bail!("git {} timed out", args.join(" ")); + }; let fut = async { // allow-unbounded-git: bounded by the tokio::time::timeout wrapping // this future in verify_recovery_prereqs_bounded, plus kill_on_drop. @@ -184,7 +192,7 @@ pub async fn verify_recovery_prereqs_bounded( cmd.kill_on_drop(true); cmd.args(args).current_dir(repo_path).output().await }; - tokio::time::timeout(timeout, fut) + tokio::time::timeout(bound, fut) .await .map_err(|_| anyhow::anyhow!("git {} timed out", args.join(" ")))? .map_err(|e| anyhow::anyhow!("git {} spawn failed: {e:#}", args.join(" "))) @@ -193,7 +201,7 @@ pub async fn verify_recovery_prereqs_bounded( git_bin, repo_path, &["config", "--get", "core.logAllRefUpdates"], - timeout, + deadline, ) .await?; let cur = String::from_utf8_lossy(&get.stdout).trim().to_string(); @@ -202,7 +210,7 @@ pub async fn verify_recovery_prereqs_bounded( git_bin, repo_path, &["config", "core.logAllRefUpdates", "always"], - timeout, + deadline, ) .await?; if !out.status.success() { @@ -216,7 +224,7 @@ pub async fn verify_recovery_prereqs_bounded( ("uploadpack.hideRefs", "refs/gitlawb/requests/"), ("transfer.hideRefs", "refs/gitlawb/requests/"), ] { - let current = run_git(git_bin, repo_path, &["config", "--get-all", key], timeout).await?; + let current = run_git(git_bin, repo_path, &["config", "--get-all", key], deadline).await?; let already = String::from_utf8_lossy(¤t.stdout) .lines() .any(|l| l.trim() == value); @@ -225,7 +233,7 @@ pub async fn verify_recovery_prereqs_bounded( git_bin, repo_path, &["config", "--add", key, value], - timeout, + deadline, ) .await?; if !out.status.success() { diff --git a/crates/twigpine-node/src/lib.rs b/crates/twigpine-node/src/lib.rs index 2173aa00..f4054e63 100644 --- a/crates/twigpine-node/src/lib.rs +++ b/crates/twigpine-node/src/lib.rs @@ -415,7 +415,8 @@ pub async fn run() -> Result<()> { tigris, lock_pool, std::time::Duration::from_secs(config.lock_held_transfer_timeout_secs), - ); + ) + .with_lock_hold_timeout(config.write_lock_hold_timeout()); // Per-DID limiter for the creation endpoints. Keyed on the authenticated // DID (attacker-varied), so bound its key set to cap memory. Max and window @@ -1026,7 +1027,8 @@ async fn run_admin_command(command: config::Command, config: &Config) -> Result< object_store, lock_pool, std::time::Duration::from_secs(config.lock_held_transfer_timeout_secs), - ); + ) + .with_lock_hold_timeout(config.write_lock_hold_timeout()); admin::run_purge_spam(&db, &repo_store, &config.repos_dir, execute) .await .map(|_| ()) diff --git a/crates/twigpine-node/src/upstream_mirror.rs b/crates/twigpine-node/src/upstream_mirror.rs index 6d3c8bbd..227ec813 100644 --- a/crates/twigpine-node/src/upstream_mirror.rs +++ b/crates/twigpine-node/src/upstream_mirror.rs @@ -295,7 +295,14 @@ fn run_fetch( repo_path: &Path, upstream: &PinnedUpstream, timeout: Duration, + lock_hold_deadline: Instant, ) -> Result<()> { + // An already-expired hold deadline starts nothing: without this check + // the preflight probes and the fetch child would each spawn once + // before their own zero deadlines noticed. + if Instant::now() >= lock_hold_deadline { + anyhow::bail!("upstream mirror fetch: the lock-hold deadline already expired"); + } let command = fetch_command(upstream); let args: Vec<&str> = command.args.iter().map(String::as_str).collect(); let config: Vec<(&str, &str)> = command @@ -303,17 +310,22 @@ fn run_fetch( .iter() .map(|(key, value)| (key.as_str(), value.as_str())) .collect(); - let rewrite_deadline = Instant::now() - .checked_add(PREFLIGHT_PROBE_TIMEOUT) - .context("mirror URL-rewrite probe timeout is not representable")?; + // Every phase under the write lock consumes the guard's lock-hold + // deadline: each keeps its own bound, recomputed per phase through + // `hold_deadline` so an expiry during one probe means the next child + // never spawns instead of starting on a deadline already past (#282). + let expired = + || anyhow::anyhow!("upstream mirror fetch: the lock-hold deadline already expired"); + let rewrite_deadline = + crate::git::repo_store::hold_deadline(lock_hold_deadline, PREFLIGHT_PROBE_TIMEOUT) + .ok_or_else(expired)?; reject_local_url_rewrite(git_bin, repo_path, &upstream.url, rewrite_deadline)?; - let config_deadline = Instant::now() - .checked_add(PREFLIGHT_PROBE_TIMEOUT) - .context("mirror local-config probe timeout is not representable")?; + let config_deadline = + crate::git::repo_store::hold_deadline(lock_hold_deadline, PREFLIGHT_PROBE_TIMEOUT) + .ok_or_else(expired)?; reject_sensitive_local_config(git_bin, repo_path, config_deadline)?; - let deadline = Instant::now() - .checked_add(timeout) - .context("upstream mirror fetch timeout is not representable")?; + let deadline = + crate::git::repo_store::hold_deadline(lock_hold_deadline, timeout).ok_or_else(expired)?; let (status, _stdout, stderr) = crate::git::visibility_pack::run_bounded_git_raw_isolated_https_with_config( git_bin, &args, repo_path, b"", deadline, &config, @@ -407,9 +419,21 @@ impl Worker { } let git_bin = self.git_bin.clone(); + // The fetch holds the write lock the whole time, so it consumes the + // guard's lock-hold deadline rather than a fresh budget: a fetch + // running past that deadline pins its lock-pool slot exactly like a + // slow push (#282). `run_fetch` clamps each phase's own bound + // against this deadline internally. let fetch_timeout = Duration::from_secs(self.config.upstream_mirror_fetch_timeout_secs); + let lock_hold_deadline = guard.lock_hold_deadline(); let fetch = tokio::task::spawn_blocking(move || { - run_fetch(&git_bin, &repo_path, &upstream, fetch_timeout) + run_fetch( + &git_bin, + &repo_path, + &upstream, + fetch_timeout, + lock_hold_deadline, + ) }) .await .context("upstream mirror fetch task panicked")?; @@ -865,7 +889,14 @@ esac url: format!("https://127.0.0.1:{port}/org/repo=mirror.git"), curlopt_resolve: None, }; - let error = run_fetch("git", repo.path(), &upstream, Duration::from_secs(5)).unwrap_err(); + let error = run_fetch( + "git", + repo.path(), + &upstream, + Duration::from_secs(5), + Instant::now() + Duration::from_secs(600), + ) + .unwrap_err(); assert!( accepted.join().unwrap(), "real Git must reach the HTTPS transport; got {error:#}" @@ -965,4 +996,81 @@ esac assert_eq!(calls.lines().collect::>(), ["fail", "ok"]); assert!(worker.repo_write_leases.is_empty()); } + + /// #282: a mirror fetch holds the write lock for its whole run, so it + /// consumes the guard's lock-hold deadline, not a fresh + /// `upstream_mirror_fetch_timeout_secs` budget. With the hold bound below + /// the fetch timeout, a sleeping fetch must die at the hold bound, not at + /// the fetch timeout. Load-bearing on the clamp: without the + /// `lock_hold_deadline` phase clamps the fetch runs to the 5s timeout and + /// the elapsed assertion below fails. + #[cfg(unix)] + #[sqlx::test] + async fn fetch_respects_the_guard_lock_hold_deadline(pool: sqlx::PgPool) { + let bootstrap = tempfile::tempdir().unwrap(); + let git = bootstrap.path().join("fake-git"); + let fetch_marker = bootstrap.path().join("fetch-started"); + // Answer the preflight probes like `fake_git` does (a bare repo with + // no rewrites and no sensitive config), then mark and sleep on the + // fetch so the test can tell "the fetch child ran" from "the run + // bailed before spawning anything". + std::fs::write( + &git, + format!( + "#!/bin/sh\n\ + if [ \"$1\" = \"ls-remote\" ]; then printf '%s\\n' \"$3\"; exit 0; fi\n\ + if [ \"$1\" = \"config\" ]; then exit 1; fi\n\ + : > \"{}\"\n\ + sleep 30\n", + fetch_marker.display() + ), + ) + .unwrap(); + { + use std::os::unix::fs::PermissionsExt; + let mut permissions = std::fs::metadata(&git).unwrap().permissions(); + permissions.set_mode(0o700); + std::fs::set_permissions(&git, permissions).unwrap(); + } + + // Fetch timeout is 5s from `test_worker`; the hold bound is tighter. + let (mut worker, db, _repos) = test_worker(pool, git, 1).await; + worker.repo_store = worker + .repo_store + .with_lock_hold_timeout(Duration::from_millis(800)); + let target = add_inbound_mirror( + &worker, + &db, + "00000000-0000-0000-0000-000000000003", + "slowfetch", + "https://8.8.8.8/org/slow.git", + ) + .await; + + let started = Instant::now(); + let result = worker.fetch_target(target).await; + let elapsed = started.elapsed(); + assert!( + result.is_err(), + "a fetch cut by the lock-hold deadline must fail, got {result:?}" + ); + // The preflight shims answer instantly, so a fast failure would + // return in milliseconds. The shim's marker plus reaching ~800ms + // prove the fetch child really started and the deadline is what cut + // it, not an expired-deadline bail before the spawn. + assert!( + fetch_marker.exists(), + "the fetch child must have started for the deadline to have cut it" + ); + assert!( + elapsed >= Duration::from_millis(700), + "a fetch that never started would fail fast, took only {elapsed:?}" + ); + assert!( + elapsed < Duration::from_secs(4), + "a 800ms lock-hold bound must end the fetch well inside the 5s \ + fetch timeout, took {elapsed:?}" + ); + assert!(worker.repo_write_leases.is_empty()); + } } diff --git a/crates/twigpine-node/tests/inv22_gates.rs b/crates/twigpine-node/tests/inv22_gates.rs index 7a2f4694..59651cfd 100644 --- a/crates/twigpine-node/tests/inv22_gates.rs +++ b/crates/twigpine-node/tests/inv22_gates.rs @@ -946,21 +946,23 @@ fn issue_26_namespace_gate_and_refusal_wiring() { ); // P1 (cont): every explicit pre-git refusal must call refuse_receive_pack_before_git - // to terminalize the intent. Exactly 6 occurrences: the five inline call + // to terminalize the intent. Exactly 8 occurrences: the seven inline call // sites (lease cap, per-source cap, write pool, acquire timeout, acquire - // error) plus the drop-guard's own pre-git spawn. The post-git guard phase - // calls mark_receive_pack_interrupted (a different name), so deleting any - // one inline site drops the count to 5 and turns this red. The parse and - // namespace refusals correctly carry no call: they return before the - // intent insert, so there is no aggregate to terminalize. + // error, and the two expired lock-hold deadline sheds, one before the + // tree is handed out and one at the receive-pack re-clamp) plus the + // drop-guard's own pre-git spawn. The post-git guard phase calls mark_receive_pack_interrupted (a + // different name), so deleting any one inline site drops the count to 7 + // and turns this red. The parse and namespace refusals correctly carry no + // call: they return before the intent insert, so there is no aggregate to + // terminalize. assert_eq!( repos.matches("refuse_receive_pack_before_git(").count(), - 6, - "P1 wiring: exactly 6 refuse_receive_pack_before_git calls expected (5 \ + 8, + "P1 wiring: exactly 8 refuse_receive_pack_before_git calls expected (7 \ inline refusal sites + guard pre-git spawn); deleting any inline site \ must turn this red" ); - // Pin the five inline sites by their reason literals, so a site that stops + // Pin the seven inline sites by their reason literals, so a site that stops // terminalizing (or is renumbered away) fails by name, not just by count. for reason in [ "repo write-lease waiter cap reached", @@ -968,6 +970,7 @@ fn issue_26_namespace_gate_and_refusal_wiring() { "git write pool saturated", "acquire_write timed out", "acquire_write failed", + "write-lock hold deadline expired before git", "handler dropped before git started", ] { assert!( From f9b73d6c038e667ce8b88ff3975033d17b821a18 Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Wed, 18 Nov 2026 21:31:03 -0600 Subject: [PATCH 2/4] fix(node): release the write guard on hold-deadline sheds and bound the mirror recheck A deadline shed that landed after the tree was handed out dropped the settlement at stage Idle and quarantined a tree receive-pack never ran on. Both receive-pack sheds now release the guard before the refusal's DB call, and the later one disarms the settlement. The upstream mirror's stale-target recheck now consumes the guard's lock-hold deadline like every other phase under the lock. Tests: a shed after path hand-out leaves no quarantine marker and frees the lock; the mirror recheck refuses on an expired deadline without spawning git; the expired-release publish test seeds a repo and gains a live-deadline control; the teardown fixture hangs only on receive-pack. --- crates/twigpine-node/src/api/repos.rs | 153 +++++++++++++++++++- crates/twigpine-node/src/git/repo_store.rs | 90 +++++++++--- crates/twigpine-node/src/upstream_mirror.rs | 92 +++++++++++- 3 files changed, 311 insertions(+), 24 deletions(-) diff --git a/crates/twigpine-node/src/api/repos.rs b/crates/twigpine-node/src/api/repos.rs index 7fe48278..c88305c1 100644 --- a/crates/twigpine-node/src/api/repos.rs +++ b/crates/twigpine-node/src/api/repos.rs @@ -3023,6 +3023,10 @@ pub async fn git_receive_pack( let Some(git_timeout) = crate::git::repo_store::hold_bounded(lock_hold_deadline, git_service_timeout) else { + // Free the lock and its pool slot before the refusal's DB round trip: + // the refusal does not need the lock, and every other pre-git refusal + // runs with none held. + let _ = guard.release(false).await; if let Err(e) = state .db .refuse_receive_pack_before_git( @@ -3188,6 +3192,22 @@ pub async fn git_receive_pack( // instead of spawning a receive-pack that is killed on the first poll. let Some(run_bound) = crate::git::repo_store::hold_bounded(lock_hold_deadline, git_timeout) else { + // No git child exists yet, so the admission bundle has nothing to + // reap: drop it, reclaim the guard and release it before the refusal's + // DB round trip, so the lock and its pool slot do not outlive the + // deadline. `release(false)` classifies the tree itself (receive-pack + // never ran on it), so this future's settlement copy is disarmed + // rather than left to quarantine an untouched tree on drop. + drop(admission); + let reclaimed = guard + .lock() + .expect("repo write-lock mutex poisoned") + .take() + .expect("the write lock is only taken here, and only once"); + let _ = reclaimed.release(false).await; + if let Some(settlement) = settlement.as_mut() { + settlement.disarm(); + } if let Err(e) = state .db .refuse_receive_pack_before_git( @@ -19555,8 +19575,11 @@ exit 0 let descfile = tmp.path().join("desc.pid"); let body = format!( "#!/bin/sh\n\ + case \"$1\" in receive-pack)\n\ sh -c 'trap \"\" TERM; echo $$ > \"{}\"; i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done' &\n\ - wait\n", + wait; exit 0 ;;\n\ + esac\n\ + exec git \"$@\"\n", descfile.display() ); let git_bin = write_fake_git(tmp.path(), &body); @@ -19714,6 +19737,134 @@ exit 0 tokio::time::sleep(std::time::Duration::from_millis(25)).await; } } + + /// #282, the shed that lands after the tree was handed out. When the + /// pre-git phases (recovery prereqs, marker) spend the whole lock-hold + /// deadline, the handler sheds just before receive-pack would start. By + /// then `guard.path()` has declared the tree handed out and the settlement + /// token sits in this future, so a bare return would drop it at stage + /// `Idle` and quarantine a tree git never ran on, and would hold the lock + /// across the refusal's DB call. The shed must release the guard and + /// disarm the settlement first. + /// + /// A real (mock) object store is configured so the stage starts `Idle`; + /// with no backend it starts `NoBackend` and settlement never quarantines. + /// + /// MUTATION (RED): drop the teardown (release and disarm) from the second + /// shed and the quarantine-marker assertion fails. + #[cfg(unix)] + #[sqlx::test] + async fn receive_pack_deadline_shed_after_path_handout_releases_and_does_not_quarantine( + pool: sqlx::PgPool, + ) { + use axum::extract::{Path, State}; + use axum::Extension; + use std::net::SocketAddr; + + let owner = "z6shedtwo"; + let name = "shed2"; + let mock = crate::git::repo_store::tests::S3Mock::start().await; + let repos_dir = tempfile::TempDir::new().unwrap(); + let tmp = tempfile::TempDir::new().unwrap(); + let sentinel = tmp.path().join("receive-pack-ran"); + // Every pre-git child (prereq `config`, marker `hash-object` and + // `update-ref`) hangs past the hold deadline and is cut by it; only + // `receive-pack` would leave the sentinel behind. + let git_bin = write_fake_git( + tmp.path(), + &format!( + "#!/bin/sh\ncase \"$1\" in receive-pack) : > \"{}\"; exit 0 ;; esac\nexec sleep 30\n", + sentinel.display() + ), + ); + + let mut state = crate::test_support::test_state(pool.clone()).await; + state.git_bin = git_bin; + state.repo_store = crate::git::repo_store::RepoStore::for_testing_with_tigris( + repos_dir.path().to_path_buf(), + pool.clone(), + crate::git::repo_store::tests::mock_tigris(&mock), + ) + .with_lock_hold_timeout(std::time::Duration::from_secs(2)); + let mut cfg = (*state.config).clone(); + cfg.git_service_timeout_secs = 600; + cfg.enforce_owner_push = false; + state.config = std::sync::Arc::new(cfg); + state + .db + .upsert_mirror_repo(owner, name, "/tmp/z6shedtwo-shed2", None, false) + .await + .unwrap(); + let rec = state.db.get_repo(owner, name).await.unwrap().unwrap(); + // The tree must exist so the pre-git children actually spawn in it. + let bare = crate::git::repo_store::validated_repo_disk_path( + repos_dir.path(), + &rec.owner_did, + name, + ) + .expect("test repo path"); + crate::git::store::init_bare(&bare).expect("a bare repo on disk"); + let quarantine = p3_quarantine_marker(repos_dir.path(), &rec.owner_did, name); + + use crate::git::repo_store::advisory_lock_key; + let key = advisory_lock_key(&owner.replace([':', '/'], "_"), name); + let probe = sqlx::postgres::PgPoolOptions::new() + .max_connections(2) + .connect_lazy_with((*pool.connect_options()).clone()); + + let result = git_receive_pack( + State(state.clone()), + Path((owner.to_string(), name.to_string())), + Extension(crate::auth::AuthenticatedDid( + "did:key:z6shedtwo".to_string(), + )), + None, + crate::rate_limit::PeerAddr(Some("203.0.113.83:5000".parse::().unwrap())), + axum::http::HeaderMap::new(), + axum::body::Bytes::from_static(b"0000"), + ) + .await; + + let status = match result { + Ok(r) => panic!("a deadline-shed push must fail, got {}", r.status()), + Err(e) => axum::response::IntoResponse::into_response(e).status(), + }; + assert_eq!( + status, + axum::http::StatusCode::GATEWAY_TIMEOUT, + "the pre-git deadline shed answers as a git timeout" + ); + assert!( + !sentinel.exists(), + "receive-pack must never start once the hold deadline is spent" + ); + assert!( + !quarantine.exists(), + "a shed before receive-pack must not quarantine a tree git never ran on" + ); + let free_deadline = std::time::Instant::now() + std::time::Duration::from_secs(2); + while !write_lock_is_takeable(&probe, key).await { + assert!( + std::time::Instant::now() < free_deadline, + "the write lock must be free once the shed returns" + ); + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + } + let rows: Vec<(String, Option)> = + sqlx::query_as("SELECT state, last_error FROM receive_pack_requests") + .fetch_all(&pool) + .await + .unwrap(); + assert_eq!( + rows, + vec![( + crate::db::request_state::REJECTED_AT_GIT.to_string(), + Some("write-lock hold deadline expired before git".to_string()), + )], + "the shed must terminalize its request as a pre-git refusal" + ); + mock.shutdown(); + } } #[cfg(test)] diff --git a/crates/twigpine-node/src/git/repo_store.rs b/crates/twigpine-node/src/git/repo_store.rs index 29b43d31..02d80d4e 100644 --- a/crates/twigpine-node/src/git/repo_store.rs +++ b/crates/twigpine-node/src/git/repo_store.rs @@ -9375,16 +9375,19 @@ pub(crate) mod tests { mock.shutdown(); } - /// #282: every phase under the lock consumes the guard's one lock-hold - /// deadline. A guard whose deadline has already passed must not START a - /// publish the bound cannot finish: the `None` (timed out) arm routes - /// through `resolve_unfinished_publish`, where stage `Idle` reads as a - /// definite non-publication (`UploadFailed`), and no PUT is dispatched. + /// Release a hand-built guard over a SEEDED bare repo against a mock store + /// that answers 200 to everything, with the given lock-hold deadline. + /// Returns the outcome and how many PUTs reached the mock. /// - /// Load-bearing on the clamp: without the `hold_bounded` deadline the - /// publish would run against the mock and the counter below would move. - #[sqlx::test] - async fn release_after_lock_hold_deadline_starts_no_publish(pool: sqlx::PgPool) { + /// The repo is seeded so archive preparation succeeds: without it a + /// publish that wrongly started would still fail before any PUT and read + /// as `UploadFailed`, which is the outcome the expired case asserts. + async fn release_with_lock_hold_deadline( + pool: &sqlx::PgPool, + repo_name: &str, + lock_key: i64, + lock_hold_deadline: Instant, + ) -> (ReleaseOutcome, usize) { use axum::response::IntoResponse; let puts = Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -9408,24 +9411,21 @@ pub(crate) mod tests { }); let dir = tempfile::TempDir::new().unwrap(); + let local_path = validated_repo_disk_path(dir.path(), "did:key:z6MkHoldExpired", repo_name) + .expect("test path"); + seed_bare_repo(&local_path); let guard = RepoWriteGuard { owner_slug: "did_key_z6MkHoldExpired".to_string(), - repo_name: "expired-hold".to_string(), - local_path: validated_repo_disk_path( - dir.path(), - "did:key:z6MkHoldExpired", - "expired-hold", - ) - .expect("test path"), - lock_key: 995_777, + repo_name: repo_name.to_string(), + local_path, + lock_key, conn: Some(pool.acquire().await.unwrap()), tigris: Some(TigrisClient::for_testing_with_endpoint( "test-bucket", &endpoint, )), lock_held_transfer_timeout: Duration::from_secs(300), - // Already past: the publish's budget must clamp to zero. - lock_hold_deadline: Instant::now() - Duration::from_secs(1), + lock_hold_deadline, publish_fence: UploadPrecondition::Unconditional, refresh_swap_authority: None, publish_stage: Arc::new(PublishStageCell::new()), @@ -9439,17 +9439,63 @@ pub(crate) mod tests { }; let outcome = guard.release(true).await; + let put_count = puts.load(std::sync::atomic::Ordering::SeqCst); + server.abort(); + (outcome, put_count) + } + + /// #282: every phase under the lock consumes the guard's one lock-hold + /// deadline. A guard whose deadline has already passed must not START a + /// publish the bound cannot finish: the `None` (timed out) arm routes + /// through `resolve_unfinished_publish`, where stage `Idle` reads as a + /// definite non-publication (`UploadFailed`), and no PUT is dispatched. + /// + /// Load-bearing on the clamp: the repo is seeded and the mock accepts + /// every PUT, so without the `hold_bounded` deadline the publish would + /// succeed (`Released`) and the counter would move. The sibling test + /// below proves that same fixture does publish when the deadline is live. + #[sqlx::test] + async fn release_after_lock_hold_deadline_starts_no_publish(pool: sqlx::PgPool) { + // Already past: the publish's budget must clamp to zero. + let (outcome, puts) = release_with_lock_hold_deadline( + &pool, + "expired-hold", + 995_777, + Instant::now() - Duration::from_secs(1), + ) + .await; assert_eq!( outcome, ReleaseOutcome::UploadFailed, "an expired lock-hold deadline makes the publish a definite non-publication, got {outcome:?}" ); assert_eq!( - puts.load(std::sync::atomic::Ordering::SeqCst), - 0, + puts, 0, "no PUT may be dispatched once the lock-hold deadline has passed" ); - server.abort(); + } + + /// Control for the test above: the same fixture with a live lock-hold + /// deadline publishes. Without it, a fixture that could never publish + /// would make the expired case pass for the wrong reason. + #[sqlx::test] + async fn release_before_lock_hold_deadline_publishes(pool: sqlx::PgPool) { + let (outcome, puts) = release_with_lock_hold_deadline( + &pool, + "live-hold", + 995_778, + Instant::now() + Duration::from_secs(300), + ) + .await; + assert_eq!( + outcome, + ReleaseOutcome::Released, + "a live lock-hold deadline lets the publish run to completion, got {outcome:?}" + ); + assert!( + puts > 0, + "the publish must reach the store while the lock-hold deadline is live" + ); } /// #282: the unlock is also a phase under the deadline. A release entered diff --git a/crates/twigpine-node/src/upstream_mirror.rs b/crates/twigpine-node/src/upstream_mirror.rs index 227ec813..c5c1b42e 100644 --- a/crates/twigpine-node/src/upstream_mirror.rs +++ b/crates/twigpine-node/src/upstream_mirror.rs @@ -290,6 +290,12 @@ fn reject_sensitive_local_config(git_bin: &str, repo_path: &Path, deadline: Inst /// only the external pack transfer it documents. const PREFLIGHT_PROBE_TIMEOUT: Duration = Duration::from_secs(30); +/// Own bound for the under-lock stale-target recheck in +/// [`Worker::fetch_target`]: one indexed row read, so a short bound is +/// generous. It is clamped to the guard's lock-hold deadline like every +/// other phase under the write lock (#282). +const STALE_RECHECK_TIMEOUT: Duration = Duration::from_secs(30); + fn run_fetch( git_bin: &str, repo_path: &Path, @@ -392,7 +398,31 @@ impl Worker { // are held. Every future transition path must take these same locks // before changing authority, so no other node can commit a transition // between this check and the fetch. - let current = match self.db.get_repo_mirror_state(&target.repo_id).await { + // + // The recheck runs under the write lock, so it consumes the guard's + // lock-hold deadline like every later phase: an expired deadline + // starts no query, and a slow read cannot pin the lock and its + // lock-pool session past the deadline (#282). + let Some(recheck_bound) = + crate::git::repo_store::hold_bounded(guard.lock_hold_deadline(), STALE_RECHECK_TIMEOUT) + else { + let _ = guard.release(false).await; + anyhow::bail!( + "upstream mirror fetch: the lock-hold deadline expired before the stale-target recheck" + ); + }; + let recheck = tokio::time::timeout( + recheck_bound, + self.db.get_repo_mirror_state(&target.repo_id), + ) + .await; + let Ok(recheck) = recheck else { + let _ = guard.release(false).await; + anyhow::bail!( + "upstream mirror fetch: the stale-target recheck did not finish inside the lock-hold budget" + ); + }; + let current = match recheck { Ok(Some(current)) => current, Ok(None) => { let _ = guard.release(false).await; @@ -1073,4 +1103,64 @@ esac ); assert!(worker.repo_write_leases.is_empty()); } + + /// #282: the stale-target recheck runs under the write lock, so it + /// consumes the guard's lock-hold deadline too. With a zero hold bound + /// the deadline has passed by the time the recheck would run: the + /// recheck must refuse, release the lock, and spawn no git child. + /// Load-bearing on the recheck's `hold_bounded` check: without it the + /// recheck runs and `run_fetch` bails with its own expired-deadline + /// message instead, so the message assertion below fails. + #[cfg(unix)] + #[sqlx::test] + async fn stale_recheck_respects_the_guard_lock_hold_deadline(pool: sqlx::PgPool) { + let bootstrap = tempfile::tempdir().unwrap(); + let git = bootstrap.path().join("fake-git"); + let git_marker = bootstrap.path().join("git-invoked"); + std::fs::write( + &git, + format!("#!/bin/sh\n: > \"{}\"\nexit 1\n", git_marker.display()), + ) + .unwrap(); + { + use std::os::unix::fs::PermissionsExt; + let mut permissions = std::fs::metadata(&git).unwrap().permissions(); + permissions.set_mode(0o700); + std::fs::set_permissions(&git, permissions).unwrap(); + } + + let (mut worker, db, _repos) = test_worker(pool, git, 1).await; + worker.repo_store = worker.repo_store.with_lock_hold_timeout(Duration::ZERO); + let target = add_inbound_mirror( + &worker, + &db, + "00000000-0000-0000-0000-000000000004", + "expiredrecheck", + "https://8.8.8.8/org/expired.git", + ) + .await; + let (owner_did, name) = (target.owner_did.clone(), target.name.clone()); + + let error = worker.fetch_target(target).await.unwrap_err(); + assert_eq!( + error.to_string(), + "upstream mirror fetch: the lock-hold deadline expired before the stale-target recheck" + ); + assert!( + !git_marker.exists(), + "no git child may run once the lock-hold deadline has expired" + ); + assert!(worker.repo_write_leases.is_empty()); + + // The refusal released the advisory write lock: a fresh acquire on + // the same store takes it at once instead of waiting it out. + let reacquired = tokio::time::timeout( + Duration::from_secs(5), + worker.repo_store.acquire_write(&owner_did, &name), + ) + .await + .expect("the write lock must be free after the recheck refused") + .expect("acquire after the refused recheck"); + let _ = reacquired.release(false).await; + } } From 373f20fc19dd74f899c588a5bc618dfdf0857e09 Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Wed, 18 Nov 2026 22:28:45 -0600 Subject: [PATCH 3/4] fix(node): take merge and issue write guards without a lock-hold deadline Merge and issue create/close run synchronous git under their guards and never consult the hold deadline, but release still clamped their publish by it. A slow merge that finished past the deadline had its publish refused and its completed write rolled back, on every retry. These writers now acquire through acquire_write_without_hold_deadline, so each phase keeps its own bound as before; push and the upstream mirror keep the deadline. --- .env.example | 14 ++-- README.md | 2 +- crates/twigpine-node/src/api/issues.rs | 54 +++++++++++- crates/twigpine-node/src/api/pulls.rs | 82 +++++++++++++++++- crates/twigpine-node/src/api/repos.rs | 4 +- crates/twigpine-node/src/git/repo_store.rs | 92 +++++++++++++++++++-- crates/twigpine-node/src/upstream_mirror.rs | 13 ++- 7 files changed, 237 insertions(+), 24 deletions(-) diff --git a/.env.example b/.env.example index 1d4b9d22..55124ee7 100644 --- a/.env.example +++ b/.env.example @@ -101,12 +101,14 @@ TWIGPINE_LOCK_HELD_TRANSFER_TIMEOUT_SECS=300 # ~32 concurrent slow pushes deny every write on the node. Does NOT bound the # acquire-side lock wait (TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS), post-git outcome # bookkeeping, the ~4s the detached reaper holds the session while the git -# process group dies, or a caller's synchronous under-lock work that never -# consults the deadline (merge/issue writes still run unbounded git under -# their guards). Unset, the effective bound is TWIGPINE_GIT_SERVICE_TIMEOUT_SECS -# itself (600 by default), so it tracks the git budget up or down. Set it only -# to tighten the hold below the git budget; a value above -# TWIGPINE_GIT_SERVICE_TIMEOUT_SECS is rejected at boot. +# process group dies, or merge and issue writes, which take their guards +# without this deadline: their git runs unbounded under the lock, and their +# refresh, publish and unlock keep their own bounds, so a slow merge is never +# rolled back for overrunning it. Unset, the effective bound is +# TWIGPINE_GIT_SERVICE_TIMEOUT_SECS itself (600 by default), so it tracks the +# git budget up or down. Set it only to tighten the hold below the git +# budget; a value above TWIGPINE_GIT_SERVICE_TIMEOUT_SECS is rejected at +# boot. #TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS=300 # Seconds a request waits for a pool connection before failing with 503. TWIGPINE_DB_ACQUIRE_TIMEOUT_SECS=5 diff --git a/README.md b/README.md index d8cdf2cb..d7bc29cb 100644 --- a/README.md +++ b/README.md @@ -433,7 +433,7 @@ Important node settings: | `TWIGPINE_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. | | `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. | | `TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. | -| `TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS` | Max seconds a repo write guard may hold its advisory-lock pool connection, measured from lock-take. One deadline covers every phase under the lock: the under-lock refresh, the caller's under-lock work (for a push, the whole `receive-pack` subprocess, clamped to the lesser of `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), and the release-side upload/unlock. It bounds the lock-pool slot's hold, so a slow push cannot starve other writers for the sum of every per-phase bound. It does not bound the acquire-side lock wait (`TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS`), post-git outcome bookkeeping, the few seconds the detached reaper holds the session while the git process group dies, or a caller's synchronous under-lock work that never consults the deadline (merge/issue writes still run unbounded git under their guards). Unset, the effective bound is `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` itself (600 by default), tracking the git budget; set it only to tighten the hold below the git budget. An explicit value above `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` is rejected at boot. | +| `TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS` | Max seconds a repo write guard may hold its advisory-lock pool connection, measured from lock-take. One deadline covers every phase under the lock: the under-lock refresh, the caller's under-lock work (for a push, the whole `receive-pack` subprocess, clamped to the lesser of `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), and the release-side upload/unlock. It bounds the lock-pool slot's hold, so a slow push cannot starve other writers for the sum of every per-phase bound. It does not bound the acquire-side lock wait (`TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS`), post-git outcome bookkeeping, the few seconds the detached reaper holds the session while the git process group dies, or merge and issue writes, which take their guards without this deadline: their git runs unbounded under the lock, and their refresh, publish and unlock keep their own bounds, so a slow merge is never rolled back for overrunning it. Unset, the effective bound is `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` itself (600 by default), tracking the git budget; set it only to tighten the hold below the git budget. An explicit value above `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` is rejected at boot. | | `TWIGPINE_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `TWIGPINE_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. | | `TWIGPINE_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `TWIGPINE_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `TWIGPINE_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | diff --git a/crates/twigpine-node/src/api/issues.rs b/crates/twigpine-node/src/api/issues.rs index b7b5b4a0..1c645b45 100644 --- a/crates/twigpine-node/src/api/issues.rs +++ b/crates/twigpine-node/src/api/issues.rs @@ -63,7 +63,7 @@ pub async fn create_issue( let guard = state .repo_store - .acquire_write(&record.owner_did, &record.name) + .acquire_write_without_hold_deadline(&record.owner_did, &record.name) .await .map_err(|e| crate::api::repos::acquire_write_app_error(&e, &repo))?; @@ -425,7 +425,7 @@ pub async fn close_issue( // 503. Calling .to_string() first destroys that and reports both as a 500. let guard = state .repo_store - .acquire_write(&record.owner_did, &record.name) + .acquire_write_without_hold_deadline(&record.owner_did, &record.name) .await .map_err(|e| crate::api::repos::acquire_write_app_error(&e, &repo))?; let disk_path = guard.path().to_path_buf(); @@ -1605,6 +1605,56 @@ mod lock_pool_shed_tests { server.abort(); } + /// #282: issue writes never consult the hold deadline, so create takes its + /// guard without one (`acquire_write_without_hold_deadline`). Under a + /// zero-length operator timeout the create still publishes and keeps its + /// ref; a stamped deadline would only roll the finished write back at + /// release and 5xx the caller. + #[sqlx::test] + async fn create_issue_without_hold_deadline_publishes(pool: PgPool) { + use crate::git::repo_store::tests as rs_tests; + + let mock = rs_tests::S3Mock::start().await; + let owner = "did:key:zISSUENOHOLDAAAAAAAAAAAAAAAAAAAAAAAA"; + let repo_name = "nohold-create"; + let mut state = crate::test_support::test_state(pool.clone()).await; + let disk = tempfile::TempDir::new().unwrap(); + let mut seeded = seed_repo(owner, repo_name); + let live = crate::git::repo_store::validated_repo_disk_path(disk.path(), owner, repo_name) + .expect("repo path"); + seeded.disk_path = live.to_string_lossy().to_string(); + state.db.create_repo(&seeded).await.expect("seed repo"); + crate::git::store::init_bare(&live).expect("init bare repo"); + state.repo_store = crate::git::repo_store::RepoStore::for_testing_with_tigris( + disk.path().to_path_buf(), + crate::git::repo_store::build_lock_pool(&pool, 2, std::time::Duration::from_secs(1)), + rs_tests::mock_tigris(&mock), + ) + .with_lock_hold_timeout(std::time::Duration::ZERO); + + let res = create_issue( + State(state.clone()), + Extension(AuthenticatedDid(owner.to_string())), + Path((owner.to_string(), repo_name.to_string())), + Json(CreateIssueRequest { + title: "t".to_string(), + body: None, + signed_payload: None, + }), + ) + .await; + let (status, Json(issue)) = res.expect("a create with no hold deadline must publish"); + assert_eq!(status, StatusCode::CREATED); + assert!( + git_issues::get_issue(&live, &issue.id) + .expect("get_issue") + .is_some(), + "the issue ref must survive on disk, not be rolled back" + ); + + mock.shutdown(); + } + /// The compensating rewrite after a refused publish must land on the ref the /// close actually mutated. A close addressed by an 8-char prefix resolves to /// the full id and closes `refs/gitlawb/issues/{full_id}`; a compensator diff --git a/crates/twigpine-node/src/api/pulls.rs b/crates/twigpine-node/src/api/pulls.rs index b2f0e03d..4f6e5d37 100644 --- a/crates/twigpine-node/src/api/pulls.rs +++ b/crates/twigpine-node/src/api/pulls.rs @@ -221,7 +221,7 @@ pub async fn merge_pr( let guard = state .repo_store - .acquire_write(&record.owner_did, &record.name) + .acquire_write_without_hold_deadline(&record.owner_did, &record.name) .await .map_err(|e| crate::api::repos::acquire_write_app_error(&e, &name))?; let disk_path = guard.path().to_path_buf(); @@ -878,6 +878,86 @@ mod merge_rollback_tests { "a refused publish must rewind the local merge to the pre-merge tip" ); } + + /// #282: merge never consults the hold deadline, so it takes its guard + /// without one (`acquire_write_without_hold_deadline`). A merge that lands + /// under a zero-length operator timeout still publishes; a stamped + /// deadline would rewind the completed merge and 5xx the caller. + #[sqlx::test] + async fn merge_pr_without_hold_deadline_publishes(pool: PgPool) { + use crate::git::repo_store::tests as rs_tests; + + let mock = rs_tests::S3Mock::start().await; + let owner = "did:key:zMERGENOHOLDAAAAAAAAAAAAAAAAAAAAAAAA"; + let repo_name = "nohold-merge"; + let mut state = crate::test_support::test_state(pool.clone()).await; + let disk = tempfile::TempDir::new().unwrap(); + state.repo_store = crate::git::repo_store::RepoStore::for_testing_with_tigris( + disk.path().to_path_buf(), + crate::git::repo_store::build_lock_pool(&pool, 2, std::time::Duration::from_secs(1)), + rs_tests::mock_tigris(&mock), + ) + .with_lock_hold_timeout(std::time::Duration::ZERO); + let live = crate::git::repo_store::validated_repo_disk_path(disk.path(), owner, repo_name) + .expect("repo path"); + let (main_tip, _feature_tip) = seeded_merge_repo(&live); + + let mut seeded = crate::db::RepoRecord { + id: Uuid::new_v4().to_string(), + name: repo_name.to_string(), + owner_did: owner.to_string(), + description: None, + is_public: true, + default_branch: "main".to_string(), + created_at: Utc::now(), + updated_at: Utc::now(), + disk_path: String::new(), + forked_from: None, + machine_id: None, + }; + seeded.disk_path = live.to_string_lossy().to_string(); + let repo_id = seeded.id.clone(); + state.db.create_repo(&seeded).await.expect("seed repo"); + let now = Utc::now().to_rfc3339(); + state + .db + .create_pr(&PullRequest { + id: Uuid::new_v4().to_string(), + repo_id: repo_id.clone(), + number: 1, + title: "nohold".to_string(), + body: None, + author_did: owner.to_string(), + source_branch: "feature".to_string(), + target_branch: "main".to_string(), + status: "open".to_string(), + merged_by_did: None, + merged_at: None, + head_commit: None, + created_at: now.clone(), + updated_at: now, + }) + .await + .expect("seed PR"); + + let res = merge_pr( + State(state.clone()), + Extension(AuthenticatedDid(owner.to_string())), + Path((owner.to_string(), repo_name.to_string(), 1)), + ) + .await; + let _ = res.expect("a merge with no hold deadline must publish"); + + let merged = state.db.get_pr(&repo_id, 1).await.unwrap().unwrap(); + assert_eq!(merged.status, "merged"); + let tip_after = git(&live, &["rev-parse", "refs/heads/main"]); + assert_ne!( + tip_after, main_tip, + "the merge commit must stay on main, not be rewound" + ); + + mock.shutdown(); + } } /// #173 F1 follow-up: merge holds no admission permit, so an exhausted write-lock diff --git a/crates/twigpine-node/src/api/repos.rs b/crates/twigpine-node/src/api/repos.rs index c88305c1..00b105fb 100644 --- a/crates/twigpine-node/src/api/repos.rs +++ b/crates/twigpine-node/src/api/repos.rs @@ -3012,7 +3012,9 @@ pub async fn git_receive_pack( // rather than carried as a duration that silently ages: a slow push is // cut at the deadline, so it cannot pin its lock-pool slot for the sum // of every per-phase bound (#282). - let lock_hold_deadline = guard.lock_hold_deadline(); + let lock_hold_deadline = guard + .lock_hold_deadline() + .expect("acquire_write stamps a lock-hold deadline"); let git_service_timeout = std::time::Duration::from_secs(state.config.git_service_timeout_secs); // `timeout(0, drive_git_child)` would still spawn git on its first poll, // so an expired deadline sheds here rather than racing inside the bound. diff --git a/crates/twigpine-node/src/git/repo_store.rs b/crates/twigpine-node/src/git/repo_store.rs index 02d80d4e..edae2ea8 100644 --- a/crates/twigpine-node/src/git/repo_store.rs +++ b/crates/twigpine-node/src/git/repo_store.rs @@ -508,6 +508,28 @@ impl RepoStore { /// and drop the legacy one a release later; that's optional given the /// accepted-window path. pub async fn acquire_write(&self, owner_did: &str, repo_name: &str) -> Result { + self.acquire_write_inner(owner_did, repo_name, Some(self.lock_hold_timeout)) + .await + } + + /// For writers whose under-lock work never consults the hold deadline + /// (merge, issue create/close). Stamping one would only turn a slow but + /// completed write into a refused publish and a rollback at release, while + /// their unbounded git already holds the lock past it. + pub async fn acquire_write_without_hold_deadline( + &self, + owner_did: &str, + repo_name: &str, + ) -> Result { + self.acquire_write_inner(owner_did, repo_name, None).await + } + + async fn acquire_write_inner( + &self, + owner_did: &str, + repo_name: &str, + hold_timeout: Option, + ) -> Result { let (owner_slug, local_path) = self.local_path(owner_did, repo_name)?; let lock_key = advisory_lock_key(&owner_slug, repo_name); @@ -666,7 +688,7 @@ impl RepoStore { conn: Some(lock_conn), tigris: self.tigris.clone(), lock_held_transfer_timeout: self.lock_held_transfer_timeout, - lock_hold_deadline: Instant::now() + self.lock_hold_timeout, + lock_hold_deadline: hold_timeout.map(|t| Instant::now() + t), // Overwritten by the refresh below with the generation actually // observed under the lock. Only reachable unset when no backend is // configured, in which case `release` publishes nothing at all. @@ -2100,8 +2122,9 @@ pub struct RepoWriteGuard { /// lock is held consumes `min(its own bound, the deadline's remaining /// budget)` through `hold_bounded`, so one deadline covers refresh, the /// caller's under-lock work, and release rather than letting each spend a - /// full budget of its own. - lock_hold_deadline: Instant, + /// full budget of its own. `None` carries no deadline: every phase keeps + /// its own bound, exactly as before the deadline existed. + lock_hold_deadline: Option, /// The generation of the stored archive as observed by the HEAD inside /// `acquire_write`, under the lock. `release` publishes fenced on it, so a /// PUT abandoned by an earlier writer's timeout cannot land on top of a @@ -2160,14 +2183,18 @@ impl RepoWriteGuard { /// guard in shared state (the receive-pack reaper bundle) can clamp every /// under-lock phase against the same instant instead of a duration that /// silently ages. - pub(crate) fn lock_hold_deadline(&self) -> Instant { + pub(crate) fn lock_hold_deadline(&self) -> Option { self.lock_hold_deadline } /// `own` clamped to what this guard's lock-hold deadline still funds; see - /// the module-level `hold_bounded` for the rule a `None` carries. + /// the module-level `hold_bounded` for the rule a `None` carries. A guard + /// with no deadline keeps each phase's own bound. fn hold_bounded(&self, own: Duration) -> Option { - hold_bounded(self.lock_hold_deadline, own) + match self.lock_hold_deadline { + Some(d) => hold_bounded(d, own), + None => Some(own), + } } /// Backend pid of the session holding the lock. Test-only observable for the @@ -4214,7 +4241,7 @@ pub(crate) mod tests { conn: Some(pool.acquire().await.expect("conn")), tigris: None, lock_held_transfer_timeout: Duration::from_secs(300), - lock_hold_deadline: Instant::now() + Duration::from_secs(600), + lock_hold_deadline: Some(Instant::now() + Duration::from_secs(600)), publish_fence: UploadPrecondition::Unconditional, refresh_swap_authority: None, publish_stage: Arc::new(PublishStageCell::new()), @@ -4585,7 +4612,7 @@ pub(crate) mod tests { conn: Some(lock_pool.acquire().await.unwrap()), tigris: None, lock_held_transfer_timeout: Duration::from_secs(300), - lock_hold_deadline: Instant::now() + Duration::from_secs(600), + lock_hold_deadline: Some(Instant::now() + Duration::from_secs(600)), // No backend, so nothing is ever published and the fence is unread. publish_fence: UploadPrecondition::Unconditional, refresh_swap_authority: None, @@ -9425,7 +9452,7 @@ pub(crate) mod tests { &endpoint, )), lock_held_transfer_timeout: Duration::from_secs(300), - lock_hold_deadline, + lock_hold_deadline: Some(lock_hold_deadline), publish_fence: UploadPrecondition::Unconditional, refresh_swap_authority: None, publish_stage: Arc::new(PublishStageCell::new()), @@ -9498,6 +9525,53 @@ pub(crate) mod tests { ); } + /// #282: merge and issue writes never consult the hold deadline, so their + /// guards carry none (`acquire_write_without_hold_deadline`). Under a + /// zero-length operator timeout such a guard still refreshes and + /// publishes on each phase's own bound, while the same store's + /// `acquire_write` stamps the already-expired deadline and dies at the + /// under-lock refresh. The release-side refusal a stamped deadline would + /// deal a completed write is pinned by + /// `release_after_lock_hold_deadline_starts_no_publish`. + #[sqlx::test] + async fn write_without_hold_deadline_publishes_after_the_hold_expires(pool: PgPool) { + let mock = S3Mock::start().await; + let opts = (*pool.connect_options()).clone(); + let repos = TempDir::new().unwrap(); + let store = fenced_store(&mock, &opts, repos.path()) + .await + .with_lock_hold_timeout(Duration::ZERO); + let owner = "did:key:z6MkNoHoldDeadline"; + + let guard = store + .acquire_write_without_hold_deadline(owner, "repo") + .await + .expect("acquire"); + let _ = guard.path(); + seed_bare_repo(&guard.local_path); + let outcome = guard.release(true).await; + assert_eq!( + outcome, + ReleaseOutcome::Released, + "a guard carrying no hold deadline must publish, got {outcome:?}" + ); + assert!( + !mock.put_attempts().is_empty(), + "the publish must reach the store" + ); + + // Control: on the same store a guard that DOES stamp the zero + // deadline cannot even start its under-lock refresh, which is exactly + // why these writers take no deadline at all. + let refused = store.acquire_write(owner, "repo").await; + assert!( + refused.is_err(), + "a stamped-and-expired hold deadline must refuse the write" + ); + + mock.shutdown(); + } + /// #282: the unlock is also a phase under the deadline. A release entered /// after the deadline has passed leaves no budget to await the wire, so /// the session must close rather than return to the pool, freeing lock diff --git a/crates/twigpine-node/src/upstream_mirror.rs b/crates/twigpine-node/src/upstream_mirror.rs index c5c1b42e..813c94fe 100644 --- a/crates/twigpine-node/src/upstream_mirror.rs +++ b/crates/twigpine-node/src/upstream_mirror.rs @@ -403,9 +403,12 @@ impl Worker { // lock-hold deadline like every later phase: an expired deadline // starts no query, and a slow read cannot pin the lock and its // lock-pool session past the deadline (#282). - let Some(recheck_bound) = - crate::git::repo_store::hold_bounded(guard.lock_hold_deadline(), STALE_RECHECK_TIMEOUT) - else { + let Some(recheck_bound) = crate::git::repo_store::hold_bounded( + guard + .lock_hold_deadline() + .expect("acquire_write stamps a lock-hold deadline"), + STALE_RECHECK_TIMEOUT, + ) else { let _ = guard.release(false).await; anyhow::bail!( "upstream mirror fetch: the lock-hold deadline expired before the stale-target recheck" @@ -455,7 +458,9 @@ impl Worker { // slow push (#282). `run_fetch` clamps each phase's own bound // against this deadline internally. let fetch_timeout = Duration::from_secs(self.config.upstream_mirror_fetch_timeout_secs); - let lock_hold_deadline = guard.lock_hold_deadline(); + let lock_hold_deadline = guard + .lock_hold_deadline() + .expect("acquire_write stamps a lock-hold deadline"); let fetch = tokio::task::spawn_blocking(move || { run_fetch( &git_bin, From 273aac501d68cdfe1efe5694c069eaae55533849 Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Wed, 18 Nov 2026 22:46:47 -0600 Subject: [PATCH 4/4] fix(node): keep the lock-hold deadline on issue writes create_issue is reachable by any signed reader of a public repo, and its git is one ref write, so the hold deadline is what bounds its refresh and publish. Only merge, whose git runs unbounded either way, takes its guard without the deadline. --- .env.example | 16 ++++++------ README.md | 2 +- crates/twigpine-node/src/api/issues.rs | 29 +++++++++++----------- crates/twigpine-node/src/git/repo_store.rs | 12 +++++---- 4 files changed, 31 insertions(+), 28 deletions(-) diff --git a/.env.example b/.env.example index 55124ee7..b1ce7fe9 100644 --- a/.env.example +++ b/.env.example @@ -101,14 +101,14 @@ TWIGPINE_LOCK_HELD_TRANSFER_TIMEOUT_SECS=300 # ~32 concurrent slow pushes deny every write on the node. Does NOT bound the # acquire-side lock wait (TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS), post-git outcome # bookkeeping, the ~4s the detached reaper holds the session while the git -# process group dies, or merge and issue writes, which take their guards -# without this deadline: their git runs unbounded under the lock, and their -# refresh, publish and unlock keep their own bounds, so a slow merge is never -# rolled back for overrunning it. Unset, the effective bound is -# TWIGPINE_GIT_SERVICE_TIMEOUT_SECS itself (600 by default), so it tracks the -# git budget up or down. Set it only to tighten the hold below the git -# budget; a value above TWIGPINE_GIT_SERVICE_TIMEOUT_SECS is rejected at -# boot. +# process group dies, or merge writes, which take their guards without this +# deadline: their git runs unbounded under the lock, and their refresh, +# publish and unlock keep their own bounds, so a slow merge is never rolled +# back for overrunning it. Issue writes keep the deadline. Unset, the +# effective bound is TWIGPINE_GIT_SERVICE_TIMEOUT_SECS itself (600 by +# default), so it tracks the git budget up or down. Set it only to tighten +# the hold below the git budget; a value above +# TWIGPINE_GIT_SERVICE_TIMEOUT_SECS is rejected at boot. #TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS=300 # Seconds a request waits for a pool connection before failing with 503. TWIGPINE_DB_ACQUIRE_TIMEOUT_SECS=5 diff --git a/README.md b/README.md index d7bc29cb..d37dad54 100644 --- a/README.md +++ b/README.md @@ -433,7 +433,7 @@ Important node settings: | `TWIGPINE_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. | | `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. | | `TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. | -| `TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS` | Max seconds a repo write guard may hold its advisory-lock pool connection, measured from lock-take. One deadline covers every phase under the lock: the under-lock refresh, the caller's under-lock work (for a push, the whole `receive-pack` subprocess, clamped to the lesser of `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), and the release-side upload/unlock. It bounds the lock-pool slot's hold, so a slow push cannot starve other writers for the sum of every per-phase bound. It does not bound the acquire-side lock wait (`TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS`), post-git outcome bookkeeping, the few seconds the detached reaper holds the session while the git process group dies, or merge and issue writes, which take their guards without this deadline: their git runs unbounded under the lock, and their refresh, publish and unlock keep their own bounds, so a slow merge is never rolled back for overrunning it. Unset, the effective bound is `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` itself (600 by default), tracking the git budget; set it only to tighten the hold below the git budget. An explicit value above `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` is rejected at boot. | +| `TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS` | Max seconds a repo write guard may hold its advisory-lock pool connection, measured from lock-take. One deadline covers every phase under the lock: the under-lock refresh, the caller's under-lock work (for a push, the whole `receive-pack` subprocess, clamped to the lesser of `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), and the release-side upload/unlock. It bounds the lock-pool slot's hold, so a slow push cannot starve other writers for the sum of every per-phase bound. It does not bound the acquire-side lock wait (`TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS`), post-git outcome bookkeeping, the few seconds the detached reaper holds the session while the git process group dies, or merge writes, which take their guards without this deadline: their git runs unbounded under the lock, and their refresh, publish and unlock keep their own bounds, so a slow merge is never rolled back for overrunning it. Issue writes keep the deadline. Unset, the effective bound is `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` itself (600 by default), tracking the git budget; set it only to tighten the hold below the git budget. An explicit value above `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` is rejected at boot. | | `TWIGPINE_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `TWIGPINE_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. | | `TWIGPINE_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `TWIGPINE_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `TWIGPINE_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | diff --git a/crates/twigpine-node/src/api/issues.rs b/crates/twigpine-node/src/api/issues.rs index 1c645b45..cb3dad2e 100644 --- a/crates/twigpine-node/src/api/issues.rs +++ b/crates/twigpine-node/src/api/issues.rs @@ -63,7 +63,7 @@ pub async fn create_issue( let guard = state .repo_store - .acquire_write_without_hold_deadline(&record.owner_did, &record.name) + .acquire_write(&record.owner_did, &record.name) .await .map_err(|e| crate::api::repos::acquire_write_app_error(&e, &repo))?; @@ -425,7 +425,7 @@ pub async fn close_issue( // 503. Calling .to_string() first destroys that and reports both as a 500. let guard = state .repo_store - .acquire_write_without_hold_deadline(&record.owner_did, &record.name) + .acquire_write(&record.owner_did, &record.name) .await .map_err(|e| crate::api::repos::acquire_write_app_error(&e, &repo))?; let disk_path = guard.path().to_path_buf(); @@ -1605,13 +1605,13 @@ mod lock_pool_shed_tests { server.abort(); } - /// #282: issue writes never consult the hold deadline, so create takes its - /// guard without one (`acquire_write_without_hold_deadline`). Under a - /// zero-length operator timeout the create still publishes and keeps its - /// ref; a stamped deadline would only roll the finished write back at - /// release and 5xx the caller. + /// #282: issue writes keep the lock-hold deadline. `create_issue` is + /// reachable by any signed DID on a public repo, and its git is one small + /// ref write, so the stamped deadline is what bounds its refresh and + /// publish. Under a zero-length operator timeout the create must refuse, + /// and no issue ref may be left on disk. #[sqlx::test] - async fn create_issue_without_hold_deadline_publishes(pool: PgPool) { + async fn create_issue_keeps_the_lock_hold_deadline(pool: PgPool) { use crate::git::repo_store::tests as rs_tests; let mock = rs_tests::S3Mock::start().await; @@ -1643,13 +1643,14 @@ mod lock_pool_shed_tests { }), ) .await; - let (status, Json(issue)) = res.expect("a create with no hold deadline must publish"); - assert_eq!(status, StatusCode::CREATED); assert!( - git_issues::get_issue(&live, &issue.id) - .expect("get_issue") - .is_some(), - "the issue ref must survive on disk, not be rolled back" + res.is_err(), + "a stamped-and-expired hold deadline must refuse the create" + ); + let issues = git_issues::list_issues(&live).expect("list issues"); + assert!( + issues.is_empty(), + "no issue ref may be left on disk after the refused create, got {issues:?}" ); mock.shutdown(); diff --git a/crates/twigpine-node/src/git/repo_store.rs b/crates/twigpine-node/src/git/repo_store.rs index edae2ea8..f4bfbf4f 100644 --- a/crates/twigpine-node/src/git/repo_store.rs +++ b/crates/twigpine-node/src/git/repo_store.rs @@ -513,9 +513,11 @@ impl RepoStore { } /// For writers whose under-lock work never consults the hold deadline - /// (merge, issue create/close). Stamping one would only turn a slow but - /// completed write into a refused publish and a rollback at release, while - /// their unbounded git already holds the lock past it. + /// (merge). Stamping one would only turn a slow but completed merge into a + /// refused publish and a rollback at release, while its unbounded git + /// already holds the lock past it. Issue writes keep the deadline: their + /// git is one ref write, so the deadline is what bounds their refresh and + /// publish. pub async fn acquire_write_without_hold_deadline( &self, owner_did: &str, @@ -9525,8 +9527,8 @@ pub(crate) mod tests { ); } - /// #282: merge and issue writes never consult the hold deadline, so their - /// guards carry none (`acquire_write_without_hold_deadline`). Under a + /// #282: merge never consults the hold deadline, so its guard carries + /// none (`acquire_write_without_hold_deadline`). Under a /// zero-length operator timeout such a guard still refreshes and /// publishes on each phase's own bound, while the same store's /// `acquire_write` stamps the already-expired deadline and dies at the