diff --git a/.env.example b/.env.example index d18c7324..b1ce7fe9 100644 --- a/.env.example +++ b/.env.example @@ -91,6 +91,25 @@ TWIGPINE_DB_LOCK_POOL_MAX_CONNECTIONS=40 # twice this value plus the git work between them. Read it together with the pool # size above and with TWIGPINE_GIT_SERVICE_TIMEOUT_SECS. TWIGPINE_LOCK_HELD_TRANSFER_TIMEOUT_SECS=300 +# Maximum seconds a repo write guard may hold its advisory-lock pool +# connection, measured from the moment the lock is taken. Every phase under +# the lock consumes this ONE deadline: the under-lock refresh, the caller's +# own under-lock work (for a push, the whole receive-pack subprocess, clamped +# to the lesser of TWIGPINE_GIT_SERVICE_TIMEOUT_SECS and the remaining +# budget), and the release-side upload/unlock. Without it a slow push pins a +# lock-pool slot for the SUM of every per-phase bound (~1500s worst case), so +# ~32 concurrent slow pushes deny every write on the node. Does NOT bound the +# acquire-side lock wait (TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS), post-git outcome +# bookkeeping, the ~4s the detached reaper holds the session while the git +# process group dies, or merge writes, which take their guards without this +# deadline: their git runs unbounded under the lock, and their refresh, +# publish and unlock keep their own bounds, so a slow merge is never rolled +# back for overrunning it. Issue writes keep the deadline. Unset, the +# effective bound is TWIGPINE_GIT_SERVICE_TIMEOUT_SECS itself (600 by +# default), so it tracks the git budget up or down. Set it only to tighten +# the hold below the git budget; a value above +# TWIGPINE_GIT_SERVICE_TIMEOUT_SECS is rejected at boot. +#TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS=300 # Seconds a request waits for a pool connection before failing with 503. TWIGPINE_DB_ACQUIRE_TIMEOUT_SECS=5 # Upper bound on each startup connect+migrate attempt, in seconds. Keep it diff --git a/README.md b/README.md index 107cc23b..d37dad54 100644 --- a/README.md +++ b/README.md @@ -67,8 +67,8 @@ Good today: Known limitations: -- Repository write authorization is one switch: `TWIGPINE_ENFORCE_OWNER_PUSH`, which defaults to `true`. With it off — rolling upgrades only — a valid HTTP Signature identifies a pusher but does not authorize one, so every signed caller may push everywhere. -- UCAN capabilities are consulted on the push path only. There, the chain's root issuer is anchored to the repository owner, so an owner-rooted, time-bounded delegation that names the repository grants scoped collaborator access for pushing to it, for the `git/push`, `*`, or `repo/admin` actions. A `*` *resource* is refused: a delegation must name the repository it applies to, so it cannot grow to cover repos created later. Push-only cuts the other way too: a delegation grants no read access, so on a private repository the delegate must also be a reader (`twigpine visibility set`) — the two gates a push crosses are described in [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). The rest is unchanged: there is no revocation path, `nb` constraints are refused rather than interpreted, and no other route — reads, pull requests, issues, agents — consults capabilities at all. +- Repository write authorization is one switch: `TWIGPINE_ENFORCE_OWNER_PUSH`, which defaults to `true`. With it off (rolling upgrades only) a valid HTTP Signature identifies a pusher but does not authorize one, so every signed caller may push everywhere. +- UCAN capabilities are consulted on the push path only. There, the chain's root issuer is anchored to the repository owner, so an owner-rooted, time-bounded delegation that names the repository grants scoped collaborator access for pushing to it, for the `git/push`, `*`, or `repo/admin` actions. A `*` *resource* is refused: a delegation must name the repository it applies to, so it cannot grow to cover repos created later. Push-only cuts the other way too: a delegation grants no read access, so on a private repository the delegate must also be a reader (`twigpine visibility set`); the two gates a push crosses are described in [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). The rest is unchanged: there is no revocation path, `nb` constraints are refused rather than interpreted, and no other route (reads, pull requests, issues, agents) consults capabilities at all. - Agent lifecycle revocation is not enforced by HTTP Signature authorization; do not rely on removing or revoking an agent record to block a compromised signer. - Task list/get reads (REST and GraphQL) are repository/task-gated and omit `ucan_token`. Slash-form `repo_id` rows (mirror-hosted tasks) can appear on list for the delegator or trimmed assignee, but `GET /api/v1/tasks/{id}` and GraphQL `task` fail closed for every caller because there is no local repository backing to serve. Read visibility is not a blanket data-classification boundary: IPFS-pin and Arweave-anchor listings are not repository-gated; withheld path names can be visible to a root reader; and later visibility changes cannot retract content already announced or externally anchored. - Peer writes are signed by upgraded nodes, but strict signed-peer enforcement is opt-in during rolling upgrades. @@ -427,12 +427,13 @@ Important node settings: | `TWIGPINE_BOOTSTRAP_DISABLE_SEEDS` | Disable embedded seed peers for isolated dev/test networks. | | `TWIGPINE_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. Defaults to `false` during the staged rollout below. | | `TWIGPINE_REQUIRE_SIGNATURE_NONCE` | Require a signed `nonce` on write routes. Default false; see the staged rollout note above. | -| `TWIGPINE_ENFORCE_OWNER_PUSH` | Require the pusher on `git-receive-pack` to be the repo owner, or to present an owner-rooted, time-bounded UCAN `git/push` capability naming the repository (`X-Ucan`). **Defaults to `true`.** A `did:key` signature is authentication, not authorization — anyone can mint a key and sign — so with this off every signed caller may push to every repository, private ones included. A CI or delegated key with no such delegation counts as a non-owner and cannot push while this is on. The delegation clears the push gate only; on a private repository the delegate must also be a reader. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). | +| `TWIGPINE_ENFORCE_OWNER_PUSH` | Require the pusher on `git-receive-pack` to be the repo owner, or to present an owner-rooted, time-bounded UCAN `git/push` capability naming the repository (`X-Ucan`). **Defaults to `true`.** A `did:key` signature is authentication, not authorization (anyone can mint a key and sign), so with this off every signed caller may push to every repository, private ones included. A CI or delegated key with no such delegation counts as a non-owner and cannot push while this is on. The delegation clears the push gate only; on a private repository the delegate must also be a reader. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). | | `TWIGPINE_AUTO_SYNC` | Enable automatic sync from known peers. | | `TWIGPINE_RECONCILIATION_SWEEP` | Enable the hourly durability sweep that re-pins/backstops missing objects (default `true`; disabled when no IPFS/Pinata backend is configured). Public pin repair runs against any configured backend (IPFS, Pinata, or both). Encrypted recovery repair requires local IPFS (`TWIGPINE_IPFS_API`); Pinata-only nodes reconcile public pins only. | | `TWIGPINE_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. | | `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. | | `TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. | +| `TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS` | Max seconds a repo write guard may hold its advisory-lock pool connection, measured from lock-take. One deadline covers every phase under the lock: the under-lock refresh, the caller's under-lock work (for a push, the whole `receive-pack` subprocess, clamped to the lesser of `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), and the release-side upload/unlock. It bounds the lock-pool slot's hold, so a slow push cannot starve other writers for the sum of every per-phase bound. It does not bound the acquire-side lock wait (`TWIGPINE_GIT_ACQUIRE_TIMEOUT_SECS`), post-git outcome bookkeeping, the few seconds the detached reaper holds the session while the git process group dies, or merge writes, which take their guards without this deadline: their git runs unbounded under the lock, and their refresh, publish and unlock keep their own bounds, so a slow merge is never rolled back for overrunning it. Issue writes keep the deadline. Unset, the effective bound is `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` itself (600 by default), tracking the git budget; set it only to tighten the hold below the git budget. An explicit value above `TWIGPINE_GIT_SERVICE_TIMEOUT_SECS` is rejected at boot. | | `TWIGPINE_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `TWIGPINE_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. | | `TWIGPINE_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `TWIGPINE_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `TWIGPINE_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | diff --git a/crates/twigpine-node/src/api/issues.rs b/crates/twigpine-node/src/api/issues.rs index b7b5b4a0..cb3dad2e 100644 --- a/crates/twigpine-node/src/api/issues.rs +++ b/crates/twigpine-node/src/api/issues.rs @@ -1605,6 +1605,57 @@ mod lock_pool_shed_tests { server.abort(); } + /// #282: issue writes keep the lock-hold deadline. `create_issue` is + /// reachable by any signed DID on a public repo, and its git is one small + /// ref write, so the stamped deadline is what bounds its refresh and + /// publish. Under a zero-length operator timeout the create must refuse, + /// and no issue ref may be left on disk. + #[sqlx::test] + async fn create_issue_keeps_the_lock_hold_deadline(pool: PgPool) { + use crate::git::repo_store::tests as rs_tests; + + let mock = rs_tests::S3Mock::start().await; + let owner = "did:key:zISSUENOHOLDAAAAAAAAAAAAAAAAAAAAAAAA"; + let repo_name = "nohold-create"; + let mut state = crate::test_support::test_state(pool.clone()).await; + let disk = tempfile::TempDir::new().unwrap(); + let mut seeded = seed_repo(owner, repo_name); + let live = crate::git::repo_store::validated_repo_disk_path(disk.path(), owner, repo_name) + .expect("repo path"); + seeded.disk_path = live.to_string_lossy().to_string(); + state.db.create_repo(&seeded).await.expect("seed repo"); + crate::git::store::init_bare(&live).expect("init bare repo"); + state.repo_store = crate::git::repo_store::RepoStore::for_testing_with_tigris( + disk.path().to_path_buf(), + crate::git::repo_store::build_lock_pool(&pool, 2, std::time::Duration::from_secs(1)), + rs_tests::mock_tigris(&mock), + ) + .with_lock_hold_timeout(std::time::Duration::ZERO); + + let res = create_issue( + State(state.clone()), + Extension(AuthenticatedDid(owner.to_string())), + Path((owner.to_string(), repo_name.to_string())), + Json(CreateIssueRequest { + title: "t".to_string(), + body: None, + signed_payload: None, + }), + ) + .await; + assert!( + res.is_err(), + "a stamped-and-expired hold deadline must refuse the create" + ); + let issues = git_issues::list_issues(&live).expect("list issues"); + assert!( + issues.is_empty(), + "no issue ref may be left on disk after the refused create, got {issues:?}" + ); + + mock.shutdown(); + } + /// The compensating rewrite after a refused publish must land on the ref the /// close actually mutated. A close addressed by an 8-char prefix resolves to /// the full id and closes `refs/gitlawb/issues/{full_id}`; a compensator diff --git a/crates/twigpine-node/src/api/pulls.rs b/crates/twigpine-node/src/api/pulls.rs index b2f0e03d..4f6e5d37 100644 --- a/crates/twigpine-node/src/api/pulls.rs +++ b/crates/twigpine-node/src/api/pulls.rs @@ -221,7 +221,7 @@ pub async fn merge_pr( let guard = state .repo_store - .acquire_write(&record.owner_did, &record.name) + .acquire_write_without_hold_deadline(&record.owner_did, &record.name) .await .map_err(|e| crate::api::repos::acquire_write_app_error(&e, &name))?; let disk_path = guard.path().to_path_buf(); @@ -878,6 +878,86 @@ mod merge_rollback_tests { "a refused publish must rewind the local merge to the pre-merge tip" ); } + + /// #282: merge never consults the hold deadline, so it takes its guard + /// without one (`acquire_write_without_hold_deadline`). A merge that lands + /// under a zero-length operator timeout still publishes; a stamped + /// deadline would rewind the completed merge and 5xx the caller. + #[sqlx::test] + async fn merge_pr_without_hold_deadline_publishes(pool: PgPool) { + use crate::git::repo_store::tests as rs_tests; + + let mock = rs_tests::S3Mock::start().await; + let owner = "did:key:zMERGENOHOLDAAAAAAAAAAAAAAAAAAAAAAAA"; + let repo_name = "nohold-merge"; + let mut state = crate::test_support::test_state(pool.clone()).await; + let disk = tempfile::TempDir::new().unwrap(); + state.repo_store = crate::git::repo_store::RepoStore::for_testing_with_tigris( + disk.path().to_path_buf(), + crate::git::repo_store::build_lock_pool(&pool, 2, std::time::Duration::from_secs(1)), + rs_tests::mock_tigris(&mock), + ) + .with_lock_hold_timeout(std::time::Duration::ZERO); + let live = crate::git::repo_store::validated_repo_disk_path(disk.path(), owner, repo_name) + .expect("repo path"); + let (main_tip, _feature_tip) = seeded_merge_repo(&live); + + let mut seeded = crate::db::RepoRecord { + id: Uuid::new_v4().to_string(), + name: repo_name.to_string(), + owner_did: owner.to_string(), + description: None, + is_public: true, + default_branch: "main".to_string(), + created_at: Utc::now(), + updated_at: Utc::now(), + disk_path: String::new(), + forked_from: None, + machine_id: None, + }; + seeded.disk_path = live.to_string_lossy().to_string(); + let repo_id = seeded.id.clone(); + state.db.create_repo(&seeded).await.expect("seed repo"); + let now = Utc::now().to_rfc3339(); + state + .db + .create_pr(&PullRequest { + id: Uuid::new_v4().to_string(), + repo_id: repo_id.clone(), + number: 1, + title: "nohold".to_string(), + body: None, + author_did: owner.to_string(), + source_branch: "feature".to_string(), + target_branch: "main".to_string(), + status: "open".to_string(), + merged_by_did: None, + merged_at: None, + head_commit: None, + created_at: now.clone(), + updated_at: now, + }) + .await + .expect("seed PR"); + + let res = merge_pr( + State(state.clone()), + Extension(AuthenticatedDid(owner.to_string())), + Path((owner.to_string(), repo_name.to_string(), 1)), + ) + .await; + let _ = res.expect("a merge with no hold deadline must publish"); + + let merged = state.db.get_pr(&repo_id, 1).await.unwrap().unwrap(); + assert_eq!(merged.status, "merged"); + let tip_after = git(&live, &["rev-parse", "refs/heads/main"]); + assert_ne!( + tip_after, main_tip, + "the merge commit must stay on main, not be rewound" + ); + + mock.shutdown(); + } } /// #173 F1 follow-up: merge holds no admission permit, so an exhausted write-lock diff --git a/crates/twigpine-node/src/api/repos.rs b/crates/twigpine-node/src/api/repos.rs index bfe73d24..00b105fb 100644 --- a/crates/twigpine-node/src/api/repos.rs +++ b/crates/twigpine-node/src/api/repos.rs @@ -3006,10 +3006,46 @@ pub async fn git_receive_pack( } Ok(Ok(g)) => g, }; + // The guard stamps its lock-hold deadline at lock-take inside + // acquire_write. Every phase that runs while the lock is held consumes + // the remaining budget of that one instant, recomputed at each clamp + // rather than carried as a duration that silently ages: a slow push is + // cut at the deadline, so it cannot pin its lock-pool slot for the sum + // of every per-phase bound (#282). + let lock_hold_deadline = guard + .lock_hold_deadline() + .expect("acquire_write stamps a lock-hold deadline"); + let git_service_timeout = std::time::Duration::from_secs(state.config.git_service_timeout_secs); + // `timeout(0, drive_git_child)` would still spawn git on its first poll, + // so an expired deadline sheds here rather than racing inside the bound. + // Shed as a git timeout, matching what a mid-git deadline expiry returns. + // The shed must precede `guard.path()`: taking the path declares the + // tree handed out, and a dropped guard would quarantine a tree no phase + // ever touched. + let Some(git_timeout) = + crate::git::repo_store::hold_bounded(lock_hold_deadline, git_service_timeout) + else { + // Free the lock and its pool slot before the refusal's DB round trip: + // the refusal does not need the lock, and every other pre-git refusal + // runs with none held. + let _ = guard.release(false).await; + if let Err(e) = state + .db + .refuse_receive_pack_before_git( + &request_id, + "write-lock hold deadline expired before git", + ) + .await + { + tracing::warn!(err = %e, request_id = %request_id, "pre-git refusal cleanup failed"); + } + return Err(AppError::Timeout( + "write-lock hold deadline expired before git could start".into(), + )); + }; let disk_path = guard.path().to_path_buf(); tracing::debug!(repo = %name, path = %disk_path.display(), "running git receive-pack"); let body_len = body.len(); - let git_timeout = std::time::Duration::from_secs(state.config.git_service_timeout_secs); // Recovery upgrade invariant (best-effort): before the first // durable intent or marker relies on reflogs/hidden refs, // idempotently enable them for new and upgraded repos. Failures @@ -3021,11 +3057,17 @@ pub async fn git_receive_pack( // Bounded through the configured git binary: the unbounded // synchronous variant would run several `git config` children with // no timeout while holding both admission permits, the lease, and - // the repo write lock. + // the repo write lock. The bound is an Instant, not a duration: the + // helper runs several children, and one shared deadline stops a slow + // sequence from spending the budget several times over. if let Err(e) = crate::git::store::verify_recovery_prereqs_bounded( &state.git_bin, &disk_path, - std::time::Duration::from_secs(10), + crate::git::repo_store::hold_deadline( + lock_hold_deadline, + std::time::Duration::from_secs(10), + ) + .unwrap_or_else(std::time::Instant::now), ) .await { @@ -3079,40 +3121,51 @@ pub async fn git_receive_pack( // marker through the bounded runners using the configured git binary: // both hold admission permits, the lease, and the write lock, so an // unbounded `hash-object` could pin them all on a hung child. - match crate::git::store::marker_value_for_bounded( - &state.git_bin, - &disk_path, - &req_row.request_bytes_hash, - std::time::Duration::from_secs(10), - ) - .await + // `hold_bounded` returns None on an expired deadline, which skips the + // call outright rather than letting `timeout(0)` spawn the child once. + if let Some(marker_value_bound) = + crate::git::repo_store::hold_bounded(lock_hold_deadline, std::time::Duration::from_secs(10)) { - Ok(marker_value) => { - if let Err(e) = crate::git::store::write_marker_bounded( - &state.git_bin, - &disk_path, - &request_id, - &marker_value, - std::time::Duration::from_secs(10), - ) - .await - { + match crate::git::store::marker_value_for_bounded( + &state.git_bin, + &disk_path, + &req_row.request_bytes_hash, + marker_value_bound, + ) + .await + { + Ok(marker_value) => { + if let Some(marker_write_bound) = crate::git::repo_store::hold_bounded( + lock_hold_deadline, + std::time::Duration::from_secs(10), + ) { + if let Err(e) = crate::git::store::write_marker_bounded( + &state.git_bin, + &disk_path, + &request_id, + &marker_value, + marker_write_bound, + ) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "bounded marker write failed; reconcile will quarantine this request" + ); + } + } + } + Err(e) => { tracing::warn!( err = %e, request_id = %request_id, repo = %name, - "bounded marker write failed; reconcile will quarantine this request" + "marker value computation failed; reconcile will quarantine this request" ); } } - Err(e) => { - tracing::warn!( - err = %e, - request_id = %request_id, - repo = %name, - "marker value computation failed; reconcile will quarantine this request" - ); - } } // P1 (reviewer-1/2 round 3): use receive_pack_raw to get the raw @@ -3134,12 +3187,49 @@ pub async fn git_receive_pack( // guard then marks children uncertain and terminalizes the parent, which // the mark_uncertain path below cannot do on a drop — it never runs, and // the detached reaper does process teardown only, no DB writes. + // Re-clamp at the call: the prereq and marker phases above spent the + // same shared deadline, and a bound stamped before them would let + // receive-pack overshoot it by whatever they took. `hold_bounded` is + // also the zero check: an expired deadline sheds as a git timeout + // instead of spawning a receive-pack that is killed on the first poll. + let Some(run_bound) = crate::git::repo_store::hold_bounded(lock_hold_deadline, git_timeout) + else { + // No git child exists yet, so the admission bundle has nothing to + // reap: drop it, reclaim the guard and release it before the refusal's + // DB round trip, so the lock and its pool slot do not outlive the + // deadline. `release(false)` classifies the tree itself (receive-pack + // never ran on it), so this future's settlement copy is disarmed + // rather than left to quarantine an untouched tree on drop. + drop(admission); + let reclaimed = guard + .lock() + .expect("repo write-lock mutex poisoned") + .take() + .expect("the write lock is only taken here, and only once"); + let _ = reclaimed.release(false).await; + if let Some(settlement) = settlement.as_mut() { + settlement.disarm(); + } + if let Err(e) = state + .db + .refuse_receive_pack_before_git( + &request_id, + "write-lock hold deadline expired before git", + ) + .await + { + tracing::warn!(err = %e, request_id = %request_id, "pre-git refusal cleanup failed"); + } + return Err(AppError::Timeout( + "write-lock hold deadline expired before git could start".into(), + )); + }; _intent_guard.notify_git_started(); let (receive_raw, exit_ok) = match smart_http::receive_pack_raw_with_reflog( &state.git_bin, &disk_path, body, - git_timeout, + run_bound, Some(admission), Some(&reflog_action), ) @@ -19296,6 +19386,487 @@ exit 0 "a damaged ref must surface as a git error, not an empty listing" ); } + /// #282 REPRO (RED on main): a push's write guard pins one dedicated + /// lock-pool connection for the WHOLE op: taken at `acquire_write`, held + /// across `smart_http::receive_pack` under `git_service_timeout_secs` + /// (default 600), released only at `release`'s upload tail, with no + /// hold-time bound of its own. A saturated pool then starves every other + /// repo's `acquire_write`, not just the slow pusher's. + /// + /// A one-connection lock pool stands in for "the dedicated pool is full": + /// one held guard is the smallest case of the flood the issue scales to + /// pool size. The shim sleeps on `receive-pack` alone (a slow-but-legit + /// push; prereq and marker children stay real and fast), so the held + /// window is the git phase itself. The probe asserts the desired + /// behaviour (an unrelated write is served while the slow push still + /// runs) and goes RED on main, which is the defect. + #[cfg(unix)] + #[sqlx::test] + async fn repro_282_slow_push_lock_hold_starves_unrelated_writes(pool: sqlx::PgPool) { + use axum::extract::{Path, State}; + use axum::Extension; + use std::net::SocketAddr; + + let tmp = tempfile::TempDir::new().unwrap(); + let repos_dir = tmp.path().join("repos"); + std::fs::create_dir_all(&repos_dir).unwrap(); + let lock_pool = sqlx::postgres::PgPoolOptions::new() + .max_connections(1) + .connect_with((*pool.connect_options()).clone()) + .await + .expect("size-1 lock pool on the test database"); + let mut state = crate::test_support::test_state(pool.clone()).await; + state.repo_store = + crate::git::repo_store::RepoStore::for_testing(repos_dir, lock_pool.clone()) + .with_lock_hold_timeout(std::time::Duration::from_secs(2)); + state.push_limiter_trust = crate::rate_limit::TrustedProxy::None; + let shim_dir = tempfile::TempDir::new().unwrap(); + let marker = shim_dir.path().join("receive-pack-started"); + state.git_bin = write_fake_git( + shim_dir.path(), + &format!( + "#!/bin/sh\ncase \" $*\" in *receive-pack*) : > \"{}\"; sleep 15;; esac\nexec git \"$@\"\n", + marker.display() + ), + ); + + let owner_a = "z6hold282a".to_string(); + let name_a = "slowrepo"; + let owner_b = "z6hold282b".to_string(); + let name_b = "otherrepo"; + let mut repo_a_id = String::new(); + for (owner, name) in [(&owner_a, name_a), (&owner_b, name_b)] { + state + .db + .upsert_mirror_repo(owner, name, &format!("/unused/{owner}/{name}"), None, false) + .await + .unwrap(); + let rec = state.db.get_repo(owner, name).await.unwrap().unwrap(); + state + .repo_store + .init(&rec.owner_did, &rec.name) + .await + .unwrap(); + if name == name_a { + repo_a_id = rec.id.clone(); + } + } + + let state_a = state.clone(); + let owner_push = owner_a.clone(); + let push = tokio::spawn(async move { + git_receive_pack( + State(state_a), + Path((owner_push.clone(), name_a.to_string())), + Extension(crate::auth::AuthenticatedDid(format!( + "did:key:{owner_push}" + ))), + None, + crate::rate_limit::PeerAddr(Some( + "203.0.113.77:5000".parse::().unwrap(), + )), + axum::http::HeaderMap::new(), + ref_update_body(&"a".repeat(40)), + ) + .await + }); + + // The push must reach its git phase (the only lock-pool connection + // pinned) before the unrelated write is probed. + let pin_deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while lock_pool.num_idle() != 0 { + if push.is_finished() { + let r = push.await; + panic!("push returned before pinning the lock pool: {r:?}"); + } + assert!( + std::time::Instant::now() < pin_deadline, + "push never pinned the lock pool" + ); + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + } + + // The pin alone does not prove the git phase: `acquire_write` checks + // the connection out before the advisory lock is even attempted, so + // an early-dead push would free the slot without the bound ever + // firing and the probe below would pass vacuously. Wait for the + // shim's own receive-pack marker so the starved write is only probed + // while the push is genuinely sleeping inside git. + let marker_deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); + while !marker.exists() { + if push.is_finished() { + let r = push.await; + panic!("push returned before reaching its git phase: {r:?}"); + } + assert!( + std::time::Instant::now() < marker_deadline, + "push never reached git receive-pack" + ); + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + } + + // The hold bound (2s) plus process-group reap (~4s cap) frees the + // slot well inside this window; the shim's own 15s sleep is what the + // bound is racing, so a window shorter than the sleep proves the + // bound, not the sleep's end, freed the slot. + let starved = tokio::time::timeout( + std::time::Duration::from_secs(9), + state.repo_store.acquire_write(&owner_b, name_b), + ) + .await; + let served = matches!(&starved, Ok(Ok(_))); + assert!( + served, + "#282: an unrelated repo's acquire_write must be served once the \ + slow push's lock-hold deadline passes (still pending after 9s of \ + a 15s git sleep with a 2s hold bound)" + ); + let push_result = push.await.expect("push task"); + assert!( + push_result.is_err(), + "a push cut by its lock-hold deadline must fail, not report success" + ); + + // A push cut mid-git cannot prove its refs did not land, so the + // intent children must go reconcile-owned `uncertain` (or stay + // `prepared` for startup reconcile), never terminal `cancelled`. + let states: Vec<(String,)> = + sqlx::query_as("SELECT state FROM pending_ref_transitions WHERE repo_id = $1") + .bind(&repo_a_id) + .fetch_all(&pool) + .await + .expect("pending_ref_transitions rows for repo A"); + assert!( + !states.is_empty(), + "the push carried one ref update, so an intent child must exist" + ); + for (state_str,) in &states { + assert!( + state_str == crate::db::pending_state::UNCERTAIN + || state_str == crate::db::pending_state::PREPARED, + "a deadline-cut push's outcome rows must stay reconcile-owned, got {state_str}" + ); + } + } + + /// #282, the other half of the bound: it must never free the lock EARLY. + /// When the hold deadline ends a push the existing git-timeout machinery + /// tears the process group down, and the reaper-carried guard keeps the + /// advisory lock until the group is confirmed gone, the same invariant + /// `receive_pack_disconnect_holds_the_write_lock_until_the_group_is_reaped` + /// pins for the disconnect path, here driven by the deadline instead. + /// + /// A fake git whose descendant ignores SIGTERM widens the teardown window: + /// after the handler's `git_timeout` clamp fires, the group survives the + /// reaper's SIGTERM grace and dies only on the SIGKILL escalation. The lock + /// probed from an independent session must stay untakeable the whole time + /// the descendant lives, and must be takeable once it is gone. + #[cfg(unix)] + #[sqlx::test] + async fn receive_pack_lock_hold_deadline_keeps_the_lock_until_the_group_is_reaped( + pool: sqlx::PgPool, + ) { + use axum::extract::{Path, State}; + use axum::Extension; + use std::net::SocketAddr; + + let owner = "z6deadline"; + let name = "dl1"; + let repos_dir = tempfile::TempDir::new().unwrap(); + let tmp = tempfile::TempDir::new().unwrap(); + let descfile = tmp.path().join("desc.pid"); + let body = format!( + "#!/bin/sh\n\ + case \"$1\" in receive-pack)\n\ + sh -c 'trap \"\" TERM; echo $$ > \"{}\"; i=0; while [ $i -lt 30 ]; do sleep 1; i=$((i+1)); done' &\n\ + wait; exit 0 ;;\n\ + esac\n\ + exec git \"$@\"\n", + descfile.display() + ); + let git_bin = write_fake_git(tmp.path(), &body); + + let mut state = crate::test_support::test_state(pool.clone()).await; + state.git_bin = git_bin; + state.repo_store = crate::git::repo_store::RepoStore::new( + repos_dir.path().to_path_buf(), + None, + crate::git::repo_store::build_lock_pool(&pool, 4, std::time::Duration::from_secs(5)), + std::time::Duration::from_secs(300), + ) + .with_lock_hold_timeout(std::time::Duration::from_secs(2)); + let mut cfg = (*state.config).clone(); + // Long enough that the service timeout is never what ends this push; + // the lock-hold deadline is. + cfg.git_service_timeout_secs = 600; + cfg.enforce_owner_push = false; + state.config = std::sync::Arc::new(cfg); + state + .db + .upsert_mirror_repo(owner, name, "/tmp/z6deadline-dl1", None, false) + .await + .unwrap(); + + use crate::git::repo_store::advisory_lock_key; + let key = advisory_lock_key(&owner.replace([':', '/'], "_"), name); + let probe = sqlx::postgres::PgPoolOptions::new() + .max_connections(2) + .connect_lazy_with((*pool.connect_options()).clone()); + + assert!( + write_lock_is_takeable(&probe, key).await, + "the write lock must be free before the push" + ); + + // Same spawn-retry shape as the disconnect test: a freshly written + // fake `git` can transiently fail to exec under fork-storm load, and a + // losing attempt's future is dropped, which cleans up after itself. + const SPAWN_ATTEMPTS: u64 = 12; + let (mut fut, desc) = { + let mut attempt = 0u64; + loop { + attempt += 1; + let _ = std::fs::remove_file(&descfile); + let mut fut = Box::pin(git_receive_pack( + State(state.clone()), + Path((owner.to_string(), name.to_string())), + Extension(crate::auth::AuthenticatedDid( + "did:key:z6deadline".to_string(), + )), + None, + crate::rate_limit::PeerAddr(Some( + "203.0.113.82:5000".parse::().unwrap(), + )), + axum::http::HeaderMap::new(), + axum::body::Bytes::from_static(b"0000"), + )); + let mut found: Option = None; + for _ in 0..500 { + let finished = + tokio::time::timeout(std::time::Duration::from_millis(10), &mut fut) + .await + .is_ok(); + if let Some(p) = std::fs::read_to_string(&descfile) + .ok() + .and_then(|s| s.trim().parse::().ok()) + { + found = Some(p); + break; + } + if finished { + break; + } + } + match found { + Some(p) => break (fut, p), + None => { + drop(fut); + assert!( + attempt < SPAWN_ATTEMPTS, + "the push never reached receive-pack after {SPAWN_ATTEMPTS} \ + attempts (persistent failure, not a transient runner miss)" + ); + tokio::time::sleep(std::time::Duration::from_millis(100 * attempt)).await; + } + } + } + }; + let _cleanup = KillOnPanic(vec![desc]); + assert!( + pid_alive(desc), + "the receive-pack group must be running when the deadline fires" + ); + + // The 2s hold deadline ends the push. On the timeout path the future + // stays pending while `reap_group_on_timeout` gives the group its + // SIGTERM grace and then SIGKILLs: the SIGTERM-ignoring descendant + // lives through that whole window, and the write lock (carried by the + // guard inside the admission bundle) must stay untakeable for all of + // it. A bound that freed early would show up here as takeable while + // the group is still on the repo. + let mut takeable_while_group_alive = false; + let mut saw_live_group_window = false; + let poll_deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); + let result = loop { + let takeable = write_lock_is_takeable(&probe, key).await; + let group_alive = pid_alive(desc); + if group_alive { + saw_live_group_window = true; + } + if takeable && group_alive { + takeable_while_group_alive = true; + } + if let Ok(done) = + tokio::time::timeout(std::time::Duration::from_millis(10), &mut fut).await + { + break done; + } + assert!( + std::time::Instant::now() < poll_deadline, + "the deadline-cut push never returned within 20s" + ); + }; + // Clean up regardless so a failure leaves no orphan behind. + // SAFETY: kill(2) takes integers only. + unsafe { + libc::kill(desc, libc::SIGKILL); + } + assert!( + result.is_err(), + "a push cut by its lock-hold deadline must fail, not report success" + ); + assert!( + saw_live_group_window, + "the SIGTERM-ignoring descendant should outlive part of teardown; \ + without that the early-free check below observed nothing" + ); + assert!( + !takeable_while_group_alive, + "the repo write lock was takeable while the deadline-cut push's \ + receive-pack group was still alive: the hold bound must never \ + free the lock early (#282)" + ); + // Postgres frees the advisory lock when it notices the closed session, + // which trails the connection drop by a scheduler's tick: poll rather + // than probe once, so a loaded database cannot make this flap. + let free_deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); + while !write_lock_is_takeable(&probe, key).await { + assert!( + std::time::Instant::now() < free_deadline, + "once the deadline-cut push's group is reaped the write lock must \ + be free for the next writer" + ); + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + } + } + + /// #282, the shed that lands after the tree was handed out. When the + /// pre-git phases (recovery prereqs, marker) spend the whole lock-hold + /// deadline, the handler sheds just before receive-pack would start. By + /// then `guard.path()` has declared the tree handed out and the settlement + /// token sits in this future, so a bare return would drop it at stage + /// `Idle` and quarantine a tree git never ran on, and would hold the lock + /// across the refusal's DB call. The shed must release the guard and + /// disarm the settlement first. + /// + /// A real (mock) object store is configured so the stage starts `Idle`; + /// with no backend it starts `NoBackend` and settlement never quarantines. + /// + /// MUTATION (RED): drop the teardown (release and disarm) from the second + /// shed and the quarantine-marker assertion fails. + #[cfg(unix)] + #[sqlx::test] + async fn receive_pack_deadline_shed_after_path_handout_releases_and_does_not_quarantine( + pool: sqlx::PgPool, + ) { + use axum::extract::{Path, State}; + use axum::Extension; + use std::net::SocketAddr; + + let owner = "z6shedtwo"; + let name = "shed2"; + let mock = crate::git::repo_store::tests::S3Mock::start().await; + let repos_dir = tempfile::TempDir::new().unwrap(); + let tmp = tempfile::TempDir::new().unwrap(); + let sentinel = tmp.path().join("receive-pack-ran"); + // Every pre-git child (prereq `config`, marker `hash-object` and + // `update-ref`) hangs past the hold deadline and is cut by it; only + // `receive-pack` would leave the sentinel behind. + let git_bin = write_fake_git( + tmp.path(), + &format!( + "#!/bin/sh\ncase \"$1\" in receive-pack) : > \"{}\"; exit 0 ;; esac\nexec sleep 30\n", + sentinel.display() + ), + ); + + let mut state = crate::test_support::test_state(pool.clone()).await; + state.git_bin = git_bin; + state.repo_store = crate::git::repo_store::RepoStore::for_testing_with_tigris( + repos_dir.path().to_path_buf(), + pool.clone(), + crate::git::repo_store::tests::mock_tigris(&mock), + ) + .with_lock_hold_timeout(std::time::Duration::from_secs(2)); + let mut cfg = (*state.config).clone(); + cfg.git_service_timeout_secs = 600; + cfg.enforce_owner_push = false; + state.config = std::sync::Arc::new(cfg); + state + .db + .upsert_mirror_repo(owner, name, "/tmp/z6shedtwo-shed2", None, false) + .await + .unwrap(); + let rec = state.db.get_repo(owner, name).await.unwrap().unwrap(); + // The tree must exist so the pre-git children actually spawn in it. + let bare = crate::git::repo_store::validated_repo_disk_path( + repos_dir.path(), + &rec.owner_did, + name, + ) + .expect("test repo path"); + crate::git::store::init_bare(&bare).expect("a bare repo on disk"); + let quarantine = p3_quarantine_marker(repos_dir.path(), &rec.owner_did, name); + + use crate::git::repo_store::advisory_lock_key; + let key = advisory_lock_key(&owner.replace([':', '/'], "_"), name); + let probe = sqlx::postgres::PgPoolOptions::new() + .max_connections(2) + .connect_lazy_with((*pool.connect_options()).clone()); + + let result = git_receive_pack( + State(state.clone()), + Path((owner.to_string(), name.to_string())), + Extension(crate::auth::AuthenticatedDid( + "did:key:z6shedtwo".to_string(), + )), + None, + crate::rate_limit::PeerAddr(Some("203.0.113.83:5000".parse::().unwrap())), + axum::http::HeaderMap::new(), + axum::body::Bytes::from_static(b"0000"), + ) + .await; + + let status = match result { + Ok(r) => panic!("a deadline-shed push must fail, got {}", r.status()), + Err(e) => axum::response::IntoResponse::into_response(e).status(), + }; + assert_eq!( + status, + axum::http::StatusCode::GATEWAY_TIMEOUT, + "the pre-git deadline shed answers as a git timeout" + ); + assert!( + !sentinel.exists(), + "receive-pack must never start once the hold deadline is spent" + ); + assert!( + !quarantine.exists(), + "a shed before receive-pack must not quarantine a tree git never ran on" + ); + let free_deadline = std::time::Instant::now() + std::time::Duration::from_secs(2); + while !write_lock_is_takeable(&probe, key).await { + assert!( + std::time::Instant::now() < free_deadline, + "the write lock must be free once the shed returns" + ); + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + } + let rows: Vec<(String, Option)> = + sqlx::query_as("SELECT state, last_error FROM receive_pack_requests") + .fetch_all(&pool) + .await + .unwrap(); + assert_eq!( + rows, + vec![( + crate::db::request_state::REJECTED_AT_GIT.to_string(), + Some("write-lock hold deadline expired before git".to_string()), + )], + "the shed must terminalize its request as a pre-git refusal" + ); + mock.shutdown(); + } } #[cfg(test)] diff --git a/crates/twigpine-node/src/config.rs b/crates/twigpine-node/src/config.rs index 728ec717..4bce7b6a 100644 --- a/crates/twigpine-node/src/config.rs +++ b/crates/twigpine-node/src/config.rs @@ -459,6 +459,34 @@ pub struct Config { )] pub lock_held_transfer_timeout_secs: u64, + /// Maximum wall-clock time a `RepoWriteGuard` may hold its advisory-lock + /// pool connection, measured from the instant `pg_try_advisory_lock` + /// succeeds, in seconds. Every phase that runs while the lock is held + /// (the under-lock Tigris refresh, the caller's own under-lock work; for a + /// push, the whole `receive-pack` subprocess, clamped to the lesser of + /// `git_service_timeout_secs` and the remaining budget, and `release`'s + /// bounded upload/unlock) consumes this one deadline, so a slow or + /// stalled push cannot pin a lock-pool slot for the sum of every + /// per-phase bound (~1500s worst case before this knob existed). On + /// expiry the in-flight git op is torn down and the write fails rather + /// than publishing late. Does NOT bound the acquire-side wait for the + /// lock (`git_acquire_timeout_secs`), the post-git outcome bookkeeping on + /// the main pool, the ~2-4s the detached reaper holds the session while + /// the process group dies, or a caller's synchronous under-lock work that + /// never consults the deadline (the merge/issue write paths still run + /// unbounded git under their guards). Must be positive and at most + /// `git_service_timeout_secs` (a hold bound above the git timeout can + /// never bind before it). Unset (the default) the effective bound is + /// `git_service_timeout_secs` itself (600s shipped), so every push keeps + /// its stated budget and lowering the git timeout lowers the hold bound + /// with it; set explicitly only to tighten the hold below the git budget. + #[arg( + long, + env = "TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS", + value_parser = clap::value_parser!(u64).range(1..=GIT_SERVICE_TIMEOUT_SECS_MAX) + )] + pub write_lock_hold_timeout_secs: Option, + /// Maximum time a request waits for a pool connection before failing with /// 503, in seconds. Bounds queueing when the database is slow or down. #[arg( @@ -1083,6 +1111,16 @@ impl Config { /// lock-pool exhaustion on unrelated repositories. pub const DB_LOCK_POOL_NON_PUSH_HEADROOM: u32 = 8; + /// Effective write-lock hold bound: the explicit knob when set, else + /// `git_service_timeout_secs`, so an unset bound always tracks the push + /// budget (and a lowered git timeout lowers it too). + pub fn write_lock_hold_timeout(&self) -> std::time::Duration { + std::time::Duration::from_secs( + self.write_lock_hold_timeout_secs + .unwrap_or(self.git_service_timeout_secs), + ) + } + /// Cross-field boot validation. Single-field ranges are enforced by clap; this /// catches combinations that ship a denial-of-service under otherwise-valid /// values. Call once at startup and fail fast on `Err`. @@ -1110,6 +1148,16 @@ impl Config { Self::DB_LOCK_POOL_NON_PUSH_HEADROOM )); } + if let Some(hold) = self.write_lock_hold_timeout_secs { + if hold > self.git_service_timeout_secs { + return Err(format!( + "TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS ({}) must not exceed \ + TWIGPINE_GIT_SERVICE_TIMEOUT_SECS ({}): a hold bound above the \ + git timeout can never bind before it", + hold, self.git_service_timeout_secs + )); + } + } if self.upstream_mirror_enabled && !self.enforce_owner_push { return Err( "TWIGPINE_UPSTREAM_MIRROR_ENABLED requires TWIGPINE_ENFORCE_OWNER_PUSH=true: \ @@ -1397,6 +1445,89 @@ mod tests { ); } + #[test] + fn write_lock_hold_timeout_defaults_to_git_budget_and_rejects_zero() { + let default = Config::parse_from(["twigpine-node"]); + assert_eq!(default.write_lock_hold_timeout_secs, None); + assert_eq!( + default.write_lock_hold_timeout(), + std::time::Duration::from_secs(default.git_service_timeout_secs), + "an unset hold bound tracks the git service budget so every push keeps its stated budget" + ); + assert_eq!( + Config::parse_from(["twigpine-node", "--write-lock-hold-timeout-secs", "120"]) + .write_lock_hold_timeout_secs, + Some(120) + ); + // 0 would expire every write guard the instant its lock lands; clap + // must reject it rather than boot a node that can never write. + assert!( + Config::try_parse_from(["twigpine-node", "--write-lock-hold-timeout-secs", "0"]) + .is_err() + ); + // Above the 100-year ceiling every derived deadline can represent. + assert!(Config::try_parse_from([ + "twigpine-node", + "--write-lock-hold-timeout-secs", + "3153600001" + ]) + .is_err()); + } + + /// An unset hold bound must not turn a lowered git timeout into a boot + /// failure: the effective bound simply tracks the configured git budget + /// down with it. + #[test] + fn write_lock_hold_timeout_unset_tracks_a_lowered_git_budget() { + let mut config = Config::parse_from(["twigpine-node", "--git-service-timeout-secs", "120"]); + config + .validate() + .expect("a lowered git timeout with the hold bound unset must validate"); + assert_eq!( + config.write_lock_hold_timeout(), + std::time::Duration::from_secs(120) + ); + } + + /// `validate()` rejects a hold bound above the git service timeout: such a + /// bound can never bind before the git op's own timeout, so accepting it + /// would let an operator believe a bound exists that never fires. + #[test] + fn write_lock_hold_timeout_must_not_exceed_the_git_budget() { + // Equality is legal: the bound fires exactly at the git timeout. + Config::parse_from([ + "twigpine-node", + "--git-service-timeout-secs", + "600", + "--write-lock-hold-timeout-secs", + "600", + ]) + .validate() + .expect("hold bound equal to the git budget must validate"); + + Config::parse_from([ + "twigpine-node", + "--git-service-timeout-secs", + "600", + "--write-lock-hold-timeout-secs", + "120", + ]) + .validate() + .expect("hold bound below the git budget must validate"); + + let mut over = Config::parse_from([ + "twigpine-node", + "--git-service-timeout-secs", + "60", + "--write-lock-hold-timeout-secs", + "61", + ]); + assert!( + over.validate().is_err(), + "a hold bound above the git service timeout must be rejected at boot" + ); + } + #[test] fn require_signature_nonce_ships_off() { assert!( diff --git a/crates/twigpine-node/src/git/repo_store.rs b/crates/twigpine-node/src/git/repo_store.rs index 986aa391..f4bfbf4f 100644 --- a/crates/twigpine-node/src/git/repo_store.rs +++ b/crates/twigpine-node/src/git/repo_store.rs @@ -13,7 +13,7 @@ use std::collections::HashSet; use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; -use std::time::Duration; +use std::time::{Duration, Instant}; use anyhow::{Context, Result}; use sqlx::postgres::PgPoolOptions; @@ -64,6 +64,12 @@ pub struct RepoStore { lock_pool: PgPool, /// Bound on any object-storage transfer that runs while the lock is HELD. lock_held_transfer_timeout: Duration, + /// Wall-clock cap on HOLDING the lock, stamped onto every guard at + /// lock-take. One deadline covers every phase under the lock, so a slow + /// caller cannot pin a lock-pool slot for the sum of every per-phase + /// bound. A field rather than a bare const so the bound is configurable + /// (`TWIGPINE_WRITE_LOCK_HOLD_TIMEOUT_SECS`) and testable. + lock_hold_timeout: Duration, /// Wall-clock cap on WAITING for the lock. A field rather than a bare const so /// the busy path can be driven in a test without a 90s wait. lock_acquire_deadline: Duration, @@ -116,6 +122,17 @@ impl RepoStore { self } + /// Bound on how long a `RepoWriteGuard` from this store may hold its lock, + /// stamped as a deadline at lock-take. Every phase under the lock consumes + /// `min(its own bound, the deadline's remaining budget)`. + /// + /// Production: set from `Config::write_lock_hold_timeout_secs`. Tests: a + /// small value drives the expired-deadline path without a 600s wait. + pub fn with_lock_hold_timeout(mut self, timeout: Duration) -> Self { + self.lock_hold_timeout = timeout; + self + } + /// Test-only: how many write guards from this store have reached the Tigris upload /// site. See [`RepoWriteGuard::release`]. #[cfg(all(test, unix))] @@ -135,6 +152,7 @@ impl RepoStore { tigris, lock_pool, lock_held_transfer_timeout, + lock_hold_timeout: LOCK_HOLD_TIMEOUT, lock_acquire_deadline: LOCK_ACQUIRE_DEADLINE, migrated: Arc::new(Mutex::new(HashSet::new())), #[cfg(test)] @@ -490,6 +508,30 @@ impl RepoStore { /// and drop the legacy one a release later; that's optional given the /// accepted-window path. pub async fn acquire_write(&self, owner_did: &str, repo_name: &str) -> Result { + self.acquire_write_inner(owner_did, repo_name, Some(self.lock_hold_timeout)) + .await + } + + /// For writers whose under-lock work never consults the hold deadline + /// (merge). Stamping one would only turn a slow but completed merge into a + /// refused publish and a rollback at release, while its unbounded git + /// already holds the lock past it. Issue writes keep the deadline: their + /// git is one ref write, so the deadline is what bounds their refresh and + /// publish. + pub async fn acquire_write_without_hold_deadline( + &self, + owner_did: &str, + repo_name: &str, + ) -> Result { + self.acquire_write_inner(owner_did, repo_name, None).await + } + + async fn acquire_write_inner( + &self, + owner_did: &str, + repo_name: &str, + hold_timeout: Option, + ) -> Result { let (owner_slug, local_path) = self.local_path(owner_did, repo_name)?; let lock_key = advisory_lock_key(&owner_slug, repo_name); @@ -648,6 +690,7 @@ impl RepoStore { conn: Some(lock_conn), tigris: self.tigris.clone(), lock_held_transfer_timeout: self.lock_held_transfer_timeout, + lock_hold_deadline: hold_timeout.map(|t| Instant::now() + t), // Overwritten by the refresh below with the generation actually // observed under the lock. Only reachable unset when no backend is // configured, in which case `release` publishes nothing at all. @@ -678,11 +721,17 @@ impl RepoStore { // bounding only the download would leave a mute endpoint able to hold // both indefinitely on the HEAD, and bounding them separately would make // worst-case occupancy two budgets instead of one. - let refreshed = bounded_transfer( - "acquire-refresh", - repo_name, - self.lock_held_transfer_timeout, - async { + // The refresh's own bound, clamped to what is left of this + // guard's lock-hold deadline: the two compose, they do not add. + let refresh_bound = guard.hold_bounded(self.lock_held_transfer_timeout); + let refreshed = match refresh_bound { + None => None, + Some(bound) => { + bounded_transfer( + "acquire-refresh", + repo_name, + bound, + async { // The HEAD and the download fail for epistemically DIFFERENT // reasons, so they are kept apart rather than collapsed into one // `Result`. A failed HEAD leaves us not knowing whether an archive @@ -721,8 +770,10 @@ impl RepoStore { Err(e) => Err(RefreshFailure::Unknown(e)), } }, - ) - .await; + ) + .await + } + }; match refreshed { Some(Ok(fence)) => { @@ -784,16 +835,30 @@ impl RepoStore { // the whole bound is not that: it pinned a lock-pool slot for the // full duration, and this raise-site `error!` is what keeps it // paging. Do NOT "fix" it to match the arm above. - tracing::error!( - repo = %repo_name, - owner = %owner_slug, - bound_secs = self.lock_held_transfer_timeout.as_secs(), - "under-lock tigris refresh exceeded the transfer bound, refusing the write" - ); - return Err(anyhow::Error::new(RepoUnavailable).context(format!( - "tigris refresh exceeded the {}s under-lock bound for {owner_slug}/{repo_name}", - self.lock_held_transfer_timeout.as_secs() - ))); + match refresh_bound { + Some(bound) => { + tracing::error!( + repo = %repo_name, + owner = %owner_slug, + bound_secs = bound.as_secs(), + "under-lock tigris refresh exceeded its bound, refusing the write" + ); + return Err(anyhow::Error::new(RepoUnavailable).context(format!( + "tigris refresh exceeded the {}s under-lock bound for {owner_slug}/{repo_name}", + bound.as_secs() + ))); + } + None => { + tracing::error!( + repo = %repo_name, + owner = %owner_slug, + "under-lock tigris refresh never started: the lock-hold deadline left no budget, refusing the write" + ); + return Err(anyhow::Error::new(RepoUnavailable).context(format!( + "the lock-hold deadline left no budget for the under-lock tigris refresh for {owner_slug}/{repo_name}" + ))); + } + } } } } @@ -2054,6 +2119,14 @@ pub struct RepoWriteGuard { tigris: Option, /// Bound on the release-side upload, which runs with the lock still held. lock_held_transfer_timeout: Duration, + /// The instant this guard's lock-pool hold expires, stamped at lock-take + /// from `RepoStore::lock_hold_timeout`. Every phase that runs while the + /// lock is held consumes `min(its own bound, the deadline's remaining + /// budget)` through `hold_bounded`, so one deadline covers refresh, the + /// caller's under-lock work, and release rather than letting each spend a + /// full budget of its own. `None` carries no deadline: every phase keeps + /// its own bound, exactly as before the deadline existed. + lock_hold_deadline: Option, /// The generation of the stored archive as observed by the HEAD inside /// `acquire_write`, under the lock. `release` publishes fenced on it, so a /// PUT abandoned by an earlier writer's timeout cannot land on top of a @@ -2108,6 +2181,24 @@ pub struct RepoWriteGuard { } impl RepoWriteGuard { + /// The deadline stamped at lock-take. Exposed so a caller that keeps the + /// guard in shared state (the receive-pack reaper bundle) can clamp every + /// under-lock phase against the same instant instead of a duration that + /// silently ages. + pub(crate) fn lock_hold_deadline(&self) -> Option { + self.lock_hold_deadline + } + + /// `own` clamped to what this guard's lock-hold deadline still funds; see + /// the module-level `hold_bounded` for the rule a `None` carries. A guard + /// with no deadline keeps each phase's own bound. + fn hold_bounded(&self, own: Duration) -> Option { + match self.lock_hold_deadline { + Some(d) => hold_bounded(d, own), + None => Some(own), + } + } + /// Backend pid of the session holding the lock. Test-only observable for the /// must-not-over-close check: if `release` closed the session instead of /// returning it, consecutive writes would report different pids. @@ -2124,6 +2215,25 @@ impl RepoWriteGuard { pid.0 } + /// Backend pid PLUS its start timestamp. `pg_backend_pid` alone cannot tell + /// "the same session" from "the kernel recycled the pid for the next + /// backend", which the pool-exhaustion tests hit under parallel load; a + /// recycled pid still gets a fresh `backend_start`. + #[cfg(test)] + async fn backend_identity_for_test(&mut self) -> (i32, String) { + let conn = self + .conn + .as_mut() + .expect("guard still holds its connection"); + sqlx::query_as( + "SELECT pid, backend_start::text FROM pg_stat_activity \ + WHERE pid = pg_backend_pid()", + ) + .fetch_one(&mut **conn) + .await + .expect("backend identity") + } + /// Path to the bare repo on local disk. /// /// Handing the tree out is also the DECLARATION that a write may land on it. @@ -2315,15 +2425,23 @@ impl RepoWriteGuard { // A separate, deliberately small slice rather than a share of the // already-exhausted publish budget: this runs with the lock still held, // and a store that just stalled for the whole bound must not be able to - // hold it for a second one. - match bounded_transfer( - "release-reconcile", - &self.repo_name, - RECONCILE_BOUND, - tigris.attempt_landed(&self.owner_slug, &self.repo_name, &attempt), - ) - .await - { + // hold it for a second one. An expired lock-hold deadline starts + // nothing: `hold_bounded` returns None rather than let `timeout(0)` + // poll the probe once and possibly land an answer the deadline no + // longer funds. + let reconciled = match self.hold_bounded(RECONCILE_BOUND) { + None => None, + Some(bound) => { + bounded_transfer( + "release-reconcile", + &self.repo_name, + bound, + tigris.attempt_landed(&self.owner_slug, &self.repo_name, &attempt), + ) + .await + } + }; + match reconciled { Some(Ok(true)) => { info!( repo = %self.repo_name, @@ -2382,14 +2500,24 @@ impl RepoWriteGuard { // refresh uses one for its HEAD and download together: this runs // with the lock held and a lock-pool slot pinned, so bounding // each attempt separately would double the worst-case occupancy. - match bounded_transfer( - "release-upload", - &self.repo_name, - self.lock_held_transfer_timeout, - self.publish(&tigris), - ) - .await - { + // The bound also composes with the lock-hold deadline: a guard + // whose deadline has passed starts nothing, classified through + // `resolve_unfinished_publish` as definite non-publication at + // stage Idle. The skip is explicit because `timeout(0)` would + // still poll the publish once. + let published = match self.hold_bounded(self.lock_held_transfer_timeout) { + None => None, + Some(bound) => { + bounded_transfer( + "release-upload", + &self.repo_name, + bound, + self.publish(&tigris), + ) + .await + } + }; + match published { Some(Ok(())) => {} // Both attempts were definitively refused. The raise site // already logged which and why, so this only has to carry @@ -2519,18 +2647,26 @@ impl RepoWriteGuard { if let Some(gate) = self.test_pre_unlock_gate.clone() { gate.notified().await; } - let unlock = if let Some(conn) = self.conn.as_mut() { - bounded_transfer( - "advisory-unlock", - &self.repo_name, - self.lock_held_transfer_timeout, - sqlx::query_as::<_, (bool,)>("SELECT pg_advisory_unlock($1)") - .bind(lock_key) - .fetch_one(&mut **conn), - ) - .await - } else { - None + // A lock-hold deadline already past is excluded by `hold_bounded` + // rather than left to `timeout(0)`, which would still poll the + // query once and could land it. + let unlock_bound = self.hold_bounded(self.lock_held_transfer_timeout); + let unlock = match (self.conn.as_mut(), unlock_bound) { + (Some(conn), Some(bound)) => { + bounded_transfer( + "advisory-unlock", + &self.repo_name, + bound, + sqlx::query_as::<_, (bool,)>("SELECT pg_advisory_unlock($1)") + .bind(lock_key) + .fetch_one(&mut **conn), + ) + .await + } + // No connection, or an expired deadline with nothing left to + // fund the wire call: the `None` arms below close the session, + // which frees lock and slot regardless. + _ => None, }; match unlock { Some(Ok((true,))) => { @@ -2553,11 +2689,14 @@ impl RepoWriteGuard { ); } None if self.conn.is_some() => { + // `unlock` is None on a bounded_transfer timeout AND on a + // deadline-expired skip; log the effective bound so a + // zero says which happened. warn!( repo = %self.repo_name, lock_key, - bound_secs = self.lock_held_transfer_timeout.as_secs(), - "advisory unlock exceeded its bound — closing the session so the lock-pool slot is not held longer" + bound_secs = unlock_bound.map(|b| b.as_secs()).unwrap_or(0), + "advisory unlock did not finish inside its bound; closing the session so the lock-pool slot is not held longer" ); } None => {} @@ -2740,6 +2879,29 @@ impl Drop for RepoWriteGuard { /// `infra/fly/fly.toml`; it is not, and reconciling the two is tracked separately. const LOCK_ACQUIRE_DEADLINE: Duration = Duration::from_secs(90); +/// Production `RepoStore::new` sites override this from +/// `Config::write_lock_hold_timeout_secs`; the fallback matches that knob's +/// default so a store built without it keeps the shipped bound. +const LOCK_HOLD_TIMEOUT: Duration = Duration::from_secs(600); + +/// `own` clamped to what `deadline` still funds, or `None` once nothing is +/// left. This is the one body that carries the lock-hold rule every under-lock +/// phase follows: a `tokio::time::timeout` at exactly zero polls the future +/// once before noticing (and a process runner has already spawned by then), so +/// an expired deadline must short-circuit here rather than race inside the +/// bound. Callers keep their own `None` policy: refuse, warn-and-skip, or bail. +pub(crate) fn hold_bounded(deadline: Instant, own: Duration) -> Option { + let bound = own.min(deadline.saturating_duration_since(Instant::now())); + (!bound.is_zero()).then_some(bound) +} + +/// `hold_bounded` for callees that take an `Instant` deadline rather than a +/// `Duration`: the phase's own bound as an instant, never past `deadline`, or +/// `None` once the hold deadline has passed. +pub(crate) fn hold_deadline(deadline: Instant, own: Duration) -> Option { + Instant::now().checked_add(hold_bounded(deadline, own)?) +} + /// Why an under-lock refresh did not complete, split by what it leaves us knowing. /// /// `Unknown` (the existence check failed) and `Download` (the archive is there and @@ -4081,6 +4243,7 @@ pub(crate) mod tests { conn: Some(pool.acquire().await.expect("conn")), tigris: None, lock_held_transfer_timeout: Duration::from_secs(300), + lock_hold_deadline: Some(Instant::now() + Duration::from_secs(600)), publish_fence: UploadPrecondition::Unconditional, refresh_swap_authority: None, publish_stage: Arc::new(PublishStageCell::new()), @@ -4451,6 +4614,7 @@ pub(crate) mod tests { conn: Some(lock_pool.acquire().await.unwrap()), tigris: None, lock_held_transfer_timeout: Duration::from_secs(300), + lock_hold_deadline: Some(Instant::now() + Duration::from_secs(600)), // No backend, so nothing is ever published and the fence is unread. publish_fence: UploadPrecondition::Unconditional, refresh_swap_authority: None, @@ -9239,4 +9403,215 @@ pub(crate) mod tests { mock.shutdown(); } + + /// Release a hand-built guard over a SEEDED bare repo against a mock store + /// that answers 200 to everything, with the given lock-hold deadline. + /// Returns the outcome and how many PUTs reached the mock. + /// + /// The repo is seeded so archive preparation succeeds: without it a + /// publish that wrongly started would still fail before any PUT and read + /// as `UploadFailed`, which is the outcome the expired case asserts. + async fn release_with_lock_hold_deadline( + pool: &sqlx::PgPool, + repo_name: &str, + lock_key: i64, + lock_hold_deadline: Instant, + ) -> (ReleaseOutcome, usize) { + use axum::response::IntoResponse; + + let puts = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let puts_in = Arc::clone(&puts); + let app = axum::Router::new().route( + "/{*key}", + axum::routing::any(move |method: axum::http::Method| { + let puts = Arc::clone(&puts_in); + async move { + if method == axum::http::Method::PUT { + puts.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + } + axum::http::StatusCode::OK.into_response() + } + }), + ); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let endpoint = format!("http://127.0.0.1:{}", listener.local_addr().unwrap().port()); + let server = tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + + let dir = tempfile::TempDir::new().unwrap(); + let local_path = validated_repo_disk_path(dir.path(), "did:key:z6MkHoldExpired", repo_name) + .expect("test path"); + seed_bare_repo(&local_path); + let guard = RepoWriteGuard { + owner_slug: "did_key_z6MkHoldExpired".to_string(), + repo_name: repo_name.to_string(), + local_path, + lock_key, + conn: Some(pool.acquire().await.unwrap()), + tigris: Some(TigrisClient::for_testing_with_endpoint( + "test-bucket", + &endpoint, + )), + lock_held_transfer_timeout: Duration::from_secs(300), + lock_hold_deadline: Some(lock_hold_deadline), + publish_fence: UploadPrecondition::Unconditional, + refresh_swap_authority: None, + publish_stage: Arc::new(PublishStageCell::new()), + tree_settled: false, + settlement_armed: Arc::new(AtomicBool::new(true)), + path_handed_out: Arc::new(AtomicBool::new(true)), + #[cfg(test)] + test_pre_unlock_gate: None, + #[cfg(test)] + upload_site_reached: Arc::new(std::sync::atomic::AtomicUsize::new(0)), + }; + + let outcome = guard.release(true).await; + let put_count = puts.load(std::sync::atomic::Ordering::SeqCst); + server.abort(); + (outcome, put_count) + } + + /// #282: every phase under the lock consumes the guard's one lock-hold + /// deadline. A guard whose deadline has already passed must not START a + /// publish the bound cannot finish: the `None` (timed out) arm routes + /// through `resolve_unfinished_publish`, where stage `Idle` reads as a + /// definite non-publication (`UploadFailed`), and no PUT is dispatched. + /// + /// Load-bearing on the clamp: the repo is seeded and the mock accepts + /// every PUT, so without the `hold_bounded` deadline the publish would + /// succeed (`Released`) and the counter would move. The sibling test + /// below proves that same fixture does publish when the deadline is live. + #[sqlx::test] + async fn release_after_lock_hold_deadline_starts_no_publish(pool: sqlx::PgPool) { + // Already past: the publish's budget must clamp to zero. + let (outcome, puts) = release_with_lock_hold_deadline( + &pool, + "expired-hold", + 995_777, + Instant::now() - Duration::from_secs(1), + ) + .await; + assert_eq!( + outcome, + ReleaseOutcome::UploadFailed, + "an expired lock-hold deadline makes the publish a definite non-publication, got {outcome:?}" + ); + assert_eq!( + puts, 0, + "no PUT may be dispatched once the lock-hold deadline has passed" + ); + } + + /// Control for the test above: the same fixture with a live lock-hold + /// deadline publishes. Without it, a fixture that could never publish + /// would make the expired case pass for the wrong reason. + #[sqlx::test] + async fn release_before_lock_hold_deadline_publishes(pool: sqlx::PgPool) { + let (outcome, puts) = release_with_lock_hold_deadline( + &pool, + "live-hold", + 995_778, + Instant::now() + Duration::from_secs(300), + ) + .await; + assert_eq!( + outcome, + ReleaseOutcome::Released, + "a live lock-hold deadline lets the publish run to completion, got {outcome:?}" + ); + assert!( + puts > 0, + "the publish must reach the store while the lock-hold deadline is live" + ); + } + + /// #282: merge never consults the hold deadline, so its guard carries + /// none (`acquire_write_without_hold_deadline`). Under a + /// zero-length operator timeout such a guard still refreshes and + /// publishes on each phase's own bound, while the same store's + /// `acquire_write` stamps the already-expired deadline and dies at the + /// under-lock refresh. The release-side refusal a stamped deadline would + /// deal a completed write is pinned by + /// `release_after_lock_hold_deadline_starts_no_publish`. + #[sqlx::test] + async fn write_without_hold_deadline_publishes_after_the_hold_expires(pool: PgPool) { + let mock = S3Mock::start().await; + let opts = (*pool.connect_options()).clone(); + let repos = TempDir::new().unwrap(); + let store = fenced_store(&mock, &opts, repos.path()) + .await + .with_lock_hold_timeout(Duration::ZERO); + let owner = "did:key:z6MkNoHoldDeadline"; + + let guard = store + .acquire_write_without_hold_deadline(owner, "repo") + .await + .expect("acquire"); + let _ = guard.path(); + seed_bare_repo(&guard.local_path); + let outcome = guard.release(true).await; + assert_eq!( + outcome, + ReleaseOutcome::Released, + "a guard carrying no hold deadline must publish, got {outcome:?}" + ); + assert!( + !mock.put_attempts().is_empty(), + "the publish must reach the store" + ); + + // Control: on the same store a guard that DOES stamp the zero + // deadline cannot even start its under-lock refresh, which is exactly + // why these writers take no deadline at all. + let refused = store.acquire_write(owner, "repo").await; + assert!( + refused.is_err(), + "a stamped-and-expired hold deadline must refuse the write" + ); + + mock.shutdown(); + } + + /// #282: the unlock is also a phase under the deadline. A release entered + /// after the deadline has passed leaves no budget to await the wire, so + /// the session must close rather than return to the pool, freeing lock + /// AND slot even though `pg_advisory_unlock` never ran. + /// + /// The observable is the backend identity on a one-connection pool: a + /// returned connection is handed back as the same session; a closed one + /// forces the next acquire onto a fresh backend. The identity carries + /// `backend_start` alongside the pid because the kernel recycles pids, so + /// an equal pid alone cannot separate same-session from new-backend. + /// Load-bearing on the clamp: without it the unlock succeeds, the + /// connection returns, and the identity repeats. + #[sqlx::test] + async fn release_after_lock_hold_deadline_closes_the_session(pool: sqlx::PgPool) { + let opts = (*pool.connect_options()).clone(); + let lock_pool = no_reap_pool(&opts, 1).await; + let dir = tempfile::TempDir::new().unwrap(); + let store = RepoStore::for_testing(dir.path().to_path_buf(), lock_pool.clone()) + .with_lock_hold_timeout(Duration::from_millis(200)); + + let owner = "did:key:z6MkHoldDeadlineSessionAAAAAAAAAAAAAAAAA"; + let name = "holddeadline"; + let mut guard = store.acquire_write(owner, name).await.expect("acquire"); + let identity_before = guard.backend_identity_for_test().await; + + // Let the deadline pass, then release: the unlock gets no budget. + tokio::time::sleep(Duration::from_millis(300)).await; + let _ = guard.release(false).await; + + let mut next = store + .acquire_write(owner, name) + .await + .expect("the lock must be free once the expired session closes"); + let identity_after = next.backend_identity_for_test().await; + assert_ne!( + identity_before, identity_after, + "an expired-deadline release must close the session, not pool it" + ); + let _ = next.release(false).await; + } } diff --git a/crates/twigpine-node/src/git/store.rs b/crates/twigpine-node/src/git/store.rs index bfedeb72..c296eb0b 100644 --- a/crates/twigpine-node/src/git/store.rs +++ b/crates/twigpine-node/src/git/store.rs @@ -168,15 +168,23 @@ pub fn ensure_marker_hidden_checked(repo_path: &Path) -> Result<()> { pub async fn verify_recovery_prereqs_bounded( git_bin: &str, repo_path: &Path, - timeout: std::time::Duration, + deadline: std::time::Instant, ) -> Result<()> { use tokio::process::Command as AsyncCommand; async fn run_git( git_bin: &str, repo_path: &Path, args: &[&str], - timeout: std::time::Duration, + deadline: std::time::Instant, ) -> Result { + // Recompute per child: several `git config` calls run in sequence + // and one shared deadline must cover them all, not the same + // duration spent once per child. `hold_bounded` returns None on an + // expired deadline so the child is never spawned. + let Some(bound) = super::repo_store::hold_bounded(deadline, std::time::Duration::MAX) + else { + anyhow::bail!("git {} timed out", args.join(" ")); + }; let fut = async { // allow-unbounded-git: bounded by the tokio::time::timeout wrapping // this future in verify_recovery_prereqs_bounded, plus kill_on_drop. @@ -184,7 +192,7 @@ pub async fn verify_recovery_prereqs_bounded( cmd.kill_on_drop(true); cmd.args(args).current_dir(repo_path).output().await }; - tokio::time::timeout(timeout, fut) + tokio::time::timeout(bound, fut) .await .map_err(|_| anyhow::anyhow!("git {} timed out", args.join(" ")))? .map_err(|e| anyhow::anyhow!("git {} spawn failed: {e:#}", args.join(" "))) @@ -193,7 +201,7 @@ pub async fn verify_recovery_prereqs_bounded( git_bin, repo_path, &["config", "--get", "core.logAllRefUpdates"], - timeout, + deadline, ) .await?; let cur = String::from_utf8_lossy(&get.stdout).trim().to_string(); @@ -202,7 +210,7 @@ pub async fn verify_recovery_prereqs_bounded( git_bin, repo_path, &["config", "core.logAllRefUpdates", "always"], - timeout, + deadline, ) .await?; if !out.status.success() { @@ -216,7 +224,7 @@ pub async fn verify_recovery_prereqs_bounded( ("uploadpack.hideRefs", "refs/gitlawb/requests/"), ("transfer.hideRefs", "refs/gitlawb/requests/"), ] { - let current = run_git(git_bin, repo_path, &["config", "--get-all", key], timeout).await?; + let current = run_git(git_bin, repo_path, &["config", "--get-all", key], deadline).await?; let already = String::from_utf8_lossy(¤t.stdout) .lines() .any(|l| l.trim() == value); @@ -225,7 +233,7 @@ pub async fn verify_recovery_prereqs_bounded( git_bin, repo_path, &["config", "--add", key, value], - timeout, + deadline, ) .await?; if !out.status.success() { diff --git a/crates/twigpine-node/src/lib.rs b/crates/twigpine-node/src/lib.rs index 2173aa00..f4054e63 100644 --- a/crates/twigpine-node/src/lib.rs +++ b/crates/twigpine-node/src/lib.rs @@ -415,7 +415,8 @@ pub async fn run() -> Result<()> { tigris, lock_pool, std::time::Duration::from_secs(config.lock_held_transfer_timeout_secs), - ); + ) + .with_lock_hold_timeout(config.write_lock_hold_timeout()); // Per-DID limiter for the creation endpoints. Keyed on the authenticated // DID (attacker-varied), so bound its key set to cap memory. Max and window @@ -1026,7 +1027,8 @@ async fn run_admin_command(command: config::Command, config: &Config) -> Result< object_store, lock_pool, std::time::Duration::from_secs(config.lock_held_transfer_timeout_secs), - ); + ) + .with_lock_hold_timeout(config.write_lock_hold_timeout()); admin::run_purge_spam(&db, &repo_store, &config.repos_dir, execute) .await .map(|_| ()) diff --git a/crates/twigpine-node/src/upstream_mirror.rs b/crates/twigpine-node/src/upstream_mirror.rs index 6d3c8bbd..813c94fe 100644 --- a/crates/twigpine-node/src/upstream_mirror.rs +++ b/crates/twigpine-node/src/upstream_mirror.rs @@ -290,12 +290,25 @@ fn reject_sensitive_local_config(git_bin: &str, repo_path: &Path, deadline: Inst /// only the external pack transfer it documents. const PREFLIGHT_PROBE_TIMEOUT: Duration = Duration::from_secs(30); +/// Own bound for the under-lock stale-target recheck in +/// [`Worker::fetch_target`]: one indexed row read, so a short bound is +/// generous. It is clamped to the guard's lock-hold deadline like every +/// other phase under the write lock (#282). +const STALE_RECHECK_TIMEOUT: Duration = Duration::from_secs(30); + fn run_fetch( git_bin: &str, repo_path: &Path, upstream: &PinnedUpstream, timeout: Duration, + lock_hold_deadline: Instant, ) -> Result<()> { + // An already-expired hold deadline starts nothing: without this check + // the preflight probes and the fetch child would each spawn once + // before their own zero deadlines noticed. + if Instant::now() >= lock_hold_deadline { + anyhow::bail!("upstream mirror fetch: the lock-hold deadline already expired"); + } let command = fetch_command(upstream); let args: Vec<&str> = command.args.iter().map(String::as_str).collect(); let config: Vec<(&str, &str)> = command @@ -303,17 +316,22 @@ fn run_fetch( .iter() .map(|(key, value)| (key.as_str(), value.as_str())) .collect(); - let rewrite_deadline = Instant::now() - .checked_add(PREFLIGHT_PROBE_TIMEOUT) - .context("mirror URL-rewrite probe timeout is not representable")?; + // Every phase under the write lock consumes the guard's lock-hold + // deadline: each keeps its own bound, recomputed per phase through + // `hold_deadline` so an expiry during one probe means the next child + // never spawns instead of starting on a deadline already past (#282). + let expired = + || anyhow::anyhow!("upstream mirror fetch: the lock-hold deadline already expired"); + let rewrite_deadline = + crate::git::repo_store::hold_deadline(lock_hold_deadline, PREFLIGHT_PROBE_TIMEOUT) + .ok_or_else(expired)?; reject_local_url_rewrite(git_bin, repo_path, &upstream.url, rewrite_deadline)?; - let config_deadline = Instant::now() - .checked_add(PREFLIGHT_PROBE_TIMEOUT) - .context("mirror local-config probe timeout is not representable")?; + let config_deadline = + crate::git::repo_store::hold_deadline(lock_hold_deadline, PREFLIGHT_PROBE_TIMEOUT) + .ok_or_else(expired)?; reject_sensitive_local_config(git_bin, repo_path, config_deadline)?; - let deadline = Instant::now() - .checked_add(timeout) - .context("upstream mirror fetch timeout is not representable")?; + let deadline = + crate::git::repo_store::hold_deadline(lock_hold_deadline, timeout).ok_or_else(expired)?; let (status, _stdout, stderr) = crate::git::visibility_pack::run_bounded_git_raw_isolated_https_with_config( git_bin, &args, repo_path, b"", deadline, &config, @@ -380,7 +398,34 @@ impl Worker { // are held. Every future transition path must take these same locks // before changing authority, so no other node can commit a transition // between this check and the fetch. - let current = match self.db.get_repo_mirror_state(&target.repo_id).await { + // + // The recheck runs under the write lock, so it consumes the guard's + // lock-hold deadline like every later phase: an expired deadline + // starts no query, and a slow read cannot pin the lock and its + // lock-pool session past the deadline (#282). + let Some(recheck_bound) = crate::git::repo_store::hold_bounded( + guard + .lock_hold_deadline() + .expect("acquire_write stamps a lock-hold deadline"), + STALE_RECHECK_TIMEOUT, + ) else { + let _ = guard.release(false).await; + anyhow::bail!( + "upstream mirror fetch: the lock-hold deadline expired before the stale-target recheck" + ); + }; + let recheck = tokio::time::timeout( + recheck_bound, + self.db.get_repo_mirror_state(&target.repo_id), + ) + .await; + let Ok(recheck) = recheck else { + let _ = guard.release(false).await; + anyhow::bail!( + "upstream mirror fetch: the stale-target recheck did not finish inside the lock-hold budget" + ); + }; + let current = match recheck { Ok(Some(current)) => current, Ok(None) => { let _ = guard.release(false).await; @@ -407,9 +452,23 @@ impl Worker { } let git_bin = self.git_bin.clone(); + // The fetch holds the write lock the whole time, so it consumes the + // guard's lock-hold deadline rather than a fresh budget: a fetch + // running past that deadline pins its lock-pool slot exactly like a + // slow push (#282). `run_fetch` clamps each phase's own bound + // against this deadline internally. let fetch_timeout = Duration::from_secs(self.config.upstream_mirror_fetch_timeout_secs); + let lock_hold_deadline = guard + .lock_hold_deadline() + .expect("acquire_write stamps a lock-hold deadline"); let fetch = tokio::task::spawn_blocking(move || { - run_fetch(&git_bin, &repo_path, &upstream, fetch_timeout) + run_fetch( + &git_bin, + &repo_path, + &upstream, + fetch_timeout, + lock_hold_deadline, + ) }) .await .context("upstream mirror fetch task panicked")?; @@ -865,7 +924,14 @@ esac url: format!("https://127.0.0.1:{port}/org/repo=mirror.git"), curlopt_resolve: None, }; - let error = run_fetch("git", repo.path(), &upstream, Duration::from_secs(5)).unwrap_err(); + let error = run_fetch( + "git", + repo.path(), + &upstream, + Duration::from_secs(5), + Instant::now() + Duration::from_secs(600), + ) + .unwrap_err(); assert!( accepted.join().unwrap(), "real Git must reach the HTTPS transport; got {error:#}" @@ -965,4 +1031,141 @@ esac assert_eq!(calls.lines().collect::>(), ["fail", "ok"]); assert!(worker.repo_write_leases.is_empty()); } + + /// #282: a mirror fetch holds the write lock for its whole run, so it + /// consumes the guard's lock-hold deadline, not a fresh + /// `upstream_mirror_fetch_timeout_secs` budget. With the hold bound below + /// the fetch timeout, a sleeping fetch must die at the hold bound, not at + /// the fetch timeout. Load-bearing on the clamp: without the + /// `lock_hold_deadline` phase clamps the fetch runs to the 5s timeout and + /// the elapsed assertion below fails. + #[cfg(unix)] + #[sqlx::test] + async fn fetch_respects_the_guard_lock_hold_deadline(pool: sqlx::PgPool) { + let bootstrap = tempfile::tempdir().unwrap(); + let git = bootstrap.path().join("fake-git"); + let fetch_marker = bootstrap.path().join("fetch-started"); + // Answer the preflight probes like `fake_git` does (a bare repo with + // no rewrites and no sensitive config), then mark and sleep on the + // fetch so the test can tell "the fetch child ran" from "the run + // bailed before spawning anything". + std::fs::write( + &git, + format!( + "#!/bin/sh\n\ + if [ \"$1\" = \"ls-remote\" ]; then printf '%s\\n' \"$3\"; exit 0; fi\n\ + if [ \"$1\" = \"config\" ]; then exit 1; fi\n\ + : > \"{}\"\n\ + sleep 30\n", + fetch_marker.display() + ), + ) + .unwrap(); + { + use std::os::unix::fs::PermissionsExt; + let mut permissions = std::fs::metadata(&git).unwrap().permissions(); + permissions.set_mode(0o700); + std::fs::set_permissions(&git, permissions).unwrap(); + } + + // Fetch timeout is 5s from `test_worker`; the hold bound is tighter. + let (mut worker, db, _repos) = test_worker(pool, git, 1).await; + worker.repo_store = worker + .repo_store + .with_lock_hold_timeout(Duration::from_millis(800)); + let target = add_inbound_mirror( + &worker, + &db, + "00000000-0000-0000-0000-000000000003", + "slowfetch", + "https://8.8.8.8/org/slow.git", + ) + .await; + + let started = Instant::now(); + let result = worker.fetch_target(target).await; + let elapsed = started.elapsed(); + assert!( + result.is_err(), + "a fetch cut by the lock-hold deadline must fail, got {result:?}" + ); + // The preflight shims answer instantly, so a fast failure would + // return in milliseconds. The shim's marker plus reaching ~800ms + // prove the fetch child really started and the deadline is what cut + // it, not an expired-deadline bail before the spawn. + assert!( + fetch_marker.exists(), + "the fetch child must have started for the deadline to have cut it" + ); + assert!( + elapsed >= Duration::from_millis(700), + "a fetch that never started would fail fast, took only {elapsed:?}" + ); + assert!( + elapsed < Duration::from_secs(4), + "a 800ms lock-hold bound must end the fetch well inside the 5s \ + fetch timeout, took {elapsed:?}" + ); + assert!(worker.repo_write_leases.is_empty()); + } + + /// #282: the stale-target recheck runs under the write lock, so it + /// consumes the guard's lock-hold deadline too. With a zero hold bound + /// the deadline has passed by the time the recheck would run: the + /// recheck must refuse, release the lock, and spawn no git child. + /// Load-bearing on the recheck's `hold_bounded` check: without it the + /// recheck runs and `run_fetch` bails with its own expired-deadline + /// message instead, so the message assertion below fails. + #[cfg(unix)] + #[sqlx::test] + async fn stale_recheck_respects_the_guard_lock_hold_deadline(pool: sqlx::PgPool) { + let bootstrap = tempfile::tempdir().unwrap(); + let git = bootstrap.path().join("fake-git"); + let git_marker = bootstrap.path().join("git-invoked"); + std::fs::write( + &git, + format!("#!/bin/sh\n: > \"{}\"\nexit 1\n", git_marker.display()), + ) + .unwrap(); + { + use std::os::unix::fs::PermissionsExt; + let mut permissions = std::fs::metadata(&git).unwrap().permissions(); + permissions.set_mode(0o700); + std::fs::set_permissions(&git, permissions).unwrap(); + } + + let (mut worker, db, _repos) = test_worker(pool, git, 1).await; + worker.repo_store = worker.repo_store.with_lock_hold_timeout(Duration::ZERO); + let target = add_inbound_mirror( + &worker, + &db, + "00000000-0000-0000-0000-000000000004", + "expiredrecheck", + "https://8.8.8.8/org/expired.git", + ) + .await; + let (owner_did, name) = (target.owner_did.clone(), target.name.clone()); + + let error = worker.fetch_target(target).await.unwrap_err(); + assert_eq!( + error.to_string(), + "upstream mirror fetch: the lock-hold deadline expired before the stale-target recheck" + ); + assert!( + !git_marker.exists(), + "no git child may run once the lock-hold deadline has expired" + ); + assert!(worker.repo_write_leases.is_empty()); + + // The refusal released the advisory write lock: a fresh acquire on + // the same store takes it at once instead of waiting it out. + let reacquired = tokio::time::timeout( + Duration::from_secs(5), + worker.repo_store.acquire_write(&owner_did, &name), + ) + .await + .expect("the write lock must be free after the recheck refused") + .expect("acquire after the refused recheck"); + let _ = reacquired.release(false).await; + } } diff --git a/crates/twigpine-node/tests/inv22_gates.rs b/crates/twigpine-node/tests/inv22_gates.rs index 7a2f4694..59651cfd 100644 --- a/crates/twigpine-node/tests/inv22_gates.rs +++ b/crates/twigpine-node/tests/inv22_gates.rs @@ -946,21 +946,23 @@ fn issue_26_namespace_gate_and_refusal_wiring() { ); // P1 (cont): every explicit pre-git refusal must call refuse_receive_pack_before_git - // to terminalize the intent. Exactly 6 occurrences: the five inline call + // to terminalize the intent. Exactly 8 occurrences: the seven inline call // sites (lease cap, per-source cap, write pool, acquire timeout, acquire - // error) plus the drop-guard's own pre-git spawn. The post-git guard phase - // calls mark_receive_pack_interrupted (a different name), so deleting any - // one inline site drops the count to 5 and turns this red. The parse and - // namespace refusals correctly carry no call: they return before the - // intent insert, so there is no aggregate to terminalize. + // error, and the two expired lock-hold deadline sheds, one before the + // tree is handed out and one at the receive-pack re-clamp) plus the + // drop-guard's own pre-git spawn. The post-git guard phase calls mark_receive_pack_interrupted (a + // different name), so deleting any one inline site drops the count to 7 + // and turns this red. The parse and namespace refusals correctly carry no + // call: they return before the intent insert, so there is no aggregate to + // terminalize. assert_eq!( repos.matches("refuse_receive_pack_before_git(").count(), - 6, - "P1 wiring: exactly 6 refuse_receive_pack_before_git calls expected (5 \ + 8, + "P1 wiring: exactly 8 refuse_receive_pack_before_git calls expected (7 \ inline refusal sites + guard pre-git spawn); deleting any inline site \ must turn this red" ); - // Pin the five inline sites by their reason literals, so a site that stops + // Pin the seven inline sites by their reason literals, so a site that stops // terminalizing (or is renumbered away) fails by name, not just by count. for reason in [ "repo write-lease waiter cap reached", @@ -968,6 +970,7 @@ fn issue_26_namespace_gate_and_refusal_wiring() { "git write pool saturated", "acquire_write timed out", "acquire_write failed", + "write-lock hold deadline expired before git", "handler dropped before git started", ] { assert!(