Skip to content

Cloud Agent loses uncommitted work on every resume after ~5 min idle (rebuild costs time + credits) #5901

Description

@maphew

Summary

Cloud Agent destroys the shared worktree/sandbox after ~5 minutes of idle and does not persist uncommitted filesystem state. The next message silently rebuilds the environment and the agent has to re-establish and redo its work. For users who return to a thread after more than a few minutes, every resume becomes a paid re-do.

Root cause

  • Control-plane idle stop = 5 min: DEADLINE_MS.idleStop = 5 * 60_000 in services/cloud-agent-next/src/sandbox-control/deadlines.ts:22, fired via beginStop('idle') -> deps.destroy() (Cloudflare) / stopSession() (Vercel).
  • Stacked on top: 15-min container sleep (SANDBOX_SLEEP_AFTER_SECONDS = 900, src/core/lease.ts:25) and 15-min Kilo server idle timeout (KILO_SERVER_IDLE_TIMEOUT_MS_DEFAULT, CloudAgentSession.ts:211). The 5-min control-plane stop is the binding constraint.
  • No mechanism persists uncommitted files. Only the transcript (DO events) and a 24-hour R2 backup of the prepared workspace (clone + setup only, WORKSPACE_BACKUP_TTL_MS, src/workspace-backup-cache.ts:10) survive. All edits, untracked files, installed dependencies, and build artifacts are lost.
  • This is documented, not a regression: .specs/cloud-agent-session.md:299 — "Uncommitted files are not guaranteed to survive replacement of the shared physical environment."

User impact

  • Returning after >5 minutes always rebuilds. Reporter: "this probably happens on 75% or more of my threads since I almost never return to a thread in less than 15 min" — i.e. ~100% of resumed threads.
  • Per loss-and-resume cycle:
    • Wall clock: ~1–3.5 min before the first token (startup deadline 2 min + wrapper readiness 90 s, deadlines.ts), vs ~0 if the sandbox survived.
    • Inference: the agent must redo work — 1–3 extra turns (git status/diff, re-read files, re-apply edits, re-run builds/tests). Roughly 150–250k extra input tokens + a few thousand output tokens per recovery → ~$0.5–$3 of inference credits per resume (model-dependent), multiplied across every resumed thread every day.
    • Non-determinism/quality: the redo can diverge from the original work.
    • Container time: a few cents (metered per second via cloud_billing_sku.rate_cents_per_unit), negligible next to inference.

Where exact numbers live

  • Container time: container_usage_interval joined to cloud_billing_sku.rate_cents_per_unit (cents/second); idle-stop closures recorded as close_reason = 'activity_expired'.
  • Inference: session-ingest metrics already aggregate totalTokens.{input,output,reasoning,cacheRead,cacheWrite}, totalCost, totalTurns, totalSteps, compactionCount → persisted as cli_sessions_v2.total_cost_microdollars.

Proposed fixes

  1. Persist uncommitted state across idle stop (snapshot/tarball the worktree on stop and restore on rebuild) instead of only backing up the freshly-cloned prepared workspace.
  2. Lengthen / make the idle stop configurable per session, plan, or org (idleStop, SANDBOX_SLEEP_AFTER_SECONDS, KILO_SERVER_IDLE_TIMEOUT_MS).
  3. Auto-commit / auto-stash on idle so work survives in git (cheap, imperfect for untracked files).
  4. At minimum, surface the imminent stop so users know uncommitted work will be lost.

Related

  • .specs/cloud-agent-session.md (Persistence) — the current non-guarantee.
  • Kilo-Org/cloud#4393 — adjacent loss mechanism (hourly GitHub token expiry stranding work on 401).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions