Skip to content

Stream uploads straight to S3 instead of staging them on disk - #251

Open
s3rot0n1n wants to merge 6 commits into
mainfrom
feat/stream-uploads-250
Open

s3rot0n1n wants to merge 6 commits into
mainfrom
feat/stream-uploads-250

Conversation

@s3rot0n1n

@s3rot0n1n s3rot0n1n commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Closes #250. Also supersedes #163, removes the causes of #202, and includes the "share post-processing between tus and form-data" half of #160.

What changes

Uploads no longer go to the API server's disk first. They stream straight into an S3 multipart upload at their final key, {prefix}/i/{mediaId}/main.{ext}.

  • Form-data (POST /media/create): busboy streams the file field into @aws-sdk/lib-storage. express-fileupload is removed. At most part size × concurrency is held in memory per upload.

  • tus (/media/create/resumable): @tus/s3-store replaces @tus/file-store. The upload ID is the object key, base64url-encoded in URLs and checked against the key pattern. tus keeps its resume bookkeeping in a .info object next to the file, deleted when the upload finishes. Any API instance can resume an upload.

  • One shared finish step (apps/api/src/media/ingest.ts) for form-data, tus and MCP:

    • converts images to WebP when the app asks for it, replacing the original;
    • rechecks storage against the stored size;
    • makes thumbnails, reading videos through a presigned URL so ffmpeg fetches only what it needs;
    • tags objects (respects DISABLE_TAGGING);
    • creates the media record.

    If any step but the thumbnail fails, everything it stored is deleted.

  • Size and quota are checked when an upload is created, enforced while it streams, and checked again when it finishes. tus uploads without Upload-Length are rejected.

  • The expiry job aborts the multipart upload of an unfinished tus upload.

  • Migration 0009: tus_uploads gains media_id and drops upload_offset and temp_file_path.

  • Crash safety. Every upload (form-data, tus, MCP) writes its temp media row when it starts, with uploaded = false and its declared size, before any byte reaches the bucket. The finish step completes the row; a rejected upload deletes it. Unfinished rows are hidden from lookups, lists and sealing, but count toward storage, so concurrent uploads reserve quota up front. The temp cleanup job aborts the multipart upload of an expired unfinished row and deletes its folder, so a server crash mid-upload leaves nothing behind. Multipart uploads are aborted by exact key, because MinIO only lists them when the prefix is the full key.

  • Stale uploads. Form-data uploads refresh their row every 5 minutes while streaming; one with no activity for UPLOAD_STALE_MINUTES (30) is released by a sweep every 15 minutes (multipart aborted, objects deleted, reserved storage freed). tus uploads stay resumable until their tus row expires, and neither the sweep nor the 24-hour temp expiry touches them before then.

  • Long uploads. No overall request timeout; a connection closes after REQUEST_IDLE_TIMEOUT_MS (2 min) with no data. Part concurrency defaults to 2, and MAX_CONCURRENT_FORM_UPLOADS (16) answers 503 with Retry-After past the cap. @medialit/uploader sends 48 MiB chunks by default and retries 408 and 429.

  • Migration 0010: media gains uploaded (default true, so existing rows count as finished).

  • New settings UPLOAD_PART_SIZE_BYTES (8 MiB) and UPLOAD_PART_CONCURRENCY (4), in .env.example. The self-hosting docs now cover a bucket lifecycle rule for incomplete multipart uploads.

API responses keep the same shape, including thumbnails.

Differences from the issue

  • The S3 store completes the multipart upload itself, so the final storage check runs right after completion and deletes the object if it fails, rather than before CompleteMultipartUpload. The outcome is the same: nothing is kept and no media record is created.
  • Errors when finishing a tus upload still return 403, as before.

Testing

  • API unit tests: 172 pass. New tests cover the streaming receiver (including one that drives the real lib-storage uploader and checks it sends AbortMultipartUpload when a file goes over the limit), tus ID encoding, deferred-length rejection and the tus create path.
  • bun run test:integration:stack (Compose stack from docker-compose.local.yml): REST, MCP and CLI suites pass.

Browser smoke tests

Run on the dashboard against AWS S3 (the QA buckets) and against MinIO:

  • Dashboard uploads (tus): PNG, JPEG, a 22 MB MP4 with its index at the end of the file, a 20 MiB binary and a Unicode filename. Sizes and content types match, thumbnails appear, and no .info objects or unfinished multipart uploads are left.
  • Form-data with a signature: works, including fields sent after the file. A form with no file returns 400.
  • WebP on: only main.webp is kept, and size is the WebP file's size, on both paths.

MinIO only:

  • Resume on another instance: an upload started on one API instance finished on a second one.
  • Upload size limit: refused at create on both paths. A chunked form upload with no Content-Length is cut off.
  • Client disconnects and abandoned tus uploads leave nothing behind once the expiry job runs.
  • Storage limit: an upload is refused when storage runs out before it finishes, and new uploads are refused when storage is full.

Crash test on the Compose stack: two slow uploads (form-data and tus) with the API killed by SIGKILL mid-upload left two unfinished rows and two unfinished multipart uploads; the temp and tus expiry jobs then removed all of it, leaving nothing in the bucket.

Load test on QA (from a home connection, up to ~104 Mbit/s)

  • 4 × 256 MiB and 12 × 64 MiB concurrent form-data uploads: all passed, exact sizes, response 4–7 s after the last byte, API pod peaked at 495 MiB and 623 MiB.
  • 24 × 64 MiB concurrent form-data uploads: the 1 GiB QA pod was OOM-killed. Each in-flight form-data upload holds parts in memory (roughly 40–45 MiB each in practice).
  • 4 × 1 GiB concurrent tus uploads: requests longer than 5 minutes got a 408 from the server's default requestTimeout (it predates this PR). Fixed here; see below.

Load test on QA after the fixes (pr-251-aeefee3, files at the 2 GiB Pro limit)

  • tus, 3 × 2 GiB, each a single 23.5-minute request: all passed, exact sizes, pod ≤ 230 MiB.
  • form-data, 3 × 2 GiB, ~9 minutes each: all passed, exact sizes, ~91 Mbit/s total, pod ≤ 581 MiB.
  • tus, 4 × 2 GiB in 48 MiB chunks: 3 passed; the 4th failed on the load generator's DNS, not the API.
  • form-data, 40 × 64 MiB at once: 15 passed, 25 got 503, pod ≤ 717 MiB, no OOM (24 uploads OOM-killed it before).
  • 2 GiB + 1 byte: tus refused at create (413); form-data refused with 400 at exactly the limit after 8 minutes of streaming.

Crash test for the stale-upload sweep: with the API killed mid-upload and the rows idle for 45 minutes, the sweep released the form-data upload completely and kept the tus one, which then resumed from its last part after a restart and completed.

The smoke run found one bug, fixed here: a form-data upload cut off at the size limit or by a disconnect left an unfinished multipart upload behind.

Rollout notes

  • In-flight tus uploads stored on disk can't resume after deploy. Clients get a 404 and tus-js-client starts a fresh upload. Delete the old $TEMP_FILE_DIR_FOR_UPLOADS/tus-uploads folder by hand.
  • Migration 0009 drops columns the old code writes, so during a rolling deploy old instances can't create tus uploads once it has run.
  • Migration 0010 adds a column with a default, so it is safe during a rolling deploy.
  • AWS: add a lifecycle rule to the private bucket that aborts incomplete multipart uploads after 1 day.
  • Not tested: Cloudflare R2 and Backblaze B2. R2 needs equal part sizes, which minPartSize enforces.

🤖 Generated with Claude Code

Form-data uploads stream through busboy into an S3 multipart upload, and
tus uploads use @tus/s3-store with the object's final key as the upload
ID. Both paths, and MCP uploads, finish through one shared step that
converts to WebP, rechecks storage against the stored size, makes
thumbnails (videos through a presigned URL) and creates the media record.

- Media ID and key are chosen when an upload starts; nothing is copied
  once it finishes.
- With WebP on, the original is replaced and size is the WebP's size.
- tus uploads must declare Upload-Length; expired ones have their
  multipart upload aborted.
- tus_uploads gains media_id and drops upload_offset and temp_file_path.
- Part size and concurrency are configurable.

Closes #250

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-07T11:37:10.128055Z 3f1e412 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f1e412feb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/api/src/media/ingest.ts Outdated
Comment on lines +108 to +109
const workDir = await mkdtemp(
path.join(tempFileDirForUploads || os.tmpdir(), "medialit-finalize-"),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Create the configured temp directory before using it

When TEMP_FILE_DIR_FOR_UPLOADS points to a directory that does not exist, as the provided /tmp/medialit example does in a fresh container, mkdtemp fails because it only creates the final directory. This runs for every upload and before the cleanup try block, so form, MCP, and tus uploads fail after transfer and can leave the completed S3 object behind. Ensure the configured parent exists before calling mkdtemp.

Useful? React with 👍 / 👎.

Comment thread apps/api/src/media/multipart-upload.ts Outdated
Bucket: cloudBucket,
}).done();
// Handled in "close" or fail(); avoid an unhandled rejection.
uploadDone.catch(() => {});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fail the request when the S3 upload rejects

If Upload.done() rejects before a large request body has been consumed, for example because S3 rejects multipart creation or a part, this handler swallows the rejection and only calls fail() after parser.close. With no consumer draining body, backpressure can stall Busboy before it closes, leaving the request hanging instead of returning the error and discarding the upload. Route the rejection into fail() promptly.

Useful? React with 👍 / 👎.

Rajat and others added 5 commits November 7, 2026 18:20
A form-data upload kept its media ID only in memory, so a server crash
mid-upload left an unfinished multipart upload, or a finished object
with no record, that nothing would ever clean up.

Every upload (form-data, tus, MCP) now writes its temp media row when it
starts, with a new `uploaded` flag set to false and its declared size,
before any byte reaches the bucket. The finish step completes that row;
a rejected upload deletes it. Unfinished rows are hidden from lookups,
lists and sealing, but count toward storage, so concurrent uploads
reserve their quota up front.

The temp cleanup job now also aborts the multipart upload of an
unfinished row, by exact key since MinIO only lists multipart uploads
whose key matches the prefix exactly.

Also fixes a crash this change would have introduced: a file going over
the limit before streaming started destroyed an unheard body stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Load testing on QA found two limits:

- Any request still running after five minutes got 408 from the
  server's default requestTimeout, so big uploads on slow links failed
  however steadily data flowed. The API now has no overall request
  timeout and instead closes a connection after REQUEST_IDLE_TIMEOUT_MS
  (2 minutes) with no data; headersTimeout still guards slow headers.
- 24 concurrent form uploads ran a 1 GiB pod out of memory and took
  every upload with it. Part concurrency now defaults to 2, and
  MAX_CONCURRENT_FORM_UPLOADS (16) answers 503 with Retry-After past the
  cap.

@medialit/uploader now sends files in 48 MiB chunks by default, so each
request is short and an interrupted upload resumes from the last chunk,
and it retries 408 and 429 as well as tus-js-client's defaults.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unfinished upload reserves its declared size from the moment it
starts. After a server crash that reservation stayed until the 24-hour
temp expiry, so an account could look full for a day.

Uploads are now judged by activity, not age:

- A form-data upload refreshes its row's updated_at every 5 minutes
  while it streams. One with no activity for UPLOAD_STALE_MINUTES (30)
  can only have been lost, and a sweep every 15 minutes aborts its
  multipart upload, deletes its objects and frees its storage.
- A tus upload stays resumable while its tus row is unexpired, so the
  sweep and the temp expiry leave it alone; the tus cleanup removes it
  when it expires. This also stops the 24-hour temp expiry from cutting
  short a tus upload with a longer resume window.

The sweep deletes the row before any object, and only while it is still
unfinished, so an upload that completes meanwhile keeps its files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Fail a form-data upload as soon as the bucket rejects it. Waiting for
  the form to end hung the request: nothing drained the body, so busboy
  stalled until the idle timeout.
- Create TEMP_FILE_DIR_FOR_UPLOADS before finalizing an upload, and do
  it inside the cleanup scope, so a missing directory no longer fails
  every upload and leaves its object behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main already runs Bun 1.4.4 in the managed workflow and next 16.4.5,
which platform-cli 0.3.0 rejects. 0.8.2 ships the Bun 1.4.4 workflow and
treats external pins as minimums.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream uploads straight to S3 instead of staging them on disk

1 participant