Repository navigation
Conversation
Form-data uploads stream through busboy into an S3 multipart upload, and tus uploads use @tus/s3-store with the object's final key as the upload ID. Both paths, and MCP uploads, finish through one shared step that converts to WebP, rechecks storage against the stored size, makes thumbnails (videos through a presigned URL) and creates the media record. - Media ID and key are chosen when an upload starts; nothing is copied once it finishes. - With WebP on, the original is replaced and size is the WebP's size. - tus uploads must declare Upload-Length; expired ones have their multipart upload aborted. - tus_uploads gains media_id and drops upload_offset and temp_file_path. - Part size and concurrency are configurable. Closes #250 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f1e412feb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const workDir = await mkdtemp( | ||
| path.join(tempFileDirForUploads || os.tmpdir(), "medialit-finalize-"), |
There was a problem hiding this comment.
Create the configured temp directory before using it
When TEMP_FILE_DIR_FOR_UPLOADS points to a directory that does not exist, as the provided /tmp/medialit example does in a fresh container, mkdtemp fails because it only creates the final directory. This runs for every upload and before the cleanup try block, so form, MCP, and tus uploads fail after transfer and can leave the completed S3 object behind. Ensure the configured parent exists before calling mkdtemp.
Useful? React with 👍 / 👎.
| Bucket: cloudBucket, | ||
| }).done(); | ||
| // Handled in "close" or fail(); avoid an unhandled rejection. | ||
| uploadDone.catch(() => {}); |
There was a problem hiding this comment.
Fail the request when the S3 upload rejects
If Upload.done() rejects before a large request body has been consumed, for example because S3 rejects multipart creation or a part, this handler swallows the rejection and only calls fail() after parser.close. With no consumer draining body, backpressure can stall Busboy before it closes, leaving the request hanging instead of returning the error and discarding the upload. Route the rejection into fail() promptly.
Useful? React with 👍 / 👎.
A form-data upload kept its media ID only in memory, so a server crash mid-upload left an unfinished multipart upload, or a finished object with no record, that nothing would ever clean up. Every upload (form-data, tus, MCP) now writes its temp media row when it starts, with a new `uploaded` flag set to false and its declared size, before any byte reaches the bucket. The finish step completes that row; a rejected upload deletes it. Unfinished rows are hidden from lookups, lists and sealing, but count toward storage, so concurrent uploads reserve their quota up front. The temp cleanup job now also aborts the multipart upload of an unfinished row, by exact key since MinIO only lists multipart uploads whose key matches the prefix exactly. Also fixes a crash this change would have introduced: a file going over the limit before streaming started destroyed an unheard body stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Load testing on QA found two limits: - Any request still running after five minutes got 408 from the server's default requestTimeout, so big uploads on slow links failed however steadily data flowed. The API now has no overall request timeout and instead closes a connection after REQUEST_IDLE_TIMEOUT_MS (2 minutes) with no data; headersTimeout still guards slow headers. - 24 concurrent form uploads ran a 1 GiB pod out of memory and took every upload with it. Part concurrency now defaults to 2, and MAX_CONCURRENT_FORM_UPLOADS (16) answers 503 with Retry-After past the cap. @medialit/uploader now sends files in 48 MiB chunks by default, so each request is short and an interrupted upload resumes from the last chunk, and it retries 408 and 429 as well as tus-js-client's defaults. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unfinished upload reserves its declared size from the moment it starts. After a server crash that reservation stayed until the 24-hour temp expiry, so an account could look full for a day. Uploads are now judged by activity, not age: - A form-data upload refreshes its row's updated_at every 5 minutes while it streams. One with no activity for UPLOAD_STALE_MINUTES (30) can only have been lost, and a sweep every 15 minutes aborts its multipart upload, deletes its objects and frees its storage. - A tus upload stays resumable while its tus row is unexpired, so the sweep and the temp expiry leave it alone; the tus cleanup removes it when it expires. This also stops the 24-hour temp expiry from cutting short a tus upload with a longer resume window. The sweep deletes the row before any object, and only while it is still unfinished, so an upload that completes meanwhile keeps its files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Fail a form-data upload as soon as the bucket rejects it. Waiting for the form to end hung the request: nothing drained the body, so busboy stalled until the idle timeout. - Create TEMP_FILE_DIR_FOR_UPLOADS before finalizing an upload, and do it inside the cleanup scope, so a missing directory no longer fails every upload and leaves its object behind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main already runs Bun 1.4.4 in the managed workflow and next 16.4.5, which platform-cli 0.3.0 rejects. 0.8.2 ships the Bun 1.4.4 workflow and treats external pins as minimums. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #250. Also supersedes #163, removes the causes of #202, and includes the "share post-processing between tus and form-data" half of #160.
What changes
Uploads no longer go to the API server's disk first. They stream straight into an S3 multipart upload at their final key,
{prefix}/i/{mediaId}/main.{ext}.Form-data (
POST /media/create): busboy streams thefilefield into@aws-sdk/lib-storage.express-fileuploadis removed. At most part size × concurrency is held in memory per upload.tus (
/media/create/resumable):@tus/s3-storereplaces@tus/file-store. The upload ID is the object key, base64url-encoded in URLs and checked against the key pattern. tus keeps its resume bookkeeping in a.infoobject next to the file, deleted when the upload finishes. Any API instance can resume an upload.One shared finish step (
apps/api/src/media/ingest.ts) for form-data, tus and MCP:DISABLE_TAGGING);If any step but the thumbnail fails, everything it stored is deleted.
Size and quota are checked when an upload is created, enforced while it streams, and checked again when it finishes. tus uploads without
Upload-Lengthare rejected.The expiry job aborts the multipart upload of an unfinished tus upload.
Migration 0009:
tus_uploadsgainsmedia_idand dropsupload_offsetandtemp_file_path.Crash safety. Every upload (form-data, tus, MCP) writes its temp media row when it starts, with
uploaded = falseand its declared size, before any byte reaches the bucket. The finish step completes the row; a rejected upload deletes it. Unfinished rows are hidden from lookups, lists and sealing, but count toward storage, so concurrent uploads reserve quota up front. The temp cleanup job aborts the multipart upload of an expired unfinished row and deletes its folder, so a server crash mid-upload leaves nothing behind. Multipart uploads are aborted by exact key, because MinIO only lists them when the prefix is the full key.Stale uploads. Form-data uploads refresh their row every 5 minutes while streaming; one with no activity for
UPLOAD_STALE_MINUTES(30) is released by a sweep every 15 minutes (multipart aborted, objects deleted, reserved storage freed). tus uploads stay resumable until their tus row expires, and neither the sweep nor the 24-hour temp expiry touches them before then.Long uploads. No overall request timeout; a connection closes after
REQUEST_IDLE_TIMEOUT_MS(2 min) with no data. Part concurrency defaults to 2, andMAX_CONCURRENT_FORM_UPLOADS(16) answers 503 withRetry-Afterpast the cap.@medialit/uploadersends 48 MiB chunks by default and retries 408 and 429.Migration 0010:
mediagainsuploaded(defaulttrue, so existing rows count as finished).New settings
UPLOAD_PART_SIZE_BYTES(8 MiB) andUPLOAD_PART_CONCURRENCY(4), in.env.example. The self-hosting docs now cover a bucket lifecycle rule for incomplete multipart uploads.API responses keep the same shape, including thumbnails.
Differences from the issue
CompleteMultipartUpload. The outcome is the same: nothing is kept and no media record is created.Testing
lib-storageuploader and checks it sendsAbortMultipartUploadwhen a file goes over the limit), tus ID encoding, deferred-length rejection and the tus create path.bun run test:integration:stack(Compose stack fromdocker-compose.local.yml): REST, MCP and CLI suites pass.Browser smoke tests
Run on the dashboard against AWS S3 (the QA buckets) and against MinIO:
.infoobjects or unfinished multipart uploads are left.main.webpis kept, andsizeis the WebP file's size, on both paths.MinIO only:
Content-Lengthis cut off.Crash test on the Compose stack: two slow uploads (form-data and tus) with the API killed by
SIGKILLmid-upload left two unfinished rows and two unfinished multipart uploads; the temp and tus expiry jobs then removed all of it, leaving nothing in the bucket.Load test on QA (from a home connection, up to ~104 Mbit/s)
requestTimeout(it predates this PR). Fixed here; see below.Load test on QA after the fixes (
pr-251-aeefee3, files at the 2 GiB Pro limit)Crash test for the stale-upload sweep: with the API killed mid-upload and the rows idle for 45 minutes, the sweep released the form-data upload completely and kept the tus one, which then resumed from its last part after a restart and completed.
The smoke run found one bug, fixed here: a form-data upload cut off at the size limit or by a disconnect left an unfinished multipart upload behind.
Rollout notes
tus-js-clientstarts a fresh upload. Delete the old$TEMP_FILE_DIR_FOR_UPLOADS/tus-uploadsfolder by hand.minPartSizeenforces.🤖 Generated with Claude Code