Skip to content

fix(documents): split Yjs state that exceeds the storage value limit - #869

Draft
posthog[bot] wants to merge 2 commits into
mainfrom
posthog-self-driving/fixdocuments-chunk-yjs-state-to-stay-273eeb
Draft

posthog[bot] wants to merge 2 commits into
mainfrom
posthog-self-driving/fixdocuments-chunk-yjs-state-to-stay-273eeb

Conversation

@posthog

@posthog posthog Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • Users with a large document cannot use AI reads on it. Every workspace_read_items call on that document fails with SQLITE_TOOBIG.
  • DocumentSession.persistYDoc writes the full Yjs state as one Durable Object value. SQLite-backed storage rejects a value over 2 MB.
  • Commit d2591d6 removed the chunked write because text-only documents could not reach 2 MB. Widget blocks (4832c6f) can hold large HTML, so documents can now go past the limit.

Warning

onSave uses the same write. When it throws, the Postgres checkpoint does not run, so edits to a large document can stop saving.

Origin

  • Error tracking: issue
  • First signal: 2026-11-15
  • Inbox report: open
  • Task started by: auto-start, after the report was rated P2 and ready to fix

Changes

  • Size-aware persistence for the Yjs key, in document-yjs-persistence.ts:
    • A state up to 1 MB stays one value at document-session:yjs-update. Small documents keep the same storage layout.
    • A larger state goes into 1 MB parts, with a manifest at the same key. Each write deletes the parts of the previous generation in the same transaction.
  • onLoad, persistYDoc, applyEdits and undoDocumentEditReceipt use the new read and write helpers.
  • A new worker test runs against real SQLite Durable Object storage. It shows that a single put of a Yjs state over 2 MB throws SQLITE_TOOBIG. It also shows that the chunked write round-trips that state and removes the old parts when the state gets smaller.
State size Before After
≤ 1 MB One value One value (no change)
1–2 MB One value Manifest + parts
> 2 MB SQLITE_TOOBIG Manifest + parts

Testing

  • vp test --run --project=workers src/features/workspaces/documents/: pass
  • vp test --run --project=node src/features/workspaces/documents/: pass
  • vp check (format, lint, typecheck): pass
  • I did not run the full pnpm verify (ciBuild included).

Agent context

  • I did not add a total-size cap. With a cap, a document above the cap fails in the same way at a higher threshold.
  • Possible follow-up (not in this PR): each write reads the previous value (up to 1 MB), only to find old parts to delete. A separate small manifest key would remove that read.
  • Out of scope: edit receipts still drop beforeDocument above 1.5 MB, so AI-edit undo is not available on very large documents.
  • Existing documents need no migration. They hold a single value under 2 MB, and the reader still accepts that format.

Closes #868


Created with PostHog Desktop from this inbox report.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

SQLite-backed Durable Object storage rejects a value over 2 MB, and widget
blocks can push a document's Yjs state past it. DocumentSession then throws
SQLITE_TOOBIG on every persist, so AI reads fail and onSave skips the
Postgres checkpoint.

States up to 1 MB stay a single value at the update key. Larger states are
split into 1 MB parts behind a manifest at that key, and each write removes
the parts of the previous generation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: 6b295967-a1c2-4c8a-9794-448858ca72f8
@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5a8966b5-af17-46e8-8c0b-ac07dcbb11e4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

React Doctor found no new issues. 🎉

Reviewed by React Doctor for commit 23c0fae.

Only this module writes the manifest, so the version field and the full
manifest guard add no protection. The part count follows from the total
size. The shrink test now checks that the large write created parts first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: 6b295967-a1c2-4c8a-9794-448858ca72f8

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

Error

0 participants