diff --git a/apps/docs/content/docs/workflows/blocks/function.mdx b/apps/docs/content/docs/workflows/blocks/function.mdx index 64c4a02f3e5..ef5d27b755b 100644 --- a/apps/docs/content/docs/workflows/blocks/function.mdx +++ b/apps/docs/content/docs/workflows/blocks/function.mdx @@ -427,9 +427,9 @@ const chunk = await sim.files.readTextChunk(file, { offset: 0, length: 1024 * 10 const bytes = await sim.files.readBase64Chunk(file, { offset: 0, length: 1024 * 1024 }); ``` -`sim.files.readText`, `readBase64`, and the `…Chunk` variants stream from execution storage under memory caps. `sim.values.read(ref)` and `sim.values.readArray(ref)` read large value and array references. Chunk `offset` and `length` are byte-based, so for exact Unicode parsing prefer smaller structured references. For large generated data in a remote Function block, write files under `/tmp/sim/outputs` and pass them to a compatible File or Table action. The separate Function tool exposes `outputPath`, `outputSandboxPath`, and `outputTable` for explicit persistence; those are not builder fields. +`sim.files` reads a file of up to 64 MiB and returns at most 10 MiB per call, so read larger files in chunks. `sim.values.read(ref)` and `sim.values.readArray(ref)` read large value and array references. Chunk `offset` and `length` are byte-based, so for exact Unicode parsing prefer smaller structured references. For large generated data in a remote Function block, write files under `/tmp/sim/outputs` and pass them to a compatible File or Table action. The separate Function tool exposes `outputPath`, `outputSandboxPath`, and `outputTable` for explicit persistence; those are not builder fields. -The lazy `sim.files` and `sim.values` helpers are available only in JavaScript functions without imports. JavaScript with imports, Python, and Shell do not support them yet. +The lazy `sim.files` and `sim.values` helpers are available only in JavaScript functions without imports. JavaScript with imports, Python, and Shell do not support them yet. See [Large data](/workflows/large-data) for every size limit. ## Errors and timeout diff --git a/apps/docs/content/docs/workflows/blocks/loop.mdx b/apps/docs/content/docs/workflows/blocks/loop.mdx index dc3837424c4..959b4815746 100644 --- a/apps/docs/content/docs/workflows/blocks/loop.mdx +++ b/apps/docs/content/docs/workflows/blocks/loop.mdx @@ -53,7 +53,7 @@ Results group the inner block outputs by iteration. For a single Agent in each i ] ``` -Use `` for the first iteration's first block result. Additional blocks can contribute additional outputs per iteration. +Use `` for the first iteration's first block result. Additional blocks can contribute additional outputs per iteration, so results grow with every block inside the loop; see [keep loop results small](/workflows/large-data#keep-loop-results-small). ## Behavior and limits diff --git a/apps/docs/content/docs/workflows/blocks/parallel.mdx b/apps/docs/content/docs/workflows/blocks/parallel.mdx index 90dd0605266..81dbcaa6a63 100644 --- a/apps/docs/content/docs/workflows/blocks/parallel.mdx +++ b/apps/docs/content/docs/workflows/blocks/parallel.mdx @@ -38,7 +38,7 @@ Arrays yield their values as `currentItem`. Objects yield `[key, value]` pairs, After completion, use the container's normalized name: ``. Generic `` references select an enclosing parallel context, not a completed container outside it. -Results retain input/instance order, even when branches finish out of order. Each entry contains that branch's block outputs. For a single API per branch, read a field with ``. Indexed paths hydrate oversized entries only when needed. +Results retain input/instance order, even when branches finish out of order. Each entry contains that branch's block outputs. For a single API per branch, read a field with ``. Indexed paths hydrate oversized entries only when needed. See [keep loop results small](/workflows/large-data#keep-loop-results-small). ## State and failures diff --git a/apps/docs/content/docs/workflows/large-data.mdx b/apps/docs/content/docs/workflows/large-data.mdx new file mode 100644 index 00000000000..33405ddc0cd --- /dev/null +++ b/apps/docs/content/docs/workflows/large-data.mdx @@ -0,0 +1,50 @@ +--- +title: Large data +description: Pass large outputs, loop results, and files between blocks within Sim's size limits. +--- + +Sim stores block outputs larger than 8 MiB with the run and loads them only when a block reads them. Read the fields a block needs rather than a whole large output, and keep loop results small. + +## Size limits + +| Limit | Size | +| --- | --- | +| One stored value | 64 MiB | +| One stored value loaded by a block at once | 16 MiB | +| One file read by Function code | 64 MiB | +| One `sim.files` read or chunk | 10 MiB | + +Sim stores large fields and array chunks as separate values, so a whole block output can exceed 64 MiB when no single piece does. A block that exceeds a limit fails with an `Execution memory limit exceeded` error that names the size and the limit. + +## Reference fields, not whole outputs + +Use `` instead of ``. Sim stores large arrays in chunks, so an indexed path loads only the chunk it reads. A field inside a large object still loads that whole object, so the object must fit the 16 MiB limit; when it does not, have the block that produces it return less. + +Only JavaScript Function code without imports can receive a whole large value; Sim loads it when the code runs, up to 16 MiB. Agent, API, and other block inputs, Python and Shell code, and JavaScript with imports must reference a smaller field. A reference to an object that contains a large value fails with `contains nested large values`; reference the nested field instead. + +An Agent prompt is also bound by its model's context window. Pass a summary or the fields the model needs, not a full dataset. + +## Keep loop results small + +A Loop or Parallel `results` value holds every block output from every iteration. It grows with both the iteration count and the number of blocks inside, so a loop that reads a table on each pass stores a copy of those rows for every iteration. + +- Read data that does not change between iterations once, before the loop. +- Read one entry with an indexed path, such as ``. Indexed paths load only that entry. +- To collect a summary from a Loop, have its last block return a compact record and append it to a [workflow variable](/workflows/variables) or a Table row. Read that after the loop instead of `results`. In a Parallel, write each branch's record to a Table rather than a shared variable. + +## Large files + +A file output is metadata: `` and `` do not load its contents. In JavaScript Function code without imports, read contents with the [`sim.files` helpers](/workflows/blocks/function#large-inputs). For files over 10 MiB, read in chunks with `readTextChunk` or `readBase64Chunk`. + +## Keep reusable datasets in a Table + +For data that several blocks or runs need, write it to a [Table](/tables) and query the rows each step needs. Pass row IDs or filters between blocks instead of the rows themselves. + +## Related docs + +- [How blocks pass data](/workflows/data-flow) +- [Connection syntax](/workflows/connections) +- [Loop](/workflows/blocks/loop) +- [Parallel](/workflows/blocks/parallel) +- [Function](/workflows/blocks/function) +- [Tables in workflows](/tables/using-in-workflows) diff --git a/apps/docs/content/docs/workflows/meta.json b/apps/docs/content/docs/workflows/meta.json index 6c05d4d8842..8762490d5cb 100644 --- a/apps/docs/content/docs/workflows/meta.json +++ b/apps/docs/content/docs/workflows/meta.json @@ -5,6 +5,7 @@ "how-it-runs", "connections", "variables", + "large-data", "code", "workspace-code", "deployment", diff --git a/apps/sim/lib/mothership/generated/docs-manifest.ts b/apps/sim/lib/mothership/generated/docs-manifest.ts index 2f5bf5ebde6..9bdd324f9e9 100644 --- a/apps/sim/lib/mothership/generated/docs-manifest.ts +++ b/apps/sim/lib/mothership/generated/docs-manifest.ts @@ -509,6 +509,7 @@ export const DOCS_MANIFEST: readonly string[] = [ 'workflows/deployment/chat.mdx', 'workflows/deployment/mcp.mdx', 'workflows/how-it-runs.mdx', + 'workflows/large-data.mdx', 'workflows/triggers/rss.mdx', 'workflows/triggers/schedule.mdx', 'workflows/triggers/sim.mdx',