Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions apps/docs/content/docs/workflows/blocks/function.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -427,9 +427,9 @@ const chunk = await sim.files.readTextChunk(file, { offset: 0, length: 1024 * 10
const bytes = await sim.files.readBase64Chunk(file, { offset: 0, length: 1024 * 1024 });
```

`sim.files.readText`, `readBase64`, and the `…Chunk` variants stream from execution storage under memory caps. `sim.values.read(ref)` and `sim.values.readArray(ref)` read large value and array references. Chunk `offset` and `length` are byte-based, so for exact Unicode parsing prefer smaller structured references. For large generated data in a remote Function block, write files under `/tmp/sim/outputs` and pass them to a compatible File or Table action. The separate Function tool exposes `outputPath`, `outputSandboxPath`, and `outputTable` for explicit persistence; those are not builder fields.
`sim.files` reads a file of up to 64 MiB and returns at most 10 MiB per call, so read larger files in chunks. `sim.values.read(ref)` and `sim.values.readArray(ref)` read large value and array references. Chunk `offset` and `length` are byte-based, so for exact Unicode parsing prefer smaller structured references. For large generated data in a remote Function block, write files under `/tmp/sim/outputs` and pass them to a compatible File or Table action. The separate Function tool exposes `outputPath`, `outputSandboxPath`, and `outputTable` for explicit persistence; those are not builder fields.

The lazy `sim.files` and `sim.values` helpers are available only in JavaScript functions without imports. JavaScript with imports, Python, and Shell do not support them yet.
The lazy `sim.files` and `sim.values` helpers are available only in JavaScript functions without imports. JavaScript with imports, Python, and Shell do not support them yet. See [Large data](/workflows/large-data) for every size limit.

## Errors and timeout

Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/docs/workflows/blocks/loop.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Results group the inner block outputs by iteration. For a single Agent in each i
]
```

Use `<processreviews.results[0][0].content>` for the first iteration's first block result. Additional blocks can contribute additional outputs per iteration.
Use `<processreviews.results[0][0].content>` for the first iteration's first block result. Additional blocks can contribute additional outputs per iteration, so results grow with every block inside the loop; see [keep loop results small](/workflows/large-data#keep-loop-results-small).

## Behavior and limits

Expand Down
2 changes: 1 addition & 1 deletion apps/docs/content/docs/workflows/blocks/parallel.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Arrays yield their values as `currentItem`. Objects yield `[key, value]` pairs,

After completion, use the container's normalized name: `<processtasks.results>`. Generic `<parallel.*>` references select an enclosing parallel context, not a completed container outside it.

Results retain input/instance order, even when branches finish out of order. Each entry contains that branch's block outputs. For a single API per branch, read a field with `<processtasks.results[10][0].data.id>`. Indexed paths hydrate oversized entries only when needed.
Results retain input/instance order, even when branches finish out of order. Each entry contains that branch's block outputs. For a single API per branch, read a field with `<processtasks.results[10][0].data.id>`. Indexed paths hydrate oversized entries only when needed. See [keep loop results small](/workflows/large-data#keep-loop-results-small).

## State and failures

Expand Down
50 changes: 50 additions & 0 deletions apps/docs/content/docs/workflows/large-data.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: Large data
description: Pass large outputs, loop results, and files between blocks within Sim's size limits.
---

Sim stores block outputs larger than 8 MiB with the run and loads them only when a block reads them. Read the fields a block needs rather than a whole large output, and keep loop results small.

## Size limits

| Limit | Size |
| --- | --- |
| One stored value | 64 MiB |
| One stored value loaded by a block at once | 16 MiB |
| One file read by Function code | 64 MiB |
| One `sim.files` read or chunk | 10 MiB |

Sim stores large fields and array chunks as separate values, so a whole block output can exceed 64 MiB when no single piece does. A block that exceeds a limit fails with an `Execution memory limit exceeded` error that names the size and the limit.

## Reference fields, not whole outputs

Use `<api.data.items[0].id>` instead of `<api.data>`. Sim stores large arrays in chunks, so an indexed path loads only the chunk it reads. A field inside a large object still loads that whole object, so the object must fit the 16 MiB limit; when it does not, have the block that produces it return less.

Only JavaScript Function code without imports can receive a whole large value; Sim loads it when the code runs, up to 16 MiB. Agent, API, and other block inputs, Python and Shell code, and JavaScript with imports must reference a smaller field. A reference to an object that contains a large value fails with `contains nested large values`; reference the nested field instead.

An Agent prompt is also bound by its model's context window. Pass a summary or the fields the model needs, not a full dataset.

## Keep loop results small

A Loop or Parallel `results` value holds every block output from every iteration. It grows with both the iteration count and the number of blocks inside, so a loop that reads a table on each pass stores a copy of those rows for every iteration.

- Read data that does not change between iterations once, before the loop.
- Read one entry with an indexed path, such as `<processreviews.results[0][0].content>`. Indexed paths load only that entry.
- To collect a summary from a Loop, have its last block return a compact record and append it to a [workflow variable](/workflows/variables) or a Table row. Read that after the loop instead of `results`. In a Parallel, write each branch's record to a Table rather than a shared variable.

## Large files

A file output is metadata: `<file.name>` and `<file.url>` do not load its contents. In JavaScript Function code without imports, read contents with the [`sim.files` helpers](/workflows/blocks/function#large-inputs). For files over 10 MiB, read in chunks with `readTextChunk` or `readBase64Chunk`.

## Keep reusable datasets in a Table

For data that several blocks or runs need, write it to a [Table](/tables) and query the rows each step needs. Pass row IDs or filters between blocks instead of the rows themselves.

## Related docs

- [How blocks pass data](/workflows/data-flow)
- [Connection syntax](/workflows/connections)
- [Loop](/workflows/blocks/loop)
- [Parallel](/workflows/blocks/parallel)
- [Function](/workflows/blocks/function)
- [Tables in workflows](/tables/using-in-workflows)
1 change: 1 addition & 0 deletions apps/docs/content/docs/workflows/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
"how-it-runs",
"connections",
"variables",
"large-data",
"code",
"workspace-code",
"deployment",
Expand Down
1 change: 1 addition & 0 deletions apps/sim/lib/mothership/generated/docs-manifest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,7 @@ export const DOCS_MANIFEST: readonly string[] = [
'workflows/deployment/chat.mdx',
'workflows/deployment/mcp.mdx',
'workflows/how-it-runs.mdx',
'workflows/large-data.mdx',
'workflows/triggers/rss.mdx',
'workflows/triggers/schedule.mdx',
'workflows/triggers/sim.mdx',
Expand Down
Loading