diff --git a/apps/docs/content/docs/cli/reference.mdx b/apps/docs/content/docs/cli/reference.mdx
index c66a09de33b..8957d0e13fe 100644
--- a/apps/docs/content/docs/cli/reference.mdx
+++ b/apps/docs/content/docs/cli/reference.mdx
@@ -8573,16 +8573,18 @@ sim workflows source apply [workflowId] [options]
| Option | Required | Description |
| --- | --- | --- |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
| `--dir
` | Yes | Source directory: workflow.ts, functions/*, .sim/lock.json and .sim/base.json. |
| `--dry-run` | No | Check the source and show the outcome without saving or writing files. |
| `-y, --yes` | No | Apply without asking for confirmation. |
+| `--deploy` | No | After the write is saved, deploy the workflow (as `sim workflows deploy ` does). |
| `--allow-delete` | No | Delete blocks the source no longer declares; without it, an apply that deletes a block is refused. |
-Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); 6 the workflow or its folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again. Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.
+Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); 6 the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it). Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check.
-API request and response schemas: [applyWorkflowSource](/api-reference/workflows/applyWorkflowSource).
+API request and response schemas: [applyWorkflowSource](/api-reference/workflows/applyWorkflowSource), [deployWorkflow](/api-reference/workflows/deployWorkflow).
### sim workflows source create
@@ -8598,17 +8600,19 @@ sim workflows source create [options]
| Option | Required | Description |
| --- | --- | --- |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
| `--dir ` | Yes | Source directory: workflow.ts and functions/*. Without workflow.ts, writes a starter and the types instead (an error with --dry-run or --yes). |
| `--dry-run` | No | Check the source and show the outcome without creating anything. |
| `--name ` | No | The workflow name (default: the source's). |
| `--folder ` | No | The folder to create it in (default: the root). |
| `-y, --yes` | No | Create without asking for confirmation. |
+| `--deploy` | No | After the write is saved, deploy the workflow (as `sim workflows deploy ` does). |
-Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed; 6 the folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again. Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.
+Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the workspace its types were written for unless --workspace names another); 6 the folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was created (run the same command with --yes). Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check.
-API request and response schemas: [createWorkflowFromSource](/api-reference/workflows/createWorkflowFromSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog), [getSourceResources](/api-reference/workflows/getSourceResources).
+API request and response schemas: [createWorkflowFromSource](/api-reference/workflows/createWorkflowFromSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog), [getSourceResources](/api-reference/workflows/getSourceResources), [deployWorkflow](/api-reference/workflows/deployWorkflow).
### sim workflows source check
@@ -8627,6 +8631,7 @@ sim workflows source check [options]
| `--dir ` | No | Source directory: workflow.ts, functions/* and .sim/. Defaults to `.`. |
| `--strict` | No | Fail on the warnings listed below in every block, not only new ones. |
| `--no-strict` | No | Never fail on a warning. |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
@@ -8634,7 +8639,7 @@ Compiles workflow.ts (and the function files it names) with the compiler the ser
Exit codes: 0 no errors; 1 errors (each printed as file:line:column with its frame and hint), or no cached catalog (run the get once). With --output json, stdout is one document that also lists every finding under diagnostics.
-These warnings, which an apply saves as written, fail the check as errors on every block the lock does not bind (a block added since the get, or every block of a new workflow): W_REQUIRED_MISSING, W_UNKNOWN_OPTION, W_UNKNOWN_OUTPUT_PATH, W_VALUE_KIND, W_OUT_OF_RANGE, W_LITERAL_SECRET, W_TABLE_UNKNOWN_COLUMN, W_CHILD_INPUT_UNKNOWN, W_BRANCH_UNWIRED. --strict fails on them in every block, except W_BRANCH_UNWIRED on a block the lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never fails on a warning.
+These warnings, which an apply saves as written, fail the check as errors on every block the lock does not bind (a block added since the get, or every block of a new workflow): W_REQUIRED_MISSING, W_UNKNOWN_OPTION, W_UNKNOWN_OUTPUT_PATH, W_VALUE_KIND, W_OUT_OF_RANGE, W_LITERAL_SECRET, W_TABLE_UNKNOWN_COLUMN, W_CHILD_INPUT_UNKNOWN, W_BRANCH_UNWIRED, W_UNRESOLVED_REF. --strict fails on them in every block, except W_BRANCH_UNWIRED on a block the lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never fails on a warning. W_JSON_REF_UNTYPED is reported only on blocks the lock does not bind, and with --strict on every block.
### sim workflows source status
@@ -8656,7 +8661,7 @@ sim workflows source status [options]
-Exit codes: 0 clean (nothing to apply, and the server changed nothing since the directory was read); 11 local changes only (applying writes them); 12 the server changed since the directory was read: drift (a resource edited, added or deleted there that the directory leaves alone) or a conflict (also changed here, so applying it is refused as stale), with or without local changes; 7 the source is too large for the server to read in one dry run; 1 the command failed. A server that cannot be reached is not a failure: the status shows the local changes only and says so (exit 0 or 11).
+Exit codes: 0 clean (nothing to apply, and the server changed nothing since the directory was read); 11 local changes only (applying writes them); 12 the server changed since the directory was read: drift (a resource edited, added or deleted there that the directory leaves alone) or a conflict (also changed here, so applying it is refused as stale), with or without local changes; 7 the source is too large for the server to read in one dry run; 1 the command failed; 13 not logged in (AUTH_REQUIRED) or the credential was refused: the status shows the local changes only, then run `sim login`. A server that cannot be reached is not a failure: the status shows the local changes only and says so (exit 0 or 11).
API request and response schemas: [applyWorkflowSource](/api-reference/workflows/applyWorkflowSource).
@@ -8694,6 +8699,38 @@ Without --server, each file created, edited or deleted since the directory was r
API request and response schemas: [getWorkflowSource](/api-reference/workflows/getWorkflowSource).
+### sim workflows source params
+
+Print a builder's params and outputs from the cached block catalog (offline)
+
+```bash
+sim workflows source params [options]
+```
+
+**Arguments**
+
+
+
+| Argument | Required | Description |
+| --- | --- | --- |
+| `builder` | Yes | Builder path (table.upsertRow, agent, trigger.schedule) or namespace |
+
+
+
+**Options**
+
+
+
+| Option | Required | Description |
+| --- | --- | --- |
+| `--dir ` | No | Source directory whose catalog to read. Defaults to `.`. |
+
+
+
+Prints a builder's params (required first, with enums and defaults), the gate field that picks which others apply, and its outputs, from the block catalog the last get, create or check cached. No request is made. <builder> is the dotted builder path as source writes it: table.upsertRow, agent, trigger.schedule, slack.sendMessage (slack.tool.sendMessage names the same builder). A namespace (gmail, trigger.gmail) lists its builders; a construct core.d.ts declares (loop, parallel, t, env) prints its declaration. Run it from a workflow or workspace source directory, or pass --dir.
+
+Exit codes: 0 printed; 1 no builder or namespace of that name (the closest one is named), or no cached catalog (run a get once).
+
### sim workflows get
Get workflow
@@ -9941,18 +9978,19 @@ sim workspaces source apply [options]
| Option | Required | Description |
| --- | --- | --- |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
| `--dir ` | Yes | Workspace source directory written by `workspaces source get`. |
| `--dry-run` | No | Plan and check without writing anything. |
| `-y, --yes` | No | Apply without asking for confirmation. |
| `--allow-delete` | No | Delete resources, columns and data the directory no longer declares. |
| `--only ` | No | Apply only these kinds (comma-separated folders: secrets.ts, credentials.ts, tools, skills, mcp, tables, knowledge, files, workflows, deployments). |
-| `--to ` | No | Promote the directory to another workspace (dev to prod): its resources bind by key to that workspace's own and are updated in place; the directory is not changed. |
+| `--to ` | No | Promote the directory to another workspace (dev to prod): its resources bind by key to that workspace's own and are updated in place. Only .sim/targets/<workspaceId>.json changes, the record the next promotion compares with: a resource changed there since is kept, or refused as stale (exit 5) when the directory changed it too. |
-Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint), or changes were held back because they use a declared secret the workspace has no value for or a declared credential key nothing binds to (the rest was applied: set the secret or connect the credential, then apply again), the directory was read from another workspace than the profile's (SOURCE_OTHER_WORKSPACE: pass --workspace to choose, or --to to promote it), its base records binary files no get downloaded (SOURCE_BLOBS_NOT_FETCHED: get again first), or the command failed; 5 a resource changed or was deleted on the server since the directory was read (SOURCE_STALE: get it again into a fresh directory, reapply the edit, retry); 6 a resource is locked; 7 the directory is too large; 8 the apply stopped part-way on a server failure: what was written is saved into the directory, and applying again continues. An apply a resource refused part-way (stale, locked, forbidden) saves what was written too, and exits as that refusal: 5, 6 or 1; 9 rate limited (HTTP 429) after the CLI's retries, before anything was written: wait a minute and apply again (binary files already uploaded are not sent again); 10 another apply is writing the workspace (SOURCE_BUSY), or another command is using the directory (SOURCE_DIR_BUSY): nothing was written, run the command again once the other finished.
+Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint), or changes were held back because they use a declared secret the workspace has no value for or a declared credential key nothing binds to (the rest was applied: set the secret or connect the credential, then apply again), the directory was read from another workspace than the one --workspace names (SOURCE_OTHER_WORKSPACE: drop --workspace, or promote it with --to; without --workspace an apply targets the workspace the directory was read from), its base records binary files no get downloaded (SOURCE_BLOBS_NOT_FETCHED: get again first), or the command failed; 5 a resource changed or was deleted on the server since the directory was read (SOURCE_STALE: get it again into a fresh directory, reapply the edit, retry); 6 a resource (a workflow or its folder, a table) is locked (a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the directory is too large; 8 the apply stopped part-way on a server failure: what was written is saved into the directory, and applying again continues. An apply a resource refused part-way (stale, locked, forbidden) saves what was written too, and exits as that refusal: 5, 6 or 1; 9 rate limited (HTTP 429) after the CLI's retries, before anything was written: wait a minute and apply again (binary files already uploaded are not sent again); 10 another apply is writing the workspace (SOURCE_BUSY), or another command is using the directory (SOURCE_DIR_BUSY): nothing was written, run the command again once the other finished; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it). Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check.
-API request and response schemas: [applyWorkspaceSource](/api-reference/workflows/applyWorkspaceSource), [findMissingWorkspaceSourceBlobs](/api-reference/workflows/findMissingWorkspaceSourceBlobs), [stageWorkspaceSourceBlobs](/api-reference/workflows/stageWorkspaceSourceBlobs), [getWorkspaceSource](/api-reference/workflows/getWorkspaceSource).
+API request and response schemas: [applyWorkspaceSource](/api-reference/workflows/applyWorkspaceSource), [findMissingWorkspaceSourceBlobs](/api-reference/workflows/findMissingWorkspaceSourceBlobs), [stageWorkspaceSourceBlobs](/api-reference/workflows/stageWorkspaceSourceBlobs), [getWorkspaceSource](/api-reference/workflows/getWorkspaceSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog).
### sim workspaces source check
@@ -9971,6 +10009,7 @@ sim workspaces source check [options]
| `--dir ` | No | Workspace source directory: the kind folders and .sim/base.json. Defaults to `.`. |
| `--strict` | No | Fail on the warnings listed below in every block, not only new ones. |
| `--no-strict` | No | Never fail on a warning. |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
@@ -9978,7 +10017,7 @@ Compiles every workflows/<key>/workflow.ts (and the function files each na
Exit codes: 0 no errors; 1 errors (each printed as file:line:column with its frame and hint), or no cached catalog (run the get once). With --output json, stdout is one document that also lists every finding under diagnostics.
-These warnings, which an apply saves as written, fail the check as errors on every block the lock does not bind (a block added since the get, or every block of a new workflow): W_REQUIRED_MISSING, W_UNKNOWN_OPTION, W_UNKNOWN_OUTPUT_PATH, W_VALUE_KIND, W_OUT_OF_RANGE, W_LITERAL_SECRET, W_TABLE_UNKNOWN_COLUMN, W_CHILD_INPUT_UNKNOWN, W_BRANCH_UNWIRED. --strict fails on them in every block, except W_BRANCH_UNWIRED on a block the lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never fails on a warning.
+These warnings, which an apply saves as written, fail the check as errors on every block the lock does not bind (a block added since the get, or every block of a new workflow): W_REQUIRED_MISSING, W_UNKNOWN_OPTION, W_UNKNOWN_OUTPUT_PATH, W_VALUE_KIND, W_OUT_OF_RANGE, W_LITERAL_SECRET, W_TABLE_UNKNOWN_COLUMN, W_CHILD_INPUT_UNKNOWN, W_BRANCH_UNWIRED, W_UNRESOLVED_REF. --strict fails on them in every block, except W_BRANCH_UNWIRED on a block the lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never fails on a warning. W_JSON_REF_UNTYPED is reported only on blocks the lock does not bind, and with --strict on every block.
Exit 10 (busy, retry): another sim workspaces source command is using this directory (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.
@@ -10005,7 +10044,7 @@ sim workspaces source status [options]
-Exit codes: 0 clean (nothing to apply, and the server changed nothing since the directory was read); 11 local changes only (applying writes them); 12 the server changed since the directory was read: drift (a resource edited, added or deleted there that the directory leaves alone) or a conflict (also changed here, so applying it is refused as stale), with or without local changes; 7 the source is too large for the server to read in one dry run; 1 the command failed. A server that cannot be reached is not a failure: the status shows the local changes only and says so (exit 0 or 11).
+Exit codes: 0 clean (nothing to apply, and the server changed nothing since the directory was read); 11 local changes only (applying writes them); 12 the server changed since the directory was read: drift (a resource edited, added or deleted there that the directory leaves alone) or a conflict (also changed here, so applying it is refused as stale), with or without local changes; 7 the source is too large for the server to read in one dry run; 1 the command failed; 13 not logged in (AUTH_REQUIRED) or the credential was refused: the status shows the local changes only, then run `sim login`. A server that cannot be reached is not a failure: the status shows the local changes only and says so (exit 0 or 11).
Exit 10 (busy, retry): another sim workspaces source command is using this directory (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.
diff --git a/apps/docs/content/docs/cli/scripting.mdx b/apps/docs/content/docs/cli/scripting.mdx
index e3e8540d3b0..1214f561331 100644
--- a/apps/docs/content/docs/cli/scripting.mdx
+++ b/apps/docs/content/docs/cli/scripting.mdx
@@ -170,10 +170,22 @@ The CLI resends a request the API answers `429` only when sending it twice is sa
| `0` | Applied or created, or a dry run with no errors |
| `1` | The source has errors (each printed as `file:line:column` with its frame and hint), or command/request failure |
| `5` | The workflow changed since the directory was read (`409 SOURCE_STALE`): get it again, reapply the edit, and retry |
-| `6` | The workflow is locked (`423`) |
+| `6` | The workflow or its folder is locked (`423`, `E_APPLY_LOCKED`); a locked block exits `1` (`E_APPLY_LOCKED_BLOCK`) |
| `7` | The source is over the size limits (`413`) |
| `9` | Still rate limited (`429`) after the CLI's retries; nothing was written |
| `10` | `--dir` is a workflow folder of a workspace directory another `sim` command is using (`SOURCE_DIR_BUSY`); nothing was written: run it again once the other finished |
+| `13` | `AUTH_REQUIRED`: not logged in, or the server refused the credential; nothing was sent |
+| `14` | Run without `--yes` and with no terminal to ask: the plan was printed with no errors and nothing was written; run the same command with `--yes` |
+| `130` | Interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check |
+
+| Code | `sim workflows source status` or `sim workspaces source status` |
+| --- | --- |
+| `0` | Clean: nothing to apply, and the server changed nothing since the directory was read |
+| `1` | Command/request failure |
+| `7` | The source is too large for the server to read in one dry run |
+| `11` | Local changes only: applying writes them |
+| `12` | The server changed since the directory was read (drift, or a conflict with local changes) |
+| `13` | `AUTH_REQUIRED`: not logged in, or the server refused the credential; the local changes are still shown |
| Code | `sim workflows source check` or `sim workspaces source check` |
| --- | --- |
@@ -192,11 +204,13 @@ The CLI resends a request the API answers `429` only when sending it twice is sa
| `0` | Applied, or a dry run with no errors |
| `1` | The source has errors, changes were held back for a secret or credential the workspace cannot satisfy, or command/request failure |
| `5` | A resource changed since the directory was read (`409 SOURCE_STALE`): get it again into a fresh directory, reapply the edit, and retry |
-| `6` | A resource is locked (`423`) |
+| `6` | A resource (a workflow or its folder, a table) is locked (`423`); a locked block exits `1` (`E_APPLY_LOCKED_BLOCK`) |
| `7` | The directory is over the size limits (`413`) |
| `8` | The apply stopped part-way: what was written is saved into the directory; apply again |
| `9` | Still rate limited (`429`) after the CLI's retries; nothing was written |
| `10` | Busy, retry: another apply is writing the workspace (`409 SOURCE_BUSY`), or another `sim workspaces source` command is using the same directory (`SOURCE_DIR_BUSY`); nothing was written: run the command again once the other finished |
+| `13` | `AUTH_REQUIRED`: not logged in, or the server refused the credential; nothing was sent |
+| `130` | Interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check |
Both waits default to 3600 seconds; `--wait-timeout 0` waits indefinitely. `SIM_TIMEOUT_SECONDS` independently bounds one API request. A timeout does not cancel the underlying run or operation.
@@ -403,12 +417,14 @@ jobs:
`check` runs first because it is offline and fast, and it regenerates `.sim/sdk/workspace.d.ts`, so the `tsc` step after it types resources the pull request adds. A workspace API key can apply tables, rows, knowledge bases, files, custom tools, MCP servers and credential bindings. Workflows, skills, secrets and deployments need a member's personal API key, and deployments need an admin. Apply rewrites files under the directory (each `.sim/base.json`, ids it allocated, values the server normalized), and the next apply compares with them, so the last step commits them back to the branch; `[skip ci]` keeps that commit from running the job again. Where the branch is protected, run `sim workspaces source get --dir workspace` on a new branch instead and open a pull request. After merging two branches that each applied, a dry run with no changes means the files match the workspace. Exit `5` means a resource changed since this branch's base: get the workspace into a fresh directory and carry the edit over. Exit `10` means another apply is writing the workspace (or another command is using the directory), so retry; the `concurrency` key keeps two runs of this job from overlapping.
-To promote the same directory to a second workspace, add a step with `--to`:
+To promote the same directory to a second workspace, add a step with `--to` before the step that commits the files back:
```bash
sim workspaces source apply --dir workspace --to "$PROD_WORKSPACE_ID" --yes
```
+The promotion writes `workspace/.sim/targets/.json`, what it left in that workspace. Committing it lets the next run tell a change made in that workspace since (kept, or exit `5` when the directory changed the same resource) from one to promote.
+
## Related docs
- [Output formats](/cli/output)
diff --git a/apps/docs/content/docs/cli/workflows.mdx b/apps/docs/content/docs/cli/workflows.mdx
index 318297758b1..f90bb57ad7e 100644
--- a/apps/docs/content/docs/cli/workflows.mdx
+++ b/apps/docs/content/docs/cli/workflows.mdx
@@ -1337,7 +1337,7 @@ sim workflows source status [options]
-Exit codes: 0 clean (nothing to apply, and the server changed nothing since the directory was read); 11 local changes only (applying writes them); 12 the server changed since the directory was read: drift (a resource edited, added or deleted there that the directory leaves alone) or a conflict (also changed here, so applying it is refused as stale), with or without local changes; 7 the source is too large for the server to read in one dry run; 1 the command failed. A server that cannot be reached is not a failure: the status shows the local changes only and says so (exit 0 or 11).
+Exit codes: 0 clean (nothing to apply, and the server changed nothing since the directory was read); 11 local changes only (applying writes them); 12 the server changed since the directory was read: drift (a resource edited, added or deleted there that the directory leaves alone) or a conflict (also changed here, so applying it is refused as stale), with or without local changes; 7 the source is too large for the server to read in one dry run; 1 the command failed; 13 not logged in (AUTH_REQUIRED) or the credential was refused: the status shows the local changes only, then run `sim login`. A server that cannot be reached is not a failure: the status shows the local changes only and says so (exit 0 or 11).
API request and response schemas: [applyWorkflowSource](/api-reference/workflows/applyWorkflowSource).
@@ -1353,17 +1353,19 @@ sim workflows source create [options]
| Option | Required | Description |
| --- | --- | --- |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
| `--dir ` | Yes | Source directory: workflow.ts and functions/*. Without workflow.ts, writes a starter and the types instead (an error with --dry-run or --yes). |
| `--dry-run` | No | Check the source and show the outcome without creating anything. |
| `--name ` | No | The workflow name (default: the source's). |
| `--folder ` | No | The folder to create it in (default: the root). |
| `-y, --yes` | No | Create without asking for confirmation. |
+| `--deploy` | No | After the write is saved, deploy the workflow (as `sim workflows deploy ` does). |
-Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed; 6 the folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again. Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.
+Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the workspace its types were written for unless --workspace names another); 6 the folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was created (run the same command with --yes). Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check.
-API request and response schemas: [createWorkflowFromSource](/api-reference/workflows/createWorkflowFromSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog), [getSourceResources](/api-reference/workflows/getSourceResources).
+API request and response schemas: [createWorkflowFromSource](/api-reference/workflows/createWorkflowFromSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog), [getSourceResources](/api-reference/workflows/getSourceResources), [deployWorkflow](/api-reference/workflows/deployWorkflow).
### Apply a workflow source directory to a workflow draft
@@ -1387,16 +1389,18 @@ sim workflows source apply [workflowId] [options]
| Option | Required | Description |
| --- | --- | --- |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
| `--dir ` | Yes | Source directory: workflow.ts, functions/*, .sim/lock.json and .sim/base.json. |
| `--dry-run` | No | Check the source and show the outcome without saving or writing files. |
| `-y, --yes` | No | Apply without asking for confirmation. |
+| `--deploy` | No | After the write is saved, deploy the workflow (as `sim workflows deploy ` does). |
| `--allow-delete` | No | Delete blocks the source no longer declares; without it, an apply that deletes a block is refused. |
-Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); 6 the workflow or its folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again. Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.
+Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); 6 the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it). Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check.
-API request and response schemas: [applyWorkflowSource](/api-reference/workflows/applyWorkflowSource).
+API request and response schemas: [applyWorkflowSource](/api-reference/workflows/applyWorkflowSource), [deployWorkflow](/api-reference/workflows/deployWorkflow).
### Show the block catalog workflow source is checked against
@@ -1431,6 +1435,7 @@ sim workflows source check [options]
| `--dir ` | No | Source directory: workflow.ts, functions/* and .sim/. Defaults to `.`. |
| `--strict` | No | Fail on the warnings listed below in every block, not only new ones. |
| `--no-strict` | No | Never fail on a warning. |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
@@ -1438,7 +1443,7 @@ Compiles workflow.ts (and the function files it names) with the compiler the ser
Exit codes: 0 no errors; 1 errors (each printed as file:line:column with its frame and hint), or no cached catalog (run the get once). With --output json, stdout is one document that also lists every finding under diagnostics.
-These warnings, which an apply saves as written, fail the check as errors on every block the lock does not bind (a block added since the get, or every block of a new workflow): W_REQUIRED_MISSING, W_UNKNOWN_OPTION, W_UNKNOWN_OUTPUT_PATH, W_VALUE_KIND, W_OUT_OF_RANGE, W_LITERAL_SECRET, W_TABLE_UNKNOWN_COLUMN, W_CHILD_INPUT_UNKNOWN, W_BRANCH_UNWIRED. --strict fails on them in every block, except W_BRANCH_UNWIRED on a block the lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never fails on a warning.
+These warnings, which an apply saves as written, fail the check as errors on every block the lock does not bind (a block added since the get, or every block of a new workflow): W_REQUIRED_MISSING, W_UNKNOWN_OPTION, W_UNKNOWN_OUTPUT_PATH, W_VALUE_KIND, W_OUT_OF_RANGE, W_LITERAL_SECRET, W_TABLE_UNKNOWN_COLUMN, W_CHILD_INPUT_UNKNOWN, W_BRANCH_UNWIRED, W_UNRESOLVED_REF. --strict fails on them in every block, except W_BRANCH_UNWIRED on a block the lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never fails on a warning. W_JSON_REF_UNTYPED is reported only on blocks the lock does not bind, and with --strict on every block.
### Show the content of the changes in a workflow source directory
@@ -1472,6 +1477,38 @@ Without --server, each file created, edited or deleted since the directory was r
API request and response schemas: [getWorkflowSource](/api-reference/workflows/getWorkflowSource).
+### Print a builder's params and outputs from the cached block catalog
+
+Offline.
+
+```bash
+sim workflows source params [options]
+```
+
+**Arguments**
+
+
+
+| Argument | Required | Description |
+| --- | --- | --- |
+| `builder` | Yes | Builder path (table.upsertRow, agent, trigger.schedule) or namespace |
+
+
+
+**Options**
+
+
+
+| Option | Required | Description |
+| --- | --- | --- |
+| `--dir ` | No | Source directory whose catalog to read. Defaults to `.`. |
+
+
+
+Prints a builder's params (required first, with enums and defaults), the gate field that picks which others apply, and its outputs, from the block catalog the last get, create or check cached. No request is made. <builder> is the dotted builder path as source writes it: table.upsertRow, agent, trigger.schedule, slack.sendMessage (slack.tool.sendMessage names the same builder). A namespace (gmail, trigger.gmail) lists its builders; a construct core.d.ts declares (loop, parallel, t, env) prints its declaration. Run it from a workflow or workspace source directory, or pass --dir.
+
+Exit codes: 0 printed; 1 no builder or namespace of that name (the closest one is named), or no cached catalog (run a get once).
+
### List the tables, knowledge bases, credentials, workflows, skills, custom tools, MCP servers and env names source can name
```bash
diff --git a/apps/docs/content/docs/cli/workspaces.mdx b/apps/docs/content/docs/cli/workspaces.mdx
index 3cbe8c01746..7143068365b 100644
--- a/apps/docs/content/docs/cli/workspaces.mdx
+++ b/apps/docs/content/docs/cli/workspaces.mdx
@@ -733,7 +733,7 @@ sim workspaces source status [options]
-Exit codes: 0 clean (nothing to apply, and the server changed nothing since the directory was read); 11 local changes only (applying writes them); 12 the server changed since the directory was read: drift (a resource edited, added or deleted there that the directory leaves alone) or a conflict (also changed here, so applying it is refused as stale), with or without local changes; 7 the source is too large for the server to read in one dry run; 1 the command failed. A server that cannot be reached is not a failure: the status shows the local changes only and says so (exit 0 or 11).
+Exit codes: 0 clean (nothing to apply, and the server changed nothing since the directory was read); 11 local changes only (applying writes them); 12 the server changed since the directory was read: drift (a resource edited, added or deleted there that the directory leaves alone) or a conflict (also changed here, so applying it is refused as stale), with or without local changes; 7 the source is too large for the server to read in one dry run; 1 the command failed; 13 not logged in (AUTH_REQUIRED) or the credential was refused: the status shows the local changes only, then run `sim login`. A server that cannot be reached is not a failure: the status shows the local changes only and says so (exit 0 or 11).
Exit 10 (busy, retry): another sim workspaces source command is using this directory (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.
@@ -751,18 +751,19 @@ sim workspaces source apply [options]
| Option | Required | Description |
| --- | --- | --- |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
| `--dir ` | Yes | Workspace source directory written by `workspaces source get`. |
| `--dry-run` | No | Plan and check without writing anything. |
| `-y, --yes` | No | Apply without asking for confirmation. |
| `--allow-delete` | No | Delete resources, columns and data the directory no longer declares. |
| `--only ` | No | Apply only these kinds (comma-separated folders: secrets.ts, credentials.ts, tools, skills, mcp, tables, knowledge, files, workflows, deployments). |
-| `--to ` | No | Promote the directory to another workspace (dev to prod): its resources bind by key to that workspace's own and are updated in place; the directory is not changed. |
+| `--to ` | No | Promote the directory to another workspace (dev to prod): its resources bind by key to that workspace's own and are updated in place. Only .sim/targets/<workspaceId>.json changes, the record the next promotion compares with: a resource changed there since is kept, or refused as stale (exit 5) when the directory changed it too. |
-Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint), or changes were held back because they use a declared secret the workspace has no value for or a declared credential key nothing binds to (the rest was applied: set the secret or connect the credential, then apply again), the directory was read from another workspace than the profile's (SOURCE_OTHER_WORKSPACE: pass --workspace to choose, or --to to promote it), its base records binary files no get downloaded (SOURCE_BLOBS_NOT_FETCHED: get again first), or the command failed; 5 a resource changed or was deleted on the server since the directory was read (SOURCE_STALE: get it again into a fresh directory, reapply the edit, retry); 6 a resource is locked; 7 the directory is too large; 8 the apply stopped part-way on a server failure: what was written is saved into the directory, and applying again continues. An apply a resource refused part-way (stale, locked, forbidden) saves what was written too, and exits as that refusal: 5, 6 or 1; 9 rate limited (HTTP 429) after the CLI's retries, before anything was written: wait a minute and apply again (binary files already uploaded are not sent again); 10 another apply is writing the workspace (SOURCE_BUSY), or another command is using the directory (SOURCE_DIR_BUSY): nothing was written, run the command again once the other finished.
+Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint), or changes were held back because they use a declared secret the workspace has no value for or a declared credential key nothing binds to (the rest was applied: set the secret or connect the credential, then apply again), the directory was read from another workspace than the one --workspace names (SOURCE_OTHER_WORKSPACE: drop --workspace, or promote it with --to; without --workspace an apply targets the workspace the directory was read from), its base records binary files no get downloaded (SOURCE_BLOBS_NOT_FETCHED: get again first), or the command failed; 5 a resource changed or was deleted on the server since the directory was read (SOURCE_STALE: get it again into a fresh directory, reapply the edit, retry); 6 a resource (a workflow or its folder, a table) is locked (a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the directory is too large; 8 the apply stopped part-way on a server failure: what was written is saved into the directory, and applying again continues. An apply a resource refused part-way (stale, locked, forbidden) saves what was written too, and exits as that refusal: 5, 6 or 1; 9 rate limited (HTTP 429) after the CLI's retries, before anything was written: wait a minute and apply again (binary files already uploaded are not sent again); 10 another apply is writing the workspace (SOURCE_BUSY), or another command is using the directory (SOURCE_DIR_BUSY): nothing was written, run the command again once the other finished; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it). Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check.
-API request and response schemas: [applyWorkspaceSource](/api-reference/workflows/applyWorkspaceSource), [findMissingWorkspaceSourceBlobs](/api-reference/workflows/findMissingWorkspaceSourceBlobs), [stageWorkspaceSourceBlobs](/api-reference/workflows/stageWorkspaceSourceBlobs), [getWorkspaceSource](/api-reference/workflows/getWorkspaceSource).
+API request and response schemas: [applyWorkspaceSource](/api-reference/workflows/applyWorkspaceSource), [findMissingWorkspaceSourceBlobs](/api-reference/workflows/findMissingWorkspaceSourceBlobs), [stageWorkspaceSourceBlobs](/api-reference/workflows/stageWorkspaceSourceBlobs), [getWorkspaceSource](/api-reference/workflows/getWorkspaceSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog).
### Compile every workflow of a workspace source directory locally, without the server
@@ -779,6 +780,7 @@ sim workspaces source check [options]
| `--dir ` | No | Workspace source directory: the kind folders and .sim/base.json. Defaults to `.`. |
| `--strict` | No | Fail on the warnings listed below in every block, not only new ones. |
| `--no-strict` | No | Never fail on a warning. |
+| `--verbose` | No | With --output json, also list info findings (such as blocks placed by autolayout). |
@@ -786,7 +788,7 @@ Compiles every workflows/<key>/workflow.ts (and the function files each na
Exit codes: 0 no errors; 1 errors (each printed as file:line:column with its frame and hint), or no cached catalog (run the get once). With --output json, stdout is one document that also lists every finding under diagnostics.
-These warnings, which an apply saves as written, fail the check as errors on every block the lock does not bind (a block added since the get, or every block of a new workflow): W_REQUIRED_MISSING, W_UNKNOWN_OPTION, W_UNKNOWN_OUTPUT_PATH, W_VALUE_KIND, W_OUT_OF_RANGE, W_LITERAL_SECRET, W_TABLE_UNKNOWN_COLUMN, W_CHILD_INPUT_UNKNOWN, W_BRANCH_UNWIRED. --strict fails on them in every block, except W_BRANCH_UNWIRED on a block the lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never fails on a warning.
+These warnings, which an apply saves as written, fail the check as errors on every block the lock does not bind (a block added since the get, or every block of a new workflow): W_REQUIRED_MISSING, W_UNKNOWN_OPTION, W_UNKNOWN_OUTPUT_PATH, W_VALUE_KIND, W_OUT_OF_RANGE, W_LITERAL_SECRET, W_TABLE_UNKNOWN_COLUMN, W_CHILD_INPUT_UNKNOWN, W_BRANCH_UNWIRED, W_UNRESOLVED_REF. --strict fails on them in every block, except W_BRANCH_UNWIRED on a block the lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never fails on a warning. W_JSON_REF_UNTYPED is reported only on blocks the lock does not bind, and with --strict on every block.
Exit 10 (busy, retry): another sim workspaces source command is using this directory (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.
diff --git a/apps/docs/content/docs/workflows/code.mdx b/apps/docs/content/docs/workflows/code.mdx
index 8366500512f..a46903e0f7a 100644
--- a/apps/docs/content/docs/workflows/code.mdx
+++ b/apps/docs/content/docs/workflows/code.mdx
@@ -13,12 +13,14 @@ Workflows as code is in preview. When it is [enabled on your Sim deployment](/pl
## For coding agents [#for-coding-agents]
-Claude Code, Cursor, Codex and other coding agents work from a directory that `get` writes. Point the agent at `.sim/AGENTS.md` in it, which walks through the loop:
+Claude Code, Cursor, Codex and other coding agents work from a directory that `get` writes. `get` prints a short primer (the edit loop, how block identity and references work, deletes, deploy, exit codes, and the params of each builder the workflow already uses; in JSON output, its `primer` field), and the same primer opens `.sim/AGENTS.md`, the full guide. The loop:
1. `sim workflows source get --dir ` writes the workflow and the guide.
2. Edit `workflow.ts`.
3. `sim workflows source check --dir ` compiles it offline.
4. `sim workflows source status --dir ` shows what changed and what to run next.
+
+To write a block the workflow does not use yet, `sim workflows source params ` (for example `table.upsertRow` or `trigger.schedule`) prints its params, required ones first with their options and defaults, and its outputs, from the cached block catalog without a request. A namespace such as `gmail` lists its builders.
5. `sim workflows source apply --dir --dry-run`, then `--yes`, saves the draft.
For a whole workspace, use `sim workspaces source` with the same steps; see [Workspace as code](/workflows/workspace-code).
@@ -37,7 +39,7 @@ Nothing in the file runs on your machine. Sim reads it statically, checks it aga
sim workflows source get --dir ./support-triage
```
- This writes `workflow.ts`, any function code under `functions/`, and type declarations for every block and for your workspace's tables, credentials, knowledge bases, and environment variable names, with `.sim/sdk/INDEX.md` listing every builder by block and operation. It also writes a `tsconfig.json` that resolves them and type-checks the blocks the workflow uses, and a `tsconfig.full.json` that type-checks every block, unless the directory already has files of those names, which are left as they are. For coding agents it writes `.sim/AGENTS.md`, a short guide, and example workflows under `.sim/examples/` with one correct example of each common block to copy from. The examples name placeholder resources, so the tsconfig files leave them out of the type-check.
+ This writes `workflow.ts`, any function code under `functions/`, and type declarations for the blocks the workflow uses and for your workspace's tables, credentials, knowledge bases, and environment variable names, with `.sim/sdk/INDEX.md` listing those blocks' builders. The declarations of every block are generated once per block catalog into the CLI's cache (`~/.sim/cache/source-catalog/`) and shared by every directory. It also writes a `tsconfig.json` that resolves them and type-checks the blocks the workflow uses, and a `tsconfig.full.json` that type-checks every block through that cache, unless the directory already has files of those names, which are left as they are. For coding agents it writes `.sim/AGENTS.md`, the primer `get` prints followed by a reference, and example workflows under `.sim/examples/` with one correct example of each common block to copy from. The examples name placeholder resources, so the tsconfig files leave them out of the type-check.
`get` into a directory that already holds source refuses to overwrite edits you have not applied, and refuses a directory that holds a different workflow. Pass `--force` to replace them.
@@ -84,13 +86,13 @@ Nothing in the file runs on your machine. Sim reads it statically, checks it aga
sim workflows source apply --dir ./support-triage --yes
```
- The workflow builder updates for anyone who has the workflow open. Deploy when you are ready. The save rewrites `workflow.ts` in canonical form, so read it again before the next edit.
+ The workflow builder updates for anyone who has the workflow open. Deploy when you are ready, or add `--deploy` to deploy right after the save; without it, a deployed workflow whose draft now differs prints the `sim workflows deploy` command to run. `--yes` refuses everything a dry run would, so step 4 is optional: without `--yes`, and with no terminal to ask, `apply` prints the plan, writes nothing and exits `14`. The save rewrites `workflow.ts` in canonical form, so read it again before the next edit.
Editing the name or description in `workflow('…')` renames or redescribes the workflow; a name another workflow in its folder already has is refused with `E_NAME_TAKEN`. Lines starting with `note:` report something the save did beyond what the file says; read them.
A save that removes a block (its declaration is gone from `workflow.ts`) is refused with `E_APPLY_BLOCK_DELETE`, naming the blocks. Pass `--allow-delete` when the deletion is intended.
-To create a new workflow, run `sim workflows source create --dir ` on an empty directory: without `--dry-run` or `--yes` it creates nothing and writes a starter `workflow.ts` with the type declarations, `tsconfig.json` and `.sim/AGENTS.md`. Edit the starter, check it, then run `sim workflows source create --dir --dry-run` and `--yes`. Until the workflow is created the directory has no `.sim/base.json`, so `status`, `diff` and `apply` refuse it. The file starts with the `// @sim-sdk 1` line and default-exports one `workflow('Name', {}, () => { ... })`. The command prints the new workflow ID and writes the same type declarations, `tsconfig.json`, and `.sim/` files that `get` writes, so you can continue editing with `apply`. If another workflow already has that name, the command exits `1`; pass `--name` to choose another.
+To create a new workflow, run `sim workflows source create --dir --workspace ` on an empty directory: without `--dry-run` or `--yes` it creates nothing and writes a starter `workflow.ts` with that workspace's type declarations, `tsconfig.json` and `.sim/AGENTS.md`, and records the workspace in `.sim/starter.json`. Edit the starter, check it, then run `sim workflows source create --dir --workspace --dry-run` and `--yes` (the starter prints both). A create goes to the starter's workspace whatever your default workspace, unless `--workspace` names another. Until the workflow is created the directory has no `.sim/base.json`, so `status`, `diff` and `apply` refuse it. The file starts with the `// @sim-sdk 1` line and default-exports one `workflow('Name', {}, () => { ... })`. The command prints the new workflow ID and writes the same type declarations, `tsconfig.json`, and `.sim/` files that `get` writes, so you can continue editing with `apply`. If another workflow already has that name, the command exits `1`; pass `--name` to choose another.
Keep the `.sim/` folder the commands write. `.sim/lock.json` ties each block in the file to the stored block, so renaming a block keeps the same block, with its webhook URL, schedules and run history. Changing its name string keeps it through the lock; renaming the variable too keeps it when the block's type, parent and connections are unchanged, and the dry run reports that as `W_MAT_INFERRED_RENAME`. Renaming two blocks of the same type in the same place in one save is ambiguous, so do those one save at a time. `.sim/base.json` records the version you read.
@@ -195,14 +197,14 @@ Code that does not fit a block goes in a Function block, kept in its own file:
const score = fn('Score lead', { code: codeFile('./functions/score-lead.js') })
```
-In the file, `$ref.block.path` reads another block's output and `$env.NAME` reads an environment variable. `block` is the block's key: its name lowercased with all whitespace and every `.` removed (`$ref.triageagent.content` for "Triage Agent", `$ref.v2result.text` for "V2. Result"). A `$ref` that names no block is a warning, because it would not resolve when the workflow runs. Python and shell files are saved as written.
+In the file, `$ref.block.path` reads another block's output and `$env.NAME` reads an environment variable. `block` is the block's key: its name lowercased with all whitespace and every `.` removed (`$ref.triageagent.content` for "Triage Agent", `$ref.v2result.text` for "V2. Result"). Inside a loop body, `$ref.loop.currentItem` and `$ref.loop.index` read the iteration's item and position; inside a parallel body, `$ref.parallel.currentItem` and `$ref.parallel.index`. A `$ref` that names no block is a warning, because it would not resolve when the workflow runs. Python and shell files are saved as written.
## Use it in CI
Run `sim workflows source apply --dir --dry-run` in CI to catch a broken workflow before it is saved. It exits `0` when the workflow is valid and `1` when it has errors, so a failing check blocks the merge. It exits `5` when the workflow was changed in the editor after you committed the directory; run `get` again and reapply your edit.
- **Commit** `workflow.ts`, `functions/`, `.sim/lock.json`, `.sim/base.json`, `.sim/written.json`, `tsconfig.json` and `.gitattributes`. CI finds the workflow from `workflowId` in `.sim/base.json`, and `status` and `get` use `.sim/written.json` to tell your edits apart from what the CLI last wrote.
-- **Optionally gitignore** `.sim/sdk/`, the generated type declarations (thousands of files); `get` and `check` write them again. A single-workflow directory gets no `.gitignore`, so add the line yourself.
+- **Optionally gitignore** `.sim/sdk/`, the generated type declarations of the blocks in use (a few dozen files); `get` and `check` write them again. Gitignore `tsconfig.full.json`, which names the CLI's cache on your machine; `check` writes it again. A single-workflow directory gets no `.gitignore`, so add the lines yourself.
- **A whole workspace** in one repository, with dry runs on pull requests, apply on merge and promotion from one workspace to another, is covered in [Git and CI](/workflows/workspace-code#git-and-ci) and [Promote to another workspace](/workflows/workspace-code#promote-to-another-workspace).
- **Credentials:** use a personal API key (**Settings → Sim API keys**, passed as `SIM_API_KEY`) or an OAuth login, not a workspace API key. Saving and dry-running source act as a person, so a workspace key is refused with `403`. Set `SIM_WORKSPACE` unless you pass `--workspace`, and `SIM_ENDPOINT` if you use a self-hosted Sim. See [Authenticating CI](/cli/authentication#authenticating-ci).
diff --git a/apps/docs/content/docs/workflows/workspace-code.mdx b/apps/docs/content/docs/workflows/workspace-code.mdx
index bdbbc7e1df5..f5b580f4740 100644
--- a/apps/docs/content/docs/workflows/workspace-code.mdx
+++ b/apps/docs/content/docs/workflows/workspace-code.mdx
@@ -48,7 +48,7 @@ const log = table.insertRow('Log Question', {
})
```
-`get` writes `.sim/AGENTS.md`, a guide for coding agents, and `.sim/examples/workspace/`, a complete example of every kind and the workflow that uses them.
+`get` prints a short primer for coding agents (in JSON output, its `primer` field) and writes it atop `.sim/AGENTS.md`, the guide, with deletes in depth, Git and CI, and promoting to another workspace in `.sim/GUIDE-advanced.md`. It also writes `.sim/examples/workspace/`, a complete example of every kind and the workflow that uses them.
## Edit a workspace from the CLI
@@ -79,6 +79,7 @@ To apply the rest with a workspace key, such as in CI, narrow the apply to the k
- A resource the directory never managed is kept, and the apply names it in a warning.
- Something deleted in Sim is not re-created by an unchanged directory, and an edit to it is refused as stale.
- Table rows are part of the directory only when the table's `rows` opts in (`tables/.rows.jsonl`). Connector-synced documents and run history never are.
+- Seed rows (`rows: { mode: 'seed' }`) belong to the directory: `get` keeps the declaration and the rows file beside it, but Sim does not store them, so a get into a new directory has the declaration without the file.
- A delete cannot be undone by applying an older copy of the directory, such as a reverted commit. The resource stays deleted, and its rows, documents, run history, webhook URLs and schedules are gone. The dry run marks each change that deletes data and counts them as `destructive`.
## Promote to another workspace
@@ -90,7 +91,15 @@ sim workspaces source apply --dir ./my-workspace --to "$PROD_WORKSPACE_ID" --dry
sim workspaces source apply --dir ./my-workspace --to "$PROD_WORKSPACE_ID" --yes
```
-Each resource binds by key to the target's own. One the target holds under the same key is updated in place, so a workflow keeps its id and its webhook URLs. A key the target lacks is created. Secrets and credentials bind by name to the target's own, and no value is copied. Resources only the target holds are kept unless you pass `--allow-delete`. The directory is not changed and stays bound to the workspace it was read from. Without `--to`, an apply refuses a directory read from another workspace.
+Each resource binds by key to the target's own. One the target holds under the same key is updated in place, so a workflow keeps its id and its webhook URLs. A key the target lacks is created. Secrets and credentials bind by name to the target's own, and no value is copied. Resources only the target holds are kept unless you pass `--allow-delete`. The directory stays bound to the workspace it was read from. Without `--to`, `check`, `status`, `diff` and `apply` act on the workspace the directory was read from, whatever your default workspace; a `--workspace` naming another one is refused.
+
+Each promotion records what it left in the target in `.sim/targets/.json`, and the next promotion compares with that record:
+
+- A resource changed in the target since the last promotion, such as a hotfix made in Sim, is kept when the directory left it alone.
+- When the directory changed it too, the promotion is refused as stale and exits `5`. Carry the target's change into the source workspace, or undo it in the target, then promote again.
+- The first promotion, and any resource the record does not hold (one created in the target since), updates the target to match the directory. Review the dry run first.
+
+Commit the record with the rest of the directory, so a promotion from CI or another clone compares with it. Deleting it makes the next promotion take the target as it stands.
## Git and CI
@@ -102,7 +111,7 @@ Commit the whole directory, including `.sim/`: its base files are what an apply
- Renaming a skill does not rewrite the workflows that use it: each agent keeps the name it stored until its skills list is next edited.
- One apply sends at most 100 edited workflows and makes at most 1,000 changes; a directory holds at most 500 resources of a kind, and a `tables/.rows.jsonl` at most 10,000 rows. An apply over a limit is refused before anything is written.
- A document or file over 10 MB, or whose name a directory cannot hold, is left out of `get` and listed in `.sim/skipped.json`. An apply never writes or deletes it.
-- Hidden files (a name starting with a dot, such as `files/.env`) are never sent. An apply names them in a warning.
+- Hidden files (a name starting with a dot, such as `files/.env`) and names with a character no workspace path holds (such as `:` or `?`) are never sent. An apply names each one in a `W_PATH_NOT_SENT` warning.
- A knowledge base's `embeddingModel` is fixed when it is created. An apply that changes it is refused.
- An apply that stops part-way exits `8` and saves what it wrote into the directory. Run the same apply again until it exits `0`.
- An apply exits `5` when a resource changed in Sim after you read it. Run `get` into a new directory and reapply your edit.
diff --git a/apps/docs/openapi-v2-workflows.json b/apps/docs/openapi-v2-workflows.json
index 847540cbd2a..e5bf63509a0 100644
--- a/apps/docs/openapi-v2-workflows.json
+++ b/apps/docs/openapi-v2-workflows.json
@@ -6343,7 +6343,8 @@
"added": [
{
"id": "a3f1c0b2-7a44-4c1d-9d3a-2b8e5f0a1c77",
- "key": "start"
+ "key": "start",
+ "type": "start_trigger"
}
],
"removed": [],
@@ -8986,7 +8987,7 @@
"post": {
"operationId": "stageWorkspaceSourceBlobs",
"summary": "Stage workspace source blobs",
- "description": "Stage the bytes of binary files (workspace files, knowledge documents) for a workspace source apply, as multipart form data: one file part per blob, its field name the SHA-256 of its bytes. At most 1000 files and 24000000 bytes per request, 10000000 bytes per file. A part whose bytes hash to another SHA-256 refuses the whole request (`400`), nothing staged. Staged files are scoped to the workspace and the caller, capped at 20000 files and 4000000000 bytes (`413` past either), and expire a day after their last use (staged, asked for, or applied); an apply does not remove them, so applying the same bytes again uploads nothing. Staging the same bytes again only renews them. Requires workspace write access. Returns `403` while workflows as code is not enabled.\n\nOAuth scope: `api:write`.",
+ "description": "Stage the bytes of binary files (workspace files, knowledge documents) for a workspace source apply, as multipart form data: one file part per blob, its field name the SHA-256 of its bytes. At most 1000 files and 8000000 bytes per request, 10000000 bytes per file; a larger file goes in a request of its own. A body over 10001024 bytes answers `413` (`PAYLOAD_TOO_LARGE`) before it is read. A part whose bytes hash to another SHA-256 refuses the whole request (`400`), nothing staged. Staged files are scoped to the workspace and the caller, capped at 20000 files and 4000000000 bytes (`413` with `details.code` `SOURCE_STAGING_FULL` past either), and expire a day after their last use (staged, asked for, or applied); an apply does not remove them, so applying the same bytes again uploads nothing. Staging the same bytes again only renews them. Requires workspace write access. Returns `403` while workflows as code is not enabled.\n\nOAuth scope: `api:write`.",
"x-sim-operation": "workspaces.source.blobs.stage",
"x-oauth-scope": "api:write",
"tags": ["Workflows"],
@@ -18915,6 +18916,7 @@
"W_APPLY_TYPE_CHANGED",
"W_APPLY_TRIGGER_REMOVED",
"W_APPLY_CATALOG_MOVED",
+ "W_APPLY_PROBABLE_RENAME",
"E_APPLY_LINT",
"W_APPLY_LINT",
"W_TABLE_UNKNOWN_COLUMN",
@@ -19060,11 +19062,15 @@
"key": {
"type": "string",
"description": "Block key in the source."
+ },
+ "type": {
+ "type": "string",
+ "description": "Block type."
}
},
- "required": ["id", "key"],
+ "required": ["id", "key", "type"],
"additionalProperties": false,
- "description": "A block by id and source key."
+ "description": "An added or removed block by id, source key and type."
},
"description": "New blocks."
},
@@ -19080,11 +19086,15 @@
"key": {
"type": "string",
"description": "Block key in the source."
+ },
+ "type": {
+ "type": "string",
+ "description": "Block type."
}
},
- "required": ["id", "key"],
+ "required": ["id", "key", "type"],
"additionalProperties": false,
- "description": "A block by id and source key."
+ "description": "An added or removed block by id, source key and type."
},
"description": "Deleted blocks."
},
@@ -19350,6 +19360,7 @@
"W_APPLY_TYPE_CHANGED",
"W_APPLY_TRIGGER_REMOVED",
"W_APPLY_CATALOG_MOVED",
+ "W_APPLY_PROBABLE_RENAME",
"E_APPLY_LINT",
"W_APPLY_LINT",
"W_TABLE_UNKNOWN_COLUMN",
@@ -20012,11 +20023,15 @@
"key": {
"type": "string",
"description": "Block key in the source."
+ },
+ "type": {
+ "type": "string",
+ "description": "Block type."
}
},
- "required": ["id", "key"],
+ "required": ["id", "key", "type"],
"additionalProperties": false,
- "description": "A block by id and source key."
+ "description": "An added or removed block by id, source key and type."
},
"description": "New blocks."
},
@@ -20032,11 +20047,15 @@
"key": {
"type": "string",
"description": "Block key in the source."
+ },
+ "type": {
+ "type": "string",
+ "description": "Block type."
}
},
- "required": ["id", "key"],
+ "required": ["id", "key", "type"],
"additionalProperties": false,
- "description": "A block by id and source key."
+ "description": "An added or removed block by id, source key and type."
},
"description": "Deleted blocks."
},
@@ -20302,6 +20321,7 @@
"W_APPLY_TYPE_CHANGED",
"W_APPLY_TRIGGER_REMOVED",
"W_APPLY_CATALOG_MOVED",
+ "W_APPLY_PROBABLE_RENAME",
"E_APPLY_LINT",
"W_APPLY_LINT",
"W_TABLE_UNKNOWN_COLUMN",
@@ -20623,7 +20643,8 @@
"added": [
{
"id": "a3f1c0b2-7a44-4c1d-9d3a-2b8e5f0a1c77",
- "key": "start"
+ "key": "start",
+ "type": "start_trigger"
}
],
"removed": [],
@@ -25117,6 +25138,7 @@
"E_DECL_SHAPE",
"E_DECL_UNKNOWN_FIELD",
"E_DECL_UNKNOWN_FILE",
+ "W_PATH_NOT_SENT",
"E_BLOB_MANIFEST",
"E_BLOB_MISSING",
"E_LIMIT_EXCEEDED",
@@ -25191,6 +25213,7 @@
"E_PLAN_REF_DELETED",
"E_PLAN_REF_HELD",
"W_RESOURCE_IN_USE",
+ "W_TABLE_COLUMN_RENAMED_IN_USE",
"E_NAME_TAKEN",
"E_WORKFLOW_FILES_MISSING",
"E_WORKFLOW_FOLDER_COPIED",
@@ -25340,6 +25363,7 @@
"W_APPLY_TYPE_CHANGED",
"W_APPLY_TRIGGER_REMOVED",
"W_APPLY_CATALOG_MOVED",
+ "W_APPLY_PROBABLE_RENAME",
"E_APPLY_LINT",
"W_APPLY_LINT",
"W_TABLE_UNKNOWN_COLUMN",
@@ -25680,6 +25704,7 @@
"E_DECL_SHAPE",
"E_DECL_UNKNOWN_FIELD",
"E_DECL_UNKNOWN_FILE",
+ "W_PATH_NOT_SENT",
"E_BLOB_MANIFEST",
"E_BLOB_MISSING",
"E_LIMIT_EXCEEDED",
@@ -25754,6 +25779,7 @@
"E_PLAN_REF_DELETED",
"E_PLAN_REF_HELD",
"W_RESOURCE_IN_USE",
+ "W_TABLE_COLUMN_RENAMED_IN_USE",
"E_NAME_TAKEN",
"E_WORKFLOW_FILES_MISSING",
"E_WORKFLOW_FOLDER_COPIED",
@@ -25903,6 +25929,7 @@
"W_APPLY_TYPE_CHANGED",
"W_APPLY_TRIGGER_REMOVED",
"W_APPLY_CATALOG_MOVED",
+ "W_APPLY_PROBABLE_RENAME",
"E_APPLY_LINT",
"W_APPLY_LINT",
"W_TABLE_UNKNOWN_COLUMN",
@@ -26086,6 +26113,7 @@
"E_DECL_SHAPE",
"E_DECL_UNKNOWN_FIELD",
"E_DECL_UNKNOWN_FILE",
+ "W_PATH_NOT_SENT",
"E_BLOB_MANIFEST",
"E_BLOB_MISSING",
"E_LIMIT_EXCEEDED",
@@ -26160,6 +26188,7 @@
"E_PLAN_REF_DELETED",
"E_PLAN_REF_HELD",
"W_RESOURCE_IN_USE",
+ "W_TABLE_COLUMN_RENAMED_IN_USE",
"E_NAME_TAKEN",
"E_WORKFLOW_FILES_MISSING",
"E_WORKFLOW_FOLDER_COPIED",
@@ -26309,6 +26338,7 @@
"W_APPLY_TYPE_CHANGED",
"W_APPLY_TRIGGER_REMOVED",
"W_APPLY_CATALOG_MOVED",
+ "W_APPLY_PROBABLE_RENAME",
"E_APPLY_LINT",
"W_APPLY_LINT",
"W_TABLE_UNKNOWN_COLUMN",
diff --git a/apps/docs/scripts/fixtures/content-baseline.json b/apps/docs/scripts/fixtures/content-baseline.json
index 6b9edc3c27b..54cc31249cc 100644
--- a/apps/docs/scripts/fixtures/content-baseline.json
+++ b/apps/docs/scripts/fixtures/content-baseline.json
@@ -13189,6 +13189,13 @@
"bytes": null,
"line": null
},
+ {
+ "lang": "bash",
+ "meta": null,
+ "sha256": "2b6c40dcef94f6188188d6369383e982f5a6dc88654b8ab76664c189f9af8ab0",
+ "bytes": null,
+ "line": null
+ },
{
"lang": "bash",
"meta": null,
@@ -17768,6 +17775,13 @@
"bytes": null,
"line": null
},
+ {
+ "lang": "bash",
+ "meta": null,
+ "sha256": "2b6c40dcef94f6188188d6369383e982f5a6dc88654b8ab76664c189f9af8ab0",
+ "bytes": null,
+ "line": null
+ },
{
"lang": "bash",
"meta": null,
diff --git a/apps/sim/app/api/v2/workflows/[workflowId]/source/route.test.ts b/apps/sim/app/api/v2/workflows/[workflowId]/source/route.test.ts
index f1b1b2ca24d..741d8ffe85e 100644
--- a/apps/sim/app/api/v2/workflows/[workflowId]/source/route.test.ts
+++ b/apps/sim/app/api/v2/workflows/[workflowId]/source/route.test.ts
@@ -215,7 +215,7 @@ describe('POST /api/v2/workflows/{workflowId}/source', () => {
it('answers 409 with the current fingerprint when the draft moved', async () => {
const current = `sf1:${'f'.repeat(64)}`
- mocks.apply.mockRejectedValue(new WorkflowSourceStaleError(current))
+ mocks.apply.mockRejectedValue(new WorkflowSourceStaleError(current, 'Lead Scoring'))
const response = await POST(applyRequest(applyBody), context)
const body = await response.json()
diff --git a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/blobs/route.test.ts b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/blobs/route.test.ts
new file mode 100644
index 00000000000..a025ec5f6d0
--- /dev/null
+++ b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/blobs/route.test.ts
@@ -0,0 +1,127 @@
+import { createHash } from 'node:crypto'
+import {
+ V2_OPERATION_RATE_LIMIT_ALLOWED,
+ V2_PREAUTH_RATE_LIMIT_ALLOWED,
+ v2ApiKeyAuthModuleMock,
+ v2RateLimiterModuleMock,
+ v2RouteMocks,
+} from '@sim/testing'
+import { NextRequest } from 'next/server'
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+
+const mocks = vi.hoisted(() => ({ admit: vi.fn(), stage: vi.fn() }))
+
+vi.mock('@/lib/api/server/routes/v2-api-key-auth', () => v2ApiKeyAuthModuleMock)
+vi.mock('@/lib/core/rate-limiter', () => v2RateLimiterModuleMock)
+vi.mock('@/lib/workspace-source/application/stage-blobs', () => ({
+ admitSourceBlobStage: {
+ operation: { id: 'workspaces.source.blobs.stage' },
+ execute: mocks.admit,
+ },
+ stageSourceBlobs: { operation: { id: 'workspaces.source.blobs.stage' }, execute: mocks.stage },
+}))
+
+import { SourceStagingFullError } from '@/lib/workspace-source/staging/store'
+import { POST } from '@/app/api/v2/workspaces/[workspaceId]/source/blobs/route'
+
+const WORKSPACE_ID = '11111111-2222-4333-8444-555555555555'
+
+const auth = {
+ principal: { kind: 'personal_api_key' as const, userId: 'user-1', keyId: 'key-2' },
+ rateLimitSubjectIds: ['api-key:key-2'] as const,
+ rateLimitSubscription: null,
+ keyType: 'personal' as const,
+}
+
+const context = { params: Promise.resolve({ workspaceId: WORKSPACE_ID }) }
+
+/** A staging request carrying one part per buffer, as the CLI builds it. */
+async function stagingRequest(buffers: Buffer[]): Promise {
+ const form = new FormData()
+ for (const bytes of buffers) {
+ const sha256 = createHash('sha256').update(bytes).digest('hex')
+ form.append(sha256, new Blob([new Uint8Array(bytes)]), sha256)
+ }
+ const encoded = new Response(form)
+ const body = Buffer.from(await encoded.arrayBuffer())
+ return new NextRequest(`http://localhost:3000/api/v2/workspaces/${WORKSPACE_ID}/source/blobs`, {
+ method: 'POST',
+ headers: {
+ 'x-api-key': 'key',
+ 'content-type': encoded.headers.get('content-type') ?? '',
+ 'content-length': String(body.byteLength),
+ },
+ body,
+ })
+}
+
+describe('POST /api/v2/workspaces/{workspaceId}/source/blobs', () => {
+ beforeEach(() => {
+ v2RouteMocks.authenticate.mockResolvedValue(auth)
+ v2RouteMocks.preauthRate.mockResolvedValue(V2_PREAUTH_RATE_LIMIT_ALLOWED)
+ v2RouteMocks.operationRate.mockResolvedValue(V2_OPERATION_RATE_LIMIT_ALLOWED)
+ mocks.admit.mockReset().mockResolvedValue({ workspaceId: WORKSPACE_ID })
+ mocks.stage.mockReset().mockResolvedValue({ workspaceId: WORKSPACE_ID, staged: [] })
+ })
+
+ it('answers a body the proxy would cut off with 413 PAYLOAD_TOO_LARGE naming the limit, before reading it', async () => {
+ const response = await POST(
+ await stagingRequest([Buffer.alloc(6_000_000, 1), Buffer.alloc(6_000_000, 2)]),
+ context
+ )
+
+ expect(response.status).toBe(413)
+ const { error } = await response.json()
+ expect(error.code).toBe('PAYLOAD_TOO_LARGE')
+ expect(error.message).toMatch(/at most \d+ bytes; this one carries \d+/)
+ expect(mocks.stage).not.toHaveBeenCalled()
+ })
+
+ it('answers a chunked body past the limit with 413 saying it carries more, not a count it never read', async () => {
+ const declared = await stagingRequest([Buffer.alloc(6_000_000, 1), Buffer.alloc(6_000_000, 2)])
+ const body = Buffer.from(await declared.arrayBuffer())
+ const headers = new Headers(declared.headers)
+ headers.delete('content-length')
+ const chunked = new NextRequest(declared.url, {
+ method: 'POST',
+ headers,
+ body: new ReadableStream({
+ start(controller) {
+ for (let at = 0; at < body.byteLength; at += 1_000_000) {
+ controller.enqueue(new Uint8Array(body.subarray(at, at + 1_000_000)))
+ }
+ controller.close()
+ },
+ }),
+ duplex: 'half',
+ } as ConstructorParameters[1])
+
+ const response = await POST(chunked, context)
+
+ expect(response.status).toBe(413)
+ const { error } = await response.json()
+ expect(error.message).toMatch(/at most (\d+) bytes; this one carries more than \1\./)
+ expect(mocks.stage).not.toHaveBeenCalled()
+ })
+
+ it('answers a caller whose staging area is full with 413 SOURCE_STAGING_FULL, apart from a body over the limit', async () => {
+ mocks.admit.mockRejectedValue(
+ new SourceStagingFullError(
+ 'Staging 1 more binary files would hold 5001, over the 5000-file limit.'
+ )
+ )
+
+ const response = await POST(await stagingRequest([Buffer.from('a')]), context)
+
+ expect(response.status).toBe(413)
+ const { error } = await response.json()
+ expect(error.details).toEqual({ code: 'SOURCE_STAGING_FULL' })
+ })
+
+ it('stages a single file of the largest size a directory holds', async () => {
+ const response = await POST(await stagingRequest([Buffer.alloc(10_000_000, 3)]), context)
+
+ expect(response.status).toBe(200)
+ expect(mocks.stage.mock.calls[0][0].input.blobs).toHaveLength(1)
+ })
+})
diff --git a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/blobs/route.ts b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/blobs/route.ts
index 6c8845c5a0e..f0db210cc38 100644
--- a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/blobs/route.ts
+++ b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/blobs/route.ts
@@ -1,4 +1,7 @@
-import { WORKSPACE_SOURCE_STAGE_MAX_BYTES } from '@sim/workspace-code/limits'
+import {
+ WORKSPACE_SOURCE_LIMITS,
+ WORKSPACE_SOURCE_STAGE_MAX_BYTES,
+} from '@sim/workspace-code/limits'
import { v2StageWorkspaceSourceBlobsContract } from '@/lib/api/contracts/v2/workspace-source'
import { defineV2BodyLifecycleRoute, v2ApiKeyAuth, v2RateLimits } from '@/lib/api/server/routes'
import { OrchestrationError } from '@/lib/core/orchestration/types'
@@ -20,11 +23,23 @@ export const revalidate = 0
const SHA256 = /^[0-9a-f]{64}$/
+/**
+ * What a body over the limit carried, for its refusal: the declared length, or for a chunked body
+ * (read only until it passed the limit) that it carried more than the limit.
+ */
+function carried(request: Request, observedBytes: number | undefined): string {
+ if (request.headers.get('content-length')) {
+ return observedBytes === undefined ? '' : `; this one carries ${observedBytes}`
+ }
+ return `; this one carries more than ${WORKSPACE_SOURCE_STAGE_MAX_BYTES}`
+}
+
/**
* POST /api/v2/workspaces/[workspaceId]/source/blobs — stages the binary files of a workspace
* source apply: a multipart form with one file part per blob, named by its SHA-256. The caller's
- * write access and staging room are checked before the body is read; a part whose bytes hash to
- * another SHA-256 refuses the batch. Draws on its own staging budget, not the source-write one.
+ * write access and staging room are checked before the body is read; a body over
+ * `WORKSPACE_SOURCE_STAGE_MAX_BYTES` (a declared length, or one the proxy truncated at its ceiling)
+ * answers `413 PAYLOAD_TOO_LARGE`, and a part whose bytes hash to another SHA-256 refuses the batch. Draws on its own staging budget, not the source-write one.
*/
export const POST = defineV2BodyLifecycleRoute({
contract: v2StageWorkspaceSourceBlobsContract,
@@ -44,7 +59,12 @@ export const POST = defineV2BodyLifecycleRoute({
label: 'workspace source staging body',
})
} catch (error) {
- if (isPayloadSizeLimitError(error)) throw error
+ if (isPayloadSizeLimitError(error)) {
+ throw new OrchestrationError(
+ 'payload_too_large',
+ `A staging request body carries at most ${WORKSPACE_SOURCE_STAGE_MAX_BYTES} bytes${carried(request, error.observedBytes)}. Send fewer files per request, a file over ${WORKSPACE_SOURCE_LIMITS.maxBlobBytesPerRequest} bytes in a request of its own.`
+ )
+ }
if (isMultipartFieldValidationError(error)) {
throw new OrchestrationError('validation', error.message)
}
@@ -61,7 +81,7 @@ export const POST = defineV2BodyLifecycleRoute({
blobs.push({
sha256: name,
content: await readFileToBufferWithLimit(value, {
- maxBytes: WORKSPACE_SOURCE_STAGE_MAX_BYTES,
+ maxBytes: WORKSPACE_SOURCE_LIMITS.maxBlobBytes,
label: 'workspace source blob',
}),
})
diff --git a/apps/sim/app/api/v2/workspaces/[workspaceId]/workflows/source/route.test.ts b/apps/sim/app/api/v2/workspaces/[workspaceId]/workflows/source/route.test.ts
index 229dfc3afc3..f82f20c20d1 100644
--- a/apps/sim/app/api/v2/workspaces/[workspaceId]/workflows/source/route.test.ts
+++ b/apps/sim/app/api/v2/workspaces/[workspaceId]/workflows/source/route.test.ts
@@ -49,7 +49,12 @@ const created = {
applied: true,
diff: {
noop: false,
- blocks: { added: [{ id: 'b-1', key: 'start' }], removed: [], renamed: [], modified: [] },
+ blocks: {
+ added: [{ id: 'b-1', key: 'start', type: 'start_trigger' }],
+ removed: [],
+ renamed: [],
+ modified: [],
+ },
edges: { added: 0, removed: 0 },
variables: { added: [], removed: [], modified: [] },
},
diff --git a/apps/sim/lib/api/contracts/v2/openapi/workflow-source.ts b/apps/sim/lib/api/contracts/v2/openapi/workflow-source.ts
index 0c5da604194..6a25d2449a0 100644
--- a/apps/sim/lib/api/contracts/v2/openapi/workflow-source.ts
+++ b/apps/sim/lib/api/contracts/v2/openapi/workflow-source.ts
@@ -181,7 +181,7 @@ export const workflowSourceOpenApiRoutes = [
diff: {
noop: false,
blocks: {
- added: [{ id: BLOCK_ID, key: 'start' }],
+ added: [{ id: BLOCK_ID, key: 'start', type: 'start_trigger' }],
removed: [],
renamed: [],
modified: [],
diff --git a/apps/sim/lib/api/contracts/v2/openapi/workspace-source.ts b/apps/sim/lib/api/contracts/v2/openapi/workspace-source.ts
index f1b12e7500a..ad5182a0343 100644
--- a/apps/sim/lib/api/contracts/v2/openapi/workspace-source.ts
+++ b/apps/sim/lib/api/contracts/v2/openapi/workspace-source.ts
@@ -1,4 +1,7 @@
-import { WORKSPACE_SOURCE_LIMITS } from '@sim/workspace-code/limits'
+import {
+ WORKSPACE_SOURCE_LIMITS,
+ WORKSPACE_SOURCE_STAGE_MAX_BYTES,
+} from '@sim/workspace-code/limits'
import {
documentedSchema,
RATE_LIMIT_HEADERS,
@@ -192,7 +195,7 @@ export const workspaceSourceOpenApiRoutes = [
applicationOperation: workspaceSourceOperations.stageBlobs,
operationId: 'stageWorkspaceSourceBlobs',
summary: 'Stage workspace source blobs',
- description: `Stage the bytes of binary files (workspace files, knowledge documents) for a workspace source apply, as multipart form data: one file part per blob, its field name the SHA-256 of its bytes. At most ${WORKSPACE_SOURCE_LIMITS.maxBlobsPerRequest} files and ${WORKSPACE_SOURCE_LIMITS.maxBlobBytesPerRequest} bytes per request, ${WORKSPACE_SOURCE_LIMITS.maxBlobBytes} bytes per file. A part whose bytes hash to another SHA-256 refuses the whole request (\`400\`), nothing staged. Staged files are scoped to the workspace and the caller, capped at ${WORKSPACE_SOURCE_LIMITS.maxStagedBlobs} files and ${WORKSPACE_SOURCE_LIMITS.maxStagedBytes} bytes (\`413\` past either), and expire a day after their last use (staged, asked for, or applied); an apply does not remove them, so applying the same bytes again uploads nothing. Staging the same bytes again only renews them. Requires workspace write access. Returns \`403\` while workflows as code is not enabled.`,
+ description: `Stage the bytes of binary files (workspace files, knowledge documents) for a workspace source apply, as multipart form data: one file part per blob, its field name the SHA-256 of its bytes. At most ${WORKSPACE_SOURCE_LIMITS.maxBlobsPerRequest} files and ${WORKSPACE_SOURCE_LIMITS.maxBlobBytesPerRequest} bytes per request, ${WORKSPACE_SOURCE_LIMITS.maxBlobBytes} bytes per file; a larger file goes in a request of its own. A body over ${WORKSPACE_SOURCE_STAGE_MAX_BYTES} bytes answers \`413\` (\`PAYLOAD_TOO_LARGE\`) before it is read. A part whose bytes hash to another SHA-256 refuses the whole request (\`400\`), nothing staged. Staged files are scoped to the workspace and the caller, capped at ${WORKSPACE_SOURCE_LIMITS.maxStagedBlobs} files and ${WORKSPACE_SOURCE_LIMITS.maxStagedBytes} bytes (\`413\` with \`details.code\` \`SOURCE_STAGING_FULL\` past either), and expire a day after their last use (staged, asked for, or applied); an apply does not remove them, so applying the same bytes again uploads nothing. Staging the same bytes again only renews them. Requires workspace write access. Returns \`403\` while workflows as code is not enabled.`,
tags: ['Workflows'],
errors: [...RESOURCE_MUTATION_ERRORS, 'PayloadTooLarge'],
success: { description: 'The binary files staged.', headers: RATE_LIMIT_HEADERS },
diff --git a/apps/sim/lib/api/contracts/v2/workflow-source.ts b/apps/sim/lib/api/contracts/v2/workflow-source.ts
index 003f57b94a7..42b34189693 100644
--- a/apps/sim/lib/api/contracts/v2/workflow-source.ts
+++ b/apps/sim/lib/api/contracts/v2/workflow-source.ts
@@ -123,6 +123,10 @@ const blockRefSchema = z
})
.describe('A block by id and source key.')
+const typedBlockRefSchema = blockRefSchema
+ .extend({ type: z.string().describe('Block type.') })
+ .describe('An added or removed block by id, source key and type.')
+
const sourceDiffSchema = z
.object({
noop: z.boolean().describe('True when the source matches the saved draft.'),
@@ -132,8 +136,8 @@ const sourceDiffSchema = z
.describe('The workflow name and description fields the source changes.'),
blocks: z
.object({
- added: z.array(blockRefSchema).describe('New blocks.'),
- removed: z.array(blockRefSchema).describe('Deleted blocks.'),
+ added: z.array(typedBlockRefSchema).describe('New blocks.'),
+ removed: z.array(typedBlockRefSchema).describe('Deleted blocks.'),
renamed: z
.array(
z
diff --git a/apps/sim/lib/api/contracts/v2/workspace-source.ts b/apps/sim/lib/api/contracts/v2/workspace-source.ts
index f21adf6277e..4a6199e535a 100644
--- a/apps/sim/lib/api/contracts/v2/workspace-source.ts
+++ b/apps/sim/lib/api/contracts/v2/workspace-source.ts
@@ -354,7 +354,7 @@ export const v2StageWorkspaceSourceBlobsFormSchema = z
z.file().max(WORKSPACE_SOURCE_LIMITS.maxBlobBytes).describe('The bytes of one binary file.')
)
.describe(
- `One file part per binary file, named by its SHA-256: at most ${WORKSPACE_SOURCE_LIMITS.maxBlobsPerRequest} files and ${WORKSPACE_SOURCE_LIMITS.maxBlobBytesPerRequest} bytes per request.`
+ `One file part per binary file, named by its SHA-256: at most ${WORKSPACE_SOURCE_LIMITS.maxBlobsPerRequest} files and ${WORKSPACE_SOURCE_LIMITS.maxBlobBytesPerRequest} bytes per request, a larger file alone.`
)
/** `POST /api/v2/workspaces/{workspaceId}/source/blobs`: stage binary files for an apply. */
diff --git a/apps/sim/lib/table/application/reshape.ts b/apps/sim/lib/table/application/reshape.ts
index a97cff3853f..f61f12172e4 100644
--- a/apps/sim/lib/table/application/reshape.ts
+++ b/apps/sim/lib/table/application/reshape.ts
@@ -79,7 +79,12 @@ export interface ReshapeTableInput {
* Marks the rows as managed by workspace source, matched by the column named `keyColumn` as the
* reshape leaves the columns (`null` clears the mark); absent leaves it alone.
*/
- sourceRows?: { mode: 'managed'; keyColumn: string } | null
+ sourceRows?: { mode: 'managed'; keyColumn: string } | { mode: 'seed' } | null
+ /**
+ * Records each source column's key, by column name as the reshape leaves the columns
+ * (`metadata.sourceColumnKeys`, replaced whole); absent leaves the record alone.
+ */
+ sourceColumnKeys?: Record
/**
* The row version the caller read the rows at. Checked under the table's lock before anything
* changes: a row written since refuses the reshape (`TableRowsChangedError`), so `rows`
@@ -127,6 +132,7 @@ function sourceRowsMark(
idByName: ReadonlyMap
): TableSourceRows | null {
if (!input) return null
+ if (input.mode === 'seed') return input
const keyColumnId = idByName.get(input.keyColumn)
if (!keyColumnId) {
throw new OrchestrationError('validation', `Column "${input.keyColumn}" does not exist`)
@@ -143,6 +149,20 @@ function withSourceRows(
return sourceRows ? { ...rest, sourceRows } : rest
}
+/** Source column keys by column name, as `metadata.sourceColumnKeys` holds them: by column id. */
+function sourceColumnKeysById(
+ keysByName: Readonly>,
+ idByName: ReadonlyMap
+): Record {
+ const byId: Record = {}
+ for (const [name, key] of Object.entries(keysByName)) {
+ const id = idByName.get(name)
+ if (!id) throw new OrchestrationError('validation', `Column "${name}" does not exist`)
+ byId[id] = key
+ }
+ return byId
+}
+
/** A column name no column of `table` holds, to park a column under during a rename cycle. */
function freeColumnName(table: TableDefinition): string {
const taken = new Set(table.schema.columns.map((column) => column.name.toLowerCase()))
@@ -376,11 +396,17 @@ export const reshapeTableUseCase = defineAuthorizedTableUseCase({
})
}
const idByName = buildIdByName(table.schema)
- if (input.sourceRows !== undefined) {
- const metadata = withSourceRows(
- table.metadata,
- sourceRowsMark(input.sourceRows, idByName)
- )
+ if (input.sourceRows !== undefined || input.sourceColumnKeys !== undefined) {
+ let metadata: TableMetadata = table.metadata ?? {}
+ if (input.sourceRows !== undefined) {
+ metadata = withSourceRows(metadata, sourceRowsMark(input.sourceRows, idByName))
+ }
+ if (input.sourceColumnKeys !== undefined) {
+ metadata = {
+ ...metadata,
+ sourceColumnKeys: sourceColumnKeysById(input.sourceColumnKeys, idByName),
+ }
+ }
await tx.trx
.update(userTableDefinitions)
.set({ metadata })
diff --git a/apps/sim/lib/table/types.ts b/apps/sim/lib/table/types.ts
index 9e1abd5533c..7ad7a797cf5 100644
--- a/apps/sim/lib/table/types.ts
+++ b/apps/sim/lib/table/types.ts
@@ -299,18 +299,33 @@ export interface TableMetadata {
*/
hiddenColumns?: string[]
/**
- * Set while workspace source manages the table's rows (`rows: { mode: 'managed' }`), so reading
- * the workspace as source prints them. Written only by the reshape that applies them.
+ * Set while a workspace source declaration opts the table's rows in (`rows: { mode: 'managed' }`
+ * or `{ mode: 'seed' }`), so reading the workspace as source prints them, or for seed rows (which
+ * the directory keeps) the declaration that keeps its rows file. Written only by the reshape that
+ * applies them.
*/
sourceRows?: TableSourceRows
+ /**
+ * Each source column's key (`columns: { : ... }` in workspace source) by **column id**, as
+ * the workspace source write that last wrote the table declared it, so reading the table as
+ * source prints a column under its key even after a rename. Written only by that write; a column
+ * without an entry is keyed by its name.
+ */
+ sourceColumnKeys?: Record
}
-/** Rows workspace source manages: the whole table, each row matched by its key column. */
-export interface TableSourceRows {
- mode: 'managed'
- /** The id (`getColumnId`) of the unique column each row is matched by. */
- keyColumnId: string
-}
+/**
+ * Rows a workspace source declaration opts in: `managed`, the whole table, each row matched by its
+ * key column; `seed`, rows written into the table when source created it, which the directory
+ * keeps and the server does not.
+ */
+export type TableSourceRows =
+ | {
+ mode: 'managed'
+ /** The id (`getColumnId`) of the unique column each row is matched by. */
+ keyColumnId: string
+ }
+ | { mode: 'seed' }
/**
* The saved shape of a table view: everything `TableMetadata` covers plus the
diff --git a/apps/sim/lib/workflows/application/apply-workflow-source.fidelity.integration.ts b/apps/sim/lib/workflows/application/apply-workflow-source.fidelity.integration.ts
index 95f68dbb334..83472ee5cbf 100644
--- a/apps/sim/lib/workflows/application/apply-workflow-source.fidelity.integration.ts
+++ b/apps/sim/lib/workflows/application/apply-workflow-source.fidelity.integration.ts
@@ -483,6 +483,53 @@ check(
}
)
+check(
+ 'a field the model hides, added in source, is reported as dropped and apply keeps the stored value',
+ async () => {
+ const workflowId = await insertWorkflow('Fidelity hidden field')
+ const start = generateId()
+ const agentId = generateId()
+ await putState(workflowId, {
+ blocks: {
+ [start]: block(start, 'start_trigger', 'Start'),
+ [agentId]: {
+ ...block(agentId, 'agent', 'Answer', {
+ model: { id: 'model', type: 'combobox', value: 'gpt-5.4' },
+ }),
+ advancedMode: true,
+ },
+ },
+ edges: [
+ {
+ id: generateId(),
+ source: start,
+ target: agentId,
+ sourceHandle: 'source',
+ targetHandle: 'target',
+ },
+ ],
+ loops: {},
+ parallels: {},
+ variables: {},
+ })
+ const source = await getSource(workflowId)
+ expect(source.files['workflow.ts']).not.toContain('temperature')
+ const before = await snapshot(workflowId)
+
+ const result = await applySource(
+ source,
+ editEntry(source.files, "model: 'gpt-5.4' }", "model: 'gpt-5.4', temperature: 0.5 }")
+ )
+ const dropped = result.diagnostics.filter((d) => d.code === 'W_FIELD_NOT_APPLICABLE')
+ expect(dropped.map((d) => d.message)).toEqual([
+ 'temperature is hidden (model gpt-5.4 does not use it), so it never runs: an apply does not save it over the stored value, and the source it writes back drops this line.',
+ ])
+ expect(result.diff.noop).toBe(true)
+ expect(result.files['workflow.ts']).not.toContain('temperature')
+ expect(await snapshot(workflowId)).toEqual(before)
+ }
+)
+
check(
'a workflow built by editing operations and one created from its source read the same through the JSON API',
async () => {
@@ -530,7 +577,63 @@ check(
const canvas = byName(await readGraph(canvasId))
const fromSource = byName(await readGraph(created.workflowId))
- expect(findDifferences(canvas, fromSource, 20)).toEqual([])
+ // The loop's `iterations`/`loopType` subBlocks the editing operations leave beside
+ // `data.count` never run, so source does not carry them to the created workflow.
+ const differences = findDifferences(canvas, fromSource, 20).map((d) => d.path)
+ expect(differences.sort()).toEqual([
+ '/blocks/Retry/subBlocks/iterations',
+ '/blocks/Retry/subBlocks/loopType',
+ ])
+ }
+)
+
+check(
+ 'a loop the editing operations built prints no superseded subBlocks, and a no-op apply keeps them stored',
+ async () => {
+ const workflowId = await insertWorkflow('Fidelity operations loop')
+ const start = generateId()
+ const loop = generateId()
+ await applyOperations(workflowId, [
+ { operation_type: 'add', block_id: start, params: { type: 'start_trigger', name: 'Start' } },
+ {
+ operation_type: 'add',
+ block_id: loop,
+ params: { type: 'loop', name: 'Retry', inputs: { loopType: 'for', iterations: 3 } },
+ },
+ {
+ operation_type: 'insert_into_subflow',
+ block_id: generateId(),
+ params: {
+ subflowId: loop,
+ type: 'function',
+ name: 'Attempt',
+ inputs: { code: 'return 1' },
+ },
+ },
+ { operation_type: 'edit', block_id: start, params: { connections: { source: loop } } },
+ ])
+ const stored = (await readGraph(workflowId)).blocks[loop]
+ expect(Object.keys(stored.subBlocks ?? {}).sort()).toEqual(['iterations', 'loopType'])
+ expect(stored.data?.count).toBe(3)
+
+ const source = await getSource(workflowId)
+ expect(source.files['workflow.ts']).toContain("loop.for('Retry', { count: 3 }")
+ expect(source.files['workflow.ts']).not.toContain('iterations')
+ expect(source.files['workflow.ts']).not.toContain('extra')
+
+ const before = await snapshot(workflowId)
+ const unchanged = await applySource(source, source.files)
+ expect(unchanged.diff.noop).toBe(true)
+ expect(await snapshot(workflowId)).toEqual(before)
+
+ const counted = await applySource(
+ source,
+ editEntry(source.files, '{ count: 3 }', '{ count: 5 }')
+ )
+ expect(counted.applied).toBe(true)
+ const after = (await readGraph(workflowId)).blocks[loop]
+ expect(after.data?.count).toBe(5)
+ expect(after.subBlocks).toEqual(stored.subBlocks)
}
)
diff --git a/apps/sim/lib/workflows/application/apply-workflow-source.integration.ts b/apps/sim/lib/workflows/application/apply-workflow-source.integration.ts
index e2d516f70a7..9be934a0bd2 100644
--- a/apps/sim/lib/workflows/application/apply-workflow-source.integration.ts
+++ b/apps/sim/lib/workflows/application/apply-workflow-source.integration.ts
@@ -38,7 +38,7 @@ import { realtimeNotifyMock, realtimeNotifyMockFns } from '@sim/testing/mocks/re
import { getErrorMessage } from '@sim/utils/errors'
import { sleep } from '@sim/utils/helpers'
import { generateId } from '@sim/utils/id'
-import { isRecordLike } from '@sim/utils/object'
+import { isRecordLike, omit } from '@sim/utils/object'
import { SOURCE_LIMITS } from '@sim/workflow-code/limits'
import { saveWorkflowToNormalizedTables } from '@sim/workflow-persistence/save'
import { CONTAINER_DIMENSIONS } from '@sim/workflow-renderer/dimensions'
@@ -174,6 +174,14 @@ function apply(
)
}
+/** The 1-based line and column where `needle` first starts in `text`. */
+function positionOf(text: string, needle: string): { line: number; column: number } {
+ const index = text.indexOf(needle)
+ expect(index, `the text holds "${needle}"`).toBeGreaterThanOrEqual(0)
+ const before = text.slice(0, index)
+ return { line: before.split('\n').length, column: index - before.lastIndexOf('\n') }
+}
+
/** Replaces the one occurrence of `from` across the source files. */
function edit(files: Record, from: string, to: string): Record {
const hits = Object.entries(files).filter(([, text]) => text.split(from).length === 2)
@@ -1605,7 +1613,7 @@ describe('source apply edge cases', () => {
)
check(
- 'a workflow renamed in source to a name its folder holds is refused at the header',
+ 'a workflow renamed in source to a name its folder holds is refused at the name',
async () => {
const seeded = await seedWorkflow()
const other = await seedWorkflow()
@@ -1614,13 +1622,11 @@ describe('source apply edge cases', () => {
const taken = await storedName(other.workflowId)
const before = await snapshot(seeded.workflowId)
- const error = await expectRejected(
- apply(source, edit(source.files, `'${name}'`, `'${taken}'`)),
- 'E_NAME_TAKEN'
- )
+ const files = edit(source.files, `'${name}'`, `'${taken}'`)
+ const error = await expectRejected(apply(source, files), 'E_NAME_TAKEN')
expect(error.diagnostics.find((d) => d.code === 'E_NAME_TAKEN')).toMatchObject({
file: 'workflow.ts',
- line: expect.any(Number),
+ ...positionOf(files['workflow.ts'], `'${taken}'`),
})
expect(await storedName(seeded.workflowId)).toBe(name)
expect(await snapshot(seeded.workflowId)).toEqual(before)
@@ -1789,9 +1795,11 @@ export default workflow('', {}, () => {
}
const error = await expectRejected(apply(source, emptied), 'E_APPLY_BLOCK_DELETE')
- expect(error.diagnostics.find((d) => d.code === 'E_APPLY_BLOCK_DELETE')?.message).toBe(
- 'This apply deletes 2 blocks the source no longer declares: Start, Score.'
- )
+ expect(error.diagnostics.find((d) => d.code === 'E_APPLY_BLOCK_DELETE')).toMatchObject({
+ message: 'This apply deletes 2 blocks the source no longer declares: Start, Score.',
+ file: 'workflow.ts',
+ ...positionOf(emptied['workflow.ts'], "'Source apply edge"),
+ })
expect(await snapshot(workflowId)).toEqual(before)
const dryRun = await apply(source, emptied, { dryRun: true })
@@ -1971,7 +1979,11 @@ export default workflow('', {}, () => {
expect(result.dryRun).toBe(true)
expect(result.applied).toBe(false)
expect(result.diagnostics).toContainEqual(
- expect.objectContaining({ code: 'E_APPLY_STALE', severity: 'error' })
+ expect.objectContaining({
+ code: 'E_APPLY_STALE',
+ severity: 'error',
+ message: `Workflow "Source apply ${seeded.workflowId}" changed since this source was read. Get the source again, reapply your edit and retry.`,
+ })
)
expect(result.base.fingerprint).toBe(source.base.fingerprint)
expect(await snapshot(seeded.workflowId)).toEqual(before)
@@ -2199,3 +2211,268 @@ export default workflow('', {}, () => {
expect(await snapshot(seeded.workflowId)).toEqual(before)
})
})
+
+describe('renames in source', () => {
+ /** Start → Boom → Tally → Report, each a function; Report's code reads Boom and Tally by $ref. */
+ async function seedChain() {
+ const workflowId = generateId()
+ const now = new Date()
+ await db.insert(workflow).values({
+ id: workflowId,
+ userId: ownerId,
+ workspaceId,
+ name: `Source renames ${workflowId}`,
+ lastSynced: now,
+ createdAt: now,
+ updatedAt: now,
+ })
+ const fn = (name: string, code: string): BlockState => ({
+ id: generateId(),
+ type: 'function',
+ name,
+ position: { x: 0, y: 0 },
+ subBlocks: { code: { id: 'code', type: 'code', value: code } },
+ outputs: {},
+ enabled: true,
+ })
+ const start: BlockState = { ...fn('Start', ''), type: 'start_trigger', subBlocks: {} }
+ const boom = fn('Boom', 'return 1')
+ const tally = fn('Tally', 'return 2')
+ const report = fn('Report', 'return + ')
+ const chain = [start, boom, tally, report]
+ const saved = await saveWorkflowToNormalizedTables(workflowId, {
+ blocks: Object.fromEntries(chain.map((block) => [block.id, block])),
+ edges: chain.slice(1).map((block, index) => ({
+ id: generateId(),
+ source: chain[index].id,
+ target: block.id,
+ })),
+ loops: {},
+ parallels: {},
+ })
+ if (!saved.success) throw new Error(`Seeding the workflow failed: ${saved.error}`)
+ return { workflowId, ids: { boom: boom.id, tally: tally.id, report: report.id } }
+ }
+
+ const codes = (diagnostics: ReadonlyArray<{ code: string }>) => diagnostics.map((d) => d.code)
+
+ /** Replaces every whole-word `from` in `workflow.ts`. */
+ function renameIdent(files: Record, from: string, to: string) {
+ return {
+ ...files,
+ 'workflow.ts': files['workflow.ts'].replace(new RegExp(`\\b${from}\\b(?!\\.js)`, 'g'), to),
+ }
+ }
+
+ check('two display-name renames in one save keep both ids and resolve their $refs', async () => {
+ const seeded = await seedChain()
+ const source = await getSource(seeded.workflowId)
+ const files = edit(edit(source.files, "fn('Boom'", "fn('Bang'"), "fn('Tally'", "fn('Total'")
+
+ const dryRun = await apply(source, files, { dryRun: true })
+ expect(codes(dryRun.diagnostics)).toEqual([])
+ const result = await apply(source, files)
+
+ expect(result.applied).toBe(true)
+ expect(result.diff.blocks.removed).toEqual([])
+ expect(result.diff.blocks.renamed).toEqual([
+ { id: seeded.ids.boom, from: 'boom', to: 'bang' },
+ { id: seeded.ids.tally, from: 'tally', to: 'total' },
+ ])
+ const after = await snapshot(seeded.workflowId)
+ expect(after.state?.blocks[seeded.ids.report]?.subBlocks.code?.value).toBe(
+ 'return + '
+ )
+ expect(result.files['functions/report.js']).toBe('return $ref.bang.result + $ref.total.result')
+ })
+
+ check(
+ 'a block whose name and const both change while its wiring changes keeps its id',
+ async () => {
+ const seeded = await seedChain()
+ const source = await getSource(seeded.workflowId)
+ const files = edit(
+ edit(renameIdent(source.files, 'boom', 'bang'), "fn('Boom'", "fn('Bang'"),
+ ' tally.to(report)',
+ ' tally.to(report)\n bang.to(report)'
+ )
+
+ const result = await apply(source, files)
+
+ expect(result.applied).toBe(true)
+ expect(result.diff.blocks.removed).toEqual([])
+ expect(result.diff.blocks.renamed).toEqual([
+ { id: seeded.ids.boom, from: 'boom', to: 'bang' },
+ ])
+ expect(codes(result.diagnostics)).toContain('W_MAT_INFERRED_RENAME')
+ }
+ )
+
+ check('three blocks of one type renamed name and const in one save keep their ids', async () => {
+ const seeded = await seedChain()
+ const source = await getSource(seeded.workflowId)
+ let files = source.files
+ for (const [ident, name, newIdent, newName] of [
+ ['boom', 'Boom', 'bang', 'Bang'],
+ ['tally', 'Tally', 'total', 'Total'],
+ ['report', 'Report', 'summary', 'Summary'],
+ ]) {
+ files = edit(renameIdent(files, ident, newIdent), `fn('${name}'`, `fn('${newName}'`)
+ }
+
+ const result = await apply(source, files)
+
+ expect(result.applied).toBe(true)
+ expect(result.diff.blocks.removed).toEqual([])
+ expect(result.diff.blocks.renamed.map((r) => r.id).sort()).toEqual(
+ [seeded.ids.boom, seeded.ids.tally, seeded.ids.report].sort()
+ )
+ expect(codes(result.diagnostics)).not.toContain('W_UNRESOLVED_REF')
+ const after = await snapshot(seeded.workflowId)
+ expect(after.state?.blocks[seeded.ids.report]?.subBlocks.code?.value).toBe(
+ 'return + '
+ )
+ })
+
+ check(
+ 'renames nothing ties to the stored blocks are refused with how to keep the ids',
+ async () => {
+ const seeded = await seedChain()
+ const source = await getSource(seeded.workflowId)
+ let files = source.files
+ for (const [ident, name, newIdent, newName] of [
+ ['boom', 'Boom', 'bang', 'Bang'],
+ ['tally', 'Tally', 'total', 'Total'],
+ ]) {
+ files = edit(renameIdent(files, ident, newIdent), `fn('${name}'`, `fn('${newName}'`)
+ }
+ files = edit(edit(files, 'return 1', 'return 10'), 'return 2', 'return 20')
+ files = edit(
+ files,
+ ' start.to(bang)\n bang.to(total)\n total.to(report)',
+ ' start.to(report)\n report.to(bang)\n report.to(total)'
+ )
+
+ const dryRun = await apply(source, files, { dryRun: true })
+
+ const refusal = dryRun.diagnostics.find((d) => d.code === 'E_APPLY_BLOCK_DELETE')
+ expect(refusal?.message).toBe(
+ 'This apply deletes 2 blocks the source no longer declares: Boom, Tally. It adds Bang, Total of the same type: if those are the same blocks renamed, both their name and const changed, so nothing ties them to the stored blocks and they would get new ids.'
+ )
+ expect(refusal?.hint).toBe(
+ 'To rename a block and keep its id, keep its const name in this apply (change it in the next), or rename the references to it with it; to replace it with a new block, pass --allow-delete.'
+ )
+
+ const allowed = await apply(source, files, { dryRun: true, allowDelete: true })
+ expect(allowed.diff.blocks.removed.map(({ key, type }) => `${key} ${type}`)).toEqual([
+ 'boom function',
+ 'tally function',
+ ])
+ expect(allowed.diff.blocks.added.map(({ key, type }) => `${key} ${type}`)).toEqual([
+ 'bang function',
+ 'total function',
+ ])
+ expect(
+ allowed.diagnostics
+ .filter((d) => d.code === 'W_APPLY_PROBABLE_RENAME')
+ .map((d) => d.message)
+ ).toEqual([
+ 'Boom (function) is deleted and Bang (function) added: probably a rename, but its name and const both changed, so it gets a new id and loses its run history and webhook URL.',
+ 'Tally (function) is deleted and Total (function) added: probably a rename, but its name and const both changed, so it gets a new id and loses its run history and webhook URL.',
+ ])
+ }
+ )
+
+ check(
+ 'an unrelated delete and add of one type is refused without --allow-delete, keeping no id',
+ async () => {
+ const seeded = await seedChain()
+ const source = await getSource(seeded.workflowId)
+ const files = omit, string>(
+ {
+ ...edit(
+ edit(
+ edit(
+ source.files,
+ " const tally = fn('Tally', { language: null, code: codeFile('./functions/tally.js') })\n",
+ " const fresh = fn('Fresh', { code: js`return 'unrelated'` })\n"
+ ),
+ ' boom.to(tally)\n tally.to(report)',
+ ' boom.to(report)\n start.to(fresh)'
+ ),
+ 'import { codeFile, fn,',
+ 'import { codeFile, fn, js,'
+ ),
+ 'functions/report.js': 'return $ref.boom.result',
+ },
+ ['functions/tally.js']
+ )
+
+ const dryRun = await apply(source, files, { dryRun: true })
+
+ expect(codes(dryRun.diagnostics)).toContain('E_APPLY_BLOCK_DELETE')
+ expect(codes(dryRun.diagnostics)).not.toContain('W_MAT_INFERRED_RENAME')
+ const allowed = await apply(source, files, { allowDelete: true })
+ expect(allowed.diff.blocks.removed.map((block) => block.id)).toEqual([seeded.ids.tally])
+ }
+ )
+
+ check(
+ 'a rename of name and const that rewires, changes its code and renames its references keeps its id',
+ async () => {
+ const seeded = await seedChain()
+ const source = await getSource(seeded.workflowId)
+ const renamed = edit(renameIdent(source.files, 'tally', 'total'), "fn('Tally'", "fn('Total'")
+ const files = {
+ ...edit(renamed, ' boom.to(total)\n', ' start.to(total)\n'),
+ 'functions/tally.js': 'return 3',
+ 'functions/report.js': 'return $ref.boom.result + $ref.total.result',
+ }
+
+ const result = await apply(source, files)
+
+ expect(result.applied).toBe(true)
+ expect(result.diff.blocks.removed).toEqual([])
+ expect(result.diff.blocks.renamed).toEqual([
+ { id: seeded.ids.tally, from: 'tally', to: 'total' },
+ ])
+ expect(result.diagnostics.find((d) => d.code === 'W_MAT_INFERRED_RENAME')?.message).toContain(
+ 'the references to it renamed with it'
+ )
+ }
+ )
+
+ check('a trigger renamed in name and const with changed settings is refused', async () => {
+ const seeded = await seedChain()
+ const source = await getSource(seeded.workflowId)
+ const files = edit(
+ edit(
+ source.files,
+ "const start = trigger.start('Start', {})",
+ "const begin = trigger.start('Begin', { inputFormat: { question: 'string' } })"
+ ),
+ ' start.to(boom)',
+ ' begin.to(boom)'
+ )
+
+ const dryRun = await apply(source, files, { dryRun: true })
+
+ expect(codes(dryRun.diagnostics)).toContain('E_APPLY_BLOCK_DELETE')
+ expect(codes(dryRun.diagnostics)).not.toContain('W_MAT_INFERRED_RENAME')
+ })
+
+ check(
+ 'a $ref to a missing block in a function file is the same warning in check and dry run, at the file',
+ async () => {
+ const seeded = await seedChain()
+ const source = await getSource(seeded.workflowId)
+ const files = { ...source.files, 'functions/boom.js': 'return $ref.nosuchblock.value\n' }
+
+ const dryRun = await apply(source, files, { dryRun: true })
+
+ expect(
+ dryRun.diagnostics.map((d) => ({ code: d.code, severity: d.severity, file: d.file }))
+ ).toEqual([{ code: 'W_UNRESOLVED_REF', severity: 'warning', file: 'functions/boom.js' }])
+ }
+ )
+})
diff --git a/apps/sim/lib/workflows/application/apply-workflow-source.ts b/apps/sim/lib/workflows/application/apply-workflow-source.ts
index a8f7f34280d..428edf4a5e5 100644
--- a/apps/sim/lib/workflows/application/apply-workflow-source.ts
+++ b/apps/sim/lib/workflows/application/apply-workflow-source.ts
@@ -49,10 +49,10 @@ import {
} from '@/lib/workflows/application/source-commit'
import { emptyDiffSummary, type SourceDiffSummary } from '@/lib/workflows/application/source-diff'
import {
- WORKFLOW_SOURCE_STALE_MESSAGE,
type WorkflowSourceDiagnostic,
WorkflowSourceRejectedError,
WorkflowSourceStaleError,
+ workflowSourceStaleMessage,
} from '@/lib/workflows/application/source-error'
import {
evaluateSource,
@@ -559,12 +559,12 @@ async function runSourceApply(
])
}
if (target.kind === 'existing' && storedFingerprint !== target.base.fingerprint) {
- if (!dryRun) throw new WorkflowSourceStaleError(storedFingerprint)
+ if (!dryRun) throw new WorkflowSourceStaleError(storedFingerprint, stored.meta.name)
return refuse([
{
code: 'E_APPLY_STALE',
severity: 'error',
- message: WORKFLOW_SOURCE_STALE_MESSAGE,
+ message: workflowSourceStaleMessage(stored.meta.name),
},
])
}
diff --git a/apps/sim/lib/workflows/application/block-visibility-writes.integration.ts b/apps/sim/lib/workflows/application/block-visibility-writes.integration.ts
new file mode 100644
index 00000000000..d42554fe47a
--- /dev/null
+++ b/apps/sim/lib/workflows/application/block-visibility-writes.integration.ts
@@ -0,0 +1,402 @@
+/**
+ * Every write that places a block, against real PostgreSQL, under a `block-visibility` document
+ * that kill-switches `api` for everyone but platform admins and one allowlisted user. A caller who
+ * cannot see a type cannot add it (source apply, editing operations, state replace), a block
+ * already of that type stays editable, and a caller the rule reveals it to adds it. The document
+ * goes through the real evaluation and the admin flag through the real user row; only the
+ * AppConfig fetch is replaced.
+ */
+import { mkdir, writeFile } from 'node:fs/promises'
+import { dirname, resolve } from 'node:path'
+import { db } from '@sim/db'
+import { member, organization, permissions, user, workflow, workspace } from '@sim/db/schema'
+import { deleteWorkspaceFixture, insertWorkspaceFixture } from '@sim/db/testing/workspace-fixtures'
+import { createSessionPrincipal } from '@sim/testing/factories/principal.factory'
+import { envFlagsMock, resetEnvFlagsMock, setEnvFlags } from '@sim/testing/mocks/env-flags.mock'
+import { featureFlagsMock, featureFlagsMockFns } from '@sim/testing/mocks/feature-flags.mock'
+import { realtimeNotifyMock } from '@sim/testing/mocks/realtime-notify.mock'
+import { getErrorMessage } from '@sim/utils/errors'
+import { generateId } from '@sim/utils/id'
+import { saveWorkflowToNormalizedTables } from '@sim/workflow-persistence/save'
+import type { BlockState } from '@sim/workflow-types/workflow'
+import { eq, inArray } from 'drizzle-orm'
+import { afterAll, beforeAll, expect, it, vi } from 'vitest'
+
+const { mockFetchProfile } = vi.hoisted(() => ({ mockFetchProfile: vi.fn() }))
+
+vi.mock('@/lib/core/config/env-flags', () => envFlagsMock)
+vi.mock('@/lib/core/config/feature-flags', () => featureFlagsMock)
+vi.mock('@/lib/core/config/appconfig', () => ({ fetchAppConfigProfile: mockFetchProfile }))
+vi.mock('@/lib/realtime/notify', () => realtimeNotifyMock)
+
+import { withPermissionGroupScope } from '@/lib/permission-groups/request-scope.server'
+import { applyWorkflowOperations } from '@/lib/workflows/application/apply-workflow-operations'
+import { applyWorkflowSource } from '@/lib/workflows/application/apply-workflow-source'
+import { readWorkflowSource } from '@/lib/workflows/application/read-workflow-source'
+import { replaceWorkflowState } from '@/lib/workflows/application/replace-workflow-state'
+import { WorkflowSourceRejectedError } from '@/lib/workflows/application/source-error'
+import { loadWorkflowFromNormalizedTables } from '@/lib/workflows/persistence/utils'
+
+const organizationId = generateId()
+const adminId = generateId()
+const memberId = generateId()
+const revealedId = generateId()
+const workspaceId = generateId()
+const userIds = [adminId, memberId, revealedId]
+const admin = createSessionPrincipal({ userId: adminId, sessionId: generateId() })
+const hiddenFrom = createSessionPrincipal({ userId: memberId, sessionId: generateId() })
+const revealedTo = createSessionPrincipal({ userId: revealedId, sessionId: generateId() })
+
+type Principal = typeof admin
+
+const VISIBILITY_DOCUMENT = {
+ api: { enabled: false, adminEnabled: true, userIds: [revealedId] },
+}
+
+const checks: { name: string; status: string; durationMs: number; error?: string }[] = []
+
+function check(name: string, run: () => Promise) {
+ it(name, async () => {
+ const started = performance.now()
+ try {
+ await run()
+ checks.push({ name, status: 'passed', durationMs: performance.now() - started })
+ } catch (error) {
+ checks.push({
+ name,
+ status: 'failed',
+ durationMs: performance.now() - started,
+ error: getErrorMessage(error),
+ })
+ throw error
+ }
+ })
+}
+
+function seedBlock(
+ type: string,
+ name: string,
+ subBlocks: Record = {}
+): BlockState {
+ return {
+ id: generateId(),
+ type,
+ name,
+ position: { x: 0, y: 0 },
+ subBlocks: Object.fromEntries(
+ Object.entries(subBlocks).map(([key, [subType, value]]) => [
+ key,
+ { id: key, type: subType, value },
+ ])
+ ) as BlockState['subBlocks'],
+ outputs: {},
+ enabled: true,
+ }
+}
+
+function seedApi(url: string): BlockState {
+ return seedBlock('api', 'Fetch', {
+ url: ['short-input', url],
+ method: ['dropdown', 'GET'],
+ })
+}
+
+/** A workflow whose first block feeds every other block. */
+async function seedGraph(blocks: BlockState[]): Promise {
+ const workflowId = generateId()
+ const now = new Date()
+ await db.insert(workflow).values({
+ id: workflowId,
+ userId: adminId,
+ workspaceId,
+ name: `Block visibility ${workflowId}`,
+ lastSynced: now,
+ createdAt: now,
+ updatedAt: now,
+ })
+ const [first, ...rest] = blocks
+ const saved = await saveWorkflowToNormalizedTables(workflowId, {
+ blocks: Object.fromEntries(blocks.map((block) => [block.id, block])),
+ edges: rest.map((block) => ({ id: generateId(), source: first.id, target: block.id })),
+ loops: {},
+ parallels: {},
+ })
+ if (!saved.success) throw new Error(`Seeding the workflow failed: ${saved.error}`)
+ return workflowId
+}
+
+function seedPlainGraph(): Promise {
+ return seedGraph([
+ seedBlock('start_trigger', 'Start'),
+ seedBlock('function', 'Score', { code: ['code', 'return 1'] }),
+ ])
+}
+
+function getSource(principal: Principal, workflowId: string) {
+ return withPermissionGroupScope(() =>
+ readWorkflowSource.execute({ principal, input: { workflowId, functionCode: 'inline' } })
+ )
+}
+
+function applySource(
+ principal: Principal,
+ source: Awaited>,
+ files: Record
+) {
+ return withPermissionGroupScope(() =>
+ applyWorkflowSource.execute({
+ principal,
+ input: { workflowId: source.workflowId, files, lock: source.lock, base: source.base },
+ })
+ )
+}
+
+/** The source with a generic `api` block declared just before the workflow body closes. */
+function withAddedApi(files: Record): Record {
+ const text = files['workflow.ts'].replace(/import \{ /, 'import { block, ')
+ const close = text.lastIndexOf('})')
+ expect(close, 'workflow.ts closes its workflow body').toBeGreaterThan(0)
+ const added =
+ " block('api', 'Added fetch', { url: 'https://example.com/added', method: 'GET' })\n"
+ return { ...files, 'workflow.ts': `${text.slice(0, close)}${added}${text.slice(close)}` }
+}
+
+function edit(files: Record, from: string, to: string): Record {
+ const text = files['workflow.ts']
+ expect(text.split(from), `workflow.ts holds "${from}" once`).toHaveLength(2)
+ return { ...files, 'workflow.ts': text.replace(from, to) }
+}
+
+function addApiOperation(principal: Principal, workflowId: string) {
+ return withPermissionGroupScope(() =>
+ applyWorkflowOperations.execute({
+ principal,
+ input: {
+ workflowId,
+ layout: 'none',
+ operations: [
+ {
+ operation_type: 'add',
+ block_id: generateId(),
+ params: {
+ type: 'api',
+ name: 'Added fetch',
+ inputs: { url: 'https://example.com/added', method: 'GET' },
+ },
+ },
+ ],
+ },
+ })
+ )
+}
+
+function replaceState(
+ principal: Principal,
+ workflowId: string,
+ stored: NonNullable>>,
+ blocks: Record
+) {
+ return withPermissionGroupScope(() =>
+ replaceWorkflowState.execute({
+ principal,
+ input: { workflowId, blocks, edges: stored.edges },
+ })
+ )
+}
+
+async function storedTypes(workflowId: string): Promise {
+ const stored = await loadWorkflowFromNormalizedTables(workflowId)
+ return Object.values(stored?.blocks ?? {}).map((block) => block.type)
+}
+
+async function loadStored(workflowId: string) {
+ const stored = await loadWorkflowFromNormalizedTables(workflowId)
+ if (!stored) throw new Error('The seeded workflow has no stored graph')
+ return stored
+}
+
+beforeAll(async () => {
+ featureFlagsMockFns.mockIsFeatureEnabled.mockImplementation(
+ async (flag: string) => flag === 'workflow-code'
+ )
+ setEnvFlags({ isAppConfigEnabled: true })
+ mockFetchProfile.mockImplementation(
+ async (ids: { profile: string }, parse: (doc: unknown) => unknown) =>
+ parse(ids.profile === 'block-visibility' ? VISIBILITY_DOCUMENT : {})
+ )
+ const now = new Date()
+ await db.insert(user).values(
+ userIds.map((id) => ({
+ id,
+ name: 'Block visibility fixture',
+ email: `${id}@block-visibility.invalid`,
+ emailVerified: true,
+ role: id === adminId ? 'admin' : 'user',
+ createdAt: now,
+ updatedAt: now,
+ }))
+ )
+ await db.insert(organization).values({
+ id: organizationId,
+ name: 'Block visibility fixture',
+ slug: organizationId,
+ createdAt: now,
+ updatedAt: now,
+ })
+ await db.insert(member).values(
+ userIds.map((userId) => ({
+ id: generateId(),
+ userId,
+ organizationId,
+ role: userId === adminId ? 'owner' : 'member',
+ createdAt: now,
+ }))
+ )
+ await insertWorkspaceFixture(db, [
+ {
+ id: workspaceId,
+ name: 'Block visibility fixture',
+ ownerId: adminId,
+ billedAccountUserId: adminId,
+ organizationId,
+ workspaceMode: 'organization' as const,
+ },
+ ])
+ await db.insert(permissions).values(
+ userIds.map((userId) => ({
+ id: generateId(),
+ userId,
+ entityType: 'workspace',
+ entityId: workspaceId,
+ permissionType: userId === adminId ? ('admin' as const) : ('write' as const),
+ }))
+ )
+})
+
+afterAll(async () => {
+ const reportPath =
+ process.env.BLOCK_VISIBILITY_WRITES_REPORT_PATH ??
+ resolve('test-results/block-visibility-writes.json')
+ await mkdir(dirname(reportPath), { recursive: true })
+ await writeFile(reportPath, JSON.stringify({ checks }, null, 2))
+ resetEnvFlagsMock()
+ await deleteWorkspaceFixture(db, eq(workspace.organizationId, organizationId))
+ await db.delete(organization).where(eq(organization.id, organizationId))
+ await db.delete(user).where(inArray(user.id, userIds))
+})
+
+check('source apply refuses adding a kill-switched type the caller cannot see', async () => {
+ const workflowId = await seedPlainGraph()
+ const source = await getSource(hiddenFrom, workflowId)
+ const before = await loadWorkflowFromNormalizedTables(workflowId)
+
+ const refused = await applySource(hiddenFrom, source, withAddedApi(source.files)).then(
+ () => null,
+ (error: unknown) => error
+ )
+
+ expect(refused).toBeInstanceOf(WorkflowSourceRejectedError)
+ expect((refused as WorkflowSourceRejectedError).diagnostics).toContainEqual(
+ expect.objectContaining({
+ code: 'E_APPLY_BLOCK_NOT_ALLOWED',
+ message: expect.stringContaining('Block type "api" is not available to you'),
+ })
+ )
+ expect(await loadWorkflowFromNormalizedTables(workflowId)).toEqual(before)
+})
+
+check('editing operations withhold adding a kill-switched type the caller cannot see', async () => {
+ const workflowId = await seedPlainGraph()
+
+ const result = await addApiOperation(hiddenFrom, workflowId)
+
+ expect(result.skipped).toContainEqual(
+ expect.objectContaining({
+ type: 'block_not_allowed',
+ reason: expect.stringContaining('Block type "api" is not available to you'),
+ })
+ )
+ expect(await storedTypes(workflowId)).not.toContain('api')
+})
+
+check(
+ 'state replace refuses a graph adding a kill-switched type the caller cannot see',
+ async () => {
+ const workflowId = await seedPlainGraph()
+ const stored = await loadStored(workflowId)
+ const added = seedApi('https://example.com/added')
+
+ await expect(
+ replaceState(hiddenFrom, workflowId, stored, { ...stored.blocks, [added.id]: added })
+ ).rejects.toMatchObject({
+ code: 'forbidden',
+ message: expect.stringContaining('Block type "api" is not available to you'),
+ })
+ expect(await storedTypes(workflowId)).not.toContain('api')
+ }
+)
+
+check('a placed block of a hidden type stays editable on every write path', async () => {
+ const placed = seedApi('https://example.com/before')
+ const workflowId = await seedGraph([seedBlock('start_trigger', 'Start'), placed])
+
+ const source = await getSource(hiddenFrom, workflowId)
+ const applied = await applySource(
+ hiddenFrom,
+ source,
+ edit(source.files, 'https://example.com/before', 'https://example.com/source')
+ )
+ expect(applied.applied).toBe(true)
+
+ const operated = await withPermissionGroupScope(() =>
+ applyWorkflowOperations.execute({
+ principal: hiddenFrom,
+ input: {
+ workflowId,
+ layout: 'none',
+ operations: [
+ {
+ operation_type: 'edit',
+ block_id: placed.id,
+ params: { type: 'api', inputs: { url: 'https://example.com/operations' } },
+ },
+ ],
+ },
+ })
+ )
+ expect(operated.skipped).toEqual([])
+ expect((await loadStored(workflowId)).blocks[placed.id]?.subBlocks.url?.value).toBe(
+ 'https://example.com/operations'
+ )
+
+ const stored = await loadStored(workflowId)
+ const replaced = structuredClone(stored.blocks)
+ replaced[placed.id].subBlocks.url.value = 'https://example.com/state'
+ await replaceState(hiddenFrom, workflowId, stored, replaced)
+ expect((await loadStored(workflowId)).blocks[placed.id]?.subBlocks.url?.value).toBe(
+ 'https://example.com/state'
+ )
+})
+
+check('a platform admin the rule reveals the type to adds it through source', async () => {
+ const workflowId = await seedPlainGraph()
+ const source = await getSource(admin, workflowId)
+
+ const result = await applySource(admin, source, withAddedApi(source.files))
+
+ expect(result.applied).toBe(true)
+ expect(await storedTypes(workflowId)).toContain('api')
+})
+
+check('an allowlisted user adds the type through operations and state replace', async () => {
+ const workflowId = await seedPlainGraph()
+
+ const result = await addApiOperation(revealedTo, workflowId)
+ expect(result.skipped).toEqual([])
+ expect(await storedTypes(workflowId)).toContain('api')
+
+ const stored = await loadStored(workflowId)
+ const added = seedApi('https://example.com/second')
+ await replaceState(revealedTo, workflowId, stored, { ...stored.blocks, [added.id]: added })
+ expect((await storedTypes(workflowId)).filter((type) => type === 'api')).toHaveLength(2)
+})
diff --git a/apps/sim/lib/workflows/application/replace-workflow-state.ts b/apps/sim/lib/workflows/application/replace-workflow-state.ts
index 4bcafb107b9..2fd779f4487 100644
--- a/apps/sim/lib/workflows/application/replace-workflow-state.ts
+++ b/apps/sim/lib/workflows/application/replace-workflow-state.ts
@@ -10,6 +10,7 @@ import { createLogger } from '@sim/logger'
import { evaluateSubBlockCondition } from '@sim/workflow-code/catalog/canonical-index'
import type { BlockState, WorkflowState } from '@sim/workflow-types/workflow'
import { principalAuditSource } from '@/lib/core/application'
+import { getBlockVisibility } from '@/lib/core/config/block-visibility'
import { OrchestrationError } from '@/lib/core/orchestration/types'
import { notifyWorkflowUpdated } from '@/lib/realtime/notify'
import { defineAuthorizedWorkflowUseCase } from '@/lib/workflows/application/authorized-workflow-use-case'
@@ -36,6 +37,7 @@ import {
validateToolBindingAuthoring,
} from '@/lib/workflows/tool-input/authoring'
import { getBlock } from '@/blocks/registry'
+import { isHiddenUnder } from '@/blocks/visibility/context'
const logger = createLogger('ReplaceWorkflowState')
@@ -61,6 +63,36 @@ function humanSubjectUserId(principal: Principal): string | null {
}
}
+/**
+ * Refuses a graph that places a block type the acting user cannot see (an unrevealed `preview`
+ * block or a kill-switched one): on a block the stored graph lacks, or one it holds with another
+ * type. The editing operations and source apply refuse the same placement through their
+ * visibility gate; a block the stored graph already holds with that type stays writable, so a
+ * workflow keeps saving after its type is hidden. The stored graph is read only when the body
+ * names a hidden type at all.
+ */
+async function assertNoHiddenPlacement(
+ blocks: Record,
+ context: { workflowId: string; workspaceOrganizationId: string | null },
+ subjectUserId: string | null,
+ previous: Awaited> | undefined
+): Promise {
+ const visibility = await getBlockVisibility({
+ ...(subjectUserId ? { userId: subjectUserId } : {}),
+ ...(context.workspaceOrganizationId ? { orgId: context.workspaceOrganizationId } : {}),
+ })
+ const hidden = Object.values(blocks).filter((block) => {
+ const config = getBlock(block.type)
+ return config !== undefined && isHiddenUnder(visibility, config)
+ })
+ if (hidden.length === 0) return
+ const stored = previous ?? (await loadWorkflowFromNormalizedTables(context.workflowId))
+ const placed = hidden.find((block) => stored?.blocks[block.id]?.type !== block.type)
+ if (placed) {
+ throw new OrchestrationError('forbidden', `Block type "${placed.type}" is not available to you`)
+ }
+}
+
export interface ReplaceWorkflowStateInput {
workflowId: string
assertedWorkspaceId?: string
@@ -177,6 +209,9 @@ export const replaceWorkflowState = defineAuthorizedWorkflowUseCase({
}
}
+ const subjectUserId = humanSubjectUserId(principal)
+ await assertNoHiddenPlacement(blocks, context, subjectUserId, previous)
+
const graph = {
blocks,
edges: sanitized.edges as WorkflowState['edges'],
@@ -189,8 +224,6 @@ export const replaceWorkflowState = defineAuthorizedWorkflowUseCase({
* the reference pass is skipped for them rather than resolved against the
* billing owner. See {@link buildWorkflowLintReport}.
*/
- const subjectUserId = humanSubjectUserId(principal)
-
const lint = await buildWorkflowLintReport(graph, {
workflowId: context.workflowId,
workspaceId: context.workspaceId,
diff --git a/apps/sim/lib/workflows/application/source-analytics.test.ts b/apps/sim/lib/workflows/application/source-analytics.test.ts
index 742fdd95586..29edc1a30ea 100644
--- a/apps/sim/lib/workflows/application/source-analytics.test.ts
+++ b/apps/sim/lib/workflows/application/source-analytics.test.ts
@@ -35,7 +35,7 @@ describe('recordSourceRefused', () => {
beforeEach(() => capture.mockClear())
it.each([
- ['stale', new WorkflowSourceStaleError('sf1:x')],
+ ['stale', new WorkflowSourceStaleError('sf1:x', 'Lead Scoring')],
['rejected', new WorkflowSourceRejectedError([])],
['locked', new OrchestrationError('locked', 'Workflow is locked')],
])('records a %s refusal as ok:false with its error_kind', (kind, error) => {
@@ -74,7 +74,7 @@ describe('recordSourceRefused', () => {
principal: createWorkspaceApiKeyPrincipal({ workspaceId: 'ws-1' }),
workflowId: 'wf-1',
},
- new WorkflowSourceStaleError('sf1:x')
+ new WorkflowSourceStaleError('sf1:x', 'Lead Scoring')
)
expect(capture).not.toHaveBeenCalled()
@@ -86,7 +86,7 @@ describe('recordSourceWritten', () => {
it('counts a create from source as a workflow creation, attributed to the Copilot subject', () => {
const diff = emptyDiffSummary()
- diff.blocks.added.push({ id: 'b-1', key: 'start' })
+ diff.blocks.added.push({ id: 'b-1', key: 'start', type: 'start_trigger' })
recordSourceWritten(
{
@@ -125,7 +125,7 @@ describe('recordSourceDryRun', () => {
it('records a clean dry run as ok with the changes a commit would make', () => {
const diff = emptyDiffSummary()
- diff.blocks.added.push({ id: 'b-1', key: 'start' })
+ diff.blocks.added.push({ id: 'b-1', key: 'start', type: 'start_trigger' })
diff.edges.added = 2
recordSourceDryRun(
diff --git a/apps/sim/lib/workflows/application/source-commit.ts b/apps/sim/lib/workflows/application/source-commit.ts
index 603d1cd6e24..5f86fc8a561 100644
--- a/apps/sim/lib/workflows/application/source-commit.ts
+++ b/apps/sim/lib/workflows/application/source-commit.ts
@@ -227,9 +227,11 @@ async function recheckAndWriteSourceMeta(
canonicalize(current.state, { catalog, meta: current.meta, resources: plan.resources }),
plan.resources
)
- if (fingerprint !== args.storedFingerprint) throw new WorkflowSourceStaleError(fingerprint)
+ if (fingerprint !== args.storedFingerprint) {
+ throw new WorkflowSourceStaleError(fingerprint, current.meta.name)
+ }
if (options.layoutOnly && !isEqual(withoutBlockLayout(current), withoutBlockLayout(stored))) {
- throw new WorkflowSourceStaleError(fingerprint)
+ throw new WorkflowSourceStaleError(fingerprint, current.meta.name)
}
}
await writeWorkflowSourceMeta(
diff --git a/apps/sim/lib/workflows/application/source-diff.ts b/apps/sim/lib/workflows/application/source-diff.ts
index 305ad10d919..7ba815b8cb2 100644
--- a/apps/sim/lib/workflows/application/source-diff.ts
+++ b/apps/sim/lib/workflows/application/source-diff.ts
@@ -1,16 +1,21 @@
-import type { CanonicalDiff, LockIdentity } from '@sim/workflow-code'
+import type { CanonicalDiff, CanonicalWorkflow, LockIdentity } from '@sim/workflow-code'
interface SourceBlockRef {
id: string
key: string
}
+/** An added or removed block, with its type so a reader can spot a probable rename. */
+interface TypedSourceBlockRef extends SourceBlockRef {
+ type: string
+}
+
/** What the apply changed (or, on a dry run, would change), by stored id and source key. */
export interface SourceDiffSummary {
noop: boolean
blocks: {
- added: SourceBlockRef[]
- removed: SourceBlockRef[]
+ added: TypedSourceBlockRef[]
+ removed: TypedSourceBlockRef[]
renamed: Array<{ id: string; from: string; to: string }>
modified: Array
}
@@ -26,19 +31,27 @@ export interface SourceDiffSummary {
*/
export function summarizeDiff(
diff: CanonicalDiff,
- baseIdentity: LockIdentity,
- targetIdentity: LockIdentity,
+ base: { workflow: CanonicalWorkflow; identity: LockIdentity },
+ target: { workflow: CanonicalWorkflow; identity: LockIdentity },
noop: boolean,
meta: ReadonlyArray<'name' | 'description'> = []
): SourceDiffSummary {
- const targetId = (key: string) => targetIdentity.blocks[key]?.id ?? ''
- const baseId = (key: string) => baseIdentity.blocks[key]?.id ?? ''
+ const targetId = (key: string) => target.identity.blocks[key]?.id ?? ''
+ const baseId = (key: string) => base.identity.blocks[key]?.id ?? ''
return {
noop: noop && meta.length === 0,
...(meta.length > 0 ? { meta: [...meta] } : {}),
blocks: {
- added: diff.blocks.added.map((key) => ({ id: targetId(key), key })),
- removed: diff.blocks.removed.map((key) => ({ id: baseId(key), key })),
+ added: diff.blocks.added.map((key) => ({
+ id: targetId(key),
+ key,
+ type: target.workflow.blocks[key]?.type ?? '',
+ })),
+ removed: diff.blocks.removed.map((key) => ({
+ id: baseId(key),
+ key,
+ type: base.workflow.blocks[key]?.type ?? '',
+ })),
renamed: diff.blocks.renamed.map(({ from, to }) => ({ id: targetId(to), from, to })),
modified: diff.blocks.modified.map((change) => {
const fields = change.changes.map((item) =>
diff --git a/apps/sim/lib/workflows/application/source-error.ts b/apps/sim/lib/workflows/application/source-error.ts
index b7b69406c4c..aa1ee7724e4 100644
--- a/apps/sim/lib/workflows/application/source-error.ts
+++ b/apps/sim/lib/workflows/application/source-error.ts
@@ -49,13 +49,20 @@ export class WorkflowSourceRejectedError extends OrchestrationError {
* edited the workflow, toggled a block lock, or a migration changed it. Re-read the source,
* rebase the edit onto it and apply again.
*/
-/** Why a source apply refuses a draft changed since its base, as both its 409 and dry run say. */
-export const WORKFLOW_SOURCE_STALE_MESSAGE =
- 'The workflow changed since this source was read. Get the source again, reapply your edit and retry.'
+/**
+ * Why a source apply refuses a draft changed since its base, as both its 409 and dry run say,
+ * naming the workflow so a workspace apply's line for it says which one.
+ */
+export function workflowSourceStaleMessage(workflowName: string): string {
+ return `Workflow ${JSON.stringify(workflowName)} changed since this source was read. Get the source again, reapply your edit and retry.`
+}
export class WorkflowSourceStaleError extends OrchestrationError {
- constructor(readonly currentFingerprint: string) {
- super('conflict', WORKFLOW_SOURCE_STALE_MESSAGE)
+ constructor(
+ readonly currentFingerprint: string,
+ workflowName: string
+ ) {
+ super('conflict', workflowSourceStaleMessage(workflowName))
this.name = 'WorkflowSourceStaleError'
}
}
diff --git a/apps/sim/lib/workflows/application/source-evaluate.ts b/apps/sim/lib/workflows/application/source-evaluate.ts
index 7892e1ecef7..d794cb00cc3 100644
--- a/apps/sim/lib/workflows/application/source-evaluate.ts
+++ b/apps/sim/lib/workflows/application/source-evaluate.ts
@@ -4,10 +4,13 @@ import { workflowBlocks } from '@sim/db/schema'
import { isRecordLike } from '@sim/utils/object'
import {
type BlockCatalog,
+ type CanonicalBlock,
type CanonicalWorkflow,
type CanonicalWorkflowMeta,
+ type CompileResult,
canonicalize,
canonicalizeUnderLock,
+ compileWorkflow,
type Diagnostic,
diffCanonical,
encodeAnchor,
@@ -20,6 +23,7 @@ import {
renamesBetween,
type SourceBase,
type SourceHints,
+ type SourceSpan,
type WorkflowLockBlock,
type WorkflowLockV1,
type WorkflowSourceMetaV1,
@@ -116,6 +120,8 @@ export interface SourceEvaluationInput {
*/
export interface SourceReference extends ResourceKeyRef {
member?: string
+ /** Where the entry file writes the reference, for an MCP tool: the agent's `tools`. */
+ span?: SourceSpan
}
/** A resource an enclosing apply creates or adopts, by key, with the id that stands in for it. */
@@ -229,6 +235,126 @@ function withoutPendingReferences(
}
}
+/**
+ * `lint` without the block-output references compile already reported in a function's code
+ * file (`W_UNRESOLVED_REF` on that block, located at the file line): the same dangling `$ref`
+ * is one warning there, as an offline check reports it, not also a lint error on the
+ * `codeFile()` call.
+ */
+function withoutCodeFileReferences(
+ lint: WorkflowLintReport,
+ compiled: CompileResult['diagnostics'],
+ keyOf: (id: string) => string | undefined
+): WorkflowLintReport {
+ const reported = new Set(
+ compiled.flatMap((diagnostic) =>
+ diagnostic.code === 'W_UNRESOLVED_REF' &&
+ diagnostic.blockKey !== undefined &&
+ diagnostic.span.file !== SOURCE_ENTRY_FILE
+ ? [diagnostic.blockKey]
+ : []
+ )
+ )
+ if (reported.size === 0) return lint
+ return {
+ ...lint,
+ unresolvedReferences: lint.unresolvedReferences.filter((reference) => {
+ const key = keyOf(reference.blockId)
+ return !(
+ reference.kind === 'block-output' &&
+ reference.field === 'code' &&
+ key !== undefined &&
+ reported.has(key)
+ )
+ }),
+ }
+}
+
+/**
+ * `diagnostics` without the code-file `W_UNRESOLVED_REF` findings a block rename resolves: an
+ * apply rewrites every `` to the new key, so `$ref.` runs. The compile
+ * already knew the renames the lock names by ident; a rename materialize inferred (name and
+ * const both changed) is known only now, so the source is compiled again with every rename and
+ * a finding stays only where that compile still reports it.
+ */
+function withoutRenamedCodeFileRefs(
+ diagnostics: CompileResult['diagnostics'],
+ renames: ReadonlyMap,
+ recompile: () => CompileResult
+): CompileResult['diagnostics'] {
+ if (renames.size === 0 || !diagnostics.some((d) => d.code === 'W_UNRESOLVED_REF')) {
+ return diagnostics
+ }
+ const at = (d: CompileResult['diagnostics'][number]) =>
+ `${d.span.file}:${d.span.line}:${d.span.column}`
+ const still = new Set(
+ recompile()
+ .diagnostics.filter((d) => d.code === 'W_UNRESOLVED_REF')
+ .map(at)
+ )
+ return diagnostics.filter((d) => d.code !== 'W_UNRESOLVED_REF' || still.has(at(d)))
+}
+
+/** `{ span }` when there is one, to spread into a diagnostic. */
+function withSpan(span: SourceSpan | undefined): { span?: SourceSpan } {
+ return span ? { span } : {}
+}
+
+/** Names in a refusal, the first ten and a count of the rest. */
+function listNames(names: readonly string[]): string {
+ const shown = names.slice(0, 10).join(', ')
+ return names.length > 10 ? `${shown}, and ${names.length - 10} more` : shown
+}
+
+/**
+ * `E_APPLY_BLOCK_DELETE` for the stored blocks an apply removes. When the source adds blocks of
+ * a removed block's type, they may be renames whose name and const both changed with no identity
+ * signal tying them to the stored blocks (settings, connections or renamed references): the
+ * finding names them and says how to keep the ids, or to pass `--allow-delete` to replace them.
+ */
+/**
+ * Each removed block with the added block of the same type and parent it most probably became:
+ * one pair per removed block, in order, each added block used once.
+ */
+function probableRenames(
+ removed: readonly CanonicalBlock[],
+ added: readonly CanonicalBlock[]
+): Array<{ removed: CanonicalBlock; added: CanonicalBlock }> {
+ const free = [...added]
+ return removed.flatMap((block) => {
+ const index = free.findIndex(
+ (candidate) => candidate.type === block.type && candidate.parent === block.parent
+ )
+ if (index === -1) return []
+ const [match] = free.splice(index, 1)
+ return [{ removed: block, added: match }]
+ })
+}
+
+function blockDeleteRefusal(
+ removed: readonly CanonicalBlock[],
+ added: readonly CanonicalBlock[],
+ span: SourceSpan | undefined
+): Diagnostic {
+ const removedTypes = new Set(removed.map((block) => block.type))
+ const lookalikes = added.filter((block) => removedTypes.has(block.type))
+ const deletes = `This apply deletes ${removed.length} block${removed.length === 1 ? '' : 's'} the source no longer declares: ${listNames(removed.map((block) => block.name))}.`
+ return {
+ code: 'E_APPLY_BLOCK_DELETE',
+ severity: 'error',
+ ...(lookalikes.length > 0
+ ? {
+ message: `${deletes} It adds ${listNames(lookalikes.map((block) => block.name))} of the same type: if those are the same blocks renamed, both their name and const changed, so nothing ties them to the stored blocks and they would get new ids.`,
+ hint: 'To rename a block and keep its id, keep its const name in this apply (change it in the next), or rename the references to it with it; to replace it with a new block, pass --allow-delete.',
+ }
+ : {
+ message: deletes,
+ hint: 'Pass --allow-delete (allowDelete=true) to delete them, or declare them again to keep them.',
+ }),
+ ...withSpan(span),
+ }
+}
+
/** Everything a commit (or a dry run) writes and reports, once every check passed. */
export interface SourcePlan {
/**
@@ -292,9 +418,13 @@ function isResourceKind(kind: string): kind is ResourceKind {
* Every `(kind, key)` resource reference in a compiled model, for ident lookup, and the tool each
* MCP agent tool (`mcp(key).tool(name)`) names on its server.
*/
-function collectResourceKeys(value: unknown, refs: SourceReference[]): void {
+function collectResourceKeys(
+ value: unknown,
+ refs: SourceReference[],
+ span: SourceSpan | undefined
+): void {
if (Array.isArray(value)) {
- for (const item of value) collectResourceKeys(item, refs)
+ for (const item of value) collectResourceKeys(item, refs, span)
return
}
if (value === null || typeof value !== 'object') return
@@ -308,8 +438,8 @@ function collectResourceKeys(value: unknown, refs: SourceReference[]): void {
refs.push({ kind: record.kind, key: record.key })
}
const tool = mcpToolReference(record)
- if (tool) refs.push(tool)
- for (const nested of Object.values(record)) collectResourceKeys(nested, refs)
+ if (tool) refs.push(span ? { ...tool, span } : tool)
+ for (const nested of Object.values(record)) collectResourceKeys(nested, refs, span)
}
/** The server and tool an MCP agent tool entry names, or `null` for anything else. */
@@ -359,14 +489,18 @@ function boundLock(identity: LockIdentity, submitted: WorkflowLockV1): WorkflowL
function metaLimitRefusals(
meta: CanonicalWorkflowMeta,
fields: readonly (keyof CanonicalWorkflowMeta)[]
-): Diagnostic[] {
- const refusals: Diagnostic[] = []
- const refuse = (message: string, hint: string) =>
- refusals.push({ code: 'E_WORKFLOW_META_INVALID', severity: 'error', message, hint })
+): Array<{ field: keyof CanonicalWorkflowMeta; refusal: Diagnostic }> {
+ const refusals: Array<{ field: keyof CanonicalWorkflowMeta; refusal: Diagnostic }> = []
+ const refuse = (field: keyof CanonicalWorkflowMeta, message: string, hint: string) =>
+ refusals.push({
+ field,
+ refusal: { code: 'E_WORKFLOW_META_INVALID', severity: 'error', message, hint },
+ })
if (fields.includes('name') && meta.name.trim() === '') {
- refuse('The workflow name is empty.', "Name the workflow: workflow('Name', ...).")
+ refuse('name', 'The workflow name is empty.', "Name the workflow: workflow('Name', ...).")
} else if (fields.includes('name') && meta.name.length > V2_WORKFLOW_NAME_MAX_LENGTH) {
refuse(
+ 'name',
`The workflow name is ${meta.name.length} characters, over the ${V2_WORKFLOW_NAME_MAX_LENGTH}-character limit.`,
`Shorten the name to ${V2_WORKFLOW_NAME_MAX_LENGTH} characters or fewer.`
)
@@ -374,6 +508,7 @@ function metaLimitRefusals(
const description = meta.description ?? ''
if (fields.includes('description') && description.length > V2_WORKFLOW_DESCRIPTION_MAX_LENGTH) {
refuse(
+ 'description',
`The workflow description is ${description.length} characters, over the ${V2_WORKFLOW_DESCRIPTION_MAX_LENGTH}-character limit.`,
`Shorten the description to ${V2_WORKFLOW_DESCRIPTION_MAX_LENGTH} characters or fewer.`
)
@@ -411,7 +546,7 @@ export async function evaluateSource(input: SourceEvaluationInput): Promise path !== SOURCE_ENTRY_FILE)),
catalog,
@@ -420,10 +555,9 @@ export async function evaluateSource(input: SourceEvaluationInput): Promise storedResources.toId(kind, key) === null)
if (newKeys.length > 0) {
@@ -609,13 +748,21 @@ export async function evaluateSource(input: SourceEvaluationInput): Promise [storedKey, key]))
+ const compileDiagnostics = withoutRenamedCodeFileRefs(compiled.diagnostics, renamedKeys, () =>
+ compileWorkflow({ ...compileInput, stored: compiled.stored, renamedKeys })
)
const sourceRefusals: Diagnostic[] = []
const headerSpan = compiled.spans[encodeAnchor({ kind: 'header' })]
+ /** Where a finding about the workflow itself points: the field it names, else the name. */
+ const metaSpan = (field: keyof CanonicalWorkflowMeta = 'name') =>
+ compiled.metaSpans[field] ?? compiled.metaSpans.name ?? headerSpan
if (
metaChanged.includes('name') &&
(await workflowNameExistsInFolder({
@@ -630,27 +777,32 @@ export async function evaluateSource(input: SourceEvaluationInput): Promise ({
+ ...metaLimitRefusals(meta, isNew ? META_FIELDS : metaChanged).map(({ field, refusal }) => ({
...refusal,
- ...(headerSpan ? { span: headerSpan } : {}),
+ ...withSpan(metaSpan(field)),
}))
)
const removedKeys = isNew ? [] : canonicalDiff.blocks.removed
+ const removedBlocks = removedKeys.map((key) => storedCanonical.workflow.blocks[key])
+ const addedBlocks = canonicalDiff.blocks.added.map((key) => materialized.workflow.blocks[key])
if (removedKeys.length > 0 && !input.allowDelete) {
- const names = removedKeys.map((key) => storedCanonical.workflow.blocks[key]?.name ?? key)
- const shown = names.slice(0, 10).join(', ')
- sourceRefusals.push({
- code: 'E_APPLY_BLOCK_DELETE',
- severity: 'error',
- message: `This apply deletes ${removedKeys.length} block${removedKeys.length === 1 ? '' : 's'} the source no longer declares: ${shown}${names.length > 10 ? `, and ${names.length - 10} more` : ''}.`,
- hint: 'Pass --allow-delete (allowDelete=true) to delete them, or declare them again to keep them.',
- ...(headerSpan ? { span: headerSpan } : {}),
- })
+ sourceRefusals.push(blockDeleteRefusal(removedBlocks, addedBlocks, metaSpan()))
}
+ const renameWarnings = input.allowDelete
+ ? probableRenames(removedBlocks, addedBlocks).map(
+ ({ removed, added }): Diagnostic => ({
+ code: 'W_APPLY_PROBABLE_RENAME',
+ severity: 'warning',
+ message: `${removed.name} (${removed.type}) is deleted and ${added.name} (${added.type}) added: probably a rename, but its name and const both changed, so it gets a new id and loses its run history and webhook URL.`,
+ hint: `To keep the id, apply ${added.name} under the const or name ${removed.name} had first, then change the other.`,
+ blockKey: added.key,
+ })
+ )
+ : []
const storedGraph: SourceGraph = { blocks: state.blocks, edges: state.edges }
let blocks = materialized.state.blocks
@@ -678,6 +830,7 @@ export async function evaluateSource(input: SourceEvaluationInput): Promise m.id)])
const diagnostics = [
- ...compiled.diagnostics,
+ ...compileDiagnostics,
...keptDiagnostics,
...materializeDiagnostics,
...refusals,
...normalized.diagnostics,
...lintDiagnostics(
- withoutPendingReferences(lint, input.pendingResources ?? []),
+ withoutCodeFileReferences(
+ withoutPendingReferences(lint, input.pendingResources ?? []),
+ compileDiagnostics,
+ (id) => keyById.get(id)
+ ),
changedIds,
(id) => keyById.get(id)
),
@@ -761,8 +918,8 @@ export async function evaluateSource(input: SourceEvaluationInput): Promise = {
},
}
+/**
+ * Refuses placing a type the acting user cannot see: an unrevealed `preview` block or one the
+ * `block-visibility` kill switch names, by the catalog's own predicate ({@link isHiddenUnder}), so a
+ * write cannot add a block the catalog hides. Placed instances keep executing and stay editable, so
+ * a block keeping its own type is not refused. Reads the visibility in scope: outside
+ * `withBlockVisibility` preview blocks stay hidden and nothing is kill-switched.
+ */
+const VISIBLE_TYPE: PlacementGate<'block_not_allowed'> = {
+ subject: 'placement',
+ skipTypes: ['block_not_allowed'],
+ check: ({ blockType, previousType, blockId, operationType, outcome, details }) => {
+ if (blockType === previousType) return null
+ const block = getBlock(blockType)
+ if (!block || !isHiddenUnder(overlayVisibility(), block)) return null
+ return {
+ type: 'block_not_allowed',
+ operationType,
+ blockId,
+ reason: `Block type "${blockType}" is not available to you - ${outcome}`,
+ details: { requestedType: blockType, ...details },
+ }
+ },
+}
+
/**
* Refuses placing a sunset (`legacy` / `deprecated`) type and names its successor. Placed
* instances keep executing and stay editable, so a block keeping its own type is not refused.
@@ -373,6 +397,7 @@ const PERMITTED_TOOL: ToolEntryGate<'tool_not_allowed'> = {
export const BLOCK_GATES: readonly BlockGate[] = [
REGISTERED_TYPE,
AVAILABLE_TYPE,
+ VISIBLE_TYPE,
ACTIVE_TYPE,
SINGLE_TRIGGER,
SINGLE_INSTANCE,
diff --git a/apps/sim/lib/workspace-source/api/route-policies.ts b/apps/sim/lib/workspace-source/api/route-policies.ts
index 3722b7c8273..ca9dfc24e18 100644
--- a/apps/sim/lib/workspace-source/api/route-policies.ts
+++ b/apps/sim/lib/workspace-source/api/route-policies.ts
@@ -9,6 +9,7 @@ import { createV2ResourceConcealmentPolicy } from '@/lib/api/server/routes'
import type { ApplicationOperation, AuthorizingUseCase } from '@/lib/core/application'
import { WorkspaceSourceRejectedError } from '@/lib/workspace-source/application/errors'
import { WorkspaceSourceBusyError } from '@/lib/workspace-source/lease/store'
+import { SourceStagingFullError } from '@/lib/workspace-source/staging/store'
import {
v2CaughtOrchestrationError,
v2Error,
@@ -66,7 +67,8 @@ export function withSourceSlot(
* status (`400`, `409` stale, `423` locked, `413` over a limit) with `details.diagnostics`;
* `details.code` is `SOURCE_STALE` when a resource changed since the directory was read, else
* `SOURCE_REJECTED`. An apply while another one writes the workspace answers `409` with
- * `details.code` `SOURCE_BUSY`.
+ * `details.code` `SOURCE_BUSY`. Staging past the caller's staging caps answers `413` with
+ * `details.code` `SOURCE_STAGING_FULL`, apart from a body over the request limit.
*/
export const v2WorkspaceSourceErrorPolicy = createV2ResourceConcealmentPolicy({
notFoundMessage: 'Workspace not found',
@@ -85,6 +87,9 @@ export const v2WorkspaceSourceErrorPolicy = createV2ResourceConcealmentPolicy({
if (error instanceof WorkspaceSourceBusyError) {
return v2ErrorForOrchestration(error.code, error.message, { code: 'SOURCE_BUSY' })
}
+ if (error instanceof SourceStagingFullError) {
+ return v2ErrorForOrchestration(error.code, error.message, { code: 'SOURCE_STAGING_FULL' })
+ }
return v2CaughtOrchestrationError(error)
},
})
diff --git a/apps/sim/lib/workspace-source/application/cross-kind.integration.ts b/apps/sim/lib/workspace-source/application/cross-kind.integration.ts
index 78388601d77..d1cc50a3570 100644
--- a/apps/sim/lib/workspace-source/application/cross-kind.integration.ts
+++ b/apps/sim/lib/workspace-source/application/cross-kind.integration.ts
@@ -84,7 +84,10 @@ import {
applyWorkspaceSource,
} from '@/lib/workspace-source/application/apply-workspace-source'
import { readWorkspaceSource } from '@/lib/workspace-source/application/read-workspace-source'
-import { findMissingSourceBlobs } from '@/lib/workspace-source/application/stage-blobs'
+import {
+ findMissingSourceBlobs,
+ stageSourceBlobs,
+} from '@/lib/workspace-source/application/stage-blobs'
import {
blobManifest,
codes,
@@ -630,6 +633,41 @@ export default workflow('Answer', {}, () => {
}
)
+ check(
+ 'renaming a table column an unchanged declared workflow writes warns at the line that writes it',
+ async () => {
+ const { workspaceId, dir, base } = await setUp()
+ const printed = (await get(workspaceId)).files['workflows/ingest/workflow.ts']
+ const lines = printed.split('\n')
+ const dataLine = lines.findIndex((line) => line.includes('email: start.email')) + 1
+ const files = {
+ ...dir.files,
+ 'tables/leads.ts': dir.files['tables/leads.ts'].replace(
+ 'column.string({ required: true })',
+ "column.string({ required: true, name: 'email_address' })"
+ ),
+ }
+ expect(files['tables/leads.ts']).toContain('email_address')
+
+ const planned = await apply(workspaceId, { ...dir, files }, { base, dryRun: true })
+
+ const renamed = planned.diagnostics.filter(
+ (finding) => finding.code === 'W_TABLE_COLUMN_RENAMED_IN_USE'
+ )
+ expect(renamed).toEqual([
+ expect.objectContaining({
+ severity: 'warning',
+ kind: 'table',
+ key: 'leads',
+ message:
+ 'tables/leads renames column "email" to "email_address", but block "Save Lead" of workflow "Ingest" still writes "email"; the run drops it.',
+ file: 'workflows/ingest/workflow.ts',
+ line: dataLine,
+ }),
+ ])
+ }
+ )
+
check(
'deleting a table a workflow outside the directory uses warns, naming that workflow',
async () => {
@@ -1009,7 +1047,7 @@ export default workflow('Big', {}, () => {
check('workspace files and knowledge documents hash within one per-request budget', async () => {
const workspaceId = await newWorkspace()
- const size = Math.floor(WORKSPACE_SOURCE_LIMITS.maxBlobBytesPerRequest / 2.5)
+ const size = Math.floor(WORKSPACE_SOURCE_LIMITS.maxHashBytesPerRequest / 2.5)
for (const name of ['a.txt', 'b.txt']) {
await uploadWorkspaceFile(
workspaceId,
@@ -1053,10 +1091,17 @@ export default workflow('Big', {}, () => {
() => true
)
- const refused = await stageBlobsAs(
- owner,
- workspaceId,
- blobs.map((blob) => ({ sha256: blob.sha256, content: blob.content ?? '' }))
+ const refused = await withPermissionGroupScope(() =>
+ stageSourceBlobs.execute({
+ principal: owner,
+ input: {
+ workspaceId,
+ blobs: blobs.map((blob) => ({
+ sha256: blob.sha256,
+ content: Buffer.from(blob.content ?? '', 'base64'),
+ })),
+ },
+ })
).then(
() => null,
(error: unknown) => error
@@ -1070,6 +1115,19 @@ export default workflow('Big', {}, () => {
})
)
expect(missing.missing).toHaveLength(3)
+
+ const [alone] = blobManifest(
+ { 'files/alone.bin': Buffer.alloc(WORKSPACE_SOURCE_LIMITS.maxBlobBytes, 'd') },
+ () => true
+ )
+ await stageBlobsAs(owner, workspaceId, [{ sha256: alone.sha256, content: alone.content ?? '' }])
+ const stillMissing = await withPermissionGroupScope(() =>
+ findMissingSourceBlobs.execute({
+ principal: owner,
+ input: { workspaceId, blobs: [{ sha256: alone.sha256, size: alone.size }] },
+ })
+ )
+ expect(stillMissing.missing).toEqual([])
})
check('a plan already refused authorizes no change', async () => {
diff --git a/apps/sim/lib/workspace-source/application/deployments.integration.ts b/apps/sim/lib/workspace-source/application/deployments.integration.ts
index 9b58488697c..592d012ead2 100644
--- a/apps/sim/lib/workspace-source/application/deployments.integration.ts
+++ b/apps/sim/lib/workspace-source/application/deployments.integration.ts
@@ -1055,6 +1055,61 @@ describe('workspace source: deployments', () => {
}
)
+ check(
+ 'warns when a column rename breaks the live version of a workflow whose draft writes the new name, unless the apply redeploys',
+ async () => {
+ const workspaceId = await workspaces.create({ name: 'Deployed column user', ownerId })
+ const kinds: WorkspaceKind[] = ['table', 'workflow', 'deployment']
+ const created = await apply(
+ workspaceId,
+ {
+ 'tables/leads.ts': LEADS_TABLE,
+ 'workflows/ingest/workflow.ts': INGEST_LEADS,
+ 'deployments/ingest.ts': DRAFT,
+ },
+ { kinds }
+ )
+ expect(created.failed).toBeUndefined()
+ const read = await getAll(workspaceId, kinds)
+ // The table renames email, and the draft writes the new name; the live version still writes email.
+ const renamed: Files = {
+ ...read.files,
+ 'tables/leads.ts': LEADS_TABLE.replace(
+ 'email: column.string(),',
+ "email: column.string({ name: 'email_address' }),"
+ ),
+ 'workflows/ingest/workflow.ts': INGEST_LEADS.replace(
+ 'data: { email: start.email }',
+ 'data: { email_address: start.email }'
+ ),
+ }
+
+ const { 'deployments/ingest.ts': _deployment, ...undeclaredDeployment } = renamed
+ const kept = await apply(workspaceId, undeclaredDeployment, {
+ kinds: ['table', 'workflow'],
+ base: read.base,
+ dryRun: true,
+ })
+ expect(codes(kept.diagnostics)).toEqual([])
+ const warning = kept.diagnostics.find(
+ (finding) => finding.code === 'W_TABLE_COLUMN_RENAMED_IN_USE'
+ )
+ expect(warning).toMatchObject({ kind: 'table', key: 'leads' })
+ expect(warning?.message).toContain('live version')
+
+ const redeployed = await apply(workspaceId, renamed, {
+ kinds,
+ base: read.base,
+ dryRun: true,
+ })
+ expect(codes(redeployed.diagnostics)).toEqual([])
+ expect(actions(redeployed.plan)).toContain('update deployment:ingest')
+ expect(codes(redeployed.diagnostics, 'warning')).not.toContain(
+ 'W_TABLE_COLUMN_RENAMED_IN_USE'
+ )
+ }
+ )
+
check('refuses a deployment of a workflow the workspace does not hold', async () => {
const workspaceId = await workspaces.create({ name: 'Deploy nothing', ownerId })
const refused = await refusal(apply(workspaceId, { 'deployments/missing.ts': DRAFT }))
diff --git a/apps/sim/lib/workspace-source/application/files.integration.ts b/apps/sim/lib/workspace-source/application/files.integration.ts
index 256337c7be5..b049c8cce33 100644
--- a/apps/sim/lib/workspace-source/application/files.integration.ts
+++ b/apps/sim/lib/workspace-source/application/files.integration.ts
@@ -775,7 +775,7 @@ describe('workspace source files', () => {
'server hashing stops at the per-request budget: a get lists the rest without a hash, and a plan refuses them rather than creating',
async () => {
const workspaceId = await newWorkspace()
- const size = Math.floor(WORKSPACE_SOURCE_LIMITS.maxBlobBytesPerRequest / 2.5)
+ const size = Math.floor(WORKSPACE_SOURCE_LIMITS.maxHashBytesPerRequest / 2.5)
const directory: Record = {}
for (const name of ['a.bin', 'b.bin', 'c.bin']) {
const bytes = Buffer.alloc(size, name)
diff --git a/apps/sim/lib/workspace-source/application/hash-budget.ts b/apps/sim/lib/workspace-source/application/hash-budget.ts
index 922d95fedf4..fa4e147af5a 100644
--- a/apps/sim/lib/workspace-source/application/hash-budget.ts
+++ b/apps/sim/lib/workspace-source/application/hash-budget.ts
@@ -18,7 +18,7 @@ export interface HashBudget {
/** A full hashing budget for one request. */
export function newHashBudget(): HashBudget {
- return { bytes: WORKSPACE_SOURCE_LIMITS.maxBlobBytesPerRequest, count: MAX_HASHED_PER_REQUEST }
+ return { bytes: WORKSPACE_SOURCE_LIMITS.maxHashBytesPerRequest, count: MAX_HASHED_PER_REQUEST }
}
/** Downloads one plan or get runs at once while hashing. */
diff --git a/apps/sim/lib/workspace-source/application/in-use.ts b/apps/sim/lib/workspace-source/application/in-use.ts
index 2e6314363c4..98b99bb09bc 100644
--- a/apps/sim/lib/workspace-source/application/in-use.ts
+++ b/apps/sim/lib/workspace-source/application/in-use.ts
@@ -136,7 +136,7 @@ export function inUseUncheckedFinding(change: Pick {
+export function liveVersionsReplaced(changes: readonly PlannedChange[]): Set {
const replaced = new Set()
for (const change of changes) {
if (change.kind !== 'deployment' || change.held) continue
diff --git a/apps/sim/lib/workspace-source/application/kinds/knowledge/diff.ts b/apps/sim/lib/workspace-source/application/kinds/knowledge/diff.ts
index 83a439bc37e..3a82b53167e 100644
--- a/apps/sim/lib/workspace-source/application/kinds/knowledge/diff.ts
+++ b/apps/sim/lib/workspace-source/application/kinds/knowledge/diff.ts
@@ -166,7 +166,10 @@ export async function hashDocs(
return [...left, ...failed]
}
-/** The info line a change gets for the documents it indexes, and for chunking it changes. */
+/**
+ * The info line a change gets for the documents it indexes, and for chunking it changes:
+ * `storedDocs` counts the stored documents that keep their chunks (neither replaced nor deleted).
+ */
export function indexingFindings(
key: string,
uploads: readonly DocUpload[],
@@ -355,6 +358,16 @@ export async function diffKnowledgeBase(input: {
)
}
}
- changes.diagnostics.push(...indexingFindings(key, changes.uploads, chunkingChanged, byName.size))
+ const reindexed = new Set([
+ ...changes.uploads.flatMap((upload) => (upload.replaces ? [upload.replaces.id] : [])),
+ ...changes.finishes.map((finish) => finish.replaces.id),
+ ...changes.deletes.map((doc) => doc.id),
+ ])
+ // A finished replacement is already stored and indexed, so it keeps its chunks too.
+ const keepingChunks =
+ [...byName.values()].filter((doc) => !reindexed.has(doc.id)).length + changes.finishes.length
+ changes.diagnostics.push(
+ ...indexingFindings(key, changes.uploads, chunkingChanged, keepingChunks)
+ )
return changes
}
diff --git a/apps/sim/lib/workspace-source/application/kinds/mcp/adapter.ts b/apps/sim/lib/workspace-source/application/kinds/mcp/adapter.ts
index 761c9de0d78..2ae18d1be15 100644
--- a/apps/sim/lib/workspace-source/application/kinds/mcp/adapter.ts
+++ b/apps/sim/lib/workspace-source/application/kinds/mcp/adapter.ts
@@ -23,9 +23,14 @@ import { createLogger } from '@sim/logger'
import { mapWithConcurrency } from '@sim/utils/concurrency'
import { getErrorMessage } from '@sim/utils/errors'
import { isRecordLike } from '@sim/utils/object'
-import { didYouMean, type Json } from '@sim/workflow-code'
+import { didYouMean, type Json, type SourceSpan } from '@sim/workflow-code'
import { SOURCE_ENTRY_FILE } from '@sim/workflow-code/limits'
-import { type BaseResource, fingerprintOf, resourceAddress } from '@sim/workspace-code'
+import {
+ type BaseResource,
+ type FlatDiagnostic,
+ fingerprintOf,
+ resourceAddress,
+} from '@sim/workspace-code'
import {
MCP_DEFAULTS,
type McpServerDecl,
@@ -50,6 +55,7 @@ import type { McpServerRow } from '@/lib/mcp/queries'
import { generateMcpServerId } from '@/lib/mcp/utils'
import { admitOnce } from '@/lib/workspace-source/application/context'
import {
+ finding,
kindFinding,
planUndeclared,
serverGoneFinding,
@@ -427,8 +433,11 @@ async function listTools(
/** Each change that calls tools of an MCP server, with the tools, by server key. */
function toolUsers(
changes: readonly PlannedChange[]
-): Map }>> {
- const byServer = new Map }>>()
+): Map }>> {
+ const byServer = new Map<
+ string,
+ Array<{ change: PlannedChange; tools: Map }>
+ >()
for (const change of changes) {
if (change.action === 'delete') continue
for (const ref of change.refs ?? []) {
@@ -436,21 +445,33 @@ function toolUsers(
const users = byServer.get(ref.key) ?? []
let user = users.find((entry) => entry.change === change)
if (!user) {
- user = { change, tools: new Set() }
+ user = { change, tools: new Map() }
users.push(user)
}
- user.tools.add(ref.member)
+ if (!user.tools.has(ref.member)) user.tools.set(ref.member, ref.span)
byServer.set(ref.key, users)
}
}
return byServer
}
-/** Where a finding about the tools a change calls points: its workflow file. */
-function fileOf(change: PlannedChange): { file?: string } {
- return change.kind === 'workflow'
- ? { file: `${workflowFolder(change.key)}/${SOURCE_ENTRY_FILE}` }
- : {}
+/**
+ * Where a finding about the tools a change calls points: its workflow file, at the line that
+ * writes the tool when the workflow's compile located it.
+ */
+function fileOf(change: PlannedChange, span?: SourceSpan): Partial {
+ if (change.kind !== 'workflow') return {}
+ return {
+ file: `${workflowFolder(change.key)}/${SOURCE_ENTRY_FILE}`,
+ ...(span
+ ? {
+ line: span.line,
+ column: span.column,
+ endLine: span.endLine,
+ endColumn: span.endColumn,
+ }
+ : {}),
+ }
}
export const mcpAdapter: KindAdapter<'mcp-server', McpServerRow> = {
@@ -631,18 +652,18 @@ export const mcpAdapter: KindAdapter<'mcp-server', McpServerRow> = {
}
const names = new Set(listed.names)
for (const { change, tools } of callers) {
- for (const tool of tools) {
+ for (const [tool, span] of tools) {
if (names.has(tool)) continue
const suggestion = didYouMean(tool, names, (name) => `"${name}"`)
change.diagnostics.push(
- kindFinding(
- change.kind,
- change.key,
+ finding(
'E_MCP_TOOL_UNKNOWN',
`${where(change.kind, change.key)} calls tool "${tool}" on MCP server "${key}", which does not list it.${suggestion ? ` ${suggestion}` : ''}`,
{
hint: `The server lists: ${[...names].slice(0, 20).join(', ') || 'no tools'}.`,
- ...fileOf(change),
+ ...fileOf(change, span),
+ kind: change.kind,
+ key: change.key,
}
)
)
diff --git a/apps/sim/lib/workspace-source/application/kinds/table/adapter.ts b/apps/sim/lib/workspace-source/application/kinds/table/adapter.ts
index 7824cbb7272..8a7944b076b 100644
--- a/apps/sim/lib/workspace-source/application/kinds/table/adapter.ts
+++ b/apps/sim/lib/workspace-source/application/kinds/table/adapter.ts
@@ -49,6 +49,7 @@ import {
deleteTableUseCase,
listTableDefinitionsUseCase,
} from '@/lib/table/application/tables'
+import { getColumnId } from '@/lib/table/column-keys'
import { getTableById } from '@/lib/table/service'
import type { RowData } from '@/lib/table/types'
import { validateRowSize } from '@/lib/table/validation'
@@ -70,6 +71,7 @@ import {
isColumnOp,
type TableChanges,
} from '@/lib/workspace-source/application/kinds/table/diff'
+import { checkColumnRenamesInUse } from '@/lib/workspace-source/application/kinds/table/renames'
import {
baseRows,
namedRows,
@@ -80,6 +82,7 @@ import {
import {
printStoredTable,
type StoredTableState,
+ sourceColumns,
storedFingerprint,
tableAsDecl,
} from '@/lib/workspace-source/application/kinds/table/stored'
@@ -98,12 +101,18 @@ import type {
interface RowsPlan {
/** The rows the reshape writes (only those that differ); absent leaves the rows alone. */
write?: ReshapeRows
- /** The managed mark to set, by key column name, or `null` to clear it; absent leaves it. */
- mark?: { mode: 'managed'; keyColumn: string } | null
+ /**
+ * The rows mark to set (managed rows by key column name, or seed rows), or `null` to clear it;
+ * absent leaves it.
+ */
+ mark?: RowsMark | null
/** The row version the plan read the stored rows at; the apply refuses one that moved. */
version?: number
}
+/** How a declaration opts its rows in, as the table's mark records it. */
+type RowsMark = { mode: 'managed'; keyColumn: string } | { mode: 'seed' }
+
/** What the apply carries out for one table, decided at plan time. */
export interface TablePlan {
/** The changes an update or adopt makes; absent for a create or delete. */
@@ -111,6 +120,11 @@ export interface TablePlan {
/** The base's column bindings the plan's fingerprint was read under. */
bindings?: Record
rows?: RowsPlan
+ /**
+ * The column keys the write records (`metadata.sourceColumnKeys`), by stored column name: each
+ * declared column whose key is not its name. Absent leaves the record alone.
+ */
+ keys?: Record
}
type TableChange = PlannedChange<'table'>
@@ -162,8 +176,60 @@ function namesByKey(decl: TableDecl): Map {
)
}
-/** The managed mark a declaration asks for, by its key column's stored name. */
-function markOf(decl: TableDecl): { mode: 'managed'; keyColumn: string } | null {
+/** The keys `decl` gives its columns where a key is not the column's name, by column name. */
+function declaredKeys(decl: TableDecl): Record {
+ const keys: Record = {}
+ for (const [key, column] of Object.entries(decl.columns)) {
+ const name = columnName(key, column)
+ if (name !== key) setOwn(keys, name, key)
+ }
+ return keys
+}
+
+/**
+ * Whether a read of the stored table without a base (a fresh get, or a promotion's read of the
+ * target) would print a declared column under another key than `decl` gives it.
+ */
+function keysUnrecorded(state: StoredTableState, decl: TableDecl): boolean {
+ const { keyById } = tableAsDecl(state)
+ const idByName = new Map(
+ sourceColumns(state.table).map((column) => [column.name, getColumnId(column)])
+ )
+ return Object.entries(decl.columns).some(([key, column]) => {
+ const id = idByName.get(columnName(key, column))
+ return id !== undefined && keyById.get(id) !== key
+ })
+}
+
+/**
+ * `change` recording `decl`'s column keys: every table write records them, and a no-op becomes a
+ * write of the record alone when a read without a base would print a column under another key.
+ */
+function withKeys(
+ change: TableChange,
+ state: StoredTableState,
+ decl: TableDecl,
+ entry: BaseResource | undefined
+): TableChange {
+ const keys = declaredKeys(decl)
+ if (change.action === 'update' || change.action === 'adopt') {
+ return { ...change, detail: { ...change.detail, keys } }
+ }
+ if (change.action !== 'noop' || !keysUnrecorded(state, decl)) return change
+ const bindings = entry?.id === state.table.id ? entry.columns : undefined
+ return {
+ ...change,
+ action: 'update',
+ storedId: state.table.id,
+ fields: ['column keys'],
+ storedFingerprint: storedFingerprint(state, bindings),
+ detail: { ...(bindings ? { bindings } : {}), keys },
+ }
+}
+
+/** The rows mark a declaration asks for: managed rows by their key column's stored name, or seed. */
+function markOf(decl: TableDecl): RowsMark | null {
+ if (decl.rows?.mode === 'seed') return { mode: 'seed' }
const key = decl.rows?.mode === 'managed' ? decl.rows.key : undefined
const column = key !== undefined ? decl.columns[key] : undefined
return key !== undefined && column
@@ -177,7 +243,7 @@ function rowsWrite(decl: TableDecl, allowDelete: boolean): ReshapeRows | undefin
if (!rows) return undefined
const named = namedRows(rows, namesByKey(decl))
const mark = markOf(decl)
- return rows.mode === 'managed' && mark
+ return rows.mode === 'managed' && mark?.mode === 'managed'
? {
mode: 'managed',
keyColumn: mark.keyColumn,
@@ -285,9 +351,16 @@ async function planRows(
const { decl: storedDecl, keyById } = tableAsDecl(state, entry?.columns)
const storedKey = storedDecl.rows?.key
if (rows?.mode !== 'managed') {
- if (storedKey !== undefined) {
- outcome.plan = { mark: null }
- outcome.fields.push('rows (no longer managed)')
+ const storedMode = storedDecl.rows?.mode
+ if (storedMode !== rows?.mode) {
+ outcome.plan = { mark: markOf(decl) }
+ outcome.fields.push(
+ storedMode === 'managed'
+ ? 'rows (no longer managed)'
+ : rows
+ ? 'rows (seed)'
+ : 'rows (no longer seeded)'
+ )
}
if (rows?.mode === 'seed' && rows.data.length > 0 && !entry) {
if (state.table.rowCount === 0) {
@@ -456,7 +529,10 @@ async function planTable(
// A table whose declaration is unchanged can still hold rows to write.
const settled = planned.settled
const resource = resolution.action === 'noop' ? resolution.stored : undefined
- if (!resource || settled.action !== 'noop' || !decl.rows) return settled
+ if (!resource || settled.action !== 'noop') return settled
+ if (!decl.rows && !resource.state.table.metadata?.sourceRows) {
+ return withKeys(settled, resource.state, decl, entry)
+ }
const rows = await planRows(key, decl, resource.state, entry, ctx)
const renamed = rows.plan?.write ? columnsRenamedOnServer(decl, resource.state, entry) : []
if (renamed.length > 0) {
@@ -471,7 +547,7 @@ async function planTable(
)
return settled
}
- return withRows(settled, rows, resource.state, entry)
+ return withKeys(withRows(settled, rows, resource.state, entry), resource.state, decl, entry)
}
switch (planned.action) {
case 'create': {
@@ -491,7 +567,7 @@ async function planTable(
...rows.fields,
],
pending: pendingOf({ ...change, action: 'create' }, { columns: declaredColumnNames(decl) }),
- detail: { rows: rows.plan },
+ detail: { rows: rows.plan, keys: declaredKeys(decl) },
}
}
case 'adopt':
@@ -533,45 +609,51 @@ async function planTable(
}
const action = planned.action === 'adopt' ? 'adopt' : changed ? 'update' : 'noop'
const storedId = planned.stored.id
- return {
- ...change,
- action,
- storedId,
- // A later workflow's dry run checks its row writes against the columns this declares.
- ...(action !== 'noop'
- ? {
- pending: pendingOf(
- { ...change, action, storedId },
- { columns: declaredColumnNames(decl) }
- ),
- }
- : {}),
- ...(changes.fields.length + rows.fields.length > 0
- ? { fields: [...changes.fields, ...rows.fields] }
- : {}),
- ...(dropped > 0 || changes.lostCells > 0 || rows.removed > 0
- ? {
- destructive: {
- what: 'cells',
- counts: {
- columns: dropped,
- ...(changes.removedOptions > 0 ? { options: changes.removedOptions } : {}),
- cells: changes.lostCells,
- ...(rows.removed > 0 ? { rows: rows.removed } : {}),
+ const entryOfStored = planned.action === 'update' ? planned.base : undefined
+ return withKeys(
+ {
+ ...change,
+ action,
+ storedId,
+ // A later workflow's dry run checks its row writes against the columns this declares.
+ ...(action !== 'noop'
+ ? {
+ pending: pendingOf(
+ { ...change, action, storedId },
+ { columns: declaredColumnNames(decl) }
+ ),
+ }
+ : {}),
+ ...(changes.fields.length + rows.fields.length > 0
+ ? { fields: [...changes.fields, ...rows.fields] }
+ : {}),
+ ...(dropped > 0 || changes.lostCells > 0 || rows.removed > 0
+ ? {
+ destructive: {
+ what: 'cells',
+ counts: {
+ columns: dropped,
+ ...(changes.removedOptions > 0 ? { options: changes.removedOptions } : {}),
+ cells: changes.lostCells,
+ ...(rows.removed > 0 ? { rows: rows.removed } : {}),
+ },
},
- },
- }
- : {}),
- storedFingerprint: planFingerprint(storedFingerprint(state, bindings), rows.plan),
- ...(action === 'noop' && rows.base
- ? { base: { ...printStoredTable(state, key, bindings).base, rows: rows.base } }
- : {}),
- detail: {
- changes,
- ...(bindings ? { bindings } : {}),
- ...(rows.plan ? { rows: rows.plan } : {}),
+ }
+ : {}),
+ storedFingerprint: planFingerprint(storedFingerprint(state, bindings), rows.plan),
+ ...(action === 'noop' && rows.base
+ ? { base: { ...printStoredTable(state, key, bindings).base, rows: rows.base } }
+ : {}),
+ detail: {
+ changes,
+ ...(bindings ? { bindings } : {}),
+ ...(rows.plan ? { rows: rows.plan } : {}),
+ },
},
- }
+ state,
+ decl,
+ entryOfStored
+ )
}
}
}
@@ -663,6 +745,7 @@ function reshapeInput(
...(rows?.write ? { rows: rows.write } : {}),
...(rows?.mark !== undefined ? { sourceRows: rows.mark } : {}),
...(rows?.version !== undefined ? { expectedRowsVersion: rows.version } : {}),
+ ...(plan?.keys !== undefined ? { sourceColumnKeys: plan.keys } : {}),
}
if (!plan?.changes) return { tableId, workspaceId, ...rowsInput }
const { table, columns } = plan.changes
@@ -700,6 +783,7 @@ async function managedRowsMovedSinceBase(
}
export const tableAdapter: KindAdapter<'table', StoredTableState> = {
+ checkRefs: checkColumnRenamesInUse,
codec: tableCodec,
async load(ctx, { allocate }) {
@@ -804,17 +888,21 @@ export const tableAdapter: KindAdapter<'table', StoredTableState> = {
},
})
const rows = change.detail?.rows
- if (rows?.write || rows?.mark) {
+ const keys = change.detail?.keys
+ const recordsKeys = keys !== undefined && Object.keys(keys).length > 0
+ if (rows?.write || rows?.mark || recordsKeys) {
await reshapeTableUseCase.execute({
principal,
- input: reshapeInput(ctx, table.id, { rows }),
+ input: reshapeInput(ctx, table.id, { rows, ...(recordsKeys ? { keys } : {}) }),
})
}
return written(ctx, change.key, table.id, decl)
}
const tableId = change.storedId
const plan = change.detail
- if (!tableId || !(plan?.changes || plan?.rows)) throw new Error('A table update needs its plan')
+ if (!tableId || !(plan?.changes || plan?.rows || plan?.keys)) {
+ throw new Error('A table update needs its plan')
+ }
await reshapeTableUseCase.execute({ principal, input: reshapeInput(ctx, tableId, plan) })
return written(ctx, change.key, tableId, decl)
},
diff --git a/apps/sim/lib/workspace-source/application/kinds/table/diff.ts b/apps/sim/lib/workspace-source/application/kinds/table/diff.ts
index 88e09ec06c1..6f53119171f 100644
--- a/apps/sim/lib/workspace-source/application/kinds/table/diff.ts
+++ b/apps/sim/lib/workspace-source/application/kinds/table/diff.ts
@@ -66,6 +66,10 @@ export interface TableChanges {
columns: ColumnChange[]
/** Fields that change, for the plan display. */
fields: string[]
+ /** Stored columns this update renames, old name → new name. */
+ renamedColumns: Array<{ from: string; to: string }>
+ /** Every column name the table holds after the update, in declaration order. */
+ columnNames: string[]
/** Cells a drop empties, and cells a removed option takes a selection from. */
lostCells: number
/**
@@ -249,6 +253,10 @@ export async function diffTable(
table: {},
columns: [],
fields: [],
+ renamedColumns: [],
+ columnNames: Object.entries(decl.columns).map(([columnKey, column]) =>
+ columnName(columnKey, column)
+ ),
lostCells: 0,
removedOptions: 0,
diagnostics,
@@ -320,6 +328,7 @@ export async function diffTable(
if (target.name !== current.name) {
updates.name = target.name
fields.push('name')
+ changes.renamedColumns.push({ from: current.name, to: target.name })
}
const retyped = target.type !== current.type
const becomesUnique = Boolean(target.unique) && !current.unique
diff --git a/apps/sim/lib/workspace-source/application/kinds/table/renames.ts b/apps/sim/lib/workspace-source/application/kinds/table/renames.ts
new file mode 100644
index 00000000000..790ee72e3b9
--- /dev/null
+++ b/apps/sim/lib/workspace-source/application/kinds/table/renames.ts
@@ -0,0 +1,184 @@
+/**
+ * Workflows that still write a table column by the name a plan renames. A row write, filter or
+ * sort names a column by its name, so after the rename the run drops the cell (or matches
+ * nothing); the plan warns on each block, at the line its workflow file writes the old name when
+ * the directory declares that workflow. A workflow folder the apply evaluates is checked by its own
+ * lint against the declared columns, so only workflows the apply leaves as stored are scanned here.
+ */
+import { escapeRegExp } from '@sim/utils/string'
+import { splitLines } from '@sim/workflow-code'
+import { SOURCE_ENTRY_FILE } from '@sim/workflow-code/limits'
+import { workflowFolder } from '@sim/workspace-code/kinds/workflow'
+import { readWorkflowSource } from '@/lib/workflows/application/read-workflow-source'
+import { collectTableBlockFieldIssues } from '@/lib/workflows/editing/lint'
+import {
+ loadDeployedWorkflowState,
+ loadWorkflowFromNormalizedTables,
+ NoActiveDeploymentError,
+} from '@/lib/workflows/persistence/utils'
+import { finding, where } from '@/lib/workspace-source/application/findings'
+import { liveVersionsReplaced, workflowsUsing } from '@/lib/workspace-source/application/in-use'
+import type { TablePlan } from '@/lib/workspace-source/application/kinds/table/adapter'
+import type { PlanContext, PlannedChange } from '@/lib/workspace-source/application/types'
+
+/** A table this plan renames columns of: its id, name, and each old name → new name. */
+interface RenamingTable {
+ change: PlannedChange
+ tableId: string
+ tableName: string
+ columnNames: string[]
+ renamed: Map
+}
+
+function renamingTables(changes: readonly PlannedChange[]): RenamingTable[] {
+ return changes.flatMap((change) => {
+ if (change.kind !== 'table' || change.held || change.storedId === undefined) return []
+ const plan = change.detail as TablePlan | undefined
+ const renamed = new Map(
+ (plan?.changes?.renamedColumns ?? []).map(({ from, to }) => [from, to] as const)
+ )
+ if (renamed.size === 0 || !plan?.changes) return []
+ return [
+ {
+ change,
+ tableId: change.storedId,
+ tableName: plan.changes.table.name ?? change.key,
+ columnNames: plan.changes.columnNames,
+ renamed,
+ },
+ ]
+ })
+}
+
+/**
+ * The 1-based line and column of the first `name` key written after the declaration of block
+ * `blockName` in `text`, else of the declaration itself; `undefined` when neither is found.
+ */
+function locateWrite(
+ text: string,
+ blockName: string,
+ name: string
+): { line: number; column: number } | undefined {
+ const lines = splitLines(text)
+ const quoted = [`'${blockName}'`, `"${blockName}"`]
+ const declaration = lines.findIndex((line) => quoted.some((name) => line.includes(name)))
+ if (declaration === -1) return undefined
+ const keyPattern = new RegExp(`(^|[\\s{,])(['"]?)${escapeRegExp(name)}\\2\\s*:`)
+ for (let index = declaration; index < lines.length; index++) {
+ const match = keyPattern.exec(lines[index])
+ if (match) return { line: index + 1, column: match.index + match[1].length + 1 }
+ }
+ return { line: declaration + 1, column: 1 }
+}
+
+/** The live version of `workflowId`'s deployment, or `null` when it has none. */
+async function liveBlocks(workflowId: string) {
+ try {
+ return (await loadDeployedWorkflowState(workflowId)).blocks
+ } catch (error) {
+ if (error instanceof NoActiveDeploymentError) return null
+ throw error
+ }
+}
+
+/**
+ * Warns `W_TABLE_COLUMN_RENAMED_IN_USE` on each table change that renames a column some stored
+ * workflow's Table block still writes, filters or sorts by the old name: one finding per block,
+ * naming the workflow and the block, pinned to the declared workflow's file and line. A deployed
+ * workflow's live version runs apart from its draft, so it is checked too (an evaluated draft
+ * included), unless the apply publishes the draft or undeploys it.
+ */
+export async function checkColumnRenamesInUse(
+ changes: readonly PlannedChange[],
+ ctx: PlanContext
+): Promise {
+ const tables = renamingTables(changes)
+ if (tables.length === 0) return
+ const using = await workflowsUsing(
+ ctx.workspace.workspaceId,
+ tables.map((table) => table.tableId)
+ )
+ if (using === null) return
+ /** Declared workflows by stored id: their folder key, and whether the apply evaluates them. */
+ const declared = new Map()
+ for (const change of changes) {
+ if (change.kind !== 'workflow' || change.storedId === undefined) continue
+ declared.set(change.storedId, { key: change.key, evaluated: change.refs !== undefined })
+ }
+ const replacedLive = liveVersionsReplaced(changes)
+ for (const table of tables) {
+ const columns = new Map([
+ [table.tableId, { name: table.tableName, columnNames: table.columnNames }],
+ ])
+ for (const workflow of using.get(table.tableId) ?? []) {
+ const folder = declared.get(workflow.id)
+ if (
+ changes.some(
+ (c) => c.kind === 'workflow' && c.storedId === workflow.id && c.action === 'delete'
+ )
+ ) {
+ continue
+ }
+ /** `blockId:field` of each finding already made from the draft. */
+ const reported = new Set()
+ if (workflow.draft && !folder?.evaluated) {
+ const loaded = await loadWorkflowFromNormalizedTables(workflow.id)
+ const issues = loaded
+ ? collectTableBlockFieldIssues(loaded.blocks, columns).filter((issue) =>
+ table.renamed.has(issue.field)
+ )
+ : []
+ const text =
+ folder && issues.length > 0
+ ? (
+ await readWorkflowSource.execute({
+ principal: ctx.principal,
+ input: { workflowId: workflow.id },
+ })
+ ).files[SOURCE_ENTRY_FILE]
+ : undefined
+ for (const issue of issues) {
+ reported.add(`${issue.blockId}:${issue.field}`)
+ const renamedTo = table.renamed.get(issue.field)
+ const blockName = issue.blockName ?? issue.blockId
+ const at = text !== undefined ? locateWrite(text, blockName, issue.field) : undefined
+ table.change.diagnostics.push(
+ finding(
+ 'W_TABLE_COLUMN_RENAMED_IN_USE',
+ `${where('table', table.change.key)} renames column "${issue.field}" to "${renamedTo}", but block "${blockName}" of workflow "${workflow.name}" still ${issue.rowWrite ? 'writes' : 'reads'} "${issue.field}"; the run drops it.`,
+ {
+ hint: `Write "${renamedTo}" in that block${folder ? ` (${workflowFolder(folder.key)}/${SOURCE_ENTRY_FILE})` : ''}, in this apply or right after it.`,
+ ...(folder
+ ? { file: `${workflowFolder(folder.key)}/${SOURCE_ENTRY_FILE}`, ...(at ?? {}) }
+ : {}),
+ kind: 'table',
+ key: table.change.key,
+ }
+ )
+ )
+ }
+ }
+ if (!workflow.live || replacedLive.has(workflow.id)) continue
+ const live = await liveBlocks(workflow.id)
+ if (!live) continue
+ for (const issue of collectTableBlockFieldIssues(live, columns)) {
+ if (!table.renamed.has(issue.field) || reported.has(`${issue.blockId}:${issue.field}`)) {
+ continue
+ }
+ const renamedTo = table.renamed.get(issue.field)
+ const blockName = issue.blockName ?? issue.blockId
+ table.change.diagnostics.push(
+ finding(
+ 'W_TABLE_COLUMN_RENAMED_IN_USE',
+ `${where('table', table.change.key)} renames column "${issue.field}" to "${renamedTo}", but block "${blockName}" of workflow "${workflow.name}" (live version) still ${issue.rowWrite ? 'writes' : 'reads'} "${issue.field}"; its runs drop it until the workflow is deployed again.`,
+ {
+ hint: `Write "${renamedTo}" in that block and deploy the workflow (\`deployments/.ts\` with live: 'draft'), in this apply or right after it.`,
+ kind: 'table',
+ key: table.change.key,
+ }
+ )
+ )
+ }
+ }
+ }
+}
diff --git a/apps/sim/lib/workspace-source/application/kinds/table/stored.ts b/apps/sim/lib/workspace-source/application/kinds/table/stored.ts
index d632e0425da..81c109b6608 100644
--- a/apps/sim/lib/workspace-source/application/kinds/table/stored.ts
+++ b/apps/sim/lib/workspace-source/application/kinds/table/stored.ts
@@ -55,23 +55,34 @@ function columnDecl(column: ColumnDefinition, key: string): TableColumnDecl {
/**
* A stored table as the declaration it prints as. A column keeps the key `bindings` (the base's
- * column key → id) gives it; any other column is keyed by its name, so a fresh directory matches
- * columns by name.
+ * column key → id) gives it, else the key the last source write recorded for it
+ * (`metadata.sourceColumnKeys`), so a fresh directory and another workspace's read print the key
+ * the declaration gave it; any other column is keyed by its name.
*/
export function tableAsDecl(
state: StoredTableState,
bindings?: Readonly>
): TableAsDecl {
const boundKey = new Map(Object.entries(bindings ?? {}).map(([key, column]) => [column.id, key]))
+ const recordedKey = state.table.metadata?.sourceColumnKeys ?? {}
const stored = sourceColumns(state.table)
const keyById = new Map()
const taken = new Set()
- // Bound keys first, so a column keyed by its name never takes a renamed column's key.
- for (const column of stored) {
- const key = boundKey.get(getColumnId(column))
- if (key === undefined || !isDeclarableColumnKey(key) || taken.has(key)) continue
- keyById.set(getColumnId(column), key)
- taken.add(key)
+ // Bound keys first, then recorded ones, so a column keyed by its name never takes a renamed
+ // column's key.
+ for (const keyOf of [
+ (id: string) => boundKey.get(id),
+ (id: string) => (Object.hasOwn(recordedKey, id) ? recordedKey[id] : undefined),
+ ]) {
+ for (const column of stored) {
+ const id = getColumnId(column)
+ const key = keyOf(id)
+ if (keyById.has(id) || key === undefined || !isDeclarableColumnKey(key) || taken.has(key)) {
+ continue
+ }
+ keyById.set(id, key)
+ taken.add(key)
+ }
}
for (const column of stored) {
const id = getColumnId(column)
@@ -88,15 +99,20 @@ export function tableAsDecl(
const key = keyById.get(getColumnId(column)) ?? column.name
setOwn(columns, key, columnDecl(column, key))
}
- // Rows source manages print as `rows`, keyed by the column the marker names while it exists.
+ // Rows source manages print as `rows`, keyed by the column the marker names while it exists;
+ // seed rows print as their `rows` alone, since the directory keeps their file.
const marker = state.table.metadata?.sourceRows
- const rowsKey = marker ? keyById.get(marker.keyColumnId) : undefined
+ const rowsKey = marker?.mode === 'managed' ? keyById.get(marker.keyColumnId) : undefined
const decl: TableDecl = {
name: state.table.name,
...(state.table.description ? { description: state.table.description } : {}),
...(state.folderPath !== '/' ? { folder: state.folderPath } : {}),
columns,
- ...(rowsKey !== undefined ? { rows: { mode: 'managed', key: rowsKey, data: [] } } : {}),
+ ...(rowsKey !== undefined
+ ? { rows: { mode: 'managed', key: rowsKey, data: [] } }
+ : marker?.mode === 'seed'
+ ? { rows: { mode: 'seed', data: [] } }
+ : {}),
}
return { decl, keyById, fingerprint: fingerprintOf(tableCodec.canonical(decl)) }
}
@@ -134,7 +150,7 @@ export function printStoredTable(
const { decl, keyById, fingerprint } = tableAsDecl(state, bindings)
let rowsEntry: BaseRows | undefined
if (rows && 'seed' in rows) {
- if (!decl.rows) decl.rows = rows.seed
+ if (decl.rows?.mode !== 'managed') decl.rows = rows.seed
} else if (decl.rows && rows) {
decl.rows = { ...decl.rows, data: rows.data }
rowsEntry = baseRows(decl.rows, rows.version)
diff --git a/apps/sim/lib/workspace-source/application/kinds/workflow/adapter.ts b/apps/sim/lib/workspace-source/application/kinds/workflow/adapter.ts
index e83e9c99daf..963ac7223ba 100644
--- a/apps/sim/lib/workspace-source/application/kinds/workflow/adapter.ts
+++ b/apps/sim/lib/workspace-source/application/kinds/workflow/adapter.ts
@@ -209,13 +209,16 @@ function sourceRefs(
evaluated: EvaluatedSource | undefined
): NonNullable {
const refs = new Map[number]>()
- for (const { kind, key, member } of references) {
+ for (const { kind, key, member, span } of references) {
if (!isWorkspaceKind(kind)) continue
if (isResourceKeyed(kind)) {
- refs.set(JSON.stringify([kind, key, member ?? null]), {
+ const id = JSON.stringify([kind, key, member ?? null])
+ if (refs.has(id)) continue
+ refs.set(id, {
kind,
key,
...(member !== undefined ? { member } : {}),
+ ...(span !== undefined ? { span } : {}),
})
continue
}
diff --git a/apps/sim/lib/workspace-source/application/knowledge.integration.ts b/apps/sim/lib/workspace-source/application/knowledge.integration.ts
index 56f8e4b5cc0..6161edc554c 100644
--- a/apps/sim/lib/workspace-source/application/knowledge.integration.ts
+++ b/apps/sim/lib/workspace-source/application/knowledge.integration.ts
@@ -517,6 +517,27 @@ describe('knowledge bases as workspace source', () => {
}
)
+ check(
+ 'a chunking change counts only the stored documents that keep their chunks, not the ones it replaces',
+ async () => {
+ const { workspaceId, read } = await setUp()
+
+ const planned = await apply(
+ workspaceId,
+ {
+ files: {
+ [POLICIES_FILE]: declaration('Policies', '\n chunking: { maxSize: 512 },'),
+ },
+ docs: { ...POLICIES.docs, [REFUNDS]: Buffer.from('# Refunds\nWithin 60 days.\n') },
+ },
+ { base: read.base, dryRun: true, carry: 'none' }
+ )
+
+ const chunking = planned.diagnostics.find((finding) => finding.code === 'I_KB_CHUNKING')
+ expect(chunking?.message).toContain('1 stored document(s) keep their chunks')
+ }
+ )
+
check('a get reproduces the apply, and re-applying it changes nothing', async () => {
const workspaceId = await newWorkspace()
const created = await apply(workspaceId, POLICIES)
@@ -1027,6 +1048,38 @@ describe('knowledge bases as workspace source', () => {
}
)
+ check(
+ 'a chunking change counts the replacement a stopped replace left among the documents keeping their chunks',
+ async () => {
+ const { workspaceId, read, dir } = await setUp()
+ const edited = withDoc(dir, REFUNDS, '# Refunds\nWithin 60 days.\n')
+ const failing = vi
+ .spyOn(deleteKnowledgeDocument, 'execute')
+ .mockRejectedValueOnce(new Error('storage unavailable'))
+ .mockRejectedValueOnce(new Error('storage unavailable'))
+ const first = await apply(workspaceId, edited, { base: read.base }).finally(() =>
+ failing.mockRestore()
+ )
+ expect(first.failed).toMatchObject({ key: 'policies' })
+
+ const planned = await apply(
+ workspaceId,
+ {
+ ...edited,
+ files: {
+ ...edited.files,
+ [POLICIES_FILE]: declaration('Policies', '\n chunking: { maxSize: 512 },'),
+ },
+ },
+ { base: first.base, dryRun: true, carry: 'none' }
+ )
+
+ // The finished replacement and shipping.txt keep their chunks; the replaced refunds.md goes.
+ const chunking = planned.diagnostics.find((finding) => finding.code === 'I_KB_CHUNKING')
+ expect(chunking?.message).toContain('2 stored document(s) keep their chunks')
+ }
+ )
+
check('a create that fails partway is adopted and finished by the next apply', async () => {
const workspaceId = await newWorkspace()
const failing = failUpload(2)
diff --git a/apps/sim/lib/workspace-source/application/mcp.integration.ts b/apps/sim/lib/workspace-source/application/mcp.integration.ts
index 63e55d86bee..cdbd3876195 100644
--- a/apps/sim/lib/workspace-source/application/mcp.integration.ts
+++ b/apps/sim/lib/workspace-source/application/mcp.integration.ts
@@ -309,6 +309,8 @@ describe('MCP servers as workspace source', () => {
kind: 'workflow',
key: 'triage',
file: 'workflows/triage/workflow.ts',
+ line: 9,
+ column: 5,
})
expect(unknown?.message).toContain('Did you mean "create_issue"?')
}
diff --git a/apps/sim/lib/workspace-source/application/promote.integration.ts b/apps/sim/lib/workspace-source/application/promote.integration.ts
index aa9a00df6de..2813a116021 100644
--- a/apps/sim/lib/workspace-source/application/promote.integration.ts
+++ b/apps/sim/lib/workspace-source/application/promote.integration.ts
@@ -11,7 +11,11 @@ import { featureFlagsMock, featureFlagsMockFns } from '@sim/testing/mocks/featur
import { posthogServerMock } from '@sim/testing/mocks/posthog-server.mock'
import { realtimeNotifyMock } from '@sim/testing/mocks/realtime-notify.mock'
import { generateId } from '@sim/utils/id'
-import { promotionSubmission } from '@sim/workspace-code/promote'
+import {
+ type PromotionRecord,
+ promotionRecord,
+ promotionSubmission,
+} from '@sim/workspace-code/promote'
import { and, eq, inArray, isNull } from 'drizzle-orm'
import { afterAll, beforeAll, describe, expect, vi } from 'vitest'
import {
@@ -72,6 +76,14 @@ description: House tone
Be brief.
`
+const NOTIFY = `// @sim-sdk 1
+import { trigger, workflow } from '@sim/workflow-sdk'
+
+export default workflow('Notify', {}, () => {
+ trigger.start('Start', { inputFormat: { email: 'string' } })
+})
+`
+
const DEV: Record = {
'tables/leads.ts': LEADS,
'skills/tone.md': TONE,
@@ -94,6 +106,41 @@ async function promote(dev: Read, target: string, options: { allowDelete?: boole
return runner.apply(target, { files: composed.files, base: composed.base, ...options })
}
+/**
+ * Promotes `dev` to `target` as `apply --to` does with the target's promotion record, returning
+ * the apply's result and the record it leaves.
+ */
+async function promoteRecorded(dev: Read, target: string, record: PromotionRecord | undefined) {
+ const prod = await runner.get(target)
+ const composed = promotionSubmission({
+ files: dev.files,
+ blobPaths: [],
+ kinds: dev.kinds,
+ target: prod,
+ record,
+ allowDelete: false,
+ })
+ const result = await runner.apply(target, { files: composed.files, base: composed.base })
+ return {
+ result,
+ record: promotionRecord({
+ workspaceId: target,
+ previous: record,
+ kinds: dev.kinds,
+ files: dev.files,
+ submission: composed,
+ applied: result,
+ }),
+ }
+}
+
+/** `dev` with `edits` applied to it, read back as the directory would hold it. */
+async function editDev(dev: string, edits: Record) {
+ const read = await runner.get(dev)
+ await runner.apply(dev, { files: { ...read.files, ...edits }, base: read.base })
+ return runner.get(dev)
+}
+
async function blockIds(workspaceId: string) {
const rows = await db
.select({ id: workflowBlocks.id, name: workflowBlocks.name })
@@ -200,6 +247,142 @@ describe('workspace source promotion', () => {
}
)
+ check(
+ 'a change made in the target since the last promotion is kept when the directory left it alone, and refused as stale when the directory changed it too',
+ async () => {
+ const dev = await workspaces.create({ name: 'Dev', ownerId })
+ const prod = await workspaces.create({ name: 'Prod', ownerId })
+ await runner.apply(dev, { files: DEV })
+ const first = await promoteRecorded(await runner.get(dev), prod, undefined)
+ expect(first.result.failed).toBeUndefined()
+
+ const prodRead = await runner.get(prod)
+ await runner.apply(prod, {
+ files: {
+ ...prodRead.files,
+ 'skills/tone.md': TONE.replace('Be brief.', 'Be brief, prod.'),
+ 'workflows/ingest/workflow.ts': ingest('Prod hotfix.'),
+ },
+ base: prodRead.base,
+ })
+
+ const untouched = await promoteRecorded(await runner.get(dev), prod, first.record)
+ expect(untouched.result.applied).toEqual([])
+ const kept = await runner.get(prod)
+ expect(kept.files['skills/tone.md']).toContain('Be brief, prod.')
+ expect(kept.files['workflows/ingest/workflow.ts']).toContain('Prod hotfix.')
+
+ const edited = await editDev(dev, {
+ 'skills/tone.md': TONE.replace('Be brief.', 'Be very brief.'),
+ 'workflows/ingest/workflow.ts': ingest('Be terse.'),
+ })
+ const stale = await refusal(promoteRecorded(edited, prod, untouched.record))
+ expect(stale.stale).toBe(true)
+ expect(codes(stale.diagnostics)).toContain('E_APPLY_STALE')
+ const unchanged = await runner.get(prod)
+ expect(unchanged.files['skills/tone.md']).toContain('Be brief, prod.')
+ expect(unchanged.files['workflows/ingest/workflow.ts']).toContain('Prod hotfix.')
+ }
+ )
+
+ check(
+ 'a resource deleted in the target since the last promotion stays deleted when the directory left it alone, and an edit of it since is refused as stale',
+ async () => {
+ const dev = await workspaces.create({ name: 'Dev', ownerId })
+ const prod = await workspaces.create({ name: 'Prod', ownerId })
+ await runner.apply(dev, { files: { ...DEV, 'workflows/notify/workflow.ts': NOTIFY } })
+ const first = await promoteRecorded(await runner.get(dev), prod, undefined)
+ expect(first.result.failed).toBeUndefined()
+
+ const prodRead = await runner.get(prod)
+ const removed = await runner.apply(prod, {
+ files: Object.fromEntries(
+ Object.entries(prodRead.files).filter(
+ ([path]) => path !== 'skills/tone.md' && !path.startsWith('workflows/notify/')
+ )
+ ),
+ base: prodRead.base,
+ allowDelete: true,
+ })
+ expect(
+ removed.applied.map((change) => `${change.action} ${change.kind}:${change.key}`).sort()
+ ).toEqual(['delete skill:tone', 'delete workflow:notify'])
+
+ const untouched = await promoteRecorded(await runner.get(dev), prod, first.record)
+ expect(untouched.result.applied).toEqual([])
+ expect(codes(untouched.result.diagnostics, 'info').sort()).toEqual([
+ 'I_SERVER_DELETED',
+ 'I_SERVER_DELETED',
+ ])
+ expect((await liveWorkflows(prod)).map((row) => row.name)).toEqual(['Ingest'])
+ expect((await runner.get(prod)).files['skills/tone.md']).toBeUndefined()
+
+ const edited = await editDev(dev, {
+ 'skills/tone.md': TONE.replace('Be brief.', 'Be very brief.'),
+ 'workflows/notify/workflow.ts': NOTIFY.replace("'Notify'", "'Notify v2'"),
+ })
+ const stale = await refusal(promoteRecorded(edited, prod, untouched.record))
+ expect(stale.stale).toBe(true)
+ expect(codes(stale.diagnostics)).toEqual(['E_APPLY_STALE', 'E_APPLY_STALE'])
+ expect((await liveWorkflows(prod)).map((row) => row.name)).toEqual(['Ingest'])
+ }
+ )
+
+ check(
+ 'with the record, a promotion that changed nothing in the target since converges, and the next edit updates it in place',
+ async () => {
+ const dev = await workspaces.create({ name: 'Dev', ownerId })
+ const prod = await workspaces.create({ name: 'Prod', ownerId })
+ await runner.apply(dev, { files: DEV })
+ const first = await promoteRecorded(await runner.get(dev), prod, undefined)
+ const again = await promoteRecorded(await runner.get(dev), prod, first.record)
+ expect(again.result.applied).toEqual([])
+
+ const edited = await editDev(dev, {
+ 'skills/tone.md': TONE.replace('Be brief.', 'Be very brief.'),
+ 'workflows/ingest/workflow.ts': ingest('Be terse.'),
+ })
+ const updated = await promoteRecorded(edited, prod, again.record)
+ expect(
+ updated.result.applied
+ .map((change) => `${change.action} ${change.kind}:${change.key}`)
+ .sort()
+ ).toEqual(['update skill:tone', 'update workflow:ingest'])
+ const settled = await promoteRecorded(edited, prod, updated.record)
+ expect(settled.result.applied).toEqual([])
+ }
+ )
+
+ check('promoting managed rows converges: the second promotion changes nothing', async () => {
+ const dev = await workspaces.create({ name: 'Dev', ownerId })
+ const prod = await workspaces.create({ name: 'Prod', ownerId })
+ const directory = {
+ 'tables/stock.ts': `// @sim-sdk 1
+import { column, defineTable } from '@sim/workflow-sdk/workspace'
+
+export default defineTable({
+ name: 'stock',
+ columns: {
+ sku: column.string({ required: true, unique: true }),
+ status: column.select({ options: ['new', 'won'] }),
+ note_text: column.string({ name: 'memo' }),
+ },
+ rows: { mode: 'managed', key: 'sku' },
+})
+`,
+ 'tables/stock.rows.jsonl': '{"sku":"A-1","status":"new","note_text":"hi"}\n{"sku":"A-2"}\n',
+ }
+ const applied = await runner.apply(dev, { files: directory, kinds: ['table'] })
+ const devDir = { ...(await runner.get(dev)), files: applied.files }
+
+ const first = await promote(devDir, prod)
+ expect(first.applied.map((change) => `${change.action} ${change.kind}:${change.key}`)).toEqual([
+ 'create table:stock',
+ ])
+ const second = await promote(devDir, prod)
+ expect(second.applied).toEqual([])
+ })
+
check(
"the target's own resources are kept, with a warning, unless the promotion allows deletes",
async () => {
diff --git a/apps/sim/lib/workspace-source/application/rows.integration.ts b/apps/sim/lib/workspace-source/application/rows.integration.ts
index 444d1a986fc..9dce9850bac 100644
--- a/apps/sim/lib/workspace-source/application/rows.integration.ts
+++ b/apps/sim/lib/workspace-source/application/rows.integration.ts
@@ -19,6 +19,7 @@ import { generateId } from '@sim/utils/id'
import { compareStrings } from '@sim/utils/string'
import { and, eq, inArray, isNull, sql } from 'drizzle-orm'
import { afterAll, beforeAll, describe, expect, vi } from 'vitest'
+import { updateTableColumnUseCase } from '@/lib/table/application/columns'
import { createTableRows } from '@/lib/table/application/rows'
import { readCurrentRowsVersion } from '@/lib/table/row-changes'
import { getTableById } from '@/lib/table/service'
@@ -347,6 +348,112 @@ describe('table rows as workspace source', () => {
expect(firedEvents()).toEqual([])
})
+ check(
+ "a column's key survives a fresh get and a rename in Sim, and a table written before keys were recorded records them",
+ async () => {
+ const workspaceId = await newWorkspace()
+ const keyed = `// @sim-sdk 1
+import { column, defineTable } from '@sim/workflow-sdk/workspace'
+
+export default defineTable({
+ name: 'keyed',
+ columns: {
+ sku: column.string({ required: true, unique: true }),
+ note: column.string({ name: 'note_text' }),
+ },
+})
+`
+ const created = await apply(workspaceId, { 'tables/keyed.ts': keyed })
+ expect(created.failed).toBeUndefined()
+ expect((await runner.get(workspaceId)).files['tables/keyed.ts']).toContain(
+ "note: column.string({ name: 'note_text' })"
+ )
+
+ const [table] = await db
+ .select({ id: userTableDefinitions.id, metadata: userTableDefinitions.metadata })
+ .from(userTableDefinitions)
+ .where(
+ and(
+ eq(userTableDefinitions.workspaceId, workspaceId),
+ isNull(userTableDefinitions.archivedAt)
+ )
+ )
+ await updateTableColumnUseCase.execute({
+ principal: owner,
+ input: {
+ tableId: table.id,
+ workspaceId,
+ columnName: 'note_text',
+ updates: { name: 'memo' },
+ },
+ })
+ expect((await runner.get(workspaceId)).files['tables/keyed.ts']).toContain(
+ "note: column.string({ name: 'memo' })"
+ )
+
+ await db
+ .update(userTableDefinitions)
+ .set({ metadata: {} })
+ .where(eq(userTableDefinitions.id, table.id))
+ const legacy = await runner.get(workspaceId)
+ expect(legacy.files['tables/keyed.ts']).toContain('memo: column.string()')
+ const recorded = await apply(
+ workspaceId,
+ { 'tables/keyed.ts': keyed.replace("name: 'note_text'", "name: 'memo'") },
+ { base: legacy.base }
+ )
+ expect(recorded.applied).toMatchObject([
+ { kind: 'table', key: 'keyed', action: 'update', fields: ['column keys'] },
+ ])
+ expect((await runner.get(workspaceId)).files['tables/keyed.ts']).toContain(
+ "note: column.string({ name: 'memo' })"
+ )
+ }
+ )
+
+ check(
+ 'managed rows resent against a fresh read plan as no change, a column keyed apart from its name included, as a promotion resends them',
+ async () => {
+ const workspaceId = await newWorkspace()
+ const typed = `// @sim-sdk 1
+import { column, defineTable } from '@sim/workflow-sdk/workspace'
+
+export default defineTable({
+ name: 'typed',
+ columns: {
+ sku: column.string({ required: true, unique: true }),
+ shipped: column.date(),
+ expires: column.ttl(),
+ price: column.currency({ currencyCode: 'EUR' }),
+ labels: column.select({ options: ['red', 'green', 'blue'], multiple: true }),
+ status: column.select({ options: ['new', 'won'] }),
+ meta: column.json(),
+ note_text: column.string({ name: 'memo' }),
+ },
+ rows: { mode: 'managed', key: 'sku' },
+})
+`
+ const directory = {
+ 'tables/typed.ts': typed,
+ 'tables/typed.rows.jsonl': [
+ '{"sku":"A-1","shipped":"2026-11-02T10:30:00Z","expires":"2027-01-01T00:00:00-00:00","price":19.99,"labels":["red","blue"],"status":"new","meta":{"a":[1,{"b":null}]}}',
+ '{"sku":"A-2","shipped":"2026-11-01","labels":[],"status":"won","meta":[1,"x",true]}',
+ '{"sku":"A-9","note_text":"line1\\nline2"}',
+ ].join('\n'),
+ }
+ const created = await apply(workspaceId, directory)
+ expect(created.failed).toBeUndefined()
+ const versionBefore = await version(workspaceId)
+ const read = await runner.get(workspaceId)
+ expect(read.files['tables/typed.rows.jsonl']).not.toBe(directory['tables/typed.rows.jsonl'])
+
+ const resent = await apply(workspaceId, directory, { base: read.base })
+
+ expect(resent.applied).toEqual([])
+ expect(await version(workspaceId)).toBe(versionBefore)
+ }
+ )
+
check('a key rename is a delete and an insert, gated by allow-delete', async () => {
const workspaceId = await newWorkspace()
const read = await managedPlans(workspaceId)
@@ -638,8 +745,10 @@ export default defineTable({
expect(again.applied).toEqual([])
expect((await storedRows(workspaceId))[0]).toMatchObject({ seats: 5 })
- // A get leaves seed rows out: they were never the table's identity.
+ // A get keeps the declaration of seed rows but leaves the rows out: the directory holds them,
+ // and they were never the table's identity.
const read = await runner.get(workspaceId)
+ expect(read.files['tables/plans.ts']).toContain("rows: { mode: 'seed' }")
expect(read.files['tables/plans.rows.jsonl']).toBeUndefined()
expect((await apply(workspaceId, read.files, { base: read.base })).applied).toEqual([])
@@ -649,6 +758,43 @@ export default defineTable({
expect(await storedRows(workspaceId)).toHaveLength(2)
})
+ check(
+ 'a declaration that stops seeding rows clears it, and one that starts records it',
+ async () => {
+ const workspaceId = await newWorkspace()
+ const seeded = await apply(workspaceId, {
+ 'tables/plans.ts': SEED,
+ 'tables/plans.rows.jsonl': ROWS,
+ })
+ const unseeded = PLANS('undefined').replace(' rows: undefined,\n', '')
+
+ const dropped = await apply(
+ workspaceId,
+ { 'tables/plans.ts': unseeded },
+ { base: seeded.base }
+ )
+ expect(dropped.applied).toMatchObject([
+ { key: 'plans', action: 'update', fields: ['rows (no longer seeded)'] },
+ ])
+ expect((await runner.get(workspaceId)).files['tables/plans.ts']).not.toContain('rows:')
+
+ const read = await runner.get(workspaceId)
+ const again = await apply(
+ workspaceId,
+ { 'tables/plans.ts': SEED, 'tables/plans.rows.jsonl': ROWS },
+ { base: read.base }
+ )
+ expect(again.applied).toMatchObject([
+ { key: 'plans', action: 'update', fields: ['rows (seed)'] },
+ ])
+ expect(codes(again.diagnostics, 'info')).not.toContain('I_TABLE_SEED_SKIPPED')
+ expect((await runner.get(workspaceId)).files['tables/plans.ts']).toContain(
+ "rows: { mode: 'seed' }"
+ )
+ expect(await storedRows(workspaceId)).toHaveLength(2)
+ }
+ )
+
check('makes an existing column required in the apply whose managed rows fill it', async () => {
const workspaceId = await newWorkspace()
const gappy = '{"code":"solo","seats":1,"tier":"basic"}\n{"code":"team","tier":"pro"}\n'
diff --git a/apps/sim/lib/workspace-source/application/stage-blobs.ts b/apps/sim/lib/workspace-source/application/stage-blobs.ts
index 736d41cbe8f..f3a50d06bb2 100644
--- a/apps/sim/lib/workspace-source/application/stage-blobs.ts
+++ b/apps/sim/lib/workspace-source/application/stage-blobs.ts
@@ -71,7 +71,8 @@ export interface StageSourceBlobsResult {
}
/**
- * Stages a batch of blobs: refuses one over `maxBlobBytes`, a batch over the request caps, and any
+ * Stages a batch of blobs: refuses one over `maxBlobBytes`, a batch over the request caps (a single
+ * blob may fill its request alone), and any
* blob whose bytes do not hash to the SHA-256 it was named by (nothing of the batch is staged),
* then stores the rest within the scope's caps. Staging a blob already staged only renews it.
*/
@@ -108,10 +109,10 @@ export const stageSourceBlobs = defineAuthorizedWorkspaceUseCase({
)
}
}
- if (total > maxBlobBytesPerRequest) {
+ if (input.blobs.length > 1 && total > maxBlobBytesPerRequest) {
throw new OrchestrationError(
'payload_too_large',
- `A staging request carries at most ${maxBlobBytesPerRequest} bytes; this one carries ${total}.`
+ `A staging request of more than one file carries at most ${maxBlobBytesPerRequest} bytes; this one carries ${total}. Send a larger file in a request of its own.`
)
}
const staged = await stageBlobs(
diff --git a/apps/sim/lib/workspace-source/application/types.ts b/apps/sim/lib/workspace-source/application/types.ts
index a033afdea00..4f8422e34a1 100644
--- a/apps/sim/lib/workspace-source/application/types.ts
+++ b/apps/sim/lib/workspace-source/application/types.ts
@@ -1,4 +1,5 @@
import type { Principal } from '@sim/auth/principal'
+import type { SourceSpan } from '@sim/workflow-code'
import type {
BaseResource,
BlobEntry,
@@ -143,6 +144,8 @@ interface ChangeFields {
key: string
/** The part of the resource the reference names: an MCP server's tool. */
member?: string
+ /** Where the declaration's entry file writes the reference, when the pipeline located it. */
+ span?: SourceSpan
/**
* The resource's id, for a kind keyed otherwise than a workflow names it (a workspace file,
* keyed by its path): the plan finds that kind's change by id and names it by its key.
diff --git a/apps/sim/lib/workspace-source/application/workspace-source.test-helpers.ts b/apps/sim/lib/workspace-source/application/workspace-source.test-helpers.ts
index 05a4713fb6c..62a331e7e94 100644
--- a/apps/sim/lib/workspace-source/application/workspace-source.test-helpers.ts
+++ b/apps/sim/lib/workspace-source/application/workspace-source.test-helpers.ts
@@ -13,6 +13,7 @@ import { sha256Hex } from '@sim/security/hash'
import { getErrorMessage } from '@sim/utils/errors'
import { generateId } from '@sim/utils/id'
import type { WorkspaceKind } from '@sim/workspace-code'
+import { stageBatches } from '@sim/workspace-code/limits'
import { inArray, sql } from 'drizzle-orm'
import { expect, it } from 'vitest'
import { withPermissionGroupScope } from '@/lib/permission-groups/request-scope.server'
@@ -124,25 +125,30 @@ export type SourceSubmission = Omit<
principal?: Principal
}
-/** Stages `blobs` (base64 bytes by SHA-256) in `workspaceId` as `principal`, as the CLI does. */
+/**
+ * Stages `blobs` (base64 bytes by SHA-256) in `workspaceId` as `principal`, as the CLI does: in
+ * the requests `stageBatches` splits them into.
+ */
export async function stageBlobsAs(
principal: Principal,
workspaceId: string,
blobs: ReadonlyArray<{ sha256: string; content: string }>
) {
- if (blobs.length === 0) return
- await withPermissionGroupScope(() =>
- stageSourceBlobs.execute({
- principal,
- input: {
- workspaceId,
- blobs: blobs.map((blob) => ({
- sha256: blob.sha256,
- content: Buffer.from(blob.content, 'base64'),
- })),
- },
- })
- )
+ const decoded = blobs.map((blob) => {
+ const content = Buffer.from(blob.content, 'base64')
+ return { sha256: blob.sha256, content, size: content.byteLength }
+ })
+ for (const batch of stageBatches(decoded)) {
+ await withPermissionGroupScope(() =>
+ stageSourceBlobs.execute({
+ principal,
+ input: {
+ workspaceId,
+ blobs: batch.map(({ sha256, content }) => ({ sha256, content })),
+ },
+ })
+ )
+ }
}
/**
diff --git a/apps/sim/lib/workspace-source/staging/store.ts b/apps/sim/lib/workspace-source/staging/store.ts
index da607b45774..d2bd7a7548d 100644
--- a/apps/sim/lib/workspace-source/staging/store.ts
+++ b/apps/sim/lib/workspace-source/staging/store.ts
@@ -100,6 +100,18 @@ export async function stagedUsage(
return { count: Number(usage?.count ?? 0), bytes: Number(usage?.bytes ?? 0) }
}
+/**
+ * A caller's staging area has no room for what it stages: `413` like a body over the limit, told
+ * apart by `details.code` `SOURCE_STAGING_FULL`, since freeing staged files (or waiting for them
+ * to expire) fixes it and sending smaller requests does not.
+ */
+export class SourceStagingFullError extends OrchestrationError {
+ constructor(message: string) {
+ super('payload_too_large', message)
+ this.name = 'SourceStagingFullError'
+ }
+}
+
/** Refuses staging that would take `target` past its blob or byte cap. */
export function assertStagingFits(
usage: { count: number; bytes: number },
@@ -107,14 +119,12 @@ export function assertStagingFits(
): void {
const { maxStagedBlobs, maxStagedBytes } = WORKSPACE_SOURCE_LIMITS
if (usage.count + adding.count > maxStagedBlobs) {
- throw new OrchestrationError(
- 'payload_too_large',
+ throw new SourceStagingFullError(
`Staging ${adding.count} more binary files would hold ${usage.count + adding.count}, over the ${maxStagedBlobs}-file limit. Staged files expire a day after their last use.`
)
}
if (usage.bytes + adding.bytes > maxStagedBytes) {
- throw new OrchestrationError(
- 'payload_too_large',
+ throw new SourceStagingFullError(
`Staging ${adding.bytes} more bytes would hold ${usage.bytes + adding.bytes}, over the ${maxStagedBytes}-byte limit. Staged files expire a day after their last use.`
)
}
diff --git a/apps/sim/providers/tool-identity.test.ts b/apps/sim/providers/tool-identity.test.ts
index a8b12b4e54a..e425df15cd9 100644
--- a/apps/sim/providers/tool-identity.test.ts
+++ b/apps/sim/providers/tool-identity.test.ts
@@ -78,9 +78,10 @@ describe('provider tool identities', () => {
const longId = `tool_${'a'.repeat(80)}`
const tools = [providerTool(longId, 'a'), providerTool(longId, 'b')]
- assignProviderToolIdentities(tools)
+ const identities = assignProviderToolIdentities(tools)
- expect(tools[0].id).toBe(longId)
+ expect(tools[0].id).toBe(longId.slice(0, 64))
+ expect(identities.toolIdByWireId.get(tools[0].id)).toBe(longId)
expect(tools[1].id).toHaveLength(64)
expect(tools[1].id).toMatch(/__sim_2$/)
})
@@ -171,4 +172,69 @@ describe('provider tool identities', () => {
expect(output.toolCalls?.list[0].name).toBe('gmail_send')
expect(output.providerTiming?.timeSegments?.[0].name).toBe('gmail_send')
})
+
+ it('gives a custom tool titled with spaces and unicode a provider-safe name that maps back', () => {
+ const PROVIDER_NAME = /^[a-zA-Z0-9_-]{1,64}$/
+ const spaced = providerTool('custom_Get Weather', 'a')
+ const unicode = providerTool('custom_Météo du jour ☀', 'b')
+
+ const identities = assignProviderToolIdentities([spaced, unicode])
+
+ expect(spaced.id).toMatch(PROVIDER_NAME)
+ expect(unicode.id).toMatch(PROVIDER_NAME)
+ expect(identities.toolIdByWireId.get(spaced.id)).toBe('custom_Get Weather')
+ expect(identities.toolIdByWireId.get(unicode.id)).toBe('custom_Météo du jour ☀')
+
+ const response: ProviderResponse = {
+ content: 'done',
+ model: 'test-model',
+ toolCalls: [{ name: spaced.id, arguments: {} }],
+ }
+ projectProviderResponseToolIdentities(response, identities)
+ expect(response.toolCalls?.[0].name).toBe('custom_Get Weather')
+ })
+
+ it('keeps custom tools whose safe names collide distinct, and a safe id unchanged', () => {
+ const tools = [
+ providerTool('custom_Get Weather', 'a'),
+ providerTool('custom_Get/Weather', 'b'),
+ providerTool('custom_Get_Weather', 'c'),
+ ]
+
+ const identities = assignProviderToolIdentities(tools)
+ const wireIds = tools.map((tool) => tool.id)
+
+ expect(new Set(wireIds).size).toBe(3)
+ expect(tools[2].id).toBe('custom_Get_Weather')
+ expect(identities.toolIdByWireId.get(wireIds[0])).toBe('custom_Get Weather')
+ expect(identities.toolIdByWireId.get(wireIds[1])).toBe('custom_Get/Weather')
+ expect(identities.toolIdByWireId.has('custom_Get_Weather')).toBe(false)
+
+ assignProviderToolIdentities(tools)
+ expect(tools.map((tool) => tool.id)).toEqual(wireIds)
+ })
+
+ it('bounds a long custom tool title to the provider name limit, keeping tools that share a prefix distinct', () => {
+ const PROVIDER_NAME = /^[a-zA-Z0-9_-]{1,64}$/
+ const prefix = `custom_${'Look up the customer account '.repeat(3)}`
+ const tools = [
+ providerTool(`${prefix}by email`, 'a'),
+ providerTool(`${prefix}by phone`, 'b'),
+ providerTool(`custom_${'x'.repeat(80)}`, 'c'),
+ ]
+
+ const identities = assignProviderToolIdentities(tools)
+ const wireIds = tools.map((tool) => tool.id)
+
+ for (const wireId of wireIds) expect(wireId).toMatch(PROVIDER_NAME)
+ expect(new Set(wireIds).size).toBe(3)
+ expect(wireIds.map((wireId) => identities.toolIdByWireId.get(wireId))).toEqual([
+ `${prefix}by email`,
+ `${prefix}by phone`,
+ `custom_${'x'.repeat(80)}`,
+ ])
+
+ assignProviderToolIdentities(tools)
+ expect(tools.map((tool) => tool.id)).toEqual(wireIds)
+ })
})
diff --git a/apps/sim/providers/tool-identity.ts b/apps/sim/providers/tool-identity.ts
index 28b53e98fdc..1be25e13209 100644
--- a/apps/sim/providers/tool-identity.ts
+++ b/apps/sim/providers/tool-identity.ts
@@ -4,6 +4,17 @@ import type { ProviderResponse, ProviderToolConfig } from '@/providers/types'
const MAX_PROVIDER_TOOL_ID_LENGTH = 64
const PROVIDER_ALIAS_MARKER = '__sim_'
+/** Characters outside every provider's function-name alphabet (Anthropic: `^[a-zA-Z0-9_-]{1,128}$`). */
+const PROVIDER_UNSAFE_TOOL_ID_CHARS = /[^a-zA-Z0-9_-]/g
+
+/**
+ * The id with every character no provider accepts in a function name replaced by `_`. A custom
+ * tool's canonical id carries its display title (`custom_Get Weather`), which execution resolves by
+ * title, so only the wire id is made safe and the canonical id keeps the title.
+ */
+function toProviderSafeToolId(toolId: string): string {
+ return toolId.replace(PROVIDER_UNSAFE_TOOL_ID_CHARS, '_')
+}
export interface ProviderToolIdentityMap {
/** Provider-only tool id to the canonical registry id used for execution and observability. */
@@ -18,10 +29,12 @@ function buildProviderAlias(toolId: string, occurrence: number, attempt: number)
}
/**
- * Gives duplicate configured tools deterministic provider-safe wire ids.
+ * Gives configured tools deterministic provider-safe wire ids.
*
- * The first occurrence and every already-unique tool keep their existing id for backwards
- * compatibility. Later occurrences receive opaque ordinal aliases; resource and credential ids
+ * The first occurrence and every already-unique tool with a provider-safe id keep it for backwards
+ * compatibility. An id with characters no provider accepts (a custom tool titled `Get Weather`) is
+ * sent with them replaced by `_`, and one longer than the strictest provider limit is cut to it;
+ * when either collides with another tool's id it is aliased like a later occurrence. Later occurrences receive opaque ordinal aliases; resource and credential ids
* never enter the provider-visible name. Tool objects are updated in place so their instance-bound
* params and secret provenance remain attached to the exact object selected by provider adapters.
*/
@@ -46,11 +59,12 @@ export function assignProviderToolIdentities<
const occurrence = (occurrences.get(canonicalId) ?? 0) + 1
occurrences.set(canonicalId, occurrence)
- let wireId = canonicalId
- if (usedWireIds.has(wireId)) {
+ const safeId = toProviderSafeToolId(canonicalId)
+ let wireId = safeId.slice(0, MAX_PROVIDER_TOOL_ID_LENGTH)
+ if (usedWireIds.has(wireId) || (wireId !== canonicalId && reservedIds.has(wireId))) {
let attempt = 0
do {
- wireId = buildProviderAlias(canonicalId, occurrence, attempt)
+ wireId = buildProviderAlias(safeId, occurrence, attempt)
attempt += 1
} while (reservedIds.has(wireId) || usedWireIds.has(wireId))
}
diff --git a/packages/sim-cli/README.md b/packages/sim-cli/README.md
index 016e5b84b0f..be28a0b24b9 100644
--- a/packages/sim-cli/README.md
+++ b/packages/sim-cli/README.md
@@ -322,8 +322,11 @@ Success exits 0 and most failures exit 1. Two failures any command can hit have
`401` (run `sim login`), and 9 when the API answered `429`. Every `GET`, and the source applies,
creates and dry runs, are resent up to 6 times first, each wait the longer of `Retry-After` (up to
60 seconds) and a backoff of about 1 to 30 seconds; any other write exits 9 at once. Before this
-release both exited 1. Command-specific codes:
-[Exit codes](https://docs.sim.ai/cli/scripting#exit-codes).
+release both exited 1. Ctrl-C exits 130; a source apply or create already sent is finished and
+saved into its directory first. The `workflows source` and `workspaces source` commands also use
+5 (stale), 6 (a locked workflow or folder; a locked block exits 1), 7 (too large), 8 (a workspace
+apply stopped part-way), 10 (busy), 11 (status: local changes) and 12 (status: the server
+changed). Command-specific codes: [Exit codes](https://docs.sim.ai/cli/scripting#exit-codes).
## Workflows and workspaces as code
@@ -342,8 +345,8 @@ sim workflows source apply --dir ./support-triage --yes # save the draft
`sim workspaces source get|check|status|diff|apply --dir ` does the same for
tables, knowledge bases, files, custom tools, skills, MCP servers, workflows and
-deployments in one directory. `sim workflows source create --dir ` starts a
-new workflow.
+deployments in one directory. `sim workflows source create --dir --workspace `
+starts a new workflow in that workspace.
Coding agents (Claude Code, Cursor, Codex) start at `.sim/AGENTS.md`, which `get`
writes into the directory. The loop is get, edit `workflow.ts`, check, status,
diff --git a/packages/sim-cli/src/commands/auth.test.ts b/packages/sim-cli/src/commands/auth.test.ts
index 700c1e39ad8..c19a122dcff 100644
--- a/packages/sim-cli/src/commands/auth.test.ts
+++ b/packages/sim-cli/src/commands/auth.test.ts
@@ -218,6 +218,7 @@ describe('login command', () => {
mocks.createInterface.mockReturnValue({
question: vi.fn(async () => 'yes'),
close: vi.fn(),
+ once: vi.fn(),
})
vi.spyOn(console, 'log').mockImplementation(() => {})
})
@@ -269,6 +270,7 @@ describe('login command', () => {
mocks.createInterface.mockReturnValue({
question: vi.fn(async () => 'no'),
close: vi.fn(),
+ once: vi.fn(),
})
await login()
@@ -463,6 +465,7 @@ describe('profiles command', () => {
mocks.createInterface.mockReturnValue({
question: vi.fn(async () => '1'),
close: vi.fn(),
+ once: vi.fn(),
})
vi.spyOn(console, 'log').mockImplementation(() => {})
})
diff --git a/packages/sim-cli/src/commands/auth.ts b/packages/sim-cli/src/commands/auth.ts
index fb10fa017b1..0641916a2bd 100644
--- a/packages/sim-cli/src/commands/auth.ts
+++ b/packages/sim-cli/src/commands/auth.ts
@@ -1,11 +1,11 @@
import { spawn } from 'node:child_process'
import { randomBytes } from 'node:crypto'
-import { createInterface } from 'node:readline/promises'
import { getErrorMessage } from '@sim/utils/errors'
import { toRecord } from '@sim/utils/object'
import { Command, Option } from 'commander'
import { printLine } from '#sim-cli/output/io'
import { styles } from '#sim-cli/output/presentation'
+import { askLine } from '#sim-cli/terminal/line-input'
import {
buildApprovalUrl,
createAuthRequest,
@@ -120,15 +120,14 @@ async function confirmProfileOverwrite(profileName: string): Promise {
)
}
- const prompt = createInterface({ input: process.stdin, output: process.stderr })
- try {
- const answer = await prompt.question(
+ const answer = (
+ await askLine(
`Profile "${redact(profileName)}" already exists. Replace its login and defaults? (y/N) `
)
- return answer.trim().toLowerCase() === 'y' || answer.trim().toLowerCase() === 'yes'
- } finally {
- prompt.close()
- }
+ )
+ ?.trim()
+ .toLowerCase()
+ return answer === 'y' || answer === 'yes'
}
function selectedProfileName(command: Command): string {
@@ -309,20 +308,16 @@ async function chooseWorkspace(
printLine(` ${index + 1}) ${safeOneLine(workspace.name)} (${workspace.id})`)
}
- const prompt = createInterface({ input: process.stdin, output: process.stderr })
- try {
- const answer = await prompt.question(`Choose a workspace [1-${workspaces.length}]: `)
- const selected = Number(answer.trim())
- if (!Number.isInteger(selected) || selected < 1 || selected > workspaces.length) {
- throw new SimApiError(
- `Invalid workspace selection "${safeOneLine(answer)}". Choose a number from 1 to ${workspaces.length}.`,
- 0
- )
- }
- return workspaces[selected - 1]
- } finally {
- prompt.close()
+ const answer = await askLine(`Choose a workspace [1-${workspaces.length}]: `)
+ if (answer === null) throw new SimApiError('No workspace was chosen.', 0, 'CANCELLED')
+ const selected = Number(answer.trim())
+ if (!Number.isInteger(selected) || selected < 1 || selected > workspaces.length) {
+ throw new SimApiError(
+ `Invalid workspace selection "${safeOneLine(answer)}". Choose a number from 1 to ${workspaces.length}.`,
+ 0
+ )
}
+ return workspaces[selected - 1]
}
function addProfileCommand(): Command {
diff --git a/packages/sim-cli/src/commands/protocol/files-get.ts b/packages/sim-cli/src/commands/protocol/files-get.ts
index c6b35a34ae9..c587ea7b245 100644
--- a/packages/sim-cli/src/commands/protocol/files-get.ts
+++ b/packages/sim-cli/src/commands/protocol/files-get.ts
@@ -6,6 +6,7 @@ import { Readable, type Writable } from 'node:stream'
import { pipeline } from 'node:stream/promises'
import type { Command } from 'commander'
import { writeStdout } from '#sim-cli/output/io'
+import { interruptsDeferred } from '#sim-cli/terminal/interrupts'
import { embedStore } from '../../embed-context'
import { V2_OPERATIONS } from '../../generated/v2-api'
import {
@@ -125,6 +126,9 @@ function reRaise(signal: NodeJS.Signals): void {
* The removal is synchronous because the termination that follows gives an
* async `rm` no turn to run.
*
+ * An interrupt held by {@link interruptsDeferred} is left alone: the deferral
+ * raises it again when its write is done, and the cleanup runs then.
+ *
* Exported for its own test: driving it through a real interrupt would take the
* test runner down with it.
*/
@@ -134,6 +138,7 @@ export function removeStagingOnSignal(
): () => void {
const installed = STAGE_SIGNALS.map((signal) => {
const onSignal = () => {
+ if (interruptsDeferred()) return
process.off(signal, onSignal)
const directory = stagingDirectory()
if (directory) {
diff --git a/packages/sim-cli/src/commands/protocol/index.ts b/packages/sim-cli/src/commands/protocol/index.ts
index 31782f51591..d9b37e51f24 100644
--- a/packages/sim-cli/src/commands/protocol/index.ts
+++ b/packages/sim-cli/src/commands/protocol/index.ts
@@ -21,6 +21,7 @@ import { attachWorkflowRunWait } from './workflow-run-wait'
import { attachWorkflowSourceApply, attachWorkflowSourceCreate } from './workflow-source-apply'
import { attachWorkflowSourceCheck } from './workflow-source-check'
import { attachWorkflowSourceGet } from './workflow-source-get'
+import { attachWorkflowSourceParams } from './workflow-source-params'
import { attachWorkflowSourceDiff, attachWorkflowSourceStatus } from './workflow-source-status'
import { attachWorkspaceOperationWait } from './workspace-operation-wait'
import { attachWorkspaceSourceApply } from './workspace-source-apply'
@@ -98,6 +99,7 @@ export function attachProtocolCommands(program: Command): void {
attachWorkflowSourceCheck(source)
attachWorkflowSourceStatus(source)
attachWorkflowSourceDiff(source)
+ attachWorkflowSourceParams(source)
const workspaces = group(program, 'workspaces')
attachWorkspaceOperationWait(group(workspaces, 'operations'))
diff --git a/packages/sim-cli/src/commands/protocol/result.test.ts b/packages/sim-cli/src/commands/protocol/result.test.ts
new file mode 100644
index 00000000000..9e8dd331e65
--- /dev/null
+++ b/packages/sim-cli/src/commands/protocol/result.test.ts
@@ -0,0 +1,32 @@
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+import { printProtocolResult } from './result'
+
+const ESC = String.fromCharCode(27)
+const BEL = String.fromCharCode(7)
+
+let logged: string[]
+
+beforeEach(() => {
+ logged = []
+ vi.spyOn(console, 'log').mockImplementation((line: string) => {
+ logged.push(line)
+ })
+})
+
+describe('printProtocolResult', () => {
+ it.each(['text', 'table'] as const)(
+ 'sanitizes a display override built from server text in %s output',
+ (format) => {
+ const key = `leads${ESC}]0;pwned${BEL}${ESC}[2A`
+ printProtocolResult(
+ format,
+ { created: [{ kind: 'table', key }] },
+ { created: `table ${key}` }
+ )
+
+ expect(logged.join('\n')).not.toContain(ESC)
+ expect(logged.join('\n')).not.toContain(BEL)
+ expect(logged.join('\n')).toContain('table leads')
+ }
+ )
+})
diff --git a/packages/sim-cli/src/commands/protocol/result.ts b/packages/sim-cli/src/commands/protocol/result.ts
index 8034aa06e22..d7d39852fc8 100644
--- a/packages/sim-cli/src/commands/protocol/result.ts
+++ b/packages/sim-cli/src/commands/protocol/result.ts
@@ -1,6 +1,6 @@
import { isRecordLike } from '@sim/utils/object'
import type { OutputFormat } from '../../config/index'
-import { printRecord, text } from '../../output/render'
+import { printRecord, sanitize, text } from '../../output/render'
/** One result value as a field: a record, or a list holding one, as compact JSON. */
function fieldText(value: unknown): string {
@@ -8,10 +8,20 @@ function fieldText(value: unknown): string {
return text(nested ? JSON.stringify(value) : value)
}
-export function printProtocolResult(format: OutputFormat, result: Record): void {
+/**
+ * Prints a protocol command's result: the record itself in a machine format, else one line per
+ * field, with `display` replacing the text of the fields it names (a list a person reads as a
+ * sentence rather than as JSON). An override is built from server text (resource keys, ids), so it
+ * is sanitized like every other field.
+ */
+export function printProtocolResult(
+ format: OutputFormat,
+ result: Record,
+ display: Readonly> = {}
+): void {
const fields = Object.entries(result).map<[string, string]>(([key, value]) => [
key,
- fieldText(value),
+ display[key] === undefined ? fieldText(value) : sanitize(display[key]),
])
printRecord(format, fields, result)
}
diff --git a/packages/sim-cli/src/commands/protocol/source-apply-interrupt.test.ts b/packages/sim-cli/src/commands/protocol/source-apply-interrupt.test.ts
new file mode 100644
index 00000000000..b3a2fd47a1e
--- /dev/null
+++ b/packages/sim-cli/src/commands/protocol/source-apply-interrupt.test.ts
@@ -0,0 +1,158 @@
+import { spawn } from 'node:child_process'
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
+import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'
+import type { AddressInfo } from 'node:net'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { afterEach, beforeEach, describe, expect, it } from 'vitest'
+
+const WORKFLOW_ID = '00000000-0000-4000-8000-00000000000a'
+const WORKSPACE_ID = 'ws_local'
+const BASE = {
+ workflowId: WORKFLOW_ID,
+ workspaceId: WORKSPACE_ID,
+ fingerprint: 'sf1:a',
+ catalogHash: 'h',
+}
+const LOCK = { version: 1, blocks: {}, variables: {} }
+const ENTRY = "// @sim-sdk 1\nexport default workflow('W', () => {})\n"
+/** How long the server holds the committing request before it answers. */
+const COMMIT_DELAY_MS = 1_500
+
+function applied() {
+ return {
+ data: {
+ workflowId: WORKFLOW_ID,
+ workspaceId: WORKSPACE_ID,
+ dryRun: false,
+ applied: true,
+ diff: {
+ noop: false,
+ blocks: { added: [{ id: 'b-1', key: 'start' }], removed: [], renamed: [], modified: [] },
+ edges: { added: 1, removed: 0 },
+ variables: { added: [], removed: [], modified: [] },
+ },
+ diagnostics: [],
+ files: { 'workflow.ts': `${ENTRY}// stored\n` },
+ lock: LOCK,
+ base: { ...BASE, fingerprint: 'sf1:b' },
+ mintedIds: {},
+ notes: [],
+ needsRedeployment: true,
+ },
+ }
+}
+
+interface InterruptedRun {
+ code: number | null
+ signal: NodeJS.Signals | null
+ stderr: string
+}
+
+let dir: string
+let server: Server
+let endpoint: string
+/** Resolves the moment the server receives the committing request. */
+let onCommit: () => void
+
+beforeEach(async () => {
+ dir = mkdtempSync(join(tmpdir(), 'sim-source-interrupt-'))
+ mkdirSync(join(dir, 'src'))
+ mkdirSync(join(dir, 'src/.sim'))
+ writeFileSync(join(dir, 'src/workflow.ts'), ENTRY)
+ writeFileSync(join(dir, 'src/.sim/lock.json'), JSON.stringify(LOCK))
+ writeFileSync(join(dir, 'src/.sim/base.json'), JSON.stringify(BASE))
+ server = createServer((request: IncomingMessage, response: ServerResponse) => {
+ request.resume()
+ request.on('end', () => {
+ onCommit()
+ setTimeout(() => {
+ response.setHeader('content-type', 'application/json')
+ response.end(JSON.stringify(applied()))
+ }, COMMIT_DELAY_MS)
+ })
+ })
+ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve))
+ endpoint = `http://127.0.0.1:${(server.address() as AddressInfo).port}`
+})
+
+afterEach(async () => {
+ server.closeAllConnections()
+ await new Promise((resolve) => server.close(() => resolve()))
+ rmSync(dir, { recursive: true, force: true })
+})
+
+/**
+ * Runs `sim workflows source apply --yes` as a real process and sends it SIGINT `interrupts`
+ * times once the server holds its commit, 100 ms apart.
+ */
+function applyInterrupted(interrupts: number): Promise {
+ const child = spawn(
+ 'bun',
+ [
+ join(import.meta.dirname, '../../index.ts'),
+ '--output',
+ 'json',
+ 'workflows',
+ 'source',
+ 'apply',
+ '--yes',
+ '--dir',
+ join(dir, 'src'),
+ ],
+ {
+ env: {
+ ...process.env,
+ SIM_CONFIG_DIR: join(dir, 'config'),
+ SIM_ENDPOINT: endpoint,
+ SIM_API_KEY: 'sk-test',
+ SIM_WORKSPACE: WORKSPACE_ID,
+ SIM_NO_UPDATE_CHECK: '1',
+ SIM_CLI_TELEMETRY_DISABLED: '1',
+ },
+ stdio: ['ignore', 'pipe', 'pipe'],
+ }
+ )
+ let stderr = ''
+ child.stderr.setEncoding('utf8').on('data', (chunk: string) => {
+ stderr += chunk
+ })
+ child.stdout.resume()
+ onCommit = () => {
+ for (let index = 0; index < interrupts; index += 1) {
+ setTimeout(() => child.kill('SIGINT'), 100 * (index + 1))
+ }
+ }
+ return new Promise((resolve, reject) => {
+ child.on('error', reject)
+ child.on('close', (code, signal) => resolve({ code, signal, stderr }))
+ })
+}
+
+function savedFingerprint(): string {
+ return JSON.parse(readFileSync(join(dir, 'src/.sim/base.json'), 'utf8')).fingerprint
+}
+
+describe('Ctrl-C while a workflow apply is committing', () => {
+ it('waits for the answer, saves it into the directory, then ends by the interrupt', async () => {
+ const run = await applyInterrupted(1)
+
+ expect(run.signal ?? run.code).toSatisfy(
+ (status: unknown) => status === 'SIGINT' || status === 130
+ )
+ expect(run.stderr).toContain('the apply was already sent')
+ expect(savedFingerprint()).toBe('sf1:b')
+ expect(readFileSync(join(dir, 'src/workflow.ts'), 'utf8')).toContain('// stored')
+ }, 30_000)
+
+ it('stops at a second Ctrl-C and says the apply may have been saved and how to recover', async () => {
+ const run = await applyInterrupted(2)
+
+ expect(run.signal ?? run.code).toSatisfy(
+ (status: unknown) => status === 'SIGINT' || status === 130
+ )
+ expect(run.stderr).toContain('Stopped before the apply answered. It may have been saved.')
+ expect(run.stderr).toContain(`sim workflows source get ${WORKFLOW_ID}`)
+ expect(savedFingerprint()).toBe('sf1:a')
+ }, 30_000)
+})
diff --git a/packages/sim-cli/src/commands/protocol/source-dir.test.ts b/packages/sim-cli/src/commands/protocol/source-dir.test.ts
index f53b24a849e..0f01c8f1ade 100644
--- a/packages/sim-cli/src/commands/protocol/source-dir.test.ts
+++ b/packages/sim-cli/src/commands/protocol/source-dir.test.ts
@@ -150,27 +150,55 @@ describe('withSourceDirLock', () => {
expect(existsSync(join(dir, SOURCE_DIR_LOCK))).toBe(false)
})
- it('prints what the interrupted command may have done, then ends with the signal', () => {
- const script = `
+ /**
+ * The shape of `workspaces source apply`: the commit and its write-back run deferred inside the
+ * directory lock, whose own interrupt handler must wait for them too.
+ */
+ function deferredWriteScript(interrupts: number): string {
+ return `
+ import { writeFileSync } from 'node:fs'
+ import { join } from 'node:path'
+ import { setTimeout as sleep } from 'node:timers/promises'
import { withSourceDirLock } from ${JSON.stringify(join(import.meta.dirname, 'source-dir.ts'))}
- let sending = false
- await withSourceDirLock(
- process.env.SOURCE_DIR,
- async () => {
- sending = true
- process.kill(process.pid, 'SIGINT')
- await new Promise(() => {})
- },
- { interruptNote: () => (sending ? 'It may have been written.' : null) }
- )
+ import { deferInterrupts } from ${JSON.stringify(join(import.meta.dirname, '../../terminal/interrupts.ts'))}
+ await withSourceDirLock(process.env.SOURCE_DIR, async () => {
+ await deferInterrupts(
+ async () => {
+ for (let sent = 0; sent < ${interrupts}; sent += 1) {
+ process.kill(process.pid, 'SIGINT')
+ await sleep(50)
+ }
+ await sleep(300)
+ writeFileSync(join(process.env.SOURCE_DIR, 'saved'), 'yes')
+ },
+ { waiting: 'Waiting for the write.', abandoned: 'Stopped before the write answered.' }
+ )
+ await new Promise(() => {})
+ })
`
- const interrupted = spawnSync('bun', ['-e', script], {
+ }
+
+ it('finishes a deferred write at an interrupt, then releases the lock and ends with the signal', () => {
+ const interrupted = spawnSync('bun', ['-e', deferredWriteScript(1)], {
+ env: { ...process.env, SOURCE_DIR: dir },
+ timeout: 20_000,
+ })
+
+ expect(interrupted.signal).toBe('SIGINT')
+ expect(interrupted.stderr.toString()).toContain('Waiting for the write.')
+ expect(existsSync(join(dir, 'saved'))).toBe(true)
+ expect(existsSync(join(dir, SOURCE_DIR_LOCK))).toBe(false)
+ })
+
+ it('abandons a deferred write at a second interrupt, saying so, and releases the lock', () => {
+ const interrupted = spawnSync('bun', ['-e', deferredWriteScript(2)], {
env: { ...process.env, SOURCE_DIR: dir },
timeout: 20_000,
})
expect(interrupted.signal).toBe('SIGINT')
- expect(interrupted.stderr.toString()).toContain('It may have been written.')
+ expect(interrupted.stderr.toString()).toContain('Stopped before the write answered.')
+ expect(existsSync(join(dir, 'saved'))).toBe(false)
expect(existsSync(join(dir, SOURCE_DIR_LOCK))).toBe(false)
})
diff --git a/packages/sim-cli/src/commands/protocol/source-dir.ts b/packages/sim-cli/src/commands/protocol/source-dir.ts
index cd9c8561523..e3d4d6400ea 100644
--- a/packages/sim-cli/src/commands/protocol/source-dir.ts
+++ b/packages/sim-cli/src/commands/protocol/source-dir.ts
@@ -37,14 +37,15 @@ import {
SOURCE_LOCK_FILE,
} from '@sim/workflow-code/limits'
import { fingerprintOf, KIND_DIRS } from '@sim/workspace-code'
-import { printError } from '#sim-cli/output/io'
+import type { Command } from 'commander'
+import type { ResolvedProfile } from '../../config/index'
import { embedStore } from '../../embed-context'
import { SOURCE_BUSY_EXIT_CODE } from '../../exit-codes'
import { SimApiError } from '../../http/client'
import { embeddedFileContent } from '../../transfer/local-file'
import { removeStagingOnSignal, saveToFile } from './files-get'
-/** Generated SDK declarations: every block in the workspace catalog, plus `workspace.d.ts`. */
+/** Generated SDK declarations: the blocks the source uses (every block's are cached), plus `workspace.d.ts`. */
export const SOURCE_SDK_DIR = '.sim/sdk'
/** A short guide for an agent working in the directory; never a top-level `AGENTS.md`. */
export const SOURCE_AGENTS_FILE = '.sim/AGENTS.md'
@@ -63,6 +64,18 @@ export const SOURCE_GITIGNORE_FILE = '.gitignore'
* edit from a file it may replace. The CLI's own record: never sent to the server.
*/
export const SOURCE_WRITTEN_FILE = '.sim/written.json'
+/**
+ * The workspace a starter written by `workflows source create` was typed for, so the create that
+ * follows targets that workspace. The CLI's own record: never sent, and removed once the workflow
+ * exists.
+ */
+export const SOURCE_STARTER_FILE = '.sim/starter.json'
+/**
+ * The edges of the workflow a get or apply printed, by lock key, with the base fingerprint they
+ * belong to, so an offline check tells an edge the stored workflow holds from one the source
+ * adds. The CLI's own record: never sent to the server.
+ */
+export const SOURCE_STORED_EDGES_FILE = '.sim/stored-edges.json'
/**
* Where a workspace get keeps the blobs it verified until they move into place. A get that fails
* leaves them there, so the next one downloads only the rest; the one that completes removes the
@@ -99,7 +112,7 @@ const GITATTRIBUTES_LINES = [
`${SOURCE_ENTRY_FILE} text eol=lf`,
`${SOURCE_FUNCTIONS_DIR}/** text eol=lf`,
]
-/** Files read or written at once on a local disk; the declarations run to thousands of small files. */
+/** Files read or written at once on a local disk; a catalog's declarations run to hundreds of small files. */
export const FILE_CONCURRENCY = 32
/**
@@ -170,6 +183,51 @@ export async function readSourceJson(dir: string, path: string): Promise {
+ const dir: unknown = action.getOptionValue('dir')
+ if (typeof dir === 'string' && !embedStore.getStore()) {
+ action.setOptionValue('dir', resolve(dir))
+ }
+ })
+}
+
+/**
+ * The workspace a source command acts on for a directory read from `readFrom`: that workspace,
+ * whatever the profile's default, so a directory stays bound to where it came from. Only a
+ * `--workspace` naming another one is refused, with `next` saying what to run instead (given the
+ * workspace the directory was read from and the one named). A directory that records no workspace
+ * takes the profile's (`null` when none is selected).
+ */
+export function directoryWorkspace(
+ dir: string,
+ readFrom: string | null,
+ profile: Pick,
+ next: (readFrom: string, named: string) => string
+): string | null {
+ const named = profile.workspaceId
+ if (readFrom === null) return named
+ if (profile.sources.workspaceId !== 'flag' || named === null || named === readFrom) {
+ return readFrom
+ }
+ throw new SimApiError(
+ `${dir} was read from workspace ${readFrom}, but --workspace names ${named}. ${next(readFrom, named)}`,
+ 0,
+ 'SOURCE_OTHER_WORKSPACE'
+ )
+}
+
+/** The workspace {@link SOURCE_STARTER_FILE} names, or `null` when the directory holds no starter. */
+export async function readStarterWorkspace(dir: string): Promise {
+ const workspaceId = toRecord(await readSourceJson(dir, SOURCE_STARTER_FILE)).workspaceId
+ return typeof workspaceId === 'string' ? workspaceId : null
+}
+
/**
* A line file (`.gitattributes`, `.gitignore`) with `lines` appended where missing, or `null`
* when it has them all.
@@ -283,12 +341,17 @@ export interface SourceDirChange {
}
/**
- * The files that record what the directory was read from: each workflow's lock, base and
- * `.sim/written.json`, and the workspace base. They move into place only after every other file,
+ * The files that record what the directory was read from: each workflow's lock, base,
+ * `.sim/written.json` and stored edges, and the workspace base. They move into place only after every other file,
* so a write cut short leaves the old records beside whatever text did land: the next apply is
* refused as stale, never sent as if the old text were current.
*/
-const RECORD_FILES = [SOURCE_BASE_FILE, SOURCE_LOCK_FILE, SOURCE_WRITTEN_FILE]
+const RECORD_FILES = [
+ SOURCE_BASE_FILE,
+ SOURCE_LOCK_FILE,
+ SOURCE_WRITTEN_FILE,
+ SOURCE_STORED_EDGES_FILE,
+]
function isRecordFile(path: string): boolean {
return RECORD_FILES.some((file) => path === file || path.endsWith(`/${file}`))
@@ -824,14 +887,7 @@ export async function takeOverStaleLock(lock: string, seen: LockSighting): Promi
export async function withSourceDirLock(
dir: string,
run: () => Promise,
- options: {
- readOnly?: boolean
- /**
- * What to print when an interrupt (Ctrl-C) ends the command: read when it arrives, so a
- * command can name what a write in flight may have done; `null` prints nothing.
- */
- interruptNote?: () => string | null
- } = {}
+ options: { readOnly?: boolean } = {}
): Promise {
if (embedStore.getStore()) return run()
const absent = async (path: string) =>
@@ -888,14 +944,7 @@ export async function withSourceDirLock(
const owner: LockOwner = { pid: process.pid, host: hostname() }
// An interrupt (Ctrl-C) releases the lock too: a lock left behind is taken over only by a
// command on the same host, so one left on a folder shared with a container would stay.
- const disposeSignalRelease = removeStagingOnSignal(
- () => lock,
- (signal) => {
- const note = options.interruptNote?.()
- if (note) printError(note)
- process.kill(process.pid, signal)
- }
- )
+ const disposeSignalRelease = removeStagingOnSignal(() => lock)
try {
await writeFile(join(lock, LOCK_OWNER_FILE), JSON.stringify(owner))
return await run()
diff --git a/packages/sim-cli/src/commands/protocol/source-exit-codes.test.ts b/packages/sim-cli/src/commands/protocol/source-exit-codes.test.ts
new file mode 100644
index 00000000000..e2d3979b4e8
--- /dev/null
+++ b/packages/sim-cli/src/commands/protocol/source-exit-codes.test.ts
@@ -0,0 +1,35 @@
+import { readFileSync } from 'node:fs'
+import { join } from 'node:path'
+import { describe, expect, it } from 'vitest'
+import * as exitCodes from '../../exit-codes'
+import { WORKFLOW_TREE } from './workflow-source-get'
+import { WORKSPACE_TREE } from './workspace-source-dir'
+
+/**
+ * Every code a source command can exit with: each `*_EXIT_CODE` the CLI defines except the
+ * run-wait timeout (4), which no source command returns, plus 0, 1 and Ctrl-C's 130.
+ */
+const SOURCE_EXIT_CODES = [
+ 0,
+ 1,
+ ...Object.entries(exitCodes)
+ .filter(([name]) => name.endsWith('_EXIT_CODE') && name !== 'STILL_RUNNING_EXIT_CODE')
+ .map(([, code]) => code as number),
+ 130,
+].sort((a, b) => a - b)
+
+const WORKFLOW_SDK_README = readFileSync(
+ join(import.meta.dirname, '../../../../workflow-sdk/README.md'),
+ 'utf8'
+)
+
+describe('the exit-code tables an agent or a reader relies on', () => {
+ it.each([
+ ['the workflow directory guide', WORKFLOW_TREE.agents],
+ ['the workspace directory guide', WORKSPACE_TREE.agents],
+ ['the workflow SDK README', WORKFLOW_SDK_README],
+ ])('%s lists every code a source command exits with', (_name, text) => {
+ const listed = [...text.matchAll(/^\| (\d+) \|/gm)].map((match) => Number(match[1]))
+ expect(listed).toEqual(SOURCE_EXIT_CODES)
+ })
+})
diff --git a/packages/sim-cli/src/commands/protocol/source-exit-codes.ts b/packages/sim-cli/src/commands/protocol/source-exit-codes.ts
new file mode 100644
index 00000000000..9a4fed61347
--- /dev/null
+++ b/packages/sim-cli/src/commands/protocol/source-exit-codes.ts
@@ -0,0 +1,105 @@
+import { DEFERRED_INTERRUPT_BOUND_MS } from '#sim-cli/terminal/interrupts'
+import {
+ AUTH_REQUIRED_EXIT_CODE,
+ RATE_LIMITED_EXIT_CODE,
+ SOURCE_BUSY_EXIT_CODE,
+ SOURCE_DRIFT_EXIT_CODE,
+ SOURCE_LOCAL_CHANGES_EXIT_CODE,
+ SOURCE_LOCKED_EXIT_CODE,
+ SOURCE_PARTIAL_EXIT_CODE,
+ SOURCE_PLANNED_EXIT_CODE,
+ SOURCE_STALE_EXIT_CODE,
+ SOURCE_TOO_LARGE_EXIT_CODE,
+} from '../../exit-codes'
+
+/** The status a shell reports for a command Ctrl-C ended (128 + SIGINT). */
+const INTERRUPTED_EXIT_CODE = 130
+
+/** How long an interrupted write waits for its answer before it stops without it. */
+const DEFERRED_SECONDS = DEFERRED_INTERRUPT_BOUND_MS / 1000
+
+/**
+ * Every exit code of the `workflows source` and `workspaces source` commands, as the Markdown
+ * table both directory guides (`.sim/AGENTS.md`) print, built from the codes the CLI exits with so
+ * the guides cannot drift from them. `prefix` is the command group the guide is for.
+ */
+export function sourceExitCodesTable(
+ prefix: 'sim workflows source' | 'sim workspaces source'
+): string {
+ const rows: Array<[number, string, string]> = [
+ [0, 'Applied or created, a dry run or check with no errors, or a clean status', 'Go on'],
+ [
+ 1,
+ 'The source has errors (each printed as `file:line:column` with a hint), a locked block (`E_APPLY_LOCKED_BLOCK`), a name another workflow holds (`SOURCE_NAME_TAKEN`), a cancelled prompt, or the command failed',
+ 'Fix what it printed and run it again',
+ ],
+ [
+ SOURCE_STALE_EXIT_CODE,
+ 'Stale (`SOURCE_STALE`, or `E_APPLY_STALE` on a dry run): the server changed what this directory was read from, and nothing was written',
+ `\`${prefix} get --dir \`, reapply your edit there, dry-run, apply`,
+ ],
+ [
+ SOURCE_LOCKED_EXIT_CODE,
+ 'A workflow or its folder is locked (`E_APPLY_LOCKED`); a locked block exits 1 instead',
+ 'Ask someone to unlock it in the editor',
+ ],
+ [
+ SOURCE_TOO_LARGE_EXIT_CODE,
+ 'The source is over the size limits (on a get, status or diff too)',
+ 'Split function code, or remove blocks or resources',
+ ],
+ [
+ SOURCE_PARTIAL_EXIT_CODE,
+ '`sim workspaces source apply` stopped part-way: what was written is saved into the directory',
+ 'Apply again until it exits 0',
+ ],
+ [
+ RATE_LIMITED_EXIT_CODE,
+ "Still rate limited (HTTP 429) after the CLI's retries (up to 6, at most a minute apart); nothing more was written",
+ 'Wait a minute and run the same command again',
+ ],
+ [
+ SOURCE_BUSY_EXIT_CODE,
+ 'Busy: another apply is writing the workspace (`SOURCE_BUSY`), or another sim command is using this directory (`SOURCE_DIR_BUSY`); nothing was written',
+ 'Run it again once the other finished',
+ ],
+ [
+ SOURCE_LOCAL_CHANGES_EXIT_CODE,
+ '`status`: only your edits, which applying writes',
+ 'Dry-run, then apply',
+ ],
+ [
+ SOURCE_DRIFT_EXIT_CODE,
+ '`status`: the server changed since the get (drift, or a conflict with your edits)',
+ 'Follow the `Next:` line it prints',
+ ],
+ [
+ AUTH_REQUIRED_EXIT_CODE,
+ 'Not logged in, the login expired, or the server refused the credential (`AUTH_REQUIRED`); nothing was sent',
+ '`sim login`, or fix or unset `SIM_API_KEY` when the key comes from it; then the same command again',
+ ],
+ [
+ SOURCE_PLANNED_EXIT_CODE,
+ '`apply` or `create` without `--yes` and no terminal to ask: it printed the plan, found no errors, and wrote nothing',
+ 'Run the same command with `--yes`',
+ ],
+ [
+ INTERRUPTED_EXIT_CODE,
+ `Interrupted (Ctrl-C). An apply or create already sent is finished and saved into the directory first, waiting up to ${DEFERRED_SECONDS}s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written`,
+ `\`${prefix} status --dir .\``,
+ ],
+ ]
+ return [
+ '| Code | Meaning | Do |',
+ '|---|---|---|',
+ ...rows.map(([code, meaning, next]) => `| ${code} | ${meaning} | ${next} |`),
+ ].join('\n')
+}
+
+/** The exit codes every source write (apply, create) shares, appended to its help. */
+export const SOURCE_WRITE_EXIT_CODES_HELP = `
+Exit ${AUTH_REQUIRED_EXIT_CODE}: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent.
+Run \`sim login\`, or fix or unset SIM_API_KEY when the key comes from it. Exit ${INTERRUPTED_EXIT_CODE}: interrupted
+(Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to
+${DEFERRED_SECONDS}s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may
+have been written and how to check.`
diff --git a/packages/sim-cli/src/commands/protocol/source-primer.ts b/packages/sim-cli/src/commands/protocol/source-primer.ts
new file mode 100644
index 00000000000..3d77d91b74a
--- /dev/null
+++ b/packages/sim-cli/src/commands/protocol/source-primer.ts
@@ -0,0 +1,111 @@
+/**
+ * The primer a source get prints and writes atop `.sim/AGENTS.md`: the few rules an agent needs to
+ * edit a source directory without reading the guide, plus a snippet for each builder the source
+ * already calls, so the blocks in front of it need no lookup. The guide below it is the reference.
+ */
+import type { BlockCatalog } from '@sim/workflow-code'
+import { SOURCE_BASE_FILE, SOURCE_ENTRY_FILE, SOURCE_LOCK_FILE } from '@sim/workflow-code/limits'
+import { SDK_SPECIAL_BUILDERS } from '@sim/workflow-code/sdk'
+import {
+ builderSnippet,
+ describeBuilder,
+ referencedBuilders,
+ typecheckCommand,
+} from '@sim/workflow-sdk/codegen'
+import {
+ RATE_LIMITED_EXIT_CODE,
+ SOURCE_BUSY_EXIT_CODE,
+ SOURCE_DRIFT_EXIT_CODE,
+ SOURCE_LOCAL_CHANGES_EXIT_CODE,
+ SOURCE_STALE_EXIT_CODE,
+} from '../../exit-codes'
+import { SOURCE_AGENTS_FILE } from './source-dir'
+
+/** Which directory a primer is for. */
+export type PrimerKind = 'workflow' | 'workspace'
+
+/** The primer without its builder snippets stays within this many bytes. */
+export const PRIMER_MAX_BYTES = 3072
+
+/** At most this many snippets, and this many bytes of them; the rest are named for `params`. */
+const MAX_SNIPPETS = 12
+const MAX_SNIPPET_BYTES = 4096
+
+/** The command that prints any builder's params (`workflow-source-params.ts`). */
+const PARAMS_COMMAND = 'sim workflows source params'
+
+/** The rules every source directory shares, after the kind's own edit loop. */
+function sharedRules(group: string): string {
+ return `- **Identity**: a block is its entry in \`${SOURCE_LOCK_FILE}\`, which binds it by its const name or its display name. Change the display name (first argument) freely: the block keeps its id. Renaming the const too keeps the id when its settings, neighbours or references still tie it to the old block; otherwise the apply refuses it as a delete (\`E_APPLY_BLOCK_DELETE\`), and the dry run names it. Safest: keep the const in the same apply.
+- **References**: in \`${SOURCE_ENTRY_FILE}\` read an output as \`x.content\`, a loop or parallel item as \`p.currentItem\` (\`p.index\`), text as \`\` t\`…\` \`\`, env as \`env.NAME\`. In a function code file (\`codeFile('./functions/x.js')\`) read an output as \`$ref..\`, the key being the display name lowercased with whitespace and dots removed ("Triage Agent" → \`$ref.triageagent.content\`); in a loop body \`$ref.loop.currentItem\` / \`$ref.loop.index\`, in a parallel \`$ref.parallel.currentItem\` / \`$ref.parallel.index\`; env as \`$env.NAME\`.
+- **Look up a builder**: \`${PARAMS_COMMAND} \` (\`table.upsertRow\`, \`agent\`, \`trigger.schedule\`; a namespace such as \`gmail\` lists its builders), offline.
+- **Exit codes**: 0 ok, 1 errors (fix what it printed), ${SOURCE_STALE_EXIT_CODE} stale (\`${group} get --dir \`, redo the edit there), ${RATE_LIMITED_EXIT_CODE} rate limited, ${SOURCE_BUSY_EXIT_CODE} busy, ${SOURCE_LOCAL_CHANGES_EXIT_CODE}/${SOURCE_DRIFT_EXIT_CODE} \`status\`: your edits / the server changed. Every code: \`${SOURCE_AGENTS_FILE}\`.`
+}
+
+const WORKFLOW_PRIMER = `# Sim workflow source: primer
+
+\`${SOURCE_ENTRY_FILE}\` is a Sim workflow as TypeScript; the server compiles it statically and nothing in it runs. Commands read the workflow id from \`${SOURCE_BASE_FILE}\`; run them from this directory.
+
+**Edit loop**: edit \`${SOURCE_ENTRY_FILE}\` (and \`functions/*\`) → \`sim workflows source check --dir .\` (offline; it also adds new blocks to \`tsconfig.json\`) → \`${typecheckCommand()}\` → \`sim workflows source apply --dir . --dry-run\` → \`--yes\`. Rerun each step until it exits 0. Apply saves the draft in canonical form: read \`${SOURCE_ENTRY_FILE}\` again before the next edit.
+
+- **Deletes**: removing a block's declaration deletes the block only with \`apply --allow-delete\`; without it the apply is refused (\`E_APPLY_BLOCK_DELETE\`).
+- **Deploy**: apply saves a draft; \`apply --deploy\` also deploys it, or \`sim workflows deploy \` publishes the saved draft later.
+${sharedRules('sim workflows source')}
+`
+
+const WORKSPACE_PRIMER = `# Sim workspace source: primer
+
+This directory is a Sim workspace as files (\`tables/\`, \`workflows//workflow.ts\`, …); the server reads them statically and nothing runs. Commands read the workspace id from \`${SOURCE_BASE_FILE}\`; run them from this directory.
+
+**Edit loop**: edit the files → \`sim workspaces source check --dir .\` (offline; regenerates \`.sim/sdk/workspace.d.ts\`, adds new blocks to \`tsconfig.json\`) → \`${typecheckCommand()}\` → \`sim workspaces source apply --dir . --dry-run\` (the plan) → \`--yes\`. Rerun each step until it exits 0. Apply rewrites the directory from what it stored: read a file again before the next edit.
+
+- **Deletes**: a removed file (a resource this directory managed), table column or workflow block is deleted only with \`apply --allow-delete\`; without it the apply is refused. A kind is managed only when its folder exists. Deletes cannot be undone.
+- **Deploy**: \`deployments/.ts\` (\`defineDeployment({ live: 'draft' })\`) deploys that workflow on apply; without one, apply never deploys.
+${sharedRules('sim workspaces source')}
+`
+
+/** The primer for a `kind` directory, without builder snippets. */
+export function sourcePrimer(kind: PrimerKind): string {
+ return kind === 'workflow' ? WORKFLOW_PRIMER : WORKSPACE_PRIMER
+}
+
+/**
+ * A snippet for each builder `sources` call ({@link builderSnippet}), integration builders before
+ * the core ones (`agent`, `fn`), whose shape the source shows. Over the cap, the rest are named
+ * for `params`. Empty when the source calls none.
+ */
+function builderSnippets(catalog: BlockCatalog, sources: Iterable): string {
+ const core = (key: string) => Object.hasOwn(SDK_SPECIAL_BUILDERS, key) || key === 'agent'
+ const keys = referencedBuilders(catalog, sources)
+ const ordered = [...keys.filter((key) => !core(key)), ...keys.filter(core)]
+ const snippets: string[] = []
+ const skipped: string[] = []
+ let bytes = 0
+ for (const key of ordered) {
+ const reference = describeBuilder(catalog, key)
+ if (!reference) continue
+ const snippet = builderSnippet(reference)
+ if (snippets.length >= MAX_SNIPPETS || bytes + snippet.length > MAX_SNIPPET_BYTES) {
+ skipped.push(key)
+ continue
+ }
+ snippets.push(snippet)
+ bytes += snippet.length
+ }
+ if (snippets.length === 0) return ''
+ const rest =
+ skipped.length > 0
+ ? `\nAlso used (\`${PARAMS_COMMAND} \`): ${skipped.map((key) => `\`${key}\``).join(', ')}.\n`
+ : ''
+ return `## Builders this source uses\n\n${snippets.join('\n')}\n${rest}`
+}
+
+/** The primer with the snippets of the builders `sources` call. */
+export function primerFor(
+ kind: PrimerKind,
+ catalog: BlockCatalog,
+ sources: Iterable
+): string {
+ const snippets = builderSnippets(catalog, sources)
+ return snippets ? `${sourcePrimer(kind)}\n${snippets}` : sourcePrimer(kind)
+}
diff --git a/packages/sim-cli/src/commands/protocol/source-status.ts b/packages/sim-cli/src/commands/protocol/source-status.ts
index 96e0d092ea6..820dbcc78e1 100644
--- a/packages/sim-cli/src/commands/protocol/source-status.ts
+++ b/packages/sim-cli/src/commands/protocol/source-status.ts
@@ -12,6 +12,7 @@ import { createTwoFilesPatch, FILE_HEADERS_ONLY } from 'diff'
import type { OutputFormat } from '../../config/index'
import {
AUTH_REQUIRED_CODE,
+ AUTH_REQUIRED_EXIT_CODE,
SOURCE_DRIFT_EXIT_CODE,
SOURCE_LOCAL_CHANGES_EXIT_CODE,
SOURCE_TOO_LARGE_EXIT_CODE,
@@ -47,8 +48,9 @@ read); ${SOURCE_LOCAL_CHANGES_EXIT_CODE} local changes only (applying writes the
directory was read: drift (a resource edited, added or deleted there that the directory leaves
alone) or a conflict (also changed here, so applying it is refused as stale), with or without
local changes; ${SOURCE_TOO_LARGE_EXIT_CODE} the source is too large for the server to read in one dry run; 1 the command
-failed. A server that cannot be reached is not a failure: the status shows the local changes only
-and says so (exit 0 or ${SOURCE_LOCAL_CHANGES_EXIT_CODE}).`
+failed; ${AUTH_REQUIRED_EXIT_CODE} not logged in (AUTH_REQUIRED) or the credential was refused: the status shows the
+local changes only, then run \`sim login\`. A server that cannot be reached is not a failure: the
+status shows the local changes only and says so (exit 0 or ${SOURCE_LOCAL_CHANGES_EXIT_CODE}).`
/** What `diff` compares, and its exit codes. */
export const DIFF_HELP = `
@@ -60,8 +62,12 @@ file or folder (and in a workspace directory by kind:key, such as table:leads).
differences were shown, whether or not there were any; ${SOURCE_TOO_LARGE_EXIT_CODE} the source is too large for the server
to read; 1 the command failed.`
-/** The exit status of a status report: drift or conflict, else local changes, else clean. */
-export function statusExitCode(state: SourceStatus['state']): number {
+/**
+ * The exit status of a status report: no login (`login` holds the command to run), else drift or
+ * conflict, else local changes, else clean.
+ */
+export function statusExitCode(state: SourceStatus['state'], login: string | null): number {
+ if (login) return AUTH_REQUIRED_EXIT_CODE
if (state === 'conflict' || state === 'drift') return SOURCE_DRIFT_EXIT_CODE
return state === 'local' ? SOURCE_LOCAL_CHANGES_EXIT_CODE : 0
}
@@ -69,28 +75,34 @@ export function statusExitCode(state: SourceStatus['state']): number {
/** Gateway statuses that mean the server was not reached, rather than that it refused. */
const UNREACHABLE_STATUSES = new Set([502, 503, 504])
+/**
+ * Whether `error` says this profile cannot authenticate: the CLI's own refusal of a profile with
+ * no login, or a credential the server refused (`401`, which exits 13 like it).
+ */
+export function credentialRefused(error: unknown): boolean {
+ return (
+ error instanceof SimApiError &&
+ ((error.status === 0 && error.code === AUTH_REQUIRED_CODE) || error.status === 401)
+ )
+}
+
/**
* Why the server could not be reached (no connection, a timeout, a gateway that gave up, no login
- * on the profile), as a sentence, or `null` for any other failure (a cancelled request included),
- * which a status reports rather than hides.
+ * on the profile or a credential it refused), as a sentence, or `null` for any other failure (a
+ * cancelled request included), which a status reports rather than hides.
*/
export function unreachableReason(error: unknown): string | null {
if (!(error instanceof SimApiError)) return null
- const offline =
- noAnswer(error, UNREACHABLE_STATUSES) ||
- (error.status === 0 && error.code === AUTH_REQUIRED_CODE)
- if (!offline) return null
+ if (!noAnswer(error, UNREACHABLE_STATUSES) && !credentialRefused(error)) return null
return /[.!?]$/.test(error.message) ? error.message : `${error.message}.`
}
/**
- * The login a status recommends when `error` is the CLI's own refusal of a profile with no login
- * (which a status treats as an unreachable server), or `null` for any other failure.
+ * The login a status recommends when `error` says the profile cannot authenticate (which a status
+ * treats as an unreachable server), or `null` for any other failure.
*/
export function loginCommand(error: unknown, authProfile: string): string | null {
- return error instanceof SimApiError && error.status === 0 && error.code === AUTH_REQUIRED_CODE
- ? `sim login --profile ${authProfile}`
- : null
+ return credentialRefused(error) ? `sim login --profile ${authProfile}` : null
}
const SERVER_WORDS = {
diff --git a/packages/sim-cli/src/commands/protocol/source-tree.ts b/packages/sim-cli/src/commands/protocol/source-tree.ts
index bbc729c84df..f5ba410c59f 100644
--- a/packages/sim-cli/src/commands/protocol/source-tree.ts
+++ b/packages/sim-cli/src/commands/protocol/source-tree.ts
@@ -5,14 +5,19 @@
* the agent guide and the LF lines.
*/
import { createHash } from 'node:crypto'
-import { join } from 'node:path'
-import { isRecordLike } from '@sim/utils/object'
+import { rmSync } from 'node:fs'
+import { mkdir, mkdtemp, readFile, rename, rm, writeFile } from 'node:fs/promises'
+import { dirname, join } from 'node:path'
+import { mapWithConcurrency } from '@sim/utils/concurrency'
+import { isRecordLike, toRecord } from '@sim/utils/object'
import { escapeRegExp } from '@sim/utils/string'
import type { BlockCatalog } from '@sim/workflow-code'
import { catalogBlock } from '@sim/workflow-code/catalog/view'
+import { SOURCE_BASE_FILE } from '@sim/workflow-code/limits'
import {
COOKBOOK_EXAMPLES,
CORE_DTS,
+ checkedBlockTypes,
defaultBlockFiles,
fullBlockFiles,
generateSdkTypes,
@@ -24,7 +29,8 @@ import {
sdkIndexHeader,
type WorkspaceManifest,
} from '@sim/workflow-sdk/codegen'
-import { printError } from '#sim-cli/output/io'
+import { printError, printLine } from '#sim-cli/output/io'
+import type { OutputFormat } from '../../config/index'
import { readJsonFile, writeJsonFile } from '../../config/json-file'
import { configDir } from '../../config/paths'
import { embedStore } from '../../embed-context'
@@ -33,7 +39,10 @@ import type { GetSourceResourcesResponse, GetWorkflowSourceResponse } from '../.
import { SimApiError } from '../../http/client'
import type { OperationClient } from '../../runtime/called-operations'
import { cliVersion } from '../../version'
+import { printProtocolResult } from './result'
import {
+ FILE_CONCURRENCY,
+ readSourceJson,
readSourceText,
SOURCE_SDK_DIR as SDK_DIR,
SOURCE_AGENTS_FILE,
@@ -44,6 +53,7 @@ import {
updateSourceDir,
withMissingLines,
} from './source-dir'
+import { type PrimerKind, primerFor } from './source-primer'
const TSCONFIG_FILE = 'tsconfig.json'
/** Type-checks the source against every current block, for a block the default one leaves out. */
@@ -75,26 +85,61 @@ function isBlockCatalog(value: unknown): value is BlockCatalog {
)
}
-/** Workspace ids are uuids; anything else is never used to name a cache file. */
+/** Workspace ids are uuids and catalog hashes hex; anything else never names a cache file. */
const CACHE_KEY = /^[\w-]+$/
+/** The CLI's catalog cache: every catalog by its hash, and each workspace's current hash. */
+function catalogCacheDir(): string {
+ return join(configDir(), 'cache', 'source-catalog')
+}
+
/**
- * Where the workspace's catalog is cached: one file per workspace in the CLI's config directory,
- * so every `source get` in the workspace (whatever the workflow or output directory) revalidates
- * the same copy. Callers gate the id with `CACHE_KEY` first.
+ * Where a catalog is cached: by its content hash, so the workspace catalog and every per-workflow
+ * catalog share one file per distinct catalog, however many workflows and workspaces name it.
+ * Callers gate the hash with `CACHE_KEY` first.
*/
-function catalogCachePath(workspaceId: string): string {
- return join(configDir(), 'cache', 'source-catalog', `${workspaceId}.json`)
+function catalogCachePath(hash: string): string {
+ return join(catalogCacheDir(), `${hash}.json`)
}
/**
- * The cached catalog, or `null`. Standalone runs only: an embedded run has no local disk of its
- * own to cache on. The file is read without following a link and under a size cap.
+ * Where the hash of a workspace's catalog is recorded, so a get revalidates it by ETag and an
+ * offline command finds it. Callers gate the id with `CACHE_KEY` first.
*/
+function workspaceCatalogPointerPath(workspaceId: string): string {
+ return join(catalogCacheDir(), 'workspaces', `${workspaceId}.json`)
+}
+
+/** A pointer is one hash; a larger file was not written by this CLI. */
+const MAX_POINTER_BYTES = 4096
+
+/**
+ * The cached catalog of `hash`, or `null`. Standalone runs only: an embedded run has no local disk
+ * of its own to cache on. The file is read without following a link and under a size cap, and
+ * kept only when the catalog in it names the hash its file does.
+ */
+export function cachedCatalogByHash(hash: unknown): BlockCatalog | null {
+ if (embedStore.getStore() || typeof hash !== 'string' || !CACHE_KEY.test(hash)) return null
+ const cached = readJsonFile(catalogCachePath(hash), MAX_CATALOG_CACHE_BYTES)
+ return isBlockCatalog(cached) && cached.hash === hash ? cached : null
+}
+
+/** The cached catalog of the workspace, by the hash its last catalog read recorded, or `null`. */
export function cachedCatalog(workspaceId: string): BlockCatalog | null {
if (embedStore.getStore() || !CACHE_KEY.test(workspaceId)) return null
- const cached = readJsonFile(catalogCachePath(workspaceId), MAX_CATALOG_CACHE_BYTES)
- return isBlockCatalog(cached) ? cached : null
+ const pointer = readJsonFile(workspaceCatalogPointerPath(workspaceId), MAX_POINTER_BYTES)
+ return isRecordLike(pointer) ? cachedCatalogByHash(pointer.hash) : null
+}
+
+/**
+ * The cached catalog a workflow source directory was printed against, by the hash its
+ * `.sim/base.json` records (the workspace's, or the workflow's own for a workflow holding a type
+ * the workspace catalog lacks), or `null` when the directory has no base or that catalog is not
+ * cached. An offline check compiles against it, so it agrees with the dry run, which compiles
+ * against the same catalog.
+ */
+export async function cachedSourceCatalog(dir: string): Promise {
+ return cachedCatalogByHash(toRecord(await readSourceJson(dir, SOURCE_BASE_FILE)).catalogHash)
}
/** The catalog a `getSourceCatalog` answer carries. */
@@ -105,22 +150,71 @@ function catalogFrom(response: { data: unknown }): BlockCatalog {
return response.data
}
-/** Replaces the cached catalog atomically; a failed write only costs the next get a download. */
-function writeCachedCatalog(workspaceId: string, catalog: BlockCatalog): void {
+/**
+ * Caches `catalog` by its hash, replacing the file atomically; a failed write only costs the next
+ * get a download. An unchanged catalog is not rewritten.
+ */
+function writeCachedCatalog(catalog: BlockCatalog): void {
+ if (embedStore.getStore() || !CACHE_KEY.test(catalog.hash)) return
+ if (cachedCatalogByHash(catalog.hash)) return
+ writeJsonFile(catalogCachePath(catalog.hash), catalog)
+}
+
+/**
+ * Records `hash` as the workspace's catalog, dropping the per-workspace copy and the
+ * per-workflow copies an older CLI cached under the workspace id.
+ */
+function writeWorkspaceCatalogPointer(workspaceId: string, hash: string): void {
if (embedStore.getStore() || !CACHE_KEY.test(workspaceId)) return
- writeJsonFile(catalogCachePath(workspaceId), catalog)
+ writeJsonFile(workspaceCatalogPointerPath(workspaceId), { hash })
+ for (const legacy of [`${workspaceId}.json`, workspaceId]) {
+ if (legacy === `${hash}.json`) continue
+ try {
+ rmSync(join(catalogCacheDir(), legacy), { recursive: true, force: true })
+ } catch {}
+ }
}
/**
- * The workspace's block catalog, revalidated by ETag against the per-workspace cache: a `304`
- * reuses it, anything else replaces it. A workflow can hold a type the workspace no longer offers
- * (a legacy version, a block an allowlist now withholds); only then is that workflow's compile
- * catalog fetched instead, uncached, so its types still resolve.
+ * The catalog of the workflow `workflowId` (the workspace catalog plus the types it stores that the
+ * workspace catalog lacks), fetched and cached by hash; `knownHash`, a hash the workflow's catalog
+ * may have, is sent as its ETag, so a catalog already cached answers `304` with no body.
+ */
+export async function loadWorkflowCatalog(
+ client: OperationClient<'getSourceCatalog'>,
+ workspaceId: string,
+ workflowId: string,
+ knownHash?: string
+): Promise {
+ const known = cachedCatalogByHash(knownHash)
+ try {
+ const catalog = catalogFrom(
+ await client.request<{ data: unknown }>('getSourceCatalog', {
+ params: { workspaceId },
+ query: { workflowId },
+ ...(known ? { headers: { 'if-none-match': `"${known.hash}"` } } : {}),
+ })
+ )
+ writeCachedCatalog(catalog)
+ return catalog
+ } catch (error) {
+ if (!known || !(error instanceof SimApiError) || error.status !== 304) throw error
+ return known
+ }
+}
+
+/**
+ * The workspace's block catalog, revalidated by ETag against the cached one: a `304` reuses it,
+ * anything else replaces it. A workflow can hold a type the workspace no longer offers (a legacy
+ * version, a block an allowlist now withholds); only then is that workflow's own catalog used
+ * instead, so its types still resolve: the one cached under the hash its base records
+ * (`catalogHash`), else fetched ({@link loadWorkflowCatalog}).
*/
export async function loadCatalog(
client: OperationClient<'getSourceCatalog'>,
source: Pick & {
workflowId?: string
+ catalogHash?: string
}
): Promise {
const cached = cachedCatalog(source.workspaceId)
@@ -132,23 +226,18 @@ export async function loadCatalog(
...(cached ? { headers: { 'if-none-match': `"${cached.hash}"` } } : {}),
})
)
- writeCachedCatalog(source.workspaceId, catalog)
+ writeCachedCatalog(catalog)
} catch (error) {
if (!cached || !(error instanceof SimApiError) || error.status !== 304) throw error
catalog = cached
}
- if (
- source.workflowId === undefined ||
- source.blockTypes.every((type) => catalogBlock(catalog, type) !== undefined)
- ) {
- return catalog
- }
- return catalogFrom(
- await client.request<{ data: unknown }>('getSourceCatalog', {
- params: { workspaceId: source.workspaceId },
- query: { workflowId: source.workflowId },
- })
- )
+ writeWorkspaceCatalogPointer(source.workspaceId, catalog.hash)
+ const declaresAll = (candidate: BlockCatalog) =>
+ source.blockTypes.every((type) => catalogBlock(candidate, type) !== undefined)
+ if (source.workflowId === undefined || declaresAll(catalog)) return catalog
+ const printedWith = cachedCatalogByHash(source.catalogHash)
+ if (printedWith && declaresAll(printedWith)) return printedWith
+ return loadWorkflowCatalog(client, source.workspaceId, source.workflowId)
}
/**
@@ -334,16 +423,72 @@ function sdkGeneratorLine(): string {
return `// generated by sim ${cliVersion()} · core.d.ts ${core}`
}
-/** The generated declarations, with the generator line under the index's catalog header. */
-function sdkTypeFiles(catalog: BlockCatalog): Array {
+/** Under the index's catalog header, the generator line; `blockTypes` as `generateSdkTypes` takes it. */
+function sdkTypeFiles(
+ catalog: BlockCatalog,
+ blockTypes?: ReadonlySet
+): Array {
const generator = sdkGeneratorLine()
- return generateSdkTypes(catalog).map((file) => {
- if (file.path !== SDK_INDEX_FILE) return [`${SDK_DIR}/${file.path}`, file.text] as const
+ return generateSdkTypes(catalog, blockTypes ? { blockTypes } : {}).map((file) => {
+ if (file.path !== SDK_INDEX_FILE) return [file.path, file.text] as const
const [header, ...rest] = file.text.split('\n')
- return [`${SDK_DIR}/${file.path}`, [header, generator, ...rest].join('\n')] as const
+ return [file.path, [header, generator, ...rest].join('\n')] as const
})
}
+/** The head a current `index.d.ts` starts with: the catalog header and the generator line. */
+function sdkIndexHead(catalog: BlockCatalog): string {
+ return `${sdkIndexHeader(catalog)}\n${sdkGeneratorLine()}`
+}
+
+/**
+ * Where the declarations of every block of `catalog` are cached, once per catalog and generator
+ * (the generator line's digest), beside the catalog itself. `tsconfig.full.json` includes the
+ * `index.d.ts` there, so a directory holds only the blocks it uses. Callers gate the hash with
+ * `CACHE_KEY` first.
+ */
+function sdkCacheDir(catalog: BlockCatalog): string {
+ const generator = createHash('sha256').update(sdkGeneratorLine(), 'utf8').digest('hex')
+ return join(catalogCacheDir(), catalog.hash, `sdk-${generator.slice(0, 16)}`)
+}
+
+/**
+ * The cached full declarations' `index.d.ts` for `catalog` (an absolute path), generating them
+ * into a staging folder moved into place on the first call; later calls only check the index.
+ * `null` in an embedded run, which has no disk of its own, or when the cache cannot be written:
+ * the directory then holds every block itself, as before the cache.
+ */
+async function sdkCacheIndex(catalog: BlockCatalog): Promise {
+ if (embedStore.getStore() || !CACHE_KEY.test(catalog.hash)) return null
+ const dir = sdkCacheDir(catalog)
+ const index = join(dir, SDK_INDEX_FILE)
+ const head = sdkIndexHead(catalog)
+ const current = await readFile(index, 'utf8').catch(() => null)
+ if (current?.startsWith(`${head}\n`)) return index
+ try {
+ await mkdir(dirname(dir), { recursive: true })
+ const staging = await mkdtemp(`${dir}.staging-`)
+ try {
+ const files = sdkTypeFiles(catalog)
+ const parents = new Set(files.map(([path]) => dirname(join(staging, path))))
+ for (const parent of parents) await mkdir(parent, { recursive: true })
+ await mapWithConcurrency(files, FILE_CONCURRENCY, ([path, text]) =>
+ writeFile(join(staging, path), text)
+ )
+ await rm(dir, { recursive: true, force: true })
+ // Another command that generated the same files first already moved them into place.
+ await rename(staging, dir).catch(async (error: unknown) => {
+ if (!(await readFile(index, 'utf8').catch(() => null))?.startsWith(`${head}\n`)) throw error
+ })
+ } finally {
+ await rm(staging, { recursive: true, force: true })
+ }
+ return index
+ } catch {
+ return null
+ }
+}
+
/** What a tree write reports: the same fields after a get and a create scaffold. */
export interface SourceTree {
/** How many files were written. */
@@ -352,6 +497,8 @@ export interface SourceTree {
workspaceTypes: boolean
/** The agent guide to read before editing. */
guide: typeof SOURCE_AGENTS_FILE
+ /** The primer atop the guide (`source-primer.ts`), printed so an agent needs no other read. */
+ primer: string
}
/** Reports a `tsconfig.json` the CLI left alone and summarizes a tree write. */
@@ -359,20 +506,46 @@ export function finishTree(
dir: string,
written: number,
workspaceTypes: WorkspaceTypes | null,
- ownsTsconfig: boolean
+ tree: { ownsTsconfig: boolean; primer: string }
): SourceTree {
- if (!ownsTsconfig) {
+ if (!tree.ownsTsconfig) {
printError(
`note: ${TSCONFIG_FILE} in ${dir} is not the CLI's, so it was left as it is. To type-check the source, map "@sim/workflow-sdk" to ["./${SDK_DIR}/core.d.ts"] in its compilerOptions.paths.`
)
}
- return { files: written, workspaceTypes: workspaceTypes !== null, guide: SOURCE_AGENTS_FILE }
+ return {
+ files: written,
+ workspaceTypes: workspaceTypes !== null,
+ guide: SOURCE_AGENTS_FILE,
+ primer: tree.primer,
+ }
+}
+
+/**
+ * Prints a tree write's result: in JSON or YAML one document with the primer as its `primer`
+ * field, otherwise the fields and then the primer as text, so it reads as written.
+ */
+export function printTreeResult(
+ format: OutputFormat,
+ result: Record & { primer: string }
+): void {
+ if (format === 'json' || format === 'yaml') {
+ printProtocolResult(format, result)
+ return
+ }
+ const { primer, ...fields } = result
+ printProtocolResult(format, fields)
+ printLine(`\n${primer}`)
}
/** What a directory's editable tree holds beside the printed source. */
export interface SourceTreeProfile {
- /** `.sim/AGENTS.md`. */
+ /** Which primer opens `.sim/AGENTS.md`. */
+ kind: PrimerKind
+ /** The reference in `.sim/AGENTS.md`, below the primer. */
agents: string
+ /** More guides beside `.sim/AGENTS.md` (path → text), rewritten every time like it. */
+ guides?: ReadonlyArray
/**
* A `tsconfig.json` the CLI writes, starting with {@link TSCONFIG_MARKER}, that type-checks the
* source with the SDK declarations `sdkFiles` (directory-relative paths).
@@ -391,13 +564,15 @@ export interface SourceTreeProfile {
/**
* The SDK files each written tsconfig includes for source that uses the `used` block types:
* `tsconfig.json` the models, `workspace.d.ts` and only those blocks (plus the core builders'),
- * so `tsc` reads a few dozen files instead of every block; `tsconfig.full.json` every current
- * block for adding one the source does not use yet.
+ * which the directory holds, so `tsc` reads a few dozen files instead of every block;
+ * `tsconfig.full.json` every current block for adding one the source does not use yet, through
+ * `fullIndex` (the cached index, else the directory's own).
*/
function sdkIncludes(
catalog: BlockCatalog,
used: readonly string[],
- hasWorkspaceTypes: boolean
+ hasWorkspaceTypes: boolean,
+ fullIndex: string
): { tsconfig: string[]; full: string[] } {
const workspace = hasWorkspaceTypes ? [WORKSPACE_TYPES_FILE] : []
const inSdk = (files: readonly string[]) => files.map((file) => `${SDK_DIR}/${file}`)
@@ -407,25 +582,33 @@ function sdkIncludes(
...workspace,
...inSdk(defaultBlockFiles(catalog, used)),
],
- full: [...inSdk(fullBlockFiles(catalog, used)), ...workspace],
+ full: [
+ ...fullBlockFiles(catalog, used).map((file) =>
+ file === SDK_INDEX_FILE ? fullIndex : `${SDK_DIR}/${file}`
+ ),
+ ...workspace,
+ ],
}
}
+/** The block types source that uses `use` names: the server's, and the builders it calls. */
+function usedBlockTypes(catalog: BlockCatalog, use: SourceTreeUse): string[] {
+ return [...new Set([...(use.blockTypes ?? []), ...referencedBlockTypes(catalog, use.sources)])]
+}
+
/**
- * The CLI's tsconfig files for source that uses `use`: each written when it is absent or still
+ * The CLI's tsconfig files for the `used` block types: each written when it is absent or still
* starts with {@link TSCONFIG_MARKER}, so a person's own file is never replaced.
*/
function tsconfigWrites(
catalog: BlockCatalog,
hasWorkspaceTypes: boolean,
profile: Pick,
- use: SourceTreeUse,
+ used: readonly string[],
+ fullIndex: string,
current: { tsconfig: string | null; full: string | null }
): { write: Array; ownsTsconfig: boolean } {
- const used = [
- ...new Set([...(use.blockTypes ?? []), ...referencedBlockTypes(catalog, use.sources)]),
- ]
- const includes = sdkIncludes(catalog, used, hasWorkspaceTypes)
+ const includes = sdkIncludes(catalog, used, hasWorkspaceTypes, fullIndex)
const owned = (text: string | null) => text === null || text.startsWith(TSCONFIG_MARKER)
const write: Array = []
const ownsTsconfig = owned(current.tsconfig)
@@ -435,13 +618,36 @@ function tsconfigWrites(
}
/**
- * Rewrites the CLI's tsconfig files for the source `use` names: after an apply, which keeps the
- * declarations but may have added a block `tsconfig.json` should now include, and on a check, so
- * a block the edit added type-checks before anything is applied. `incremental` (a check) keeps
- * the blocks the files include now, since a check scans only the source text and not the
- * server's block types, and rewrites only files that exist. It reads the workspace's cached
- * catalog, so it costs no request; without one nothing changes until the next get. A file whose
- * text would not change is not rewritten.
+ * The generated declarations a directory holds for the `used` block types, under the SDK
+ * directory: the models, an index and `INDEX.md` of its blocks, and the file of each block
+ * `tsconfig.json` includes. Without a cache (`cached` false) every block, as the full config then
+ * reads the directory's own index.
+ */
+function localSdkFiles(
+ catalog: BlockCatalog,
+ used: readonly string[],
+ cached: boolean
+): Array {
+ const blockTypes = cached ? new Set(checkedBlockTypes(catalog, used)) : undefined
+ return sdkTypeFiles(catalog, blockTypes).map(
+ ([path, text]) => [`${SDK_DIR}/${path}`, text] as const
+ )
+}
+
+/** `tsconfig.full.json`'s index: the cached one, else the directory's own. */
+function fullIndexPath(cacheIndex: string | null): string {
+ return cacheIndex ?? SDK_INDEX_PATH
+}
+
+/**
+ * Brings the CLI's tsconfig files and the block declarations the directory holds up to date with
+ * the source `use` names: after an apply, which may have added a block, and on a check, so a block
+ * the edit added type-checks before anything is applied. `incremental` (a check) keeps the blocks
+ * `tsconfig.json` includes now, since a check scans only the source text and not the server's
+ * block types, and rewrites `tsconfig.json` only when it exists (`tsconfig.full.json`, which names
+ * this machine's cache, is written again when missing, as a fresh clone lacks it). It reads the
+ * cached catalog, so it costs no request; without one nothing changes until the next get. A file
+ * whose text would not change is not rewritten.
*/
export async function refreshTsconfigs(
dir: string,
@@ -450,27 +656,45 @@ export async function refreshTsconfigs(
use: SourceTreeUse,
options: { incremental?: boolean } = {}
): Promise {
- const catalog = cachedCatalog(workspaceId)
+ const catalog = (await cachedSourceCatalog(dir)) ?? cachedCatalog(workspaceId)
if (!catalog) return
- const [tsconfig, full, workspaceTypes] = await Promise.all([
+ const [tsconfig, full, workspaceTypes, index, cacheIndex] = await Promise.all([
readSourceText(dir, TSCONFIG_FILE),
readSourceText(dir, TSCONFIG_FULL_FILE),
readSourceText(dir, WORKSPACE_TYPES_FILE),
+ readSourceText(dir, SDK_INDEX_PATH),
+ sdkCacheIndex(catalog),
])
const kept = options.incremental ? includedBlockTypes(tsconfig) : []
+ const used = usedBlockTypes(catalog, {
+ ...use,
+ blockTypes: [...(use.blockTypes ?? []), ...kept],
+ })
const { write } = tsconfigWrites(
catalog,
workspaceTypes !== null,
profile,
- { ...use, blockTypes: [...(use.blockTypes ?? []), ...kept] },
+ used,
+ fullIndexPath(cacheIndex),
{ tsconfig, full }
)
- const current: Record = {
- [TSCONFIG_FILE]: tsconfig,
- [TSCONFIG_FULL_FILE]: full,
- }
- const changed = write.filter(
- ([path, text]) => current[path] !== text && (!options.incremental || current[path] !== null)
+ const current = new Map([
+ [TSCONFIG_FILE, tsconfig],
+ [TSCONFIG_FULL_FILE, full],
+ ])
+ // Declarations of another catalog or generator are left for the next get, which replaces them.
+ const declarations =
+ cacheIndex !== null && index?.startsWith(`${sdkIndexHead(catalog)}\n`)
+ ? localSdkFiles(catalog, used, true).filter(([path]) => path.startsWith(`${SDK_DIR}/blocks/`))
+ : []
+ const texts = await mapWithConcurrency(declarations, FILE_CONCURRENCY, ([path]) =>
+ readSourceText(dir, path)
+ )
+ declarations.forEach(([path], position) => current.set(path, texts[position]))
+ const changed = [...write, ...declarations].filter(
+ ([path, text]) =>
+ current.get(path) !== text &&
+ (!options.incremental || path !== TSCONFIG_FILE || current.get(path) !== null)
)
if (changed.length > 0) await updateSourceDir(dir, { write: changed })
}
@@ -502,12 +726,15 @@ export interface SourceTreeUse {
}
/**
- * Everything beside the printed source that makes a directory editable: the agent guide and the
- * example workflows (both rewritten every time and never in `.sim/written.json`), the
- * declarations (regenerated only when the catalog or generator changed), `workspace.d.ts`, the
- * CLI's `tsconfig.json` and `tsconfig.full.json`, and the LF lines. `incremental` (a write that
- * leaves workflows on disk unread, such as a get of only some kinds) keeps the blocks
- * `tsconfig.json` includes now, as {@link refreshTsconfigs} does, adding the ones `use` needs.
+ * Everything beside the printed source that makes a directory editable: the agent guide (the
+ * primer for the builders `use` calls, then the profile's reference) and the example workflows
+ * (both rewritten every time and never in `.sim/written.json`), the declarations of the blocks
+ * the source uses (every block's are cached once per catalog, for `tsconfig.full.json`),
+ * `workspace.d.ts`, the CLI's `tsconfig.json` and `tsconfig.full.json`, and the LF lines.
+ * Declarations of another catalog or generator are removed whole; otherwise only the block files,
+ * so a block the source no longer uses loses its file. `incremental` (a write that leaves
+ * workflows on disk unread, such as a get of only some kinds) keeps the blocks `tsconfig.json`
+ * includes now, as {@link refreshTsconfigs} does, adding the ones `use` needs.
*/
export async function sourceTreeChange(
dir: string,
@@ -516,9 +743,12 @@ export async function sourceTreeChange(
profile: SourceTreeProfile,
use: SourceTreeUse,
options: { incremental?: boolean } = {}
-): Promise<{ change: SourceDirChange; ownsTsconfig: boolean }> {
+): Promise<{ change: SourceDirChange; ownsTsconfig: boolean; primer: string }> {
+ const sources = [...use.sources]
+ const primer = primerFor(profile.kind, catalog, sources)
const write: Array = [
- [SOURCE_AGENTS_FILE, profile.agents],
+ [SOURCE_AGENTS_FILE, `${primer}\n${profile.agents}`],
+ ...(profile.guides ?? []),
...Object.entries(COOKBOOK_EXAMPLES).map(
([file, text]) => [`${SOURCE_EXAMPLES_DIR}/${file}`, text] as const
),
@@ -527,19 +757,25 @@ export async function sourceTreeChange(
...(profile.sdk ?? []),
]
const remove: string[] = []
- const [gitattributes, gitignore, index, tsconfig, tsconfigFull] = await Promise.all([
+ const [gitattributes, gitignore, index, tsconfig, tsconfigFull, cacheIndex] = await Promise.all([
readSourceText(dir, SOURCE_GITATTRIBUTES_FILE),
profile.gitignore ? readSourceText(dir, SOURCE_GITIGNORE_FILE) : null,
readSourceText(dir, SDK_INDEX_PATH),
readSourceText(dir, TSCONFIG_FILE),
readSourceText(dir, TSCONFIG_FULL_FILE),
+ sdkCacheIndex(catalog),
])
const kept = options.incremental ? includedBlockTypes(tsconfig) : []
+ const used = usedBlockTypes(catalog, {
+ sources,
+ blockTypes: [...(use.blockTypes ?? []), ...kept],
+ })
const tsconfigs = tsconfigWrites(
catalog,
workspaceTypes !== null,
profile,
- { ...use, blockTypes: [...(use.blockTypes ?? []), ...kept] },
+ used,
+ fullIndexPath(cacheIndex),
{ tsconfig, full: tsconfigFull }
)
write.push(...tsconfigs.write)
@@ -548,10 +784,13 @@ export async function sourceTreeChange(
if (mergedAttributes !== null) write.push([SOURCE_GITATTRIBUTES_FILE, mergedAttributes])
const mergedIgnore = profile.gitignore ? withMissingLines(gitignore, profile.gitignore) : null
if (mergedIgnore !== null) write.push([SOURCE_GITIGNORE_FILE, mergedIgnore])
- const indexHead = index?.split('\n', 2).join('\n')
- if (indexHead !== `${sdkIndexHeader(catalog)}\n${sdkGeneratorLine()}`) {
- remove.push(SDK_DIR)
- write.push(...sdkTypeFiles(catalog))
+ const current = index?.startsWith(`${sdkIndexHead(catalog)}\n`) === true
+ if (!current) remove.push(SDK_DIR)
+ if (cacheIndex !== null) {
+ if (current) remove.push(`${SDK_DIR}/blocks`)
+ write.push(...localSdkFiles(catalog, used, true))
+ } else if (!current) {
+ write.push(...localSdkFiles(catalog, used, false))
}
if (workspaceTypes === null) remove.push(WORKSPACE_TYPES_FILE)
else write.push([WORKSPACE_TYPES_FILE, workspaceTypes.text])
@@ -560,7 +799,7 @@ export async function sourceTreeChange(
} else {
remove.push(WORKSPACE_MANIFEST_FILE)
}
- return { change: { write, remove }, ownsTsconfig }
+ return { change: { write, remove }, ownsTsconfig, primer }
}
/** A read the server refused as over the source size limits exits 7, as a refused apply does. */
diff --git a/packages/sim-cli/src/commands/protocol/source-ui.ts b/packages/sim-cli/src/commands/protocol/source-ui.ts
index a9ec4068f05..c39e7eb14f2 100644
--- a/packages/sim-cli/src/commands/protocol/source-ui.ts
+++ b/packages/sim-cli/src/commands/protocol/source-ui.ts
@@ -3,16 +3,18 @@
* findings (each with its location, frame and hint), the confirmation prompt, the refusal of a
* write nobody confirmed, and an API refusal re-thrown with the exit status a script branches on.
*/
-import { createInterface } from 'node:readline/promises'
import { isRecordLike, toRecord } from '@sim/utils/object'
import { SOURCE_LOCK_FILE } from '@sim/workflow-code/limits'
import { type FlatDiagnostic, type RefusalClass, refusalClass } from '@sim/workspace-code'
+import type { Command } from 'commander'
import { printError } from '#sim-cli/output/io'
+import { askLine } from '#sim-cli/terminal/line-input'
import type { OutputFormat } from '../../config/index'
-import { embedStore } from '../../embed-context'
+import { embedStore, setSoftExitCode } from '../../embed-context'
import {
SOURCE_BUSY_EXIT_CODE,
SOURCE_LOCKED_EXIT_CODE,
+ SOURCE_PLANNED_EXIT_CODE,
SOURCE_STALE_EXIT_CODE,
SOURCE_TOO_LARGE_EXIT_CODE,
} from '../../exit-codes'
@@ -26,7 +28,7 @@ import { noAnswer, REQUEST_CANCELLED, SimApiError } from '../../http/client'
type WorkflowDiagnostic = ApplyWorkflowSourceResponse['data']['diagnostics'][number]
/** A workflow finding, or a workspace one: the same fields, over both code vocabularies. */
-type Diagnostic = Omit & {
+export type Diagnostic = Omit & {
code:
| WorkflowDiagnostic['code']
| ApplyWorkspaceSourceResponse['data']['diagnostics'][number]['code']
@@ -74,28 +76,58 @@ function formatDiagnostic(diagnostic: Diagnostic): string {
return lines.join('\n')
}
+/** How many other locations a collapsed line lists before counting the rest. */
+const COLLAPSED_LOCATIONS = 8
+
+/** Where a finding sits, as its printed line starts with: `file:line:column`, or empty. */
+function locationOf(diagnostic: Diagnostic): string {
+ return formatDiagnostic({ ...diagnostic, frame: undefined, hint: undefined })
+ .split(` ${diagnostic.severity} `)[0]
+ .trim()
+}
+
/**
- * Prints a write's findings: every error and warning with its location, frame and hint, and each
- * info code folded into one line (a new workflow places every block by autolayout, which is no
- * reason to print a frame per block and bury the warnings).
+ * Prints a write's findings on stderr for a person: every error with its location, frame and
+ * hint; the first warning of each code the same way, and the rest of that code on one line
+ * listing where they are, each with its own message when it differs from the first (one fix
+ * usually serves them all, and twenty frames of the same warning buried the errors); each info
+ * code folded into one line.
*/
-function printDiagnostics(diagnostics: Diagnostic[]): void {
- const infos = new Map()
+function printDiagnostics(diagnostics: readonly Diagnostic[]): void {
+ const groups = new Map()
for (const diagnostic of diagnostics) {
- if (diagnostic.severity !== 'info') {
+ if (diagnostic.severity === 'error') {
printError(formatDiagnostic(diagnostic))
continue
}
- const group = infos.get(diagnostic.code)
+ const key = `${diagnostic.severity}:${diagnostic.code}`
+ const group = groups.get(key)
if (group) group.push(diagnostic)
- else infos.set(diagnostic.code, [diagnostic])
+ else groups.set(key, [diagnostic])
}
- for (const [code, group] of infos) {
- const [first] = group
+ for (const [first, ...rest] of groups.values()) {
+ if (first.severity === 'info') {
+ printError(
+ rest.length === 0
+ ? formatDiagnostic({ ...first, frame: undefined })
+ : `info ${first.code} on ${rest.length + 1} blocks: ${first.message} (and ${rest.length} more like it)`
+ )
+ continue
+ }
+ printError(formatDiagnostic(first))
+ if (rest.length === 0) continue
+ // A collapsed finding saying something else than the first keeps its words beside its place.
+ const listed = rest
+ .slice(0, COLLAPSED_LOCATIONS)
+ .map((diagnostic) => {
+ const location = locationOf(diagnostic)
+ if (diagnostic.message === first.message) return location
+ return location ? `${location} (${diagnostic.message})` : diagnostic.message
+ })
+ .filter(Boolean)
+ const unlisted = rest.length - listed.length
printError(
- group.length === 1
- ? formatDiagnostic({ ...first, frame: undefined })
- : `info ${code} on ${group.length} blocks: ${first.message} (and ${group.length - 1} more like it)`
+ ` ${rest.length} more ${first.code} like it${listed.length > 0 ? `: ${listed.join(', ')}` : ''}${unlisted > 0 && listed.length > 0 ? `, and ${unlisted} more` : ''}`
)
}
}
@@ -103,16 +135,31 @@ function printDiagnostics(diagnostics: Diagnostic[]): void {
/** A finding in a source command's `--output json` (or `yaml`) result: the frame stays human-only. */
type ResultDiagnostic = Omit & { code: Diagnostic['code'] }
+/** Whether `format` prints one machine document on stdout rather than lines for a person. */
+function isMachine(format: OutputFormat): boolean {
+ return format === 'json' || format === 'yaml'
+}
+
/**
* The `diagnostics` a source command's result carries in a machine format, so the one document on
- * stdout holds every finding; a display format prints them on stderr only.
+ * stdout holds every finding: errors and warnings, and info only with `verbose` (a new workflow
+ * places every block by autolayout, which is no finding to act on), except `I_FINDINGS_TRUNCATED`,
+ * which says the list is incomplete. A display format prints them on stderr only.
*/
-export function resultDiagnostics(
+function resultDiagnostics(
format: OutputFormat,
- diagnostics: readonly Diagnostic[]
+ diagnostics: readonly Diagnostic[],
+ verbose = false
): { diagnostics?: ResultDiagnostic[] } {
- if (format !== 'json' && format !== 'yaml') return {}
- return { diagnostics: diagnostics.map(({ frame: _frame, ...finding }) => finding) }
+ if (!isMachine(format)) return {}
+ return {
+ diagnostics: diagnostics
+ .filter(
+ (finding) =>
+ verbose || finding.severity !== 'info' || finding.code === 'I_FINDINGS_TRUNCATED'
+ )
+ .map(({ frame: _frame, ...finding }) => finding),
+ }
}
function findingKey(diagnostic: Diagnostic): string {
@@ -128,11 +175,64 @@ export function fresh(printed: Set, key: string): boolean {
}
/**
- * Prints the findings `printed` does not hold yet and records them, so a finding the preview
- * showed is not shown again by the write that follows it.
+ * Where a source command's findings go, chosen once by its output format, so each is shown once:
+ * a display format prints them on stderr as frames (and never in the result), a machine format
+ * only in the one document on stdout (and never as frames, which an agent reading both streams
+ * read twice).
*/
-export function printNewDiagnostics(diagnostics: Diagnostic[], printed: Set): void {
- printDiagnostics(diagnostics.filter((diagnostic) => fresh(printed, findingKey(diagnostic))))
+export interface FindingsOutput {
+ /** Whether the result is one machine document on stdout. */
+ machine: boolean
+ /**
+ * Prints the findings not printed yet on stderr in a display format, so a finding the preview
+ * showed is not shown again by the write that follows it; nothing in a machine format.
+ */
+ print(diagnostics: readonly Diagnostic[]): void
+ /** The result's `diagnostics` field: see {@link resultDiagnostics}. */
+ result(diagnostics: readonly Diagnostic[]): { diagnostics?: ResultDiagnostic[] }
+}
+
+/** The {@link FindingsOutput} of `format`; `verbose` keeps info findings in a machine result. */
+export function findingsOutput(format: OutputFormat, verbose = false): FindingsOutput {
+ const printed = new Set()
+ const machine = isMachine(format)
+ return {
+ machine,
+ print(diagnostics) {
+ const unseen = diagnostics.filter((diagnostic) => fresh(printed, findingKey(diagnostic)))
+ if (!machine) printDiagnostics(unseen)
+ },
+ result: (diagnostics) => resultDiagnostics(format, diagnostics, verbose),
+ }
+}
+
+/** A resource a source write created, as its result lists it; `id` once it is written. */
+export interface CreatedResource {
+ kind: string
+ key?: string
+ id?: string
+}
+
+/** How many created resources the display line names before counting the rest. */
+const NAMED_CREATED = 5
+
+/** The created resources as one line for a person: `workflow (abc), block start (b-1), …`. */
+export function createdText(created: readonly CreatedResource[]): string {
+ if (created.length === 0) return 'none'
+ const named = created.slice(0, NAMED_CREATED).map(({ kind, key, id }) => {
+ const label = [kind, key].filter((part) => part !== undefined).join(' ')
+ return id === undefined ? label : `${label} (${id})`
+ })
+ const more = created.length - named.length
+ return `${named.join(', ')}${more > 0 ? `, and ${more} more` : ''}`
+}
+
+/** Adds `--verbose` to a source command whose result can carry info findings. */
+export function verboseFlag(command: Command): Command {
+ return command.option(
+ '--verbose',
+ 'With --output json, also list info findings (such as blocks placed by autolayout)'
+ )
}
function isDiagnosticList(value: unknown): value is Diagnostic[] {
@@ -146,13 +246,8 @@ function canPrompt(): boolean {
/** Asks `question` on the terminal; anything but `y` or `yes` is no. */
export async function confirm(question: string): Promise {
- const prompt = createInterface({ input: process.stdin, output: process.stderr })
- try {
- const answer = (await prompt.question(`${question} (y/N) `)).trim().toLowerCase()
- return answer === 'y' || answer === 'yes'
- } finally {
- prompt.close()
- }
+ const answer = (await askLine(`${question} (y/N) `))?.trim().toLowerCase()
+ return answer === 'y' || answer === 'yes'
}
/** The exit status of each `409` the server tells apart by `details.code`; any other exits 1. */
@@ -185,19 +280,35 @@ function answerLost(error: SimApiError): boolean {
return noAnswer(error, LOST_ANSWER_STATUSES) || error.code === REQUEST_CANCELLED
}
+/**
+ * The server's pointer to the findings in the error document (`See details.diagnostics.`): a
+ * display format prints them instead, so the sentence names a JSON field the reader never sees.
+ */
+const DETAILS_REFERENCE = / See details\.diagnostics\.$/
+
/**
* Re-throws an API refusal after printing its diagnostics, so each shows its source line, with
* the exit status a script branches on. In a machine `format` the error document keeps the
* refusal's details with each finding as a success result carries it ({@link resultDiagnostics});
- * a display format prints the findings once, above the error. `ifAnswerLost` is the next step for
- * a write whose answer never came back, appended to its message.
+ * a display format prints the findings once, above the error, and `verbose` keeps info findings in
+ * the machine document as it does in a result. `ifAnswerLost` is the next step for a write whose
+ * answer never came back, appended to its message.
*/
-export function printRefusal(error: unknown, format: OutputFormat, ifAnswerLost?: string): never {
+export function printRefusal(
+ error: unknown,
+ format: OutputFormat,
+ options: { ifAnswerLost?: string; verbose?: boolean } = {}
+): never {
+ const { ifAnswerLost, verbose = false } = options
if (!(error instanceof SimApiError)) throw error
const details = toRecord(error.details)
const diagnostics = details.diagnostics
const printed = isDiagnosticList(diagnostics)
- if (printed) printDiagnostics(diagnostics)
+ const machine = isMachine(format)
+ if (printed && !machine) printDiagnostics(diagnostics)
+ const message = machine
+ ? error.message
+ : error.message.replace(DETAILS_REFERENCE, printed ? ' Each finding is printed above.' : '')
const nameTaken = error.status === 409 && details.code === 'SOURCE_NAME_TAKEN'
const hint = nameTaken
? 'Pass --name to create it under another name, or apply to that workflow.'
@@ -205,10 +316,10 @@ export function printRefusal(error: unknown, format: OutputFormat, ifAnswerLost?
? ifAnswerLost
: null
throw new SimApiError(
- hint ? `${error.message} ${hint}` : error.message,
+ hint ? `${message} ${hint}` : message,
error.status,
error.code,
- printed ? refusalDetails(format, details, diagnostics) : error.details,
+ printed ? refusalDetails(format, details, diagnostics, verbose) : error.details,
refusalExitCode(error, details.code),
error.requestId
)
@@ -218,26 +329,42 @@ export function printRefusal(error: unknown, format: OutputFormat, ifAnswerLost?
function refusalDetails(
format: OutputFormat,
details: Record,
- diagnostics: Diagnostic[]
+ diagnostics: Diagnostic[],
+ verbose: boolean
): Record | undefined {
- const flattened = resultDiagnostics(format, diagnostics)
+ const flattened = resultDiagnostics(format, diagnostics, verbose)
return flattened.diagnostics ? { ...details, ...flattened } : undefined
}
/**
- * A write without `--yes` or `--dry-run` where no terminal can ask: the generic refusal, naming
- * what the write does (`writes the workflow draft`) and its verb (`apply`).
+ * How a source write proceeds: `dry-run` checks only; `write` writes at once (`--yes`); `ask`
+ * previews and asks on a terminal; `plan` (no `--yes` and no terminal to ask) previews, prints the
+ * plan and stops with {@link SOURCE_PLANNED_EXIT_CODE}, so an agent's edit loop sees the plan
+ * without a separate dry run and applies it with `--yes`.
*/
-export function requireConsent(
- options: { dryRun?: boolean; yes?: boolean },
- action: string,
- verb: string
-): void {
- if (!options.dryRun && !options.yes && !canPrompt()) {
- throw new SimApiError(
- `This ${action}. Re-run with --yes to ${verb} it, or with --dry-run to check it without writing.`,
- 0,
- 'CONFIRMATION_REQUIRED'
- )
+export type WriteMode = 'dry-run' | 'write' | 'ask' | 'plan'
+
+/** The {@link WriteMode} of a write's `--dry-run` and `--yes` flags, here. */
+export function writeMode(options: { dryRun?: boolean; yes?: boolean }): WriteMode {
+ if (options.dryRun) return 'dry-run'
+ if (options.yes) return 'write'
+ return canPrompt() ? 'ask' : 'plan'
+}
+
+/** `value` as one shell word: as it is when it needs no quoting, else single-quoted. */
+export function shellWord(value: string): string {
+ return /^[\w@%+=:,./-]+$/.test(value) ? value : `'${value.replaceAll("'", "'\\''")}'`
+}
+
+/** The next step a `plan` run names: the same write with `--yes`. */
+export function planNext(command: string): { command: string; reason: string } {
+ return {
+ command: `${command} --yes`,
+ reason: 'Nothing was written: this is the plan. Run it with --yes to apply it.',
}
}
+
+/** Marks a `plan` run that found no errors: nothing was written, applying needs `--yes`. */
+export function stopAtPlan(): void {
+ setSoftExitCode(SOURCE_PLANNED_EXIT_CODE)
+}
diff --git a/packages/sim-cli/src/commands/protocol/stored-edges.ts b/packages/sim-cli/src/commands/protocol/stored-edges.ts
new file mode 100644
index 00000000000..04306d11abb
--- /dev/null
+++ b/packages/sim-cli/src/commands/protocol/stored-edges.ts
@@ -0,0 +1,180 @@
+import { isRecordLike, toRecord } from '@sim/utils/object'
+import {
+ type BlockCatalog,
+ compileWorkflow,
+ parseWorkflowLock,
+ type StoredEdgeRef,
+ setOwn,
+} from '@sim/workflow-code'
+import { SOURCE_BASE_FILE, SOURCE_ENTRY_FILE, SOURCE_LOCK_FILE } from '@sim/workflow-code/limits'
+import { KIND_DIRS } from '@sim/workspace-code'
+import type { OperationClient } from '../../runtime/called-operations'
+import { readSourceJson, SOURCE_STORED_EDGES_FILE } from './source-dir'
+import { cachedCatalogByHash, loadWorkflowCatalog } from './source-tree'
+
+/** A printed source as a get, apply or create answers it. */
+interface PrintedAnswer {
+ files: Record
+ lock: unknown
+ base: unknown
+}
+
+/**
+ * The `.sim/stored-edges.json` write for a printed source: its edges, compiled against `catalog`
+ * (the one it was printed with), tagged with its base fingerprint. `null` when the answer has no
+ * fingerprint or entry, or nothing to compile it with; the record left on disk then names an
+ * older fingerprint, which a check ignores.
+ */
+export function storedEdgesWrite(
+ answer: PrintedAnswer,
+ catalog: BlockCatalog | null,
+ folder = ''
+): readonly [string, string] | null {
+ const { fingerprint } = toRecord(answer.base)
+ const entry = answer.files[SOURCE_ENTRY_FILE]
+ const { lock } = parseWorkflowLock(answer.lock)
+ if (!catalog || typeof fingerprint !== 'string' || entry === undefined || !lock) return null
+ const { edges } = compileWorkflow({
+ entry: { path: SOURCE_ENTRY_FILE, text: entry },
+ files: new Map(Object.entries(answer.files).filter(([path]) => path !== SOURCE_ENTRY_FILE)),
+ catalog,
+ sdkMajor: catalog.sdkMajor,
+ knownKeys: new Set(Object.keys(lock.blocks)),
+ storedUnread: lock,
+ })
+ return [
+ `${folder}${SOURCE_STORED_EDGES_FILE}`,
+ `${JSON.stringify({ version: 1, fingerprint, edges }, null, 2)}\n`,
+ ]
+}
+
+/** The workflow folders a workspace answer's files hold, by key, each as a printed source. */
+function printedWorkflowFolders(files: Record): Map {
+ const folders = new Map()
+ const prefix = `${KIND_DIRS.workflow}/`
+ for (const [path, text] of Object.entries(files)) {
+ if (!path.startsWith(prefix)) continue
+ const slash = path.indexOf('/', prefix.length)
+ if (slash < 0) continue
+ const key = path.slice(prefix.length, slash)
+ const relative = path.slice(slash + 1)
+ const folder = folders.get(key) ?? { files: {}, lock: undefined, base: undefined }
+ if (relative === SOURCE_LOCK_FILE) folder.lock = parseJson(text)
+ else if (relative === SOURCE_BASE_FILE) folder.base = parseJson(text)
+ else setOwn(folder.files, relative, text)
+ folders.set(key, folder)
+ }
+ return folders
+}
+
+function parseJson(text: string): unknown {
+ try {
+ return JSON.parse(text)
+ } catch {
+ return undefined
+ }
+}
+
+/** The stored-edges records of a workspace answer, and the folders left without one. */
+export interface WorkflowFolderRecords {
+ records: Array
+ /** Folder keys whose catalog could not be fetched: their record is skipped, not failed. */
+ unrecorded: string[]
+}
+
+/**
+ * The `.sim/stored-edges.json` write of each workflow folder a workspace get or apply answered
+ * with, each compiled against the catalog its base names. A workflow holding a type the workspace
+ * catalog lacks was printed with its own catalog: one not cached yet is fetched, once per distinct
+ * hash, and cached, so an offline check compiles that folder against it too. The records are a
+ * cache fill, never part of the write: a catalog fetch that fails skips the folders printed with
+ * that catalog (named in `unrecorded`) rather than failing a write the server already committed.
+ */
+export async function workflowFolderRecords(
+ client: OperationClient<'getSourceCatalog'>,
+ workspaceId: string,
+ workspaceCatalog: BlockCatalog,
+ files: Record
+): Promise {
+ const folders = printedWorkflowFolders(files)
+ const uncached = new Map()
+ for (const { base } of folders.values()) {
+ const { workflowId, catalogHash } = toRecord(base)
+ if (typeof workflowId !== 'string' || typeof catalogHash !== 'string') continue
+ if (catalogHash === workspaceCatalog.hash || cachedCatalogByHash(catalogHash)) continue
+ if (!uncached.has(catalogHash)) uncached.set(catalogHash, workflowId)
+ }
+ const fetched = new Map()
+ for (const [catalogHash, workflowId] of uncached) {
+ const catalog = await loadWorkflowCatalog(client, workspaceId, workflowId).catch(() => null)
+ if (catalog) fetched.set(catalogHash, catalog)
+ }
+ const records: Array = []
+ const unrecorded: string[] = []
+ for (const [key, folder] of folders) {
+ const { catalogHash } = toRecord(folder.base)
+ const catalog =
+ catalogHash === workspaceCatalog.hash
+ ? workspaceCatalog
+ : typeof catalogHash === 'string'
+ ? (fetched.get(catalogHash) ?? cachedCatalogByHash(catalogHash))
+ : null
+ if (typeof catalogHash === 'string' && uncached.has(catalogHash) && !catalog) {
+ unrecorded.push(key)
+ continue
+ }
+ const write = storedEdgesWrite(folder, catalog, `${KIND_DIRS.workflow}/${key}/`)
+ if (write) records.push(write)
+ }
+ return { records, unrecorded }
+}
+
+/**
+ * The warning for workflow folders {@link workflowFolderRecords} could not record, or `null`.
+ * Their edges are unrecorded, so an offline check of each one compiles without its stored edges
+ * until the next get records them.
+ */
+export function unrecordedFoldersWarning(unrecorded: readonly string[]): string | null {
+ if (unrecorded.length === 0) return null
+ return `Warning: could not fetch the block catalog of ${unrecorded.map((key) => `${KIND_DIRS.workflow}/${key}`).join(', ')}, so its stored edges are not recorded; everything else was saved. The next \`sim workspaces source get\` records them.`
+}
+
+function isStoredEdge(value: unknown): value is StoredEdgeRef {
+ return (
+ isRecordLike(value) &&
+ typeof value.from === 'string' &&
+ typeof value.port === 'string' &&
+ typeof value.to === 'string'
+ )
+}
+
+/**
+ * The stored edges recorded for the base `dir` holds now, or `undefined` when none are: no
+ * record, one an older CLI or another base wrote, or a malformed one.
+ */
+export async function readStoredEdges(dir: string): Promise {
+ const [record, base] = await Promise.all([
+ readSourceJson(dir, SOURCE_STORED_EDGES_FILE),
+ readSourceJson(dir, SOURCE_BASE_FILE),
+ ])
+ const { version, fingerprint, edges } = toRecord(record)
+ if (
+ version !== 1 ||
+ typeof fingerprint !== 'string' ||
+ fingerprint !== toRecord(base).fingerprint ||
+ !Array.isArray(edges) ||
+ !edges.every(isStoredEdge)
+ ) {
+ return undefined
+ }
+ return edges
+}
+
+/** The stored-edges records of the workflow folders whose base a write removes (`paths`). */
+export function removedWorkflowRecords(paths: Iterable): string[] {
+ return [...paths].flatMap((path) =>
+ path.startsWith(`${KIND_DIRS.workflow}/`) && path.endsWith(`/${SOURCE_BASE_FILE}`)
+ ? [`${path.slice(0, -SOURCE_BASE_FILE.length)}${SOURCE_STORED_EDGES_FILE}`]
+ : []
+ )
+}
diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-apply.test.ts b/packages/sim-cli/src/commands/protocol/workflow-source-apply.test.ts
index ec73df948bb..5a34c301403 100644
--- a/packages/sim-cli/src/commands/protocol/workflow-source-apply.test.ts
+++ b/packages/sim-cli/src/commands/protocol/workflow-source-apply.test.ts
@@ -1,23 +1,39 @@
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { hostname, tmpdir } from 'node:os'
import { join } from 'node:path'
+import { toRecord } from '@sim/utils/object'
import { compileWorkflow } from '@sim/workflow-code'
import { fingerprintOf } from '@sim/workspace-code'
import { Command } from 'commander'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { SimApiError } from '../../http/client'
import { buildGeneratedCommands } from '../../runtime/build'
+import { cacheCatalogText } from '../../test/catalog-cache'
import { contextMockFns, contextMockState } from '../../test/context-mock'
import { digest } from './source-dir'
import { TSCONFIG_MARKER } from './source-tree'
import { attachWorkflowSourceApply, attachWorkflowSourceCreate } from './workflow-source-apply'
-const prompt = vi.hoisted(() => ({ question: vi.fn() }))
+const prompt = vi.hoisted(() => ({
+ question: vi.fn(),
+ /** The open readline interface, to end its input as Ctrl-D does. */
+ open: null as import('node:events').EventEmitter | null,
+}))
vi.mock('../../context', async () => (await import('../../test/context-mock')).contextMock)
-vi.mock('node:readline/promises', () => ({
- createInterface: () => ({ question: prompt.question, close: () => {} }),
-}))
+vi.mock('node:readline/promises', async () => {
+ const { EventEmitter } = await import('node:events')
+ return {
+ createInterface: () => {
+ const readline = Object.assign(new EventEmitter(), {
+ question: prompt.question,
+ close: () => readline.emit('close'),
+ })
+ prompt.open = readline
+ return readline
+ },
+ }
+})
const { mockRequest: request } = contextMockFns
@@ -47,13 +63,17 @@ function outcome(overrides: { dryRun?: boolean; diagnostics?: unknown[]; notes?:
diff: {
noop: false,
blocks: {
- added: [{ id: 'b-1', key: 'start' }],
- removed: [],
+ added: [{ id: 'b-1', key: 'start', type: 'start_trigger' }] as Array<{
+ id: string
+ key: string
+ type: string
+ }>,
+ removed: [] as Array<{ id: string; key: string; type: string }>,
renamed: [],
modified: [] as Array<{ id: string; key: string; fields: string[] }>,
},
edges: { added: 1, removed: 0 },
- variables: { added: [], removed: [], modified: [] },
+ variables: { added: [] as string[], removed: [] as string[], modified: [] as string[] },
},
diagnostics: overrides.diagnostics ?? [],
files: { 'workflow.ts': `${ENTRY}// stored\n` },
@@ -143,28 +163,62 @@ function dryRunFlags() {
}
describe('sim workflows source apply', () => {
- it('refuses without --yes when nobody is at a terminal, sending nothing', async () => {
+ it('without --yes and nobody at a terminal, prints the plan, writes nothing and exits 14', async () => {
setTerminal(false)
+ request.mockResolvedValue(outcome({ dryRun: true }))
- const error = await apply(WORKFLOW_ID, '--dir', dir).catch((caught: unknown) => caught)
+ await apply(WORKFLOW_ID, '--dir', dir, '--allow-delete')
- expect(error).toBeInstanceOf(SimApiError)
- expect((error as SimApiError).code).toBe('CONFIRMATION_REQUIRED')
- expect((error as SimApiError).message).toContain('--yes')
- expect(request).not.toHaveBeenCalled()
+ expect(dryRunFlags()).toEqual([true])
+ expect(process.exitCode).toBe(14)
+ expect(readFileSync(join(dir, 'workflow.ts'), 'utf8')).toBe(ENTRY)
+ expect(stdoutJson()).toMatchObject({
+ applied: false,
+ dryRun: true,
+ next: { command: `sim workflows source apply --dir ${dir} --allow-delete --yes` },
+ })
+ expect(stderrLines().join('\n')).toContain('Blocks: 1 added')
})
- it('refuses a create without --yes and sends nothing', async () => {
+ it('without --yes, exits with the refusal and names no next step when the plan has errors', async () => {
setTerminal(false)
+ request.mockResolvedValue(
+ outcome({
+ dryRun: true,
+ diagnostics: [{ code: 'E_UNKNOWN_PARAM', severity: 'error', message: 'No param x.' }],
+ })
+ )
- const error = await program()
- .parseAsync(['node', 'sim', 'workflows', 'source', 'create', '--dir', dir])
- .catch((caught: unknown) => caught)
+ await apply(WORKFLOW_ID, '--dir', dir)
- expect(error).toBeInstanceOf(SimApiError)
- expect((error as SimApiError).code).toBe('CONFIRMATION_REQUIRED')
- expect((error as SimApiError).message).toContain('--yes to create')
- expect(request).not.toHaveBeenCalled()
+ expect(process.exitCode).toBe(1)
+ expect(stdoutJson()).not.toHaveProperty('next')
+ })
+
+ it('plans a create without --yes and creates nothing', async () => {
+ setTerminal(false)
+ rmSync(join(dir, '.sim/base.json'))
+ request.mockResolvedValue(outcome({ dryRun: true }))
+
+ await program().parseAsync(['node', 'sim', 'workflows', 'source', 'create', '--dir', dir])
+
+ expect(dryRunFlags()).toEqual([true])
+ expect(process.exitCode).toBe(14)
+ expect(stdoutJson()).toMatchObject({
+ next: {
+ command: `sim workflows source create --dir ${dir} --workspace ${WORKSPACE_ID} --yes`,
+ },
+ })
+ })
+
+ it('with --yes, prints the summary of what it applied', async () => {
+ setTerminal(false)
+ request.mockResolvedValue(outcome())
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--yes')
+
+ expect(dryRunFlags()).toEqual([false])
+ expect(stderrLines().join('\n')).toContain('Blocks: 1 added, 0 removed')
})
it('applies at once with --yes and saves what the server stored', async () => {
@@ -203,11 +257,7 @@ describe('sim workflows source apply', () => {
it('points tsconfig.json at a block the applied source added, from the cached catalog', async () => {
setTerminal(false)
vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config'))
- mkdirSync(join(dir, 'config/cache/source-catalog'), { recursive: true })
- writeFileSync(
- join(dir, `config/cache/source-catalog/${WORKSPACE_ID}.json`),
- readFileSync(FIXTURE_CATALOG, 'utf8')
- )
+ cacheCatalogText(join(dir, 'config'), WORKSPACE_ID, readFileSync(FIXTURE_CATALOG, 'utf8'))
writeFileSync(join(dir, 'tsconfig.json'), `${TSCONFIG_MARKER}\n{}\n`)
const stored = outcome()
stored.data.files = { 'workflow.ts': `${ENTRY}// slack.sendMessage('Post', {})\n` }
@@ -257,6 +307,71 @@ describe('sim workflows source apply', () => {
expect(readFileSync(join(dir, 'workflow.ts'), 'utf8')).toBe(ENTRY)
})
+ it.each([
+ [
+ 'Node closes the prompt',
+ () => {
+ queueMicrotask(() => prompt.open?.emit('close'))
+ return new Promise(() => {})
+ },
+ ],
+ [
+ 'Bun aborts the question',
+ async () => {
+ throw Object.assign(new Error('The operation was aborted.'), { name: 'AbortError' })
+ },
+ ],
+ ])(
+ 'at a terminal, applies nothing when input ends at the prompt (%s)',
+ async (_runtime, answer) => {
+ setTerminal(true)
+ request.mockResolvedValue(outcome({ dryRun: true }))
+ prompt.question.mockImplementation(answer)
+
+ await expect(apply(WORKFLOW_ID, '--dir', dir)).rejects.toMatchObject({
+ code: 'CANCELLED',
+ exitCode: 1,
+ message: 'Nothing was applied.',
+ })
+
+ expect(dryRunFlags()).toEqual([true])
+ expect(readFileSync(join(dir, 'workflow.ts'), 'utf8')).toBe(ENTRY)
+ }
+ )
+
+ it.each([
+ [
+ 'table',
+ 'The workflow source has 1 error(s); nothing was written. Each finding is printed above.',
+ ],
+ ['json', 'The workflow source has 1 error(s); nothing was written. See details.diagnostics.'],
+ ] as const)('names where the findings of a refusal are in %s output', async (output, message) => {
+ contextMockState.output = output
+ setTerminal(false)
+ request.mockRejectedValue(
+ new SimApiError(
+ 'The workflow source has 1 error(s); nothing was written. See details.diagnostics.',
+ 400,
+ 'BAD_REQUEST',
+ {
+ code: 'SOURCE_REJECTED',
+ diagnostics: [
+ {
+ code: 'E_PARSE',
+ severity: 'error',
+ message: 'Unexpected token.',
+ file: 'workflow.ts',
+ line: 2,
+ column: 1,
+ },
+ ],
+ }
+ )
+ )
+
+ await expect(apply(WORKFLOW_ID, '--dir', dir, '--yes')).rejects.toMatchObject({ message })
+ })
+
it('at a terminal, does not ask when the dry run already found errors', async () => {
setTerminal(true)
request.mockResolvedValue(
@@ -328,6 +443,45 @@ describe('sim workflows source apply', () => {
})
})
+ it('names each added and removed block with its type in the result and the preview', async () => {
+ setTerminal(true)
+ const changed = outcome({ dryRun: true })
+ changed.data.diff.blocks.added = [{ id: 'b-3', key: 'bang', type: 'function' }]
+ changed.data.diff.blocks.removed = [{ id: 'b-2', key: 'boom', type: 'function' }]
+ request.mockResolvedValue(changed)
+ prompt.question.mockResolvedValue('n')
+
+ await apply(WORKFLOW_ID, '--dir', dir).catch(() => {})
+ await apply(WORKFLOW_ID, '--dir', dir, '--dry-run')
+
+ expect(stderrLines().some((line) => line.includes('Added: bang (function)'))).toBe(true)
+ expect(stderrLines().some((line) => line.includes('Removed: boom (function)'))).toBe(true)
+ const printed = vi.mocked(console.log).mock.calls.at(-1)?.join(' ') ?? ''
+ expect(JSON.parse(printed)).toMatchObject({
+ added: [{ block: 'bang', type: 'function' }],
+ removed: [{ block: 'boom', type: 'function' }],
+ })
+ })
+
+ it('names each changed variable in the result and the preview', async () => {
+ setTerminal(true)
+ const changed = outcome({ dryRun: true })
+ changed.data.diff.variables = { added: ['retries'], removed: [], modified: ['threshold'] }
+ request.mockResolvedValue(changed)
+ prompt.question.mockResolvedValue('n')
+
+ await apply(WORKFLOW_ID, '--dir', dir).catch(() => {})
+ await apply(WORKFLOW_ID, '--dir', dir, '--dry-run')
+
+ expect(
+ stderrLines().some((line) => line.includes('Variables: added retries; modified threshold'))
+ ).toBe(true)
+ const printed = vi.mocked(console.log).mock.calls.at(-1)?.join(' ') ?? ''
+ expect(JSON.parse(printed)).toMatchObject({
+ variables: { added: ['retries'], removed: [], modified: ['threshold'] },
+ })
+ })
+
it('prints the modified blocks readably in text output', async () => {
contextMockState.output = 'text'
const modified = outcome({ dryRun: true })
@@ -359,12 +513,17 @@ describe('sim workflows source apply', () => {
])
const printed = JSON.parse(vi.mocked(console.log).mock.calls.at(-1)?.join(' ') ?? '{}')
- expect(printed).toMatchObject({ created: false, dryRun: true })
+ expect(printed).toMatchObject({
+ dryRun: true,
+ created: [{ kind: 'workflow' }, { kind: 'block', key: 'start' }],
+ })
expect(printed).not.toHaveProperty('workflowId')
+ expect(JSON.stringify(printed.created)).not.toContain('"id"')
})
it('folds info findings into one line per code and keeps frames on warnings', async () => {
setTerminal(false)
+ contextMockState.output = 'text'
const layout = (key: string, line: number) => ({
code: 'I_MAT_NEEDS_LAYOUT',
severity: 'info',
@@ -404,6 +563,7 @@ describe('sim workflows source apply', () => {
it('prints every note on a dry run, so a kept name or a dropped tool is never silent', async () => {
setTerminal(false)
+ contextMockState.output = 'text'
const note = "The workflow name 'New' differs from the stored one and was not changed."
request.mockResolvedValue(outcome({ dryRun: true, notes: [note] }))
@@ -412,18 +572,19 @@ describe('sim workflows source apply', () => {
expect(stderrLines()).toContain(`note: ${note}`)
})
- it('counts notes apart from warnings in the result', async () => {
+ it('lists notes apart from warnings in the JSON result, and prints none on stderr', async () => {
setTerminal(false)
request.mockResolvedValue(outcome({ dryRun: true, notes: ['a kept name'] }))
await apply(WORKFLOW_ID, '--dir', dir, '--dry-run')
- const printed = JSON.parse(vi.mocked(console.log).mock.calls.at(-1)?.join(' ') ?? '{}')
- expect(printed).toMatchObject({ warnings: 0, notes: 1 })
+ expect(stdoutJson()).toMatchObject({ warnings: 0, notes: ['a kept name'] })
+ expect(stderrLines()).toEqual([])
})
it('at a terminal, prints the notes of the preview and of the applied write', async () => {
setTerminal(true)
+ contextMockState.output = 'text'
request.mockImplementation(async (_path: string, options: { query?: { dryRun?: boolean } }) =>
outcome({
dryRun: options.query?.dryRun === true,
@@ -441,6 +602,7 @@ describe('sim workflows source apply', () => {
it('at a terminal, prints a warning and a note the preview showed once, not again on the write', async () => {
setTerminal(true)
+ contextMockState.output = 'text'
const warning = {
code: 'W_REQUIRED_MISSING',
severity: 'warning',
@@ -483,6 +645,236 @@ describe('sim workflows source apply', () => {
})
})
+ describe('findings and results an agent reads', () => {
+ const untyped = (line: number) => ({
+ code: 'W_JSON_REF_UNTYPED',
+ severity: 'warning',
+ message: `Nothing declares the type of .`,
+ file: 'workflow.ts',
+ line,
+ column: 7,
+ frame: `> ${line} | n: score.result.n${line},`,
+ hint: 'Leave it bare when it holds a number.',
+ })
+ const layout = {
+ code: 'I_MAT_NEEDS_LAYOUT',
+ severity: 'info',
+ message: 'Block "start" is placed by autolayout',
+ file: 'workflow.ts',
+ line: 3,
+ column: 3,
+ }
+
+ it('with --output json, prints findings only in the JSON on stdout, never as frames on stderr', async () => {
+ setTerminal(false)
+ request.mockResolvedValue(outcome({ dryRun: true, diagnostics: [untyped(4)] }))
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--dry-run')
+
+ expect(stdoutJson()).toMatchObject({ diagnostics: [{ code: 'W_JSON_REF_UNTYPED' }] })
+ expect(stderrLines().join('\n')).not.toContain('W_JSON_REF_UNTYPED')
+ })
+
+ it('leaves info findings out of the JSON unless --verbose', async () => {
+ setTerminal(false)
+ request.mockResolvedValue(outcome({ dryRun: true, diagnostics: [layout, untyped(4)] }))
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--dry-run')
+ expect(stdoutJson().diagnostics).toEqual([expect.objectContaining({ severity: 'warning' })])
+
+ vi.mocked(console.log).mockClear()
+ await apply(WORKFLOW_ID, '--dir', dir, '--dry-run', '--verbose')
+ expect(stdoutJson().diagnostics).toEqual([
+ expect.objectContaining({ code: 'I_MAT_NEEDS_LAYOUT' }),
+ expect.objectContaining({ code: 'W_JSON_REF_UNTYPED' }),
+ ])
+ })
+
+ it('in a display format, prints the first warning of a code whole and the rest on one line', async () => {
+ setTerminal(false)
+ contextMockState.output = 'text'
+ request.mockResolvedValue(
+ outcome({ dryRun: true, diagnostics: [4, 5, 6, 7].map((line) => untyped(line)) })
+ )
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--dry-run')
+
+ const lines = stderrLines()
+ expect(lines.filter((line) => line.includes('> '))).toHaveLength(1)
+ expect(lines).toContain(
+ ' 3 more W_JSON_REF_UNTYPED like it: workflow.ts:5:7 (Nothing declares the type of .), workflow.ts:6:7 (Nothing declares the type of .), workflow.ts:7:7 (Nothing declares the type of .)'
+ )
+ })
+
+ it('in a display format, names each collapsed warning by location alone when they say the same', async () => {
+ setTerminal(false)
+ contextMockState.output = 'text'
+ request.mockResolvedValue(
+ outcome({
+ dryRun: true,
+ diagnostics: [4, 5].map((line) => ({ ...untyped(line), message: 'Same words.' })),
+ })
+ )
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--dry-run')
+
+ expect(stderrLines()).toContain(' 1 more W_JSON_REF_UNTYPED like it: workflow.ts:5:7')
+ })
+
+ it('with --output json and --verbose, keeps info findings in a refusal too', async () => {
+ setTerminal(false)
+ request.mockRejectedValue(
+ new SimApiError('Refused. See details.diagnostics.', 422, 'SOURCE_REJECTED', {
+ diagnostics: [layout, STALE],
+ })
+ )
+
+ const refused = await apply(WORKFLOW_ID, '--dir', dir, '--yes', '--verbose').then(
+ () => null,
+ (error: unknown) => error as SimApiError
+ )
+
+ expect(toRecord(refused?.details).diagnostics).toEqual([
+ expect.objectContaining({ code: 'I_MAT_NEEDS_LAYOUT' }),
+ expect.objectContaining({ code: 'E_APPLY_STALE' }),
+ ])
+ })
+
+ it('lists the workflow and blocks a create made, with their ids', async () => {
+ setTerminal(false)
+ vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config'))
+ rmSync(join(dir, '.sim/base.json'))
+ const catalog = JSON.parse(readFileSync(FIXTURE_CATALOG, 'utf8'))
+ request.mockImplementation(async (path: string) => {
+ if (path.endsWith('/workflows/source/catalog')) return { data: catalog }
+ if (path.endsWith('/workflows/source/resources')) {
+ return {
+ data: {
+ workspaceId: WORKSPACE_ID,
+ tables: [],
+ knowledgeBases: [],
+ credentials: [],
+ workflows: [],
+ env: [],
+ truncated: false,
+ },
+ }
+ }
+ return outcome()
+ })
+
+ await program().parseAsync([
+ 'node',
+ 'sim',
+ 'workflows',
+ 'source',
+ 'create',
+ '--dir',
+ dir,
+ '--yes',
+ ])
+
+ expect(stdoutJson()).toMatchObject({
+ workflowId: WORKFLOW_ID,
+ created: [
+ { kind: 'workflow', id: WORKFLOW_ID },
+ { kind: 'block', key: 'start', id: 'b-1' },
+ ],
+ })
+ })
+
+ it('names what an apply created on one line in a display format', async () => {
+ setTerminal(false)
+ contextMockState.output = 'text'
+ request.mockResolvedValue(outcome())
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--yes')
+
+ expect(vi.mocked(console.log).mock.calls.map((call) => call.join(' '))).toContain(
+ 'created\tblock start (b-1)'
+ )
+ })
+ })
+
+ describe('--deploy', () => {
+ const DEPLOY_PATH = `/api/v2/workflows/${WORKFLOW_ID}/deploy`
+ const deployed = {
+ data: {
+ id: WORKFLOW_ID,
+ isDeployed: true,
+ deployedAt: null,
+ warnings: [],
+ activeDeployment: null,
+ latestDeploymentAttempt: null,
+ version: 4,
+ lint: null,
+ },
+ }
+
+ it('deploys through the deploy operation once the apply is saved', async () => {
+ setTerminal(false)
+ request.mockImplementation(async (path: string) =>
+ path === DEPLOY_PATH ? deployed : outcome()
+ )
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--yes', '--deploy')
+
+ expect(request.mock.calls.map((call) => call[0])).toEqual([APPLY_PATH, DEPLOY_PATH])
+ expect(readFileSync(join(dir, 'workflow.ts'), 'utf8')).toBe(`${ENTRY}// stored\n`)
+ expect(stdoutJson()).toMatchObject({
+ applied: true,
+ deployed: { version: 4, isDeployed: true },
+ needsRedeployment: false,
+ })
+ })
+
+ it('without --deploy, names the deploy command when the deployed workflow now differs', async () => {
+ setTerminal(false)
+ request.mockResolvedValue(outcome())
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--yes')
+
+ expect(request.mock.calls.map((call) => call[0])).toEqual([APPLY_PATH])
+ expect(stdoutJson()).toMatchObject({
+ needsRedeployment: true,
+ next: { command: `sim workflows deploy ${WORKFLOW_ID}` },
+ })
+ })
+
+ it('deploys nothing after a refused apply, and refuses beside --dry-run', async () => {
+ setTerminal(false)
+ request.mockResolvedValue(outcome({ diagnostics: [STALE] }))
+
+ await apply(WORKFLOW_ID, '--dir', dir, '--yes', '--deploy')
+ expect(request.mock.calls.map((call) => call[0])).toEqual([APPLY_PATH])
+
+ request.mockClear()
+ await expect(apply(WORKFLOW_ID, '--dir', dir, '--dry-run', '--deploy')).rejects.toThrow(
+ /--dry-run saves nothing/
+ )
+ expect(request).not.toHaveBeenCalled()
+ })
+
+ it('says the apply was saved when the deploy after it fails', async () => {
+ setTerminal(false)
+ request.mockImplementation(async (path: string) => {
+ if (path === DEPLOY_PATH) throw new SimApiError('Forbidden', 403, 'FORBIDDEN')
+ return outcome()
+ })
+
+ const error = await apply(WORKFLOW_ID, '--dir', dir, '--yes', '--deploy').catch(
+ (caught: unknown) => caught
+ )
+
+ expect(error).toMatchObject({ status: 403, code: 'FORBIDDEN' })
+ expect((error as Error).message).toContain(`The apply was saved into ${dir}`)
+ expect((error as Error).message).toContain(`sim workflows deploy ${WORKFLOW_ID}`)
+ expect(readFileSync(join(dir, 'workflow.ts'), 'utf8')).toBe(`${ENTRY}// stored\n`)
+ // The saved write's result still prints, before the deploy's failure.
+ expect(stdoutJson()).toMatchObject({ workflowId: WORKFLOW_ID, applied: true })
+ })
+ })
+
it('exits 5 when a dry run finds the directory stale', async () => {
setTerminal(false)
request.mockResolvedValue(outcome({ dryRun: true, diagnostics: [STALE] }))
@@ -755,6 +1147,75 @@ describe('sim workflows source create on a directory with no workflow.ts', () =>
expect(printed.files).toBeGreaterThan(0)
})
+ /** Answers the starter's catalog and manifest reads, and a create with a stored workflow. */
+ function answerStarterAndCreate() {
+ request.mockImplementation(async (path: string, init: { query?: { dryRun?: boolean } }) => {
+ if (path.endsWith('/workflows/source/catalog')) return { data: CATALOG }
+ if (path.endsWith('/workflows/source/resources')) return MANIFEST
+ if (path.endsWith('/workflows/source')) return outcome({ dryRun: init.query?.dryRun })
+ throw new Error(`unexpected request ${path}`)
+ })
+ }
+
+ function createCalls(): string[] {
+ return request.mock.calls
+ .map((call) => call[0] as string)
+ .filter((path) => path.endsWith('/workflows/source'))
+ }
+
+ it('records the workspace the starter was typed for and names it in the next step', async () => {
+ setTerminal(false)
+ vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config'))
+ const empty = join(dir, 'new')
+ answerStarterAndCreate()
+
+ await create('--dir', empty)
+
+ expect(JSON.parse(readFileSync(join(empty, '.sim/starter.json'), 'utf8'))).toEqual({
+ workspaceId: WORKSPACE_ID,
+ })
+ expect(stderrLines().join('\n')).toContain(
+ `sim workflows source create --dir ${empty} --workspace ${WORKSPACE_ID} --yes`
+ )
+ })
+
+ it('creates a starter in the workspace it was typed for, whatever the profile default', async () => {
+ setTerminal(false)
+ vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config'))
+ const starter = join(dir, 'new')
+ answerStarterAndCreate()
+ await create('--dir', starter)
+ contextMockState.workspaceId = 'ws_other'
+ contextMockState.workspaceSource = 'config'
+
+ try {
+ await create('--dir', starter, '--yes')
+ expect(createCalls()).toEqual([`/api/v2/workspaces/${WORKSPACE_ID}/workflows/source`])
+ } finally {
+ contextMockState.workspaceId = WORKSPACE_ID
+ }
+ })
+
+ it('creates a starter where --workspace names, and drops the starter record once created', async () => {
+ setTerminal(false)
+ vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config'))
+ const starter = join(dir, 'new')
+ answerStarterAndCreate()
+ await create('--dir', starter)
+ contextMockState.workspaceId = 'ws_other'
+ contextMockState.workspaceSource = 'flag'
+
+ try {
+ await create('--dir', starter, '--yes')
+ expect(createCalls()).toEqual(['/api/v2/workspaces/ws_other/workflows/source'])
+ expect(existsSync(join(starter, '.sim/starter.json'))).toBe(false)
+ expect(existsSync(join(starter, '.sim/base.json'))).toBe(true)
+ } finally {
+ contextMockState.workspaceId = WORKSPACE_ID
+ contextMockState.workspaceSource = 'config'
+ }
+ })
+
it.each(['--dry-run', '--yes'])('refuses %s instead of scaffolding', async (flag) => {
setTerminal(false)
vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config'))
diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-apply.ts b/packages/sim-cli/src/commands/protocol/workflow-source-apply.ts
index 69b7f704ce1..ee3233cefd7 100644
--- a/packages/sim-cli/src/commands/protocol/workflow-source-apply.ts
+++ b/packages/sim-cli/src/commands/protocol/workflow-source-apply.ts
@@ -1,4 +1,5 @@
import { mapWithConcurrency } from '@sim/utils/concurrency'
+import { toRecord } from '@sim/utils/object'
import { parseSourceBase, parseWorkflowLock, type WorkflowLockV1 } from '@sim/workflow-code'
import { codeFileReferences } from '@sim/workflow-code/code-files/references'
import {
@@ -9,34 +10,49 @@ import {
} from '@sim/workflow-code/limits'
import type { Command } from 'commander'
import { printError } from '#sim-cli/output/io'
+import { DEFERRED_INTERRUPT_BOUND_MS, deferInterrupts } from '#sim-cli/terminal/interrupts'
import { setSoftExitCode } from '../../embed-context'
import type {
ApplyWorkflowSourceBody,
ApplyWorkflowSourceResponse,
CreateWorkflowFromSourceBody,
+ DeployWorkflowResponse,
} from '../../generated/v2-api'
import { SimApiError } from '../../http/client'
import { apiCommand, type OperationClient } from '../../runtime/called-operations'
import { printProtocolResult } from './result'
import {
+ absoluteDirOption,
FILE_CONCURRENCY,
lockCodeFiles,
readSourceText,
+ readStarterWorkspace,
SOURCE_AGENTS_FILE,
+ SOURCE_STARTER_FILE,
sourceRelativePath,
+ updateSourceDir,
WORKSPACE_FOLDER_BUSY_HELP,
withWorkspaceFolderLock,
writeBack,
} from './source-dir'
+import { SOURCE_WRITE_EXIT_CODES_HELP } from './source-exit-codes'
+import { cachedCatalogByHash, printTreeResult } from './source-tree'
import {
+ type CreatedResource,
confirm,
+ createdText,
+ type FindingsOutput,
+ findingsOutput,
fresh,
- printNewDiagnostics,
+ planNext,
printRefusal,
refusedExitCode,
- requireConsent,
- resultDiagnostics,
+ shellWord,
+ stopAtPlan,
+ verboseFlag,
+ writeMode,
} from './source-ui'
+import { storedEdgesWrite } from './stored-edges'
import {
refreshWorkflowTsconfigs,
writeSourceDirectory,
@@ -50,15 +66,20 @@ const APPLY_EXIT_CODES_HELP = `
Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as
file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the
directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry);
-6 the workflow or its folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after
-the CLI's retries, so nothing was written: wait a minute and run it again.`
+6 the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1,
+E_APPLY_LOCKED_BLOCK); 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's
+retries, so nothing was written: wait a minute and run it again; 14 without --yes and with no
+terminal to ask: the plan was printed with no errors and nothing was written (run the same
+command with --yes to apply it).`
const CREATE_EXIT_CODES_HELP = `
Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each
printed as file:line:column with its frame and hint), the workflow name is taken
-(SOURCE_NAME_TAKEN: pass --name), or the command failed; 6 the folder is locked; 7 the source
-is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a
-minute and run it again.`
+(SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the
+workspace its types were written for unless --workspace names another); 6 the folder is locked; 7 the
+source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created:
+wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was
+printed with no errors and nothing was created (run the same command with --yes).`
type SourceApplyData = ApplyWorkflowSourceResponse['data']
@@ -69,13 +90,26 @@ interface SourceWriteOptions {
dryRun?: boolean
yes?: boolean
allowDelete?: boolean
+ verbose?: boolean
+ deploy?: boolean
}
-/** The text of a file the directory must hold, or a refusal that says how to write it. */
+/**
+ * The text of a file the directory must hold, or a refusal that says how to write it: `hint`, or
+ * for a starter no workflow was created from yet, the create that makes it one.
+ */
async function requireText(directory: string, path: string, hint: string): Promise {
const text = await readSourceText(directory, path)
- if (text === null) throw new SimApiError(`${directory} has no ${path}. ${hint}`, 0)
- return text
+ if (text !== null) return text
+ const starterWorkspace = await readStarterWorkspace(directory)
+ throw new SimApiError(
+ `${directory} has no ${path}. ${
+ starterWorkspace === null
+ ? hint
+ : `It holds a starter no workflow was created from yet: run \`${createCommand(directory, starterWorkspace)} --dry-run\`, then the same with --yes.`
+ }`,
+ 0
+ )
}
async function readJson(directory: string, path: string, hint: string): Promise {
@@ -105,14 +139,15 @@ function codeFilePaths(entry: string, lock: WorkflowLockV1 | null): string[] {
}
/**
- * Prints a write's diagnostics and notes (such as a custom tool the save could not store),
- * skipping any already in `printed` and adding the rest: after a terminal preview, the
+ * Prints a write's diagnostics and, in a display format, its notes (such as a custom tool the
+ * save could not store; a machine result lists them), each once: after a terminal preview, the
* write it confirmed prints only what the preview did not.
*/
-function printFindings(data: SourceApplyData, printed: Set): void {
- printNewDiagnostics(data.diagnostics, printed)
+function printFindings(data: SourceApplyData, findings: FindingsOutput, notes: Set): void {
+ findings.print(data.diagnostics)
+ if (findings.machine) return
for (const note of data.notes) {
- if (fresh(printed, `note:${note}`)) printError(`note: ${note}`)
+ if (fresh(notes, note)) printError(`note: ${note}`)
}
}
@@ -126,12 +161,17 @@ function diffSummary(data: SourceApplyData): string {
`${blocks.renamed.length} renamed`,
`${blocks.modified.length} modified`,
]
- const variableCount =
- variables.added.length + variables.removed.length + variables.modified.length
+ const variableChanges = (['added', 'removed', 'modified'] as const).flatMap((change) =>
+ variables[change].length > 0 ? [`${change} ${variables[change].join(', ')}`] : []
+ )
const modified = blocks.modified.map(({ key, fields }) => `${key} (${fields.join(', ')})`)
+ const typed = (list: ReadonlyArray<{ key: string; type: string }>) =>
+ list.map(({ key, type }) => `${key} (${type})`).join('; ')
return `Blocks: ${parts.join(', ')}. Connections: +${edges.added} -${edges.removed}.${
- variableCount > 0 ? ` Variables changed: ${variableCount}.` : ''
- }${modified.length > 0 ? `\nModified: ${modified.join('; ')}` : ''}`
+ blocks.added.length > 0 ? `\nAdded: ${typed(blocks.added)}` : ''
+ }${blocks.removed.length > 0 ? `\nRemoved: ${typed(blocks.removed)}` : ''}${
+ modified.length > 0 ? `\nModified: ${modified.join('; ')}` : ''
+ }${variableChanges.length > 0 ? `\nVariables: ${variableChanges.join('; ')}` : ''}`
}
const GET_HINT = 'Write the directory with `sim workflows source get` first.'
@@ -176,6 +216,11 @@ export async function readLock(directory: string): Promise {
)
}
+/** The directory's `workflow.ts`, or a refusal naming the command that writes one. */
+export function requireEntry(directory: string): Promise {
+ return requireText(directory, ENTRY_FILE, GET_HINT)
+}
+
/**
* The files a write sends: the entry file and the code files it or the lock references. A
* referenced file that is missing is left out, and the server reports it with its source line.
@@ -184,7 +229,7 @@ export async function readSourceFiles(
directory: string,
lock: WorkflowLockV1 | null
): Promise> {
- const entry = await requireText(directory, ENTRY_FILE, GET_HINT)
+ const entry = await requireEntry(directory)
const files: Record = { [ENTRY_FILE]: entry }
const paths = codeFilePaths(entry, lock)
const texts = await mapWithConcurrency(paths, FILE_CONCURRENCY, (path) =>
@@ -202,13 +247,15 @@ type SourceWriteOperation = 'applyWorkflowSource' | 'createWorkflowFromSource'
/**
* The shared write flow of `apply` and `create`: refuse a write without consent when no terminal
* can ask, preview and ask on a terminal, send, print the diagnostics, and after a successful
- * write save it into the directory with the caller's `save`.
+ * write save it into the directory with the caller's `save`. From the send through the save an
+ * interrupt waits for them ({@link deferInterrupts}), so Ctrl-C never strands a stored write the
+ * directory did not record.
*/
async function runSourceWrite(args: {
options: SourceWriteOptions
creating: boolean
question: string
- client: OperationClient
+ client: OperationClient
operation: Operation
params: Record
body: ApplyWorkflowSourceBody | CreateWorkflowFromSourceBody
@@ -217,6 +264,8 @@ async function runSourceWrite(args: {
save: (data: SourceApplyData) => Promise
/** The next step when the write's answer never came back, so it may have been stored. */
ifAnswerLost: string
+ /** This write as a command, without `--yes`: what a `plan` run names next with it. */
+ command: string
}): Promise {
const { options, creating } = args
/** Sends the write (or its dry run); a refusal prints its diagnostics and rethrows. */
@@ -234,38 +283,161 @@ async function runSourceWrite(args: {
})
return response.data
} catch (error) {
- return printRefusal(error, args.output, dryRun ? undefined : args.ifAnswerLost)
+ return printRefusal(error, args.output, {
+ ...(dryRun ? {} : { ifAnswerLost: args.ifAnswerLost }),
+ verbose: options.verbose === true,
+ })
}
}
- const printed = new Set()
- if (!options.dryRun && !options.yes) {
+ const findings = findingsOutput(args.output, options.verbose)
+ const notes = new Set()
+ const report = (data: SourceApplyData, deployment: DeployOutcome = {}) => {
+ const result = { ...summarize(data, creating, findings.machine), ...deployment }
+ printProtocolResult(
+ args.output,
+ { ...result, ...findings.result(data.diagnostics) },
+ {
+ ...(result.created ? { created: createdText(result.created) } : {}),
+ ...(deployment.next ? { next: deployment.next.command } : {}),
+ ...(deployment.deployed
+ ? { deployed: `version ${deployment.deployed.version ?? '?'}` }
+ : {}),
+ }
+ )
+ }
+ const mode = writeMode(options)
+ if (mode === 'ask' || mode === 'plan') {
const preview = await send(true)
- printFindings(preview, printed)
+ printFindings(preview, findings, notes)
if (preview.diagnostics.some((diagnostic) => diagnostic.severity === 'error')) {
setSoftExitCode(refusedExitCode(preview.diagnostics))
- printProtocolResult(args.output, {
- ...summarize(preview, creating),
- ...resultDiagnostics(args.output, preview.diagnostics),
- })
+ report(preview)
return
}
printError(diffSummary(preview))
+ if (mode === 'plan') {
+ stopAtPlan()
+ report(preview, { next: planNext(args.command) })
+ return
+ }
if (!(await confirm(args.question))) {
throw new SimApiError('Nothing was applied.', 0, 'CANCELLED')
}
}
- const data = await send(options.dryRun === true)
- printFindings(data, printed)
- const refused = data.diagnostics.some((diagnostic) => diagnostic.severity === 'error')
- if (refused) setSoftExitCode(refusedExitCode(data.diagnostics))
+ const verb = creating ? 'create' : 'apply'
+ let refused = false
+ const sendAndSave = async (): Promise => {
+ const data = await send(options.dryRun === true)
+ printFindings(data, findings, notes)
+ refused = data.diagnostics.some((diagnostic) => diagnostic.severity === 'error')
+ if (refused) setSoftExitCode(refusedExitCode(data.diagnostics))
+ if (!data.dryRun && !refused && (data.applied || !creating)) await args.save(data)
+ return data
+ }
+ const data = options.dryRun
+ ? await sendAndSave()
+ : await deferInterrupts(sendAndSave, {
+ waiting: `Interrupted: the ${verb} was already sent. Waiting up to ${DEFERRED_INTERRUPT_BOUND_MS / 1000}s for its answer and saving it into ${options.dir}, so the directory matches the workflow; press Ctrl-C again to stop now.`,
+ abandoned: `Stopped before the ${verb} answered. ${args.ifAnswerLost}`,
+ })
- if (!data.dryRun && !refused && (data.applied || !creating)) await args.save(data)
+ const stored = !data.dryRun && !refused && (data.applied || !creating)
+ if (mode === 'write' && stored) printError(diffSummary(data))
+ let deployment: DeployOutcome = {}
+ if (stored) {
+ try {
+ deployment = await deployStep(args.client, data, {
+ ...options,
+ verb,
+ machine: findings.machine,
+ })
+ } catch (error) {
+ // The write stands: its result prints before the deploy's failure.
+ report(data)
+ throw error
+ }
+ }
+ report(data, deployment)
+}
- printProtocolResult(args.output, {
- ...summarize(data, creating),
- ...resultDiagnostics(args.output, data.diagnostics),
- })
+/** What a write adds to its result about deploying: the deploy it ran, or the one to run. */
+interface DeployOutcome {
+ deployed?: { version?: number; isDeployed: boolean; warnings?: string[] }
+ needsRedeployment?: boolean
+ next?: { command: string; reason: string }
+}
+
+/**
+ * After a stored write: with `--deploy`, deploys the workflow through the same operation as
+ * `sim workflows deploy` (so the same permission check, on the server); without it, names that
+ * command as the next step when the workflow is deployed and its draft now differs. A failed
+ * deploy says the write itself was saved.
+ */
+async function deployStep(
+ client: OperationClient<'deployWorkflow'>,
+ data: SourceApplyData,
+ options: { deploy?: boolean; dir: string; verb: string; machine: boolean }
+): Promise {
+ const deployCommand = `sim workflows deploy ${data.workflowId}`
+ if (!options.deploy) {
+ if (!data.needsRedeployment) return {}
+ return {
+ next: {
+ command: deployCommand,
+ reason: `The workflow is deployed and its draft now differs from the live version: deploy to make the change live (or ${options.verb} with --deploy).`,
+ },
+ }
+ }
+ const deployed = await client
+ .request('deployWorkflow', {
+ params: { workflowId: data.workflowId },
+ body: {},
+ })
+ .then(
+ (response) => response.data,
+ (error: unknown) => {
+ throw deployFailure(error, options, deployCommand)
+ }
+ )
+ if (!options.machine) {
+ for (const warning of deployed.warnings) printError(`deploy warning: ${warning}`)
+ }
+ return {
+ deployed: {
+ ...(deployed.version !== undefined ? { version: deployed.version } : {}),
+ isDeployed: deployed.isDeployed,
+ ...(options.machine && deployed.warnings.length > 0 ? { warnings: deployed.warnings } : {}),
+ },
+ needsRedeployment: false,
+ }
+}
+
+/** A failed deploy after a saved write, saying the write itself stands and how to deploy it. */
+function deployFailure(
+ error: unknown,
+ options: { dir: string; verb: string },
+ deployCommand: string
+): unknown {
+ if (!(error instanceof SimApiError)) return error
+ return new SimApiError(
+ `The ${options.verb} was saved into ${options.dir}, but the deploy failed: ${error.message} Fix that, then run \`${deployCommand}\`.`,
+ error.status,
+ error.code,
+ error.details,
+ error.exitCode,
+ error.requestId
+ )
+}
+
+/** Refuses `--deploy` beside `--dry-run`: a dry run stores nothing to deploy. */
+function assertDeployable(options: SourceWriteOptions): void {
+ if (options.deploy && options.dryRun) {
+ throw new SimApiError(
+ '--deploy deploys what the write saves, and --dry-run saves nothing: drop one of them.',
+ 0
+ )
+ }
}
const DIR_DESCRIPTION =
@@ -282,7 +454,12 @@ const DIR_DESCRIPTION =
* needs `--yes`. `--dry-run` writes nothing, so it never asks.
*/
export function attachWorkflowSourceApply(source: Command): void {
- const [command, connectApply] = apiCommand(source, 'apply', ['applyWorkflowSource'])
+ const [command, connectApply] = apiCommand(source, 'apply', [
+ 'applyWorkflowSource',
+ 'deployWorkflow',
+ ])
+ absoluteDirOption(command)
+ verboseFlag(command)
command
.argument(
'[workflowId]',
@@ -291,16 +468,23 @@ export function attachWorkflowSourceApply(source: Command): void {
.requiredOption('--dir ', DIR_DESCRIPTION)
.option('--dry-run', 'Check the source and show the outcome without saving or writing files')
.option('-y, --yes', 'Apply without asking for confirmation')
+ .option(
+ '--deploy',
+ 'After the write is saved, deploy the workflow (as `sim workflows deploy ` does)'
+ )
.option(
'--allow-delete',
'Delete blocks the source no longer declares; without it, an apply that deletes a block is refused'
)
.allowExcessArguments(false)
.description('Apply a workflow source directory to a workflow draft')
- .addHelpText('after', `${APPLY_EXIT_CODES_HELP}${WORKSPACE_FOLDER_BUSY_HELP}`)
+ .addHelpText(
+ 'after',
+ `${APPLY_EXIT_CODES_HELP}${WORKSPACE_FOLDER_BUSY_HELP}${SOURCE_WRITE_EXIT_CODES_HELP}`
+ )
.action(async (named: string | undefined, options: SourceWriteOptions) =>
withWorkspaceFolderLock(options.dir, async () => {
- requireConsent(options, 'writes the workflow draft', 'apply')
+ assertDeployable(options)
const { client, profile } = connectApply()
const lock = await readLock(options.dir)
const base = await readBase(options.dir, named)
@@ -310,13 +494,18 @@ export function attachWorkflowSourceApply(source: Command): void {
options,
creating: false,
question: `Apply these changes to workflow ${workflowId}?`,
+ command: `sim workflows source apply --dir ${shellWord(options.dir)}${options.allowDelete ? ' --allow-delete' : ''}${options.deploy ? ' --deploy' : ''}`,
output: profile.output,
client,
operation: 'applyWorkflowSource',
params: { workflowId },
body: { files, lock, base, provenance: { client: 'cli' } },
save: async (data) => {
- await writeBack(options.dir, data, lock)
+ const edges = storedEdgesWrite(
+ data,
+ cachedCatalogByHash(toRecord(data.base).catalogHash)
+ )
+ await writeBack(options.dir, data, lock, { write: edges ? [edges] : [] })
await refreshWorkflowTsconfigs(options.dir, data)
},
ifAnswerLost: `It may have been saved. Run \`sim workflows source apply ${workflowId} --dir ${options.dir} --dry-run\`: exit 5 (stale) means the draft changed since this directory was read, by this apply or by someone else. Run \`sim workflows source get ${workflowId} --dir \` and compare it with ${options.dir}: if it holds your edit, \`sim workflows source get ${workflowId} --dir ${options.dir} --force\` brings this directory up to date; if not, reapply your edit in the fresh directory. Any other outcome means it was not saved: apply again.`,
@@ -336,7 +525,10 @@ export function attachWorkflowSourceCreate(source: Command): void {
'createWorkflowFromSource',
'getSourceCatalog',
'getSourceResources',
+ 'deployWorkflow',
])
+ absoluteDirOption(command)
+ verboseFlag(command)
command
.requiredOption(
'--dir ',
@@ -346,9 +538,16 @@ export function attachWorkflowSourceCreate(source: Command): void {
.option('--name ', "The workflow name (default: the source's)")
.option('--folder ', 'The folder to create it in (default: the root)')
.option('-y, --yes', 'Create without asking for confirmation')
+ .option(
+ '--deploy',
+ 'After the write is saved, deploy the workflow (as `sim workflows deploy ` does)'
+ )
.allowExcessArguments(false)
.description('Create a workflow from a workflow source directory')
- .addHelpText('after', `${CREATE_EXIT_CODES_HELP}${WORKSPACE_FOLDER_BUSY_HELP}`)
+ .addHelpText(
+ 'after',
+ `${CREATE_EXIT_CODES_HELP}${WORKSPACE_FOLDER_BUSY_HELP}${SOURCE_WRITE_EXIT_CODES_HELP}`
+ )
.action(async (options: SourceWriteOptions & { name?: string; folder?: string }) =>
withWorkspaceFolderLock(options.dir, async () => {
if ((await readSourceText(options.dir, ENTRY_FILE)) === null) {
@@ -359,21 +558,27 @@ export function attachWorkflowSourceCreate(source: Command): void {
)
}
const { client, profile } = connectCreate()
- const tree = await writeSourceScaffold(client, options.dir, client.requireWorkspace())
+ const workspaceId = client.requireWorkspace()
+ const tree = await writeSourceScaffold(client, options.dir, workspaceId)
printError(
- `${options.dir} had no ${ENTRY_FILE}, so a starter, the SDK types, tsconfig.json, .sim/AGENTS.md and the examples in .sim/examples/ were written and nothing was created. Edit ${ENTRY_FILE}, then run \`sim workflows source create --dir ${options.dir} --yes\`.`
+ `${options.dir} had no ${ENTRY_FILE}, so a starter, the SDK types for workspace ${workspaceId}, tsconfig.json, .sim/AGENTS.md and the examples in .sim/examples/ were written and nothing was created. Edit ${ENTRY_FILE}, then run \`${createCommand(options.dir, workspaceId)} --yes\`.`
)
- printProtocolResult(profile.output, { created: false, scaffolded: true, ...tree })
+ printTreeResult(profile.output, { created: false, scaffolded: true, ...tree })
return
}
- requireConsent(options, 'creates a workflow', 'create')
+ assertDeployable(options)
const { client, profile } = connectCreate()
- const workspaceId = client.requireWorkspace()
+ // A starter is bound to the workspace its types were written for, as a get directory
+ // is to the one it was read from; --workspace still chooses another.
+ const starterWorkspace =
+ profile.sources.workspaceId === 'flag' ? null : await readStarterWorkspace(options.dir)
+ const workspaceId = client.requireWorkspace(starterWorkspace ?? undefined)
const files = await readSourceFiles(options.dir, null)
await runSourceWrite<'createWorkflowFromSource'>({
options,
creating: true,
question: 'Create this workflow?',
+ command: `${createCommand(shellWord(options.dir), workspaceId)}${options.name !== undefined ? ` --name ${shellWord(options.name)}` : ''}${options.folder !== undefined ? ` --folder ${shellWord(options.folder)}` : ''}${options.deploy ? ' --deploy' : ''}`,
output: profile.output,
client,
operation: 'createWorkflowFromSource',
@@ -381,6 +586,7 @@ export function attachWorkflowSourceCreate(source: Command): void {
ifAnswerLost: `It may have been created. Run this command again: a SOURCE_NAME_TAKEN refusal means it was, so find its id with \`sim workflows list\` and run \`sim workflows source get --dir ${options.dir} --force\`.`,
save: async (data) => {
await writeSourceDirectory(client, options.dir, data, [])
+ await updateSourceDir(options.dir, { write: [], remove: [SOURCE_STARTER_FILE] })
},
body: {
files,
@@ -393,27 +599,65 @@ export function attachWorkflowSourceCreate(source: Command): void {
)
}
-/** A create that did not apply stored nothing, so its provisional workflow id names no workflow. */
-function summarize(data: SourceApplyData, created: boolean) {
+/** Whether a diff adds, removes or changes any workflow variable. */
+function variablesChanged(variables: SourceApplyData['diff']['variables']): boolean {
+ return variables.added.length > 0 || variables.removed.length > 0 || variables.modified.length > 0
+}
+
+/** The create command for `dir` in `workspaceId`, as the next step a starter prints. */
+function createCommand(dir: string, workspaceId: string): string {
+ return `sim workflows source create --dir ${dir} --workspace ${workspaceId}`
+}
+
+/**
+ * What a write created (on a dry run, would create; ids are only known once it is written): the
+ * workflow of a create, then each block added.
+ */
+function createdResources(data: SourceApplyData, creating: boolean): CreatedResource[] {
+ const written = !data.dryRun && data.applied
+ return [
+ ...(creating ? [{ kind: 'workflow', ...(written ? { id: data.workflowId } : {}) }] : []),
+ ...data.diff.blocks.added.map(({ key, id }) => ({
+ kind: 'block',
+ key,
+ ...(written ? { id } : {}),
+ })),
+ ]
+}
+
+/**
+ * The result of a write. A create that did not apply stored nothing, so its provisional workflow
+ * id names no workflow. Notes are listed in a `machine` result, and counted in a display one,
+ * which prints them on stderr.
+ */
+function summarize(data: SourceApplyData, creating: boolean, machine: boolean) {
+ const created = data.dryRun || data.applied ? createdResources(data, creating) : []
return {
- ...(created && !data.applied ? {} : { workflowId: data.workflowId }),
- ...(created ? { created: data.applied } : {}),
+ ...(creating && !data.applied ? {} : { workflowId: data.workflowId }),
applied: data.applied,
dryRun: data.dryRun,
noop: data.diff.noop,
+ ...(created.length > 0 ? { created } : {}),
blocksAdded: data.diff.blocks.added.length,
+ ...(data.diff.blocks.added.length > 0 && {
+ added: data.diff.blocks.added.map(({ key, type }) => ({ block: key, type })),
+ }),
blocksRemoved: data.diff.blocks.removed.length,
+ ...(data.diff.blocks.removed.length > 0 && {
+ removed: data.diff.blocks.removed.map(({ key, type }) => ({ block: key, type })),
+ }),
blocksModified: data.diff.blocks.modified.length,
...(data.diff.blocks.modified.length > 0 && {
modified: data.diff.blocks.modified.map(({ key, fields }) => ({ block: key, fields })),
}),
edgesAdded: data.diff.edges.added,
edgesRemoved: data.diff.edges.removed,
+ ...(variablesChanged(data.diff.variables) && { variables: data.diff.variables }),
errors: data.diagnostics.filter((diagnostic) => diagnostic.severity === 'error').length,
warnings: data.diagnostics.filter((diagnostic) => diagnostic.severity === 'warning').length,
- notes: data.notes.length,
+ notes: machine ? data.notes : data.notes.length,
fingerprint: data.base.fingerprint,
needsRedeployment: data.needsRedeployment,
- ...(created && data.applied && !data.dryRun ? { guide: SOURCE_AGENTS_FILE } : {}),
+ ...(creating && data.applied && !data.dryRun ? { guide: SOURCE_AGENTS_FILE } : {}),
}
}
diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-check.test.ts b/packages/sim-cli/src/commands/protocol/workflow-source-check.test.ts
index c597f47ff97..e2989388885 100644
--- a/packages/sim-cli/src/commands/protocol/workflow-source-check.test.ts
+++ b/packages/sim-cli/src/commands/protocol/workflow-source-check.test.ts
@@ -4,7 +4,8 @@ import { join } from 'node:path'
import { Command } from 'commander'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import type { SimApiError } from '../../http/client'
-import { contextMockFns } from '../../test/context-mock'
+import { cacheCatalogText } from '../../test/catalog-cache'
+import { contextMockFns, mockedProfile } from '../../test/context-mock'
import { attachWorkflowSourceCheck } from './workflow-source-check'
vi.mock('../../context', async () => (await import('../../test/context-mock')).contextMock)
@@ -33,11 +34,7 @@ async function check(): Promise {
}
function cacheCatalog(): void {
- mkdirSync(join(dir, 'config/cache/source-catalog'), { recursive: true })
- writeFileSync(
- join(dir, `config/cache/source-catalog/${WORKSPACE_ID}.json`),
- readFileSync(FIXTURE_CATALOG, 'utf8')
- )
+ cacheCatalogText(join(dir, 'config'), WORKSPACE_ID, readFileSync(FIXTURE_CATALOG, 'utf8'))
}
function result(): Record {
@@ -51,10 +48,18 @@ function stdoutJson(): Record {
return JSON.parse(calls[0].join(' '))
}
-function stderr(): string {
- return vi
- .mocked(console.error)
- .mock.calls.map((call) => call.join(' '))
+/**
+ * The findings of the JSON result on stdout as a person reads them on stderr in a display format
+ * (`file:line:column severity CODE: message`, then its hint): with `--output json` they are
+ * printed only there.
+ */
+function findingLines(): string {
+ const diagnostics = (result().diagnostics ?? []) as Array>
+ return diagnostics
+ .map(
+ ({ file, line, column, severity, code, message, hint }) =>
+ `${file === undefined ? '' : `${file}:${line}:${column} `}${severity} ${code}: ${message}${hint === undefined ? '' : `\n hint: ${hint}`}`
+ )
.join('\n')
}
@@ -208,7 +213,11 @@ beforeEach(() => {
JSON.stringify({ workflowId: 'wf', workspaceId: WORKSPACE_ID, fingerprint: 'sf1:a' })
)
vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config'))
- contextMockFns.mockProfileFrom.mockReturnValue({ workspaceId: 'ws_other', output: 'json' })
+ contextMockFns.mockProfileFrom.mockReturnValue({
+ ...mockedProfile(),
+ workspaceId: 'ws_other',
+ output: 'json',
+ })
contextMockFns.mockRequest.mockReset()
vi.spyOn(console, 'log').mockImplementation(() => {})
vi.spyOn(console, 'error').mockImplementation(() => {})
@@ -242,7 +251,7 @@ describe('sim workflows source check', () => {
await check()
expect(result()).toMatchObject({ errors: 1 })
- expect(stderr()).toMatch(/workflow\.ts:6:\d+ error E_UNKNOWN_PARAM/)
+ expect(findingLines()).toMatch(/workflow\.ts:6:\d+ error E_UNKNOWN_PARAM/)
expect(process.exitCode).toBe(1)
})
@@ -269,6 +278,19 @@ describe('sim workflows source check', () => {
})
})
+ it('with --output json, prints each finding in the JSON only, not again as a frame on stderr', async () => {
+ cacheCatalog()
+ writeFileSync(
+ join(dir, 'workflow.ts'),
+ CLEAN.replace('{ body: { received: start.message } }', '{ bdy: { received: start.message } }')
+ )
+
+ await check()
+
+ expect(stdoutJson()).toMatchObject({ diagnostics: [{ code: 'E_UNKNOWN_PARAM' }] })
+ expect(vi.mocked(console.error)).not.toHaveBeenCalled()
+ })
+
it('leaves stored() to the server, which holds the stored values', async () => {
cacheCatalog()
writeFileSync(
@@ -287,7 +309,7 @@ describe('sim workflows source check', () => {
await check()
- expect(stderr()).not.toContain('E_STORED_VALUE_MISSING')
+ expect(findingLines()).not.toContain('E_STORED_VALUE_MISSING')
})
it('refuses without a cached catalog, naming the get that caches one', async () => {
@@ -299,6 +321,29 @@ describe('sim workflows source check', () => {
expect((error as SimApiError).message).toContain('sim workflows source get')
})
+ it('refuses a directory without workflow.ts before looking for a catalog, by its absolute path', async () => {
+ const error = await check().catch((caught: unknown) => caught)
+
+ expect((error as SimApiError).code).not.toBe('CATALOG_NOT_CACHED')
+ expect((error as SimApiError).message).toContain(`${dir} has no workflow.ts`)
+ })
+
+ it('refuses a --workspace naming another workspace than .sim/base.json, and only that', async () => {
+ cacheCatalog()
+ writeFileSync(join(dir, 'workflow.ts'), CLEAN)
+ contextMockFns.mockProfileFrom.mockReturnValue({
+ ...mockedProfile(),
+ workspaceId: 'ws_other',
+ output: 'json',
+ sources: { workspaceId: 'flag' },
+ })
+
+ const error = await check().catch((caught: unknown) => caught)
+
+ expect((error as SimApiError).code).toBe('SOURCE_OTHER_WORKSPACE')
+ expect((error as SimApiError).message).toContain(`was read from workspace ${WORKSPACE_ID}`)
+ })
+
it('points a workspace root at the workspace check', async () => {
cacheCatalog()
writeWorkspace()
@@ -335,8 +380,10 @@ describe('sim workflows source check', () => {
await checkIn(folder)
expect(result()).toMatchObject({ errors: 2 })
- expect(stderr()).toMatch(/workflow\.ts:6:\d+ error E_MAT_UNKNOWN_RESOURCE: No table "contact"/)
- expect(stderr()).toMatch(
+ expect(findingLines()).toMatch(
+ /workflow\.ts:6:\d+ error E_MAT_UNKNOWN_RESOURCE: No table "contact"/
+ )
+ expect(findingLines()).toMatch(
/error E_MAT_UNKNOWN_RESOURCE: No credential "ops-slak"[\s\S]*Keys: ops-slack/
)
expect(process.exitCode).toBe(1)
@@ -353,7 +400,7 @@ describe('sim workflows source check', () => {
await checkIn(folder)
expect(result()).toMatchObject({ errors: 0, warnings: 1 })
- expect(stderr()).toContain('W_ENV_UNKNOWN')
+ expect(findingLines()).toContain('W_ENV_UNKNOWN')
})
it('fails a row write naming a column or select option the table does not declare', async () => {
@@ -370,10 +417,10 @@ describe('sim workflows source check', () => {
await checkIn(folder)
- expect(stderr()).toMatch(
+ expect(findingLines()).toMatch(
/error W_TABLE_UNKNOWN_COLUMN: data writes emial[\s\S]*Columns: email, status/
)
- expect(stderr()).toMatch(
+ expect(findingLines()).toMatch(
/error W_UNKNOWN_OPTION: status of table "contacts" has no option done/
)
expect(process.exitCode).toBe(1)
@@ -384,7 +431,7 @@ describe('sim workflows source check', () => {
writeFileSync(join(dir, 'workflow.ts'), UNKNOWN_METHOD)
await check()
- expect(stderr()).toMatch(/error W_UNKNOWN_OPTION/)
+ expect(findingLines()).toMatch(/error W_UNKNOWN_OPTION/)
expect(process.exitCode).toBe(1)
process.exitCode = undefined
@@ -394,6 +441,71 @@ describe('sim workflows source check', () => {
expect(process.exitCode).toBeUndefined()
})
+ it('reports an untyped JSON reference only on a block the lock does not bind, and on all with --strict', async () => {
+ cacheCatalog()
+ writeFileSync(
+ join(dir, 'workflow.ts'),
+ CLEAN.replace(
+ "const done = response('Done', { body: { received: start.message } })",
+ "const call = api('Call', { url: 'https://x', method: 'POST', body: { n: start.missing.deep } })\n const done = response('Done', { body: { received: start.message } })"
+ ).replace('import { response,', 'import { api, response,')
+ )
+ const untyped = () =>
+ ((result().diagnostics ?? []) as Array<{ code: string }>).filter(
+ (finding) => finding.code === 'W_JSON_REF_UNTYPED'
+ )
+
+ await check()
+ expect(untyped()).toHaveLength(1)
+
+ writeLock(['start', 'done', 'call'])
+ await check()
+ expect(untyped()).toEqual([])
+
+ await checkIn(dir, '--strict')
+ expect(untyped()).toHaveLength(1)
+ })
+
+ it('fails on a dangling $ref in a new function file by default, and on any with --strict', async () => {
+ cacheCatalog()
+ mkdirSync(join(dir, 'functions'), { recursive: true })
+ writeFileSync(join(dir, 'functions/total.js'), 'return $ref.ordertotals.sum\n')
+ writeFileSync(
+ join(dir, 'workflow.ts'),
+ `// @sim-sdk 1
+import { codeFile, fn, trigger, workflow } from '@sim/workflow-sdk'
+
+export default workflow('W', {}, () => {
+ const start = trigger.start('Start', {})
+ const total = fn('Total', { code: codeFile('./functions/total.js') })
+ start.to(total)
+})
+`
+ )
+
+ await check()
+ expect(result().diagnostics).toContainEqual(
+ expect.objectContaining({
+ code: 'W_UNRESOLVED_REF',
+ severity: 'error',
+ file: 'functions/total.js',
+ line: 1,
+ column: 8,
+ })
+ )
+ expect(process.exitCode).toBe(1)
+
+ process.exitCode = undefined
+ writeLock(['start', 'total'])
+ await check()
+ expect(result()).toMatchObject({ errors: 0, warnings: 1 })
+ expect(process.exitCode).toBeUndefined()
+
+ await checkIn(dir, '--strict')
+ expect(result()).toMatchObject({ errors: 1 })
+ expect(process.exitCode).toBe(1)
+ })
+
it('fails on a bound block with --strict and on nothing with --no-strict', async () => {
cacheCatalog()
writeFileSync(join(dir, 'workflow.ts'), UNKNOWN_METHOD)
@@ -416,10 +528,12 @@ describe('sim workflows source check', () => {
writeLock(['start', 'l', 'inner', 'fallback', 'a', 'route'])
await check()
- expect(stderr()).toMatch(/warning W_ERROR_PORT_STORED/)
- expect(stderr()).toMatch(/warning W_EDGE_CYCLE/)
- expect(stderr()).toMatch(/warning W_BRANCH_UNWIRED/)
- expect(stderr()).toMatch(/keeps this error edge only if the stored workflow already holds it/)
+ expect(findingLines()).toMatch(/warning W_ERROR_PORT_STORED/)
+ expect(findingLines()).toMatch(/warning W_EDGE_CYCLE/)
+ expect(findingLines()).toMatch(/warning W_BRANCH_UNWIRED/)
+ expect(findingLines()).toMatch(
+ /keeps this error edge only if the stored workflow already holds it/
+ )
expect(result()).toMatchObject({ errors: 0 })
expect(process.exitCode).toBeUndefined()
@@ -434,8 +548,8 @@ describe('sim workflows source check', () => {
writeLock(['start', 'l', 'inner', 'fallback', 'a', 'route'])
await check()
- expect(stderr()).toMatch(/warning W_ERROR_PORT_STORED/)
- expect(stderr()).toMatch(/warning W_EDGE_CYCLE/)
+ expect(findingLines()).toMatch(/warning W_ERROR_PORT_STORED/)
+ expect(findingLines()).toMatch(/warning W_EDGE_CYCLE/)
expect(result()).toMatchObject({ errors: 0 })
expect(process.exitCode).toBeUndefined()
})
@@ -449,8 +563,8 @@ describe('sim workflows source check', () => {
writeLock(['start', 'l', 'inner', 'fallback', 'a', 'route'])
await check()
- expect(stderr()).toMatch(/error E_ERROR_PORT_UNSUPPORTED/)
- expect(stderr()).toMatch(/error E_EDGE_CYCLE/)
+ expect(findingLines()).toMatch(/error E_ERROR_PORT_UNSUPPORTED/)
+ expect(findingLines()).toMatch(/error E_EDGE_CYCLE/)
expect(process.exitCode).toBe(1)
})
@@ -460,8 +574,8 @@ describe('sim workflows source check', () => {
writeLock(['start', 'inner', 'fallback', 'route'])
await check()
- expect(stderr()).toMatch(/error E_ERROR_PORT_UNSUPPORTED/)
- expect(stderr()).toMatch(/error E_EDGE_CYCLE/)
+ expect(findingLines()).toMatch(/error E_ERROR_PORT_UNSUPPORTED/)
+ expect(findingLines()).toMatch(/error E_EDGE_CYCLE/)
expect(process.exitCode).toBe(1)
})
@@ -484,6 +598,5 @@ describe('sim workflows source check', () => {
expect(includes(join(dir, 'tsconfig.json'))).toEqual(
expect.arrayContaining(['.sim/sdk/blocks/slack_v2.d.ts', '.sim/sdk/blocks/gmail_v2.d.ts'])
)
- expect(() => readFileSync(join(dir, 'tsconfig.full.json'))).toThrow()
})
})
diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-check.ts b/packages/sim-cli/src/commands/protocol/workflow-source-check.ts
index 743b8593d81..c8bb5177c82 100644
--- a/packages/sim-cli/src/commands/protocol/workflow-source-check.ts
+++ b/packages/sim-cli/src/commands/protocol/workflow-source-check.ts
@@ -23,15 +23,24 @@ import { profileFrom } from '../../context'
import { setSoftExitCode } from '../../embed-context'
import { SimApiError } from '../../http/client'
import { printProtocolResult } from './result'
-import { readSourceJson, readSourceText, workspaceFolderOf } from './source-dir'
+import {
+ absoluteDirOption,
+ directoryWorkspace,
+ readSourceJson,
+ readSourceText,
+ readStarterWorkspace,
+ workspaceFolderOf,
+} from './source-dir'
import {
cachedCatalog,
+ cachedSourceCatalog,
type LocalDeclarations,
refreshTsconfigs,
refreshWorkspaceTypes,
} from './source-tree'
-import { printNewDiagnostics, resultDiagnostics } from './source-ui'
-import { readSourceFiles } from './workflow-source-apply'
+import { type Diagnostic, findingsOutput, verboseFlag } from './source-ui'
+import { readStoredEdges } from './stored-edges'
+import { readSourceFiles, requireEntry } from './workflow-source-apply'
import { WORKFLOW_TREE } from './workflow-source-get'
import {
kindFolders,
@@ -60,6 +69,7 @@ const STRICT_CODES: ReadonlySet = new Set([
'W_TABLE_UNKNOWN_COLUMN',
'W_CHILD_INPUT_UNKNOWN',
'W_BRANCH_UNWIRED',
+ 'W_UNRESOLVED_REF',
])
/**
@@ -70,6 +80,14 @@ const STRICT_CODES: ReadonlySet = new Set([
*/
const STORED_EXEMPT_CODES: ReadonlySet = new Set(['W_BRANCH_UNWIRED'])
+/**
+ * Warnings a check reports only on a block the lock does not bind (one added since the get), and
+ * on every block with `--strict`: printed source holds them on blocks nobody edited, and the
+ * server, which compares with the stored workflow, already reports them only for an edited field.
+ * `W_JSON_REF_UNTYPED`: a bare reference nothing types, in a JSON field.
+ */
+const EDIT_SCOPED_CODES: ReadonlySet = new Set(['W_JSON_REF_UNTYPED'])
+
/**
* Which {@link STRICT_CODES} warnings fail a check: `new` those on a block the lock does not bind
* (one added since the get, or every block of a workflow that has no lock yet), `all` every one,
@@ -85,7 +103,7 @@ type StrictMode = 'new' | 'all' | 'off'
const KEPT_AS_WRITTEN = /[;,] (?:it|the value|the text) is kept as written\.$/
/** One finding of a check, as the write commands print theirs. */
-export type CheckFinding = Parameters[0][number]
+export type CheckFinding = Diagnostic
/** The `--strict` / `--no-strict` pair: absent selects the default, {@link StrictMode} `new`. */
export interface StrictOptions {
@@ -108,7 +126,8 @@ does not bind (a block added since the get, or every block of a new workflow):
${[...STRICT_CODES].map((code) => ` ${code}`).join(',\n')}.
--strict fails on them in every block, except ${[...STORED_EXEMPT_CODES].join(', ')} on a block the
lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict never
-fails on a warning.`
+fails on a warning. ${[...EDIT_SCOPED_CODES].join(', ')} is reported only on blocks the lock does not
+bind, and with --strict on every block.`
/** Adds the `--strict` / `--no-strict` pair to a check command. */
export function strictFlags(command: Command): Command {
@@ -346,16 +365,23 @@ function declarationFindings(
}
/**
- * Compiles the workflow source directory `dir` against `catalog`, checking env names against the
- * names `secrets.ts` declares when `declarations` hold them.
+ * Compiles the workflow source directory `dir` against the catalog its base names when that one is
+ * cached (a workflow holding a type the workspace catalog lacks was printed with its own), else
+ * `catalog`, checking env names against the names `secrets.ts` declares when `declarations` hold
+ * them.
*/
async function compileWorkflowDir(
dir: string,
- catalog: BlockCatalog,
+ workspaceCatalog: BlockCatalog,
declarations?: WorkspaceDeclarations
) {
const { lock } = parseWorkflowLock(await readSourceJson(dir, SOURCE_LOCK_FILE))
- const files = await readSourceFiles(dir, lock ?? null)
+ const [files, storedEdges, printedWith] = await Promise.all([
+ readSourceFiles(dir, lock ?? null),
+ readStoredEdges(dir),
+ cachedSourceCatalog(dir),
+ ])
+ const catalog = printedWith ?? workspaceCatalog
const envNames = declarations?.envNames
const compiled = compileWorkflow({
entry: { path: SOURCE_ENTRY_FILE, text: files[SOURCE_ENTRY_FILE] },
@@ -364,6 +390,7 @@ async function compileWorkflowDir(
sdkMajor: catalog.sdkMajor,
knownKeys: new Set(Object.keys(lock?.blocks ?? {})),
storedUnread: lock ?? undefined,
+ ...(storedEdges ? { storedUnreadEdges: storedEdges } : {}),
...(envNames ? { envNames, envDeclaredIn: KIND_DIRS.secret } : {}),
})
return { lock, files, compiled }
@@ -457,8 +484,11 @@ export async function checkWorkflowDir(
!(STORED_EXEMPT_CODES.has(code) && bound(blockKey)) &&
(options.strict === 'all' || (options.strict === 'new' && !bound(blockKey)))
const prefix = options.prefix ?? ''
+ const reported = (code: string, blockKey: string | undefined) =>
+ !SERVER_ONLY_CODES.has(code) &&
+ !(EDIT_SCOPED_CODES.has(code) && options.strict !== 'all' && bound(blockKey))
const findings = [
- ...compiled.diagnostics.filter((diagnostic) => !SERVER_ONLY_CODES.has(diagnostic.code)),
+ ...compiled.diagnostics.filter((diagnostic) => reported(diagnostic.code, diagnostic.blockKey)),
...declared,
].map(({ code, severity, message, hint, frame, span, blockKey }): CheckFinding => {
const raised = severity === 'warning' && fails(code, blockKey)
@@ -474,13 +504,18 @@ export async function checkWorkflowDir(
return { findings, entry: entryText }
}
-/** Prints a check's findings and result, and exits 1 on any error. */
+/**
+ * Prints a check's findings and result, and exits 1 on any error. `verbose` keeps info findings in
+ * a machine result.
+ */
export function reportCheck(
output: Parameters[0],
findings: CheckFinding[],
- result: Record
+ result: Record,
+ verbose = false
): void {
- printNewDiagnostics(findings, new Set())
+ const shown = findingsOutput(output, verbose)
+ shown.print(findings)
const errors = findings.filter((finding) => finding.severity === 'error').length
if (errors > 0) setSoftExitCode(1)
printProtocolResult(output, {
@@ -488,13 +523,16 @@ export function reportCheck(
errors,
warnings: findings.filter((finding) => finding.severity === 'warning').length,
...result,
- ...resultDiagnostics(output, findings),
+ ...shown.result(findings),
})
}
-/** The cached catalog of `workspaceId`, or a refusal naming `get`, the command that caches one. */
-export function requireCachedCatalog(workspaceId: string, get: string): BlockCatalog {
- const catalog = cachedCatalog(workspaceId)
+/** The catalog a cache read found, or a refusal naming `get`, the command that caches one. */
+function cachedOrRefuse(
+ catalog: BlockCatalog | null,
+ workspaceId: string,
+ get: string
+): BlockCatalog {
if (catalog) return catalog
throw new SimApiError(
`No block catalog is cached for workspace ${workspaceId}. Run ${get} once, then check again.`,
@@ -503,75 +541,97 @@ export function requireCachedCatalog(workspaceId: string, get: string): BlockCat
)
}
+/** The cached catalog of `workspaceId`, or a refusal naming `get`, the command that caches one. */
+export function requireCachedCatalog(workspaceId: string, get: string): BlockCatalog {
+ return cachedOrRefuse(cachedCatalog(workspaceId), workspaceId, get)
+}
+
/**
* `sim workflows source check [--dir ]`: compiles a source directory locally with the bundled
* compiler and the cached catalog, so an agent's edit loop gets syntax and builder findings in
* milliseconds and without the network. The dry run stays the authority.
*/
export function attachWorkflowSourceCheck(source: Command): void {
- strictFlags(
- source
- .command('check')
- .option('--dir ', 'Source directory: workflow.ts, functions/* and .sim/', '.')
+ verboseFlag(
+ strictFlags(
+ absoluteDirOption(source.command('check')).option(
+ '--dir ',
+ 'Source directory: workflow.ts, functions/* and .sim/',
+ '.'
+ )
+ )
)
.allowExcessArguments(false)
.description('Compile a workflow source directory locally, without the server')
.addHelpText('after', CHECK_HELP)
- .action(async (options: { dir: string } & StrictOptions, command: Command) => {
- const profile = profileFrom(command)
- const base = toRecord(await readSourceJson(options.dir, SOURCE_BASE_FILE))
- if (
- typeof base.workspaceId === 'string' &&
- base.workflowId === undefined &&
- (await readSourceText(options.dir, SOURCE_ENTRY_FILE)) === null
- ) {
- throw new SimApiError(
- `${options.dir} is a workspace source directory: check it with \`sim workspaces source check --dir ${options.dir}\`.`,
- 0
- )
- }
- const workspace = await workspaceFolderOf(options.dir)
- const workspaceId =
- typeof base.workspaceId === 'string'
- ? base.workspaceId
- : (workspace?.workspaceId ?? profile.workspaceId)
- if (!workspaceId) {
- throw new SimApiError(
- `${options.dir} has no ${SOURCE_BASE_FILE} and no workspace is selected: pass --workspace .`,
- 0
- )
- }
- const catalog = requireCachedCatalog(
- workspaceId,
- workspace
- ? `\`sim workspaces source get --dir ${workspace.root}\``
- : `\`sim workflows source get --dir ${options.dir}\` (or \`create\`)`
- )
- const declarations = workspace
- ? {
- ...(await readWorkspaceDeclarations(workspace.root)),
- workflowInputs: await workspaceWorkflowInputs(workspace.root, catalog),
- }
- : undefined
- const { findings, entry } = await checkWorkflowDir(options.dir, catalog, {
- strict: strictMode(options),
- ...(declarations ? { declarations } : {}),
- })
- if (workspace && declarations) {
- await refreshWorkspaceTypes(
- workspace.root,
- withWorkflowInputs(declarations.local, declarations.workflowInputs)
- )
- await refreshWorkspaceTsconfigs(workspace.root, workspaceId)
- } else {
- await refreshTsconfigs(
+ .action(
+ async (options: { dir: string; verbose?: boolean } & StrictOptions, command: Command) => {
+ const profile = profileFrom(command)
+ const base = toRecord(await readSourceJson(options.dir, SOURCE_BASE_FILE))
+ if (
+ typeof base.workspaceId === 'string' &&
+ base.workflowId === undefined &&
+ (await readSourceText(options.dir, SOURCE_ENTRY_FILE)) === null
+ ) {
+ throw new SimApiError(
+ `${options.dir} is a workspace source directory: check it with \`sim workspaces source check --dir ${options.dir}\`.`,
+ 0
+ )
+ }
+ await requireEntry(options.dir)
+ const workspace = await workspaceFolderOf(options.dir)
+ const recorded =
+ typeof base.workspaceId === 'string'
+ ? base.workspaceId
+ : (workspace?.workspaceId ??
+ (profile.sources.workspaceId === 'flag'
+ ? null
+ : await readStarterWorkspace(options.dir)))
+ const workspaceId = directoryWorkspace(
options.dir,
+ recorded,
+ profile,
+ (from) => `Run it without --workspace to check it against the catalog of ${from}.`
+ )
+ if (!workspaceId) {
+ throw new SimApiError(
+ `${options.dir} has no ${SOURCE_BASE_FILE} and no workspace is selected: pass --workspace .`,
+ 0
+ )
+ }
+ const catalog = cachedOrRefuse(
+ (await cachedSourceCatalog(options.dir)) ?? cachedCatalog(workspaceId),
workspaceId,
- WORKFLOW_TREE,
- { sources: [entry] },
- { incremental: true }
+ workspace
+ ? `\`sim workspaces source get --dir ${workspace.root}\``
+ : `\`sim workflows source get --dir ${options.dir}\` (or \`create\`)`
)
+ const declarations = workspace
+ ? {
+ ...(await readWorkspaceDeclarations(workspace.root)),
+ workflowInputs: await workspaceWorkflowInputs(workspace.root, catalog),
+ }
+ : undefined
+ const { findings, entry } = await checkWorkflowDir(options.dir, catalog, {
+ strict: strictMode(options),
+ ...(declarations ? { declarations } : {}),
+ })
+ if (workspace && declarations) {
+ await refreshWorkspaceTypes(
+ workspace.root,
+ withWorkflowInputs(declarations.local, declarations.workflowInputs)
+ )
+ await refreshWorkspaceTsconfigs(workspace.root, workspaceId)
+ } else {
+ await refreshTsconfigs(
+ options.dir,
+ workspaceId,
+ WORKFLOW_TREE,
+ { sources: [entry] },
+ { incremental: true }
+ )
+ }
+ reportCheck(profile.output, findings, { catalogHash: catalog.hash }, options.verbose)
}
- reportCheck(profile.output, findings, { catalogHash: catalog.hash })
- })
+ )
}
diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-get.test.ts b/packages/sim-cli/src/commands/protocol/workflow-source-get.test.ts
index 1ff6b340033..55ab1fb1b47 100644
--- a/packages/sim-cli/src/commands/protocol/workflow-source-get.test.ts
+++ b/packages/sim-cli/src/commands/protocol/workflow-source-get.test.ts
@@ -1,13 +1,16 @@
+import { execFileSync } from 'node:child_process'
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
-import { join } from 'node:path'
+import { dirname, join } from 'node:path'
import { COOKBOOK_EXAMPLES } from '@sim/workflow-sdk/codegen'
import { Command } from 'commander'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { SimApiError } from '../../http/client'
import { buildGeneratedCommands } from '../../runtime/build'
-import { contextMockFns, contextMockState } from '../../test/context-mock'
+import { contextMockFns, contextMockState, mockedProfile } from '../../test/context-mock'
import { digest } from './source-dir'
+import { PRIMER_MAX_BYTES, sourcePrimer } from './source-primer'
+import { attachWorkflowSourceCheck } from './workflow-source-check'
import { attachWorkflowSourceGet } from './workflow-source-get'
vi.mock('../../context', async () => (await import('../../test/context-mock')).contextMock)
@@ -64,6 +67,21 @@ function program(): Command {
return root
}
+/** Type-checks the output directory with one of its tsconfig files; returns tsc's output, or ''. */
+function typecheck(project: string): string {
+ try {
+ execFileSync(
+ join(__dirname, '../../../../../node_modules/.bin/tsc'),
+ ['--noEmit', '-p', join(outDir, project)],
+ { encoding: 'utf8', stdio: 'pipe' }
+ )
+ return ''
+ } catch (error) {
+ const failure = error as { stdout?: string; stderr?: string }
+ return `${failure.stdout ?? ''}${failure.stderr ?? ''}`
+ }
+}
+
/** The `include` list of a tsconfig the CLI wrote into the output directory. */
function tsconfigInclude(project: string): string[] {
const [, ...json] = readFileSync(join(outDir, project), 'utf8').split('\n')
@@ -121,7 +139,7 @@ afterEach(() => {
})
describe('sim workflows source get --dir', () => {
- it('writes the source, lock, base, an agent guide and declarations for every block in the workspace catalog', async () => {
+ it('writes the source, lock, base, an agent guide and the declarations of the blocks it uses', async () => {
serve({
[SOURCE_PATH]: () =>
source({ 'workflow.ts': '// @sim-sdk 1\n', 'functions/score.js': 'return 1\n' }),
@@ -136,7 +154,7 @@ describe('sim workflows source get --dir', () => {
)
expect(existsSync(join(outDir, '.sim/sdk/core.d.ts'))).toBe(true)
expect(existsSync(join(outDir, '.sim/sdk/blocks/agent.d.ts'))).toBe(true)
- expect(existsSync(join(outDir, '.sim/sdk/blocks/slack_v2.d.ts'))).toBe(true)
+ expect(existsSync(join(outDir, '.sim/sdk/blocks/slack_v2.d.ts'))).toBe(false)
const workspaceTypes = readFileSync(join(outDir, '.sim/sdk/workspace.d.ts'), 'utf8')
expect(workspaceTypes).toContain("'docs': true")
expect(workspaceTypes).toContain("interface WorkspaceSkills {\n 'triage': true")
@@ -145,6 +163,67 @@ describe('sim workflows source get --dir', () => {
expect(callsTo(CATALOG_PATH)[0][1].query).toBeUndefined()
})
+ it('prints the primer, which opens the guide, with a snippet for each builder the source calls and no other', async () => {
+ serve({
+ [SOURCE_PATH]: () =>
+ source({ 'workflow.ts': "// @sim-sdk 1\nconst mail = gmail.sendEmail('Mail', {})\n" }),
+ })
+
+ await get('--dir', outDir)
+
+ const { primer } = JSON.parse(vi.mocked(console.log).mock.calls.at(-1)?.join(' ') ?? '{}')
+ expect(readFileSync(join(outDir, '.sim/AGENTS.md'), 'utf8').startsWith(primer)).toBe(true)
+ expect(primer).toContain('- `gmail.sendEmail` (Gmail: Send Email)')
+ expect(primer).not.toContain('`slack.sendMessage`')
+ })
+
+ it('keeps the primer without snippets within its byte budget for both directory kinds', () => {
+ for (const kind of ['workflow', 'workspace'] as const) {
+ expect(Buffer.byteLength(sourcePrimer(kind))).toBeLessThanOrEqual(PRIMER_MAX_BYTES)
+ }
+ })
+
+ it('type-checks a builder of a block the directory does not hold: through the cache at once, and locally after a check', async () => {
+ const workflow = [
+ '// @sim-sdk 1',
+ "import { trigger, workflow } from '@sim/workflow-sdk'",
+ '',
+ "export default workflow('W', {}, () => {",
+ " const start = trigger.start('Start', {})",
+ '})',
+ '',
+ ].join('\n')
+ serve({ [SOURCE_PATH]: () => source({ 'workflow.ts': workflow }, []) })
+ await get('--dir', outDir)
+ expect(existsSync(join(outDir, '.sim/sdk/blocks/gmail_v2.d.ts'))).toBe(false)
+
+ writeFileSync(
+ join(outDir, 'workflow.ts'),
+ workflow
+ .replace('{ trigger, workflow }', '{ gmail, trigger, workflow }')
+ .replace(
+ " const start = trigger.start('Start', {})\n",
+ " const start = trigger.start('Start', {})\n const mail = gmail.sendEmail('Mail', { to: 'a@b.c', subject: 's', body: 'b', bogusParam: 1 })\n start.to(mail)\n"
+ )
+ )
+ const unknownParam = /'bogusParam' does not exist in type 'GmailSendEmailParams'/
+ expect(typecheck('tsconfig.full.json')).toMatch(unknownParam)
+
+ contextMockFns.mockProfileFrom.mockReturnValue({
+ ...mockedProfile(),
+ workspaceId: WORKSPACE_ID,
+ output: 'json',
+ })
+ const check = new Command('source')
+ attachWorkflowSourceCheck(check)
+ check.commands.forEach((command) => command.exitOverride())
+ await check.parseAsync(['node', 'source', 'check', '--dir', outDir])
+ process.exitCode = undefined
+
+ expect(existsSync(join(outDir, '.sim/sdk/blocks/gmail_v2.d.ts'))).toBe(true)
+ expect(typecheck('tsconfig.json')).toMatch(unknownParam)
+ })
+
it('writes the example workflows beside the guide, outside what the tsconfig files type-check', async () => {
serve()
@@ -160,7 +239,7 @@ describe('sim workflows source get --dir', () => {
}
})
- it('type-checks only the blocks the source uses by default, and every current block in tsconfig.full.json', async () => {
+ it('type-checks only the blocks the source uses by default, and every current block through the cache in tsconfig.full.json', async () => {
serve({
[SOURCE_PATH]: () =>
source(
@@ -193,20 +272,26 @@ describe('sim workflows source get --dir', () => {
)
expect(include).not.toContain('.sim/sdk/blocks/google_sheets_v2.d.ts')
expect(include.every((path) => existsSync(join(outDir, path)))).toBe(true)
- expect(tsconfigInclude('tsconfig.full.json')).toEqual([
- 'workflow.ts',
- '.sim/sdk/index.d.ts',
- '.sim/sdk/workspace.d.ts',
- ])
+ const [entry, fullIndex, workspace] = tsconfigInclude('tsconfig.full.json')
+ expect([entry, workspace]).toEqual(['workflow.ts', '.sim/sdk/workspace.d.ts'])
+ expect(fullIndex.startsWith(join(rootDir, 'config/cache/source-catalog', CATALOG.hash))).toBe(
+ true
+ )
+ expect(readFileSync(fullIndex, 'utf8')).toContain("import './blocks/google_sheets_v2'")
+ expect(existsSync(join(dirname(fullIndex), 'blocks/google_sheets_v2.d.ts'))).toBe(true)
})
- it('writes one declaration file per block, and the builder index', async () => {
+ it('lists the builders of the blocks it holds in the builder index, and every builder in the cached one', async () => {
serve()
await get('--dir', outDir)
expect(existsSync(join(outDir, '.sim/sdk/operations'))).toBe(false)
- expect(readFileSync(join(outDir, '.sim/sdk/INDEX.md'), 'utf8')).toContain(
+ const local = readFileSync(join(outDir, '.sim/sdk/INDEX.md'), 'utf8')
+ expect(local).toContain('- `agent` — Agent → `blocks/agent.d.ts` (`AgentParams`)')
+ expect(local).not.toContain('`gmail.sendEmail`')
+ const [, fullIndex] = tsconfigInclude('tsconfig.full.json')
+ expect(readFileSync(join(dirname(fullIndex), 'INDEX.md'), 'utf8')).toContain(
'- `gmail.sendEmail` — Gmail: Send Email → `blocks/gmail_v2.d.ts` (`GmailSendEmailParams`)'
)
})
diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-get.ts b/packages/sim-cli/src/commands/protocol/workflow-source-get.ts
index 72cb9ad3022..b4d719da980 100644
--- a/packages/sim-cli/src/commands/protocol/workflow-source-get.ts
+++ b/packages/sim-cli/src/commands/protocol/workflow-source-get.ts
@@ -11,24 +11,27 @@ import type { GetWorkflowSourceResponse } from '../../generated/v2-api'
import { SimApiError } from '../../http/client'
import { callsOperations, type OperationClient } from '../../runtime/called-operations'
import { buildRequest } from '../../runtime/request'
-import { printProtocolResult } from './result'
import {
+ absoluteDirOption,
assertPrintedSourcePaths,
locallyEditedFiles,
readPreviousLock,
readSourceJson,
SOURCE_SDK_DIR as SDK_DIR,
SOURCE_EXAMPLES_DIR,
+ SOURCE_STARTER_FILE,
updateSourceDir,
WORKSPACE_FOLDER_BUSY_HELP,
withWorkspaceFolderLock,
writeBack,
} from './source-dir'
+import { sourceExitCodesTable } from './source-exit-codes'
import {
assertSdkMajor,
finishTree,
loadCatalog,
loadWorkspaceTypes,
+ printTreeResult,
refreshTsconfigs,
type SourceTree,
type SourceTreeProfile,
@@ -39,6 +42,7 @@ import {
type WorkspaceTypes,
withSourceExitCode,
} from './source-tree'
+import { storedEdgesWrite } from './stored-edges'
const DIR_FLAG = 'dir'
const FORCE_FLAG = 'force'
@@ -66,31 +70,29 @@ function workflowTsconfig(sdkFiles: readonly string[]): string {
)}\n`
}
-/** The guide an agent working in the directory reads first; the docs page has the rest. */
-const AGENTS_GUIDE = `# Sim workflow source
-
-This directory is a Sim workflow written as TypeScript. The server compiles \`${SOURCE_ENTRY_FILE}\` statically; nothing in it runs. The workflow id is \`workflowId\` in \`${SOURCE_BASE_FILE}\`; every command below reads it from there.
-
-## Start here
+/**
+ * The reference below the primer in `.sim/AGENTS.md`, by task: the edit loop in detail, a new or
+ * forked workflow, the file shape, lookups, the rules and every exit code. The docs page has the
+ * rest.
+ */
+const AGENTS_GUIDE = `# Reference
-1. Read the examples in \`${SOURCE_EXAMPLES_DIR}/\` before writing: \`cookbook.ts\` (agent with structured output, condition, function, table query and insert, knowledge search, Slack, human in the loop, loop, response), \`integrations.ts\` (webhook trigger, weekday cron schedule, HTTP request with headers, table update by row id, Gmail, an error path) and \`agents.ts\` (router, agent tools: custom tool, MCP tool, integration operation, child workflow; run a child workflow; parallel). Their resource keys are examples; this workspace's are in \`${WORKSPACE_TYPES_FILE}\`.
-2. For any other block, find its builder in \`${SDK_DIR}/${SDK_GUIDE_FILE}\` (one line per builder: \`grep -i "gmail" ${SDK_DIR}/${SDK_GUIDE_FILE}\`). It names the builder, the file and the params type; read just that type: \`grep -n -A40 "export type GmailSendEmailParams" ${SDK_DIR}/blocks/gmail_v2.d.ts\`. Builders are named for the service, not the block type: \`gmail_v2\` is \`gmail.*\`, \`workflow_input\` is \`runWorkflow\`, the HTTP request block is \`api\`, and \`legacy.*\` builders are old versions not to write.
-3. Do not read \`${SDK_DIR}/blocks/*.d.ts\` or \`${SDK_DIR}/models.d.ts\` whole: they are large. \`grep\` them for a type or a model id.
+The primer above is enough to edit; read a section here when a step needs it.
-## Edit loop (run from this directory)
+## Edit loop in detail
-1. Edit \`${SOURCE_ENTRY_FILE}\` (and \`functions/*\`).
-2. \`sim workflows source check --dir .\`: compiles offline with the server's compiler, in milliseconds. It catches what \`tsc\` cannot (duplicate names, cycles, literal secrets), and fails (exit 1) on a missing required field, an unknown option or output path, a wrong value kind, an out-of-range value, an unwired condition or router branch or a literal secret in a block you added (\`--strict\`: in every block). It also adds the blocks the source now uses to \`tsconfig.json\`. Fix and rerun until it exits 0.
-3. \`${typecheckCommand()}\` type-checks with the TypeScript release the declarations are built for (the directory has no \`package.json\`; plain \`bunx tsc\` would run whatever version it finds). \`tsconfig.json\` includes only the blocks the source used at the last check, get or apply, which keeps it fast: a builder from another block fails with \`has no exported member\` until you run step 2 (or check with \`${typecheckCommand(TSCONFIG_FULL_FILE)}\`, every current block). The dry run is the authority: a printed line you did not edit that \`tsc\` flags and the dry run accepts is fine, so leave it as it is.
-4. \`sim workflows source status --dir .\` before applying: the files you changed (\`sim workflows source diff --dir .\` shows the edits), whether the workflow changed on the server since the get, and the command to run next. Exit 0 nothing changed, 11 only your edits, 12 the server changed it since (\`--offline\`: skip the server). Then \`sim workflows source apply --dir . --dry-run\`: fix every error it prints, rerun until it exits 0.
-5. \`sim workflows source apply --dir . --yes\`: saves the draft (never deploys). It then rewrites \`${SOURCE_ENTRY_FILE}\` in canonical form (order, quoting, defaults left out), so read the file again before the next edit rather than editing from memory.
-6. Exit 5 (\`SOURCE_STALE\`, HTTP 409, or \`E_APPLY_STALE\` on a dry run): the workflow changed and nothing was saved. Run \`sim workflows source get --dir \`, reapply your edit there, dry-run, apply. A get into this directory refuses to overwrite your unapplied edits unless you pass \`--force\`. Exit 5 right after an apply that ended without an answer (a timeout, a dropped connection) means the draft changed, by that apply or by someone else: get into a fresh directory and compare it with this one. Only when it holds your edit, run \`sim workflows source get --dir . --force\`; otherwise reapply your edit in the fresh directory.
+1. Edit \`${SOURCE_ENTRY_FILE}\` (and \`functions/*\`). \`${SOURCE_EXAMPLES_DIR}/\` holds correct examples to copy from: \`cookbook.ts\` (agent with structured output, condition, function, table query and insert, knowledge search, Slack, human in the loop, loop, response), \`integrations.ts\` (webhook trigger, weekday cron schedule, HTTP request with headers, table update by row id, Gmail, an error path) and \`agents.ts\` (router, agent tools: custom tool, MCP tool, integration operation, child workflow; run a child workflow; parallel). Their resource keys are examples; this workspace's are in \`${WORKSPACE_TYPES_FILE}\`.
+2. \`sim workflows source check --dir .\`: compiles offline with the server's compiler, in milliseconds. It catches what \`tsc\` cannot (duplicate names, cycles, literal secrets), and fails (exit 1) on a missing required field, an unknown option or output path, a wrong value kind, an out-of-range value, an unwired condition or router branch or a literal secret in a block you added (\`--strict\`: in every block). It also adds the blocks the source now uses to \`tsconfig.json\`.
+3. \`${typecheckCommand()}\`: about a second; catches what step 2 and the dry run do not (such as a reference to a start input the trigger does not declare). It runs the TypeScript release the declarations are built for (the directory has no \`package.json\`; plain \`bunx tsc\` would run whatever version it finds). \`tsconfig.json\` includes only the blocks the source used at the last check, get or apply: a builder from another block fails with \`has no exported member\` until you run step 2 (or use \`${typecheckCommand(TSCONFIG_FULL_FILE)}\`, every current block). The dry run is the authority: a printed line you did not edit that \`tsc\` flags and the dry run accepts is fine.
+4. \`sim workflows source status --dir .\`: the files you changed (\`sim workflows source diff --dir .\` shows the edits), whether the workflow changed on the server since the get, and the command to run next (\`--offline\`: skip the server). Then \`sim workflows source apply --dir . --dry-run\` until it exits 0.
+5. \`sim workflows source apply --dir . --yes\`: saves the draft, refusing errors, block deletions without \`--allow-delete\` and a stale directory, so it needs no separate dry run (without \`--yes\` and a terminal it prints the plan and exits 14). \`--deploy\` also deploys it; without it, a deployed workflow's next \`sim workflows deploy \` is printed. It then rewrites \`${SOURCE_ENTRY_FILE}\` in canonical form (order, quoting, defaults left out).
+6. Exit 5 (\`SOURCE_STALE\`, HTTP 409, or \`E_APPLY_STALE\` on a dry run): the workflow changed and nothing was saved. Run \`sim workflows source get --dir \`, reapply your edit there, dry-run, apply. A get into this directory refuses to overwrite your unapplied edits unless you pass \`--force\`. Exit 5 right after an apply that ended without an answer (a timeout, a dropped connection) means the draft changed, by that apply or by someone else: get into a fresh directory and compare. Only when it holds your edit, run \`sim workflows source get --dir . --force\`; otherwise reapply your edit in the fresh directory.
-An apply that deletes blocks (a declaration removed from \`${SOURCE_ENTRY_FILE}\`) is refused with \`E_APPLY_BLOCK_DELETE\` naming them; pass \`--allow-delete\` when the deletion is intended. Editing the \`workflow('…')\` name or description renames or redescribes the workflow; a name another workflow in its folder holds is refused with \`E_NAME_TAKEN\`.
+Editing the \`workflow('…')\` name or description renames or redescribes the workflow; a name another workflow in its folder holds is refused with \`E_NAME_TAKEN\`. Lines starting \`note:\` explain something the write did differently than the source says; read them.
-Lines starting \`note:\` explain something the write did differently than the source says; read them.
+## A new workflow, or a fork
-A new workflow: \`sim workflows source create --dir \` (without \`--dry-run\` or \`--yes\`) on a directory without \`${SOURCE_ENTRY_FILE}\` writes a starter \`${SOURCE_ENTRY_FILE}\`, these types, \`tsconfig.json\` and this guide, and creates nothing. Until the workflow exists there is no \`${SOURCE_BASE_FILE}\`, so \`status\`, \`diff\` and \`apply\` refuse the directory: edit the starter, run step 2 and 3, then \`sim workflows source create --dir --dry-run\` and \`sim workflows source create --dir --yes\`: it prints the new workflow id and writes the lock files. Exit 1 with \`SOURCE_NAME_TAKEN\` means another workflow has that name: pass \`--name\`. To fork a workflow, copy its directory, delete the copy's \`.sim/\` and run \`sim workflows source create --dir --name --yes\`.
+\`sim workflows source create --dir --workspace