diff --git a/.claude/rules/sim-workflow-code.md b/.claude/rules/sim-workflow-code.md index d513811c96a..752c3d67731 100644 --- a/.claude/rules/sim-workflow-code.md +++ b/.claude/rules/sim-workflow-code.md @@ -29,8 +29,9 @@ workspace directory). | `packages/workflow-types` | Shared workflow, block and resource-kind types (`WORKFLOW_RESOURCE_KINDS`) | | `packages/workflow-code` | Pure engines for one workflow: catalog types, canonicalize, print, compile, materialize, lock, diagnostics | | `packages/workspace-code` | Pure workspace half: `KIND_DIRS`, `APPLY_ORDER`, one codec per kind (`src/kinds/`), `.sim/base.json`, `rf1:` fingerprints, limits | +| `packages/table-cells` | Pure table cell rules (coerce, validate, currency/date/ttl parsing) shared by the table codec's offline row check and the server's writes | | `packages/workflow-sdk` | Authoring declarations (`src/core.d.ts`, `src/workspace/*.d.ts`), type codegen from the catalog (`src/codegen/`), the cookbook | -| `packages/sim-cli/src/commands/protocol/` | `workflow-source-*` and `workspace-source-*` commands; `source-dir.ts`, `source-tree.ts`, `source-ui.ts` are shared | +| `packages/sim-cli/src/commands/protocol/` | `workflow-source-*` and `workspace-source-*` commands, which never import each other's command modules; `source-dir/`, `workflow-dir.ts`, `source-check.ts`, `source-write.ts`, `source-tree.ts`, `source-ui.ts` and `workspace-source-dir.ts` (the workspace directory reader a workflow check also reads declarations through) are shared | | `apps/sim/lib/workflows/code/` | Server glue: catalog projection from the registry (`catalog/`), source-apply gates (`apply/`), stored canonical form, the feature flag, the proof corpus (`corpus/`) | | `apps/sim/lib/workflows/application/*source*` | Workflow source use cases: read, apply, create, catalog, resources | | `apps/sim/lib/workspace-source/` | Workspace source use cases (`application/`), one adapter per kind (`application/kinds//`), route policies (`api/`) | @@ -38,7 +39,7 @@ workspace directory). | `apps/sim/app/workspace/**/panel/components/code/` | The read-only Code tab | Dependencies point one way: `packages/workflow-types` → `packages/workflow-code` → -`packages/workspace-code` and `packages/workflow-sdk` → the CLI and `apps/sim`. `check:boundaries` keeps both engines pure (no +`packages/workspace-code` and `packages/workflow-sdk` → the CLI and `apps/sim`. `check:boundaries` keeps both engines and `packages/table-cells` pure (no Node built-ins, `eval`, `globalThis`, dynamic import): they run in the browser, the CLI and the server unchanged. @@ -151,7 +152,7 @@ satisfy. 4. If workflows name it by key: `WORKFLOW_RESOURCE_KINDS`, the live loader in `lib/workspaces/resource-keys.ts`, the manifest in `read-source-resources.ts` and `codegen/workspace.ts`, and `PENDABLE_REFERENCES` in - `lib/workflows/application/source-evaluate.ts`. + `lib/workflows/application/source-pipeline/evaluate/finalize.ts`. 5. SDK: `packages/workflow-sdk/src/workspace/.d.ts` plus `index.d.ts`, a cookbook example under `cookbook/workspace/`, then regenerate the embedded text. 6. CLI (`workspace-source-dir.ts`): `workspaceTsconfig` includes, the `AGENTS_GUIDE` section and diff --git a/.cursor/rules/sim-workflow-code.mdc b/.cursor/rules/sim-workflow-code.mdc index 02c9437d166..aaf39757082 100644 --- a/.cursor/rules/sim-workflow-code.mdc +++ b/.cursor/rules/sim-workflow-code.mdc @@ -19,8 +19,9 @@ workspace directory). | `packages/workflow-types` | Shared workflow, block and resource-kind types (`WORKFLOW_RESOURCE_KINDS`) | | `packages/workflow-code` | Pure engines for one workflow: catalog types, canonicalize, print, compile, materialize, lock, diagnostics | | `packages/workspace-code` | Pure workspace half: `KIND_DIRS`, `APPLY_ORDER`, one codec per kind (`src/kinds/`), `.sim/base.json`, `rf1:` fingerprints, limits | +| `packages/table-cells` | Pure table cell rules (coerce, validate, currency/date/ttl parsing) shared by the table codec's offline row check and the server's writes | | `packages/workflow-sdk` | Authoring declarations (`src/core.d.ts`, `src/workspace/*.d.ts`), type codegen from the catalog (`src/codegen/`), the cookbook | -| `packages/sim-cli/src/commands/protocol/` | `workflow-source-*` and `workspace-source-*` commands; `source-dir.ts`, `source-tree.ts`, `source-ui.ts` are shared | +| `packages/sim-cli/src/commands/protocol/` | `workflow-source-*` and `workspace-source-*` commands, which never import each other's command modules; `source-dir/`, `workflow-dir.ts`, `source-check.ts`, `source-write.ts`, `source-tree.ts`, `source-ui.ts` and `workspace-source-dir.ts` (the workspace directory reader a workflow check also reads declarations through) are shared | | `apps/sim/lib/workflows/code/` | Server glue: catalog projection from the registry (`catalog/`), source-apply gates (`apply/`), stored canonical form, the feature flag, the proof corpus (`corpus/`) | | `apps/sim/lib/workflows/application/*source*` | Workflow source use cases: read, apply, create, catalog, resources | | `apps/sim/lib/workspace-source/` | Workspace source use cases (`application/`), one adapter per kind (`application/kinds//`), route policies (`api/`) | @@ -28,7 +29,7 @@ workspace directory). | `apps/sim/app/workspace/**/panel/components/code/` | The read-only Code tab | Dependencies point one way: `packages/workflow-types` → `packages/workflow-code` → -`packages/workspace-code` and `packages/workflow-sdk` → the CLI and `apps/sim`. `check:boundaries` keeps both engines pure (no +`packages/workspace-code` and `packages/workflow-sdk` → the CLI and `apps/sim`. `check:boundaries` keeps both engines and `packages/table-cells` pure (no Node built-ins, `eval`, `globalThis`, dynamic import): they run in the browser, the CLI and the server unchanged. @@ -141,7 +142,7 @@ satisfy. 4. If workflows name it by key: `WORKFLOW_RESOURCE_KINDS`, the live loader in `lib/workspaces/resource-keys.ts`, the manifest in `read-source-resources.ts` and `codegen/workspace.ts`, and `PENDABLE_REFERENCES` in - `lib/workflows/application/source-evaluate.ts`. + `lib/workflows/application/source-pipeline/evaluate/finalize.ts`. 5. SDK: `packages/workflow-sdk/src/workspace/.d.ts` plus `index.d.ts`, a cookbook example under `cookbook/workspace/`, then regenerate the embedded text. 6. CLI (`workspace-source-dir.ts`): `workspaceTsconfig` includes, the `AGENTS_GUIDE` section and diff --git a/apps/docs/content/docs/cli/reference.mdx b/apps/docs/content/docs/cli/reference.mdx index e1c915767fa..f3c7c515d3d 100644 --- a/apps/docs/content/docs/cli/reference.mdx +++ b/apps/docs/content/docs/cli/reference.mdx @@ -8547,7 +8547,7 @@ sim workflows source get [options] -Exit codes with --dir: 0 written; 1 the command failed, the directory is a workspace source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or it holds another workflow's source (SOURCE_WORKFLOW_MISMATCH) or edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force); 7 the workflow is too large to print as source; 9 rate limited (HTTP 429) after the CLI's retries: wait a minute and run it again. Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. +Exit codes with --dir: 0 written; 1 the command failed, the directory is a workspace source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or it holds another workflow's source (SOURCE_WORKFLOW_MISMATCH) or edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force); 7 the workflow is too large to print as source; 9 rate limited (HTTP 429) after the CLI's retries: wait a minute and run it again; 10 (busy, retry) another sim command is using --dir, or the workspace directory it is a workflow folder of (SOURCE_DIR_BUSY): nothing was read or written, run it again once the other finished; 13 not logged in, the login expired, or the server refused the credential (AUTH_REQUIRED): nothing was read or written, run `sim login` and the get again. API request and response schemas: [getWorkflowSource](/api-reference/workflows/getWorkflowSource). @@ -8584,7 +8584,7 @@ sim workflows source apply [workflowId] [options] -Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); 6 the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it). Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check. +Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); 6 the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again; 10 (busy, retry) another sim command is using --dir, or the workspace directory it is a workflow folder of (SOURCE_DIR_BUSY): nothing was read or written, run it again once the other finished; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it). Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check. API request and response schemas: [applyWorkflowSource](/api-reference/workflows/applyWorkflowSource), [deployWorkflow](/api-reference/workflows/deployWorkflow), [getSourceResources](/api-reference/workflows/getSourceResources). @@ -8612,7 +8612,7 @@ sim workflows source create [options] -Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the workspace its types were written for unless --workspace names another); 6 the folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was created (run the same command with --yes). Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check. +Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the workspace its types were written for unless --workspace names another); 6 the folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again; 10 (busy, retry) another sim command is using --dir, or the workspace directory it is a workflow folder of (SOURCE_DIR_BUSY): nothing was read or written, run it again once the other finished; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was created (run the same command with --yes). Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check. API request and response schemas: [createWorkflowFromSource](/api-reference/workflows/createWorkflowFromSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog), [getSourceResources](/api-reference/workflows/getSourceResources), [deployWorkflow](/api-reference/workflows/deployWorkflow). @@ -9960,9 +9960,7 @@ sim workspaces source get [options] A directory that already holds a .sim/base.json is refreshed from the workspace it was read from, whatever the profile's default workspace. -Exit codes: 0 written; 1 the command failed, the directory holds edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force), --workspace names another workspace than the one the directory was read from (SOURCE_OTHER_WORKSPACE), the directory is a workflow source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or a downloaded binary file does not match its listing (SOURCE_BLOB_MISMATCH: nothing was written); 7 the workspace has more resources of a kind than one directory holds; 9 rate limited (HTTP 429) after the CLI's retries: nothing was written, the binary files already downloaded are kept in .sim/downloads, and running the get again downloads only the rest. - -Exit 10 (busy, retry): another sim workspaces source command is using this directory (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. +Exit codes: 0 written; 1 the command failed, the directory holds edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force), --workspace names another workspace than the one the directory was read from (SOURCE_OTHER_WORKSPACE), the directory is a workflow source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or a downloaded binary file does not match its listing (SOURCE_BLOB_MISMATCH: nothing was written); 7 the workspace has more resources of a kind than one directory holds; 9 rate limited (HTTP 429) after the CLI's retries: nothing was written, the binary files already downloaded are kept in .sim/downloads, and running the get again downloads only the rest; 10 (busy, retry) another sim workspaces source command is using this directory (SOURCE_DIR_BUSY): nothing was read or written, run it again once it finished; 13 not logged in, the login expired, or the server refused the credential (AUTH_REQUIRED): nothing was read or written, run `sim login` and the get again. API request and response schemas: [getWorkspaceSource](/api-reference/workflows/getWorkspaceSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog), [getSourceResources](/api-reference/workflows/getSourceResources), [downloadFile](/api-reference/files/downloadFile), [downloadKnowledgeDocument](/api-reference/knowledge-bases/downloadKnowledgeDocument). diff --git a/apps/docs/content/docs/cli/workflows.mdx b/apps/docs/content/docs/cli/workflows.mdx index 382c4395d7d..a376ef22740 100644 --- a/apps/docs/content/docs/cli/workflows.mdx +++ b/apps/docs/content/docs/cli/workflows.mdx @@ -1317,7 +1317,7 @@ sim workflows source get [options] -Exit codes with --dir: 0 written; 1 the command failed, the directory is a workspace source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or it holds another workflow's source (SOURCE_WORKFLOW_MISMATCH) or edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force); 7 the workflow is too large to print as source; 9 rate limited (HTTP 429) after the CLI's retries: wait a minute and run it again. Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. +Exit codes with --dir: 0 written; 1 the command failed, the directory is a workspace source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or it holds another workflow's source (SOURCE_WORKFLOW_MISMATCH) or edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force); 7 the workflow is too large to print as source; 9 rate limited (HTTP 429) after the CLI's retries: wait a minute and run it again; 10 (busy, retry) another sim command is using --dir, or the workspace directory it is a workflow folder of (SOURCE_DIR_BUSY): nothing was read or written, run it again once the other finished; 13 not logged in, the login expired, or the server refused the credential (AUTH_REQUIRED): nothing was read or written, run `sim login` and the get again. API request and response schemas: [getWorkflowSource](/api-reference/workflows/getWorkflowSource). @@ -1365,7 +1365,7 @@ sim workflows source create [options] -Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the workspace its types were written for unless --workspace names another); 6 the folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was created (run the same command with --yes). Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check. +Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the workspace its types were written for unless --workspace names another); 6 the folder is locked; 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again; 10 (busy, retry) another sim command is using --dir, or the workspace directory it is a workflow folder of (SOURCE_DIR_BUSY): nothing was read or written, run it again once the other finished; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was created (run the same command with --yes). Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check. API request and response schemas: [createWorkflowFromSource](/api-reference/workflows/createWorkflowFromSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog), [getSourceResources](/api-reference/workflows/getSourceResources), [deployWorkflow](/api-reference/workflows/deployWorkflow). @@ -1400,7 +1400,7 @@ sim workflows source apply [workflowId] [options] -Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); 6 the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it). Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check. +Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); 6 the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again; 10 (busy, retry) another sim command is using --dir, or the workspace directory it is a workflow folder of (SOURCE_DIR_BUSY): nothing was read or written, run it again once the other finished; 14 without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it). Exit 13: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. Run `sim login`, or fix or unset SIM_API_KEY when the key comes from it. Exit 130: interrupted (Ctrl-C). A write already sent is finished and saved into the directory first, waiting up to 60s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written and how to check. API request and response schemas: [applyWorkflowSource](/api-reference/workflows/applyWorkflowSource), [deployWorkflow](/api-reference/workflows/deployWorkflow), [getSourceResources](/api-reference/workflows/getSourceResources). diff --git a/apps/docs/content/docs/cli/workspaces.mdx b/apps/docs/content/docs/cli/workspaces.mdx index 083554ef028..e0002bfa79b 100644 --- a/apps/docs/content/docs/cli/workspaces.mdx +++ b/apps/docs/content/docs/cli/workspaces.mdx @@ -708,9 +708,7 @@ sim workspaces source get [options] A directory that already holds a .sim/base.json is refreshed from the workspace it was read from, whatever the profile's default workspace. -Exit codes: 0 written; 1 the command failed, the directory holds edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force), --workspace names another workspace than the one the directory was read from (SOURCE_OTHER_WORKSPACE), the directory is a workflow source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or a downloaded binary file does not match its listing (SOURCE_BLOB_MISMATCH: nothing was written); 7 the workspace has more resources of a kind than one directory holds; 9 rate limited (HTTP 429) after the CLI's retries: nothing was written, the binary files already downloaded are kept in .sim/downloads, and running the get again downloads only the rest. - -Exit 10 (busy, retry): another sim workspaces source command is using this directory (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished. +Exit codes: 0 written; 1 the command failed, the directory holds edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force), --workspace names another workspace than the one the directory was read from (SOURCE_OTHER_WORKSPACE), the directory is a workflow source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or a downloaded binary file does not match its listing (SOURCE_BLOB_MISMATCH: nothing was written); 7 the workspace has more resources of a kind than one directory holds; 9 rate limited (HTTP 429) after the CLI's retries: nothing was written, the binary files already downloaded are kept in .sim/downloads, and running the get again downloads only the rest; 10 (busy, retry) another sim workspaces source command is using this directory (SOURCE_DIR_BUSY): nothing was read or written, run it again once it finished; 13 not logged in, the login expired, or the server refused the credential (AUTH_REQUIRED): nothing was read or written, run `sim login` and the get again. API request and response schemas: [getWorkspaceSource](/api-reference/workflows/getWorkspaceSource), [getSourceCatalog](/api-reference/catalog/getSourceCatalog), [getSourceResources](/api-reference/workflows/getSourceResources), [downloadFile](/api-reference/files/downloadFile), [downloadKnowledgeDocument](/api-reference/knowledge-bases/downloadKnowledgeDocument). diff --git a/apps/docs/openapi-v2-workflows.json b/apps/docs/openapi-v2-workflows.json index d1b9e6b6b43..98e30b39fa9 100644 --- a/apps/docs/openapi-v2-workflows.json +++ b/apps/docs/openapi-v2-workflows.json @@ -19156,7 +19156,7 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", + "I_FINDINGS_TRUNCATED", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", @@ -19627,7 +19627,7 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", + "I_FINDINGS_TRUNCATED", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", @@ -20616,7 +20616,7 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", + "I_FINDINGS_TRUNCATED", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", @@ -21581,7 +21581,7 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", + "I_FINDINGS_TRUNCATED", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", @@ -26538,7 +26538,6 @@ "W_PATH_NOT_SENT", "E_BLOB_MANIFEST", "E_BLOB_MISSING", - "E_LIMIT_EXCEEDED", "E_KEY_INVALID", "W_DECL_SUSPICIOUS_CHARACTER", "E_BASE_JSON", @@ -26601,7 +26600,6 @@ "W_DEPLOY_DRAFT_NOT_SOURCE", "E_DEPLOY_DRAFT_NOT_SOURCE_CHAT", "I_DEPLOY_DRAFT_NOT_LIVE", - "E_APPLY_STALE", "E_APPLY_NO_BASE", "E_APPLY_FORBIDDEN", "E_APPLY_DELETE_NOT_ALLOWED", @@ -26613,7 +26611,6 @@ "E_PLAN_REF_HELD", "W_RESOURCE_IN_USE", "W_TABLE_COLUMN_RENAMED_IN_USE", - "E_NAME_TAKEN", "E_WORKFLOW_FILES_MISSING", "E_WORKFLOW_FOLDER_COPIED", "W_UNDECLARED", @@ -26625,6 +26622,9 @@ "W_SERVER_MOVED", "I_KEY_REBOUND", "E_KEY_RENAMED", + "E_APPLY_STALE", + "E_LIMIT_EXCEEDED", + "E_NAME_TAKEN", "I_FINDINGS_TRUNCATED", "E_SYNTAX", "E_SDK_HEADER", @@ -26636,7 +26636,6 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", @@ -27115,7 +27114,6 @@ "W_PATH_NOT_SENT", "E_BLOB_MANIFEST", "E_BLOB_MISSING", - "E_LIMIT_EXCEEDED", "E_KEY_INVALID", "W_DECL_SUSPICIOUS_CHARACTER", "E_BASE_JSON", @@ -27178,7 +27176,6 @@ "W_DEPLOY_DRAFT_NOT_SOURCE", "E_DEPLOY_DRAFT_NOT_SOURCE_CHAT", "I_DEPLOY_DRAFT_NOT_LIVE", - "E_APPLY_STALE", "E_APPLY_NO_BASE", "E_APPLY_FORBIDDEN", "E_APPLY_DELETE_NOT_ALLOWED", @@ -27190,7 +27187,6 @@ "E_PLAN_REF_HELD", "W_RESOURCE_IN_USE", "W_TABLE_COLUMN_RENAMED_IN_USE", - "E_NAME_TAKEN", "E_WORKFLOW_FILES_MISSING", "E_WORKFLOW_FOLDER_COPIED", "W_UNDECLARED", @@ -27202,6 +27198,9 @@ "W_SERVER_MOVED", "I_KEY_REBOUND", "E_KEY_RENAMED", + "E_APPLY_STALE", + "E_LIMIT_EXCEEDED", + "E_NAME_TAKEN", "I_FINDINGS_TRUNCATED", "E_SYNTAX", "E_SDK_HEADER", @@ -27213,7 +27212,6 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", @@ -27535,7 +27533,6 @@ "W_PATH_NOT_SENT", "E_BLOB_MANIFEST", "E_BLOB_MISSING", - "E_LIMIT_EXCEEDED", "E_KEY_INVALID", "W_DECL_SUSPICIOUS_CHARACTER", "E_BASE_JSON", @@ -27598,7 +27595,6 @@ "W_DEPLOY_DRAFT_NOT_SOURCE", "E_DEPLOY_DRAFT_NOT_SOURCE_CHAT", "I_DEPLOY_DRAFT_NOT_LIVE", - "E_APPLY_STALE", "E_APPLY_NO_BASE", "E_APPLY_FORBIDDEN", "E_APPLY_DELETE_NOT_ALLOWED", @@ -27610,7 +27606,6 @@ "E_PLAN_REF_HELD", "W_RESOURCE_IN_USE", "W_TABLE_COLUMN_RENAMED_IN_USE", - "E_NAME_TAKEN", "E_WORKFLOW_FILES_MISSING", "E_WORKFLOW_FOLDER_COPIED", "W_UNDECLARED", @@ -27622,6 +27617,9 @@ "W_SERVER_MOVED", "I_KEY_REBOUND", "E_KEY_RENAMED", + "E_APPLY_STALE", + "E_LIMIT_EXCEEDED", + "E_NAME_TAKEN", "I_FINDINGS_TRUNCATED", "E_SYNTAX", "E_SDK_HEADER", @@ -27633,7 +27631,6 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", @@ -28401,7 +28398,6 @@ "W_PATH_NOT_SENT", "E_BLOB_MANIFEST", "E_BLOB_MISSING", - "E_LIMIT_EXCEEDED", "E_KEY_INVALID", "W_DECL_SUSPICIOUS_CHARACTER", "E_BASE_JSON", @@ -28464,7 +28460,6 @@ "W_DEPLOY_DRAFT_NOT_SOURCE", "E_DEPLOY_DRAFT_NOT_SOURCE_CHAT", "I_DEPLOY_DRAFT_NOT_LIVE", - "E_APPLY_STALE", "E_APPLY_NO_BASE", "E_APPLY_FORBIDDEN", "E_APPLY_DELETE_NOT_ALLOWED", @@ -28476,7 +28471,6 @@ "E_PLAN_REF_HELD", "W_RESOURCE_IN_USE", "W_TABLE_COLUMN_RENAMED_IN_USE", - "E_NAME_TAKEN", "E_WORKFLOW_FILES_MISSING", "E_WORKFLOW_FOLDER_COPIED", "W_UNDECLARED", @@ -28488,6 +28482,9 @@ "W_SERVER_MOVED", "I_KEY_REBOUND", "E_KEY_RENAMED", + "E_APPLY_STALE", + "E_LIMIT_EXCEEDED", + "E_NAME_TAKEN", "I_FINDINGS_TRUNCATED", "E_SYNTAX", "E_SDK_HEADER", @@ -28499,7 +28496,6 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", @@ -28821,7 +28817,6 @@ "W_PATH_NOT_SENT", "E_BLOB_MANIFEST", "E_BLOB_MISSING", - "E_LIMIT_EXCEEDED", "E_KEY_INVALID", "W_DECL_SUSPICIOUS_CHARACTER", "E_BASE_JSON", @@ -28884,7 +28879,6 @@ "W_DEPLOY_DRAFT_NOT_SOURCE", "E_DEPLOY_DRAFT_NOT_SOURCE_CHAT", "I_DEPLOY_DRAFT_NOT_LIVE", - "E_APPLY_STALE", "E_APPLY_NO_BASE", "E_APPLY_FORBIDDEN", "E_APPLY_DELETE_NOT_ALLOWED", @@ -28896,7 +28890,6 @@ "E_PLAN_REF_HELD", "W_RESOURCE_IN_USE", "W_TABLE_COLUMN_RENAMED_IN_USE", - "E_NAME_TAKEN", "E_WORKFLOW_FILES_MISSING", "E_WORKFLOW_FOLDER_COPIED", "W_UNDECLARED", @@ -28908,6 +28901,9 @@ "W_SERVER_MOVED", "I_KEY_REBOUND", "E_KEY_RENAMED", + "E_APPLY_STALE", + "E_LIMIT_EXCEEDED", + "E_NAME_TAKEN", "I_FINDINGS_TRUNCATED", "E_SYNTAX", "E_SDK_HEADER", @@ -28919,7 +28915,6 @@ "E_NUMBER_NOT_FINITE", "W_NUMBER_PRECISION", "E_TOO_MANY_ERRORS", - "W_TOO_MANY_WARNINGS", "E_DUPLICATE_NAME", "E_RESERVED_NAME", "E_KEY_OVERRIDE", diff --git a/apps/sim/app/api/v2/workflows/[workflowId]/source/plan/route.ts b/apps/sim/app/api/v2/workflows/[workflowId]/source/plan/route.ts index 6031ead11b6..aa289f4826b 100644 --- a/apps/sim/app/api/v2/workflows/[workflowId]/source/plan/route.ts +++ b/apps/sim/app/api/v2/workflows/[workflowId]/source/plan/route.ts @@ -1,7 +1,7 @@ import { SOURCE_REQUEST_MAX_BYTES } from '@sim/workflow-code/limits' import { v2PlanWorkflowSourceContract } from '@/lib/api/contracts/v2/workflow-source' import { defineV2JsonRoute, v2ApiKeyAuth, v2RateLimits } from '@/lib/api/server/routes' -import { v2WorkflowErrorPolicies, withSourceSlot } from '@/lib/workflows/api' +import { presentSourceOutcome, v2WorkflowErrorPolicies, withSourceSlot } from '@/lib/workflows/api' import { planWorkflowSource } from '@/lib/workflows/application/apply-workflow-source' import { workflowOperations } from '@/lib/workflows/application/operations' @@ -30,5 +30,5 @@ export const POST = defineV2JsonRoute({ ...(query.allowDelete !== undefined ? { allowDelete: query.allowDelete } : {}), }), useCase: withSourceSlot(planWorkflowSource, 'plan'), - present: (result) => ({ data: result }), + present: (result) => ({ data: presentSourceOutcome(result) }), }) diff --git a/apps/sim/app/api/v2/workflows/[workflowId]/source/route.test.ts b/apps/sim/app/api/v2/workflows/[workflowId]/source/route.test.ts index 997e14b4a71..071f4a301e4 100644 --- a/apps/sim/app/api/v2/workflows/[workflowId]/source/route.test.ts +++ b/apps/sim/app/api/v2/workflows/[workflowId]/source/route.test.ts @@ -120,6 +120,23 @@ describe('GET /api/v2/workflows/{workflowId}/source', () => { expect(mocks.read).not.toHaveBeenCalled() }) + it('answers a stored draft whose canonicalize floods one warning with 20 of them and one I_FINDINGS_TRUNCATED', async () => { + const flood = Array.from({ length: 120 }, (_, index) => ({ + code: 'W_CANON_UNRESOLVED_ID' as const, + severity: 'warning' as const, + message: `Table gone-${index} no longer exists.`, + })) + mocks.read.mockResolvedValue({ ...source, diagnostics: flood }) + + const response = await GET(request(), context) + const { data } = await response.json() + + expect(response.status).toBe(200) + const codes = data.diagnostics.map((finding: { code: string }) => finding.code) + expect(codes.filter((code: string) => code === 'W_CANON_UNRESOLVED_ID')).toHaveLength(20) + expect(codes.filter((code: string) => code === 'I_FINDINGS_TRUNCATED')).toHaveLength(1) + }) + it('passes functionCode through, omitting it when absent', async () => { await GET(request('?functionCode=inline'), context) await GET(request(), context) @@ -219,6 +236,49 @@ describe('POST /api/v2/workflows/{workflowId}/source', () => { expect(body.error.details).toEqual({ code: 'SOURCE_REJECTED', diagnostics: [diagnostic] }) }) + it('answers a flood of one warning as 20 of them and one I_FINDINGS_TRUNCATED, in place of the compile’s', async () => { + const flood = Array.from({ length: 120 }, (_, index) => ({ + code: 'W_CANON_UNRESOLVED_ID' as const, + severity: 'warning' as const, + message: `Table gone-${index} no longer exists.`, + })) + const compileMarker = { + code: 'I_FINDINGS_TRUNCATED' as const, + severity: 'info' as const, + message: 'Listed 20 W_UNRESOLVED_REF warnings; the rest of them are omitted.', + } + mocks.apply.mockResolvedValue({ ...applied, diagnostics: [compileMarker, ...flood] }) + + const response = await POST(applyRequest(applyBody), context) + const { data } = await response.json() + + expect(response.status).toBe(200) + const codes = data.diagnostics.map((finding: { code: string }) => finding.code) + expect(codes.filter((code: string) => code === 'W_CANON_UNRESOLVED_ID')).toHaveLength(20) + expect(codes.filter((code: string) => code === 'I_FINDINGS_TRUNCATED')).toHaveLength(1) + expect(data.diagnostics.at(-1).message).toContain('There are 120 findings; 100 more') + }) + + it('answers a refusal with more errors than one answer lists as the first 100 and one that counts the rest', async () => { + const errors = Array.from({ length: 150 }, (_, index) => ({ + code: 'E_UNKNOWN_IDENTIFIER' as const, + severity: 'error' as const, + message: `Unknown identifier t${index}.`, + })) + mocks.apply.mockRejectedValue(new WorkflowSourceRejectedError(errors)) + + const response = await POST(applyRequest(applyBody), context) + const body = await response.json() + + expect(response.status).toBe(400) + const listed = body.error.details.diagnostics + expect(listed).toHaveLength(101) + expect(listed.at(-1)).toMatchObject({ + code: 'I_FINDINGS_TRUNCATED', + message: expect.stringContaining('50 more (50 of them errors)'), + }) + }) + it('answers 409 with the current fingerprint when the draft moved', async () => { const current = `sf1:${'f'.repeat(64)}` mocks.apply.mockRejectedValue(new WorkflowSourceStaleError(current, 'Lead Scoring')) diff --git a/apps/sim/app/api/v2/workflows/[workflowId]/source/route.ts b/apps/sim/app/api/v2/workflows/[workflowId]/source/route.ts index 90e70a43e53..b5b249a9a42 100644 --- a/apps/sim/app/api/v2/workflows/[workflowId]/source/route.ts +++ b/apps/sim/app/api/v2/workflows/[workflowId]/source/route.ts @@ -4,7 +4,12 @@ import { v2GetWorkflowSourceContract, } from '@/lib/api/contracts/v2/workflow-source' import { defineV2JsonRoute, v2ApiKeyAuth, v2RateLimits } from '@/lib/api/server/routes' -import { v2WorkflowErrorPolicies, withSourceSlot } from '@/lib/workflows/api' +import { + presentSourceOutcome, + v2WorkflowErrorPolicies, + withBoundSourceFindings, + withSourceSlot, +} from '@/lib/workflows/api' import { applyWorkflowSource } from '@/lib/workflows/application/apply-workflow-source' import { workflowOperations } from '@/lib/workflows/application/operations' import { readWorkflowSource } from '@/lib/workflows/application/read-workflow-source' @@ -33,7 +38,7 @@ export const GET = defineV2JsonRoute({ ...(query.functionCode ? { functionCode: query.functionCode } : {}), }), useCase: withSourceSlot(readWorkflowSource, 'read'), - present: (source) => ({ data: source }), + present: (source) => ({ data: withBoundSourceFindings(source) }), }) /** @@ -62,5 +67,5 @@ export const POST = defineV2JsonRoute({ ...(query.allowDelete !== undefined ? { allowDelete: query.allowDelete } : {}), }), useCase: withSourceSlot(applyWorkflowSource, 'apply'), - present: (result) => ({ data: result }), + present: (result) => ({ data: presentSourceOutcome(result) }), }) diff --git a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/plan/route.test.ts b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/plan/route.test.ts index 1c6df0fab07..59e5579c238 100644 --- a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/plan/route.test.ts +++ b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/plan/route.test.ts @@ -83,4 +83,30 @@ describe('POST /api/v2/workspaces/{workspaceId}/source/plan', () => { expect(response.status).toBe(200) expect(Math.max(...charged.values())).toBe(3) }) + + it('answers a plan with at most 1000 findings, however many its workflows hold', async () => { + mocks.plan.mockResolvedValue({ + workspaceId: WORKSPACE_ID, + dryRun: true, + plan: [], + applied: [], + needBlobs: [], + files: {}, + base: { version: 1, workspaceId: WORKSPACE_ID, resources: {} }, + diagnostics: Array.from({ length: 1_200 }, (_, index) => ({ + code: 'W_CANON_UNRESOLVED_ID' as const, + severity: 'warning' as const, + message: `Table gone-${index} no longer exists.`, + kind: 'workflow' as const, + key: `w${index}`, + })), + }) + + const response = await plan({ 'workflows/a/workflow.ts': WORKFLOW }) + const { data } = await response.json() + + expect(response.status).toBe(200) + expect(data.diagnostics).toHaveLength(1_001) + expect(data.diagnostics.at(-1)).toMatchObject({ code: 'I_FINDINGS_TRUNCATED' }) + }) }) diff --git a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/plan/route.ts b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/plan/route.ts index 39f2fcf1e15..76a44f74883 100644 --- a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/plan/route.ts +++ b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/plan/route.ts @@ -4,6 +4,7 @@ import { defineV2JsonRoute, v2ApiKeyAuth, v2RateLimits } from '@/lib/api/server/ import { withSourceSlot } from '@/lib/workflows/api' import { v2WorkspaceSourceErrorPolicy, + withBoundWorkspaceFindings, workspaceApplyRateCost, } from '@/lib/workspace-source/api/route-policies' import { planWorkspaceSource } from '@/lib/workspace-source/application/apply-workspace-source' @@ -38,5 +39,5 @@ export const POST = defineV2JsonRoute({ }), rateLimitCost: (input) => workspaceApplyRateCost(input.files, input.blobs), useCase: withSourceSlot(planWorkspaceSource, 'plan'), - present: (result) => ({ data: result }), + present: (result) => ({ data: withBoundWorkspaceFindings(result) }), }) diff --git a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/route.test.ts b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/route.test.ts new file mode 100644 index 00000000000..8838546e7d2 --- /dev/null +++ b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/route.test.ts @@ -0,0 +1,163 @@ +import { + V2_OPERATION_RATE_LIMIT_ALLOWED, + V2_PREAUTH_RATE_LIMIT_ALLOWED, + v2ApiKeyAuthModuleMock, + v2RateLimiterModuleMock, + v2RouteMocks, +} from '@sim/testing' +import { NextRequest } from 'next/server' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const mocks = vi.hoisted(() => ({ apply: vi.fn(), read: vi.fn(), authorizeRead: vi.fn() })) + +vi.mock('@/lib/api/server/routes/v2-api-key-auth', () => v2ApiKeyAuthModuleMock) +vi.mock('@/lib/core/rate-limiter', () => v2RateLimiterModuleMock) +vi.mock('@/lib/workspace-source/application/apply-workspace-source', () => ({ + applyWorkspaceSource: { operation: { id: 'workspaces.source.apply' }, execute: mocks.apply }, +})) +vi.mock('@/lib/workspace-source/application/read-workspace-source', () => ({ + readWorkspaceSource: { + operation: { id: 'workspaces.source.read' }, + authorize: mocks.authorizeRead, + execute: mocks.read, + }, +})) + +import { WorkspaceSourceRejectedError } from '@/lib/workspace-source/application/errors' +import type { WorkspaceSourceDiagnostic } from '@/lib/workspace-source/application/types' +import { GET, POST } from '@/app/api/v2/workspaces/[workspaceId]/source/route' + +const WORKSPACE_ID = '11111111-2222-4333-8444-555555555555' + +const auth = { + principal: { + kind: 'workspace_api_key' as const, + workspaceId: WORKSPACE_ID, + keyId: 'key-1', + access: 'write' as const, + }, + rateLimitSubjectIds: ['api-key:key-1', `workspace:${WORKSPACE_ID}`] as const, + rateLimitSubscription: null, + keyType: 'workspace' as const, +} + +function apply(query = '') { + return POST( + new NextRequest(`http://localhost:3000/api/v2/workspaces/${WORKSPACE_ID}/source${query}`, { + method: 'POST', + headers: { 'x-api-key': 'key', 'content-type': 'application/json' }, + body: JSON.stringify({ + files: { + 'workflows/a/workflow.ts': "// @sim-sdk 1\nexport default workflow('A', () => {})\n", + }, + kinds: ['workflow'], + }), + }), + { params: Promise.resolve({ workspaceId: WORKSPACE_ID }) } + ) +} + +/** `count` errors of one workflow folder, as a plan refusal pins them to it. */ +function folderErrors(count: number): WorkspaceSourceDiagnostic[] { + return Array.from({ length: count }, (_, index) => ({ + code: 'E_UNKNOWN_IDENTIFIER' as const, + severity: 'error' as const, + message: `Unknown identifier t${index}.`, + file: 'workflows/a/workflow.ts', + kind: 'workflow' as const, + key: 'a', + })) +} + +describe('POST /api/v2/workspaces/{workspaceId}/source', () => { + beforeEach(() => { + v2RouteMocks.authenticate.mockResolvedValue(auth) + v2RouteMocks.preauthRate.mockResolvedValue(V2_PREAUTH_RATE_LIMIT_ALLOWED) + v2RouteMocks.operationRate.mockResolvedValue(V2_OPERATION_RATE_LIMIT_ALLOWED) + }) + + it('answers a plan refused before any write with at most 1000 findings, and one that counts the rest', async () => { + mocks.apply.mockRejectedValue(new WorkspaceSourceRejectedError(folderErrors(1_500))) + + const response = await apply() + const body = await response.json() + + expect(response.status).toBe(400) + const listed = body.error.details.diagnostics + expect(listed).toHaveLength(1_001) + expect(listed.at(-1)).toMatchObject({ + code: 'I_FINDINGS_TRUNCATED', + severity: 'info', + message: expect.stringContaining('500 more (500 of them errors)'), + }) + }) + + it('answers an apply with at most 1000 findings, however many its writes added', async () => { + mocks.apply.mockResolvedValue({ + workspaceId: WORKSPACE_ID, + dryRun: false, + plan: [], + applied: [], + needBlobs: [], + files: {}, + base: { version: 1, workspaceId: WORKSPACE_ID, resources: {} }, + diagnostics: Array.from({ length: 1_200 }, (_, index) => ({ + code: 'W_CANON_UNRESOLVED_ID' as const, + severity: 'warning' as const, + message: `Table gone-${index} no longer exists.`, + kind: 'workflow' as const, + key: `w${index}`, + })), + }) + + const response = await apply() + const { data } = await response.json() + + expect(response.status).toBe(200) + expect(data.diagnostics).toHaveLength(1_001) + expect(data.diagnostics.at(-1)).toMatchObject({ code: 'I_FINDINGS_TRUNCATED' }) + }) +}) + +/** `count` warnings, one per workflow, as a stored workspace whose tables are gone reads. */ +function unresolvedWarnings(count: number): WorkspaceSourceDiagnostic[] { + return Array.from({ length: count }, (_, index) => ({ + code: 'W_CANON_UNRESOLVED_ID' as const, + severity: 'warning' as const, + message: `Table gone-${index} no longer exists.`, + kind: 'workflow' as const, + key: `w${index}`, + })) +} + +describe('GET /api/v2/workspaces/{workspaceId}/source', () => { + beforeEach(() => { + v2RouteMocks.authenticate.mockResolvedValue(auth) + v2RouteMocks.preauthRate.mockResolvedValue(V2_PREAUTH_RATE_LIMIT_ALLOWED) + v2RouteMocks.operationRate.mockResolvedValue(V2_OPERATION_RATE_LIMIT_ALLOWED) + }) + + it('answers a read with at most 1000 findings, however many its workflows hold', async () => { + mocks.read.mockResolvedValue({ + workspaceId: WORKSPACE_ID, + kinds: ['workflow'], + files: {}, + blobs: [], + base: { version: 1, workspaceId: WORKSPACE_ID, resources: {} }, + skipped: [], + diagnostics: unresolvedWarnings(1_200), + }) + + const response = await GET( + new NextRequest(`http://localhost:3000/api/v2/workspaces/${WORKSPACE_ID}/source`, { + headers: { 'x-api-key': 'key' }, + }), + { params: Promise.resolve({ workspaceId: WORKSPACE_ID }) } + ) + const { data } = await response.json() + + expect(response.status).toBe(200) + expect(data.diagnostics).toHaveLength(1_001) + expect(data.diagnostics.at(-1)).toMatchObject({ code: 'I_FINDINGS_TRUNCATED' }) + }) +}) diff --git a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/route.ts b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/route.ts index f61927065ff..4afda03230c 100644 --- a/apps/sim/app/api/v2/workspaces/[workspaceId]/source/route.ts +++ b/apps/sim/app/api/v2/workspaces/[workspaceId]/source/route.ts @@ -7,6 +7,7 @@ import { defineV2JsonRoute, v2ApiKeyAuth, v2RateLimits } from '@/lib/api/server/ import { withSourceSlot } from '@/lib/workflows/api' import { v2WorkspaceSourceErrorPolicy, + withBoundWorkspaceFindings, workspaceApplyRateCost, } from '@/lib/workspace-source/api/route-policies' import { applyWorkspaceSource } from '@/lib/workspace-source/application/apply-workspace-source' @@ -35,7 +36,7 @@ export const GET = defineV2JsonRoute({ ...(query.kinds ? { kinds: query.kinds } : {}), }), useCase: withSourceSlot(readWorkspaceSource, 'read'), - present: (source) => ({ data: source }), + present: (source) => ({ data: withBoundWorkspaceFindings(source) }), }) /** @@ -65,5 +66,5 @@ export const POST = defineV2JsonRoute({ }), rateLimitCost: (input) => workspaceApplyRateCost(input.files, input.blobs), useCase: withSourceSlot(applyWorkspaceSource, 'apply'), - present: (result) => ({ data: result }), + present: (result) => ({ data: withBoundWorkspaceFindings(result) }), }) diff --git a/apps/sim/app/api/v2/workspaces/[workspaceId]/workflows/source/route.ts b/apps/sim/app/api/v2/workspaces/[workspaceId]/workflows/source/route.ts index e92c65bb48d..f5ad727483c 100644 --- a/apps/sim/app/api/v2/workspaces/[workspaceId]/workflows/source/route.ts +++ b/apps/sim/app/api/v2/workspaces/[workspaceId]/workflows/source/route.ts @@ -1,7 +1,7 @@ import { SOURCE_REQUEST_MAX_BYTES } from '@sim/workflow-code/limits' import { v2CreateWorkflowFromSourceContract } from '@/lib/api/contracts/v2/workflow-source' import { defineV2JsonRoute, v2ApiKeyAuth, v2RateLimits } from '@/lib/api/server/routes' -import { v2WorkflowErrorPolicies } from '@/lib/workflows/api' +import { presentSourceOutcome, v2WorkflowErrorPolicies } from '@/lib/workflows/api' import { createWorkflowFromSource } from '@/lib/workflows/application/apply-workflow-source' import { workflowOperations } from '@/lib/workflows/application/operations' @@ -32,5 +32,5 @@ export const POST = defineV2JsonRoute({ ...(query.dryRun !== undefined ? { dryRun: query.dryRun } : {}), }), useCase: createWorkflowFromSource, - present: (result) => ({ data: result }), + present: (result) => ({ data: presentSourceOutcome(result) }), }) diff --git a/apps/sim/app/api/workflows/[id]/source/route.ts b/apps/sim/app/api/workflows/[id]/source/route.ts index e374f647e25..fe59cbce4fc 100644 --- a/apps/sim/app/api/workflows/[id]/source/route.ts +++ b/apps/sim/app/api/workflows/[id]/source/route.ts @@ -6,7 +6,7 @@ import { internalSessionAuth, } from '@/lib/api/server/routes' import type { TokenBucketConfig } from '@/lib/core/rate-limiter' -import { internalWorkflowErrorPolicies } from '@/lib/workflows/api' +import { internalWorkflowErrorPolicies, withBoundSourceFindings } from '@/lib/workflows/api' import { readWorkflowSource } from '@/lib/workflows/application/read-workflow-source' export const dynamic = 'force-dynamic' @@ -51,5 +51,5 @@ export const GET = defineInternalJsonRoute({ }), useCase: readWorkflowSource, entityTag: (source) => `${source.base.fingerprint}.${sha256Hex(JSON.stringify(source))}`, - present: (source) => ({ data: source }), + present: (source) => ({ data: withBoundSourceFindings(source) }), }) diff --git a/apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/chat-content/chat-content.test.ts b/apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/chat-content/chat-content.test.ts index bd5ebc531d3..10e927d7034 100644 --- a/apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/chat-content/chat-content.test.ts +++ b/apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/chat-content/chat-content.test.ts @@ -1,6 +1,25 @@ +import { expectLinearGrowth } from '@sim/testing/helpers/scaling' import { describe, expect, it } from 'vitest' import { sanitizeChatDisplayContent } from '@/app/workspace/[workspaceId]/home/components/message-content/components/chat-content/chat-sanitize' -import { scalingRatioOver4x } from '@/app/workspace/[workspaceId]/home/components/message-content/components/scaling-test-helpers' + +/** How many repetitions the smaller input of a scaling check holds. */ +const SCALING_REPEATS = 1_000 + +/** + * The scaling step for `sanitizeChatDisplayContent` over `build(times)`, built untimed. The + * default input repeats a tag name that never closes: the shape both the parser and the display + * sanitizer used to be quadratic on, because a scan allowed to cross an opener restarts from + * every opener. + */ +function sanitizeStep( + build: (times: number) => string = (times) => + 'The tag is used here. '.repeat(times) +): (times: number) => () => void { + return (times) => { + const content = build(times) + return () => sanitizeChatDisplayContent(content) + } +} describe('sanitizeChatDisplayContent', () => { it('does not let an unmatched backtick run suppress later citations', () => { @@ -61,20 +80,14 @@ describe('sanitizeChatDisplayContent', () => { // A lazy scan allowed to cross an opener restarts from every opener, which // is quadratic — 154ms for this input before the bound, on the main thread, // for every streamed chunk. - // - // Asserted as a scaling ratio, not a wall-clock ceiling — see - // {@link scalingRatioOver4x} for why. - expect(scalingRatioOver4x((content) => sanitizeChatDisplayContent(content))).toBeLessThan(8) + expectLinearGrowth(sanitizeStep(), SCALING_REPEATS) }) it('stays linear on repeated unclosed JSON chip bodies', () => { - expect( - scalingRatioOver4x((content) => - sanitizeChatDisplayContent( - content.replaceAll('The tag is used here. ', '{"snippet":"') - ) - ) - ).toBeLessThan(8) + expectLinearGrowth( + sanitizeStep((times) => '{"snippet":"'.repeat(times)), + SCALING_REPEATS + ) }) it('still unwraps a real tag that carries a stray backtick on one side only', () => { @@ -96,20 +109,20 @@ describe('sanitizeChatDisplayContent', () => { it.each(['source', 'workspace_resource'])( 'stays linear on repeated %s tags with unterminated JSON strings', (name) => { - expect( - scalingRatioOver4x(sanitizeChatDisplayContent, (times) => - `<${name}>{${String.fromCharCode(92, 34)}`.repeat(times) - ) - ).toBeLessThan(8) + expectLinearGrowth( + sanitizeStep((times) => `<${name}>{${String.fromCharCode(92, 34)}`.repeat(times)), + SCALING_REPEATS + ) } ) it.each(['"', '{"key":"'])( 'stays linear on repeated quoted payload prefix %s', (prefix) => { - expect( - scalingRatioOver4x(sanitizeChatDisplayContent, (times) => prefix.repeat(times)) - ).toBeLessThan(8) + expectLinearGrowth( + sanitizeStep((times) => prefix.repeat(times)), + SCALING_REPEATS + ) } ) }) diff --git a/apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/scaling-test-helpers.ts b/apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/scaling-test-helpers.ts deleted file mode 100644 index d925e1bcb98..00000000000 --- a/apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/scaling-test-helpers.ts +++ /dev/null @@ -1,48 +0,0 @@ -/** - * Repeated mention of a tag name that never closes — the shape both the parser - * and the display sanitizer used to be quadratic on, because a scan allowed to - * cross an opener restarts from every opener. - */ -function buildRepeatedTagMentions(times: number): string { - return 'The tag is used here. '.repeat(times) -} - -/** Fastest of five runs, so a single scheduling hiccup cannot skew the sample. */ -function fastest(run: (content: string) => void, content: string): number { - let best = Number.POSITIVE_INFINITY - for (let attempt = 0; attempt < 5; attempt++) { - const startedAt = performance.now() - run(content) - best = Math.min(best, performance.now() - startedAt) - } - return best -} - -/** - * How much slower `run` gets when its input grows 4x. - * - * Complexity is asserted as a RATIO rather than a wall-clock ceiling. A fixed - * millisecond bound measures the machine as much as the algorithm: it fails on a - * loaded CI box, and set generously enough not to, it lets a genuine quadratic - * through at the single size it happens to sample. Quadratic costs ~16x for 4x - * the input; linear costs ~4x. The median of three independent ratios is - * returned, so one noisy trial in either direction — a GC pause inflating a - * sample, or a lucky one deflating it — cannot decide the result on its own. - */ -export function scalingRatioOver4x( - run: (content: string) => void, - buildContent: (times: number) => string = buildRepeatedTagMentions -): number { - // Warm up first — the JIT would otherwise charge the whole compile to the - // small sample and flatter the ratio. - const smallContent = buildContent(2_000) - const largeContent = buildContent(8_000) - fastest(run, smallContent) - - const ratios: number[] = [] - for (let trial = 0; trial < 3; trial++) { - ratios.push(fastest(run, largeContent) / fastest(run, smallContent)) - } - ratios.sort((a, b) => a - b) - return ratios[1] -} diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-config-sidebar.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-config-sidebar.tsx index 4e6fdc63750..fec41acdd76 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-config-sidebar.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-config-sidebar.tsx @@ -13,14 +13,11 @@ import { toast, } from '@sim/emcn' import { X } from '@sim/emcn/icons' +import { DEFAULT_CURRENCY_CODE, resolveCurrencyCode } from '@sim/table-cells/currency' import { toError } from '@sim/utils/errors' -import { - DEFAULT_CURRENCY_CODE, - getCurrencyOptions, - resolveCurrencyCode, -} from '@sim/workspace-code/table-cells/currency' import { findValidationIssue, isValidationError } from '@/lib/api/client/errors' import type { ColumnDefinition, SelectOption } from '@/lib/table' +import { getCurrencyOptions } from '@/lib/table/currency-format' import { FieldError, RequiredLabel, diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/row-modal/row-modal.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/row-modal/row-modal.tsx index 6ba34ebc7f2..4aabf97e8ee 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/row-modal/row-modal.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/row-modal/row-modal.tsx @@ -16,18 +16,14 @@ import { toast, } from '@sim/emcn' import { createLogger } from '@sim/logger' +import { resolveCurrencyCode } from '@sim/table-cells/currency' import { getErrorMessage } from '@sim/utils/errors' -import { resolveCurrencyCode } from '@sim/workspace-code/table-cells/currency' -import { - todayAtTtlOffset, - ttlValueFromPicker, - ttlValueToPickerParts, -} from '@sim/workspace-code/table-cells/ttl' import { useParams } from 'next/navigation' import type { ColumnDefinition, TableInfo, TableRow } from '@/lib/table' import { getColumnId } from '@/lib/table/column-keys' import { columnTypeOf } from '@/lib/table/column-types' import { isEmptyCellValue } from '@/lib/table/deps' +import { todayAtTtlOffset, ttlValueFromPicker, ttlValueToPickerParts } from '@/lib/table/ttl-picker' import { getTimezoneEditBlockedMessage } from '@/app/workspace/[workspaceId]/tables/[tableId]/components/timezone-editing' import type { RowInsertTarget } from '@/app/workspace/[workspaceId]/tables/[tableId]/types' import { type TimezoneState, useTimezoneState } from '@/hooks/queries/general-settings' diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.test.ts b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.test.ts index 93b49bcb27b..244e3f302c3 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.test.ts +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.test.ts @@ -2,7 +2,7 @@ * @vitest-environment jsdom */ import { act, createElement, type ReactNode } from 'react' -import { TTL_FORMAT_ERROR } from '@sim/workspace-code/table-cells/ttl' +import { TTL_FORMAT_ERROR } from '@sim/table-cells/ttl' import { createRoot } from 'react-dom/client' import { beforeEach, describe, expect, it, vi } from 'vitest' import type { ColumnDefinition } from '@/lib/table' diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.tsx index ebafa6f64b5..0320c29ea18 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.tsx @@ -14,14 +14,10 @@ import { toast, } from '@sim/emcn' import { Check } from '@sim/emcn/icons' -import { isCalendarDateString } from '@sim/workspace-code/table-cells/dates' -import { - todayAtTtlOffset, - ttlValueFromPicker, - ttlValueToPickerParts, -} from '@sim/workspace-code/table-cells/ttl' +import { isCalendarDateString } from '@sim/table-cells/dates' import type { ColumnDefinition } from '@/lib/table' import { columnTypeOf } from '@/lib/table/column-types' +import { todayAtTtlOffset, ttlValueFromPicker, ttlValueToPickerParts } from '@/lib/table/ttl-picker' import { getTimezoneEditBlockedMessage } from '@/app/workspace/[workspaceId]/tables/[tableId]/components/timezone-editing' import { useTimezoneState } from '@/hooks/queries/general-settings' import type { SaveReason } from '../../../types' diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/utils.ts b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/utils.ts index c24363a0728..561c8d6f1fe 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/utils.ts +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/utils.ts @@ -1,8 +1,5 @@ +import { formatDateCellDisplay, normalizeDateCellValue } from '@sim/table-cells/dates' import { getWallClockParts } from '@sim/utils/timezone' -import { - formatDateCellDisplay, - normalizeDateCellValue, -} from '@sim/workspace-code/table-cells/dates' import type { ColumnDefinition, JsonValue } from '@/lib/table' import type { ColumnType } from '@/lib/table/column-types' import { columnTypeById, columnTypeOf } from '@/lib/table/column-types' diff --git a/apps/sim/background/cleanup-table-row-ttl.integration.ts b/apps/sim/background/cleanup-table-row-ttl.integration.ts index ea3653a0dae..843758205d0 100644 --- a/apps/sim/background/cleanup-table-row-ttl.integration.ts +++ b/apps/sim/background/cleanup-table-row-ttl.integration.ts @@ -22,10 +22,7 @@ vi.mock('@/lib/table/events', () => ({ signalTableRowsChanged: signalChanged })) vi.mock('@/lib/table/trigger', () => ({ fireTableTrigger: fireTrigger })) import { db } from '@sim/db' -import { - normalizeTtlTimestamp, - TTL_TIMESTAMP_VALIDATION, -} from '@sim/workspace-code/table-cells/ttl' +import { normalizeTtlTimestamp, TTL_TIMESTAMP_VALIDATION } from '@sim/table-cells/ttl' import { validatedTimestampSql } from '@/lib/table/column-types/timestamp-sql' import { updateColumnConstraints } from '@/lib/table/columns/service' import { getDeleteSnapshotBatchSize } from '@/lib/table/constants' diff --git a/apps/sim/background/cleanup-table-row-ttl.ts b/apps/sim/background/cleanup-table-row-ttl.ts index 6a94e45c0c2..e38c2ae0e44 100644 --- a/apps/sim/background/cleanup-table-row-ttl.ts +++ b/apps/sim/background/cleanup-table-row-ttl.ts @@ -1,7 +1,7 @@ import { dbFor } from '@sim/db' import { userTableDefinitions, userTableRows } from '@sim/db/schema' import { createLogger } from '@sim/logger' -import { TTL_TIMESTAMP_VALIDATION } from '@sim/workspace-code/table-cells/ttl' +import { TTL_TIMESTAMP_VALIDATION } from '@sim/table-cells/ttl' import { task } from '@trigger.dev/sdk' import { type SQL, sql } from 'drizzle-orm' import { asOrchestrationError } from '@/lib/core/orchestration/types' diff --git a/apps/sim/lib/chat-deployments/application/workflow-chat-deployment.ts b/apps/sim/lib/chat-deployments/application/workflow-chat-deployment.ts index 8fe59a1bc59..a3a2d64a7cd 100644 --- a/apps/sim/lib/chat-deployments/application/workflow-chat-deployment.ts +++ b/apps/sim/lib/chat-deployments/application/workflow-chat-deployment.ts @@ -29,8 +29,7 @@ import { buildChatDeploymentUrl } from '@/lib/chat-deployments/urls' import { defineAuthorizedWorkspaceUseCase } from '@/lib/core/application' import { resolveCopilotSecretReference } from '@/lib/core/application/environment-reference' import { OrchestrationError } from '@/lib/core/orchestration/types' -import { expectedDraft } from '@/lib/workflows/application/deployments' -import { resolvePrincipalWorkflowContext } from '@/lib/workflows/application/principal-scope' +import { withExpectedDraft } from '@/lib/workflows/application/deployments' import { performChatDeploy, performChatUndeploy } from '@/lib/workflows/orchestration' import { validateChatDeployAuth } from '@/ee/access-control/utils/permission-check' @@ -272,38 +271,32 @@ export const replaceWorkflowChatDeployment = defineAuthorizedWorkspaceUseCase({ const attribution = resolvePrincipalAttribution(principal, { workspaceBillingOwnerUserId: context.billedAccountUserId, }) - const draftCheck = - input.expectedDraftFingerprint !== undefined - ? expectedDraft( - principal, - await resolvePrincipalWorkflowContext({ - principal, - input: { workflowId: context.workflowId }, - }), - input.expectedDraftFingerprint - ) - : undefined - const result = await performChatDeploy({ - workflowId: context.workflowId, - userId: attribution.attributedUserId, - actorId: attribution.attributedUserId, - actor: toPrincipalActor(principal), - identifier: input.identifier, - title: input.title, - description: input.description ?? '', - customizations, - authType, - password, - allowedEmails, - outputConfigs, - includeThinking: input.includeThinking ?? false, - includeToolCalls: input.includeToolCalls ?? false, - workspaceId: context.workspaceId, - requestId: input.requestId, - projectLegacyAudit: false, - ...(draftCheck ? { admitDraft: draftCheck.admitDraft } : {}), - }).catch(chatIdentifierUniquenessConflict(input.identifier)) - draftCheck?.assertAdmitted() + const result = await withExpectedDraft( + principal, + context, + input.expectedDraftFingerprint, + (admission) => + performChatDeploy({ + workflowId: context.workflowId, + userId: attribution.attributedUserId, + actorId: attribution.attributedUserId, + actor: toPrincipalActor(principal), + identifier: input.identifier, + title: input.title, + description: input.description ?? '', + customizations, + authType, + password, + allowedEmails, + outputConfigs, + includeThinking: input.includeThinking ?? false, + includeToolCalls: input.includeToolCalls ?? false, + workspaceId: context.workspaceId, + requestId: input.requestId, + projectLegacyAudit: false, + ...admission, + }).catch(chatIdentifierUniquenessConflict(input.identifier)) + ) if (!result.success) { /** diff --git a/apps/sim/lib/table/column-conversion.test.ts b/apps/sim/lib/table/column-conversion.test.ts index c3078ec42c7..a1b2f06e904 100644 --- a/apps/sim/lib/table/column-conversion.test.ts +++ b/apps/sim/lib/table/column-conversion.test.ts @@ -3,7 +3,7 @@ * coverage and every case below was a shipped defect caught in review. */ -import { resolveSelectOptionId } from '@sim/workspace-code/table-cells/select' +import { resolveSelectOptionId } from '@sim/table-cells/select' import { describe, expect, it } from 'vitest' import { COLUMN_TYPE_REGISTRY } from '@/lib/table/column-types' import { diff --git a/apps/sim/lib/table/column-keys.ts b/apps/sim/lib/table/column-keys.ts index 58d9f006c60..516347f8b71 100644 --- a/apps/sim/lib/table/column-keys.ts +++ b/apps/sim/lib/table/column-keys.ts @@ -8,8 +8,8 @@ * map builders here. */ +import { cellKeyOf } from '@sim/table-cells/rows' import { generateId } from '@sim/utils/id' -import { cellKeyOf } from '@sim/workspace-code/table-cells/rows' import type { ColumnDefinition, Filter, diff --git a/apps/sim/lib/table/column-types/boolean.ts b/apps/sim/lib/table/column-types/boolean.ts index b056fd6bfbb..a31a25a2aa2 100644 --- a/apps/sim/lib/table/column-types/boolean.ts +++ b/apps/sim/lib/table/column-types/boolean.ts @@ -1,5 +1,5 @@ import { TypeBoolean } from '@sim/emcn/icons' -import { CELL_RULES } from '@sim/workspace-code/table-cells/rules' +import { CELL_RULES } from '@sim/table-cells/rules' import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' export const booleanColumnType: ColumnTypeDefinition = { diff --git a/apps/sim/lib/table/column-types/currency.ts b/apps/sim/lib/table/column-types/currency.ts index fcdd50d049e..7b8be863f99 100644 --- a/apps/sim/lib/table/column-types/currency.ts +++ b/apps/sim/lib/table/column-types/currency.ts @@ -1,12 +1,8 @@ import { TypeCurrency } from '@sim/emcn/icons' -import { - formatCurrencyDisplay, - formatCurrencyForInput, - isSupportedCurrencyCode, - resolveCurrencyCode, -} from '@sim/workspace-code/table-cells/currency' -import { CELL_RULES } from '@sim/workspace-code/table-cells/rules' +import { isSupportedCurrencyCode, resolveCurrencyCode } from '@sim/table-cells/currency' +import { CELL_RULES } from '@sim/table-cells/rules' import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' +import { formatCurrencyDisplay, formatCurrencyForInput } from '@/lib/table/currency-format' export const currencyColumnType: ColumnTypeDefinition = { id: 'currency', diff --git a/apps/sim/lib/table/column-types/date.ts b/apps/sim/lib/table/column-types/date.ts index 9205c0a4fec..da734b383e2 100644 --- a/apps/sim/lib/table/column-types/date.ts +++ b/apps/sim/lib/table/column-types/date.ts @@ -1,6 +1,6 @@ import { Calendar as CalendarIcon } from '@sim/emcn/icons' -import { formatDateCellDisplay, storedDateToEditable } from '@sim/workspace-code/table-cells/dates' -import { CELL_RULES } from '@sim/workspace-code/table-cells/rules' +import { formatDateCellDisplay, storedDateToEditable } from '@sim/table-cells/dates' +import { CELL_RULES } from '@sim/table-cells/rules' import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' export const dateColumnType: ColumnTypeDefinition = { diff --git a/apps/sim/lib/table/column-types/json.ts b/apps/sim/lib/table/column-types/json.ts index 6658d2f688c..7b97066da0d 100644 --- a/apps/sim/lib/table/column-types/json.ts +++ b/apps/sim/lib/table/column-types/json.ts @@ -1,5 +1,5 @@ import { TypeJson } from '@sim/emcn/icons' -import { CELL_RULES } from '@sim/workspace-code/table-cells/rules' +import { CELL_RULES } from '@sim/table-cells/rules' import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' export const jsonColumnType: ColumnTypeDefinition = { diff --git a/apps/sim/lib/table/column-types/number.ts b/apps/sim/lib/table/column-types/number.ts index 0509677bd9c..afb4ac87e1e 100644 --- a/apps/sim/lib/table/column-types/number.ts +++ b/apps/sim/lib/table/column-types/number.ts @@ -1,5 +1,5 @@ import { TypeNumber } from '@sim/emcn/icons' -import { CELL_RULES } from '@sim/workspace-code/table-cells/rules' +import { CELL_RULES } from '@sim/table-cells/rules' import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' export const numberColumnType: ColumnTypeDefinition = { diff --git a/apps/sim/lib/table/column-types/select.ts b/apps/sim/lib/table/column-types/select.ts index 0d4b9b6856d..b8df6ece469 100644 --- a/apps/sim/lib/table/column-types/select.ts +++ b/apps/sim/lib/table/column-types/select.ts @@ -1,9 +1,6 @@ import { TagIcon } from '@sim/emcn/icons' -import { CELL_RULES } from '@sim/workspace-code/table-cells/rules' -import { - resolveSelectOptionId, - splitMultiSelectInput, -} from '@sim/workspace-code/table-cells/select' +import { CELL_RULES } from '@sim/table-cells/rules' +import { resolveSelectOptionId, splitMultiSelectInput } from '@sim/table-cells/select' import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' import { MAX_SELECT_OPTIONS } from '@/lib/table/constants' import { selectValueToNames } from '@/lib/table/select-values' diff --git a/apps/sim/lib/table/column-types/string.ts b/apps/sim/lib/table/column-types/string.ts index 7b6db17b91e..39014631c08 100644 --- a/apps/sim/lib/table/column-types/string.ts +++ b/apps/sim/lib/table/column-types/string.ts @@ -1,5 +1,5 @@ import { TypeText } from '@sim/emcn/icons' -import { CELL_RULES } from '@sim/workspace-code/table-cells/rules' +import { CELL_RULES } from '@sim/table-cells/rules' import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' export const stringColumnType: ColumnTypeDefinition = { diff --git a/apps/sim/lib/table/column-types/ttl.test.ts b/apps/sim/lib/table/column-types/ttl.test.ts index 54f55577ec1..2816741eabe 100644 --- a/apps/sim/lib/table/column-types/ttl.test.ts +++ b/apps/sim/lib/table/column-types/ttl.test.ts @@ -2,15 +2,13 @@ import { isTtlTimestamp, normalizeTtlTimestamp, TTL_FORMAT_ERROR, - todayAtTtlOffset, ttlInstantForComparison, - ttlValueFromPicker, - ttlValueToPickerParts, -} from '@sim/workspace-code/table-cells/ttl' +} from '@sim/table-cells/ttl' import { describe, expect, it, vi } from 'vitest' import { isValueCompatible } from '@/lib/table/column-types' import { ttlColumnType } from '@/lib/table/column-types/ttl' import { retypeCellRewrite } from '@/lib/table/columns/service' +import { todayAtTtlOffset, ttlValueFromPicker, ttlValueToPickerParts } from '@/lib/table/ttl-picker' import type { ColumnDefinition, JsonValue } from '@/lib/table/types' import { coerceRowToSchema } from '@/lib/table/validation' diff --git a/apps/sim/lib/table/column-types/ttl.ts b/apps/sim/lib/table/column-types/ttl.ts index 1c65cd55b96..31a7c6e9d32 100644 --- a/apps/sim/lib/table/column-types/ttl.ts +++ b/apps/sim/lib/table/column-types/ttl.ts @@ -1,11 +1,11 @@ import { TypeTtl } from '@sim/emcn/icons' -import { CELL_RULES } from '@sim/workspace-code/table-cells/rules' +import { CELL_RULES } from '@sim/table-cells/rules' import { isTtlTimestamp, normalizeTtlTimestamp, TTL_FORMAT_ERROR, TTL_TIMESTAMP_VALIDATION, -} from '@sim/workspace-code/table-cells/ttl' +} from '@sim/table-cells/ttl' import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' export const ttlColumnType: ColumnTypeDefinition = { diff --git a/apps/sim/lib/table/column-types/types.ts b/apps/sim/lib/table/column-types/types.ts index 809b251361e..68d679b1345 100644 --- a/apps/sim/lib/table/column-types/types.ts +++ b/apps/sim/lib/table/column-types/types.ts @@ -20,7 +20,7 @@ */ import type React from 'react' -import { CELL_TYPES } from '@sim/workspace-code/table-cells/rules' +import { CELL_TYPES } from '@sim/table-cells/rules' import type { ColumnDefinition, JsonValue } from '@/lib/table/types' /** diff --git a/apps/sim/lib/table/columns/service.ts b/apps/sim/lib/table/columns/service.ts index 1dc11de4fd2..961d8ca0736 100644 --- a/apps/sim/lib/table/columns/service.ts +++ b/apps/sim/lib/table/columns/service.ts @@ -18,8 +18,8 @@ import { db } from '@sim/db' import { userTableDefinitions, userTableRows } from '@sim/db/schema' import { createLogger } from '@sim/logger' +import { resolveCurrencyCode } from '@sim/table-cells/currency' import { omit } from '@sim/utils/object' -import { resolveCurrencyCode } from '@sim/workspace-code/table-cells/currency' import { and, asc, count, eq, gt, sql } from 'drizzle-orm' import { OrchestrationError } from '@/lib/core/orchestration/types' import { columnMatchesRef, generateColumnId, getColumnId } from '@/lib/table/column-keys' diff --git a/apps/sim/lib/table/currency-format.test.ts b/apps/sim/lib/table/currency-format.test.ts new file mode 100644 index 00000000000..8038bb45bf6 --- /dev/null +++ b/apps/sim/lib/table/currency-format.test.ts @@ -0,0 +1,28 @@ +/** + * Guards for the `currency` column's display helpers: a rendered amount reads back as the same + * number, and a cell with no readable amount is never blanked. + */ +import { parseCurrencyInput } from '@sim/table-cells/currency' +import { describe, expect, it } from 'vitest' +import { formatCurrencyDisplay, formatCurrencyForInput } from '@/lib/table/currency-format' + +describe('formatCurrencyDisplay', () => { + it('round-trips through parseCurrencyInput', () => { + for (const amount of [0, 12, -12.5, 1234.56, 1234567.89]) { + const rendered = formatCurrencyDisplay(amount, 'USD', 'en-US') + expect(parseCurrencyInput(rendered)).toBe(amount) + } + }) + + it('renders an unreadable value verbatim rather than blanking it', () => { + // What a string → currency conversion can leave behind. + expect(formatCurrencyDisplay('pending', 'USD', 'en-US')).toBe('pending') + expect(formatCurrencyDisplay(null, 'USD', 'en-US')).toBe('') + }) +}) + +describe('formatCurrencyForInput', () => { + it('keeps unparseable text so an edit does not silently erase it', () => { + expect(formatCurrencyForInput('pending')).toBe('pending') + }) +}) diff --git a/apps/sim/lib/table/currency-format.ts b/apps/sim/lib/table/currency-format.ts new file mode 100644 index 00000000000..3b4bd72be76 --- /dev/null +++ b/apps/sim/lib/table/currency-format.ts @@ -0,0 +1,134 @@ +/** + * Display helpers for the `currency` column type: the grid's formatted amount, the editor's bare + * amount, and the column-config picker's code list. Parsing and validation live in + * `@sim/table-cells/currency`, shared with the CLI's offline row check. + */ + +import { + listSupportedCurrencyCodes, + parseCurrencyInput, + resolveCurrencyCode, +} from '@sim/table-cells/currency' +import { LRUCache } from 'lru-cache' + +/** + * Codes offered first in the picker. The rest of ICU's set is still selectable + * (and any valid code is accepted over the API) — these are just the ones worth + * reaching without typing. + */ +const PINNED_CURRENCY_CODES = [ + 'USD', + 'EUR', + 'GBP', + 'JPY', + 'CAD', + 'AUD', + 'CHF', + 'CNY', + 'INR', + 'BRL', +] as const + +/** One picker entry. */ +export interface CurrencyOption { + code: string + /** Localized currency name, e.g. `US Dollar`. Falls back to the code. */ + name: string +} + +/** + * Built on first call, not at module load: constructing `Intl.DisplayNames` and + * naming ~160 currencies costs several milliseconds of ICU work, and the only + * caller is the column-config sidebar — every table API route imports this + * module and would otherwise pay for a list it never reads. + */ +let currencyOptions: readonly CurrencyOption[] | null = null + +function currencyDisplayNames(): Intl.DisplayNames | null { + try { + return new Intl.DisplayNames(['en'], { type: 'currency' }) + } catch { + return null + } +} + +/** + * Codes for the column-config picker: the pinned set first, then every other + * code the runtime supports, alphabetically. + */ +export function getCurrencyOptions(): readonly CurrencyOption[] { + if (currencyOptions) return currencyOptions + const pinned = new Set(PINNED_CURRENCY_CODES) + const rest = listSupportedCurrencyCodes() + .filter((code) => !pinned.has(code)) + .sort() + const displayNames = currencyDisplayNames() + currencyOptions = [...PINNED_CURRENCY_CODES, ...rest].map((code) => ({ + code, + name: displayNames?.of(code) ?? code, + })) + return currencyOptions +} + +/** Formatters a grid keeps live: one per locale and code it paints. */ +const MAX_CACHED_FORMATTERS = 256 + +/** + * Formatters are cached by locale + code: a grid paints thousands of currency + * cells per scroll, and constructing an `Intl.NumberFormat` per cell is orders + * of magnitude more expensive than the format call itself. `false` records a + * code the runtime cannot format. + */ +const formatterCache = new LRUCache({ + max: MAX_CACHED_FORMATTERS, +}) + +function currencyFormatter( + currencyCode: string, + locale: string | undefined +): Intl.NumberFormat | null { + const key = `${locale ?? ''}:${currencyCode}` + const cached = formatterCache.get(key) + if (cached !== undefined) return cached || null + let formatter: Intl.NumberFormat | null + try { + formatter = new Intl.NumberFormat(locale, { style: 'currency', currency: currencyCode }) + } catch { + formatter = null + } + formatterCache.set(key, formatter ?? false) + return formatter +} + +/** + * Formats a stored cell for display — symbol placement and fraction digits come + * from the currency itself, so `JPY` renders `¥1,235` while `USD` renders + * `$1,234.56`. Values that carry no readable amount (a string left behind by a + * `string` → `currency` conversion, say) render verbatim rather than blanking, + * and an unformattable code degrades to `CODE amount`. + * + * `locale` is left to the caller's runtime by default, which means the viewer's + * own grouping/decimal conventions in the browser. + */ +export function formatCurrencyDisplay( + value: unknown, + currencyCode: string | undefined, + locale?: string +): string { + const amount = parseCurrencyInput(value) + if (amount === null) return typeof value === 'string' ? value : '' + const code = resolveCurrencyCode(currencyCode) + const formatter = currencyFormatter(code, locale) + return formatter ? formatter.format(amount) : `${code} ${amount}` +} + +/** + * Renders a stored cell for a text input: the bare amount, with no symbol or + * grouping, so editing round-trips through {@link parseCurrencyInput} exactly. + */ +export function formatCurrencyForInput(value: unknown): string { + if (value === null || value === undefined) return '' + const amount = parseCurrencyInput(value) + if (amount !== null) return String(amount) + return typeof value === 'string' ? value : '' +} diff --git a/apps/sim/lib/table/import.ts b/apps/sim/lib/table/import.ts index 72fc0c7f341..3738669530f 100644 --- a/apps/sim/lib/table/import.ts +++ b/apps/sim/lib/table/import.ts @@ -10,12 +10,9 @@ * parsers share {@link csvParseOptions} so their behavior can't drift. */ -import { parseCurrencyInput } from '@sim/workspace-code/table-cells/currency' -import { - type NormalizeDateCellOptions, - normalizeDateCellValue, -} from '@sim/workspace-code/table-cells/dates' -import { normalizeTtlTimestamp } from '@sim/workspace-code/table-cells/ttl' +import { parseCurrencyInput } from '@sim/table-cells/currency' +import { type NormalizeDateCellOptions, normalizeDateCellValue } from '@sim/table-cells/dates' +import { normalizeTtlTimestamp } from '@sim/table-cells/ttl' import type { Options as CsvParseOptions } from 'csv-parse' import { OrchestrationError } from '@/lib/core/orchestration/types' import { getColumnId } from '@/lib/table/column-keys' diff --git a/apps/sim/lib/table/index.ts b/apps/sim/lib/table/index.ts index 6e6bac7b212..3684f58623a 100644 --- a/apps/sim/lib/table/index.ts +++ b/apps/sim/lib/table/index.ts @@ -5,12 +5,13 @@ * Import hooks directly from '@/lib/table/hooks' in client components. */ -export * from '@sim/workspace-code/table-cells/currency' -export * from '@sim/workspace-code/table-cells/dates' +export * from '@sim/table-cells/currency' +export * from '@sim/table-cells/dates' export * from '@/lib/table/billing' export * from '@/lib/table/column-keys' export * from '@/lib/table/columns/service' export * from '@/lib/table/constants' +export * from '@/lib/table/currency-format' export * from '@/lib/table/errors' export * from '@/lib/table/ids' export * from '@/lib/table/import' diff --git a/apps/sim/lib/table/orchestration/columns.ts b/apps/sim/lib/table/orchestration/columns.ts index b803531ea7d..680ae82d316 100644 --- a/apps/sim/lib/table/orchestration/columns.ts +++ b/apps/sim/lib/table/orchestration/columns.ts @@ -1,6 +1,6 @@ import { AuditAction, AuditResourceType, recordAudit } from '@sim/audit' import { createLogger } from '@sim/logger' -import { isSupportedCurrencyCode } from '@sim/workspace-code/table-cells/currency' +import { isSupportedCurrencyCode } from '@sim/table-cells/currency' import { OrchestrationError, type OrchestrationErrorCode, diff --git a/apps/sim/lib/table/select-values.ts b/apps/sim/lib/table/select-values.ts index 4354a8fbab6..593684efdf3 100644 --- a/apps/sim/lib/table/select-values.ts +++ b/apps/sim/lib/table/select-values.ts @@ -9,8 +9,8 @@ * both the legacy `$` grammar and the v2 predicate tree. */ +import { resolveSelectOptionId } from '@sim/table-cells/select' import { isRecordLike } from '@sim/utils/object' -import { resolveSelectOptionId } from '@sim/workspace-code/table-cells/select' import { buildIdByName, getColumnId, predicateNamesToIds } from '@/lib/table/column-keys' import type { ColumnDefinition, diff --git a/apps/sim/lib/table/sql.ts b/apps/sim/lib/table/sql.ts index 2dae20aebf0..9719e7964d6 100644 --- a/apps/sim/lib/table/sql.ts +++ b/apps/sim/lib/table/sql.ts @@ -5,9 +5,9 @@ * Uses text extraction (->>) for comparisons and pattern matching. */ +import { normalizeDateCellValue } from '@sim/table-cells/dates' import { isRecordLike } from '@sim/utils/object' import { escapeLikePattern, truncate } from '@sim/utils/string' -import { normalizeDateCellValue } from '@sim/workspace-code/table-cells/dates' import type { SQL } from 'drizzle-orm' import { sql } from 'drizzle-orm' import { getColumnId } from '@/lib/table/column-keys' diff --git a/apps/sim/lib/table/ttl-picker.ts b/apps/sim/lib/table/ttl-picker.ts new file mode 100644 index 00000000000..076015cde8b --- /dev/null +++ b/apps/sim/lib/table/ttl-picker.ts @@ -0,0 +1,32 @@ +/** + * The expiration (`ttl`) column's date-picker helpers: split a stored timestamp into the picker's + * day, time and offset, join them back, and name today in that offset. The timestamp format itself + * lives in `@sim/table-cells/ttl`, shared with the CLI's offline row check. + */ + +import { normalizeTtlTimestamp } from '@sim/table-cells/ttl' + +/** Serializes picker fields in their existing offset. A day alone means midnight in that offset. */ +export function ttlValueFromPicker(day: string, time: string | null, offset = '-00:00'): string { + const seconds = time ? (time.length === 5 ? `${time}:00` : time) : '00:00:00' + return `${day}T${seconds}${offset}` +} + +/** Reads the stored clock and offset for the picker without converting the instant. */ +export function ttlValueToPickerParts(value: string): { + day: string | null + time: string | null + offset: string +} { + const normalized = normalizeTtlTimestamp(value) + return normalized + ? { day: normalized.slice(0, 10), time: normalized.slice(11, -6), offset: normalized.slice(-6) } + : { day: null, time: null, offset: '-00:00' } +} + +/** Today's calendar date in a fixed numeric offset, independent of profile timezone settings. */ +export function todayAtTtlOffset(offset: string): string { + const minutes = Number(offset.slice(1, 3)) * 60 + Number(offset.slice(4, 6)) + const signedMinutes = offset.startsWith('-') ? -minutes : minutes + return new Date(Date.now() + signedMinutes * 60_000).toISOString().slice(0, 10) +} diff --git a/apps/sim/lib/table/validation.ts b/apps/sim/lib/table/validation.ts index 2d524fab1c5..a8fdafe953b 100644 --- a/apps/sim/lib/table/validation.ts +++ b/apps/sim/lib/table/validation.ts @@ -4,7 +4,6 @@ import { db } from '@sim/db' import { userTableRows } from '@sim/db/schema' -import { chunkArray } from '@sim/utils/helpers' import { type UncoercibleValuePolicy as CellUncoercibleValuePolicy, cellOf as cellAt, @@ -12,11 +11,9 @@ import { coerceRowCells, rowCellErrors, scanBatchUniqueValues, -} from '@sim/workspace-code/table-cells/rows' -import { - resolveSelectOptionId, - splitMultiSelectInput, -} from '@sim/workspace-code/table-cells/select' +} from '@sim/table-cells/rows' +import { resolveSelectOptionId, splitMultiSelectInput } from '@sim/table-cells/select' +import { chunkArray } from '@sim/utils/helpers' import { and, eq, or, type SQL, sql } from 'drizzle-orm' import { NextResponse } from 'next/server' import { getColumnId } from '@/lib/table/column-keys' @@ -50,7 +47,7 @@ export type { ColumnDefinition, TableSchema, ValidationResult } /** * Re-exported so existing importers keep working; the implementations live in - * `@sim/workspace-code/table-cells/select`, free of this module's drizzle dependency. + * `@sim/table-cells/select`, free of this module's drizzle dependency. */ export { resolveSelectOptionId, splitMultiSelectInput } diff --git a/apps/sim/lib/workflows/api/index.ts b/apps/sim/lib/workflows/api/index.ts index d87ea6cedd8..37b1a68c40f 100644 --- a/apps/sim/lib/workflows/api/index.ts +++ b/apps/sim/lib/workflows/api/index.ts @@ -7,6 +7,8 @@ export { v2WorkflowErrorPolicies, WORKFLOW_NOT_FOUND_MESSAGE, } from '@/lib/workflows/api/route-policies' +export { withBoundSourceFindings } from '@/lib/workflows/api/source-findings' +export { presentSourceOutcome } from '@/lib/workflows/api/source-presenters' export { withSourceSlot } from '@/lib/workflows/api/source-slots' export { presentV2WorkflowTrigger, diff --git a/apps/sim/lib/workflows/api/route-policies.ts b/apps/sim/lib/workflows/api/route-policies.ts index b813b4b55b0..bcab1ba5fce 100644 --- a/apps/sim/lib/workflows/api/route-policies.ts +++ b/apps/sim/lib/workflows/api/route-policies.ts @@ -4,6 +4,7 @@ import type { SessionPrincipal, WorkspaceApiKeyPrincipal, } from '@sim/auth/principal' +import { boundAnswerFindings } from '@sim/workflow-code/diagnostics' import { v2CancelWorkflowRunDataSchema } from '@/lib/api/contracts/v2/workflows' import { createInternalResourceConcealmentPolicy, @@ -108,7 +109,7 @@ export const v2WorkflowErrorPolicies = { if (error instanceof WorkflowSourceRejectedError) { return v2ErrorForOrchestration(error.code, error.message, { code: 'SOURCE_REJECTED', - diagnostics: error.diagnostics, + diagnostics: boundAnswerFindings(error.diagnostics), }) } if (error instanceof WorkflowSourceStaleError) { @@ -138,7 +139,7 @@ export const v2WorkflowErrorPolicies = { if (error instanceof WorkflowSourceRejectedError) { return v2ErrorForOrchestration(error.code, error.message, { code: 'SOURCE_REJECTED', - diagnostics: error.diagnostics, + diagnostics: boundAnswerFindings(error.diagnostics), }) } if (error instanceof WorkflowSourceNameTakenError) { diff --git a/apps/sim/lib/workflows/api/source-findings.ts b/apps/sim/lib/workflows/api/source-findings.ts new file mode 100644 index 00000000000..2899d34b127 --- /dev/null +++ b/apps/sim/lib/workflows/api/source-findings.ts @@ -0,0 +1,15 @@ +import { boundAnswerFindings, type DiagnosticSeverity } from '@sim/workflow-code/diagnostics' + +/** `result` with its `diagnostics` bounded for the answer (`boundAnswerFindings`). */ +export function withBoundSourceFindings< + R extends { + diagnostics: ReadonlyArray<{ + code: string + severity: DiagnosticSeverity + message: string + hint?: string + }> + }, +>(result: R) { + return { ...result, diagnostics: boundAnswerFindings(result.diagnostics) } +} diff --git a/apps/sim/lib/workflows/api/source-presenters.ts b/apps/sim/lib/workflows/api/source-presenters.ts new file mode 100644 index 00000000000..9a1539a5672 --- /dev/null +++ b/apps/sim/lib/workflows/api/source-presenters.ts @@ -0,0 +1,29 @@ +import { boundAnswerFindings } from '@sim/workflow-code/diagnostics' +import type { ContractJsonResponse } from '@/lib/api/contracts/types' +import type { v2ApplyWorkflowSourceContract } from '@/lib/api/contracts/v2/workflow-source' +import type { SourceOutcome } from '@/lib/workflows/application/source-pipeline' + +/** The v2 wire shape of a source apply, plan or create outcome: the same for all three routes. */ +type SourceOutcomeData = ContractJsonResponse['data'] + +/** + * The v2 wire shape of a source write's outcome: exactly the fields the source routes' response + * schema exposes, so in-process fields (the workflow name, references, source meta flags) never + * reach the wire. Its findings are bounded here, once (`boundAnswerFindings`). + */ +export function presentSourceOutcome(outcome: SourceOutcome): SourceOutcomeData { + return { + workflowId: outcome.workflowId, + workspaceId: outcome.workspaceId, + dryRun: outcome.dryRun, + applied: outcome.applied, + diff: outcome.diff, + diagnostics: boundAnswerFindings(outcome.diagnostics), + files: outcome.files, + lock: outcome.lock, + base: outcome.base, + mintedIds: outcome.mintedIds, + notes: outcome.notes, + needsRedeployment: outcome.needsRedeployment, + } +} diff --git a/apps/sim/lib/workflows/application/apply-workflow-source.credentials.integration.ts b/apps/sim/lib/workflows/application/apply-workflow-source.credentials.integration.ts index 98e04b4c62a..e7cd279c789 100644 --- a/apps/sim/lib/workflows/application/apply-workflow-source.credentials.integration.ts +++ b/apps/sim/lib/workflows/application/apply-workflow-source.credentials.integration.ts @@ -236,6 +236,29 @@ check( } ) +check('a dry run over a stored credential that has no key yet allocates none for it', async () => { + const credentialId = await seedHiddenCredential() + const { workflowId } = await seedWorkflow(credentialId) + const source = await getSource(reader, workflowId) + expect(await credentialKeyRows(credentialId)).toEqual([]) + + const answer = await withPermissionGroupScope(() => + applyWorkflowSource.execute({ + principal: owner, + input: { + workflowId, + files: edit(source.files, 'js`return 1`', 'js`return 2`'), + lock: source.lock, + base: source.base, + dryRun: true, + }, + }) + ) + + expect(answer.dryRun).toBe(true) + expect(await credentialKeyRows(credentialId)).toEqual([]) +}) + check('a reader never sees the key an admin already allocated from the name', async () => { const credentialId = await seedHiddenCredential() const { workflowId } = await seedWorkflow(credentialId) diff --git a/apps/sim/lib/workflows/application/apply-workflow-source.integration.ts b/apps/sim/lib/workflows/application/apply-workflow-source.integration.ts index e9106767534..8f72c8c20cb 100644 --- a/apps/sim/lib/workflows/application/apply-workflow-source.integration.ts +++ b/apps/sim/lib/workflows/application/apply-workflow-source.integration.ts @@ -11,6 +11,7 @@ */ import { mkdir, writeFile } from 'node:fs/promises' import { dirname, resolve } from 'node:path' +import type { Principal } from '@sim/auth/principal' import { db, runOutsideTransactionContext } from '@sim/db' import { auditLog, @@ -39,6 +40,7 @@ import { getErrorMessage } from '@sim/utils/errors' import { sleep } from '@sim/utils/helpers' import { generateId } from '@sim/utils/id' import { isRecordLike, omit } from '@sim/utils/object' +import { compareStrings } from '@sim/utils/string' import { SOURCE_LIMITS } from '@sim/workflow-code/limits' import { saveWorkflowToNormalizedTables } from '@sim/workflow-persistence/save' import { CONTAINER_DIMENSIONS } from '@sim/workflow-renderer/dimensions' @@ -48,6 +50,7 @@ import { isEqual } from 'es-toolkit' import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' import type { DbOrTx } from '@/lib/db/types' import { withPermissionGroupScope } from '@/lib/permission-groups/request-scope.server' +import { createTableUseCase } from '@/lib/table/application/tables' import { type ApplyWorkflowSourceInput, applyWorkflowSource, @@ -56,11 +59,18 @@ import { } from '@/lib/workflows/application/apply-workflow-source' import { readWorkflowSource } from '@/lib/workflows/application/read-workflow-source' import { WorkflowSourceRejectedError } from '@/lib/workflows/application/source-error' +import { + type PendingResource, + PlanBindingError, + type PlannedSource, + workflowSourceUpdate, +} from '@/lib/workflows/application/source-pipeline' import { updateWorkflow } from '@/lib/workflows/application/update-workflow' import { loadWorkflowDeploymentSnapshot, loadWorkflowFromNormalizedTables, } from '@/lib/workflows/persistence/utils' +import { claimResourceKey } from '@/lib/workspaces/resource-keys' vi.mock('@/lib/core/config/feature-flags', () => featureFlagsMock) vi.mock('@/lib/realtime/notify', () => realtimeNotifyMock) @@ -1302,7 +1312,22 @@ describe('apply edges: identity, row lock, live triggers', () => { isRecordLike(properties) && properties.workflow_id === seeded.workflowId ) - await expect(run({ planOnly: true })).rejects.toMatchObject({ code: 'locked' }) + await expect( + withPermissionGroupScope(() => + workflowSourceUpdate.plan.execute({ + principal: createPersonalApiKeyPrincipal({ userId: ownerId, keyId: generateId() }), + input: { + workflowId: seeded.workflowId, + files: edit(source.files, SCORE_CODE, 'return 24'), + lock: source.lock, + base: source.base, + allowDelete: false, + pendingResources: [], + loads: {}, + }, + }) + ) + ).rejects.toMatchObject({ code: 'locked' }) expect(recorded()).toEqual([]) await expect(run({})).rejects.toMatchObject({ code: 'locked' }) @@ -1696,27 +1721,196 @@ export default workflow('', {}, () => { } ) - /** A `planOnly` evaluation of `files`, to commit after a concurrent write lands. */ - async function planned(source: ReadWorkflowSourceResult, files: Record) { - const plan = await apply(source, files, { planOnly: true }) - if (!plan.evaluated) throw new Error('The plan was refused') - return plan.evaluated + /** A workspace apply's plan of `files`, to commit after a concurrent write lands. */ + async function planned( + source: ReadWorkflowSourceResult, + files: Record, + planner: Principal = principal, + pendingResources: PendingResource[] = [] + ) { + const plan = await withPermissionGroupScope(() => + workflowSourceUpdate.plan.execute({ + principal: planner, + input: { + workflowId: source.workflowId, + files, + lock: source.lock, + base: source.base, + allowDelete: false, + pendingResources, + loads: {}, + }, + }) + ) + if (plan.kind === 'refused') throw new Error('The plan was refused') + return plan + } + + /** Commits a plan as a workspace apply does. */ + function commit(plan: PlannedSource, committer: Principal = principal) { + return withPermissionGroupScope(() => + workflowSourceUpdate.commit.execute({ principal: committer, input: { planned: plan } }) + ) + } + + check( + 'a plan made in one session is refused when another session of the same user commits it', + async () => { + const seeded = await seedWorkflow() + const source = await getSource(seeded.workflowId) + const files = edit(source.files, SCORE_CODE, 'return 44') + const plan = await planned(source, files) + const before = await snapshot(seeded.workflowId) + + expect( + await bindingRefusal( + commit(plan, createSessionPrincipal({ userId: ownerId, sessionId: generateId() })) + ) + ).toBe('PLAN_PRINCIPAL_MISMATCH') + expect(await snapshot(seeded.workflowId)).toEqual(before) + + expect((await commit(plan)).applied).toBe(true) + } + ) + + /** The binding code a commit refused with; fails the check on any other outcome. */ + async function bindingRefusal(committed: Promise): Promise { + const error = await committed.then( + () => null, + (rejection: unknown) => rejection + ) + expect(error).toBeInstanceOf(PlanBindingError) + return (error as PlanBindingError).code + } + + /** + * `source` with an Insert Row block writing the table keyed `key`, and the stand-in an enclosing + * workspace apply hands its plan for that table before creating it. + */ + function writingStandInTable(source: ReadWorkflowSourceResult, key: string) { + const withImport = Object.fromEntries( + Object.entries(source.files).map(([path, text]) => [ + path, + text.replace( + /import \{ ([^}]*) \} from '@sim\/workflow-sdk'/, + (_, names: string) => + `import { ${[...names.split(', '), 'table'].sort(compareStrings).join(', ')} } from '@sim/workflow-sdk'` + ), + ]) + ) + const files = edit( + withImport, + ' start.to(score)', + ` const save = table.insertRow('Save Order', { tableId: table('${key}'), data: { email: 'a@example.com' } })\n start.to(score)\n score.to(save)` + ) + const standIn: PendingResource = { + kind: 'table', + key, + id: generateId(), + placeholder: true, + name: key, + columns: ['email'], + } + return { files, standIn } } + /** Creates the table keyed `key`, as the enclosing apply does before the workflows. */ + async function createKeyedTable(key: string) { + const created = await createTableUseCase.execute({ + principal, + input: { + workspaceId, + name: key, + schema: { columns: [{ name: 'email', type: 'string' }] }, + }, + }) + await claimResourceKey({ + workspaceId, + kind: 'table', + key, + resourceId: created.table.id, + origin: 'adopted', + createdBy: ownerId, + }) + } + + check( + 'a plan that resolved a stand-in id is refused, before it is planned again, when another session commits it', + async () => { + const seeded = await seedWorkflow() + const source = await getSource(seeded.workflowId) + const { files, standIn } = writingStandInTable(source, 'orders') + const plan = await planned(source, files, principal, [standIn]) + expect(plan.replan).toBeDefined() + await createKeyedTable('orders') + const before = await snapshot(seeded.workflowId) + + expect( + await bindingRefusal( + commit(plan, createSessionPrincipal({ userId: ownerId, sessionId: generateId() })) + ) + ).toBe('PLAN_PRINCIPAL_MISMATCH') + expect(await snapshot(seeded.workflowId)).toEqual(before) + + expect((await commit(plan)).applied).toBe(true) + } + ) + + check( + 'a plan whose stand-in resource was never created is refused at commit, not written with the stand-in id', + async () => { + const seeded = await seedWorkflow() + const source = await getSource(seeded.workflowId) + const { files, standIn } = writingStandInTable(source, 'invoices') + const plan = await planned(source, files, principal, [standIn]) + const before = await snapshot(seeded.workflowId) + + await expect(commit(plan)).rejects.toBeInstanceOf(WorkflowSourceRejectedError) + expect(await snapshot(seeded.workflowId)).toEqual(before) + } + ) + + check('a dry-run create under a name its folder already holds is refused at plan', async () => { + const seeded = await seedWorkflow() + const taken = await storedName(seeded.workflowId) + + const created = withPermissionGroupScope(() => + createWorkflowFromSource.execute({ + principal, + input: { + workspaceId, + name: taken, + dryRun: true, + files: { + 'workflow.ts': `// @sim-sdk 1 +import { trigger, workflow } from '@sim/workflow-sdk' + +export default workflow('Fresh', {}, () => { + trigger.start('Start', {}) +}) +`, + }, + }, + }) + ) + + await expect(created).rejects.toMatchObject({ name: 'WorkflowSourceNameTakenError' }) + }) + check( 'a source rename committed after a teammate renamed the workflow is a 409 and keeps their name', async () => { const seeded = await seedWorkflow() const source = await getSource(seeded.workflowId) const files = edit(source.files, `'${await storedName(seeded.workflowId)}'`, "'From source'") - const evaluated = await planned(source, files) + const plan = await planned(source, files) const teammate = `Teammate ${generateId()}` await updateWorkflow.execute({ principal, input: { workflowId: seeded.workflowId, name: teammate }, }) - await expect(apply(source, files, { evaluated })).rejects.toMatchObject({ + await expect(commit(plan)).rejects.toMatchObject({ name: 'WorkflowSourceStaleError', code: 'conflict', }) @@ -1733,14 +1927,14 @@ export default workflow('', {}, () => { const name = await storedName(seeded.workflowId) const contested = `Contested ${generateId()}` const files = edit(edit(source.files, `'${name}'`, `'${contested}'`), SCORE_CODE, 'return 43') - const evaluated = await planned(source, files) + const plan = await planned(source, files) const before = await snapshot(seeded.workflowId) await updateWorkflow.execute({ principal, input: { workflowId: other.workflowId, name: contested }, }) - await expect(apply(source, files, { evaluated })).rejects.toMatchObject({ + await expect(commit(plan)).rejects.toMatchObject({ name: 'WorkflowSourceNameTakenError', code: 'conflict', }) diff --git a/apps/sim/lib/workflows/application/apply-workflow-source.ts b/apps/sim/lib/workflows/application/apply-workflow-source.ts index 139b3bc146e..14ed886e67d 100644 --- a/apps/sim/lib/workflows/application/apply-workflow-source.ts +++ b/apps/sim/lib/workflows/application/apply-workflow-source.ts @@ -1,33 +1,7 @@ -import { AuditAction, AuditResourceType } from '@sim/audit' -import { - type Principal, - resolvePrincipalAttribution, - resolvePrincipalSubjectUserId, -} from '@sim/auth/principal' -import { db } from '@sim/db' -import { createLogger } from '@sim/logger' +import type { Principal } from '@sim/auth/principal' import { generateId } from '@sim/utils/id' -import type { - BlockCatalog, - CanonicalizeResult, - SourceBase, - SourceFingerprint, - WorkflowLockV1, - WorkflowSourceMetaV1, -} from '@sim/workflow-code' -import { readWorkflowSourceMeta } from '@sim/workflow-persistence/source-meta' -import type { WorkflowState } from '@sim/workflow-types/workflow' -import { isEqual } from 'es-toolkit' -import { - type CatalogGate, - resolveCatalogGate, - withCatalogBlockScope, -} from '@/lib/catalog/application/context' -import { principalAuditSource } from '@/lib/core/application' -import type { DbTransaction } from '@/lib/db/types' -import { notifyWorkflowUpdated, notifyWorkspaceWorkflowsChanged } from '@/lib/realtime/notify' +import type { SourceBase, WorkflowLockV1 } from '@sim/workflow-code' import { defineAuthorizedWorkflowUseCase } from '@/lib/workflows/application/authorized-workflow-use-case' -import type { ActiveWorkflowApplicationContext } from '@/lib/workflows/application/context' import { resolveWorkflowFolderPath } from '@/lib/workflows/application/folders' import { requireMutableFolder, @@ -35,117 +9,34 @@ import { } from '@/lib/workflows/application/mutability' import { workflowOperations } from '@/lib/workflows/application/operations' import { resolvePrincipalWorkflowContext } from '@/lib/workflows/application/principal-scope' -import { loadStoredSource } from '@/lib/workflows/application/read-workflow-source' import { recordSourceDryRun, recordSourceRefused, - recordSourceWritten, } from '@/lib/workflows/application/source-analytics' import { - commitSourceCreate, - commitSourceMeta, - commitSourceReplace, - previewSourceCommit, -} from '@/lib/workflows/application/source-commit' -import { emptyDiffSummary, type SourceDiffSummary } from '@/lib/workflows/application/source-diff' -import { - type WorkflowSourceDiagnostic, - WorkflowSourceRejectedError, - WorkflowSourceStaleError, - workflowSourceStaleMessage, -} from '@/lib/workflows/application/source-error' -import { - evaluateSource, - type PendingResource, - type SourcePlan, - type SourceReference, -} from '@/lib/workflows/application/source-evaluate' -import { buildSourceCatalog, storedCatalogTypes } from '@/lib/workflows/code/catalog/workspace' -import { - type PrintedSource, - printStoredSource, - type StoredSource, - storedSourceFingerprint, -} from '@/lib/workflows/code/print-stored' -import { loadEnvNames } from '@/lib/workflows/code/resources' -import { - canonicalizeStored, - prefetchStoredResources, - type StoredCanonical, -} from '@/lib/workflows/code/stored-canonical' -import { - checkDraftNeedsRedeployment, - checkNeedsRedeployment, -} from '@/lib/workflows/deployment-status' -import { - type ActiveWorkspaceApplicationContext, - resolveActiveWorkspaceApplicationContext, -} from '@/lib/workspaces/application/context' - -const logger = createLogger('ApplyWorkflowSource') - -/** The stored draft a create applies onto: an empty graph. */ -const EMPTY_STORED_SOURCE: StoredSource = { - state: { blocks: {}, edges: [], loops: {}, parallels: {}, variables: {} }, - meta: { name: '', description: null }, -} - -/** What every source submission carries, whether it updates a draft or creates a workflow. */ -interface SourceSubmission { - /** Repo-relative path → text: `workflow.ts` plus any function-code files it references. */ - files: Record - /** Run the whole pipeline and stop before the write. Never audits or notifies. */ - dryRun?: boolean - provenance?: { client: 'cli' | 'copilot' | 'canvas' | 'api'; gitSha?: string; repo?: string } - /** Resources the enclosing workspace source apply creates or adopts first (`evaluateSource`). */ - pendingResources?: readonly PendingResource[] - /** - * Evaluate and stop, for a caller that commits later through `evaluated`: a dry run without the - * deployment check or the re-print of what a write would store, so the result's `files` echo - * the submission and `needsRedeployment` is false. A create still checks its name is free. It - * records no analytics: the caller's own commit (or its dry run) is the outcome. - */ - planOnly?: boolean - /** - * A `planOnly` run's evaluation of this same submission: the write commits it without loading or - * evaluating again. The commit still re-checks the fingerprint under the workflow row lock, so a - * draft that moved since is a 409 as ever. In-process only, never on a wire contract. - */ - evaluated?: EvaluatedSource - /** Loads shared by every workflow one request evaluates (see {@link SourceRequestShare}). */ - shared?: SourceRequestShare -} - -/** - * Loads that are the same for every workflow one request evaluates in one workspace for one - * principal (a workspace source apply): each starts on first use and is reused after. Never share - * one across principals or workspaces. - */ -export interface SourceRequestShare { - catalogGate?: Promise - envNames?: Promise -} - -/** - * What a `planOnly` run evaluated, handed back as `evaluated` so the write commits it as is: the - * stored draft and catalog it was evaluated against, the fingerprint the commit re-checks, and - * the plan. - */ -export interface EvaluatedSource { - workflowId: string - gate: CatalogGate - catalog: BlockCatalog - stored: StoredSource - storedSourceMeta: WorkflowSourceMetaV1 | null - storedCanonical: StoredCanonical - /** The stored draft's fingerprint when evaluated; the write refuses a base that differs. */ - storedFingerprint: SourceFingerprint - plan: SourcePlan -} - -export interface ApplyWorkflowSourceInput extends SourceSubmission { + afterSourceApply, + afterSourceCreate, + commitSource, + EMPTY_LOCK, + planSource, + previewSource, + refusalError, + refusedOutcome, + type SourceOutcome, + type SourceProvenance, + type SourceTarget, + sourceAnalyticsFacts, + sourceApplyAudit, + sourceCreateAudit, +} from '@/lib/workflows/application/source-pipeline' +import { checkNeedsRedeployment } from '@/lib/workflows/deployment-status' +import { resolveActiveWorkspaceApplicationContext } from '@/lib/workspaces/application/context' + +export interface ApplyWorkflowSourceInput { workflowId: string assertedWorkspaceId?: string + /** Repo-relative path → text: `workflow.ts` plus any function-code files it references. */ + files: Record /** The identity lock returned by `source get` (or a previous apply). */ lock: WorkflowLockV1 /** @@ -155,145 +46,101 @@ export interface ApplyWorkflowSourceInput extends SourceSubmission { base: SourceBase /** Allow removing blocks the source no longer declares (`--allow-delete`); refused otherwise. */ allowDelete?: boolean + /** Run the whole pipeline and stop before the write. Never audits or notifies. */ + dryRun?: boolean + provenance?: SourceProvenance } -export interface CreateWorkflowFromSourceInput extends SourceSubmission { +export interface CreateWorkflowFromSourceInput { workspaceId: string - /** - * The id the new workflow takes, minted by an enclosing workspace source apply so its other - * workflows name this one before it is written; a fresh id when omitted. In-process only, never - * on a wire contract. - */ - workflowId?: string + /** Repo-relative path → text: `workflow.ts` plus any function-code files it references. */ + files: Record /** The new workflow's name; the source's `workflow('…')` name when omitted. */ name?: string /** The new workflow's description; the source's when omitted. */ description?: string | null /** Containing folder path; the workspace root when omitted. */ folderPath?: string - /** - * Runs inside the create's transaction once the workflow and its graph are written, so what it - * writes (a workspace source apply binding the folder's key) commits or rolls back with the - * workflow. In-process only, never on a wire contract. - */ - withinCreate?: (tx: DbTransaction, workflowId: string) => Promise + /** Run the whole pipeline and stop before the write. Never audits or notifies. */ + dryRun?: boolean + provenance?: SourceProvenance } -interface ApplyWorkflowSourceResult { - workflowId: string - workflowName: string - workspaceId: string - dryRun: boolean - /** True when the apply wrote a graph, or renamed or redescribed the workflow. */ - applied: boolean - /** True when source meta was stored: with a graph write, or alone for a graph no-op. */ - wroteSourceMeta: boolean - /** - * True when a graph no-op still changes source meta (idents, order or comments): stored by a - * commit, and on a dry run what a commit would store. - */ - sourceMetaChanged: boolean - diff: SourceDiffSummary - /** Warnings on success; on a dry run, every finding including the errors a commit would refuse. */ - diagnostics: WorkflowSourceDiagnostic[] - /** Re-printed from what is (or would be) stored, so normalized values show in the source. */ - files: Record - lock: WorkflowLockV1 - /** Where what is (or would be) stored was read: the next apply's `base`. */ - base: SourceBase - /** Source key → id for every new block whose id was minted rather than taken from the lock. */ - mintedIds: Record - /** - * Every resource the submitted source names by key, as the compiler read it; empty when the - * source did not compile. - */ - references: SourceReference[] - /** - * Every env name the submitted source reads, as the compiler read it; empty when the source did - * not compile. - */ - envReferences: string[] - notes: string[] - needsRedeployment: boolean - sdkMajor: number - /** On a `planOnly` run that passed every check, what the write commits through `evaluated`. */ - evaluated?: EvaluatedSource +/** + * Whether the draft as it is differs from its deployment, for a dry run that leaves it so (a no-op + * or a refusal): it writes nothing, so this is known now and runs beside the plan. A new workflow + * has no deployment. A dry run that would change the draft compares what it would write instead. + * The `catch` only keeps it observed when an error throws first or it goes unused. + */ +function draftNeedsRedeployment(target: SourceTarget): Promise { + if (target.kind === 'new') return Promise.resolve(false) + const check = checkNeedsRedeployment(target.context.workflowId) + check.catch(() => {}) + return check } /** - * Where a source apply writes: the stored draft of an existing workflow, or a new workflow the - * apply creates, with an empty graph standing in for the stored one. + * Plans the submission onto `target`, then answers: a refused plan throws (409 when stale, else + * 400), or on a dry run answers 200 with its findings; a planned one is previewed on a dry run + * and committed otherwise. */ -type SourceApplyTarget = - | { - kind: 'existing' - context: ActiveWorkflowApplicationContext - base: SourceBase - allowDelete: boolean - } - | { - kind: 'new' - workspace: ActiveWorkspaceApplicationContext - workflowId: string - create: { name?: string; description?: string | null } - folderId: string | null - withinCreate?: CreateWorkflowFromSourceInput['withinCreate'] - } +async function writeSource( + principal: Principal, + target: SourceTarget, + submission: { files: Record; lock: WorkflowLockV1 }, + dryRun: boolean +): Promise { + if (!dryRun) { + const planned = await planSource(principal, target, submission, { allocate: true }) + if (planned.kind === 'refused') throw refusalError(planned) + return commitSource(principal, planned, target) + } + const needsRedeployment = draftNeedsRedeployment(target) + const planned = await planSource(principal, target, submission, { allocate: false }) + return planned.kind === 'refused' + ? refusedOutcome(planned, target, submission, needsRedeployment) + : previewSource(principal, planned, needsRedeployment) +} /** * Applies workflow source (TypeScript) to the stored draft. The server compiles the files - * against its live per-workflow catalog and evaluates them onto the stored graph - * (`evaluateSource`); any error refuses the whole write, with source lines. The write is one - * replace under the workflow row lock that re-checks the fingerprint (409 when the draft moved) - * and stores the submission's idents, order and comments in source meta. A source that changes - * only those is a graph no-op that stores just source meta. Draft only: it never deploys. + * against its live per-workflow catalog and evaluates them onto the stored graph; any error + * refuses the whole write, with source lines. The write is one replace under the workflow row + * lock that re-checks the fingerprint (409 when the draft moved) and stores the submission's + * idents, order and comments in source meta. A source that changes only those is a graph no-op + * that stores just source meta. Draft only: it never deploys. */ export const applyWorkflowSource = defineAuthorizedWorkflowUseCase({ operation: workflowOperations.applySource, resolveContext: resolvePrincipalWorkflowContext, - async execute({ principal, input, context }): Promise { - const facts = analyticsFacts(principal, input, 'apply', context.workspaceId) + async execute({ principal, input, context }): Promise { + const dryRun = input.dryRun === true + const facts = sourceAnalyticsFacts( + principal, + input.provenance, + 'apply', + context.workspaceId, + dryRun + ) try { await requireMutableWorkflow(context.workflowId) - const result = await runSourceApply(principal, input, { - kind: 'existing', - context, - base: input.base, - allowDelete: input.allowDelete === true, - }) - if (facts.dryRun) recordSourceDryRun({ ...facts, workflowId: context.workflowId }, result) + const result = await writeSource( + principal, + { kind: 'existing', context, base: input.base, allowDelete: input.allowDelete === true }, + { files: input.files, lock: input.lock }, + dryRun + ) + if (dryRun) recordSourceDryRun({ ...facts, workflowId: context.workflowId }, result) return result } catch (error) { - if (!input.planOnly) recordSourceRefused({ ...facts, workflowId: context.workflowId }, error) + recordSourceRefused({ ...facts, workflowId: context.workflowId }, error) throw error } }, - /** Only a committed, non-empty apply is a change worth auditing. */ projectAudit: ({ principal, input, context, result }) => - result.applied - ? ({ - action: AuditAction.WORKFLOW_UPDATED, - resourceType: AuditResourceType.WORKFLOW, - resourceId: context.workflowId, - resourceName: result.workflowName, - description: `Applied workflow source to workflow "${result.workflowName}"`, - metadata: sourceAuditMetadata(principal, input, result), - } as const) - : [], - /** Source meta alone changes what an open Code tab prints, so it notifies too. */ - afterSuccess: async ({ principal, input, context, result }) => { - if (result.applied) { - recordSourceWritten(analyticsFacts(principal, input, 'apply', context.workspaceId), result) - } - if (!result.applied && !result.wroteSourceMeta) return - await Promise.all([ - notifyWorkflowUpdated(context.workflowId), - // A rename or new description changes how the workspace lists the workflow. - result.applied && result.diff.meta?.length - ? notifyWorkspaceWorkflowsChanged(context.workspaceId) - : undefined, - ]) - }, + sourceApplyAudit(principal, input.provenance, context.workflowId, result), + afterSuccess: ({ principal, input, context, result }) => + afterSourceApply(principal, input.provenance, context, result), }) /** What {@link planWorkflowSource} reads: a submission as an apply sends it, without its knobs. */ @@ -316,16 +163,12 @@ export interface PlanWorkflowSourceInput { export const planWorkflowSource = defineAuthorizedWorkflowUseCase({ operation: workflowOperations.planSource, resolveContext: resolvePrincipalWorkflowContext, - async execute({ principal, input, context }): Promise { - return runSourceApply( + async execute({ principal, input, context }): Promise { + return writeSource( principal, - { files: input.files, lock: input.lock, dryRun: true }, - { - kind: 'existing', - context, - base: input.base, - allowDelete: input.allowDelete === true, - } + { kind: 'existing', context, base: input.base, allowDelete: input.allowDelete === true }, + { files: input.files, lock: input.lock }, + true ) }, }) @@ -340,458 +183,45 @@ export const createWorkflowFromSource = defineAuthorizedWorkflowUseCase({ operation: workflowOperations.createFromSource, resolveContext: ({ input }: { input: CreateWorkflowFromSourceInput }) => resolveActiveWorkspaceApplicationContext(input.workspaceId), - async execute({ principal, input, context }): Promise { + async execute({ principal, input, context }): Promise { const { folderId } = await resolveWorkflowFolderPath( context.workspaceId, input.folderPath ?? '/' ) - const facts = analyticsFacts(principal, input, 'create', context.workspaceId) + const dryRun = input.dryRun === true + const facts = sourceAnalyticsFacts( + principal, + input.provenance, + 'create', + context.workspaceId, + dryRun + ) try { await requireMutableFolder(folderId) - const result = await runSourceApply( + const result = await writeSource( principal, - { ...input, lock: { version: 1, blocks: {}, variables: {} } }, { kind: 'new', workspace: context, - workflowId: input.evaluated?.workflowId ?? input.workflowId ?? generateId(), + workflowId: generateId(), create: { ...(input.name !== undefined ? { name: input.name } : {}), ...(input.description !== undefined ? { description: input.description } : {}), }, folderId, - ...(input.withinCreate ? { withinCreate: input.withinCreate } : {}), - } + }, + { files: input.files, lock: EMPTY_LOCK }, + dryRun ) - if (facts.dryRun) recordSourceDryRun({ ...facts, workflowId: '' }, result) + if (dryRun) recordSourceDryRun({ ...facts, workflowId: '' }, result) return result } catch (error) { - if (!input.planOnly) recordSourceRefused({ ...facts, workflowId: '' }, error) + recordSourceRefused({ ...facts, workflowId: '' }, error) throw error } }, projectAudit: ({ principal, input, result }) => - result.applied - ? ({ - action: AuditAction.WORKFLOW_CREATED, - resourceType: AuditResourceType.WORKFLOW, - resourceId: result.workflowId, - resourceName: result.workflowName, - description: `Created workflow "${result.workflowName}" from source`, - metadata: sourceAuditMetadata(principal, input, result), - } as const) - : [], - afterSuccess: async ({ principal, input, result }) => { - if (!result.applied) return - recordSourceWritten(analyticsFacts(principal, input, 'create', result.workspaceId), result) - await Promise.all([ - notifyWorkflowUpdated(result.workflowId), - notifyWorkspaceWorkflowsChanged(result.workspaceId), - ]) - }, + sourceCreateAudit(principal, input.provenance, result), + afterSuccess: ({ principal, input, result }) => + afterSourceCreate(principal, input.provenance, result), }) - -function analyticsFacts( - principal: Principal, - input: SourceSubmission, - operation: 'apply' | 'create', - workspaceId: string -) { - return { - principal, - client: input.provenance?.client, - operation, - workspaceId, - dryRun: input.dryRun === true, - } -} - -/** The audit metadata a committed source write records, for update and create alike. */ -function sourceAuditMetadata( - principal: Principal, - input: SourceSubmission, - result: ApplyWorkflowSourceResult -) { - const { blocks, edges, variables } = result.diff - return { - op: 'apply_source', - blocksAdded: blocks.added.length, - blocksRemoved: blocks.removed.length, - blocksRenamed: blocks.renamed.length, - blocksModified: blocks.modified.length, - edgesAdded: edges.added, - edgesRemoved: edges.removed, - variablesChanged: - variables.added.length + - variables.removed.length + - variables.renamed.length + - variables.modified.length, - sdkMajor: result.sdkMajor, - catalogHash: result.base.catalogHash, - ...(input.provenance ? { client: input.provenance.client } : {}), - ...(input.provenance?.gitSha ? { gitSha: input.provenance.gitSha } : {}), - ...(input.provenance?.repo ? { repo: input.provenance.repo } : {}), - source: principalAuditSource(principal), - } -} - -/** - * The source-apply pipeline for either target: evaluate the submission onto the stored graph - * (empty for a new workflow), or take a `planOnly` run's evaluation, then write it: source meta - * alone for a graph no-op, a replace under the row lock for an existing draft, or a create that - * inserts the row and its graph in one transaction. A dry run stops before the write; a refusal - * throws, or on a dry run answers 200. - */ -async function runSourceApply( - principal: Principal, - input: SourceSubmission & { lock: WorkflowLockV1 }, - target: SourceApplyTarget -): Promise { - const context = - target.kind === 'existing' - ? target.context - : { ...target.workspace, workflowId: target.workflowId } - const isNew = target.kind === 'new' - const planOnly = input.planOnly === true - const dryRun = input.dryRun === true || planOnly - const subjectUserId = resolvePrincipalAttribution(principal, { - workspaceBillingOwnerUserId: context.billedAccountUserId, - }).attributedUserId - const createdBy = resolvePrincipalSubjectUserId(principal) ?? null - /** Who and where the write acts: what the evaluation gates and the commit re-checks run as. */ - const scope = { - workflowId: context.workflowId, - workspaceId: context.workspaceId, - workspaceOrganizationId: context.workspaceOrganizationId, - subjectUserId, - folderId: target.kind === 'existing' ? target.context.workflow.folderId : target.folderId, - } - if (input.evaluated && input.evaluated.workflowId !== context.workflowId) { - throw new Error('An evaluated source commits only to the workflow it was evaluated for') - } - - const shared: SourceRequestShare = input.shared ?? {} - const loaded = input.evaluated - ? null - : await Promise.all([ - (shared.catalogGate ??= resolveCatalogGate(principal, context, { - customBlockInputs: true, - })), - target.kind === 'existing' ? loadStoredSource(target.context) : EMPTY_STORED_SOURCE, - target.kind === 'existing' ? readWorkflowSourceMeta(db, context.workflowId) : null, - prefetchStoredResources({ - workflowId: context.workflowId, - workspaceId: context.workspaceId, - createdBy, - allocate: !dryRun, - }), - (shared.envNames ??= loadEnvNames(context.workspaceId, subjectUserId)), - ]) - /** - * Whether the draft differs from its deployment, for a dry run that leaves it as it is (a no-op - * or a refusal): it writes nothing, so this is known now and runs beside the evaluation. A dry - * run that would change the draft compares what it would write instead. The `catch` only keeps - * it observed when an error throws first or it goes unused. - */ - const dryRunNeedsRedeployment = - dryRun && !planOnly && !isNew ? checkNeedsRedeployment(context.workflowId) : null - dryRunNeedsRedeployment?.catch(() => {}) - const gate = input.evaluated?.gate ?? loaded?.[0] - if (!gate) throw new Error('A source apply needs its catalog gate') - return withCatalogBlockScope(gate, async () => { - /** The one response shape: what is (or would be) stored, printed, and the outcome. */ - const result = async ( - evaluated: Pick, - printed: Pick & Pick, - outcome: Pick & - Partial< - Pick< - ApplyWorkflowSourceResult, - | 'workflowName' - | 'mintedIds' - | 'references' - | 'envReferences' - | 'notes' - | 'wroteSourceMeta' - | 'sourceMetaChanged' - | 'evaluated' - | 'needsRedeployment' - > - > - ): Promise => ({ - workflowId: context.workflowId, - workflowName: evaluated.stored.meta.name, - workspaceId: context.workspaceId, - dryRun, - mintedIds: {}, - references: [], - envReferences: [], - notes: [], - wroteSourceMeta: false, - sourceMetaChanged: false, - ...outcome, - files: printed.files, - lock: printed.lock, - base: { - version: 1, - workflowId: context.workflowId, - workspaceId: context.workspaceId, - fingerprint: printed.fingerprint, - catalogHash: evaluated.catalog.hash, - }, - needsRedeployment: - outcome.needsRedeployment ?? - (!isNew && - !planOnly && - (await (dryRunNeedsRedeployment ?? checkNeedsRedeployment(context.workflowId)))), - sdkMajor: evaluated.catalog.sdkMajor, - }) - - let evaluated = input.evaluated - if (!evaluated) { - if (!loaded) throw new Error('A source apply needs its stored draft') - const [, stored, storedSourceMeta, prefetched, envNames] = loaded - const catalog = buildSourceCatalog({ - gate, - organizationId: context.workspaceOrganizationId, - storedTypes: storedCatalogTypes(Object.values(stored.state.blocks)), - }) - const storedCanonical = await canonicalizeStored({ - prefetched, - state: stored.state, - meta: stored.meta, - catalog, - }) - const storedFingerprint = storedSourceFingerprint( - storedCanonical.canonical, - storedCanonical.resources - ) - /** - * Refuses a commit; a dry run reports the same findings with a 200 instead. The dry run - * echoes the caller's own base fingerprint, so applying its response after a stale or - * mismatched read still fails the freshness check rather than overwriting the newer draft. - */ - const refuse = ( - diagnostics: WorkflowSourceDiagnostic[], - compiledReferences?: Pick - ) => { - if (!dryRun) throw new WorkflowSourceRejectedError(diagnostics) - const fingerprint = target.kind === 'existing' ? target.base.fingerprint : storedFingerprint - return result( - { catalog, stored }, - { files: input.files, lock: input.lock, fingerprint }, - { applied: false, diff: emptyDiffSummary(), diagnostics, ...compiledReferences } - ) - } - - if (target.kind === 'existing' && target.base.workflowId !== context.workflowId) { - return refuse([ - { - code: 'E_APPLY_WORKFLOW_MISMATCH', - severity: 'error', - message: `This source was read from workflow ${target.base.workflowId}, not ${context.workflowId}; applying it here would overwrite a different workflow.`, - hint: 'Apply it to the workflow it was read from, or get this workflow’s source into its own directory.', - path: '/base/workflowId', - }, - ]) - } - if (target.kind === 'existing' && storedFingerprint !== target.base.fingerprint) { - if (!dryRun) throw new WorkflowSourceStaleError(storedFingerprint, stored.meta.name) - return refuse([ - { - code: 'E_APPLY_STALE', - severity: 'error', - message: workflowSourceStaleMessage(stored.meta.name), - }, - ]) - } - - const evaluation = await evaluateSource({ - principal, - scope, - stored, - storedCanonical, - storedSourceMeta, - catalog, - envNames: new Set(envNames), - createdBy, - files: input.files, - lock: input.lock, - ...(input.pendingResources ? { pendingResources: input.pendingResources } : {}), - ...(target.kind === 'existing' - ? { base: target.base, allowDelete: target.allowDelete } - : {}), - ...(target.kind === 'new' ? { create: target.create } : {}), - }) - if ('refused' in evaluation) { - return refuse(evaluation.refused, evaluation.compiledReferences) - } - evaluated = { - workflowId: context.workflowId, - gate, - catalog, - stored, - storedSourceMeta, - storedCanonical, - storedFingerprint, - plan: evaluation.plan, - } - } - const { catalog, stored, storedSourceMeta, storedCanonical, storedFingerprint, plan } = - evaluated - const { diff, diagnostics, mintedIds } = plan - /** The function files the submission named, kept so a write never renames the author's. */ - const submittedCodeFiles: Record = {} - for (const [key, hint] of Object.entries(plan.compiled.hints.blocks)) { - if (hint.codeFile !== undefined) submittedCodeFiles[key] = hint.codeFile - } - /** Prints a state under the keys the submission bound, as the next `source get` will. */ - const print = ( - state: WorkflowState, - sourceMeta: WorkflowSourceMetaV1 | null, - lockless?: CanonicalizeResult - ) => - printStoredSource({ - state, - meta: plan.meta, - catalog, - resources: plan.resources, - sourceMeta, - functionCode: plan.functionCode, - identity: plan.identity, - codeFiles: submittedCodeFiles, - ...(lockless ? { lockless } : {}), - }) - const printWritten = (written: Omit) => - print({ ...written, variables: plan.storedVariables }, plan.sourceMeta) - const logContext = { - workflowId: context.workflowId, - workspaceId: context.workspaceId, - principalKind: principal.kind, - } - - if (plan.noop) { - // The stored graph under the name and description the write leaves it with. - const lockless = - plan.metaChanged.length > 0 - ? { - ...storedCanonical.canonical, - workflow: { ...storedCanonical.canonical.workflow, meta: plan.meta }, - } - : storedCanonical.canonical - const printed = print(stored.state, plan.sourceMeta, lockless) - /** Unchanged source meta prints the same text, so only changed meta prints the "before". */ - const before = isEqual(plan.sourceMeta, storedSourceMeta) - ? printed - : print(stored.state, storedSourceMeta, lockless) - const sourceMetaChanged = - !isEqual(printed.files, before.files) || !isEqual(printed.lock, before.lock) - const wroteSourceMeta = sourceMetaChanged && !dryRun - const renamed = !dryRun && plan.metaChanged.length > 0 - if ((wroteSourceMeta || renamed) && target.kind === 'existing') { - await commitSourceMeta({ - scope, - plan, - stored, - storedFingerprint, - catalog, - }) - } - return result(evaluated, printed, { - applied: renamed, - workflowName: plan.meta.name, - diff, - diagnostics, - references: plan.references, - envReferences: plan.envReferences, - wroteSourceMeta, - sourceMetaChanged, - ...(planOnly ? { evaluated } : {}), - }) - } - - if (planOnly) { - // A create's name is checked now, so a taken one refuses the plan, not the write. An - // update's ids are its own or minted (evaluation drops lock ids held elsewhere), and the - // commit re-checks them under the row lock. - if (target.kind === 'new') { - await previewSourceCommit(context.workflowId, plan, { - workspaceId: context.workspaceId, - folderId: target.folderId, - }) - } - return result( - evaluated, - { files: input.files, lock: input.lock, fingerprint: storedFingerprint }, - { - applied: false, - diff, - diagnostics, - mintedIds, - references: plan.references, - envReferences: plan.envReferences, - workflowName: plan.meta.name, - evaluated, - } - ) - } - - if (dryRun) { - const preview = await previewSourceCommit( - context.workflowId, - plan, - target.kind === 'new' - ? { workspaceId: context.workspaceId, folderId: target.folderId } - : undefined - ) - logger.info('Evaluated workflow source without persisting', logContext) - return result(evaluated, printWritten(preview.state), { - applied: false, - needsRedeployment: - !isNew && - (await checkDraftNeedsRedeployment(context.workflowId, { - ...preview.state, - variables: plan.storedVariables, - })), - diff, - diagnostics, - mintedIds, - references: plan.references, - envReferences: plan.envReferences, - notes: preview.warnings, - workflowName: plan.meta.name, - }) - } - - const committed = - target.kind === 'new' - ? await commitSourceCreate({ - scope, - plan, - folderId: target.folderId, - ...(target.withinCreate ? { withinCreate: target.withinCreate } : {}), - }) - : { - ...(await commitSourceReplace({ - scope, - plan, - stored, - storedFingerprint, - catalog, - })), - name: plan.meta.name, - } - logger.info(isNew ? 'Created workflow from source' : 'Applied workflow source', logContext) - return result(evaluated, printWritten(committed.state), { - applied: true, - wroteSourceMeta: true, - diff, - diagnostics, - mintedIds, - references: plan.references, - envReferences: plan.envReferences, - workflowName: committed.name, - notes: committed.warnings, - }) - }) -} diff --git a/apps/sim/lib/workflows/application/deployments.ts b/apps/sim/lib/workflows/application/deployments.ts index 3fc8e62f30d..95c58f03d80 100644 --- a/apps/sim/lib/workflows/application/deployments.ts +++ b/apps/sim/lib/workflows/application/deployments.ts @@ -164,32 +164,35 @@ export class WorkflowDraftChangedError extends OrchestrationError { } } +/** The draft gate a publish takes: absent when the caller holds no expected fingerprint. */ +interface DraftAdmission { + admitDraft?: (draft: WorkflowState) => Promise +} + /** - * A publish's draft check: the snapshot it publishes must print with `expected`, the fingerprint - * of the draft the caller reviewed or wrote. Every path that may publish a draft for a caller - * holding such a fingerprint (a deploy, a chat write that redeploys) admits its snapshot through - * `admitDraft`; after a failed publish, `assertAdmitted` throws {@link WorkflowDraftChangedError} - * when this check is what refused it. + * Runs a publish under a draft check: the snapshot it publishes must print with `expected`, the + * fingerprint of the draft the caller reviewed or wrote. Every path that may publish a draft for a + * caller holding such a fingerprint (a deploy, a chat write that redeploys) spreads the admission + * into its publish. Returns the publish's result, and throws {@link WorkflowDraftChangedError} + * when this check is what refused it. With no `expected`, the publish runs unchecked. */ -export function expectedDraft( +export async function withExpectedDraft( principal: Principal, context: ActiveWorkflowApplicationContext, - expected: string -): { - admitDraft: (draft: WorkflowState) => Promise - assertAdmitted: () => void -} { + expected: string | undefined, + publish: (admission: DraftAdmission) => Promise +): Promise { + if (expected === undefined) return publish({}) let refused = false - return { + const result = await publish({ async admitDraft(draft) { if ((await draftSourceFingerprint(principal, context, draft)) === expected) return null refused = true return new WorkflowDraftChangedError().message }, - assertAdmitted() { - if (refused) throw new WorkflowDraftChangedError() - }, - } + }) + if (refused) throw new WorkflowDraftChangedError() + return result } export const deployWorkflow = defineAuthorizedWorkflowUseCase({ @@ -200,23 +203,24 @@ export const deployWorkflow = defineAuthorizedWorkflowUseCase({ const attribution = resolvePrincipalAttribution(principal, { workspaceBillingOwnerUserId: context.billedAccountUserId, }) - const draftCheck = - input.expectedDraftFingerprint !== undefined - ? expectedDraft(principal, context, input.expectedDraftFingerprint) - : undefined - const result = await performFullDeploy({ - workflowId: context.workflowId, - userId: attribution.attributedUserId, - actorId: attribution.attributedUserId, - actor: toPrincipalActor(principal), - ...(principal.kind === 'delegated' ? { captureAnalytics: false as const } : {}), - versionName: input.name, - versionDescription: input.description, - requestId: input.requestId, - idempotencyKey: input.idempotencyKey, - ...(draftCheck ? { admitDraft: draftCheck.admitDraft } : {}), - }) - draftCheck?.assertAdmitted() + const result = await withExpectedDraft( + principal, + context, + input.expectedDraftFingerprint, + (admission) => + performFullDeploy({ + workflowId: context.workflowId, + userId: attribution.attributedUserId, + actorId: attribution.attributedUserId, + actor: toPrincipalActor(principal), + ...(principal.kind === 'delegated' ? { captureAnalytics: false as const } : {}), + versionName: input.name, + versionDescription: input.description, + requestId: input.requestId, + idempotencyKey: input.idempotencyKey, + ...admission, + }) + ) if (!result.success) throwDeploymentFailure(result, 'Failed to deploy workflow') const lint = input.lintDeployedVersion && result.deploymentVersionId diff --git a/apps/sim/lib/workflows/application/source-commit.ts b/apps/sim/lib/workflows/application/source-commit.ts index 5f86fc8a561..11089f3fd39 100644 --- a/apps/sim/lib/workflows/application/source-commit.ts +++ b/apps/sim/lib/workflows/application/source-commit.ts @@ -27,7 +27,7 @@ import { WorkflowSourceNameTakenError, WorkflowSourceStaleError, } from '@/lib/workflows/application/source-error' -import type { SourcePlan } from '@/lib/workflows/application/source-evaluate' +import type { SourcePlan } from '@/lib/workflows/application/source-pipeline' import { requireWorkflowTransition } from '@/lib/workflows/application/transition-result' import { assertLockedBlocksPreserved } from '@/lib/workflows/code/apply/locked' import { type StoredSource, storedSourceFingerprint } from '@/lib/workflows/code/print-stored' @@ -68,20 +68,27 @@ interface CommitScope { subjectUserId: string } +/** + * Refuses a workflow created from source under a name already taken in its folder, as the create + * would refuse it ({@link WorkflowSourceNameTakenError}): checked when the create is planned. + */ +export async function assertCreateNameFree( + plan: SourcePlan, + target: { workspaceId: string; folderId: string | null } +): Promise { + if (await workflowNameExistsInFolder({ ...target, name: plan.meta.name })) { + throw new WorkflowSourceNameTakenError(plan.meta.name) + } +} + /** * What a dry run would store: the plan's graph prepared for persistence, refused when another - * workflow already stores one of its ids, exactly as the write would refuse it. For a create - * (`newWorkflow`), a name already taken in the target folder is refused as the create would - * refuse it ({@link WorkflowSourceNameTakenError}). + * workflow already stores one of its ids, exactly as the write would refuse it. */ export async function previewSourceCommit( workflowId: string, - plan: SourcePlan, - newWorkflow?: { workspaceId: string; folderId: string | null } + plan: SourcePlan ): Promise { - if (newWorkflow && (await workflowNameExistsInFolder({ ...newWorkflow, name: plan.meta.name }))) { - throw new WorkflowSourceNameTakenError(plan.meta.name) - } const prepared = prepareWorkflowStateForPersistence({ blocks: plan.blocks, edges: plan.edges }) await assertWorkflowGraphIdsUnclaimed(db, workflowId, collectWorkflowGraphIds(prepared.state)) return prepared diff --git a/apps/sim/lib/workflows/application/source-error.ts b/apps/sim/lib/workflows/application/source-error.ts index aa1ee7724e4..40d440b6bba 100644 --- a/apps/sim/lib/workflows/application/source-error.ts +++ b/apps/sim/lib/workflows/application/source-error.ts @@ -1,4 +1,5 @@ -import type { DiagnosticCode, DiagnosticSeverity } from '@sim/workflow-code' +import type { DiagnosticCode } from '@sim/workflow-code' +import type { DiagnosticSeverity } from '@sim/workflow-code/diagnostics' import { OrchestrationError } from '@/lib/core/orchestration/types' /** diff --git a/apps/sim/lib/workflows/application/source-evaluate.ts b/apps/sim/lib/workflows/application/source-evaluate.ts deleted file mode 100644 index da81eaca216..00000000000 --- a/apps/sim/lib/workflows/application/source-evaluate.ts +++ /dev/null @@ -1,1037 +0,0 @@ -import type { Principal } from '@sim/auth/principal' -import { db } from '@sim/db' -import { workflowBlocks } from '@sim/db/schema' -import { isRecordLike } from '@sim/utils/object' -import { - type BlockCatalog, - type CanonicalBlock, - type CanonicalWorkflow, - type CanonicalWorkflowMeta, - type CompileResult, - canonicalize, - canonicalizeUnderLock, - compileWorkflow, - type Diagnostic, - diffCanonical, - encodeAnchor, - getOwn, - type LocatedDiagnostic, - type LockIdentity, - type MaterializeResult, - materialize, - parseWorkflowLock, - type ResourceKeyResolver, - type ResourceKind, - refuseEditedFindings, - renamesBetween, - replacedBetween, - type SourceBase, - type SourceHints, - type SourceSpan, - type WorkflowLockBlock, - type WorkflowLockV1, - type WorkflowSourceMetaV1, -} from '@sim/workflow-code' -import { SOURCE_ENTRY_FILE } from '@sim/workflow-code/limits' -import { WORKFLOW_RESOURCE_KINDS } from '@sim/workflow-types/resources' -import type { BlockState, WorkflowState } from '@sim/workflow-types/workflow' -import type { WorkspaceKind } from '@sim/workspace-code' -import { and, inArray, ne } from 'drizzle-orm' -import { isEqual } from 'es-toolkit' -import { - V2_WORKFLOW_DESCRIPTION_MAX_LENGTH, - V2_WORKFLOW_NAME_MAX_LENGTH, -} from '@/lib/api/contracts/v2/workflows' -import { resolvePermissionGroupConfig } from '@/lib/permission-groups/config-scope.server' -import { layoutChangedBlocks } from '@/lib/workflows/application/graph-write' -import { snapshotMcpToolOutputSchemas } from '@/lib/workflows/application/mcp-output-schemas' -import { type SourceDiffSummary, summarizeDiff } from '@/lib/workflows/application/source-diff' -import type { WorkflowSourceDiagnostic } from '@/lib/workflows/application/source-error' -import { - createLocator, - hasErrors, - lintDiagnostics, - normalizeChangedValues, -} from '@/lib/workflows/application/source-locate' -import { foreignResourceRefusals } from '@/lib/workflows/application/source-resource-scope' -import { normalizeWorkflowVariables } from '@/lib/workflows/application/workflow-variables' -import { gateSourceDiff } from '@/lib/workflows/code/apply/gates' -import { diffGraphs, type SourceGraph } from '@/lib/workflows/code/apply/graph-diff' -import { findLiveTriggerRemovals } from '@/lib/workflows/code/apply/live-triggers' -import { findLockedBlockRefusals } from '@/lib/workflows/code/apply/locked' -import { findInvalidSchedules } from '@/lib/workflows/code/apply/schedules' -import { compileSource } from '@/lib/workflows/code/compile-source' -import { keptCredentials } from '@/lib/workflows/code/credential-nodes' -import { createMaterializeHooks } from '@/lib/workflows/code/materialize-hooks' -import type { StoredSource } from '@/lib/workflows/code/print-stored' -import { loadAvailableResourceKeys, loadResourceKeyResolver } from '@/lib/workflows/code/resources' -import { mergeSourceMetaFromLock } from '@/lib/workflows/code/source-lock' -import type { StoredCanonical } from '@/lib/workflows/code/stored-canonical' -import type { - WorkflowLintReport, - WorkflowLintTableSchema, - WorkflowLintUnresolvedReference, -} from '@/lib/workflows/editing/lint' -import { buildWorkflowLintReport } from '@/lib/workflows/editing/lint-report' -import { workflowNameExistsInFolder } from '@/lib/workflows/orchestration' -import { parseWorkflowVariables } from '@/lib/workflows/variables/parse' -import type { ResourceKeyRef } from '@/lib/workspaces/resource-keys' - -const META_FIELDS: Array = ['name', 'description'] - -export interface SourceEvaluationInput { - principal: Principal - scope: { - workflowId: string - workspaceId: string - workspaceOrganizationId: string | null - /** The human the write is performed as, for gates, permission groups and lint. */ - subjectUserId: string - /** The folder the workflow sits in (`null` at the root), for a rename's name check. */ - folderId: string | null - } - /** The draft the source applies to; an empty graph for a workflow the apply creates. */ - stored: StoredSource - /** `stored` canonicalized with its resource keys. */ - storedCanonical: StoredCanonical - storedSourceMeta: WorkflowSourceMetaV1 | null - catalog: BlockCatalog - /** Env names the subject's runs resolve, for `env('…')` checks. */ - envNames: ReadonlySet - /** Attribution for resource keys a submitted reference allocates. */ - createdBy: string | null - files: Record - lock: WorkflowLockV1 - /** Where and when the source was read, its fingerprint already matched; absent for a create. */ - base?: SourceBase - /** Set when the apply creates the workflow: the caller's name and description overrides. */ - create?: { name?: string; description?: string | null } - /** - * Whether the apply may remove blocks the source no longer declares (`--allow-delete`). Without - * it, removing any block refuses the apply with `E_APPLY_BLOCK_DELETE`, as a workspace apply - * refuses deleting a resource. - */ - allowDelete?: boolean - /** - * Resources a workspace source apply creates or adopts before this workflow, by key, each with - * a stand-in id: consulted after the stored keys, so a dry run of a workflow that names a table - * created in the same apply resolves instead of failing `E_MAT_UNKNOWN_RESOURCE`. - */ - pendingResources?: readonly PendingResource[] -} - -/** - * A resource the source names by key; `member` is the part of it a reference names, when it names - * one: an MCP server's tool, for `mcp(key).tool(name)`. - */ -export interface SourceReference extends ResourceKeyRef { - member?: string - /** Where the entry file writes the reference, for an MCP tool: the agent's `tools`. */ - span?: SourceSpan -} - -/** A resource an enclosing apply creates or adopts, by key, with the id that stands in for it. */ -export interface PendingResource { - kind: ResourceKind - key: string - id: string - /** `id` stands in for a resource the apply has yet to create, not one it stores already. */ - placeholder?: boolean - /** The name the apply writes it under, which a workflow's stored reference repeats (a skill). */ - name?: string - /** - * A table's declared column names, which the lint checks a row write against instead of the - * stored schema the apply has yet to change. - */ - columns?: readonly string[] -} - -/** The schemas `pending` declares for its tables, by id, for the lint's table checks. */ -function declaredTableSchemas( - pending: readonly PendingResource[] -): Map { - const schemas = new Map() - for (const entry of pending) { - if (entry.kind !== 'table' || !entry.columns) continue - schemas.set(entry.id, { name: entry.name ?? entry.key, columnNames: entry.columns }) - } - return schemas -} - -/** - * `primary`, falling back to `fallback` for a key, id or name it does not resolve. Whether an id - * is built in is `primary`'s answer alone. - */ -function withFallback( - primary: ResourceKeyResolver, - fallback: Pick -): ResourceKeyResolver { - return { - toKey: (kind, id) => primary.toKey(kind, id) ?? fallback.toKey(kind, id), - toId: (kind, key) => primary.toId(kind, key) ?? fallback.toId(kind, key), - nameOf: (kind, id) => primary.nameOf?.(kind, id) ?? fallback.nameOf?.(kind, id) ?? null, - isBuiltin: primary.isBuiltin, - } -} - -/** `resources`, falling back to `pending` for a key, id or name it does not resolve. */ -function withPendingResources( - resources: ResourceKeyResolver, - pending: readonly PendingResource[] -): ResourceKeyResolver { - if (pending.length === 0) return resources - const byId = (kind: ResourceKind, id: string) => - pending.find((entry) => entry.kind === kind && entry.id === id) - return withFallback(resources, { - toKey: (kind, id) => byId(kind, id)?.key ?? null, - toId: (kind, key) => - pending.find((entry) => entry.kind === kind && entry.key === key)?.id ?? null, - nameOf: (kind, id) => byId(kind, id)?.name ?? null, - }) -} - -/** - * The lint reference kind an unresolved id of each workspace source kind reports as, or `null` for - * a kind no workflow names by id (a secret, read by name; a deployment, which nothing references). - * Every kind an enclosing apply can create or bind is listed, so a new one fails to compile until - * it is mapped. - */ -const PENDABLE_REFERENCE_OF: Record = - { - secret: null, - credential: 'credential', - 'custom-tool': 'custom-tool', - skill: 'skill', - table: 'resource', - 'knowledge-base': 'resource', - file: 'resource', - workflow: 'resource', - 'mcp-server': 'mcp-tool', - deployment: null, - } - -/** Unresolved reference kinds whose ids an enclosing apply may create. */ -const PENDABLE_REFERENCES: ReadonlySet = new Set( - Object.values(PENDABLE_REFERENCE_OF).filter((kind) => kind !== null) -) - -/** - * `lint` without the unresolved resource references that name only ids an enclosing apply has yet - * to write: a dry run of a directory that adds a skill and the workflow using it is not a warning. - */ -function withoutPendingReferences( - lint: WorkflowLintReport, - pending: readonly PendingResource[] -): WorkflowLintReport { - if (pending.length === 0) return lint - const pendingIds = new Set(pending.map((entry) => entry.id)) - const awaitsPending = (value: unknown) => { - const ids = typeof value === 'string' ? value.split(',').map((id) => id.trim()) : value - return ( - Array.isArray(ids) && - ids.length > 0 && - ids.every((id) => typeof id === 'string' && pendingIds.has(id)) - ) - } - return { - ...lint, - unresolvedReferences: lint.unresolvedReferences.filter( - (reference) => !PENDABLE_REFERENCES.has(reference.kind) || !awaitsPending(reference.value) - ), - } -} - -/** - * `lint` without the block-output findings compile already reported at their source line, so a - * dangling reference is one finding, as an offline check reports it: on a block whose - * `workflow.ts` reads a path no output has (`W_UNKNOWN_OUTPUT_PATH`, or its refusal), every - * block-output finding; on a block whose code file `$ref` names nothing (`W_UNRESOLVED_REF`, or - * its refusal), the one on its code. - */ -function withoutCompiledReferences( - lint: WorkflowLintReport, - compiled: readonly LocatedDiagnostic[], - keyOf: (id: string) => string | undefined -): WorkflowLintReport { - const inEntry = new Set() - const inCode = new Set() - for (const { code, blockKey, span } of compiled) { - if (blockKey === undefined) continue - if (span.file === SOURCE_ENTRY_FILE) { - if (code === 'W_UNKNOWN_OUTPUT_PATH' || code === 'E_APPLY_UNRESOLVED_REF') - inEntry.add(blockKey) - } else if (code === 'W_UNRESOLVED_REF' || code === 'E_APPLY_UNRESOLVED_REF') { - inCode.add(blockKey) - } - } - if (inEntry.size === 0 && inCode.size === 0) return lint - return { - ...lint, - unresolvedReferences: lint.unresolvedReferences.filter((reference) => { - const key = keyOf(reference.blockId) - if (reference.kind !== 'block-output' || key === undefined) return true - return !inEntry.has(key) && !(reference.field === 'code' && inCode.has(key)) - }), - } -} - -/** - * Findings a block rename can settle: a code-file `W_UNRESOLVED_REF` (an apply rewrites every - * `` to the new key, so `$ref.` runs) and a `W_JSON_REF_UNTYPED` on a JSON - * field whose only change is that rewrite. - */ -const RENAME_SETTLED_CODES: ReadonlySet = new Set([ - 'W_UNRESOLVED_REF', - 'W_JSON_REF_UNTYPED', -]) - -/** - * `diagnostics` without the {@link RENAME_SETTLED_CODES} findings a block rename settles. The - * compile already knew the renames the lock names by ident; a rename materialize inferred (name - * and const both changed) is known only now, so the source is compiled again with every rename and - * its findings of those codes, listed and withheld, replace the first compile's. Taken whole, not - * matched by position: the first compile withholds one finding per code, block and file, so a - * settled one may stand for a later finding in that file that the rename leaves unresolved. - */ -function withoutRenameSettledFindings( - compiled: Pick, - renames: ReadonlyMap, - recompile: () => CompileResult -): Pick { - const settles = (d: CompileResult['diagnostics'][number]) => RENAME_SETTLED_CODES.has(d.code) - if (renames.size === 0 || ![...compiled.diagnostics, ...compiled.withheld].some(settles)) { - return compiled - } - const again = recompile() - const unsettled = (d: CompileResult['diagnostics'][number]) => !settles(d) - return { - diagnostics: [...compiled.diagnostics.filter(unsettled), ...again.diagnostics.filter(settles)], - withheld: [...compiled.withheld.filter(unsettled), ...again.withheld.filter(settles)], - } -} - -/** `{ span }` when there is one, to spread into a diagnostic. */ -function withSpan(span: SourceSpan | undefined): { span?: SourceSpan } { - return span ? { span } : {} -} - -/** Names in a refusal, the first ten and a count of the rest. */ -function listNames(names: readonly string[]): string { - const shown = names.slice(0, 10).join(', ') - return names.length > 10 ? `${shown}, and ${names.length - 10} more` : shown -} - -/** A variable as a delete refusal names it. */ -interface NamedVariable { - name: string - type: string -} - -/** - * `E_APPLY_VARIABLE_DELETE` for the stored variables an apply removes, by name. When it removes - * one and adds one of the same type, the edit may be a rename the apply could not tie to the - * stored variable (its value changed too, or says nothing), so the hint says how to keep the id. - */ -function variableDeleteRefusal( - removed: readonly NamedVariable[], - added: readonly NamedVariable[], - span: SourceSpan | undefined -): Diagnostic { - const names = removed.map((variable) => variable.name) - const [former] = removed - const [next] = added - const rename = - removed.length === 1 && added.length === 1 && former.type === next.type - ? `If ${next.name} renames ${former.name}, apply the new key with the value ${former.name} holds first and change the value in a second apply, so the variable keeps its id; or pass --allow-delete (allowDelete=true) to replace it.` - : undefined - return { - code: 'E_APPLY_VARIABLE_DELETE', - severity: 'error', - message: `This apply deletes ${names.length} variable${names.length === 1 ? '' : 's'} the source no longer declares: ${listNames(names)}.`, - hint: - rename ?? - 'Pass --allow-delete (allowDelete=true) to delete them, or declare them again in workflow(name, { variables }) to keep them.', - ...withSpan(span), - } -} - -/** - * `E_APPLY_BLOCK_DELETE` for the stored blocks an apply removes. When the source adds blocks of - * a removed block's type, they may be renames whose name and const both changed with no identity - * signal tying them to the stored blocks (settings, connections or renamed references): the - * finding names them and says how to keep the ids, or to pass `--allow-delete` to replace them. - */ -/** - * Each removed block with the added block of the same type and parent it most probably became: - * one pair per removed block, in order, each added block used once. - */ -function probableRenames( - removed: readonly CanonicalBlock[], - added: readonly CanonicalBlock[] -): Array<{ removed: CanonicalBlock; added: CanonicalBlock }> { - const free = [...added] - return removed.flatMap((block) => { - const index = free.findIndex( - (candidate) => candidate.type === block.type && candidate.parent === block.parent - ) - if (index === -1) return [] - const [match] = free.splice(index, 1) - return [{ removed: block, added: match }] - }) -} - -function blockDeleteRefusal( - removed: readonly CanonicalBlock[], - added: readonly CanonicalBlock[], - span: SourceSpan | undefined -): Diagnostic { - const removedTypes = new Set(removed.map((block) => block.type)) - const lookalikes = added.filter((block) => removedTypes.has(block.type)) - const deletes = `This apply deletes ${removed.length} block${removed.length === 1 ? '' : 's'} the source no longer declares: ${listNames(removed.map((block) => block.name))}.` - return { - code: 'E_APPLY_BLOCK_DELETE', - severity: 'error', - ...(lookalikes.length > 0 - ? { - message: `${deletes} It adds ${listNames(lookalikes.map((block) => block.name))} of the same type: if those are the same blocks renamed, both their name and const changed, so nothing ties them to the stored blocks and they would get new ids.`, - hint: 'To rename a block and keep its id, keep its const name in this apply (change it in the next), or rename the references to it with it; to replace it with a new block, pass --allow-delete.', - } - : { - message: deletes, - hint: 'Pass --allow-delete (allowDelete=true) to delete them, or declare them again to keep them.', - }), - ...withSpan(span), - } -} - -/** Everything a commit (or a dry run) writes and reports, once every check passed. */ -export interface SourcePlan { - /** - * The workflow's name and description after the write: the source's (a stored workflow's - * differ from what is stored where `metaChanged` says), or a created workflow's. - */ - meta: CanonicalWorkflowMeta - /** The name and description fields the write changes on a stored workflow. */ - metaChanged: Array - /** The graph is unchanged; only source meta may differ. Never true for a create. */ - noop: boolean - blocks: Record - edges: WorkflowState['edges'] - /** The variables to write, or `undefined` when the source leaves them unchanged. */ - variables: ReturnType | undefined - /** Variables as the next read parses them, for re-printing what is stored. */ - storedVariables: WorkflowState['variables'] - /** Warnings (on a dry run, every finding), pinned to the submitted files. */ - diagnostics: WorkflowSourceDiagnostic[] - diff: SourceDiffSummary - /** Source key → id for every new block whose id was minted rather than taken from the lock. */ - mintedIds: Record - /** Every key the write binds, with its id: what the next print is keyed by. */ - identity: LockIdentity - /** The post-materialize lock the compiled idents and comments are re-keyed onto. */ - lock: WorkflowLockV1 - /** The compiled file's idents, order and comments. */ - compiled: { hints: SourceHints; workflow: Pick } - /** source meta after the write, merged onto the stored one. */ - sourceMeta: WorkflowSourceMetaV1 - /** How the response prints function code: as the submission did. */ - functionCode: 'inline' | 'files' - /** Answers every resource reference of both the stored and the submitted source. */ - resources: ResourceKeyResolver - /** Every resource the submitted source names by key, as the compiler read it. */ - references: SourceReference[] - /** Every env name the submitted source reads, as the compiler read it. */ - envReferences: string[] -} - -/** - * A refusal, with what the source references when it compiled (so an enclosing apply can tell a - * workflow refused for a reference it holds back), or the plan. - */ -export type SourceEvaluation = - | { - refused: WorkflowSourceDiagnostic[] - compiledReferences?: Pick - } - | { plan: SourcePlan } - -/** Whether a compiled resource node's kind is one the resolver answers. */ -function isResourceKind(kind: string): kind is ResourceKind { - return ( - kind === 'workflow' || - (kind !== 'env-var' && WORKFLOW_RESOURCE_KINDS.some((candidate) => candidate === kind)) - ) -} - -/** - * Every `(kind, key)` resource reference in a compiled model, for ident lookup, and the tool each - * MCP agent tool (`mcp(key).tool(name)`) names on its server. - */ -function collectResourceKeys( - value: unknown, - refs: SourceReference[], - span: SourceSpan | undefined -): void { - if (Array.isArray(value)) { - for (const item of value) collectResourceKeys(item, refs, span) - return - } - if (value === null || typeof value !== 'object') return - const record = value as Record - if ( - record.$sim === 'resource' && - typeof record.kind === 'string' && - isResourceKind(record.kind) && - typeof record.key === 'string' - ) { - refs.push({ kind: record.kind, key: record.key }) - } - const tool = mcpToolReference(record) - if (tool) refs.push(span ? { ...tool, span } : tool) - for (const nested of Object.values(record)) collectResourceKeys(nested, refs, span) -} - -/** The server and tool an MCP agent tool entry names, or `null` for anything else. */ -function mcpToolReference(record: Record): SourceReference | null { - if (record.type !== 'mcp' || !isRecordLike(record.params)) return null - const { serverId, toolName } = record.params - if ( - !isRecordLike(serverId) || - serverId.$sim !== 'resource' || - serverId.kind !== 'mcp-server' || - typeof serverId.key !== 'string' || - typeof toolName !== 'string' - ) { - return null - } - return { kind: 'mcp-server', key: serverId.key, member: toolName } -} - -/** Block ids another workflow already stores, among `ids`. */ -async function findBlockIdsClaimedElsewhere(workflowId: string, ids: string[]): Promise { - if (ids.length === 0) return [] - const rows = await db - .select({ id: workflowBlocks.id }) - .from(workflowBlocks) - .where(and(inArray(workflowBlocks.id, ids), ne(workflowBlocks.workflowId, workflowId))) - return rows.map((row) => row.id) -} - -/** - * The post-materialize lock `mergeSourceMetaFromLock` re-keys onto: every bound key with its id, - * and the submitted lock's ident and order for a block the file does not state them for. - */ -function boundLock(identity: LockIdentity, submitted: WorkflowLockV1): WorkflowLockV1 { - const blocks: Record = {} - for (const [key, block] of Object.entries(identity.blocks)) { - const previous = Object.hasOwn(submitted.blocks, key) ? submitted.blocks[key] : undefined - blocks[key] = { ...block, ident: previous?.ident ?? key, order: previous?.order ?? 0 } - } - return { version: 1, blocks, variables: identity.variables } -} - -/** - * The name and description fields among `fields` that v2 create and update would refuse: an empty - * name, or a name or description over its limit. Only the fields a write sets are checked, so a - * stored workflow the source keeps as it is never refuses. - */ -function metaLimitRefusals( - meta: CanonicalWorkflowMeta, - fields: readonly (keyof CanonicalWorkflowMeta)[] -): Array<{ field: keyof CanonicalWorkflowMeta; refusal: Diagnostic }> { - const refusals: Array<{ field: keyof CanonicalWorkflowMeta; refusal: Diagnostic }> = [] - const refuse = (field: keyof CanonicalWorkflowMeta, message: string, hint: string) => - refusals.push({ - field, - refusal: { code: 'E_WORKFLOW_META_INVALID', severity: 'error', message, hint }, - }) - if (fields.includes('name') && meta.name.trim() === '') { - refuse('name', 'The workflow name is empty.', "Name the workflow: workflow('Name', ...).") - } else if (fields.includes('name') && meta.name.length > V2_WORKFLOW_NAME_MAX_LENGTH) { - refuse( - 'name', - `The workflow name is ${meta.name.length} characters, over the ${V2_WORKFLOW_NAME_MAX_LENGTH}-character limit.`, - `Shorten the name to ${V2_WORKFLOW_NAME_MAX_LENGTH} characters or fewer.` - ) - } - const description = meta.description ?? '' - if (fields.includes('description') && description.length > V2_WORKFLOW_DESCRIPTION_MAX_LENGTH) { - refuse( - 'description', - `The workflow description is ${description.length} characters, over the ${V2_WORKFLOW_DESCRIPTION_MAX_LENGTH}-character limit.`, - `Shorten the description to ${V2_WORKFLOW_DESCRIPTION_MAX_LENGTH} characters or fewer.` - ) - } - return refusals -} - -/** - * Evaluates submitted source against a stored draft without writing it: compile against the live - * catalog, parse the lock, materialize onto the stored graph (keeping every unchanged value's - * stored bytes and every id the lock binds), then every gate (locked blocks, engine gates, - * types that may stay but not be added), the value checks and lint. Returns the refusals, located - * in the submitted text, or the plan the write stores. - */ -export async function evaluateSource(input: SourceEvaluationInput): Promise { - const { principal, scope, catalog, files } = input - const isNew = input.create !== undefined - - const parsedLock = parseWorkflowLock(input.lock) - if (parsedLock.diagnostics) { - return { refused: parsedLock.diagnostics.map(createLocator(files, {}, new Map())) } - } - const lock = parsedLock.lock - - const { state } = input.stored - let resources = input.storedCanonical.resources - let meta = input.stored.meta - /** - * The stored graph as the submitted lock reads it: what `stored()` keeps values from, what - * materialize matches against, and what the diff compares the compiled source against. - */ - let storedCanonical = canonicalizeUnderLock( - state, - { catalog, meta, resources }, - input.storedCanonical.canonical, - lock - ) - const compileInput = { - entry: { path: SOURCE_ENTRY_FILE, text: files[SOURCE_ENTRY_FILE] }, - files: new Map(Object.entries(files).filter(([path]) => path !== SOURCE_ENTRY_FILE)), - catalog, - sdkMajor: catalog.sdkMajor, - knownKeys: new Set([ - ...Object.keys(lock.blocks), - ...Object.keys(storedCanonical.identity.blocks), - ]), - envNames: input.envNames, - } - const compiled = compileSource({ ...compileInput, stored: storedCanonical.workflow, lock }) - const compiledWorkflow = compiled.workflow - if (!compiledWorkflow || hasErrors(compiled.diagnostics)) { - const refused: Diagnostic[] = [...compiled.diagnostics] - if (input.base && input.base.catalogHash !== catalog.hash) { - refused.push({ - code: 'W_APPLY_CATALOG_MOVED', - severity: 'warning', - message: - 'The block catalog changed since this source was read, which can explain these errors.', - hint: 'Get the source again to print it against the current catalog, then reapply your edit.', - }) - } - return { refused: refused.map(createLocator(files, compiled.spans, new Map())) } - } - - if (input.create) { - meta = { - name: input.create.name ?? compiledWorkflow.meta.name, - description: - input.create.description !== undefined - ? input.create.description - : compiledWorkflow.meta.description, - } - } - - const compiledKeys: SourceReference[] = [] - for (const [key, block] of Object.entries(compiledWorkflow.blocks)) { - const span = - compiled.spans[encodeAnchor({ kind: 'param', block: key, param: 'tools' })] ?? - compiled.spans[encodeAnchor({ kind: 'block', block: key })] - collectResourceKeys(block, compiledKeys, span) - } - const storedResources = resources - const newKeys = compiledKeys.filter(({ kind, key }) => storedResources.toId(kind, key) === null) - if (newKeys.length > 0) { - const extra = await loadResourceKeyResolver( - scope.workspaceId, - { keys: newKeys }, - { actorUserId: input.createdBy } - ) - resources = withFallback(storedResources, extra) - } - - const kept = keptCredentials( - compiledWorkflow, - compiled.stored, - (key) => - resources.toId('credential', key) === null && - !(input.pendingResources ?? []).some( - (entry) => entry.kind === 'credential' && entry.key === key - ) - ) - if (kept.length > 0) { - const idsByKey = new Map(kept.map(({ key, id }) => [key, id])) - const keysById = new Map(kept.map(({ key, id }) => [id, key])) - resources = withFallback(resources, { - toKey: (kind, id) => (kind === 'credential' ? (keysById.get(id) ?? null) : null), - toId: (kind, key) => (kind === 'credential' ? (idsByKey.get(key) ?? null) : null), - }) - // The stored graph under the keys the source gives its kept credentials: no change there. - storedCanonical = canonicalizeUnderLock( - state, - { catalog, meta, resources }, - canonicalize(state, { catalog, meta, resources }), - lock - ) - } - const keptDiagnostics: Diagnostic[] = kept.map(({ key, blockKey }) => ({ - code: 'W_APPLY_CREDENTIAL_KEPT', - severity: 'warning', - message: `credential(${JSON.stringify(key)}) stands where the workflow stores a credential you cannot see, so that credential is kept as stored.`, - hint: 'Name a credential you can see to change it, or ask a member who can see the stored one.', - blockKey, - })) - - resources = withPendingResources(resources, input.pendingResources ?? []) - - const hasLockEntries = - Object.keys(lock.blocks).length > 0 || Object.keys(lock.variables).length > 0 - const lockFreshness: 'fresh' | 'stale' | 'none' = !hasLockEntries - ? 'none' - : input.base && input.base.workspaceId !== scope.workspaceId - ? 'stale' - : 'fresh' - const compiledIdents: Record = {} - for (const [key, hint] of Object.entries(compiled.hints.blocks)) { - if (hint.ident !== undefined) compiledIdents[key] = hint.ident - } - const hooks = createMaterializeHooks() - const materializeWith = (withLock: WorkflowLockV1): MaterializeResult => - materialize(compiledWorkflow, withLock, { - catalog, - hooks, - prev: state, - prevCanonical: storedCanonical, - resources, - compiledIdents, - lockFreshness, - }) - - /** - * A lock id for a new block that another workflow already stores cannot be reused (block ids - * are global keys): drop it from the lock so materialize mints one, and say so. - */ - let materialized = materializeWith(lock) - if (materialized.diagnostics.some((d) => d.code === 'E_MAT_UNKNOWN_RESOURCE')) { - // Only on a miss: list the keys the source could have named, for the finding's did-you-mean. - const missingKinds = compiledKeys - .filter(({ kind, key }) => resources.toId(kind, key) === null) - .map(({ kind }) => kind) - if (missingKinds.length > 0) { - const available = await loadAvailableResourceKeys( - scope.workspaceId, - missingKinds, - input.createdBy - ) - const known = resources - resources = { ...known, keysOf: (kind) => available.get(kind) } - materialized = materializeWith(lock) - } - } - const claimedDiagnostics: Diagnostic[] = [] - const lockSuppliedIds = materialized.entries.flatMap((entry) => - entry.status === 'created' && entry.idSource === 'lock' ? [entry.id] : [] - ) - const [claimedIds, permissionConfig] = await Promise.all([ - findBlockIdsClaimedElsewhere(scope.workflowId, lockSuppliedIds), - resolvePermissionGroupConfig( - scope.subjectUserId, - scope.workspaceId, - scope.workspaceOrganizationId - ), - ]) - const claimed = new Set(claimedIds) - if (claimed.size > 0) { - const blocks: Record = {} - for (const [key, entry] of Object.entries(lock.blocks)) { - if (claimed.has(entry.id)) { - claimedDiagnostics.push({ - code: 'W_MAT_LOCK_ID_CLAIMED', - severity: 'warning', - message: `The lock id for block "${key}" belongs to another workflow; a new id was minted`, - blockKey: key, - }) - } else { - blocks[key] = entry - } - } - materialized = materializeWith({ ...lock, blocks }) - } - - const idByKey = new Map( - Object.entries(materialized.identity.blocks).map(([key, block]) => [key, block.id]) - ) - const keyById = new Map([...idByKey].map(([key, id]) => [id, key])) - const storedKeyById = new Map( - Object.entries(storedCanonical.identity.blocks).map(([key, block]) => [block.id, key]) - ) - /** A block's source key: the one this write binds, else the stored one for a removed block. */ - const keyOf = (id: string) => keyById.get(id) ?? storedKeyById.get(id) - const locate = createLocator(files, compiled.spans, idByKey) - const materializeDiagnostics = [...materialized.diagnostics, ...claimedDiagnostics] - if (hasErrors(materializeDiagnostics)) { - return { - refused: [...compiled.diagnostics, ...materializeDiagnostics].map(locate), - compiledReferences: { references: compiledKeys, envReferences: compiled.envReferences }, - } - } - - /** - * The name and description the source gives a stored workflow, where they differ from the - * stored ones: the write renames or redescribes the workflow (the graph diff never sees them). - */ - const metaChanged = isNew - ? [] - : META_FIELDS.filter( - (field) => (compiledWorkflow.meta[field] ?? null) !== (meta[field] ?? null) - ) - const storedMeta = meta - if (metaChanged.length > 0) { - meta = { - name: compiledWorkflow.meta.name, - description: compiledWorkflow.meta.description ?? null, - } - } - /** - * What the write stores against what is stored: the materialized CWM, not the compiled one, - * so a field source never prints (a forEach loop's empty while condition, a pair's stored - * member) that materialize keeps from the stored block is no change. - */ - const blockRenames = renamesBetween(storedCanonical.identity, materialized.identity) - const replacedKeys = replacedBetween(storedCanonical.identity, materialized.identity) - const canonicalDiff = diffCanonical( - storedCanonical.workflow, - { ...materialized.workflow, meta: storedCanonical.workflow.meta }, - blockRenames, - replacedKeys - ) - const renamedKeys = new Map([...blockRenames].map(([key, storedKey]) => [storedKey, key])) - const storedKeyOf = new Map([...renamedKeys].map(([storedKey, key]) => [key, storedKey])) - const storedCode = (key: string) => { - const storedBlock = - getOwn(compiled.stored.blocks, key) ?? - getOwn(compiled.stored.blocks, storedKeyOf.get(key) ?? key) - return storedBlock && getOwn(storedBlock.params, 'code') - } - const compileFindings = withoutRenameSettledFindings(compiled, renamedKeys, () => - compileWorkflow({ ...compileInput, stored: compiled.stored, renamedKeys }) - ) - const codeEdited = (key: string) => { - const compiledBlock = getOwn(compiledWorkflow.blocks, key) - return !isEqual(compiledBlock && getOwn(compiledBlock.params, 'code'), storedCode(key)) - } - const sourceRefusals: Diagnostic[] = [] - const headerSpan = compiled.spans[encodeAnchor({ kind: 'header' })] - /** Where a finding about the workflow itself points: the field it names, else the name. */ - const metaSpan = (field: keyof CanonicalWorkflowMeta = 'name') => - compiled.metaSpans[field] ?? compiled.metaSpans.name ?? headerSpan - if ( - metaChanged.includes('name') && - (await workflowNameExistsInFolder({ - workspaceId: scope.workspaceId, - name: meta.name, - folderId: scope.folderId, - excludeWorkflowId: scope.workflowId, - })) - ) { - sourceRefusals.push({ - code: 'E_NAME_TAKEN', - severity: 'error', - message: `Another workflow in this workflow's folder is named ${JSON.stringify(meta.name)}.`, - hint: `Choose another name, or keep workflow(${JSON.stringify(storedMeta.name)}, ...).`, - ...withSpan(metaSpan('name')), - }) - } - sourceRefusals.push( - ...metaLimitRefusals(meta, isNew ? META_FIELDS : metaChanged).map(({ field, refusal }) => ({ - ...refusal, - ...withSpan(metaSpan(field)), - })) - ) - // A block replaced under its key (a retype) is reported removed and added, but it is the - // same declaration, which W_MAT_RETYPE_RECREATED covers, so it needs no allowDelete. - const removedKeys = isNew - ? [] - : canonicalDiff.blocks.removed.filter((key) => !replacedKeys.has(key)) - const removedBlocks = removedKeys.map((key) => storedCanonical.workflow.blocks[key]) - const addedBlocks = canonicalDiff.blocks.added - .filter((key) => !replacedKeys.has(key)) - .map((key) => materialized.workflow.blocks[key]) - if (removedKeys.length > 0 && !input.allowDelete) { - sourceRefusals.push(blockDeleteRefusal(removedBlocks, addedBlocks, metaSpan())) - } - const keptVariableIds = new Set(Object.values(materialized.identity.variables)) - const storedVariableIds = new Set(Object.values(storedCanonical.identity.variables)) - const removedVariables = isNew - ? [] - : Object.entries(storedCanonical.identity.variables).flatMap(([key, id]) => { - if (keptVariableIds.has(id)) return [] - const variable = getOwn(storedCanonical.workflow.variables, key) - return [{ name: variable?.name ?? key, type: variable?.type ?? '' }] - }) - if (removedVariables.length > 0 && !input.allowDelete) { - const addedVariables = Object.entries(materialized.identity.variables).flatMap(([key, id]) => { - const variable = getOwn(materialized.workflow.variables, key) - return storedVariableIds.has(id) || !variable - ? [] - : [{ name: variable.name, type: variable.type }] - }) - sourceRefusals.push(variableDeleteRefusal(removedVariables, addedVariables, metaSpan())) - } - const renameWarnings = input.allowDelete - ? probableRenames(removedBlocks, addedBlocks).map( - ({ removed, added }): Diagnostic => ({ - code: 'W_APPLY_PROBABLE_RENAME', - severity: 'warning', - message: `${removed.name} (${removed.type}) is deleted and ${added.name} (${added.type}) added: probably a rename, but its name and const both changed, so it gets a new id and loses its run history and webhook URL.`, - hint: `To keep the id, apply ${added.name} under the const or name ${removed.name} had first, then change the other.`, - blockKey: added.key, - }) - ) - : [] - - const storedGraph: SourceGraph = { blocks: state.blocks, edges: state.edges } - let blocks = materialized.state.blocks - const edges = materialized.state.edges - const graphDiff = diffGraphs(storedGraph, { blocks, edges }) - - const typeChanged = graphDiff.modified.filter((change) => change.typeChanged).map((c) => c.id) - const [gateDiagnostics, liveTriggerRefusals] = await Promise.all([ - gateSourceDiff({ - base: storedGraph, - target: { blocks, edges }, - diff: graphDiff, - catalog, - keyOf, - permissionConfig, - subjectUserId: scope.subjectUserId, - workspaceId: scope.workspaceId, - isCopilot: principal.kind === 'delegated' && principal.serviceId === 'copilot', - }), - findLiveTriggerRemovals( - scope.workflowId, - [...graphDiff.removed, ...typeChanged].map((id) => state.blocks[id]), - keyOf - ), - ]) - const refusals: Diagnostic[] = [ - ...sourceRefusals, - ...renameWarnings, - ...findLockedBlockRefusals(storedGraph, { blocks, edges }, graphDiff, keyOf), - ...findInvalidSchedules({ blocks, edges }, graphDiff, keyOf), - ...gateDiagnostics, - ...liveTriggerRefusals, - ] - - for (const diagnostic of gateDiagnostics) { - if (diagnostic.code !== 'W_APPLY_HOSTED_KEY_DROPPED' || !diagnostic.blockId) continue - const { blockId, param } = diagnostic - const block = blocks[blockId] - if (!param || !block?.subBlocks?.[param]) continue - blocks = { - ...blocks, - [blockId]: { - ...block, - subBlocks: { ...block.subBlocks, [param]: { ...block.subBlocks[param], value: null } }, - }, - } - } - - const normalized = normalizeChangedValues(blocks, graphDiff, (id) => keyById.get(id) ?? '') - blocks = await snapshotMcpToolOutputSchemas(normalized.blocks, state.blocks, { - principal, - workspaceId: scope.workspaceId, - }) - if (materialized.needsLayout.length > 0) { - blocks = layoutChangedBlocks(scope.workflowId, storedGraph, { blocks, edges }) - } - - const lint = await buildWorkflowLintReport( - { blocks, edges }, - { - workflowId: scope.workflowId, - workspaceId: scope.workspaceId, - subjectUserId: scope.subjectUserId, - }, - { declaredTables: declaredTableSchemas(input.pendingResources ?? []), iterationScope: false } - ) - const changedIds = new Set([...graphDiff.added, ...graphDiff.modified.map((m) => m.id)]) - const changedKeys = new Set([...changedIds].flatMap((id) => keyById.get(id) ?? [])) - const keyOfId = (id: string) => keyById.get(id) - const compileDiagnostics = refuseEditedFindings(compileFindings, { - changed: (key) => changedKeys.has(key), - codeEdited, - }) - const compiledLint = withoutCompiledReferences( - withoutPendingReferences(lint, input.pendingResources ?? []), - compileDiagnostics, - keyOfId - ) - const foreignIds = await foreignResourceRefusals({ - blocks, - diff: graphDiff, - workspaceId: scope.workspaceId, - pending: input.pendingResources ?? [], - keyOf: keyOfId, - }) - const diagnostics = [ - ...compileDiagnostics, - ...keptDiagnostics, - ...materializeDiagnostics, - ...refusals, - ...normalized.diagnostics, - ...foreignIds, - ...lintDiagnostics(compiledLint, changedIds, keyOfId), - ].map(locate) - if (hasErrors(diagnostics)) { - return { - refused: diagnostics, - compiledReferences: { references: compiledKeys, envReferences: compiled.envReferences }, - } - } - - const noop = !isNew && canonicalDiff.noop - const mintedIds: Record = {} - for (const entry of materialized.entries) { - if (entry.status === 'created' && entry.idSource === 'minted') mintedIds[entry.key] = entry.id - } - const { added, removed, modified } = canonicalDiff.variables - const variables = - added.length > 0 || removed.length > 0 || modified.length > 0 - ? normalizeWorkflowVariables(materialized.state.variables ?? {}, { coerceValues: true }) - : undefined - const compiledSource = { hints: compiled.hints, workflow: compiledWorkflow } - const lockAfter = boundLock(materialized.identity, lock) - - return { - plan: { - meta, - noop, - blocks, - edges, - variables, - storedVariables: - variables === undefined ? state.variables : (parseWorkflowVariables(variables) ?? {}), - diagnostics, - diff: summarizeDiff( - canonicalDiff, - storedCanonical, - { workflow: materialized.workflow, identity: materialized.identity }, - noop, - metaChanged - ), - metaChanged, - mintedIds, - identity: materialized.identity, - lock: lockAfter, - compiled: compiledSource, - sourceMeta: mergeSourceMetaFromLock(input.storedSourceMeta, lockAfter, compiledSource), - functionCode: Object.values(compiled.hints.blocks).some((hint) => hint.codeFile) - ? 'files' - : 'inline', - resources, - references: compiledKeys, - envReferences: compiled.envReferences, - }, - } -} diff --git a/apps/sim/lib/workflows/application/source-pipeline/commit.ts b/apps/sim/lib/workflows/application/source-pipeline/commit.ts new file mode 100644 index 00000000000..d4105b3bf13 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/commit.ts @@ -0,0 +1,101 @@ +import type { Principal } from '@sim/auth/principal' +import { createLogger } from '@sim/logger' +import { withCatalogBlockScope } from '@/lib/catalog/application/context' +import type { DbTransaction } from '@/lib/db/types' +import { + commitSourceCreate, + commitSourceMeta, + commitSourceReplace, +} from '@/lib/workflows/application/source-commit' +import { + type CommitTarget, + sourceScope, + targetIds, +} from '@/lib/workflows/application/source-pipeline/load' +import { + printWritten, + type SourceOutcome, + shapeOutcome, +} from '@/lib/workflows/application/source-pipeline/outcome' +import { + assertPlannedBy, + PlanBindingError, + type PlannedSource, +} from '@/lib/workflows/application/source-pipeline/plan' +import { checkNeedsRedeployment } from '@/lib/workflows/deployment-status' + +const logger = createLogger('ApplyWorkflowSource') + +/** + * Commits a plan to `target`, the workflow it was planned for, as the principal it was planned + * by: a plan made for another workflow or principal throws `PlanBindingError` before anything is + * written. A graph no-op stores only its source meta, name and description (and nothing when none + * changed); an existing draft is replaced under the workflow row lock, which re-checks the plan's + * fingerprint (409 when the draft moved); a new workflow is inserted with its graph in one + * transaction, with `withinCreate` run inside it. + */ +export async function commitSource( + principal: Principal, + planned: PlannedSource, + target: CommitTarget, + options: { withinCreate?: (tx: DbTransaction, workflowId: string) => Promise } = {} +): Promise { + const { workflowId, workspaceId } = targetIds(target) + if (planned.workflowId !== workflowId || planned.target.kind !== target.kind) { + throw new PlanBindingError( + 'PLAN_WORKFLOW_MISMATCH', + 'A planned source commits only to the workflow it was planned for' + ) + } + assertPlannedBy(principal, planned) + const { plan, loaded } = planned + const { catalog, stored, storedFingerprint } = loaded + const scope = sourceScope(principal, target) + const frame = { target, dryRun: false, catalog, storedName: stored.meta.name } + const logContext = { workflowId, workspaceId, principalKind: principal.kind } + return withCatalogBlockScope(loaded.gate, async () => { + if (planned.noop) { + if (target.kind !== 'existing') throw new Error('A created workflow is never a graph no-op') + const { printed, sourceMetaChanged } = planned.noop + const renamed = plan.metaChanged.length > 0 + if (sourceMetaChanged || renamed) { + await commitSourceMeta({ scope, plan, stored, storedFingerprint, catalog }) + } + return shapeOutcome(frame, printed, { + applied: renamed, + workflowName: plan.meta.name, + diff: plan.diff, + diagnostics: plan.diagnostics, + wroteSourceMeta: sourceMetaChanged, + needsRedeployment: await checkNeedsRedeployment(workflowId), + }) + } + + const committed = + target.kind === 'new' + ? await commitSourceCreate({ + scope, + plan, + folderId: target.folderId, + ...(options.withinCreate ? { withinCreate: options.withinCreate } : {}), + }) + : { + ...(await commitSourceReplace({ scope, plan, stored, storedFingerprint, catalog })), + name: plan.meta.name, + } + logger.info( + target.kind === 'new' ? 'Created workflow from source' : 'Applied workflow source', + logContext + ) + return shapeOutcome(frame, printWritten(planned, committed.state), { + applied: true, + wroteSourceMeta: true, + diff: plan.diff, + diagnostics: plan.diagnostics, + mintedIds: plan.mintedIds, + workflowName: committed.name, + notes: committed.warnings, + needsRedeployment: target.kind === 'new' ? false : await checkNeedsRedeployment(workflowId), + }) + }) +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/effects.ts b/apps/sim/lib/workflows/application/source-pipeline/effects.ts new file mode 100644 index 00000000000..4a1eb492af7 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/effects.ts @@ -0,0 +1,133 @@ +import { AuditAction, AuditResourceType } from '@sim/audit' +import type { Principal } from '@sim/auth/principal' +import { principalAuditSource } from '@/lib/core/application' +import { notifyWorkflowUpdated, notifyWorkspaceWorkflowsChanged } from '@/lib/realtime/notify' +import { recordSourceWritten } from '@/lib/workflows/application/source-analytics' +import type { SourceOutcome } from '@/lib/workflows/application/source-pipeline/outcome' + +/** Where a source write says it came from: a self-reported client, and the git commit it read. */ +export interface SourceProvenance { + client: 'cli' | 'copilot' | 'canvas' | 'api' + gitSha?: string + repo?: string +} + +/** The analytics facts of a source write or dry run. */ +export function sourceAnalyticsFacts( + principal: Principal, + provenance: SourceProvenance | undefined, + operation: 'apply' | 'create', + workspaceId: string, + dryRun: boolean +) { + return { principal, client: provenance?.client, operation, workspaceId, dryRun } +} + +/** The audit metadata a committed source write records, for update and create alike. */ +function sourceAuditMetadata( + principal: Principal, + provenance: SourceProvenance | undefined, + result: SourceOutcome +) { + const { blocks, edges, variables } = result.diff + return { + op: 'apply_source', + blocksAdded: blocks.added.length, + blocksRemoved: blocks.removed.length, + blocksRenamed: blocks.renamed.length, + blocksModified: blocks.modified.length, + edgesAdded: edges.added, + edgesRemoved: edges.removed, + variablesChanged: + variables.added.length + + variables.removed.length + + variables.renamed.length + + variables.modified.length, + sdkMajor: result.sdkMajor, + catalogHash: result.base.catalogHash, + ...(provenance ? { client: provenance.client } : {}), + ...(provenance?.gitSha ? { gitSha: provenance.gitSha } : {}), + ...(provenance?.repo ? { repo: provenance.repo } : {}), + source: principalAuditSource(principal), + } +} + +/** The audit of a source write to a stored workflow: only a committed, non-empty one. */ +export function sourceApplyAudit( + principal: Principal, + provenance: SourceProvenance | undefined, + workflowId: string, + result: SourceOutcome +) { + return result.applied + ? ({ + action: AuditAction.WORKFLOW_UPDATED, + resourceType: AuditResourceType.WORKFLOW, + resourceId: workflowId, + resourceName: result.workflowName, + description: `Applied workflow source to workflow "${result.workflowName}"`, + metadata: sourceAuditMetadata(principal, provenance, result), + } as const) + : [] +} + +/** The audit of a workflow created from source: only a committed one. */ +export function sourceCreateAudit( + principal: Principal, + provenance: SourceProvenance | undefined, + result: SourceOutcome +) { + return result.applied + ? ({ + action: AuditAction.WORKFLOW_CREATED, + resourceType: AuditResourceType.WORKFLOW, + resourceId: result.workflowId, + resourceName: result.workflowName, + description: `Created workflow "${result.workflowName}" from source`, + metadata: sourceAuditMetadata(principal, provenance, result), + } as const) + : [] +} + +/** + * After a source write to a stored workflow: records a committed write and notifies. Source meta + * alone changes what an open Code tab prints, so it notifies too. + */ +export async function afterSourceApply( + principal: Principal, + provenance: SourceProvenance | undefined, + context: { workflowId: string; workspaceId: string }, + result: SourceOutcome +): Promise { + if (result.applied) { + recordSourceWritten( + sourceAnalyticsFacts(principal, provenance, 'apply', context.workspaceId, result.dryRun), + result + ) + } + if (!result.applied && !result.wroteSourceMeta) return + await Promise.all([ + notifyWorkflowUpdated(context.workflowId), + // A rename or new description changes how the workspace lists the workflow. + result.applied && result.diff.meta?.length + ? notifyWorkspaceWorkflowsChanged(context.workspaceId) + : undefined, + ]) +} + +/** After a workflow is created from source: records the write and notifies. */ +export async function afterSourceCreate( + principal: Principal, + provenance: SourceProvenance | undefined, + result: SourceOutcome +): Promise { + if (!result.applied) return + recordSourceWritten( + sourceAnalyticsFacts(principal, provenance, 'create', result.workspaceId, result.dryRun), + result + ) + await Promise.all([ + notifyWorkflowUpdated(result.workflowId), + notifyWorkspaceWorkflowsChanged(result.workspaceId), + ]) +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/evaluate/compile.ts b/apps/sim/lib/workflows/application/source-pipeline/evaluate/compile.ts new file mode 100644 index 00000000000..9095c160720 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/evaluate/compile.ts @@ -0,0 +1,96 @@ +import { + type CanonicalizeResult, + type CanonicalWorkflowMeta, + canonicalizeUnderLock, + type Diagnostic, + parseWorkflowLock, + type WorkflowLockV1, +} from '@sim/workflow-code' +import { diagnostic } from '@sim/workflow-code/diagnostics' +import { SOURCE_ENTRY_FILE } from '@sim/workflow-code/limits' +import type { WorkflowSourceDiagnostic } from '@/lib/workflows/application/source-error' +import { createLocator, hasErrors } from '@/lib/workflows/application/source-locate' +import type { SourceEvaluationInput } from '@/lib/workflows/application/source-pipeline/evaluate/types' +import { compileSource } from '@/lib/workflows/code/compile-source' + +type SourceCompile = ReturnType + +/** The compile phase's result: the submission compiled against the stored graph under its lock. */ +export interface CompiledSubmission { + lock: WorkflowLockV1 + /** + * The stored graph as the submitted lock reads it: what `stored()` keeps values from, what + * materialize matches against, and what the diff compares the compiled source against. + */ + storedCanonical: CanonicalizeResult + /** What the compile read, kept to compile again with the renames materialize infers. */ + compileInput: Omit[0], 'stored' | 'lock'> + compiled: SourceCompile + compiledWorkflow: NonNullable + /** The stored name and description, or for a create the caller's overrides of the source's. */ + meta: CanonicalWorkflowMeta +} + +/** + * Parses the submitted lock and compiles the files against the live catalog and the stored graph + * read under that lock. Refuses a lock that does not parse, or a source that does not compile + * (noting a catalog that moved since the read, which can explain the errors). + */ +export function compileSubmission( + input: SourceEvaluationInput +): { refused: WorkflowSourceDiagnostic[] } | CompiledSubmission { + const { catalog, files } = input + + const parsedLock = parseWorkflowLock(input.lock) + if (parsedLock.diagnostics) { + return { refused: parsedLock.diagnostics.map(createLocator(files, {}, new Map())) } + } + const lock = parsedLock.lock + + const { state } = input.stored + const storedCanonical = canonicalizeUnderLock( + state, + { catalog, meta: input.stored.meta, resources: input.storedCanonical.resources }, + input.storedCanonical.canonical, + lock + ) + const compileInput = { + entry: { path: SOURCE_ENTRY_FILE, text: files[SOURCE_ENTRY_FILE] }, + files: new Map(Object.entries(files).filter(([path]) => path !== SOURCE_ENTRY_FILE)), + catalog, + sdkMajor: catalog.sdkMajor, + knownKeys: new Set([ + ...Object.keys(lock.blocks), + ...Object.keys(storedCanonical.identity.blocks), + ]), + envNames: input.envNames, + } + const compiled = compileSource({ ...compileInput, stored: storedCanonical.workflow, lock }) + const compiledWorkflow = compiled.workflow + if (!compiledWorkflow || hasErrors(compiled.diagnostics)) { + const refused: Diagnostic[] = [...compiled.diagnostics] + if (input.base && input.base.catalogHash !== catalog.hash) { + refused.push( + diagnostic( + 'W_APPLY_CATALOG_MOVED', + 'The block catalog changed since this source was read, which can explain these errors.', + { + hint: 'Get the source again to print it against the current catalog, then reapply your edit.', + } + ) + ) + } + return { refused: refused.map(createLocator(files, compiled.spans, new Map())) } + } + + const meta = input.create + ? { + name: input.create.name ?? compiledWorkflow.meta.name, + description: + input.create.description !== undefined + ? input.create.description + : compiledWorkflow.meta.description, + } + : input.stored.meta + return { lock, storedCanonical, compileInput, compiled, compiledWorkflow, meta } +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/evaluate/finalize.ts b/apps/sim/lib/workflows/application/source-pipeline/evaluate/finalize.ts new file mode 100644 index 00000000000..6ede61f3432 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/evaluate/finalize.ts @@ -0,0 +1,267 @@ +import type { + LocatedDiagnostic, + LockIdentity, + WorkflowLockBlock, + WorkflowLockV1, +} from '@sim/workflow-code' +import { SOURCE_ENTRY_FILE } from '@sim/workflow-code/limits' +import type { WorkspaceKind } from '@sim/workspace-code' +import { layoutChangedBlocks } from '@/lib/workflows/application/graph-write' +import { snapshotMcpToolOutputSchemas } from '@/lib/workflows/application/mcp-output-schemas' +import { summarizeDiff } from '@/lib/workflows/application/source-diff' +import { + hasErrors, + lintDiagnostics, + normalizeChangedValues, +} from '@/lib/workflows/application/source-locate' +import type { CompiledSubmission } from '@/lib/workflows/application/source-pipeline/evaluate/compile' +import type { GatedGraph } from '@/lib/workflows/application/source-pipeline/evaluate/gates' +import { + editedCompileFindings, + type SourceRefusals, +} from '@/lib/workflows/application/source-pipeline/evaluate/refusals' +import type { ResolvedSubmission } from '@/lib/workflows/application/source-pipeline/evaluate/resolve' +import type { + PendingResource, + SourceEvaluation, + SourceEvaluationInput, +} from '@/lib/workflows/application/source-pipeline/evaluate/types' +import { foreignResourceRefusals } from '@/lib/workflows/application/source-resource-scope' +import { normalizeWorkflowVariables } from '@/lib/workflows/application/workflow-variables' +import { mergeSourceMetaFromLock } from '@/lib/workflows/code/source-lock' +import type { + WorkflowLintReport, + WorkflowLintTableSchema, + WorkflowLintUnresolvedReference, +} from '@/lib/workflows/editing/lint' +import { buildWorkflowLintReport } from '@/lib/workflows/editing/lint-report' +import { parseWorkflowVariables } from '@/lib/workflows/variables/parse' + +/** The schemas `pending` declares for its tables, by id, for the lint's table checks. */ +function declaredTableSchemas( + pending: readonly PendingResource[] +): Map { + const schemas = new Map() + for (const entry of pending) { + if (entry.kind !== 'table' || !entry.columns) continue + schemas.set(entry.id, { name: entry.name ?? entry.key, columnNames: entry.columns }) + } + return schemas +} + +/** + * The lint reference kind an unresolved id of each workspace source kind reports as, or `null` for + * a kind no workflow names by id (a secret, read by name; a deployment, which nothing references). + * Every kind an enclosing apply can create or bind is listed, so a new one fails to compile until + * it is mapped. + */ +const PENDABLE_REFERENCE_OF: Record = + { + secret: null, + credential: 'credential', + 'custom-tool': 'custom-tool', + skill: 'skill', + table: 'resource', + 'knowledge-base': 'resource', + file: 'resource', + workflow: 'resource', + 'mcp-server': 'mcp-tool', + deployment: null, + } + +/** Unresolved reference kinds whose ids an enclosing apply may create. */ +const PENDABLE_REFERENCES: ReadonlySet = new Set( + Object.values(PENDABLE_REFERENCE_OF).filter((kind) => kind !== null) +) + +/** + * `lint` without the unresolved resource references that name only ids an enclosing apply has yet + * to write: a dry run of a directory that adds a skill and the workflow using it is not a warning. + */ +function withoutPendingReferences( + lint: WorkflowLintReport, + pending: readonly PendingResource[] +): WorkflowLintReport { + if (pending.length === 0) return lint + const pendingIds = new Set(pending.map((entry) => entry.id)) + const awaitsPending = (value: unknown) => { + const ids = typeof value === 'string' ? value.split(',').map((id) => id.trim()) : value + return ( + Array.isArray(ids) && + ids.length > 0 && + ids.every((id) => typeof id === 'string' && pendingIds.has(id)) + ) + } + return { + ...lint, + unresolvedReferences: lint.unresolvedReferences.filter( + (reference) => !PENDABLE_REFERENCES.has(reference.kind) || !awaitsPending(reference.value) + ), + } +} + +/** + * `lint` without the block-output findings compile already reported at their source line, so a + * dangling reference is one finding, as an offline check reports it: on a block whose + * `workflow.ts` reads a path no output has (`W_UNKNOWN_OUTPUT_PATH`, or its refusal), every + * block-output finding; on a block whose code file `$ref` names nothing (`W_UNRESOLVED_REF`, or + * its refusal), the one on its code. + */ +function withoutCompiledReferences( + lint: WorkflowLintReport, + compiled: readonly LocatedDiagnostic[], + keyOf: (id: string) => string | undefined +): WorkflowLintReport { + const inEntry = new Set() + const inCode = new Set() + for (const { code, blockKey, span } of compiled) { + if (blockKey === undefined) continue + if (span.file === SOURCE_ENTRY_FILE) { + if (code === 'W_UNKNOWN_OUTPUT_PATH' || code === 'E_APPLY_UNRESOLVED_REF') + inEntry.add(blockKey) + } else if (code === 'W_UNRESOLVED_REF' || code === 'E_APPLY_UNRESOLVED_REF') { + inCode.add(blockKey) + } + } + if (inEntry.size === 0 && inCode.size === 0) return lint + return { + ...lint, + unresolvedReferences: lint.unresolvedReferences.filter((reference) => { + const key = keyOf(reference.blockId) + if (reference.kind !== 'block-output' || key === undefined) return true + return !inEntry.has(key) && !(reference.field === 'code' && inCode.has(key)) + }), + } +} + +/** + * The post-materialize lock `mergeSourceMetaFromLock` re-keys onto: every bound key with its id, + * and the submitted lock's ident and order for a block the file does not state them for. + */ +function boundLock(identity: LockIdentity, submitted: WorkflowLockV1): WorkflowLockV1 { + const blocks: Record = {} + for (const [key, block] of Object.entries(identity.blocks)) { + const previous = Object.hasOwn(submitted.blocks, key) ? submitted.blocks[key] : undefined + blocks[key] = { ...block, ident: previous?.ident ?? key, order: previous?.order ?? 0 } + } + return { version: 1, blocks, variables: identity.variables } +} + +/** + * Normalizes the changed values, snapshots MCP tool output schemas, lays out new blocks, lints the + * graph, refuses ids of another workspace's resources, and gathers every finding pinned to the + * submitted files. Refuses on any error; otherwise assembles the plan the write stores. + */ +export async function finalizePlan( + input: SourceEvaluationInput, + submission: CompiledSubmission, + resolved: ResolvedSubmission, + refusals: SourceRefusals, + gated: GatedGraph +): Promise { + const { principal, scope } = input + const { state } = input.stored + const isNew = input.create !== undefined + const { lock, compiled, compiledWorkflow } = submission + const { materialized, keyById, locate, compiledKeys } = resolved + const { canonicalDiff, metaChanged, meta } = refusals + const { storedGraph, graphDiff, edges } = gated + const pending = input.pendingResources ?? [] + + const normalized = normalizeChangedValues(gated.blocks, graphDiff, (id) => keyById.get(id) ?? '') + let blocks = await snapshotMcpToolOutputSchemas(normalized.blocks, state.blocks, { + principal, + workspaceId: scope.workspaceId, + }) + if (materialized.needsLayout.length > 0) { + blocks = layoutChangedBlocks(scope.workflowId, storedGraph, { blocks, edges }) + } + + const lint = await buildWorkflowLintReport( + { blocks, edges }, + { + workflowId: scope.workflowId, + workspaceId: scope.workspaceId, + subjectUserId: scope.subjectUserId, + }, + { declaredTables: declaredTableSchemas(pending), iterationScope: false } + ) + const changedIds = new Set([...graphDiff.added, ...graphDiff.modified.map((m) => m.id)]) + const changedKeys = new Set([...changedIds].flatMap((id) => keyById.get(id) ?? [])) + const keyOfId = (id: string) => keyById.get(id) + const compileDiagnostics = editedCompileFindings(refusals, changedKeys) + const compiledLint = withoutCompiledReferences( + withoutPendingReferences(lint, pending), + compileDiagnostics, + keyOfId + ) + const foreignIds = await foreignResourceRefusals({ + blocks, + diff: graphDiff, + workspaceId: scope.workspaceId, + pending, + keyOf: keyOfId, + }) + const diagnostics = [ + ...compileDiagnostics, + ...resolved.keptDiagnostics, + ...resolved.materializeDiagnostics, + ...refusals.sourceRefusals, + ...refusals.renameWarnings, + ...gated.gateFindings, + ...normalized.diagnostics, + ...foreignIds, + ...lintDiagnostics(compiledLint, changedIds, keyOfId), + ].map(locate) + if (hasErrors(diagnostics)) { + return { + refused: diagnostics, + compiledReferences: { references: compiledKeys, envReferences: compiled.envReferences }, + } + } + + const noop = !isNew && canonicalDiff.noop + const mintedIds: Record = {} + for (const entry of materialized.entries) { + if (entry.status === 'created' && entry.idSource === 'minted') mintedIds[entry.key] = entry.id + } + const { added, removed, modified } = canonicalDiff.variables + const variables = + added.length > 0 || removed.length > 0 || modified.length > 0 + ? normalizeWorkflowVariables(materialized.state.variables ?? {}, { coerceValues: true }) + : undefined + const compiledSource = { hints: compiled.hints, workflow: compiledWorkflow } + const lockAfter = boundLock(materialized.identity, lock) + + return { + plan: { + meta, + noop, + blocks, + edges, + variables, + storedVariables: + variables === undefined ? state.variables : (parseWorkflowVariables(variables) ?? {}), + diagnostics, + diff: summarizeDiff( + canonicalDiff, + resolved.storedCanonical, + { workflow: materialized.workflow, identity: materialized.identity }, + noop, + metaChanged + ), + metaChanged, + mintedIds, + identity: materialized.identity, + lock: lockAfter, + compiled: compiledSource, + sourceMeta: mergeSourceMetaFromLock(input.storedSourceMeta, lockAfter, compiledSource), + functionCode: Object.values(compiled.hints.blocks).some((hint) => hint.codeFile) + ? 'files' + : 'inline', + resources: resolved.resources, + references: compiledKeys, + envReferences: compiled.envReferences, + }, + } +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/evaluate/gates.ts b/apps/sim/lib/workflows/application/source-pipeline/evaluate/gates.ts new file mode 100644 index 00000000000..c366da5f9ea --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/evaluate/gates.ts @@ -0,0 +1,82 @@ +import type { Diagnostic } from '@sim/workflow-code' +import type { BlockState, WorkflowState } from '@sim/workflow-types/workflow' +import type { ResolvedSubmission } from '@/lib/workflows/application/source-pipeline/evaluate/resolve' +import type { SourceEvaluationInput } from '@/lib/workflows/application/source-pipeline/evaluate/types' +import { gateSourceDiff } from '@/lib/workflows/code/apply/gates' +import { diffGraphs, type SourceGraph } from '@/lib/workflows/code/apply/graph-diff' +import { findLiveTriggerRemovals } from '@/lib/workflows/code/apply/live-triggers' +import { findLockedBlockRefusals } from '@/lib/workflows/code/apply/locked' +import { findInvalidSchedules } from '@/lib/workflows/code/apply/schedules' + +/** The gates phase's result: the graph diff and every gate's findings. */ +export interface GatedGraph { + /** The stored graph. */ + storedGraph: SourceGraph + graphDiff: ReturnType + /** The materialized blocks, with every hosted key a gate drops cleared. */ + blocks: Record + edges: WorkflowState['edges'] + /** Locked blocks, invalid schedules, engine gates and live triggers removed, in that order. */ + gateFindings: Diagnostic[] +} + +/** + * Diffs the materialized graph against the stored one and runs every gate on the diff: locked + * blocks, schedules, the engine gates (block types, permission groups, hosted keys) and live + * triggers a removal or retype would drop. A hosted key a gate drops is cleared from its block. + */ +export async function gateSourceGraph( + input: SourceEvaluationInput, + resolved: ResolvedSubmission +): Promise { + const { principal, scope, catalog } = input + const { state } = input.stored + const { materialized, keyOf, permissionConfig } = resolved + + const storedGraph: SourceGraph = { blocks: state.blocks, edges: state.edges } + let blocks = materialized.state.blocks + const edges = materialized.state.edges + const graphDiff = diffGraphs(storedGraph, { blocks, edges }) + + const typeChanged = graphDiff.modified.filter((change) => change.typeChanged).map((c) => c.id) + const [gateDiagnostics, liveTriggerRefusals] = await Promise.all([ + gateSourceDiff({ + base: storedGraph, + target: { blocks, edges }, + diff: graphDiff, + catalog, + keyOf, + permissionConfig, + subjectUserId: scope.subjectUserId, + workspaceId: scope.workspaceId, + isCopilot: principal.kind === 'delegated' && principal.serviceId === 'copilot', + }), + findLiveTriggerRemovals( + scope.workflowId, + [...graphDiff.removed, ...typeChanged].map((id) => state.blocks[id]), + keyOf + ), + ]) + const gateFindings: Diagnostic[] = [ + ...findLockedBlockRefusals(storedGraph, { blocks, edges }, graphDiff, keyOf), + ...findInvalidSchedules({ blocks, edges }, graphDiff, keyOf), + ...gateDiagnostics, + ...liveTriggerRefusals, + ] + + for (const diagnostic of gateDiagnostics) { + if (diagnostic.code !== 'W_APPLY_HOSTED_KEY_DROPPED' || !diagnostic.blockId) continue + const { blockId, param } = diagnostic + const block = blocks[blockId] + if (!param || !block?.subBlocks?.[param]) continue + blocks = { + ...blocks, + [blockId]: { + ...block, + subBlocks: { ...block.subBlocks, [param]: { ...block.subBlocks[param], value: null } }, + }, + } + } + + return { storedGraph, graphDiff, blocks, edges, gateFindings } +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/evaluate/index.ts b/apps/sim/lib/workflows/application/source-pipeline/evaluate/index.ts new file mode 100644 index 00000000000..e8f15e6f806 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/evaluate/index.ts @@ -0,0 +1,8 @@ +export { evaluateSource } from './run' +export type { + CompiledReferences, + PendingResource, + SourcePlan, + SourceReference, + SourceScope, +} from './types' diff --git a/apps/sim/lib/workflows/application/source-pipeline/evaluate/refusals.ts b/apps/sim/lib/workflows/application/source-pipeline/evaluate/refusals.ts new file mode 100644 index 00000000000..94d2a521c2b --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/evaluate/refusals.ts @@ -0,0 +1,372 @@ +import { + type CanonicalBlock, + type CanonicalDiff, + type CanonicalWorkflowMeta, + type CompileResult, + compileWorkflow, + type Diagnostic, + diffCanonical, + encodeAnchor, + getOwn, + type LocatedDiagnostic, + refuseEditedFindings, + renamesBetween, + replacedBetween, + type SourceSpan, +} from '@sim/workflow-code' +import { diagnostic } from '@sim/workflow-code/diagnostics' +import { isEqual } from 'es-toolkit' +import { + V2_WORKFLOW_DESCRIPTION_MAX_LENGTH, + V2_WORKFLOW_NAME_MAX_LENGTH, +} from '@/lib/api/contracts/v2/workflows' +import type { CompiledSubmission } from '@/lib/workflows/application/source-pipeline/evaluate/compile' +import type { ResolvedSubmission } from '@/lib/workflows/application/source-pipeline/evaluate/resolve' +import type { SourceEvaluationInput } from '@/lib/workflows/application/source-pipeline/evaluate/types' +import { workflowNameExistsInFolder } from '@/lib/workflows/orchestration' + +const META_FIELDS: Array = ['name', 'description'] + +/** + * Findings a block rename can settle: a code-file `W_UNRESOLVED_REF` (an apply rewrites every + * `` to the new key, so `$ref.` runs) and a `W_JSON_REF_UNTYPED` on a JSON + * field whose only change is that rewrite. + */ +const RENAME_SETTLED_CODES: ReadonlySet = new Set([ + 'W_UNRESOLVED_REF', + 'W_JSON_REF_UNTYPED', +]) + +/** + * `diagnostics` without the {@link RENAME_SETTLED_CODES} findings a block rename settles. The + * compile already knew the renames the lock names by ident; a rename materialize inferred (name + * and const both changed) is known only now, so the source is compiled again with every rename and + * its findings of those codes, listed and withheld, replace the first compile's. Taken whole, not + * matched by position: the first compile withholds one finding per code, block and file, so a + * settled one may stand for a later finding in that file that the rename leaves unresolved. + */ +function withoutRenameSettledFindings( + compiled: Pick, + renames: ReadonlyMap, + recompile: () => CompileResult +): Pick { + const settles = (d: CompileResult['diagnostics'][number]) => RENAME_SETTLED_CODES.has(d.code) + if (renames.size === 0 || ![...compiled.diagnostics, ...compiled.withheld].some(settles)) { + return compiled + } + const again = recompile() + const unsettled = (d: CompileResult['diagnostics'][number]) => !settles(d) + return { + diagnostics: [...compiled.diagnostics.filter(unsettled), ...again.diagnostics.filter(settles)], + withheld: [...compiled.withheld.filter(unsettled), ...again.withheld.filter(settles)], + } +} + +/** `{ span }` when there is one, to spread into a diagnostic. */ +function withSpan(span: SourceSpan | undefined): { span?: SourceSpan } { + return span ? { span } : {} +} + +/** Names in a refusal, the first ten and a count of the rest. */ +function listNames(names: readonly string[]): string { + const shown = names.slice(0, 10).join(', ') + return names.length > 10 ? `${shown}, and ${names.length - 10} more` : shown +} + +/** A variable as a delete refusal names it. */ +interface NamedVariable { + name: string + type: string +} + +/** + * `E_APPLY_VARIABLE_DELETE` for the stored variables an apply removes, by name. When it removes + * one and adds one of the same type, the edit may be a rename the apply could not tie to the + * stored variable (its value changed too, or says nothing), so the hint says how to keep the id. + */ +function variableDeleteRefusal( + removed: readonly NamedVariable[], + added: readonly NamedVariable[], + span: SourceSpan | undefined +): Diagnostic { + const names = removed.map((variable) => variable.name) + const [former] = removed + const [next] = added + const rename = + removed.length === 1 && added.length === 1 && former.type === next.type + ? `If ${next.name} renames ${former.name}, apply the new key with the value ${former.name} holds first and change the value in a second apply, so the variable keeps its id; or pass --allow-delete (allowDelete=true) to replace it.` + : undefined + return diagnostic( + 'E_APPLY_VARIABLE_DELETE', + `This apply deletes ${names.length} variable${names.length === 1 ? '' : 's'} the source no longer declares: ${listNames(names)}.`, + { + hint: + rename ?? + 'Pass --allow-delete (allowDelete=true) to delete them, or declare them again in workflow(name, { variables }) to keep them.', + ...withSpan(span), + } + ) +} + +/** + * Each removed block with the added block of the same type and parent it most probably became: + * one pair per removed block, in order, each added block used once. + */ +function probableRenames( + removed: readonly CanonicalBlock[], + added: readonly CanonicalBlock[] +): Array<{ removed: CanonicalBlock; added: CanonicalBlock }> { + const free = [...added] + return removed.flatMap((block) => { + const index = free.findIndex( + (candidate) => candidate.type === block.type && candidate.parent === block.parent + ) + if (index === -1) return [] + const [match] = free.splice(index, 1) + return [{ removed: block, added: match }] + }) +} + +/** + * `E_APPLY_BLOCK_DELETE` for the stored blocks an apply removes. When the source adds blocks of + * a removed block's type, they may be renames whose name and const both changed with no identity + * signal tying them to the stored blocks (settings, connections or renamed references): the + * finding names them and says how to keep the ids, or to pass `--allow-delete` to replace them. + */ +function blockDeleteRefusal( + removed: readonly CanonicalBlock[], + added: readonly CanonicalBlock[], + span: SourceSpan | undefined +): Diagnostic { + const removedTypes = new Set(removed.map((block) => block.type)) + const lookalikes = added.filter((block) => removedTypes.has(block.type)) + const deletes = `This apply deletes ${removed.length} block${removed.length === 1 ? '' : 's'} the source no longer declares: ${listNames(removed.map((block) => block.name))}.` + if (lookalikes.length > 0) { + return diagnostic( + 'E_APPLY_BLOCK_DELETE', + `${deletes} It adds ${listNames(lookalikes.map((block) => block.name))} of the same type: if those are the same blocks renamed, both their name and const changed, so nothing ties them to the stored blocks and they would get new ids.`, + { + hint: 'To rename a block and keep its id, keep its const name in this apply (change it in the next), or rename the references to it with it; to replace it with a new block, pass --allow-delete.', + ...withSpan(span), + } + ) + } + return diagnostic('E_APPLY_BLOCK_DELETE', deletes, { + hint: 'Pass --allow-delete (allowDelete=true) to delete them, or declare them again to keep them.', + ...withSpan(span), + }) +} + +/** + * The name and description fields among `fields` that v2 create and update would refuse: an empty + * name, or a name or description over its limit. Only the fields a write sets are checked, so a + * stored workflow the source keeps as it is never refuses. + */ +function metaLimitRefusals( + meta: CanonicalWorkflowMeta, + fields: readonly (keyof CanonicalWorkflowMeta)[] +): Array<{ field: keyof CanonicalWorkflowMeta; refusal: Diagnostic }> { + const refusals: Array<{ field: keyof CanonicalWorkflowMeta; refusal: Diagnostic }> = [] + const refuse = (field: keyof CanonicalWorkflowMeta, message: string, hint: string) => + refusals.push({ + field, + refusal: diagnostic('E_WORKFLOW_META_INVALID', message, { hint }), + }) + if (fields.includes('name') && meta.name.trim() === '') { + refuse('name', 'The workflow name is empty.', "Name the workflow: workflow('Name', ...).") + } else if (fields.includes('name') && meta.name.length > V2_WORKFLOW_NAME_MAX_LENGTH) { + refuse( + 'name', + `The workflow name is ${meta.name.length} characters, over the ${V2_WORKFLOW_NAME_MAX_LENGTH}-character limit.`, + `Shorten the name to ${V2_WORKFLOW_NAME_MAX_LENGTH} characters or fewer.` + ) + } + const description = meta.description ?? '' + if (fields.includes('description') && description.length > V2_WORKFLOW_DESCRIPTION_MAX_LENGTH) { + refuse( + 'description', + `The workflow description is ${description.length} characters, over the ${V2_WORKFLOW_DESCRIPTION_MAX_LENGTH}-character limit.`, + `Shorten the description to ${V2_WORKFLOW_DESCRIPTION_MAX_LENGTH} characters or fewer.` + ) + } + return refusals +} + +/** The source-level refusals phase's result. */ +export interface SourceRefusals { + /** The name and description after the write. */ + meta: CanonicalWorkflowMeta + /** The name and description fields the write changes on a stored workflow. */ + metaChanged: Array + /** What the write stores against what is stored, on the canonical models. */ + canonicalDiff: CanonicalDiff + /** The compile's findings with those a block rename settles taken out. */ + compileFindings: Pick + /** Whether the source changed the code of block `key` from what is stored. */ + codeEdited: (key: string) => boolean + /** Name, meta limit, block delete and variable delete refusals. */ + sourceRefusals: Diagnostic[] + /** `W_APPLY_PROBABLE_RENAME` for each delete-and-add that is probably a rename. */ + renameWarnings: Diagnostic[] +} + +/** + * Diffs the materialized source against the stored graph and refuses what the source itself + * asks for and may not do: a rename onto a name taken in the folder, a name or description over + * its limit, and (without `allowDelete`) deleting blocks or variables. With `allowDelete`, a delete + * and add of the same type and parent warns that it is probably a rename. + */ +export async function sourceRefusals( + input: SourceEvaluationInput, + submission: CompiledSubmission, + resolved: ResolvedSubmission +): Promise { + const { scope } = input + const isNew = input.create !== undefined + const { compiled, compiledWorkflow, compileInput } = submission + const { storedCanonical, materialized } = resolved + let { meta } = submission + + /** + * The name and description the source gives a stored workflow, where they differ from the + * stored ones: the write renames or redescribes the workflow (the graph diff never sees them). + */ + const metaChanged = isNew + ? [] + : META_FIELDS.filter( + (field) => (compiledWorkflow.meta[field] ?? null) !== (meta[field] ?? null) + ) + const storedMeta = meta + if (metaChanged.length > 0) { + meta = { + name: compiledWorkflow.meta.name, + description: compiledWorkflow.meta.description ?? null, + } + } + /** + * What the write stores against what is stored: the materialized CWM, not the compiled one, + * so a field source never prints (a forEach loop's empty while condition, a pair's stored + * member) that materialize keeps from the stored block is no change. + */ + const blockRenames = renamesBetween(storedCanonical.identity, materialized.identity) + const replacedKeys = replacedBetween(storedCanonical.identity, materialized.identity) + const canonicalDiff = diffCanonical( + storedCanonical.workflow, + { ...materialized.workflow, meta: storedCanonical.workflow.meta }, + blockRenames, + replacedKeys + ) + const renamedKeys = new Map([...blockRenames].map(([key, storedKey]) => [storedKey, key])) + const storedKeyOf = new Map([...renamedKeys].map(([storedKey, key]) => [key, storedKey])) + const storedCode = (key: string) => { + const storedBlock = + getOwn(compiled.stored.blocks, key) ?? + getOwn(compiled.stored.blocks, storedKeyOf.get(key) ?? key) + return storedBlock && getOwn(storedBlock.params, 'code') + } + const compileFindings = withoutRenameSettledFindings(compiled, renamedKeys, () => + compileWorkflow({ ...compileInput, stored: compiled.stored, renamedKeys }) + ) + const codeEdited = (key: string) => { + const compiledBlock = getOwn(compiledWorkflow.blocks, key) + return !isEqual(compiledBlock && getOwn(compiledBlock.params, 'code'), storedCode(key)) + } + const refusals: Diagnostic[] = [] + const headerSpan = compiled.spans[encodeAnchor({ kind: 'header' })] + /** Where a finding about the workflow itself points: the field it names, else the name. */ + const metaSpan = (field: keyof CanonicalWorkflowMeta = 'name') => + compiled.metaSpans[field] ?? compiled.metaSpans.name ?? headerSpan + if ( + metaChanged.includes('name') && + (await workflowNameExistsInFolder({ + workspaceId: scope.workspaceId, + name: meta.name, + folderId: scope.folderId, + excludeWorkflowId: scope.workflowId, + })) + ) { + refusals.push( + diagnostic( + 'E_NAME_TAKEN', + `Another workflow in this workflow's folder is named ${JSON.stringify(meta.name)}.`, + { + hint: `Choose another name, or keep workflow(${JSON.stringify(storedMeta.name)}, ...).`, + ...withSpan(metaSpan('name')), + } + ) + ) + } + refusals.push( + ...metaLimitRefusals(meta, isNew ? META_FIELDS : metaChanged).map(({ field, refusal }) => ({ + ...refusal, + ...withSpan(metaSpan(field)), + })) + ) + // A block replaced under its key (a retype) is reported removed and added, but it is the + // same declaration, which W_MAT_RETYPE_RECREATED covers, so it needs no allowDelete. + const removedKeys = isNew + ? [] + : canonicalDiff.blocks.removed.filter((key) => !replacedKeys.has(key)) + const removedBlocks = removedKeys.map((key) => storedCanonical.workflow.blocks[key]) + const addedBlocks = canonicalDiff.blocks.added + .filter((key) => !replacedKeys.has(key)) + .map((key) => materialized.workflow.blocks[key]) + if (removedKeys.length > 0 && !input.allowDelete) { + refusals.push(blockDeleteRefusal(removedBlocks, addedBlocks, metaSpan())) + } + const keptVariableIds = new Set(Object.values(materialized.identity.variables)) + const storedVariableIds = new Set(Object.values(storedCanonical.identity.variables)) + const removedVariables = isNew + ? [] + : Object.entries(storedCanonical.identity.variables).flatMap(([key, id]) => { + if (keptVariableIds.has(id)) return [] + const variable = getOwn(storedCanonical.workflow.variables, key) + return [{ name: variable?.name ?? key, type: variable?.type ?? '' }] + }) + if (removedVariables.length > 0 && !input.allowDelete) { + const addedVariables = Object.entries(materialized.identity.variables).flatMap(([key, id]) => { + const variable = getOwn(materialized.workflow.variables, key) + return storedVariableIds.has(id) || !variable + ? [] + : [{ name: variable.name, type: variable.type }] + }) + refusals.push(variableDeleteRefusal(removedVariables, addedVariables, metaSpan())) + } + const renameWarnings = input.allowDelete + ? probableRenames(removedBlocks, addedBlocks).map( + ({ removed, added }): Diagnostic => + diagnostic( + 'W_APPLY_PROBABLE_RENAME', + `${removed.name} (${removed.type}) is deleted and ${added.name} (${added.type}) added: probably a rename, but its name and const both changed, so it gets a new id and loses its run history and webhook URL.`, + { + hint: `To keep the id, apply ${added.name} under the const or name ${removed.name} had first, then change the other.`, + blockKey: added.key, + } + ) + ) + : [] + + return { + meta, + metaChanged, + canonicalDiff, + compileFindings, + codeEdited, + sourceRefusals: refusals, + renameWarnings, + } +} + +/** + * The compile's findings, each on a block this write changes (or whose code it edits) refused as + * the shared edited-block rule says, so a warning the author introduced cannot be written. + */ +export function editedCompileFindings( + refusals: Pick, + changedKeys: ReadonlySet +): LocatedDiagnostic[] { + return refuseEditedFindings(refusals.compileFindings, { + changed: (key) => changedKeys.has(key), + codeEdited: refusals.codeEdited, + }) +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/evaluate/resolve.ts b/apps/sim/lib/workflows/application/source-pipeline/evaluate/resolve.ts new file mode 100644 index 00000000000..6cacb3d0f07 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/evaluate/resolve.ts @@ -0,0 +1,324 @@ +import { db } from '@sim/db' +import { workflowBlocks } from '@sim/db/schema' +import { isRecordLike } from '@sim/utils/object' +import { + type CanonicalizeResult, + canonicalize, + canonicalizeUnderLock, + type Diagnostic, + encodeAnchor, + isResourceKind, + type MaterializeResult, + materialize, + type ResourceKeyResolver, + type ResourceKind, + type SourceSpan, + type WorkflowLockBlock, + type WorkflowLockV1, +} from '@sim/workflow-code' +import { diagnostic } from '@sim/workflow-code/diagnostics' +import { and, inArray, ne } from 'drizzle-orm' +import { resolvePermissionGroupConfig } from '@/lib/permission-groups/config-scope.server' +import type { WorkflowSourceDiagnostic } from '@/lib/workflows/application/source-error' +import { createLocator, hasErrors } from '@/lib/workflows/application/source-locate' +import type { CompiledSubmission } from '@/lib/workflows/application/source-pipeline/evaluate/compile' +import type { + CompiledReferences, + PendingResource, + SourceEvaluationInput, + SourceReference, +} from '@/lib/workflows/application/source-pipeline/evaluate/types' +import { keptCredentials } from '@/lib/workflows/code/credential-nodes' +import { createMaterializeHooks } from '@/lib/workflows/code/materialize-hooks' +import { loadAvailableResourceKeys, loadResourceKeyResolver } from '@/lib/workflows/code/resources' + +/** + * `primary`, falling back to `fallback` for a key, id or name it does not resolve. Whether an id + * is built in is `primary`'s answer alone. + */ +function withFallback( + primary: ResourceKeyResolver, + fallback: Pick +): ResourceKeyResolver { + return { + toKey: (kind, id) => primary.toKey(kind, id) ?? fallback.toKey(kind, id), + toId: (kind, key) => primary.toId(kind, key) ?? fallback.toId(kind, key), + nameOf: (kind, id) => primary.nameOf?.(kind, id) ?? fallback.nameOf?.(kind, id) ?? null, + isBuiltin: primary.isBuiltin, + } +} + +/** `resources`, falling back to `pending` for a key, id or name it does not resolve. */ +function withPendingResources( + resources: ResourceKeyResolver, + pending: readonly PendingResource[] +): ResourceKeyResolver { + if (pending.length === 0) return resources + const byId = (kind: ResourceKind, id: string) => + pending.find((entry) => entry.kind === kind && entry.id === id) + return withFallback(resources, { + toKey: (kind, id) => byId(kind, id)?.key ?? null, + toId: (kind, key) => + pending.find((entry) => entry.kind === kind && entry.key === key)?.id ?? null, + nameOf: (kind, id) => byId(kind, id)?.name ?? null, + }) +} + +/** + * Every `(kind, key)` resource reference in a compiled model, for ident lookup, and the tool each + * MCP agent tool (`mcp(key).tool(name)`) names on its server. + */ +function collectResourceKeys( + value: unknown, + refs: SourceReference[], + span: SourceSpan | undefined +): void { + if (Array.isArray(value)) { + for (const item of value) collectResourceKeys(item, refs, span) + return + } + if (value === null || typeof value !== 'object') return + const record = value as Record + if ( + record.$sim === 'resource' && + typeof record.kind === 'string' && + isResourceKind(record.kind) && + typeof record.key === 'string' + ) { + refs.push({ kind: record.kind, key: record.key }) + } + const tool = mcpToolReference(record) + if (tool) refs.push(span ? { ...tool, span } : tool) + for (const nested of Object.values(record)) collectResourceKeys(nested, refs, span) +} + +/** The server and tool an MCP agent tool entry names, or `null` for anything else. */ +function mcpToolReference(record: Record): SourceReference | null { + if (record.type !== 'mcp' || !isRecordLike(record.params)) return null + const { serverId, toolName } = record.params + if ( + !isRecordLike(serverId) || + serverId.$sim !== 'resource' || + serverId.kind !== 'mcp-server' || + typeof serverId.key !== 'string' || + typeof toolName !== 'string' + ) { + return null + } + return { kind: 'mcp-server', key: serverId.key, member: toolName } +} + +/** Block ids another workflow already stores, among `ids`. */ +async function findBlockIdsClaimedElsewhere(workflowId: string, ids: string[]): Promise { + if (ids.length === 0) return [] + const rows = await db + .select({ id: workflowBlocks.id }) + .from(workflowBlocks) + .where(and(inArray(workflowBlocks.id, ids), ne(workflowBlocks.workflowId, workflowId))) + return rows.map((row) => row.id) +} + +/** The resolve phase's result: the compiled source materialized onto the stored graph. */ +export interface ResolvedSubmission { + /** Answers every resource reference of the stored and submitted source, pending ones included. */ + resources: ResourceKeyResolver + /** The stored graph under the lock, re-read under the keys of any credential the source keeps. */ + storedCanonical: CanonicalizeResult + /** Every resource the compiled source names by key. */ + compiledKeys: SourceReference[] + /** `W_APPLY_CREDENTIAL_KEPT` for each stored credential kept under a key the subject cannot see. */ + keptDiagnostics: Diagnostic[] + materialized: MaterializeResult + /** Materialize's findings and the lock ids it could not reuse. */ + materializeDiagnostics: Diagnostic[] + permissionConfig: Awaited> + /** Block id by the key this write binds. */ + idByKey: Map + /** Block key by the id this write binds. */ + keyById: Map + /** A block's source key: the one this write binds, else the stored one for a removed block. */ + keyOf: (id: string) => string | undefined + /** Pins a finding to the submitted files. */ + locate: ReturnType +} + +/** + * Resolves the resources the compiled source names (stored keys, then keys new to it, then the + * stored credentials it keeps, then what an enclosing apply creates first) and materializes it + * onto the stored graph, keeping every id the lock binds that no other workflow stores. Refuses + * the materialize errors, with what the source references. + */ +export async function resolveSubmission( + input: SourceEvaluationInput, + submission: CompiledSubmission +): Promise< + | { refused: WorkflowSourceDiagnostic[]; compiledReferences: CompiledReferences } + | ResolvedSubmission +> { + const { scope, catalog, files } = input + const { lock, compiled, compiledWorkflow, meta } = submission + const { state } = input.stored + let { storedCanonical } = submission + let resources = input.storedCanonical.resources + + const compiledKeys: SourceReference[] = [] + for (const [key, block] of Object.entries(compiledWorkflow.blocks)) { + const span = + compiled.spans[encodeAnchor({ kind: 'param', block: key, param: 'tools' })] ?? + compiled.spans[encodeAnchor({ kind: 'block', block: key })] + collectResourceKeys(block, compiledKeys, span) + } + const storedResources = resources + const newKeys = compiledKeys.filter(({ kind, key }) => storedResources.toId(kind, key) === null) + if (newKeys.length > 0) { + const extra = await loadResourceKeyResolver( + scope.workspaceId, + { keys: newKeys }, + { actorUserId: input.createdBy } + ) + resources = withFallback(storedResources, extra) + } + + const kept = keptCredentials( + compiledWorkflow, + compiled.stored, + (key) => + resources.toId('credential', key) === null && + !(input.pendingResources ?? []).some( + (entry) => entry.kind === 'credential' && entry.key === key + ) + ) + if (kept.length > 0) { + const idsByKey = new Map(kept.map(({ key, id }) => [key, id])) + const keysById = new Map(kept.map(({ key, id }) => [id, key])) + resources = withFallback(resources, { + toKey: (kind, id) => (kind === 'credential' ? (keysById.get(id) ?? null) : null), + toId: (kind, key) => (kind === 'credential' ? (idsByKey.get(key) ?? null) : null), + }) + // The stored graph under the keys the source gives its kept credentials: no change there. + storedCanonical = canonicalizeUnderLock( + state, + { catalog, meta, resources }, + canonicalize(state, { catalog, meta, resources }), + lock + ) + } + const keptDiagnostics: Diagnostic[] = kept.map(({ key, blockKey }) => + diagnostic( + 'W_APPLY_CREDENTIAL_KEPT', + `credential(${JSON.stringify(key)}) stands where the workflow stores a credential you cannot see, so that credential is kept as stored.`, + { + hint: 'Name a credential you can see to change it, or ask a member who can see the stored one.', + blockKey, + } + ) + ) + + resources = withPendingResources(resources, input.pendingResources ?? []) + + const hasLockEntries = + Object.keys(lock.blocks).length > 0 || Object.keys(lock.variables).length > 0 + const lockFreshness: 'fresh' | 'stale' | 'none' = !hasLockEntries + ? 'none' + : input.base && input.base.workspaceId !== scope.workspaceId + ? 'stale' + : 'fresh' + const compiledIdents: Record = {} + for (const [key, hint] of Object.entries(compiled.hints.blocks)) { + if (hint.ident !== undefined) compiledIdents[key] = hint.ident + } + const hooks = createMaterializeHooks() + const prevCanonical = storedCanonical + const materializeWith = (withLock: WorkflowLockV1): MaterializeResult => + materialize(compiledWorkflow, withLock, { + catalog, + hooks, + prev: state, + prevCanonical, + resources, + compiledIdents, + lockFreshness, + }) + + /** + * A lock id for a new block that another workflow already stores cannot be reused (block ids + * are global keys): drop it from the lock so materialize mints one, and say so. + */ + let materialized = materializeWith(lock) + if (materialized.diagnostics.some((d) => d.code === 'E_MAT_UNKNOWN_RESOURCE')) { + // Only on a miss: list the keys the source could have named, for the finding's did-you-mean. + const missingKinds = compiledKeys + .filter(({ kind, key }) => resources.toId(kind, key) === null) + .map(({ kind }) => kind) + if (missingKinds.length > 0) { + const available = await loadAvailableResourceKeys( + scope.workspaceId, + missingKinds, + input.createdBy + ) + const known = resources + resources = { ...known, keysOf: (kind) => available.get(kind) } + materialized = materializeWith(lock) + } + } + const claimedDiagnostics: Diagnostic[] = [] + const lockSuppliedIds = materialized.entries.flatMap((entry) => + entry.status === 'created' && entry.idSource === 'lock' ? [entry.id] : [] + ) + const [claimedIds, permissionConfig] = await Promise.all([ + findBlockIdsClaimedElsewhere(scope.workflowId, lockSuppliedIds), + resolvePermissionGroupConfig( + scope.subjectUserId, + scope.workspaceId, + scope.workspaceOrganizationId + ), + ]) + const claimed = new Set(claimedIds) + if (claimed.size > 0) { + const blocks: Record = {} + for (const [key, entry] of Object.entries(lock.blocks)) { + if (claimed.has(entry.id)) { + claimedDiagnostics.push( + diagnostic( + 'W_MAT_LOCK_ID_CLAIMED', + `The lock id for block "${key}" belongs to another workflow; a new id was minted`, + { blockKey: key } + ) + ) + } else { + blocks[key] = entry + } + } + materialized = materializeWith({ ...lock, blocks }) + } + + const idByKey = new Map( + Object.entries(materialized.identity.blocks).map(([key, block]) => [key, block.id]) + ) + const keyById = new Map([...idByKey].map(([key, id]) => [id, key])) + const storedKeyById = new Map( + Object.entries(storedCanonical.identity.blocks).map(([key, block]) => [block.id, key]) + ) + const keyOf = (id: string) => keyById.get(id) ?? storedKeyById.get(id) + const locate = createLocator(files, compiled.spans, idByKey) + const materializeDiagnostics = [...materialized.diagnostics, ...claimedDiagnostics] + if (hasErrors(materializeDiagnostics)) { + return { + refused: [...compiled.diagnostics, ...materializeDiagnostics].map(locate), + compiledReferences: { references: compiledKeys, envReferences: compiled.envReferences }, + } + } + return { + resources, + storedCanonical, + compiledKeys, + keptDiagnostics, + materialized, + materializeDiagnostics, + permissionConfig, + idByKey, + keyById, + keyOf, + locate, + } +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/evaluate/run.ts b/apps/sim/lib/workflows/application/source-pipeline/evaluate/run.ts new file mode 100644 index 00000000000..a00f70f55cc --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/evaluate/run.ts @@ -0,0 +1,26 @@ +import { compileSubmission } from '@/lib/workflows/application/source-pipeline/evaluate/compile' +import { finalizePlan } from '@/lib/workflows/application/source-pipeline/evaluate/finalize' +import { gateSourceGraph } from '@/lib/workflows/application/source-pipeline/evaluate/gates' +import { sourceRefusals } from '@/lib/workflows/application/source-pipeline/evaluate/refusals' +import { resolveSubmission } from '@/lib/workflows/application/source-pipeline/evaluate/resolve' +import type { + SourceEvaluation, + SourceEvaluationInput, +} from '@/lib/workflows/application/source-pipeline/evaluate/types' + +/** + * Evaluates submitted source against a stored draft without writing it: compile against the live + * catalog, parse the lock, materialize onto the stored graph (keeping every unchanged value's + * stored bytes and every id the lock binds), then every gate (locked blocks, engine gates, + * types that may stay but not be added), the value checks and lint. Returns the refusals, located + * in the submitted text, or the plan the write stores. + */ +export async function evaluateSource(input: SourceEvaluationInput): Promise { + const submission = compileSubmission(input) + if ('refused' in submission) return { refused: submission.refused } + const resolved = await resolveSubmission(input, submission) + if ('refused' in resolved) return resolved + const refusals = await sourceRefusals(input, submission, resolved) + const gated = await gateSourceGraph(input, resolved) + return finalizePlan(input, submission, resolved, refusals, gated) +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/evaluate/types.ts b/apps/sim/lib/workflows/application/source-pipeline/evaluate/types.ts new file mode 100644 index 00000000000..1471f23dc15 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/evaluate/types.ts @@ -0,0 +1,145 @@ +import type { Principal } from '@sim/auth/principal' +import type { + BlockCatalog, + CanonicalWorkflow, + CanonicalWorkflowMeta, + LockIdentity, + ResourceKeyResolver, + ResourceKind, + SourceBase, + SourceHints, + SourceSpan, + WorkflowLockV1, + WorkflowSourceMetaV1, +} from '@sim/workflow-code' +import type { BlockState, WorkflowState } from '@sim/workflow-types/workflow' +import type { SourceDiffSummary } from '@/lib/workflows/application/source-diff' +import type { WorkflowSourceDiagnostic } from '@/lib/workflows/application/source-error' +import type { normalizeWorkflowVariables } from '@/lib/workflows/application/workflow-variables' +import type { StoredSource } from '@/lib/workflows/code/print-stored' +import type { StoredCanonical } from '@/lib/workflows/code/stored-canonical' +import type { ResourceKeyRef } from '@/lib/workspaces/resource-keys' + +/** Who and where a source write acts: what the evaluation gates and the commit re-checks run as. */ +export interface SourceScope { + workflowId: string + workspaceId: string + workspaceOrganizationId: string | null + /** The human the write is performed as, for gates, permission groups and lint. */ + subjectUserId: string + /** The folder the workflow sits in (`null` at the root), for a rename's name check. */ + folderId: string | null +} + +export interface SourceEvaluationInput { + principal: Principal + scope: SourceScope + /** The draft the source applies to; an empty graph for a workflow the apply creates. */ + stored: StoredSource + /** `stored` canonicalized with its resource keys. */ + storedCanonical: StoredCanonical + storedSourceMeta: WorkflowSourceMetaV1 | null + catalog: BlockCatalog + /** Env names the subject's runs resolve, for `env('…')` checks. */ + envNames: ReadonlySet + /** Attribution for resource keys a submitted reference allocates. */ + createdBy: string | null + files: Record + lock: WorkflowLockV1 + /** Where and when the source was read, its fingerprint already matched; absent for a create. */ + base?: SourceBase + /** Set when the apply creates the workflow: the caller's name and description overrides. */ + create?: { name?: string; description?: string | null } + /** + * Whether the apply may remove blocks the source no longer declares (`--allow-delete`). Without + * it, removing any block refuses the apply with `E_APPLY_BLOCK_DELETE`, as a workspace apply + * refuses deleting a resource. + */ + allowDelete?: boolean + /** + * Resources a workspace source apply creates or adopts before this workflow, by key, each with + * a stand-in id: consulted after the stored keys, so a dry run of a workflow that names a table + * created in the same apply resolves instead of failing `E_MAT_UNKNOWN_RESOURCE`. + */ + pendingResources?: readonly PendingResource[] +} + +/** + * A resource the source names by key; `member` is the part of it a reference names, when it names + * one: an MCP server's tool, for `mcp(key).tool(name)`. + */ +export interface SourceReference extends ResourceKeyRef { + member?: string + /** Where the entry file writes the reference, for an MCP tool: the agent's `tools`. */ + span?: SourceSpan +} + +/** A resource an enclosing apply creates or adopts, by key, with the id that stands in for it. */ +export interface PendingResource { + kind: ResourceKind + key: string + id: string + /** `id` stands in for a resource the apply has yet to create, not one it stores already. */ + placeholder?: boolean + /** The name the apply writes it under, which a workflow's stored reference repeats (a skill). */ + name?: string + /** + * A table's declared column names, which the lint checks a row write against instead of the + * stored schema the apply has yet to change. + */ + columns?: readonly string[] +} + +/** Everything a commit (or a dry run) writes and reports, once every check passed. */ +export interface SourcePlan { + /** + * The workflow's name and description after the write: the source's (a stored workflow's + * differ from what is stored where `metaChanged` says), or a created workflow's. + */ + meta: CanonicalWorkflowMeta + /** The name and description fields the write changes on a stored workflow. */ + metaChanged: Array + /** The graph is unchanged; only source meta may differ. Never true for a create. */ + noop: boolean + blocks: Record + edges: WorkflowState['edges'] + /** The variables to write, or `undefined` when the source leaves them unchanged. */ + variables: ReturnType | undefined + /** Variables as the next read parses them, for re-printing what is stored. */ + storedVariables: WorkflowState['variables'] + /** Warnings (on a dry run, every finding), pinned to the submitted files. */ + diagnostics: WorkflowSourceDiagnostic[] + diff: SourceDiffSummary + /** Source key → id for every new block whose id was minted rather than taken from the lock. */ + mintedIds: Record + /** Every key the write binds, with its id: what the next print is keyed by. */ + identity: LockIdentity + /** The post-materialize lock the compiled idents and comments are re-keyed onto. */ + lock: WorkflowLockV1 + /** The compiled file's idents, order and comments. */ + compiled: { hints: SourceHints; workflow: Pick } + /** source meta after the write, merged onto the stored one. */ + sourceMeta: WorkflowSourceMetaV1 + /** How the response prints function code: as the submission did. */ + functionCode: 'inline' | 'files' + /** Answers every resource reference of both the stored and the submitted source. */ + resources: ResourceKeyResolver + /** Every resource the submitted source names by key, as the compiler read it. */ + references: SourceReference[] + /** Every env name the submitted source reads, as the compiler read it. */ + envReferences: string[] +} + +/** What the submitted source references, as the compiler read it. */ +export type CompiledReferences = Pick + +/** + * A refusal, with what the source references when it compiled (so an enclosing apply can tell a + * workflow refused for a reference it holds back), or the plan. + */ +export type SourceEvaluation = + | { + refused: WorkflowSourceDiagnostic[] + compiledReferences?: CompiledReferences + } + | { plan: SourcePlan } diff --git a/apps/sim/lib/workflows/application/source-pipeline/index.ts b/apps/sim/lib/workflows/application/source-pipeline/index.ts new file mode 100644 index 00000000000..9b99638ef6e --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/index.ts @@ -0,0 +1,21 @@ +export { commitSource } from './commit' +export { + afterSourceApply, + afterSourceCreate, + type SourceProvenance, + sourceAnalyticsFacts, + sourceApplyAudit, + sourceCreateAudit, +} from './effects' +export type { PendingResource, SourcePlan } from './evaluate' +export { EMPTY_LOCK, type SourceLoads, type SourceTarget } from './load' +export { previewSource, refusedOutcome, type SourceOutcome } from './outcome' +export { + PlanBindingError, + type PlannedSource, + planSource, + refusalError, + type SourcePlanSummary, + summarizePlan, +} from './plan' +export { workflowSourceCreate, workflowSourceUpdate } from './workspace-writes' diff --git a/apps/sim/lib/workflows/application/source-pipeline/load.ts b/apps/sim/lib/workflows/application/source-pipeline/load.ts new file mode 100644 index 00000000000..7fca7287b2b --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/load.ts @@ -0,0 +1,182 @@ +import { + type Principal, + resolvePrincipalAttribution, + resolvePrincipalSubjectUserId, +} from '@sim/auth/principal' +import { db } from '@sim/db' +import type { + BlockCatalog, + SourceBase, + SourceFingerprint, + WorkflowLockV1, + WorkflowSourceMetaV1, +} from '@sim/workflow-code' +import { readWorkflowSourceMeta } from '@sim/workflow-persistence/source-meta' +import { + type CatalogGate, + resolveCatalogGate, + withCatalogBlockScope, +} from '@/lib/catalog/application/context' +import type { ActiveWorkflowApplicationContext } from '@/lib/workflows/application/context' +import { loadStoredSource } from '@/lib/workflows/application/read-workflow-source' +import type { SourceScope } from '@/lib/workflows/application/source-pipeline/evaluate' +import { buildSourceCatalog, storedCatalogTypes } from '@/lib/workflows/code/catalog/workspace' +import { type StoredSource, storedSourceFingerprint } from '@/lib/workflows/code/print-stored' +import { loadEnvNames } from '@/lib/workflows/code/resources' +import { + canonicalizeStored, + prefetchStoredResources, + type StoredCanonical, +} from '@/lib/workflows/code/stored-canonical' +import type { ActiveWorkspaceApplicationContext } from '@/lib/workspaces/application/context' + +/** The stored draft a create applies onto: an empty graph. */ +const EMPTY_STORED_SOURCE: StoredSource = { + state: { blocks: {}, edges: [], loops: {}, parallels: {}, variables: {} }, + meta: { name: '', description: null }, +} + +/** The empty lock a created workflow is planned with: every id is minted. */ +export const EMPTY_LOCK: WorkflowLockV1 = { version: 1, blocks: {}, variables: {} } + +/** Where a source write lands: an existing workflow's draft, or a new workflow `workflowId`. */ +export type CommitTarget = + | { kind: 'existing'; context: ActiveWorkflowApplicationContext } + | { + kind: 'new' + workspace: ActiveWorkspaceApplicationContext + workflowId: string + /** The folder the workflow is created in; `null` at the workspace root. */ + folderId: string | null + } + +/** + * What a plan evaluates against: the stored draft of an existing workflow, read at `base`, or a + * new workflow, with an empty graph standing in for the stored one. + */ +export type SourceTarget = + | (Extract & { + /** + * Where and when the source was read. A `base.fingerprint` that no longer matches the draft + * refuses as stale; a lock read in another workspace is treated as stale. + */ + base: SourceBase + /** Allow removing blocks and variables the source no longer declares. */ + allowDelete: boolean + }) + | (Extract & { + /** The caller's name and description, over the source's. */ + create: { name?: string; description?: string | null } + }) + +/** + * Loads that are the same for every workflow one request evaluates in one workspace for one + * principal (a workspace source apply): each starts on first use and is reused after. Never share + * one across principals or workspaces. + */ +export interface SourceLoads { + catalogGate?: Promise + envNames?: Promise +} + +/** Everything a plan reads before it evaluates: the stored draft, its catalog and fingerprint. */ +export interface LoadedTarget { + gate: CatalogGate + catalog: BlockCatalog + stored: StoredSource + storedSourceMeta: WorkflowSourceMetaV1 | null + storedCanonical: StoredCanonical + /** The stored draft's fingerprint when loaded; a commit refuses a draft that moved off it. */ + storedFingerprint: SourceFingerprint + envNames: string[] + /** Attribution for resource keys the write allocates. */ + createdBy: string | null + scope: SourceScope +} + +/** The workflow a target writes and the workspace it is in. */ +export function targetIds(target: CommitTarget): { workflowId: string; workspaceId: string } { + return target.kind === 'existing' + ? { workflowId: target.context.workflowId, workspaceId: target.context.workspaceId } + : { workflowId: target.workflowId, workspaceId: target.workspace.workspaceId } +} + +/** + * Who and where a write to `target` acts: the human it is attributed to (the workspace's billing + * owner for a workspace key) and the workflow's folder. + */ +export function sourceScope(principal: Principal, target: CommitTarget): SourceScope { + const context = + target.kind === 'existing' + ? target.context + : { ...target.workspace, workflowId: target.workflowId } + return { + workflowId: context.workflowId, + workspaceId: context.workspaceId, + workspaceOrganizationId: context.workspaceOrganizationId, + subjectUserId: resolvePrincipalAttribution(principal, { + workspaceBillingOwnerUserId: context.billedAccountUserId, + }).attributedUserId, + folderId: target.kind === 'existing' ? target.context.workflow.folderId : target.folderId, + } +} + +/** + * Loads what a plan of `target` evaluates against: the catalog gate, the stored draft and its + * source meta (an empty graph for a new workflow), its resource keys (allocating missing ones + * only when the plan will be committed) and the env names its runs resolve. The stored draft is + * canonicalized and fingerprinted inside the gate's block scope. `loads` shares the catalog gate + * and env names across the workflows of one request. + */ +export async function loadSourceTarget( + principal: Principal, + target: SourceTarget, + options: { loads?: SourceLoads; allocate: boolean } +): Promise { + const scope = sourceScope(principal, target) + const createdBy = resolvePrincipalSubjectUserId(principal) ?? null + const loads = options.loads ?? {} + const context = + target.kind === 'existing' + ? target.context + : { ...target.workspace, workflowId: target.workflowId } + const [gate, stored, storedSourceMeta, prefetched, envNames] = await Promise.all([ + (loads.catalogGate ??= resolveCatalogGate(principal, context, { customBlockInputs: true })), + target.kind === 'existing' ? loadStoredSource(target.context) : EMPTY_STORED_SOURCE, + target.kind === 'existing' ? readWorkflowSourceMeta(db, scope.workflowId) : null, + prefetchStoredResources({ + workflowId: scope.workflowId, + workspaceId: scope.workspaceId, + createdBy, + allocate: options.allocate, + }), + (loads.envNames ??= loadEnvNames(scope.workspaceId, scope.subjectUserId)), + ]) + return withCatalogBlockScope(gate, async () => { + const catalog = buildSourceCatalog({ + gate, + organizationId: scope.workspaceOrganizationId, + storedTypes: storedCatalogTypes(Object.values(stored.state.blocks)), + }) + const storedCanonical = await canonicalizeStored({ + prefetched, + state: stored.state, + meta: stored.meta, + catalog, + }) + return { + gate, + catalog, + stored, + storedSourceMeta, + storedCanonical, + storedFingerprint: storedSourceFingerprint( + storedCanonical.canonical, + storedCanonical.resources + ), + envNames, + createdBy, + scope, + } + }) +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/outcome.ts b/apps/sim/lib/workflows/application/source-pipeline/outcome.ts new file mode 100644 index 00000000000..c0e3a7879ec --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/outcome.ts @@ -0,0 +1,207 @@ +import type { Principal } from '@sim/auth/principal' +import { createLogger } from '@sim/logger' +import type { + BlockCatalog, + CanonicalizeResult, + SourceBase, + WorkflowLockV1, + WorkflowSourceMetaV1, +} from '@sim/workflow-code' +import type { WorkflowState } from '@sim/workflow-types/workflow' +import { withCatalogBlockScope } from '@/lib/catalog/application/context' +import { previewSourceCommit } from '@/lib/workflows/application/source-commit' +import { emptyDiffSummary, type SourceDiffSummary } from '@/lib/workflows/application/source-diff' +import type { WorkflowSourceDiagnostic } from '@/lib/workflows/application/source-error' +import { type CommitTarget, targetIds } from '@/lib/workflows/application/source-pipeline/load' +import type { PlannedSource, RefusedSource } from '@/lib/workflows/application/source-pipeline/plan' +import { type PrintedSource, printStoredSource } from '@/lib/workflows/code/print-stored' +import { checkDraftNeedsRedeployment } from '@/lib/workflows/deployment-status' + +const logger = createLogger('ApplyWorkflowSource') + +/** The outcome of a source write, its dry run or its plan, as the use cases return it. */ +export interface SourceOutcome { + workflowId: string + workflowName: string + workspaceId: string + dryRun: boolean + /** True when the apply wrote a graph, or renamed or redescribed the workflow. */ + applied: boolean + /** True when source meta was stored: with a graph write, or alone for a graph no-op. */ + wroteSourceMeta: boolean + diff: SourceDiffSummary + /** Warnings on success; on a dry run, every finding including the errors a commit would refuse. */ + diagnostics: WorkflowSourceDiagnostic[] + /** Re-printed from what is (or would be) stored, so normalized values show in the source. */ + files: Record + lock: WorkflowLockV1 + /** Where what is (or would be) stored was read: the next apply's `base`. */ + base: SourceBase + /** Source key → id for every new block whose id was minted rather than taken from the lock. */ + mintedIds: Record + notes: string[] + needsRedeployment: boolean + sdkMajor: number +} + +/** Whose outcome is shaped: the workflow, whether it is a dry run, and the draft it was read on. */ +interface OutcomeFrame { + target: CommitTarget + dryRun: boolean + catalog: BlockCatalog + /** The stored workflow's name, which the outcome reports unless the write gives another. */ + storedName: string +} + +/** What an outcome reports beyond its frame and the printed source; the rest default to empty. */ +type OutcomeFields = Pick & + Partial> + +/** The one outcome shape: what is (or would be) stored, printed, and what the write did. */ +export function shapeOutcome( + frame: OutcomeFrame, + printed: Pick & Pick, + outcome: OutcomeFields +): SourceOutcome { + const { workflowId, workspaceId } = targetIds(frame.target) + return { + workflowId, + workflowName: frame.storedName, + workspaceId, + dryRun: frame.dryRun, + mintedIds: {}, + notes: [], + wroteSourceMeta: false, + ...outcome, + files: printed.files, + lock: printed.lock, + base: { + version: 1, + workflowId, + workspaceId, + fingerprint: printed.fingerprint, + catalogHash: frame.catalog.hash, + }, + sdkMajor: frame.catalog.sdkMajor, + } +} + +/** + * Prints `state` as the plan's write would leave it, under the keys and function files the + * submission bound, as the next `source get` will. `lockless` is the state already canonicalized. + */ +export function printPlanned( + { plan, loaded }: Pick, + state: WorkflowState, + sourceMeta: WorkflowSourceMetaV1 | null, + lockless?: CanonicalizeResult +): PrintedSource { + /** The function files the submission named, kept so a write never renames the author's. */ + const codeFiles: Record = {} + for (const [key, hint] of Object.entries(plan.compiled.hints.blocks)) { + if (hint.codeFile !== undefined) codeFiles[key] = hint.codeFile + } + return printStoredSource({ + state, + meta: plan.meta, + catalog: loaded.catalog, + resources: plan.resources, + sourceMeta, + functionCode: plan.functionCode, + identity: plan.identity, + codeFiles, + ...(lockless ? { lockless } : {}), + }) +} + +/** Prints the blocks and edges a write stores with the plan's variables and source meta. */ +export function printWritten( + planned: Pick, + written: Omit +): PrintedSource { + return printPlanned( + planned, + { ...written, variables: planned.plan.storedVariables }, + planned.plan.sourceMeta + ) +} + +/** + * The dry run of a plan: what a commit would store, re-printed, and whether the draft would then + * differ from its deployment. A graph no-op reports the source meta a commit would store, and + * `draftNeedsRedeployment`, the deployment check of the draft as it is, started beside the plan. + */ +export async function previewSource( + principal: Principal, + planned: PlannedSource, + draftNeedsRedeployment: Promise +): Promise { + const { plan, loaded, target } = planned + const frame = { + target, + dryRun: true, + catalog: loaded.catalog, + storedName: loaded.stored.meta.name, + } + const isNew = target.kind === 'new' + return withCatalogBlockScope(loaded.gate, async () => { + if (planned.noop) { + return shapeOutcome(frame, planned.noop.printed, { + applied: false, + workflowName: plan.meta.name, + diff: plan.diff, + diagnostics: plan.diagnostics, + needsRedeployment: await draftNeedsRedeployment, + }) + } + const preview = await previewSourceCommit(planned.workflowId, plan) + logger.info('Evaluated workflow source without persisting', { + workflowId: planned.workflowId, + workspaceId: loaded.scope.workspaceId, + principalKind: principal.kind, + }) + return shapeOutcome(frame, printWritten(planned, preview.state), { + applied: false, + needsRedeployment: + !isNew && + (await checkDraftNeedsRedeployment(planned.workflowId, { + ...preview.state, + variables: plan.storedVariables, + })), + diff: plan.diff, + diagnostics: plan.diagnostics, + mintedIds: plan.mintedIds, + notes: preview.warnings, + workflowName: plan.meta.name, + }) + }) +} + +/** + * The dry-run answer for a refused plan: its findings with a 200, echoing the submission. It echoes + * the caller's own base fingerprint, so applying its response after a stale or mismatched read + * still fails the freshness check rather than overwriting the newer draft, and the deployment check + * of the draft as it is (`draftNeedsRedeployment`). + */ +export async function refusedOutcome( + refused: RefusedSource, + target: CommitTarget, + submission: { files: Record; lock: WorkflowLockV1 }, + draftNeedsRedeployment: Promise +): Promise { + return shapeOutcome( + { + target, + dryRun: true, + catalog: refused.echo.catalog, + storedName: refused.echo.stored.meta.name, + }, + { files: submission.files, lock: submission.lock, fingerprint: refused.echo.fingerprint }, + { + applied: false, + diff: emptyDiffSummary(), + diagnostics: refused.diagnostics, + needsRedeployment: await draftNeedsRedeployment, + } + ) +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/plan.ts b/apps/sim/lib/workflows/application/source-pipeline/plan.ts new file mode 100644 index 00000000000..25ab5d03316 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/plan.ts @@ -0,0 +1,338 @@ +import type { Principal } from '@sim/auth/principal' +import type { + BlockCatalog, + ResourceKind, + SourceBase, + SourceFingerprint, + WorkflowLockV1, +} from '@sim/workflow-code' +import { diagnostic } from '@sim/workflow-code/diagnostics' +import { isEqual } from 'es-toolkit' +import { withCatalogBlockScope } from '@/lib/catalog/application/context' +import { assertCreateNameFree } from '@/lib/workflows/application/source-commit' +import { emptyDiffSummary, type SourceDiffSummary } from '@/lib/workflows/application/source-diff' +import { + type WorkflowSourceDiagnostic, + WorkflowSourceRejectedError, + WorkflowSourceStaleError, + workflowSourceStaleMessage, +} from '@/lib/workflows/application/source-error' +import { + type CompiledReferences, + evaluateSource, + type PendingResource, + type SourcePlan, + type SourceReference, +} from '@/lib/workflows/application/source-pipeline/evaluate' +import { + type LoadedTarget, + loadSourceTarget, + type SourceLoads, + type SourceTarget, +} from '@/lib/workflows/application/source-pipeline/load' +import { printPlanned } from '@/lib/workflows/application/source-pipeline/outcome' +import type { PrintedSource, StoredSource } from '@/lib/workflows/code/print-stored' + +/** What a source submission carries to a plan. */ +interface SourceSubmission { + /** Repo-relative path → text: `workflow.ts` plus any function-code files it references. */ + files: Record + /** The identity lock returned by `source get` (or a previous apply); empty for a create. */ + lock: WorkflowLockV1 + /** Resources the enclosing workspace source apply creates or adopts first. */ + pendingResources?: readonly PendingResource[] +} + +/** + * The identity a plan is bound to: a commit refuses a plan made for another principal. The key is + * the principal as it authenticated, so two sessions or keys of one user are two principals. + */ +function principalKey(principal: Principal): string { + return JSON.stringify(principal) +} + +/** Why a commit refuses a plan: made for another workflow or write, or by another principal. */ +type PlanBindingCode = 'PLAN_WORKFLOW_MISMATCH' | 'PLAN_PRINCIPAL_MISMATCH' + +/** + * A commit was handed a plan it may not write: a bug in the caller, not something its user can + * fix, so it stays unclassified (a 500). `code` says which binding the plan broke. + */ +export class PlanBindingError extends Error { + constructor( + readonly code: PlanBindingCode, + message: string + ) { + super(message) + this.name = 'PlanBindingError' + } +} + +/** Throws {@link PlanBindingError} unless `planned` was made by `principal`. */ +export function assertPlannedBy(principal: Principal, planned: PlannedSource): void { + if (planned.boundTo.principalKey !== principalKey(principal)) { + throw new PlanBindingError( + 'PLAN_PRINCIPAL_MISMATCH', + 'A planned source commits only as the principal it was planned by' + ) + } +} + +/** + * A plan that passed every check, bound to the workflow and principal it was made for. Outside the + * pipeline, read only `kind` and `workflowId`, and the rest through {@link summarizePlan}; the + * other fields are what the pipeline's commit and preview read. + */ +export interface PlannedSource { + kind: 'planned' + /** The workflow the plan writes: the stored one, or the id a create takes. */ + workflowId: string + boundTo: { principalKey: string } + /** The target the plan was evaluated against. */ + target: SourceTarget + loaded: LoadedTarget + plan: SourcePlan + /** + * For a graph no-op: the stored graph printed under the source meta the write leaves it with, + * and whether that differs from what is stored (new idents, order or comments). + */ + noop?: { printed: PrintedSource; sourceMetaChanged: boolean } + /** + * Set when the plan resolved a stand-in id of a resource the enclosing apply has yet to create: + * such a plan cannot be committed as it is, so the commit plans `submission` again (with `loads`) + * once that resource is written under its real id. + */ + replan?: { submission: SourceSubmission; loads?: SourceLoads } +} + +/** What a caller outside the pipeline reads of a plan, refused or not. */ +export interface SourcePlanSummary { + diagnostics: readonly WorkflowSourceDiagnostic[] + /** Empty for a refusal, which plans no change. */ + diff: SourceDiffSummary + /** What the source references, as the compiler read it; empty when it did not compile. */ + references: readonly SourceReference[] + envReferences: readonly string[] + /** Whether a graph no-op still changes source meta (idents, order or comments). */ + sourceMetaChanged: boolean + /** The workflow's name after the write; absent for a refusal. */ + workflowName?: string + /** Each block of the planned graph, by id; absent for a refusal. */ + blocks?: ReadonlyArray<{ id: string; name: string }> + /** The id the plan resolved for a resource the source names by key, or null. */ + resolveId(kind: ResourceKind, key: string): string | null +} + +/** The {@link SourcePlanSummary} of a plan or a refusal. */ +export function summarizePlan(outcome: PlannedSource | RefusedSource): SourcePlanSummary { + if (outcome.kind === 'refused') { + return { + diagnostics: outcome.diagnostics, + diff: emptyDiffSummary(), + references: outcome.compiledReferences?.references ?? [], + envReferences: outcome.compiledReferences?.envReferences ?? [], + sourceMetaChanged: false, + resolveId: () => null, + } + } + const { plan } = outcome + return { + diagnostics: plan.diagnostics, + diff: plan.diff, + references: plan.references, + envReferences: plan.envReferences, + sourceMetaChanged: outcome.noop?.sourceMetaChanged ?? false, + workflowName: plan.meta.name, + blocks: Object.values(plan.blocks).map(({ id, name }) => ({ id, name })), + resolveId: (kind, key) => plan.resources.toId(kind, key), + } +} + +/** + * Whether `references` name a resource of `pending` by a stand-in id. A workflow the enclosing + * apply creates is no stand-in: its id is minted at plan time and is the one its create stores. + */ +function resolvesPlaceholder( + references: readonly SourceReference[], + pending: readonly PendingResource[] +): boolean { + return references.some((ref) => + pending.some((entry) => entry.placeholder && entry.kind === ref.kind && entry.key === ref.key) + ) +} + +/** + * The plan a commit writes, once `planned` is shown to be the committing principal's (a foreign + * plan throws {@link PlanBindingError} before any replan): `planned` itself, or (when it resolved a + * stand-in id) the submission planned again onto `target` with every resource but the workflows + * the apply creates read as stored, since the apply writes those first; a refusal of that plan + * throws. Pending falls back only where stored resolution fails, so dropping the stand-ins is what + * refuses a resource the apply never created instead of writing its stand-in id. The workflows + * still resolve from pending, as a create of one may not have run yet. + */ +export async function planForCommit( + principal: Principal, + planned: PlannedSource, + target: SourceTarget +): Promise { + assertPlannedBy(principal, planned) + if (!planned.replan) return planned + const { submission, loads } = planned.replan + const replanned = await planSource( + principal, + target, + { + ...submission, + pendingResources: (submission.pendingResources ?? []).filter( + (entry) => entry.kind === 'workflow' + ), + }, + { ...(loads ? { loads } : {}), allocate: true } + ) + if (replanned.kind === 'refused') throw refusalError(replanned) + return replanned +} + +/** A plan refused: the findings, and what its dry-run answer echoes. */ +export interface RefusedSource { + kind: 'refused' + diagnostics: WorkflowSourceDiagnostic[] + /** Set when the draft moved off the submission's base: a commit answers 409, not 400. */ + stale?: { currentFingerprint: SourceFingerprint; workflowName: string } + /** What the source references, when it compiled. */ + compiledReferences?: CompiledReferences + /** The catalog and draft the plan read, and the base fingerprint a dry run echoes. */ + echo: { catalog: BlockCatalog; stored: StoredSource; fingerprint: SourceBase['fingerprint'] } +} + +/** The error a write throws for a refused plan: 409 when stale, else 400 (413 over a limit). */ +export function refusalError( + refused: RefusedSource +): WorkflowSourceStaleError | WorkflowSourceRejectedError { + return refused.stale + ? new WorkflowSourceStaleError(refused.stale.currentFingerprint, refused.stale.workflowName) + : new WorkflowSourceRejectedError(refused.diagnostics) +} + +/** + * Plans a source write without writing: loads the target, refuses a base read from another + * workflow (`E_APPLY_WORKFLOW_MISMATCH`) or a draft that moved since it (`E_APPLY_STALE`), then + * evaluates the submission onto the stored graph (empty for a new workflow). A new workflow's name + * is checked free in its folder now, so a taken name refuses the plan, not the write (it throws + * {@link WorkflowSourceNameTakenError}). A graph no-op is printed once here, to tell whether its + * source meta changes. `allocate` lets the load allocate missing resource keys: set it only for a + * plan that will be committed. + */ +export async function planSource( + principal: Principal, + target: SourceTarget, + submission: SourceSubmission, + options: { loads?: SourceLoads; allocate: boolean } +): Promise { + const loaded = await loadSourceTarget(principal, target, options) + const { catalog, stored, storedFingerprint, scope } = loaded + return withCatalogBlockScope(loaded.gate, async () => { + const refuse = ( + diagnostics: WorkflowSourceDiagnostic[], + extra: Pick = {} + ): RefusedSource => ({ + kind: 'refused', + diagnostics, + ...extra, + echo: { + catalog, + stored, + fingerprint: target.kind === 'existing' ? target.base.fingerprint : storedFingerprint, + }, + }) + + if (target.kind === 'existing' && target.base.workflowId !== scope.workflowId) { + return refuse([ + diagnostic( + 'E_APPLY_WORKFLOW_MISMATCH', + `This source was read from workflow ${target.base.workflowId}, not ${scope.workflowId}; applying it here would overwrite a different workflow.`, + { + hint: 'Apply it to the workflow it was read from, or get this workflow’s source into its own directory.', + path: '/base/workflowId', + } + ), + ]) + } + if (target.kind === 'existing' && storedFingerprint !== target.base.fingerprint) { + return refuse([diagnostic('E_APPLY_STALE', workflowSourceStaleMessage(stored.meta.name))], { + stale: { currentFingerprint: storedFingerprint, workflowName: stored.meta.name }, + }) + } + + const evaluation = await evaluateSource({ + principal, + scope, + stored, + storedCanonical: loaded.storedCanonical, + storedSourceMeta: loaded.storedSourceMeta, + catalog, + envNames: new Set(loaded.envNames), + createdBy: loaded.createdBy, + files: submission.files, + lock: submission.lock, + ...(submission.pendingResources ? { pendingResources: submission.pendingResources } : {}), + ...(target.kind === 'existing' + ? { base: target.base, allowDelete: target.allowDelete } + : { create: target.create }), + }) + if ('refused' in evaluation) { + return refuse( + evaluation.refused, + evaluation.compiledReferences ? { compiledReferences: evaluation.compiledReferences } : {} + ) + } + const { plan } = evaluation + const planned: PlannedSource = { + kind: 'planned', + workflowId: scope.workflowId, + boundTo: { principalKey: principalKey(principal) }, + target, + loaded, + plan, + } + if ( + submission.pendingResources && + resolvesPlaceholder(plan.references, submission.pendingResources) + ) { + planned.replan = { submission, ...(options.loads ? { loads: options.loads } : {}) } + } + if (target.kind === 'new') { + await assertCreateNameFree(plan, { + workspaceId: scope.workspaceId, + folderId: target.folderId, + }) + } + if (plan.noop) planned.noop = printNoop(planned) + return planned + }) +} + +/** + * The stored graph of a graph no-op printed under the name and description the write leaves it + * with and the source meta it stores, and whether that source meta prints differently from the + * stored one: only changed source meta prints the "before". + */ +function printNoop(planned: PlannedSource): NonNullable { + const { plan, loaded } = planned + const { storedCanonical, stored, storedSourceMeta } = loaded + const lockless = + plan.metaChanged.length > 0 + ? { + ...storedCanonical.canonical, + workflow: { ...storedCanonical.canonical.workflow, meta: plan.meta }, + } + : storedCanonical.canonical + const printed = printPlanned(planned, stored.state, plan.sourceMeta, lockless) + const before = isEqual(plan.sourceMeta, storedSourceMeta) + ? printed + : printPlanned(planned, stored.state, storedSourceMeta, lockless) + return { + printed, + sourceMetaChanged: !isEqual(printed.files, before.files) || !isEqual(printed.lock, before.lock), + } +} diff --git a/apps/sim/lib/workflows/application/source-pipeline/workspace-writes.ts b/apps/sim/lib/workflows/application/source-pipeline/workspace-writes.ts new file mode 100644 index 00000000000..ed312b3e423 --- /dev/null +++ b/apps/sim/lib/workflows/application/source-pipeline/workspace-writes.ts @@ -0,0 +1,202 @@ +import type { Principal } from '@sim/auth/principal' +import type { SourceBase, WorkflowLockV1 } from '@sim/workflow-code' +import type { DbTransaction } from '@/lib/db/types' +import { defineAuthorizedWorkflowUseCase } from '@/lib/workflows/application/authorized-workflow-use-case' +import type { ActiveWorkflowApplicationContext } from '@/lib/workflows/application/context' +import { resolveWorkflowFolderPath } from '@/lib/workflows/application/folders' +import { + requireMutableFolder, + requireMutableWorkflow, +} from '@/lib/workflows/application/mutability' +import { workflowOperations } from '@/lib/workflows/application/operations' +import { resolvePrincipalWorkflowContext } from '@/lib/workflows/application/principal-scope' +import { recordSourceRefused } from '@/lib/workflows/application/source-analytics' +import { commitSource } from '@/lib/workflows/application/source-pipeline/commit' +import { + afterSourceApply, + afterSourceCreate, + type SourceProvenance, + sourceAnalyticsFacts, + sourceApplyAudit, + sourceCreateAudit, +} from '@/lib/workflows/application/source-pipeline/effects' +import type { PendingResource } from '@/lib/workflows/application/source-pipeline/evaluate' +import { + EMPTY_LOCK, + type SourceLoads, + type SourceTarget, +} from '@/lib/workflows/application/source-pipeline/load' +import type { SourceOutcome } from '@/lib/workflows/application/source-pipeline/outcome' +import { + type PlannedSource, + planForCommit, + planSource, + type RefusedSource, +} from '@/lib/workflows/application/source-pipeline/plan' +import { + type ActiveWorkspaceApplicationContext, + resolveActiveWorkspaceApplicationContext, +} from '@/lib/workspaces/application/context' + +/** An update of a stored workflow's draft, as a workspace source apply plans it. */ +interface UpdatePlanInput { + workflowId: string + files: Record + lock: WorkflowLockV1 + base: SourceBase + allowDelete: boolean + /** The resources the workspace apply creates or adopts first, which the evaluation resolves. */ + pendingResources: readonly PendingResource[] + /** Loads shared by every workflow the workspace apply evaluates. */ + loads: SourceLoads +} + +/** A workflow a workspace source apply creates, under the id it minted for it. */ +interface CreatePlanInput { + workspaceId: string + workflowId: string + files: Record + pendingResources: readonly PendingResource[] + loads: SourceLoads +} + +/** The update target of a stored workflow's plan, under the workflow context resolved now. */ +function updateTarget( + context: ActiveWorkflowApplicationContext, + input: Pick +): SourceTarget { + return { kind: 'existing', context, base: input.base, allowDelete: input.allowDelete } +} + +/** The create target of a new workflow's plan, at the workspace root. */ +async function createTarget( + workspace: ActiveWorkspaceApplicationContext, + workflowId: string +): Promise> { + const { folderId } = await resolveWorkflowFolderPath(workspace.workspaceId, '/') + await requireMutableFolder(folderId) + return { kind: 'new', workspace, workflowId, create: {}, folderId } +} + +/** + * A workspace source apply's update of one stored workflow, in two steps under the apply + * operation's authorization, feature flag and mutability checks: `plan` evaluates the folder + * without writing and records nothing; `commit` writes that plan (or, when the plan resolved a + * stand-in id of a resource the apply has since created, plans it again first), re-checking the + * fingerprint under the workflow row lock, and audits, records and notifies exactly as + * `applyWorkflowSource` does. + */ +export const workflowSourceUpdate = { + plan: defineAuthorizedWorkflowUseCase({ + operation: workflowOperations.applySource, + resolveContext: resolvePrincipalWorkflowContext, + async execute({ principal, input, context }): Promise { + await requireMutableWorkflow(context.workflowId) + return planSource( + principal, + updateTarget(context, input), + { files: input.files, lock: input.lock, pendingResources: input.pendingResources }, + { loads: input.loads, allocate: false } + ) + }, + }), + commit: defineAuthorizedWorkflowUseCase({ + operation: workflowOperations.applySource, + resolveContext: ({ principal, input }: { principal: Principal; input: CommitInput }) => + resolvePrincipalWorkflowContext({ + principal, + input: { workflowId: input.planned.workflowId }, + }), + async execute({ principal, input, context }): Promise { + const facts = sourceAnalyticsFacts( + principal, + input.provenance, + 'apply', + context.workspaceId, + false + ) + try { + await requireMutableWorkflow(context.workflowId) + const { planned } = input + if (planned.target.kind !== 'existing') { + throw new Error('An update commits a plan made for a stored workflow') + } + const target: SourceTarget = { ...planned.target, context } + return await commitSource(principal, await planForCommit(principal, planned, target), { + kind: 'existing', + context, + }) + } catch (error) { + recordSourceRefused({ ...facts, workflowId: context.workflowId }, error) + throw error + } + }, + projectAudit: ({ principal, input, context, result }) => + sourceApplyAudit(principal, input.provenance, context.workflowId, result), + afterSuccess: ({ principal, input, context, result }) => + afterSourceApply(principal, input.provenance, context, result), + }), +} + +/** What a workspace source apply commits: a plan its `plan` step made. */ +interface CommitInput { + planned: PlannedSource + provenance?: SourceProvenance +} + +/** What a workspace source apply commits for a workflow it creates. */ +interface CreateCommitInput extends CommitInput { + /** Runs inside the create's transaction, so what it writes commits with the workflow. */ + withinCreate: (tx: DbTransaction, workflowId: string) => Promise +} + +/** + * A workspace source apply's create of one workflow under the id it minted, in two steps under + * the create operation's authorization, feature flag and folder mutability checks: `plan` + * evaluates the folder and checks its name is free, recording nothing; `commit` writes the plan + * (planning again first when it must, as {@link workflowSourceUpdate} does) in one transaction with + * `withinCreate`, and audits, records and notifies exactly as `createWorkflowFromSource` does. + */ +export const workflowSourceCreate = { + plan: defineAuthorizedWorkflowUseCase({ + operation: workflowOperations.createFromSource, + resolveContext: ({ input }: { input: CreatePlanInput }) => + resolveActiveWorkspaceApplicationContext(input.workspaceId), + async execute({ principal, input, context }): Promise { + return planSource( + principal, + await createTarget(context, input.workflowId), + { files: input.files, lock: EMPTY_LOCK, pendingResources: input.pendingResources }, + { loads: input.loads, allocate: false } + ) + }, + }), + commit: defineAuthorizedWorkflowUseCase({ + operation: workflowOperations.createFromSource, + resolveContext: ({ input }: { input: CreateCommitInput }) => + resolveActiveWorkspaceApplicationContext(input.planned.loaded.scope.workspaceId), + async execute({ principal, input, context }): Promise { + const facts = sourceAnalyticsFacts( + principal, + input.provenance, + 'create', + context.workspaceId, + false + ) + try { + const target = await createTarget(context, input.planned.workflowId) + const planned = await planForCommit(principal, input.planned, target) + return await commitSource(principal, planned, target, { + withinCreate: input.withinCreate, + }) + } catch (error) { + recordSourceRefused({ ...facts, workflowId: '' }, error) + throw error + } + }, + projectAudit: ({ principal, input, result }) => + sourceCreateAudit(principal, input.provenance, result), + afterSuccess: ({ principal, input, result }) => + afterSourceCreate(principal, input.provenance, result), + }), +} diff --git a/apps/sim/lib/workspace-source/api/route-policies.ts b/apps/sim/lib/workspace-source/api/route-policies.ts index a30b6d5cc0b..df2067f817d 100644 --- a/apps/sim/lib/workspace-source/api/route-policies.ts +++ b/apps/sim/lib/workspace-source/api/route-policies.ts @@ -1,6 +1,6 @@ +import { boundAnswerFindings } from '@sim/workflow-code/diagnostics' import { utf8Bytes } from '@sim/workflow-code/limits' -import type { BlobEntry } from '@sim/workspace-code' -import { budgetOfPath } from '@sim/workspace-code' +import { type BlobEntry, budgetOfPath, WORKSPACE_ANSWER_BOUND } from '@sim/workspace-code' import { KIND_LAYOUTS } from '@sim/workspace-code/kinds/registry' import { workflowCodec, workflowFolder } from '@sim/workspace-code/kinds/workflow' import { kindOfPath } from '@sim/workspace-code/layout' @@ -8,13 +8,15 @@ import { WORKSPACE_SOURCE_LIMITS } from '@sim/workspace-code/limits' import { createV2ResourceConcealmentPolicy } from '@/lib/api/server/routes' import { renderSourceSlotBusy } from '@/lib/workflows/api/source-slots' import { WorkspaceSourceRejectedError } from '@/lib/workspace-source/application/errors' +import type { WorkspaceSourceDiagnostic } from '@/lib/workspace-source/application/types' import { WorkspaceSourceBusyError } from '@/lib/workspace-source/lease/store' import { SourceStagingFullError } from '@/lib/workspace-source/staging/store' import { v2CaughtOrchestrationError, v2ErrorForOrchestration } from '@/app/api/v2/lib/response' /** * Workspace source routes conceal an unreachable workspace as absent. A refused apply answers its - * status (`400`, `409` stale, `423` locked, `413` over a limit) with `details.diagnostics`; + * status (`400`, `409` stale, `423` locked, `413` over a limit) with `details.diagnostics`, + * bounded as every answer's are; * `details.code` is `SOURCE_STALE` when a resource changed since the directory was read, else * `SOURCE_REJECTED`. An apply while another one writes the workspace answers `409` with * `details.code` `SOURCE_BUSY`. Staging past the caller's staging caps answers `413` with @@ -28,7 +30,7 @@ export const v2WorkspaceSourceErrorPolicy = createV2ResourceConcealmentPolicy({ if (error instanceof WorkspaceSourceRejectedError) { return v2ErrorForOrchestration(error.code, error.message, { code: error.stale ? 'SOURCE_STALE' : 'SOURCE_REJECTED', - diagnostics: error.diagnostics, + diagnostics: boundAnswerFindings(error.diagnostics, WORKSPACE_ANSWER_BOUND), }) } if (error instanceof WorkspaceSourceBusyError) { @@ -41,6 +43,16 @@ export const v2WorkspaceSourceErrorPolicy = createV2ResourceConcealmentPolicy({ }, }) +/** + * `result` as a workspace source route answers it: its findings bounded once, here at the edge + * (`WORKSPACE_ANSWER_BOUND`), whichever kinds and stages they came from. + */ +export function withBoundWorkspaceFindings< + R extends { diagnostics: readonly WorkspaceSourceDiagnostic[] }, +>(result: R) { + return { ...result, diagnostics: boundAnswerFindings(result.diagnostics, WORKSPACE_ANSWER_BOUND) } +} + /** Non-workflow text bytes (declarations and table rows) one source-write unit covers. */ const TEXT_BYTES_PER_UNIT = 2_000_000 diff --git a/apps/sim/lib/workspace-source/application/apply.integration.ts b/apps/sim/lib/workspace-source/application/apply.integration.ts index 4369d887730..ce7f98db0e2 100644 --- a/apps/sim/lib/workspace-source/application/apply.integration.ts +++ b/apps/sim/lib/workspace-source/application/apply.integration.ts @@ -299,28 +299,50 @@ describe('workspace source apply', () => { repo: 'git@github.com:acme/flows.git', } - const applied = await apply(workspaceId, STARTER, { provenance }) + const created = await apply(workspaceId, STARTER, { provenance }) + const read = await get(workspaceId) + const updated = await apply( + workspaceId, + { + ...read.files, + 'workflows/ingest/workflow.ts': read.files['workflows/ingest/workflow.ts'].replace( + "'Save Lead'", + "'Store Lead'" + ), + }, + { base: read.base, provenance } + ) + expect(updated.applied.map(({ key, action }) => `${action} ${key}`)).toEqual([ + 'update ingest', + ]) - await vi.waitFor(async () => { + const recorded = async (applyId: string | undefined, workflowAction: string) => { const rows = await db .select({ action: auditLog.action, metadata: auditLog.metadata }) .from(auditLog) .where(eq(auditLog.workspaceId, workspaceId)) - const recorded = rows + return rows .filter( (row) => isRecordLike(row.metadata) && - row.metadata.correlationId === applied.applyId && - (row.action === 'workspace.updated' || row.action === 'workflow.created') + row.metadata.correlationId === applyId && + (row.action === 'workspace.updated' || row.action === workflowAction) ) .map((row) => row.metadata) - expect(recorded).toHaveLength(2) - for (const metadata of recorded) { - expect(metadata).toMatchObject({ - client: 'cli', - gitSha: '0123abcd', - repo: provenance.repo, - }) + } + await vi.waitFor(async () => { + for (const entries of [ + await recorded(created.applyId, 'workflow.created'), + await recorded(updated.applyId, 'workflow.updated'), + ]) { + expect(entries).toHaveLength(2) + for (const metadata of entries) { + expect(metadata).toMatchObject({ + client: 'cli', + gitSha: '0123abcd', + repo: provenance.repo, + }) + } } }) } diff --git a/apps/sim/lib/workspace-source/application/context.ts b/apps/sim/lib/workspace-source/application/context.ts index 1d55f4acca3..8ee9da69f1a 100644 --- a/apps/sim/lib/workspace-source/application/context.ts +++ b/apps/sim/lib/workspace-source/application/context.ts @@ -22,7 +22,7 @@ export function sourceContext( createdBy: resolvePrincipalSubjectUserId(principal) ?? null, hashBudget: newHashBudget(), admitted: new Map(), - shared: {}, + loads: {}, listings: {}, } } diff --git a/apps/sim/lib/workspace-source/application/cross-kind.integration.ts b/apps/sim/lib/workspace-source/application/cross-kind.integration.ts index 3dc509ad6d5..af5c89493ab 100644 --- a/apps/sim/lib/workspace-source/application/cross-kind.integration.ts +++ b/apps/sim/lib/workspace-source/application/cross-kind.integration.ts @@ -78,6 +78,7 @@ import { deleteWorkflow } from '@/lib/workflows/application/delete-workflow' import { duplicateWorkflow } from '@/lib/workflows/application/duplicate-workflow' import { loadWorkflowFromNormalizedTables } from '@/lib/workflows/persistence/utils' import { deleteWorkspaceFileOperation } from '@/lib/workspace-files/application/delete-workspace-file' +import { withBoundWorkspaceFindings } from '@/lib/workspace-source/api/route-policies' import { type ApplyWorkspaceSourceInput, type ApplyWorkspaceSourceResult, @@ -1168,8 +1169,9 @@ export default workflow('Big', {}, () => { }) ) - expect(planned.diagnostics).toHaveLength(1_001) - expect(planned.diagnostics.at(-1)).toMatchObject({ + const answered = withBoundWorkspaceFindings(planned).diagnostics + expect(answered).toHaveLength(1_001) + expect(answered.at(-1)).toMatchObject({ code: 'I_FINDINGS_TRUNCATED', severity: 'info', message: expect.stringContaining('500 more (500 of them errors)'), @@ -1205,7 +1207,7 @@ export default workflow('Big', {}, () => { }) check( - 'a plan over 1000 info findings still applies, and keeps every warning ahead of the infos', + 'a plan over 1000 info findings still applies, and its answer keeps every warning while it cuts infos', async () => { const workspaceId = await newWorkspace() const notes = Array.from( @@ -1238,8 +1240,9 @@ export default workflow('Secret', {}, () => { expect(codes(result.diagnostics)).toEqual([]) expect(result.failed).toBeUndefined() expect(result.applied).toHaveLength(6) - expect(codes(result.diagnostics, 'warning')).toContain('W_LITERAL_SECRET') - expect(codes(result.diagnostics, 'info')).toContain('I_FINDINGS_TRUNCATED') + const answered = withBoundWorkspaceFindings(result).diagnostics + expect(codes(answered, 'warning')).toContain('W_LITERAL_SECRET') + expect(codes(answered, 'info')).toContain('I_FINDINGS_TRUNCATED') } ) diff --git a/apps/sim/lib/workspace-source/application/findings.ts b/apps/sim/lib/workspace-source/application/findings.ts index de2a3a4aa27..8f93d232cb8 100644 --- a/apps/sim/lib/workspace-source/application/findings.ts +++ b/apps/sim/lib/workspace-source/application/findings.ts @@ -1,13 +1,14 @@ /** * The findings every kind reports the same way, so the plan reads alike across kinds. */ + +import { diagnostic } from '@sim/workflow-code/diagnostics' import { type BaseResource, type FlatDiagnostic, isFileKind, KIND_DIRS, resourceAddress, - severityOf, WORKSPACE_KIND_SPECS, type WorkspaceDiagnosticCode, type WorkspaceKind, @@ -33,7 +34,7 @@ export function finding( message: string, extras: Omit = {} ): WorkspaceSourceDiagnostic { - return { code, severity: severityOf(code), message, ...extras } + return diagnostic(code, message, extras) } /** How to recover from a refusal that needs a fresh read. */ diff --git a/apps/sim/lib/workspace-source/application/keys.ts b/apps/sim/lib/workspace-source/application/keys.ts index 6d486bb9670..7e74f941a7c 100644 --- a/apps/sim/lib/workspace-source/application/keys.ts +++ b/apps/sim/lib/workspace-source/application/keys.ts @@ -8,7 +8,7 @@ import type { ResourceKind } from '@sim/workflow-code' import { WORKSPACE_KIND_SPECS, type WorkspaceKind } from '@sim/workspace-code' import { WORKSPACE_SOURCE_LIMITS } from '@sim/workspace-code/limits' import type { DbTransaction } from '@/lib/db/types' -import type { PendingResource } from '@/lib/workflows/application/source-evaluate' +import type { PendingResource } from '@/lib/workflows/application/source-pipeline' import { reboundFinding } from '@/lib/workspace-source/application/findings' import type { PlannedChange, diff --git a/apps/sim/lib/workspace-source/application/kinds/workflow/adapter.ts b/apps/sim/lib/workspace-source/application/kinds/workflow/adapter.ts index 59984c8a16e..1d1f072379f 100644 --- a/apps/sim/lib/workspace-source/application/kinds/workflow/adapter.ts +++ b/apps/sim/lib/workspace-source/application/kinds/workflow/adapter.ts @@ -1,7 +1,7 @@ /** * Workflows as workspace source: each `workflows//` folder is a workflow source directory, - * applied through the workflow source pipeline (`applyWorkflowSource`, `createWorkflowFromSource`) - * and archived through `deleteWorkflow`, so compile, gates, locks, the per-workflow fingerprint + * planned and committed through the workflow source pipeline (`workflowSourceUpdate`, + * `workflowSourceCreate`) and archived through `deleteWorkflow`, so compile, gates, locks, the per-workflow fingerprint * (409) and audit are exactly a single-workflow apply's. Folder placement is not managed. * * The kind's rule differs from the three-way rule the other kinds share, on purpose: @@ -49,7 +49,6 @@ import { OrchestrationError } from '@/lib/core/orchestration/types' import { applyWorkflowSource, createWorkflowFromSource, - type EvaluatedSource, } from '@/lib/workflows/application/apply-workflow-source' import { deleteWorkflow } from '@/lib/workflows/application/delete-workflow' import { listWorkflows } from '@/lib/workflows/application/list-workflows' @@ -62,7 +61,14 @@ import { } from '@/lib/workflows/application/read-workflow-source' import type { SourceDiffSummary } from '@/lib/workflows/application/source-diff' import { WorkflowSourceRejectedError } from '@/lib/workflows/application/source-error' -import type { PendingResource, SourceReference } from '@/lib/workflows/application/source-evaluate' +import { + type PendingResource, + type PlannedSource, + type SourcePlanSummary, + summarizePlan, + workflowSourceCreate, + workflowSourceUpdate, +} from '@/lib/workflows/application/source-pipeline' import { countLiveWorkflows, LAST_WORKFLOW_MESSAGE, @@ -187,16 +193,8 @@ export interface WorkflowPlan { * a workflow that calls this one back) name it before it is written. */ workflowId?: string - /** - * The workflows this apply creates, by key with their minted ids: a write that evaluates again - * resolves those not written yet, as its plan did. - */ - created?: PendingResource[] - /** - * The plan's evaluation, which the write commits without evaluating again; absent when it - * resolved a resource this apply creates first, whose stand-in id the write must replace. - */ - evaluated?: EvaluatedSource + /** The workflow's plan, which the write commits. */ + planned?: PlannedSource } type WorkflowChange = PlannedChange<'workflow'> @@ -204,17 +202,13 @@ type WorkflowChange = PlannedChange<'workflow'> /** * The resources a compiled workflow names that a workspace directory declares: by the same key * for a kind whose source key is its resource key, once per member it names (an MCP server's - * tools); by the id `evaluated` resolves for one keyed otherwise (a workspace file, whose source + * tools); by the id `planned` resolves for one keyed otherwise (a workspace file, whose source * key is its path), which the plan matches to that kind's change; and each secret it reads by name * (`env.NAME`). */ -function sourceRefs( - references: readonly SourceReference[], - envReferences: readonly string[], - evaluated: EvaluatedSource | undefined -): NonNullable { +function sourceRefs(summary: SourcePlanSummary): NonNullable { const refs = new Map[number]>() - for (const { kind, key, member, span } of references) { + for (const { kind, key, member, span } of summary.references) { if (!isWorkspaceKind(kind)) continue if (isResourceKeyed(kind)) { const id = JSON.stringify([kind, key, member ?? null]) @@ -227,19 +221,19 @@ function sourceRefs( }) continue } - const id = evaluated?.plan.resources.toId(kind, key) + const id = summary.resolveId(kind, key) if (id) refs.set(JSON.stringify([kind, key, null]), { kind, key, id }) } - for (const name of envReferences) { + for (const name of summary.envReferences) { refs.set(JSON.stringify(['secret', name, null]), { kind: 'secret', key: name }) } return [...refs.values()] } -/** The evaluated graph's blocks, id to normalized name, as a change's `blocks`. */ -function blocksOf(evaluated: EvaluatedSource | undefined): Pick { - if (!evaluated) return {} - const blocks = Object.values(evaluated.plan.blocks).map((block): [string, string] => [ +/** The planned graph's blocks, id to normalized name, as a change's `blocks`. */ +function blocksOf(summary: SourcePlanSummary): Pick { + if (!summary.blocks) return {} + const blocks = summary.blocks.map((block): [string, string] => [ block.id, normalizeWorkflowBlockName(block.name), ]) @@ -409,20 +403,6 @@ function classifyFolder( return { change, create: true } } -/** - * Whether a plan's evaluation still holds at write time: it resolved no stand-in id of a resource - * this apply creates first (those get their real ids only once written). A workflow this apply - * creates is no stand-in: its id is minted at plan time and is the one the create stores. - */ -function commitsAsPlanned( - references: readonly SourceReference[], - pending: readonly PendingResource[] -): boolean { - return !references.some((ref) => - pending.some((entry) => entry.placeholder && entry.kind === ref.kind && entry.key === ref.key) - ) -} - /** * Evaluates an edited folder of a stored workflow (plan only, no write). `pending` holds the * resources this apply creates or adopts first, which the evaluation resolves. @@ -441,7 +421,7 @@ async function planUpdate( change.diagnostics.push(...read.diagnostics) return change } - const result = await applyWorkflowSource.execute({ + const outcome = await workflowSourceUpdate.plan.execute({ principal: ctx.principal, input: { workflowId: update.workflowId, @@ -449,28 +429,27 @@ async function planUpdate( lock: read.lock, base: update.base, allowDelete: ctx.allowDelete, - planOnly: true, pendingResources: pending, - shared: ctx.shared, + loads: ctx.loads, }, }) + const result = summarizePlan(outcome) change.diagnostics.push(...inFolder(key, result.diagnostics)) - if (result.diff.meta?.includes('name')) renamed.set(key, result.workflowName) + if (result.workflowName !== undefined && result.diff.meta?.includes('name')) { + renamed.set(key, result.workflowName) + } // A graph no-op that changes only idents, order or comments still writes its source meta. const metaOnly = result.diff.noop && result.sourceMetaChanged - const evaluated = - result.evaluated && commitsAsPlanned(result.references, pending) ? result.evaluated : undefined return { ...change, action: result.diff.noop && !metaOnly ? 'noop' : 'update', storedId: update.workflowId, fields: metaOnly ? ['names, order or comments'] : diffFields(result.diff), - refs: sourceRefs(result.references, result.envReferences, result.evaluated), - ...blocksOf(result.evaluated), + refs: sourceRefs(result), + ...blocksOf(result), detail: { update: { lock: read.lock, base: update.base, allowDelete: ctx.allowDelete }, - created: pending.filter((entry) => entry.kind === 'workflow'), - ...(evaluated ? { evaluated } : {}), + ...(outcome.kind === 'planned' ? { planned: outcome } : {}), }, } } @@ -489,35 +468,32 @@ async function planCreate( ): Promise<{ change: WorkflowChange; name?: string }> { const { decl } = change if (!decl) return { change } - const result = await createWorkflowFromSource.execute({ + const outcome = await workflowSourceCreate.plan.execute({ principal: ctx.principal, input: { workspaceId: ctx.workspace.workspaceId, workflowId: created.id, files: decl.files, - planOnly: true, pendingResources: pending, - shared: ctx.shared, + loads: ctx.loads, }, }) + const result = summarizePlan(outcome) change.diagnostics.push(...inFolder(change.key, result.diagnostics)) - const evaluated = - result.evaluated && commitsAsPlanned(result.references, pending) ? result.evaluated : undefined return { change: { ...change, action: 'create', pending: created, fields: diffFields(result.diff), - refs: sourceRefs(result.references, result.envReferences, result.evaluated), - ...blocksOf(result.evaluated), + refs: sourceRefs(result), + ...blocksOf(result), detail: { workflowId: created.id, - created: pending.filter((entry) => entry.kind === 'workflow'), - ...(evaluated ? { evaluated } : {}), + ...(outcome.kind === 'planned' ? { planned: outcome } : {}), }, }, - ...(result.evaluated ? { name: result.workflowName } : {}), + ...(result.workflowName !== undefined ? { name: result.workflowName } : {}), } } @@ -846,40 +822,28 @@ export const workflowAdapter: KindAdapter<'workflow', StoredWorkflowState> = { async apply(change, ctx) { const { principal } = ctx - const workspaceId = ctx.workspace.workspaceId if (change.action === 'delete') { if (!change.storedId) throw new Error('A workflow delete needs the stored workflow id') await deleteWorkflow.execute({ principal, input: { workflowId: change.storedId } }) return { id: change.storedId, text: {} } } - const decl = change.decl - if (!decl) throw new Error('A workflow write needs its files') const update = change.detail?.update - if (change.action !== 'create' && (!update || !change.storedId)) { - throw new Error('A workflow update needs its plan') - } - const submission = { - files: decl.files, - shared: ctx.shared, - ...(change.detail?.created?.length ? { pendingResources: change.detail.created } : {}), - ...(ctx.provenance ? { provenance: ctx.provenance } : {}), - ...(change.detail?.evaluated ? { evaluated: change.detail.evaluated } : {}), + const planned = change.detail?.planned + if (!planned || (change.action !== 'create' && !update)) { + throw new Error('A workflow write needs its plan') } + const provenance = ctx.provenance ? { provenance: ctx.provenance } : {} // A new folder has no base, so no later apply could adopt a workflow whose create committed // without its key: the create claims it in its own transaction, and a failed claim undoes it. const claimed: WorkspaceSourceDiagnostic[] = [] const write = () => - update && change.storedId - ? applyWorkflowSource.execute({ - principal, - input: { workflowId: change.storedId, ...update, ...submission }, - }) - : createWorkflowFromSource.execute({ + update + ? workflowSourceUpdate.commit.execute({ principal, input: { planned, ...provenance } }) + : workflowSourceCreate.commit.execute({ principal, input: { - workspaceId, - ...(change.detail?.workflowId ? { workflowId: change.detail.workflowId } : {}), - ...submission, + planned, + ...provenance, withinCreate: async (tx, workflowId) => { claimed.splice(0, claimed.length, ...(await claimKey(change, ctx, workflowId, tx))) }, diff --git a/apps/sim/lib/workspace-source/application/plan.ts b/apps/sim/lib/workspace-source/application/plan.ts index a309d3f20fb..7bb241d948a 100644 --- a/apps/sim/lib/workspace-source/application/plan.ts +++ b/apps/sim/lib/workspace-source/application/plan.ts @@ -17,7 +17,6 @@ import { isFileKind, KIND_DIRS, parseWorkspaceBase, - refusalClass, resourceAddress, type SourceBudget, type WorkspaceBaseV1, @@ -153,49 +152,6 @@ async function resolveStaged( /** Changes a plan authorizes at once. */ const AUTHORIZE_CONCURRENCY = 6 -/** Findings one plan reports; past it, one more finding counts the rest. */ -const MAX_PLAN_FINDINGS = 1_000 - -const SEVERITY_RANK = { error: 0, warning: 1, info: 2 } as const - -/** - * How early a finding is kept: by severity, and among errors one that classes a refusal (stale, - * locked, too large) before the rest, so the refusal is classed as the uncut findings class it. - */ -function keepRank(finding: WorkspaceSourceDiagnostic): number { - const rank = SEVERITY_RANK[finding.severity] * 2 - return finding.severity === 'error' && refusalClass([finding.code]) === 'validation' - ? rank + 1 - : rank -} - -/** - * `diagnostics` within {@link MAX_PLAN_FINDINGS}: errors first so a refusal always shows why - * (those that class it first, see {@link keepRank}), then warnings, then infos. The finding that - * counts the rest is an info, so cutting a plan's findings never refuses it, nor changes how a - * refusal is classed. - */ -function capFindings(diagnostics: WorkspaceSourceDiagnostic[]): WorkspaceSourceDiagnostic[] { - if (diagnostics.length <= MAX_PLAN_FINDINGS) return diagnostics - const kept = diagnostics - .map((entry, index) => ({ entry, index })) - .sort((a, b) => keepRank(a.entry) - keepRank(b.entry) || a.index - b.index) - .slice(0, MAX_PLAN_FINDINGS) - .map(({ entry }) => entry) - const left = diagnostics.length - kept.length - const leftErrors = - diagnostics.filter((entry) => entry.severity === 'error').length - - kept.filter((entry) => entry.severity === 'error').length - return [ - ...kept, - finding( - 'I_FINDINGS_TRUNCATED', - `The plan has ${diagnostics.length} findings; ${left} more${leftErrors > 0 ? ` (${leftErrors} of them errors)` : ''} are left out.`, - { hint: 'Fix the findings shown, then apply again to see the rest.' } - ), - ] -} - /** * Names each reference made by id (`refs[].id`) by the key of the change that plans that resource * (its stored id, or the stand-in id of a create), so a delete of it and a write waiting for it @@ -331,7 +287,7 @@ export async function buildWorkspacePlan( if (diagnostics.some((finding) => finding.severity === 'error')) { return { changes: [], - diagnostics: capFindings(diagnostics), + diagnostics, base, needBlobs: [], staged: new Map(), @@ -490,7 +446,7 @@ export async function buildWorkspacePlan( return { changes, - diagnostics: capFindings(diagnostics), + diagnostics, base, needBlobs: blobs.missing, staged: blobs.staged, diff --git a/apps/sim/lib/workspace-source/application/types.ts b/apps/sim/lib/workspace-source/application/types.ts index e4dcca210e3..d7b33029204 100644 --- a/apps/sim/lib/workspace-source/application/types.ts +++ b/apps/sim/lib/workspace-source/application/types.ts @@ -19,8 +19,7 @@ import type { SkillDecl } from '@sim/workspace-code/kinds/skills' import type { TableDecl } from '@sim/workspace-code/kinds/table' import type { ToolDecl } from '@sim/workspace-code/kinds/tools' import type { WorkflowDecl } from '@sim/workspace-code/kinds/workflow' -import type { SourceRequestShare } from '@/lib/workflows/application/apply-workflow-source' -import type { PendingResource } from '@/lib/workflows/application/source-evaluate' +import type { PendingResource, SourceLoads } from '@/lib/workflows/application/source-pipeline' import type { HashBudget } from '@/lib/workspace-source/application/hash-budget' import type { CredentialPlan } from '@/lib/workspace-source/application/kinds/credentials/adapter' import type { DeploymentPlan } from '@/lib/workspace-source/application/kinds/deployment/adapter' @@ -46,7 +45,7 @@ export interface SourceContext { /** Workspace-wide admissions this request already ran, by key (see `admitOnce`). */ admitted: Map> /** What every workflow this request evaluates loads the same (its catalog gate, env names). */ - shared: SourceRequestShare + loads: SourceLoads /** * Workspace-wide listings this request already started, reused by every kind and check that * needs one; spread copies of the context share it. diff --git a/apps/sim/package.json b/apps/sim/package.json index 388727e403e..1f25af6afa7 100644 --- a/apps/sim/package.json +++ b/apps/sim/package.json @@ -169,6 +169,7 @@ "@sim/realtime-protocol": "workspace:*", "@sim/runtime-secrets": "workspace:*", "@sim/security": "workspace:*", + "@sim/table-cells": "workspace:*", "@sim/terminal-protocol": "workspace:*", "@sim/utils": "workspace:*", "@sim/workflow-code": "workspace:*", diff --git a/bun.lock b/bun.lock index 3ad519e8ab4..bab6aae6bb7 100644 --- a/bun.lock +++ b/bun.lock @@ -314,6 +314,7 @@ "@sim/realtime-protocol": "workspace:*", "@sim/runtime-secrets": "workspace:*", "@sim/security": "workspace:*", + "@sim/table-cells": "workspace:*", "@sim/terminal-protocol": "workspace:*", "@sim/utils": "workspace:*", "@sim/workflow-code": "workspace:*", @@ -803,6 +804,19 @@ "vitest": "^5.1.0", }, }, + "packages/table-cells": { + "name": "@sim/table-cells", + "version": "0.1.0", + "dependencies": { + "@sim/utils": "workspace:*", + "zod": "4.3.6", + }, + "devDependencies": { + "@sim/tsconfig": "workspace:*", + "typescript": "^7.0.2", + "vitest": "^5.1.0", + }, + }, "packages/terminal-protocol": { "name": "@sim/terminal-protocol", "version": "0.1.0", @@ -863,6 +877,7 @@ }, "devDependencies": { "@babel/types": "7.29.8", + "@sim/testing": "workspace:*", "@sim/tsconfig": "workspace:*", "@xyflow/react": "12.12.1", "fast-check": "3.23.2", @@ -955,12 +970,13 @@ "version": "0.1.0", "dependencies": { "@babel/parser": "7.29.9", + "@sim/table-cells": "workspace:*", "@sim/utils": "workspace:*", "@sim/workflow-code": "workspace:*", - "zod": "4.3.6", }, "devDependencies": { "@babel/types": "7.29.8", + "@sim/testing": "workspace:*", "@sim/tsconfig": "workspace:*", "@types/js-yaml": "4.0.9", "fast-check": "3.23.2", @@ -2130,6 +2146,8 @@ "@sim/security": ["@sim/security@workspace:packages/security"], + "@sim/table-cells": ["@sim/table-cells@workspace:packages/table-cells"], + "@sim/terminal-protocol": ["@sim/terminal-protocol@workspace:packages/terminal-protocol"], "@sim/testing": ["@sim/testing@workspace:packages/testing"], diff --git a/packages/sim-cli/src/commands/protocol/local-errors.ts b/packages/sim-cli/src/commands/protocol/local-errors.ts new file mode 100644 index 00000000000..64b35ab2248 --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/local-errors.ts @@ -0,0 +1,44 @@ +/** + * Refusals a source command raises itself, with no request behind them. They render and exit + * through the same path as an API error (`status` 0: no server answered), so a script sees the + * same message, code and exit status, while the code that raises one says it is local. + */ + +import { SimApiError } from '../../http/client' + +/** What a local refusal carries beside its message: a machine-readable code and the exit status. */ +interface LocalErrorOptions { + /** The code a script branches on (`SOURCE_SYMLINK`, `SOURCE_DIR_BUSY`); none by default. */ + code?: string + /** The process exit status; 1 by default. */ + exitCode?: number +} + +/** + * A refusal the CLI raised itself. Its `status` 0 and possibly null `code` read like a request + * that got no answer, so a check for an unreachable server or a lost answer excludes this class. + */ +export abstract class LocalRefusal extends SimApiError {} + +/** + * A source directory, its records or a file it would send or write refused by the CLI: a missing + * or unreadable file, a symbolic link, another kind of directory, a busy or unwritable directory, + * or a server answer naming a path the CLI never writes. + */ +export class SourceDirError extends LocalRefusal { + constructor(message: string, options: LocalErrorOptions = {}) { + super(message, 0, options.code ?? null, undefined, options.exitCode ?? 1) + this.name = 'SourceDirError' + } +} + +/** + * Flags or arguments a source command cannot act on: an unknown kind or path filter, flags that + * contradict each other, or a missing `--workspace`. Refused before any request is sent. + */ +export class CliUsageError extends LocalRefusal { + constructor(message: string, options: Pick = {}) { + super(message, 0, options.code ?? null) + this.name = 'CliUsageError' + } +} diff --git a/packages/sim-cli/src/commands/protocol/source-check.ts b/packages/sim-cli/src/commands/protocol/source-check.ts new file mode 100644 index 00000000000..298816947fd --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-check.ts @@ -0,0 +1,623 @@ +/** + * The local check of a workflow source directory, shared by `workflows source check` and + * `workspaces source check`: compiling a workflow directory against the cached catalog and the + * workspace's declarations, the `--strict` rule, and how a check reports. + */ + +import { join } from 'node:path' +import { isRecordLike, toRecord } from '@sim/utils/object' +import { + type BlockCatalog, + type CanonicalWorkflow, + codeFrame, + codePointLabel, + compileWorkflow, + didYouMean, + EDIT_REFUSALS, + encodeAnchor, + getOwn, + parseJsonField, + parseWorkflowLock, + raisedMessage, + refuseEditedFindings, + scriptName, + startInputs, + type Value, + type WorkflowInput, +} from '@sim/workflow-code' +import { type AnswerBound, boundAnswerFindings } from '@sim/workflow-code/diagnostics' +import { SOURCE_ENTRY_FILE, SOURCE_LOCK_FILE } from '@sim/workflow-code/limits' +import { KIND_DIRS } from '@sim/workspace-code' +import { KIND_LAYOUTS } from '@sim/workspace-code/kinds/registry' +import type { Command } from 'commander' +import { setSoftExitCode } from '../../embed-context' +import { SourceDirError } from './local-errors' +import { printProtocolResult } from './result' +import { digest, readFolderWritten, readSourceJson, readSourceText } from './source-dir' +import { + cachedCatalog, + cachedSourceCatalog, + type LocalDeclarations, + refreshTsconfigs, +} from './source-tree' +import { type Diagnostic, findingsOutput } from './source-ui' +import { blockDigest, readStoredBlockDigests, readStoredEdges } from './stored-edges' +import { readSourceFiles } from './workflow-dir' +import { kindFolders, WORKSPACE_TREE, type WorkspaceDeclarations } from './workspace-source-dir' + +/** + * A finding only the server can decide: `stored()` keeps a secret the stored workflow holds, and + * a local compile has no stored workflow to read it from. + */ +const SERVER_ONLY_CODES: ReadonlySet = new Set(['E_STORED_VALUE_MISSING']) + +/** + * Warnings an apply saves as written although the block they name is broken: a strict check + * fails on them (see {@link StrictMode}). + */ +const STRICT_CODES: ReadonlySet = new Set([ + 'W_REQUIRED_MISSING', + 'W_UNKNOWN_OPTION', + 'W_UNKNOWN_OUTPUT_PATH', + 'W_VALUE_KIND', + 'W_OUT_OF_RANGE', + 'W_LITERAL_SECRET', + 'W_TABLE_UNKNOWN_COLUMN', + 'W_CHILD_INPUT_UNKNOWN', + 'W_BRANCH_UNWIRED', + 'W_UNRESOLVED_REF', + 'W_REF_OUT_OF_SCOPE', +]) + +/** + * {@link STRICT_CODES} warnings the stored workflow exempts, which a check never fails on a block + * the lock binds, `--strict` included: the server stays quiet when the stored workflow holds the + * same state, and an offline check cannot read it. `W_BRANCH_UNWIRED`: a branch the stored + * workflow leaves unwired. + */ +const STORED_EXEMPT_CODES: ReadonlySet = new Set(['W_BRANCH_UNWIRED']) + +/** + * Warnings a check reports only on a block the lock does not bind (one added since the get), and + * on every block with `--strict`: printed source holds them on blocks nobody edited, and the + * server, which compares with the stored workflow, already reports them only for an edited field. + * `W_JSON_REF_UNTYPED`: a bare reference nothing types, in a JSON field. + */ +const EDIT_SCOPED_CODES: ReadonlySet = new Set(['W_JSON_REF_UNTYPED']) + +/** + * Which {@link STRICT_CODES} warnings fail a check: `new` those on a block the lock does not bind + * (one added since the get, or every block of a workflow that has no lock yet), `all` every one, + * `off` none. Printed source can hold such a warning on a block nobody edited (a field left empty + * in the editor), so the default fails only on blocks the edit wrote. + */ +type StrictMode = 'new' | 'all' | 'off' + +/** One finding of a check, as the write commands print theirs. */ +export type CheckFinding = Diagnostic + +/** The `--strict` / `--no-strict` pair: absent selects the default, {@link StrictMode} `new`. */ +export interface StrictOptions { + strict?: boolean +} + +/** The strict mode the `--strict` / `--no-strict` pair selects. */ +export function strictMode(options: StrictOptions): StrictMode { + if (options.strict === undefined) return 'new' + return options.strict ? 'all' : 'off' +} + +/** Where each kind of edit refusal applies, as the help says it. */ +const REFUSAL_WHERE: Record<(typeof EDIT_REFUSALS)[number]['where'], string> = { + 'changed block': 'in any file of a block the edit added or changed', + 'workflow.ts': 'in workflow.ts on a block the edit added or changed', + 'edited code file': 'in a function file changed since the get', +} + +/** Help shared by both check commands: what fails, and the strict flags. */ +export const STRICT_HELP = `Exit codes: 0 no errors; 1 errors (each printed as file:line:column with its frame and hint), or +no cached catalog (run the get once). With --output json, stdout is one document that also lists +every finding under diagnostics. + +These warnings, which an apply saves as written, fail the check as errors on every block the lock +does not bind (a block added since the get, or every block of a new workflow): +${[...STRICT_CODES].map((code) => ` ${code}`).join(',\n')}. +--strict fails on them in every block, except ${[...STORED_EXEMPT_CODES].join(', ')} on a block the +lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict fails +on none of them. In every mode, as the server refuses them: ${EDIT_REFUSALS.map( + ({ warning, refusal, where }) => `${warning} is the error ${refusal} ${REFUSAL_WHERE[where]}` +).join('; ')}. A block counts as changed when the lock does not bind it, when the source moved it +into another container, or when it compiles differently than in the source the last get or +apply printed (.sim/stored-edges.json records a digest of each block). That last comparison +needs the catalog the source was printed with in the cache: without it, a bound block that did +not move counts as unchanged, so the server may refuse what the check passes. A $ref in a +function file to a block the lock binds that the source no longer declares stays a warning, unless +a warning cap left findings out: the server may infer that block was renamed and rewrite the +reference, and refuses it otherwise. +${[...EDIT_SCOPED_CODES].join(', ')} is reported only on blocks the lock does not bind, and with +--strict on every block.` + +/** Adds the `--strict` / `--no-strict` pair to a check command. */ +export function strictFlags(command: Command): Command { + return command + .option('--strict', 'Fail on the warnings listed below in every block, not only new ones') + .option('--no-strict', 'Never fail on a warning an apply saves') +} + +/** Every workflow key of the workspace directory `root` that has a `workflow.ts`. */ +export async function workspaceWorkflowKeys(root: string): Promise { + const { paths } = await kindFolders(root, ['workflow']) + return [...KIND_LAYOUTS.workflow.group(paths).groups] + .filter(([key, folderPaths]) => folderPaths.includes(workflowEntry(key))) + .map(([key]) => key) +} + +/** A workflow's `workflow.ts`, relative to the workspace directory. */ +function workflowEntry(key: string): string { + return `${KIND_DIRS.workflow}/${key}/${SOURCE_ENTRY_FILE}` +} + +/** + * Points the workspace directory's tsconfig files at the blocks its workflows now use, keeping the + * blocks they include. + */ +export async function refreshWorkspaceTsconfigs(root: string, workspaceId: string): Promise { + const keys = await workspaceWorkflowKeys(root) + const sources = await Promise.all(keys.map((key) => readSourceText(root, workflowEntry(key)))) + await refreshTsconfigs( + root, + workspaceId, + WORKSPACE_TREE, + { sources: sources.filter((text) => text !== null) }, + { incremental: true } + ) +} + +/** A canonical resource node: what `table('k')`, `credential('k')` and the like compile to. */ +function resourceNode(value: unknown): { kind: string; key: string } | null { + if (!isRecordLike(value) || value.$sim !== 'resource') return null + const { kind, key } = value + return typeof kind === 'string' && typeof key === 'string' ? { kind, key } : null +} + +/** Every resource node inside a value, depth first. */ +function resourceNodes(value: unknown): Array<{ kind: string; key: string }> { + const node = resourceNode(value) + if (node) return [node] + if (Array.isArray(value)) return value.flatMap(resourceNodes) + if (isRecordLike(value)) return Object.values(value).flatMap(resourceNodes) + return [] +} + +/** + * Why a key can never match: ` "triаge" holds U+0430, a Cyrillic letter; …` for a key with a + * character outside lowercase ASCII (a resource key is `[a-z][a-z0-9_-]*`), which reads the same + * as the declared key it imitates; empty for a key in the key format. + */ +function keyCharacterNote(key: string): string { + const odd = [...key].find((character) => !/[a-z0-9_-]/.test(character)) + if (odd === undefined) return '' + const script = scriptName(odd) + return ` "${key}" holds ${codePointLabel(odd)}${script ? `, a ${script} letter` : ''}: a key is lowercase ASCII letters, digits, - and _, so retype it.` +} + +/** How many keys a hint lists. */ +const LISTED_KEYS = 20 + +function listed(keys: Iterable): string { + const all = [...keys] + const more = all.length > LISTED_KEYS ? `, and ${all.length - LISTED_KEYS} more` : '' + return `${all.slice(0, LISTED_KEYS).join(', ')}${more}` +} + +/** + * A row-write param's value as a JSON tree: compiled row data is JSON text whose non-string + * references sit bare (`"n": `), so it parses as a JSON field, references as leaves. + */ +function rowJson(value: Value): unknown { + return typeof value === 'string' ? parseJsonField(value) : value +} + +/** The params of a table block that hold row data: one row, or a list of rows. */ +const ROW_PARAMS = ['data', 'rows'] as const + +/** + * A finding about one block param, located by the caller: at the param, or at the param whose + * source names the resource key `names` (a source param name can differ from its canonical id). + */ +type ParamFinding = Pick & { + block: string + param: string + names?: string +} + +/** The `inputMapping` of a child workflow call, as a record of input name to value. */ +function inputMappingOf(canonical: CanonicalWorkflow['blocks'][string]): Record { + const mapping = canonical.params.inputMapping + if (mapping?.kind !== 'value') return {} + const value = rowJson(mapping.value) + return toRecord(value) +} + +/** + * A child workflow call checked against the child's own `workflow.ts` in this directory: a child + * without a Start trigger (`W_CHILD_NO_START`), and an `inputMapping` key its Start trigger does + * not declare (`W_CHILD_INPUT_UNKNOWN`). A child that did not compile is not checked. + */ +function childInputFindings( + block: string, + canonical: CanonicalWorkflow['blocks'][string], + declarations: WorkspaceDeclarations +): ParamFinding[] { + const ref = canonical.params.workflowId + const child = ref?.kind === 'value' ? resourceNode(ref.value) : null + if (child?.kind !== 'workflow') return [] + const inputs = declarations.workflowInputs?.get(child.key) + if (inputs === undefined) return [] + const entry = `${KIND_DIRS.workflow}/${child.key}/${SOURCE_ENTRY_FILE}` + if (inputs === null) { + return [ + { + code: 'W_CHILD_NO_START', + severity: 'warning', + message: `Workflow "${child.key}" has no Start trigger, so it takes no inputs and a call cannot run it.`, + hint: `Add trigger.start('Start', { inputFormat: { … } }) to ${entry}, or call a workflow that has one.`, + block, + param: 'workflowId', + names: child.key, + }, + ] + } + const names = inputs.map((input) => input.name) + return Object.keys(inputMappingOf(canonical)) + .filter((field) => !names.includes(field)) + .map((field) => ({ + code: 'W_CHILD_INPUT_UNKNOWN', + severity: 'warning', + message: `inputMapping sets ${field}, which is not an input of workflow "${child.key}"; the child never receives it.`, + hint: + names.length > 0 + ? `${didYouMean(field, names) ?? ''} Inputs (the Start trigger's inputFormat in ${entry}): ${listed(names)}.`.trimStart() + : `The Start trigger of ${entry} declares no inputs: add ${field} to its inputFormat.`, + block, + param: 'inputMapping', + })) +} + +/** + * The findings a compile cannot make without the workspace's declarations: a resource key the + * directory does not declare (an apply fails on it), and a table row write naming a column, or a + * select option, its table does not declare. + */ +function declarationFindings( + workflow: CanonicalWorkflow, + declarations: WorkspaceDeclarations +): ParamFinding[] { + const findings: ParamFinding[] = [] + for (const [block, canonical] of Object.entries(workflow.blocks)) { + const tables = new Set() + for (const [param, value] of Object.entries(canonical.params)) { + for (const node of resourceNodes(value)) { + if (node.kind === 'table') tables.add(node.key) + const declared = declarations.keys.get(node.kind) + if (!declared || declared.has(node.key)) continue + const near = didYouMean(node.key, declared) + findings.push({ + code: 'E_MAT_UNKNOWN_RESOURCE', + severity: 'error', + message: `No ${node.kind} "${node.key}" is declared in this workspace directory.${keyCharacterNote(node.key)}`, + hint: + declared.size > 0 + ? `${near ? `${near} ` : ''}Keys: ${listed(declared)}.` + : `This directory declares no ${node.kind}: declare it before naming it.`, + block, + param, + names: node.key, + }) + } + } + findings.push(...childInputFindings(block, canonical, declarations)) + if (tables.size !== 1) continue + const [table] = tables + const columns = declarations.tables.get(table) + if (!columns) continue + for (const param of ROW_PARAMS) { + const value = canonical.params[param] + if (value?.kind !== 'value') continue + const json = rowJson(value.value) + for (const row of Array.isArray(json) ? json : [json]) { + if (!isRecordLike(row)) continue + for (const [field, cell] of Object.entries(row)) { + const column = columns.get(field) + if (!column) { + findings.push({ + code: 'W_TABLE_UNKNOWN_COLUMN', + severity: 'warning', + message: `${param} writes ${field}, which is not a column of table "${table}"; the run drops it.`, + hint: `${didYouMean(field, columns.keys()) ?? ''} Columns: ${listed(columns.keys())}.`.trimStart(), + block, + param, + }) + continue + } + const { options } = column + if (!options || column.multiple || typeof cell !== 'string' || cell.includes('<')) + continue + if (options.includes(cell)) continue + findings.push({ + code: 'W_UNKNOWN_OPTION', + severity: 'warning', + message: `${field} of table "${table}" has no option ${cell}; it is kept as written.`, + hint: `Options: ${options.join(', ')}.`, + block, + param, + }) + } + } + } + } + return findings +} + +/** + * Compiles the workflow source directory `dir` against the catalog its base names when that one is + * cached (a workflow holding a type the workspace catalog lacks was printed with its own), else + * `catalog`, checking env names against the names `secrets.ts` declares when `declarations` hold + * them. + */ +async function compileWorkflowDir( + dir: string, + workspaceCatalog: BlockCatalog, + declarations?: WorkspaceDeclarations +) { + const { lock } = parseWorkflowLock(await readSourceJson(dir, SOURCE_LOCK_FILE)) + const [files, storedEdges, storedBlocks, printedWith] = await Promise.all([ + readSourceFiles(dir, lock ?? null), + readStoredEdges(dir), + readStoredBlockDigests(dir), + cachedSourceCatalog(dir), + ]) + const catalog = printedWith ?? workspaceCatalog + const envNames = declarations?.envNames + const compiled = compileWorkflow({ + entry: { path: SOURCE_ENTRY_FILE, text: files[SOURCE_ENTRY_FILE] }, + files: new Map(Object.entries(files).filter(([path]) => path !== SOURCE_ENTRY_FILE)), + catalog, + sdkMajor: catalog.sdkMajor, + knownKeys: new Set(Object.keys(lock?.blocks ?? {})), + storedUnread: lock ?? undefined, + ...(storedEdges ? { storedUnreadEdges: storedEdges } : {}), + ...(envNames ? { envNames, envDeclaredIn: KIND_DIRS.secret } : {}), + }) + return { lock, files, compiled, storedBlocks, printedWith } +} + +/** + * The inputs of each workflow of the workspace directory `root` that compiles (`startInputs`, + * `null` for one without a Start trigger), by key: what a child workflow call is checked against + * and what `workspace.d.ts` types its `inputMapping` with. + */ +export async function workspaceWorkflowInputs( + root: string, + catalog: BlockCatalog +): Promise> { + const inputs = new Map() + for (const key of await workspaceWorkflowKeys(root)) { + const { compiled } = await compileWorkflowDir(join(root, KIND_DIRS.workflow, key), catalog) + if (compiled.workflow) inputs.set(key, startInputs(compiled.workflow)) + } + return inputs +} + +/** + * The workflows of `local` typed with the inputs each declares (`inputs`), for `workspace.d.ts`. + */ +export function withWorkflowInputs( + local: LocalDeclarations, + inputs: ReadonlyMap +): LocalDeclarations { + const workflows = { ...local.workflows } + for (const [key, fields] of inputs) { + workflows[key] = { + inputs: Object.fromEntries((fields ?? []).map((field) => [field.name, field.type])), + } + } + return { ...local, workflows } +} + +/** + * Compiles one workflow source directory offline against `catalog`, with the workspace's + * declarations when the directory sits in a workspace directory, and returns its findings with + * the {@link STRICT_CODES} the strict mode fails on raised to errors, and the `workflow.ts` text. + * `prefix` goes before each file path (`workflows//` in a workspace check). + */ +export async function checkWorkflowDir( + dir: string, + catalog: BlockCatalog, + options: { declarations?: WorkspaceDeclarations; strict: StrictMode; prefix?: string } +): Promise<{ findings: CheckFinding[]; entry: string }> { + const { lock, files, compiled, storedBlocks, printedWith } = await compileWorkflowDir( + dir, + catalog, + options.declarations + ) + const entryText = files[SOURCE_ENTRY_FILE] + const locate = (block: string, param: string, names: string | undefined) => { + const prefix = encodeAnchor({ kind: 'param', block, param: '' }) + const naming = + names === undefined + ? undefined + : Object.entries(compiled.spans).find( + ([anchor, span]) => + anchor.startsWith(prefix) && + span !== undefined && + span.file === SOURCE_ENTRY_FILE && + ['"', "'", '`'].some((quote) => + entryText.slice(span.start, span.end).includes(`${quote}${names}${quote}`) + ) + )?.[1] + return ( + naming ?? + compiled.spans[encodeAnchor({ kind: 'param', block, param })] ?? + compiled.spans[encodeAnchor({ kind: 'block', block })] + ) + } + const declared = + compiled.workflow && options.declarations + ? declarationFindings(compiled.workflow, options.declarations).map( + ({ block, param, names, ...finding }) => { + const span = locate(block, param, names) + const text = span ? files[span.file] : undefined + return { + ...finding, + blockKey: block, + ...(span ? { span } : {}), + ...(span && text !== undefined ? { frame: codeFrame(text, span) } : {}), + } + } + ) + : [] + const bound = (blockKey: string | undefined) => + lock !== undefined && blockKey !== undefined && Object.hasOwn(lock.blocks, blockKey) + const written = await readFolderWritten(dir) + // A file only moved from the path the lock bound its block to is unedited while its text matches that path's digest. + const edited = (file: string, blockKey: string | undefined) => { + const text = files[file] + if (text === undefined) return true + const textDigest = digest(text) + if (written.digests[file] === textDigest) return false + const previousFile = + lock && blockKey !== undefined && Object.hasOwn(lock.blocks, blockKey) + ? lock.blocks[blockKey].codeFile + : undefined + return previousFile === undefined || written.digests[previousFile] !== textDigest + } + const moved = (blockKey: string) => { + const held = lock ? getOwn(lock.blocks, blockKey) : undefined + const block = compiled.workflow ? getOwn(compiled.workflow.blocks, blockKey) : undefined + return held !== undefined && block !== undefined && held.parent !== block.parent + } + /** + * Whether the edit added, moved or changed the block. A block's digest is compared only when + * the catalog it was printed with is cached: against another catalog every block may compile + * differently, and a check prefers a warning to a refusal the server would not make. + */ + const changed = (blockKey: string) => { + if (!bound(blockKey) || moved(blockKey)) return true + const block = compiled.workflow?.blocks[blockKey] + const printed = printedWith ? storedBlocks?.get(blockKey) : undefined + return block !== undefined && printed !== undefined && blockDigest(block) !== printed + } + const fails = (code: string, blockKey: string | undefined) => + STRICT_CODES.has(code) && + !(STORED_EXEMPT_CODES.has(code) && bound(blockKey)) && + (options.strict === 'all' || (options.strict === 'new' && !bound(blockKey))) + const prefix = options.prefix ?? '' + const reported = (code: string, blockKey: string | undefined) => + !SERVER_ONLY_CODES.has(code) && + !(EDIT_SCOPED_CODES.has(code) && options.strict !== 'all' && bound(blockKey)) + /** + * Whether the code `blockKey` runs from `file` changed, as the server compares it: the file's + * text, or a bound block now pointed at another file whose text differs from what it ran. + */ + const codeEdited = (blockKey: string, file: string) => { + if (edited(file, blockKey)) return true + const before = lock ? getOwn(lock.blocks, blockKey)?.codeFile : undefined + const text = files[file] + return ( + before !== undefined && + before !== file && + text !== undefined && + written.digests[before] !== digest(text) + ) + } + const removedKeys = new Set( + Object.keys(lock?.blocks ?? {}).filter( + (key) => !compiled.workflow || getOwn(compiled.workflow.blocks, key) === undefined + ) + ) + /** + * A refused `$ref` in a code file naming a block the lock binds that the source no longer + * declares: the server may infer that block's rename (name and const both changed) and then + * rewrite the reference, which only it can tell, so the check keeps the warning. + */ + // Past a warning cap the compile keeps one finding per code, block and file, which could stand + // for a reference no rename settles, so then nothing is exempt. + const listedEvery = !compiled.diagnostics.some((d) => d.code === 'I_FINDINGS_TRUNCATED') + const mayBeRenamed = (finding: (typeof compiled.diagnostics)[number]) => { + if (!listedEvery || finding.code !== 'W_UNRESOLVED_REF') return false + if (finding.span.file === SOURCE_ENTRY_FILE) return false + const text = files[finding.span.file]?.slice(finding.span.start, finding.span.end) ?? '' + const head = /^(?:\$ref\.|<)([A-Za-z0-9_]+)/.exec(text)?.[1] + return head !== undefined && removedKeys.has(head) + } + const refusable = (finding: (typeof compiled.diagnostics)[number]) => !mayBeRenamed(finding) + const refused = [ + ...refuseEditedFindings( + { + diagnostics: compiled.diagnostics.filter(refusable), + withheld: compiled.withheld.filter(refusable), + }, + { changed, codeEdited } + ), + ...compiled.diagnostics.filter(mayBeRenamed), + ] + const findings = [ + ...refused.filter((diagnostic) => reported(diagnostic.code, diagnostic.blockKey)), + ...declared, + ].map(({ code, severity, message, hint, frame, span, blockKey }): CheckFinding => { + const raised = severity === 'warning' && fails(code, blockKey) + return { + code, + severity: raised ? 'error' : severity, + message: raised ? raisedMessage(message) : message, + ...(hint !== undefined ? { hint } : {}), + ...(frame !== undefined ? { frame } : {}), + ...(span ? { file: `${prefix}${span.file}`, line: span.line, column: span.column } : {}), + } + }) + return { findings, entry: entryText } +} + +/** + * Prints a check's findings and result, and exits 1 on any error. The findings are bounded as the + * server bounds an answer of the same source (`bound`, the workflow answer's by default), while + * `errors` and `warnings` count every one. `verbose` keeps info findings in a machine result. + */ +export function reportCheck( + output: Parameters[0], + findings: CheckFinding[], + result: Record, + options: { verbose?: boolean; bound?: AnswerBound } = {} +): void { + const shown = findingsOutput(output, options.verbose) + const listed = boundAnswerFindings(findings, options.bound) + shown.print(listed) + const errors = findings.filter((finding) => finding.severity === 'error').length + if (errors > 0) setSoftExitCode(1) + printProtocolResult(output, { + checked: 'compile', + errors, + warnings: findings.filter((finding) => finding.severity === 'warning').length, + ...result, + ...shown.result(listed), + }) +} + +/** The catalog a cache read found, or a refusal naming `get`, the command that caches one. */ +export function cachedOrRefuse( + catalog: BlockCatalog | null, + workspaceId: string, + get: string +): BlockCatalog { + if (catalog) return catalog + throw new SourceDirError( + `No block catalog is cached for workspace ${workspaceId}. Run ${get} once, then check again.`, + { code: 'CATALOG_NOT_CACHED' } + ) +} + +/** The cached catalog of `workspaceId`, or a refusal naming `get`, the command that caches one. */ +export function requireCachedCatalog(workspaceId: string, get: string): BlockCatalog { + return cachedOrRefuse(cachedCatalog(workspaceId), workspaceId, get) +} diff --git a/packages/sim-cli/src/commands/protocol/source-dir.test.ts b/packages/sim-cli/src/commands/protocol/source-dir.test.ts index 0f01c8f1ade..a574752f667 100644 --- a/packages/sim-cli/src/commands/protocol/source-dir.test.ts +++ b/packages/sim-cli/src/commands/protocol/source-dir.test.ts @@ -18,6 +18,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest' import { type EmbedContext, embedStore } from '../../embed-context' import { EmbeddedOutput } from '../../embed-output' import { + readSourceText, SOURCE_DIR_LOCK, SOURCE_WRITTEN_FILE, takeOverStaleLock, @@ -54,6 +55,21 @@ describe('updateSourceDir', () => { for (const path of records) expect(readFileSync(join(dir, path), 'utf8')).toBe('old') }) + it('in an embedded run, a read the run stopped during throws instead of reading as a missing file', async () => { + const stopped = new AbortController() + const host: EmbedContext = { + identity: { endpoint: 'http://localhost', apiKey: 'test', signal: stopped.signal }, + stdout: new EmbeddedOutput(), + stderr: new EmbeddedOutput(), + readFile: async () => { + stopped.abort() + return 'text' + }, + } + + await expect(embedStore.run(host, () => readSourceText(dir, 'workflow.ts'))).rejects.toThrow() + }) + it('in an embedded run, writes the records after every file, fetched blobs included', async () => { const landed: string[] = [] const host: EmbedContext = { @@ -132,9 +148,44 @@ describe('withSourceDirLock', () => { expect(existsSync(lock)).toBe(false) }) + it('a failed command on a missing nested directory leaves none of the folders the lock created', async () => { + mkdirSync(join(dir, 'out')) + writeFileSync(join(dir, 'out', 'kept.txt'), 'kept') + + await expect( + withSourceDirLock(join(dir, 'out', 'a', 'b'), async () => { + throw new Error('not found') + }) + ).rejects.toThrow('not found') + + expect(existsSync(join(dir, 'out', 'a'))).toBe(false) + expect(readFileSync(join(dir, 'out', 'kept.txt'), 'utf8')).toBe('kept') + }) + + it('keeps an empty folder that was there before the lock created the ones under it', async () => { + mkdirSync(join(dir, 'out')) + + expect(await withSourceDirLock(join(dir, 'out', 'a', 'b'), async () => 'ran')).toBe('ran') + + expect(existsSync(join(dir, 'out', 'a'))).toBe(false) + expect(existsSync(join(dir, 'out'))).toBe(true) + }) + + it('removes the folders it created and keeps an empty one it did not, through a path holding ..', async () => { + mkdirSync(join(dir, 'ws', 'a', 'keep'), { recursive: true }) + + expect( + await withSourceDirLock(`${join(dir, 'ws', 'a', 'new')}/../keep/leaf`, async () => 'ran') + ).toBe('ran') + + expect(existsSync(join(dir, 'ws', 'a', 'keep'))).toBe(true) + expect(existsSync(join(dir, 'ws', 'a', 'keep', 'leaf'))).toBe(false) + expect(existsSync(join(dir, 'ws', 'a', 'new'))).toBe(false) + }) + it('releases the lock when an interrupt ends the command holding it', () => { const script = ` - import { withSourceDirLock } from ${JSON.stringify(join(import.meta.dirname, 'source-dir.ts'))} + import { withSourceDirLock } from ${JSON.stringify(join(import.meta.dirname, 'source-dir/index.ts'))} await withSourceDirLock(process.env.SOURCE_DIR, async () => { process.kill(process.pid, 'SIGINT') await new Promise(() => {}) @@ -159,7 +210,7 @@ describe('withSourceDirLock', () => { import { writeFileSync } from 'node:fs' import { join } from 'node:path' import { setTimeout as sleep } from 'node:timers/promises' - import { withSourceDirLock } from ${JSON.stringify(join(import.meta.dirname, 'source-dir.ts'))} + import { withSourceDirLock } from ${JSON.stringify(join(import.meta.dirname, 'source-dir/index.ts'))} import { deferInterrupts } from ${JSON.stringify(join(import.meta.dirname, '../../terminal/interrupts.ts'))} await withSourceDirLock(process.env.SOURCE_DIR, async () => { await deferInterrupts( diff --git a/packages/sim-cli/src/commands/protocol/source-dir.ts b/packages/sim-cli/src/commands/protocol/source-dir.ts deleted file mode 100644 index 4ddb7f39967..00000000000 --- a/packages/sim-cli/src/commands/protocol/source-dir.ts +++ /dev/null @@ -1,1114 +0,0 @@ -/** - * The source directory on disk, for `workflows source` (one workflow's directory) and - * `workspaces source` (whose `workflows//` folders are each one): its layout beside the - * printed files, and the only code that reads or writes it. One definition, so the commands - * cannot drift apart. - * - * Every path is relative to `realpath(--dir)` and no component below it may be a symbolic link, - * so nothing is read from or written to outside the directory, whatever a link in it points at. - * Writes are staged in the directory first and moved into place, so a failed write leaves the - * files it did not reach as they were. An embedded run reads and writes through its host instead. - */ - -import { createHash } from 'node:crypto' -import type { Dirent, Stats } from 'node:fs' -import { - lstat, - mkdir, - mkdtemp, - readdir, - readFile, - realpath, - rename, - rm, - rmdir, - stat, - writeFile, -} from 'node:fs/promises' -import { hostname, tmpdir } from 'node:os' -import { basename, dirname, join, posix, resolve } from 'node:path' -import { mapWithConcurrency } from '@sim/utils/concurrency' -import { isRecordLike, toRecord } from '@sim/utils/object' -import { compareStrings } from '@sim/utils/string' -import { codeFileReferences } from '@sim/workflow-code/code-files/references' -import { - isCodeFilePath, - SOURCE_BASE_FILE, - SOURCE_ENTRY_FILE, - SOURCE_CODE_DIR as SOURCE_FUNCTIONS_DIR, - SOURCE_LOCK_FILE, -} from '@sim/workflow-code/limits' -import { fingerprintOf, KIND_DIRS } from '@sim/workspace-code' -import type { Command } from 'commander' -import { printError } from '#sim-cli/output/io' -import type { ResolvedProfile } from '../../config/index' -import { embedStore } from '../../embed-context' -import { SOURCE_BUSY_EXIT_CODE } from '../../exit-codes' -import { SimApiError } from '../../http/client' -import { embeddedFileContent } from '../../transfer/local-file' -import { removeStagingOnSignal, saveToFile } from './files-get' - -/** Generated SDK declarations: the blocks the source uses (every block's are cached), plus `workspace.d.ts`. */ -export const SOURCE_SDK_DIR = '.sim/sdk' -/** A short guide for an agent working in the directory; never a top-level `AGENTS.md`. */ -export const SOURCE_AGENTS_FILE = '.sim/AGENTS.md' -/** - * Example workflows (`cookbook.ts`, `integrations.ts`, `agents.ts`): together one correct example - * of each common block, rewritten on every get like the agent guide. They name example resources - * the workspace may lack, so the written `tsconfig.json` never includes them. - */ -export const SOURCE_EXAMPLES_DIR = '.sim/examples' -/** Pins the source files to LF, merged into whatever the directory already holds. */ -export const SOURCE_GITATTRIBUTES_FILE = '.gitattributes' -/** Keeps a command's transient files out of a repository, merged like `.gitattributes`. */ -export const SOURCE_GITIGNORE_FILE = '.gitignore' -/** - * A digest of each source file as the CLI last wrote it, so a later get can tell an unapplied - * edit from a file it may replace. The CLI's own record: never sent to the server. - */ -export const SOURCE_WRITTEN_FILE = '.sim/written.json' -/** - * Machine-local cache the CLI keeps beside its records: the stat each blob digest was taken at, - * which names this machine's files (device, inode, times) and so is never committed. The folder - * ignores itself (`.sim/cache/.gitignore`), whatever the repository's own `.gitignore` says. - */ -const SOURCE_CACHE_DIR = '.sim/cache' -const SOURCE_STATS_FILE = `${SOURCE_CACHE_DIR}/stats.json` -const SOURCE_CACHE_IGNORE_FILE = `${SOURCE_CACHE_DIR}/.gitignore` -/** - * The workspace a starter written by `workflows source create` was typed for, so the create that - * follows targets that workspace. The CLI's own record: never sent, and removed once the workflow - * exists. - */ -export const SOURCE_STARTER_FILE = '.sim/starter.json' -/** - * The edges of the workflow a get or apply printed, by lock key, with the base fingerprint they - * belong to, so an offline check tells an edge the stored workflow holds from one the source - * adds. The CLI's own record: never sent to the server. - */ -export const SOURCE_STORED_EDGES_FILE = '.sim/stored-edges.json' -/** - * Where a workspace get keeps the blobs it verified until they move into place. A get that fails - * leaves them there, so the next one downloads only the rest; the one that completes removes the - * folder. - */ -export const SOURCE_DOWNLOADS_DIR = '.sim/downloads' -/** The folder a write stages its files in before moving them into place; removed when it ends. */ -export const SOURCE_STAGING_PREFIX = '.sim-staging-' - -/** - * Whether a server answer may write `path` into the source directory: the entry file, or a code - * file as {@link isCodeFilePath} (the rule the compiler applies to `codeFile()`) names one. - * Anything else (`.git/hooks/*`, `conftest.py`, a dotfile) is refused, whatever base URL answered. - */ -function isPrintedSourcePath(path: string): boolean { - return path === SOURCE_ENTRY_FILE || isCodeFilePath(path) -} - -/** Refuses a server answer naming a file the printer never writes, before anything is written. */ -export function assertPrintedSourcePaths(files: Record): void { - for (const path of Object.keys(files)) { - if (!isPrintedSourcePath(path)) { - throw new SimApiError(`The workflow source named an unsafe file path: ${path}`, 0) - } - } -} - -/** - * Refuses, before a write is sent, a code file the server would store but the CLI could never - * write back: one rule ({@link isCodeFilePath}) for both, so an apply is never stored and then - * reported as failed. - */ -export function assertSendableCodeFiles(files: Record): void { - const refused = Object.keys(files).filter((path) => !isPrintedSourcePath(path)) - if (refused.length === 0) return - throw new SimApiError( - `${refused.join(', ')} ${refused.length === 1 ? 'is not a code file path' : 'are not code file paths'} a workflow directory holds, so nothing was sent. Move the code to ${SOURCE_FUNCTIONS_DIR}/.js (or .py, .sh: letters, digits, _, $, - and dots, not starting with a dot) and update its codeFile().`, - 0, - 'E_CODE_FILE_PATH' - ) -} - -const GITATTRIBUTES_LINES = [ - `${SOURCE_ENTRY_FILE} text eol=lf`, - `${SOURCE_FUNCTIONS_DIR}/** text eol=lf`, -] -/** Files read or written at once on a local disk; a catalog's declarations run to hundreds of small files. */ -export const FILE_CONCURRENCY = 32 - -/** - * A path inside the source directory, normalized (`./functions/x.js` → `functions/x.js`), or - * `null` for an absolute path, a drive letter or a `..` escape. - */ -export function sourceRelativePath(path: string): string | null { - const normalized = posix.normalize(path.replace(/\\/g, '/')) - if ( - normalized.startsWith('/') || - /^[A-Za-z]:/.test(normalized) || - normalized === '.' || - normalized === '..' || - normalized.startsWith('../') - ) { - return null - } - return normalized.startsWith('./') ? normalized.slice(2) : normalized -} - -/** Every `codeFile` a lock records, as source-relative paths. */ -export function lockCodeFiles(lock: unknown): Set { - const paths = new Set() - const blocks = toRecord(toRecord(lock).blocks) - for (const block of Object.values(blocks)) { - const path = - isRecordLike(block) && typeof block.codeFile === 'string' - ? sourceRelativePath(block.codeFile) - : null - if (path) paths.add(path) - } - return paths -} - -/** - * The function files a write leaves behind: named by the lock the directory held (so the CLI - * wrote them), no longer named by the new lock, and not being written now. A file the person - * added and no lock ever named is never pruned. - */ -export function orphanedCodeFiles( - previousLock: unknown, - nextLock: unknown, - written: Iterable -): string[] { - const kept = lockCodeFiles(nextLock) - for (const path of written) kept.add(path) - return [...lockCodeFiles(previousLock)].filter( - (path) => path.startsWith(`${SOURCE_FUNCTIONS_DIR}/`) && !kept.has(path) - ) -} - -/** - * The lock an earlier get, apply or create left, read leniently (missing or invalid is `null`): - * it names the function files the CLI wrote, which is all a prune needs from it. - */ -export function readPreviousLock(dir: string): Promise { - return readSourceJson(dir, SOURCE_LOCK_FILE) -} - -/** A JSON file in the source directory, read leniently: `null` when it is missing or invalid. */ -export async function readSourceJson(dir: string, path: string): Promise { - const text = await readSourceText(dir, path) - if (text === null) return null - try { - return JSON.parse(text) - } catch { - return null - } -} - -/** - * Makes `command`'s `--dir` (given or defaulted) an absolute path before its action runs, so every - * message names the directory it read wherever the CLI ran: a wrapper that changes directory - * first made `.` name a folder its caller never meant. An embedded run keeps its host's path. - */ -export function absoluteDirOption(command: Command): Command { - return command.hook('preAction', (_hooked, action) => { - const dir: unknown = action.getOptionValue('dir') - if (typeof dir === 'string' && !embedStore.getStore()) { - action.setOptionValue('dir', resolve(dir)) - } - }) -} - -/** - * The workspace a source command acts on for a directory read from `readFrom`: that workspace, - * whatever the profile's default, so a directory stays bound to where it came from. Only a - * `--workspace` naming another one is refused, with `next` saying what to run instead (given the - * workspace the directory was read from and the one named). A directory that records no workspace - * takes the profile's (`null` when none is selected). - */ -export function directoryWorkspace( - dir: string, - readFrom: string | null, - profile: Pick, - next: (readFrom: string, named: string) => string -): string | null { - const named = profile.workspaceId - if (readFrom === null) return named - if (profile.sources.workspaceId !== 'flag' || named === null || named === readFrom) { - return readFrom - } - throw new SimApiError( - `${dir} was read from workspace ${readFrom}, but --workspace names ${named}. ${next(readFrom, named)}`, - 0, - 'SOURCE_OTHER_WORKSPACE' - ) -} - -/** Which source command family a directory belongs to: one workflow's, or a whole workspace's. */ -export type SourceDirKind = 'workflow' | 'workspace' - -/** - * What `dir` holds, by its `.sim/base.json` (read leniently, as an apply reads it: a base the - * parser refuses still names what it is): a workflow base names its `workflowId`, a workspace base - * only its `workspaceId`. Without a base, a `workflow.ts` or a starter record at the root is a - * workflow directory (a workspace directory keeps its workflows under `workflows//`). `null` - * for a directory that holds neither, such as a new one. - */ -async function sourceDirKind( - dir: string -): Promise<{ kind: SourceDirKind; readFrom: string | null } | null> { - const base = toRecord(await readSourceJson(dir, SOURCE_BASE_FILE)) - if (typeof base.workflowId === 'string') return { kind: 'workflow', readFrom: base.workflowId } - if (typeof base.workspaceId === 'string') { - return { kind: 'workspace', readFrom: base.workspaceId } - } - const starter = await readStarterWorkspace(dir) - if (starter !== null || (await readSourceText(dir, SOURCE_ENTRY_FILE)) !== null) { - return { kind: 'workflow', readFrom: null } - } - return null -} - -/** - * Refuses a `kind` command on a directory of the other kind, whatever `--force` says: a workflow - * get into a workspace directory (or the reverse) would replace its base, so staleness could no - * longer be detected and its unapplied edits would sit in a directory neither command reads. - */ -export async function assertSourceDirKind(dir: string, kind: SourceDirKind): Promise { - const held = await sourceDirKind(dir) - if (held === null || held.kind === kind) return - const message = - held.kind === 'workspace' - ? `${dir} is a workspace source directory${held.readFrom ? ` (workspace ${held.readFrom})` : ''}, so a \`sim workflows source\` command would replace its base. Edit a workflow of it in ${dir}/${KIND_DIRS.workflow}/ with \`sim workspaces source\` (check, status, apply) --dir ${dir}, or get the workflow into another directory.` - : `${dir} is a workflow source directory${held.readFrom ? ` (workflow ${held.readFrom})` : ''}, so a \`sim workspaces source\` command would replace its base. Use \`sim workflows source\` (get, check, status, apply) --dir ${dir}, or get the workspace into another directory.` - throw new SimApiError(message, 0, 'SOURCE_DIR_KIND_MISMATCH') -} - -/** The workspace {@link SOURCE_STARTER_FILE} names, or `null` when the directory holds no starter. */ -export async function readStarterWorkspace(dir: string): Promise { - const workspaceId = toRecord(await readSourceJson(dir, SOURCE_STARTER_FILE)).workspaceId - return typeof workspaceId === 'string' ? workspaceId : null -} - -/** - * A line file (`.gitattributes`, `.gitignore`) with `lines` appended where missing, or `null` - * when it has them all. - */ -export function withMissingLines( - existing: string | null, - lines: readonly string[] = GITATTRIBUTES_LINES -): string | null { - const present = new Set((existing ?? '').split('\n').map((line) => line.trim())) - const missing = lines.filter((line) => !present.has(line)) - if (missing.length === 0) return null - const prefix = existing && !existing.endsWith('\n') ? `${existing}\n` : (existing ?? '') - return `${prefix}${missing.join('\n')}\n` -} - -/** Whether a file-system error means the path does not exist (`ENOENT`, or a file as a folder). */ -export function isMissing(error: unknown): boolean { - const code = errorCode(error) - return code === 'ENOENT' || code === 'ENOTDIR' -} - -/** - * The absolute path of `path` under `root` (already a realpath), refusing a symbolic link at any - * component that exists. `verified` caches directories already checked across one operation. - */ -export async function checkedPath( - root: string, - path: string, - verified: Set -): Promise { - const segments = path.split('/') - let current = root - for (const [index, segment] of segments.entries()) { - current = join(current, segment) - if (verified.has(current)) continue - let stats: Awaited> - try { - stats = await lstat(current) - } catch (error) { - if (isMissing(error)) return join(root, ...segments) - throw error - } - if (stats.isSymbolicLink()) { - throw new SimApiError( - `Refusing ${segments.slice(0, index + 1).join('/')}: it is a symbolic link, and the source directory only holds plain files.`, - 0, - 'SOURCE_SYMLINK' - ) - } - if (index < segments.length - 1) verified.add(current) - } - return current -} - -async function realRoot(dir: string): Promise { - try { - return await realpath(dir) - } catch (error) { - if (isMissing(error)) return null - throw error - } -} - -/** - * Source text as the CLI reads and compares it: no leading byte-order mark and LF line endings, so - * a Windows editor or checkout never turns an unchanged source into an edit, or hides its - * `// @sim-sdk` header. - */ -function normalizeSourceText(text: string): string { - return text.replace(/^\uFEFF/, '').replace(/\r\n/g, '\n') -} - -/** - * The text of `path` in the source directory, normalized by `normalizeSourceText`, or `null` when - * it does not exist. A symbolic link anywhere below the directory is refused rather than followed. - */ -export async function readSourceText(dir: string, path: string): Promise { - const embedded = embedStore.getStore() - let text: string - if (embedded) { - try { - const content = await embeddedFileContent(embedded, posix.join(dir, path)) - text = typeof content === 'string' ? content : new TextDecoder().decode(content) - } catch { - return null - } - } else { - const root = await realRoot(dir) - if (root === null) return null - const target = await checkedPath(root, path, new Set()) - try { - text = await readFile(target, 'utf8') - } catch (error) { - if (isMissing(error)) return null - throw new SimApiError(`Cannot read ${join(dir, path)}: ${(error as Error).message}`, 0) - } - } - return normalizeSourceText(text) -} - -/** What one write does to the source directory: files to remove first, then files to write. */ -export interface SourceDirChange { - write: ReadonlyArray - /** - * Files whose bytes `stage` writes into the staged file it is handed (a download), before - * anything is moved into place: one that throws leaves the directory as it was. - */ - fetch?: ReadonlyArray Promise]> - /** Files or directories (removed recursively); an embedded run cannot remove, so skips these. */ - remove?: readonly string[] -} - -/** - * The files that record what the directory was read from: each workflow's lock, base, - * `.sim/written.json` and stored edges, and the workspace base. They move into place only after every other file, - * so a write cut short leaves the old records beside whatever text did land: the next apply is - * refused as stale, never sent as if the old text were current. - */ -const RECORD_FILES = [ - SOURCE_BASE_FILE, - SOURCE_LOCK_FILE, - SOURCE_WRITTEN_FILE, - SOURCE_STORED_EDGES_FILE, -] - -function isRecordFile(path: string): boolean { - return RECORD_FILES.some((file) => path === file || path.endsWith(`/${file}`)) -} - -/** - * Applies a change to the source directory, creating it when absent. Every target is checked - * before anything is touched; the files are written to a staging directory inside it, then the - * removals run and each file is renamed into place, never through a link, the record files last. - */ -export async function updateSourceDir(dir: string, change: SourceDirChange): Promise { - const fetches = change.fetch ?? [] - const writes = [...change.write.map(([path]) => path), ...fetches.map(([path]) => path)] - for (const path of [...writes, ...(change.remove ?? [])]) { - if (sourceRelativePath(path) !== path) { - throw new SimApiError(`Refusing to write ${path} outside ${dir}`, 0) - } - } - if (embedStore.getStore()) { - const fetched: Array]> = [] - if (fetches.length > 0) { - const scratch = await mkdtemp(join(tmpdir(), 'sim-source-fetch-')) - try { - for (const [path, stage] of fetches) { - const file = join(scratch, String(fetched.length)) - await stage(file) - fetched.push([path, new Uint8Array(await readFile(file))]) - } - } finally { - await rm(scratch, { recursive: true, force: true }) - } - } - const all = [...change.write, ...fetched] - // The host has no rename, so the records go last here too: after every file and blob. - for (const [path, content] of [ - ...all.filter(([path]) => !isRecordFile(path)), - ...all.filter(([path]) => isRecordFile(path)), - ]) { - await saveToFile(new Blob([content]).stream(), posix.join(dir, path), true) - } - return - } - - await mkdir(dir, { recursive: true }) - const root = await realpath(dir) - const verified = new Set() - const targets: string[] = [] - for (const path of writes) targets.push(await checkedPath(root, path, verified)) - const removals: string[] = [] - for (const path of change.remove ?? []) removals.push(await checkedPath(root, path, verified)) - - let staging: string | null = null - const disposeSignalCleanup = removeStagingOnSignal(() => staging) - try { - const stagingDir = await mkdtemp(join(root, SOURCE_STAGING_PREFIX)) - staging = stagingDir - const indexes = writes.map((_, index) => index) - const staged = (index: number) => join(stagingDir, String(index)) - await mapWithConcurrency(indexes, FILE_CONCURRENCY, (index) => - index < change.write.length - ? writeFile(staged(index), change.write[index][1], { flag: 'wx' }) - : fetches[index - change.write.length][1](staged(index)) - ) - for (const target of removals) await rm(target, { recursive: true, force: true }) - const parents = new Map>() - const moveIntoPlace = async (index: number) => { - const parent = dirname(targets[index]) - let created = parents.get(parent) - if (!created) { - created = mkdir(parent, { recursive: true }) - parents.set(parent, created) - } - await created - await rename(staged(index), targets[index]) - } - await mapWithConcurrency( - indexes.filter((index) => !isRecordFile(writes[index])), - FILE_CONCURRENCY, - moveIntoPlace - ) - await mapWithConcurrency( - indexes.filter((index) => isRecordFile(writes[index])), - FILE_CONCURRENCY, - moveIntoPlace - ) - } finally { - if (staging) await rm(staging, { recursive: true, force: true }) - disposeSignalCleanup() - } -} - -/** - * What identifies a file's bytes without reading them, taken when its digest was: the file (device - * and inode), its size, and its modification and change times. The change time is the one a user - * tool cannot set back (`cp -p`, `rsync -t` and `touch -d` restore only the modification time), so - * any write since the digest moves it even when the size and modification time match. - */ -export interface FileStat { - dev: number - ino: number - size: number - mtimeMs: number - ctimeMs: number -} - -/** The {@link FileStat} of a `stat` result. */ -export function fileStatOf(stats: FileStat): FileStat { - const { dev, ino, size, mtimeMs, ctimeMs } = stats - return { dev, ino, size, mtimeMs, ctimeMs } -} - -/** Whether two stats name the same, unchanged bytes. */ -export function sameFileStat(a: FileStat, b: FileStat): boolean { - return ( - a.dev === b.dev && - a.ino === b.ino && - a.size === b.size && - a.mtimeMs === b.mtimeMs && - a.ctimeMs === b.ctimeMs - ) -} - -/** A blob's SHA-256 with the stat of the file it was read from. */ -export interface StatDigest extends FileStat { - sha256: string -} - -/** - * `.sim/written.json`: each source file's digest as the CLI last wrote it, normalized, sorted by - * path. It holds nothing about this machine, so a clone reads it the same and a write that changes - * no file leaves it byte for byte. - */ -interface WrittenRecord { - version: 1 - files: Record -} - -/** - * `.sim/cache/stats.json`: for a blob, the stat its digest was taken at, beside that digest, so a - * later command on this machine reuses the digest of an untouched blob instead of reading it - * again. A stat whose digest is not the one `.sim/written.json` records is ignored. - */ -interface StatsRecord { - version: 1 - stats: Record -} - -/** - * What the CLI last wrote, as one command reads it, once: the digests from `.sim/written.json` - * and the stats from `.sim/cache/stats.json` that still match them. - */ -export interface Written { - digests: Record - stats: Record -} - -/** The digest of `text` as `readSourceText` would read it back. */ -export function digest(text: string): string { - return createHash('sha256').update(normalizeSourceText(text), 'utf8').digest('hex') -} - -/** - * What `.sim/written.json` records for `files` as they are about to be written, keeping the `kept` - * digests of files this write leaves as they are. A blob's stat is recorded only beside the digest - * it was taken with. - */ -export function writtenRecord( - files: Record, - kept: Record = {}, - stats: ReadonlyMap = new Map() -): Written { - const digests: Record = { - ...kept, - ...Object.fromEntries(Object.entries(files).map(([path, text]) => [path, digest(text)])), - } - const recorded = [...stats].flatMap(([path, { sha256, ...stat }]) => - digests[path] === sha256 ? [[path, fileStatOf(stat)] as const] : [] - ) - return { digests, stats: Object.fromEntries(recorded) } -} - -/** `record` with its keys in path order, so the same entries always print the same text. */ -function sortedByPath(record: Record): Record { - return Object.fromEntries(Object.entries(record).sort(([a], [b]) => compareStrings(a, b))) -} - -/** - * The files that record `written`: `.sim/written.json` (committed) and, when a blob stat is - * recorded, the machine-local `.sim/cache/stats.json` with the cache folder's own `.gitignore`. - */ -export function writtenFiles(written: Written): Array { - const record: WrittenRecord = { version: 1, files: sortedByPath(written.digests) } - const files: Array = [ - [SOURCE_WRITTEN_FILE, `${JSON.stringify(record, null, 2)}\n`], - ] - const stats = Object.entries(written.stats).flatMap(([path, stat]) => - Object.hasOwn(written.digests, path) - ? [[path, { sha256: written.digests[path], ...fileStatOf(stat) }] as const] - : [] - ) - if (stats.length === 0) return files - const cache: StatsRecord = { version: 1, stats: sortedByPath(Object.fromEntries(stats)) } - return [ - ...files, - [SOURCE_STATS_FILE, `${JSON.stringify(cache, null, 2)}\n`], - [SOURCE_CACHE_IGNORE_FILE, '*\n'], - ] -} - -/** What the CLI recorded for its last write, empty when it recorded nothing. */ -export async function readWritten(dir: string): Promise { - const record = toRecord(await readSourceJson(dir, SOURCE_WRITTEN_FILE)) - const digests = Object.fromEntries( - Object.entries(toRecord(record.files)).filter( - (entry): entry is [string, string] => typeof entry[1] === 'string' - ) - ) - const cache = toRecord(await readSourceJson(dir, SOURCE_STATS_FILE)) - const stats: Record = {} - for (const [path, stat] of Object.entries(toRecord(cache.stats))) { - const { sha256, dev, ino, size, mtimeMs, ctimeMs } = toRecord(stat) - if ( - typeof sha256 === 'string' && - Object.hasOwn(digests, path) && - digests[path] === sha256 && - typeof dev === 'number' && - typeof ino === 'number' && - typeof size === 'number' && - typeof mtimeMs === 'number' && - typeof ctimeMs === 'number' - ) { - stats[path] = { dev, ino, size, mtimeMs, ctimeMs } - } - } - return { digests, stats } -} - -/** The blob digests `written` holds with their stats, to carry into the next record. */ -export function recordedStats(written: Written): Map { - return new Map( - Object.entries(written.stats).flatMap(([path, stat]) => - Object.hasOwn(written.digests, path) - ? [[path, { ...stat, sha256: written.digests[path] }] as const] - : [] - ) - ) -} - -/** - * The files among `touched` (path → the text about to be written, or `null` for a removal) whose - * text on disk is neither what the CLI last wrote there nor what it is about to write: a write - * would destroy an edit no apply has saved. A file the CLI has no record of counts as edited, so a - * hand-written file is never replaced silently either; so does one the CLI wrote that is missing - * now and would be written again, a delete no apply has sent. - */ -export async function editedFiles( - dir: string, - touched: ReadonlyMap, - written: Written['digests'] -): Promise { - const entries = [...touched] - const texts = await mapWithConcurrency(entries, FILE_CONCURRENCY, ([path]) => - readSourceText(dir, path) - ) - return entries.flatMap(([path, next], index) => { - const current = texts[index] - if (current === null) return next !== null && Object.hasOwn(written, path) ? [path] : [] - if (next !== null && current === normalizeSourceText(next)) return [] - return written[path] !== digest(current) ? [path] : [] - }) -} - -/** - * The source files a write of `answer` would destroy edits in: a file it replaces or prunes whose - * text on disk is neither what the CLI last wrote there nor what it is about to write. - */ -export async function locallyEditedFiles( - dir: string, - answer: Pick, - previousLock: unknown -): Promise { - const touched = new Map(Object.entries(answer.files)) - for (const path of orphanedCodeFiles(previousLock, answer.lock, touched.keys())) { - touched.set(path, null) - } - return editedFiles(dir, touched, (await readFolderWritten(dir)).digests) -} - -/** - * A `workflows/` folder of a workspace directory: `root` holds a workspace `.sim/base.json`. - * The workspace's `.sim/written.json` and base are the folder's records too, so a single-workflow - * write keeps them instead of writing a record of its own the workspace commands never read. - */ -export interface WorkspaceFolder { - root: string - workspaceId: string - /** The folder's key under `workflows/`. */ - key: string -} - -/** The workspace directory `dir` is a workflow folder of, or `null`. */ -export async function workspaceFolderOf(dir: string): Promise { - const folder = resolve(dir) - if (basename(dirname(folder)) !== KIND_DIRS.workflow) return null - const root = dirname(dirname(folder)) - const base = toRecord(await readSourceJson(root, SOURCE_BASE_FILE)) - if (typeof base.workspaceId !== 'string' || base.workflowId !== undefined) return null - return { root, workspaceId: base.workspaceId, key: basename(folder) } -} - -/** `path` of the workflow folder `key`, relative to its workspace directory. */ -function inWorkspace(key: string, path: string): string { - return `${KIND_DIRS.workflow}/${key}/${path}` -} - -/** - * What the CLI last wrote into the workflow directory `dir`: its own `.sim/written.json`, or for a - * workflow folder of a workspace directory, the workspace record's entries for the folder (its - * `.sim/` records left out). A write into such a folder records there and never in the folder, but - * a CLI from before that left a record of the folder's own. That record wins only when it is the - * later one: when the workspace record's digest of the folder's `.sim/base.json` is not the base on - * disk, a single-workflow write replaced the base after the workspace record was taken. Otherwise - * the folder's own record is stale and ignored. - */ -export async function readFolderWritten(dir: string): Promise { - const own = await readWritten(dir) - const folder = await workspaceFolderOf(dir) - if (!folder) return own - const record = await readWritten(folder.root) - if (Object.keys(own.digests).length > 0) { - const base = await readSourceText(dir, SOURCE_BASE_FILE) - const recorded = record.digests[inWorkspace(folder.key, SOURCE_BASE_FILE)] - if (base !== null && recorded !== digest(base)) return own - } - const prefix = inWorkspace(folder.key, '') - const inFolder = (entries: Record) => - Object.fromEntries( - Object.entries(entries).flatMap(([path, value]) => { - const relative = path.startsWith(prefix) ? path.slice(prefix.length) : null - return relative !== null && !relative.startsWith('.sim/') ? [[relative, value]] : [] - }) - ) - return { digests: inFolder(record.digests), stats: inFolder(record.stats) } -} - -/** - * Records a single-workflow write into workflow folder `folder` in the workspace records: each - * written file's digest (`written`, folder-relative path → text) replaces the one before, a - * `removed` file's is dropped, and the workspace base's entry for the workflow takes the - * fingerprint of the folder's new base, as a workspace get would record it. Written after the - * folder's files, so a write cut short leaves the old records. - */ -async function recordFolderWrite( - folder: WorkspaceFolder, - written: Record, - removed: readonly string[], - base: unknown -): Promise { - const record = await readWritten(folder.root) - const dropped = new Set(removed.map((path) => inWorkspace(folder.key, path))) - const digests = { - ...Object.fromEntries(Object.entries(record.digests).filter(([path]) => !dropped.has(path))), - ...Object.fromEntries( - Object.entries(written).map(([path, text]) => [inWorkspace(folder.key, path), digest(text)]) - ), - } - const write: Array = [ - ...writtenFiles({ digests, stats: record.stats }), - ] - const workspaceBase = toRecord(await readSourceJson(folder.root, SOURCE_BASE_FILE)) - const resources = toRecord(workspaceBase.resources) - const address = `workflow:${folder.key}` - const entry = toRecord(resources[address]) - const { workflowId, fingerprint } = toRecord(base) - if (entry.id === workflowId && typeof fingerprint === 'string') { - const next = { - ...workspaceBase, - resources: { - ...resources, - [address]: { ...entry, fingerprint: fingerprintOf({ source: fingerprint }) }, - }, - } - write.push([SOURCE_BASE_FILE, `${JSON.stringify(next, null, 2)}\n`]) - } - await updateSourceDir(folder.root, { write }) -} - -/** - * Every plain file under `folder` in the source directory, as source-relative paths. A symbolic - * link anywhere is refused rather than followed; a missing folder lists nothing. - */ -export async function listSourceFiles(dir: string, folder: string): Promise { - if (embedStore.getStore()) { - throw new SimApiError('Workspace source directories need a local disk.', 0) - } - const root = await realRoot(dir) - if (root === null) return [] - const start = await checkedPath(root, folder, new Set()) - const files: string[] = [] - const walk = async (absolute: string, relative: string): Promise => { - let entries: Dirent[] - try { - entries = await readdir(absolute, { withFileTypes: true, encoding: 'utf8' }) - } catch (error) { - if (isMissing(error)) return - throw error - } - for (const entry of entries) { - const path = `${relative}/${entry.name}` - if (entry.isSymbolicLink()) { - throw new SimApiError( - `Refusing ${path}: it is a symbolic link, and the source directory only holds plain files.`, - 0, - 'SOURCE_SYMLINK' - ) - } - if (entry.isDirectory()) await walk(join(absolute, entry.name), path) - else if (entry.isFile()) files.push(path) - } - } - await walk(start, folder) - return files -} - -/** What the server answers a get, apply or create with: the printed files, lock and base. */ -interface PrintedSourceAnswer { - files: Record - lock: unknown - base: unknown -} - -/** - * The files of a stored write's answer the CLI may write: every one {@link isPrintedSourcePath} - * accepts. The write is already stored, so a path refused here is left as it is on disk (it is - * the author's own file, which the server echoed) with a warning, never thrown: the lock and base - * are still written, so the directory matches the stored workflow. - */ -function writableFiles(files: Record): Record { - const refused = Object.keys(files).filter((path) => !isPrintedSourcePath(path)) - if (refused.length === 0) return files - printError( - `warning: the answer named ${refused.join(', ')}, which the CLI never writes; the write was saved and the lock and base were written, but ${refused.length === 1 ? 'that file was' : 'those files were'} left as ${refused.length === 1 ? 'it is' : 'they are'} on disk.` - ) - return Object.fromEntries(Object.entries(files).filter(([path]) => isPrintedSourcePath(path))) -} - -/** - * Writes a server answer into the source directory in one staged update: its files, lock and base, - * plus `extra`, pruning the function files the previous lock named and the new one no longer does. - * Returns how many files it wrote. A get refuses an unsafe path first - * ({@link assertPrintedSourcePaths}); after a stored write one is skipped ({@link writableFiles}). - */ -export async function writeBack( - dir: string, - answer: PrintedSourceAnswer, - previousLock: unknown, - extra: SourceDirChange = { write: [] } -): Promise { - const files = writableFiles(answer.files) - const folder = await workspaceFolderOf(dir) - const printed: Array = [ - ...Object.entries(files), - [SOURCE_LOCK_FILE, `${JSON.stringify(answer.lock, null, 2)}\n`], - [SOURCE_BASE_FILE, `${JSON.stringify(answer.base, null, 2)}\n`], - ] - const write = [...printed, ...(folder ? [] : writtenFiles(writtenRecord(files))), ...extra.write] - const orphaned = orphanedCodeFiles(previousLock, answer.lock, Object.keys(answer.files)) - const unnamed = await unnamedCodeFiles(dir, previousLock, answer, orphaned) - await updateSourceDir(dir, { - write, - remove: [ - ...orphaned, - ...(extra.remove ?? []), - // A record of the folder's own, from before, would shadow the workspace's. - ...(folder ? [SOURCE_WRITTEN_FILE] : []), - ], - }) - if (folder) await recordFolderWrite(folder, Object.fromEntries(printed), orphaned, answer.base) - if (unnamed.length > 0) { - printError( - `warning: ${unnamed.join(', ')} ${unnamed.length === 1 ? 'is' : 'are'} no longer named by ${SOURCE_ENTRY_FILE} (the printed source names no code file there, such as code now printed as stored()), so ${unnamed.length === 1 ? 'it was' : 'they were'} left on disk as written: delete ${unnamed.length === 1 ? 'it' : 'them'}, or name ${unnamed.length === 1 ? 'it' : 'them'} again with codeFile(). Never commit one that holds a credential.` - ) - } - return write.length -} - -/** - * The code files a write leaves on disk although the printed source no longer names them: named - * by the entry file on disk before the write or by the previous lock, neither written nor pruned - * now, and still present. A file the person added is never deleted, so the caller says where it - * is instead: one that held a literal secret now printed as `stored()` must not be committed. - */ -async function unnamedCodeFiles( - dir: string, - previousLock: unknown, - answer: PrintedSourceAnswer, - pruned: readonly string[] -): Promise { - const entry = await readSourceText(dir, SOURCE_ENTRY_FILE) - const named = lockCodeFiles(previousLock) - for (const literal of entry === null ? [] : codeFileReferences(entry)) { - const path = sourceRelativePath(literal) - if (path !== null) named.add(path) - } - const kept = new Set([...Object.keys(answer.files), ...lockCodeFiles(answer.lock), ...pruned]) - const candidates = [...named].filter((path) => isCodeFilePath(path) && !kept.has(path)) - const present = await Promise.all( - candidates.map(async (path) => (await readSourceText(dir, path)) !== null) - ) - return candidates.filter((_, index) => present[index]).sort(compareStrings) -} - -/** - * Held while a command reads and rewrites a source directory, so two commands on one directory - * (an agent and a person, two terminals) never interleave their writes: each file moves into - * place atomically, but a set of them does not, and records from one command beside text from - * another would send stale text as an edit. A folder, since creating one is atomic everywhere. - */ -export const SOURCE_DIR_LOCK = '.sim/command.lock' - -/** The help line every command that holds {@link SOURCE_DIR_LOCK} adds to its exit codes. */ -export const SOURCE_DIR_BUSY_HELP = ` -Exit 10 (busy, retry): another sim workspaces source command is using this directory -(SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.` - -/** The file-system codes of a directory the user may not write: its lock cannot be taken. */ -const UNWRITABLE_CODES: ReadonlySet = new Set(['EACCES', 'EPERM', 'EROFS']) - -/** A lock whose owner never recorded itself is stale after this long (it died while taking it). */ -const UNRECORDED_LOCK_STALE_MS = 60_000 - -/** The file in the lock folder that records its owner. */ -const LOCK_OWNER_FILE = 'owner.json' -/** The folder a command creates in a lock it found stale while it takes that lock over. */ -const LOCK_TAKEOVER_DIR = 'takeover' - -/** Who holds the lock, as its owner file records it. */ -interface LockOwner { - pid: number - host: string -} - -/** A file-system or process error's code (`EEXIST`, `ESRCH`), if it has one. */ -function errorCode(error: unknown): string | undefined { - return isRecordLike(error) && typeof error.code === 'string' ? error.code : undefined -} - -function isAlive(pid: number): boolean { - try { - process.kill(pid, 0) - return true - } catch (error) { - return errorCode(error) === 'EPERM' - } -} - -/** The lock folder a command found, as it found it: which folder, and when it last changed. */ -type LockSighting = Pick - -/** - * Whether the lock at `lock`, as `seen`, was left by a command that is gone: its process on this - * machine no longer runs, or it never recorded an owner and had not changed for a minute when it - * was seen. The age is the sighting's, not the folder's now, because a takeover's own claim - * changes the folder. A lock another machine holds (a shared folder) is never taken over. - */ -async function isStaleLock(lock: string, seen: LockSighting): Promise { - const owner = toRecord( - await readFile(join(lock, LOCK_OWNER_FILE), 'utf8').then( - (text): unknown => JSON.parse(text), - () => null - ) - ) - if (typeof owner.pid === 'number' && typeof owner.host === 'string') { - return owner.host === hostname() && !isAlive(owner.pid) - } - return Date.now() - seen.mtimeMs > UNRECORDED_LOCK_STALE_MS -} - -/** - * Removes the lock at `lock`, which the caller found stale as `seen`, unless another command took - * it over first. Two commands can find one lock stale at once: one removes it and takes the lock, - * and the other, acting on what it saw before, must not remove the live lock that replaced it. So - * a takeover first claims the lock folder it means to remove (creating {@link LOCK_TAKEOVER_DIR} - * in it, which only one command can), then removes it only while it is still the folder it saw - * and still stale; a lock that replaced the stale one is another folder or records a live owner, - * so the late command leaves it. A command that died mid-takeover leaves its claim in a stale - * lock, which the busy message says to remove. - */ -export async function takeOverStaleLock(lock: string, seen: LockSighting): Promise { - const claim = join(lock, LOCK_TAKEOVER_DIR) - try { - await mkdir(claim) - } catch (error) { - if (errorCode(error) === 'EEXIST' || isMissing(error)) return - throw error - } - const now = await stat(lock).catch(() => null) - const sameFolder = now !== null && now.dev === seen.dev && now.ino === seen.ino - if (sameFolder && (await isStaleLock(lock, seen))) - await rm(lock, { recursive: true, force: true }) - else await rmdir(claim).catch(() => {}) -} - -/** - * Runs `run` holding the directory's command lock ({@link SOURCE_DIR_LOCK}). A second command on - * the directory meanwhile is refused (`SOURCE_DIR_BUSY`, exit 10) with nothing read or written; a - * lock its command left behind when it died is taken over ({@link takeOverStaleLock}). Like every - * other source path, `.sim` and the lock are never followed through a symbolic link - * (`SOURCE_SYMLINK`). An embedded run has no local directory to lock. - */ -export async function withSourceDirLock( - dir: string, - run: () => Promise, - options: { readOnly?: boolean } = {} -): Promise { - if (embedStore.getStore()) return run() - const absent = async (path: string) => - lstat(path).then( - () => false, - () => true - ) - const createdDir = await absent(dir) - let root: string - let records: string - let createdRecords: boolean - let lock: string - let held = false - try { - await mkdir(dir, { recursive: true }) - root = await realpath(dir) - records = await checkedPath(root, '.sim', new Set()) - createdRecords = createdDir || (await absent(records)) - await mkdir(records).catch((error: unknown) => { - if (errorCode(error) !== 'EEXIST') throw error - }) - lock = await checkedPath(root, SOURCE_DIR_LOCK, new Set()) - for (let attempt = 0; attempt < 2 && !held; attempt += 1) { - try { - await mkdir(lock) - held = true - } catch (error) { - if (errorCode(error) !== 'EEXIST') throw error - await checkedPath(root, SOURCE_DIR_LOCK, new Set()) - const seen = attempt === 0 ? await stat(lock).catch(() => null) : null - if (seen && (await isStaleLock(lock, seen))) await takeOverStaleLock(lock, seen) - } - } - } catch (error) { - if (!UNWRITABLE_CODES.has(errorCode(error) ?? '')) throw error - // Nothing can rewrite a directory no one may write, so a command that only reads it needs - // no lock; one that writes it is refused with the reason instead of a stack trace. - if (options.readOnly) return run() - throw new SimApiError( - `${dir} cannot be written (${errorCode(error)}), and this command writes it; nothing was read or written. Make it writable, or run the command on a writable copy.`, - 0, - 'SOURCE_DIR_UNWRITABLE' - ) - } - if (!held) { - throw new SimApiError( - `Another sim command is using ${dir}; nothing was read or written. Run this one again once it finished. If none is running, remove ${join(dir, SOURCE_DIR_LOCK)}.`, - 0, - 'SOURCE_DIR_BUSY', - undefined, - SOURCE_BUSY_EXIT_CODE - ) - } - const owner: LockOwner = { pid: process.pid, host: hostname() } - // An interrupt (Ctrl-C) releases the lock too: a lock left behind is taken over only by a - // command on the same host, so one left on a folder shared with a container would stay. - const disposeSignalRelease = removeStagingOnSignal(() => lock) - try { - await writeFile(join(lock, LOCK_OWNER_FILE), JSON.stringify(owner)) - return await run() - } finally { - disposeSignalRelease() - await rm(await checkedPath(root, SOURCE_DIR_LOCK, new Set()), { recursive: true, force: true }) - // Folders the lock alone created go with it; one the command filled stays. - if (createdRecords) await rmdir(records).catch(() => {}) - if (createdDir) await rmdir(dir).catch(() => {}) - } -} - -/** - * Runs a single-workflow command on `dir` holding the command lock of the workspace directory it - * is a workflow folder of, if any: such a command rewrites the workspace's own records (see - * {@link writeBack}), so it must not interleave with a workspace command, or with another workflow - * command in a sibling folder. Refused like a workspace command (`SOURCE_DIR_BUSY`, exit 10), with - * nothing read or written. - */ -export async function withWorkspaceFolderLock(dir: string, run: () => Promise): Promise { - const folder = await workspaceFolderOf(dir) - return folder ? withSourceDirLock(folder.root, run) : run() -} - -/** The help line every single-workflow command that writes a directory adds to its exit codes. */ -export const WORKSPACE_FOLDER_BUSY_HELP = ` -Exit 10 (busy, retry): --dir is a workflow folder of a workspace directory another sim command is -using (SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.` diff --git a/packages/sim-cli/src/commands/protocol/source-dir/command-lock.ts b/packages/sim-cli/src/commands/protocol/source-dir/command-lock.ts new file mode 100644 index 00000000000..3716ccc37b0 --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-dir/command-lock.ts @@ -0,0 +1,227 @@ +/** + * The command lock of a source directory (`.sim/command.lock`): held while a command reads and + * rewrites the directory, so two commands on it never interleave their writes. + */ + +import type { Stats } from 'node:fs' +import { lstat, mkdir, readFile, realpath, rm, rmdir, stat, writeFile } from 'node:fs/promises' +import { hostname } from 'node:os' +import { dirname, join, resolve } from 'node:path' +import { toRecord } from '@sim/utils/object' +import { embedStore } from '../../../embed-context' +import { SOURCE_BUSY_EXIT_CODE } from '../../../exit-codes' +import { removeStagingOnSignal } from '../files-get' +import { SourceDirError } from '../local-errors' +import { workspaceFolderOf } from './dir-kind' +import { checkedPath, errorCode, isMissing } from './files' + +/** + * Held while a command reads and rewrites a source directory, so two commands on one directory + * (an agent and a person, two terminals) never interleave their writes: each file moves into + * place atomically, but a set of them does not, and records from one command beside text from + * another would send stale text as an edit. A folder, since creating one is atomic everywhere. + */ +export const SOURCE_DIR_LOCK = '.sim/command.lock' + +/** The help line every command that holds {@link SOURCE_DIR_LOCK} adds to its exit codes. */ +export const SOURCE_DIR_BUSY_HELP = ` +Exit ${SOURCE_BUSY_EXIT_CODE} (busy, retry): another sim workspaces source command is using this directory +(SOURCE_DIR_BUSY); nothing was read or written. Run this one again once it finished.` + +/** The file-system codes of a directory the user may not write: its lock cannot be taken. */ +const UNWRITABLE_CODES: ReadonlySet = new Set(['EACCES', 'EPERM', 'EROFS']) + +/** A lock whose owner never recorded itself is stale after this long (it died while taking it). */ +const UNRECORDED_LOCK_STALE_MS = 60_000 + +/** The file in the lock folder that records its owner. */ +const LOCK_OWNER_FILE = 'owner.json' +/** The folder a command creates in a lock it found stale while it takes that lock over. */ +const LOCK_TAKEOVER_DIR = 'takeover' + +/** Who holds the lock, as its owner file records it. */ +interface LockOwner { + pid: number + host: string +} + +function isAlive(pid: number): boolean { + try { + process.kill(pid, 0) + return true + } catch (error) { + return errorCode(error) === 'EPERM' + } +} + +/** The lock folder a command found, as it found it: which folder, and when it last changed. */ +type LockSighting = Pick + +/** + * Whether the lock at `lock`, as `seen`, was left by a command that is gone: its process on this + * machine no longer runs, or it never recorded an owner and had not changed for a minute when it + * was seen. The age is the sighting's, not the folder's now, because a takeover's own claim + * changes the folder. A lock another machine holds (a shared folder) is never taken over. + */ +async function isStaleLock(lock: string, seen: LockSighting): Promise { + const owner = toRecord( + await readFile(join(lock, LOCK_OWNER_FILE), 'utf8').then( + (text): unknown => JSON.parse(text), + () => null + ) + ) + if (typeof owner.pid === 'number' && typeof owner.host === 'string') { + return owner.host === hostname() && !isAlive(owner.pid) + } + return Date.now() - seen.mtimeMs > UNRECORDED_LOCK_STALE_MS +} + +/** + * Removes the lock at `lock`, which the caller found stale as `seen`, unless another command took + * it over first. Two commands can find one lock stale at once: one removes it and takes the lock, + * and the other, acting on what it saw before, must not remove the live lock that replaced it. So + * a takeover first claims the lock folder it means to remove (creating {@link LOCK_TAKEOVER_DIR} + * in it, which only one command can), then removes it only while it is still the folder it saw + * and still stale; a lock that replaced the stale one is another folder or records a live owner, + * so the late command leaves it. A command that died mid-takeover leaves its claim in a stale + * lock, which the busy message says to remove. + */ +export async function takeOverStaleLock(lock: string, seen: LockSighting): Promise { + const claim = join(lock, LOCK_TAKEOVER_DIR) + try { + await mkdir(claim) + } catch (error) { + if (errorCode(error) === 'EEXIST' || isMissing(error)) return + throw error + } + const now = await stat(lock).catch(() => null) + const sameFolder = now !== null && now.dev === seen.dev && now.ino === seen.ino + if (sameFolder && (await isStaleLock(lock, seen))) + await rm(lock, { recursive: true, force: true }) + else await rmdir(claim).catch(() => {}) +} + +/** + * Runs `run` holding the directory's command lock ({@link SOURCE_DIR_LOCK}). A second command on + * the directory meanwhile is refused (`SOURCE_DIR_BUSY`, exit 10) with nothing read or written; a + * lock its command left behind when it died is taken over ({@link takeOverStaleLock}). Like every + * other source path, `.sim` and the lock are never followed through a symbolic link + * (`SOURCE_SYMLINK`). An embedded run has no local directory to lock. + */ +export async function withSourceDirLock( + dir: string, + run: () => Promise, + options: { readOnly?: boolean } = {} +): Promise { + if (embedStore.getStore()) return run() + // `mkdir` resolves `..` after following links and the cleanup climbs lexically: resolved, both + // walk one path. + dir = resolve(dir) + const absent = async (path: string) => + lstat(path).then( + () => false, + () => true + ) + /** The outermost folder the lock created on the way to `dir`, when `dir` was missing. */ + let createdFrom: string | undefined + let root: string + let records: string | undefined + let createdRecords = false + let lock: string + let held = false + /** Removes the folders the lock alone created, each only while it is empty. */ + const removeCreatedFolders = async () => { + if (createdRecords && records !== undefined) await rmdir(records).catch(() => {}) + if (createdFrom !== undefined) await removeEmptyFolders(dir, createdFrom) + } + try { + createdFrom = await mkdir(dir, { recursive: true }) + root = await realpath(dir) + records = await checkedPath(root, '.sim', new Set()) + createdRecords = createdFrom !== undefined || (await absent(records)) + await mkdir(records).catch((error: unknown) => { + if (errorCode(error) !== 'EEXIST') throw error + }) + lock = await checkedPath(root, SOURCE_DIR_LOCK, new Set()) + for (let attempt = 0; attempt < 2 && !held; attempt += 1) { + try { + await mkdir(lock) + held = true + } catch (error) { + if (errorCode(error) !== 'EEXIST') throw error + await checkedPath(root, SOURCE_DIR_LOCK, new Set()) + const seen = attempt === 0 ? await stat(lock).catch(() => null) : null + if (seen && (await isStaleLock(lock, seen))) await takeOverStaleLock(lock, seen) + } + } + } catch (error) { + if (!UNWRITABLE_CODES.has(errorCode(error) ?? '')) { + await removeCreatedFolders() + throw error + } + // Nothing can rewrite a directory no one may write, so a command that only reads it needs + // no lock; one that writes it is refused with the reason instead of a stack trace. + if (options.readOnly) { + try { + return await run() + } finally { + await removeCreatedFolders() + } + } + await removeCreatedFolders() + throw new SourceDirError( + `${dir} cannot be written (${errorCode(error)}), and this command writes it; nothing was read or written. Make it writable, or run the command on a writable copy.`, + { code: 'SOURCE_DIR_UNWRITABLE' } + ) + } + if (!held) { + await removeCreatedFolders() + throw new SourceDirError( + `Another sim command is using ${dir}; nothing was read or written. Run this one again once it finished. If none is running, remove ${join(dir, SOURCE_DIR_LOCK)}.`, + { code: 'SOURCE_DIR_BUSY', exitCode: SOURCE_BUSY_EXIT_CODE } + ) + } + const owner: LockOwner = { pid: process.pid, host: hostname() } + // An interrupt (Ctrl-C) releases the lock too: a lock left behind is taken over only by a + // command on the same host, so one left on a folder shared with a container would stay. + const disposeSignalRelease = removeStagingOnSignal(() => lock) + try { + await writeFile(join(lock, LOCK_OWNER_FILE), JSON.stringify(owner)) + return await run() + } finally { + disposeSignalRelease() + await rm(await checkedPath(root, SOURCE_DIR_LOCK, new Set()), { recursive: true, force: true }) + // Folders the lock alone created go with it; one the command filled stays. + await removeCreatedFolders() + } +} + +/** Removes `dir` and each parent up to `outermost`, stopping at the first that is not empty. */ +async function removeEmptyFolders(dir: string, outermost: string): Promise { + const last = resolve(outermost) + for (let folder = dir; ; folder = dirname(folder)) { + try { + await rmdir(folder) + } catch { + return + } + if (folder === last || dirname(folder) === folder) return + } +} + +/** + * Runs a single-workflow command on `dir` holding the command lock it shares with every command + * that may rewrite the same records: for a workflow folder of a workspace directory, the workspace + * directory's lock (such a command rewrites the workspace's own records, see {@link writeBack}, so + * it must not interleave with a workspace command, or with a workflow command in a sibling + * folder); for any other directory, its own. Refused like a workspace command (`SOURCE_DIR_BUSY`, + * exit 10), with nothing read or written. `readOnly` as {@link withSourceDirLock} takes it. + */ +export async function withSourceCommandLock( + dir: string, + run: () => Promise, + options: { readOnly?: boolean } = {} +): Promise { + const folder = await workspaceFolderOf(dir) + return withSourceDirLock(folder ? folder.root : dir, run, options) +} diff --git a/packages/sim-cli/src/commands/protocol/source-dir/dir-kind.ts b/packages/sim-cli/src/commands/protocol/source-dir/dir-kind.ts new file mode 100644 index 00000000000..edcfbaeed0a --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-dir/dir-kind.ts @@ -0,0 +1,121 @@ +/** + * What a source directory is: one workflow's, a whole workspace's, or a workflow folder of a + * workspace directory, and which workspace a command on it acts on. + */ + +import { basename, dirname, resolve } from 'node:path' +import { toRecord } from '@sim/utils/object' +import { SOURCE_BASE_FILE, SOURCE_ENTRY_FILE } from '@sim/workflow-code/limits' +import { KIND_DIRS } from '@sim/workspace-code' +import type { Command } from 'commander' +import type { ResolvedProfile } from '../../../config/index' +import { embedStore } from '../../../embed-context' +import { SourceDirError } from '../local-errors' +import { readSourceJson, readSourceText } from './files' +import { SOURCE_STARTER_FILE } from './layout' + +/** + * Makes `command`'s `--dir` (given or defaulted) an absolute path before its action runs, so every + * message names the directory it read wherever the CLI ran: a wrapper that changes directory + * first made `.` name a folder its caller never meant. An embedded run keeps its host's path. + */ +export function absoluteDirOption(command: Command): Command { + return command.hook('preAction', (_hooked, action) => { + const dir: unknown = action.getOptionValue('dir') + if (typeof dir === 'string' && !embedStore.getStore()) { + action.setOptionValue('dir', resolve(dir)) + } + }) +} + +/** + * The workspace a source command acts on for a directory read from `readFrom`: that workspace, + * whatever the profile's default, so a directory stays bound to where it came from. Only a + * `--workspace` naming another one is refused, with `next` saying what to run instead (given the + * workspace the directory was read from and the one named). A directory that records no workspace + * takes the profile's (`null` when none is selected). + */ +export function directoryWorkspace( + dir: string, + readFrom: string | null, + profile: Pick, + next: (readFrom: string, named: string) => string +): string | null { + const named = profile.workspaceId + if (readFrom === null) return named + if (profile.sources.workspaceId !== 'flag' || named === null || named === readFrom) { + return readFrom + } + throw new SourceDirError( + `${dir} was read from workspace ${readFrom}, but --workspace names ${named}. ${next(readFrom, named)}`, + { code: 'SOURCE_OTHER_WORKSPACE' } + ) +} + +/** Which source command family a directory belongs to: one workflow's, or a whole workspace's. */ +export type SourceDirKind = 'workflow' | 'workspace' + +/** + * What `dir` holds, by its `.sim/base.json` (read leniently, as an apply reads it: a base the + * parser refuses still names what it is): a workflow base names its `workflowId`, a workspace base + * only its `workspaceId`. Without a base, a `workflow.ts` or a starter record at the root is a + * workflow directory (a workspace directory keeps its workflows under `workflows//`). `null` + * for a directory that holds neither, such as a new one. + */ +async function sourceDirKind( + dir: string +): Promise<{ kind: SourceDirKind; readFrom: string | null } | null> { + const base = toRecord(await readSourceJson(dir, SOURCE_BASE_FILE)) + if (typeof base.workflowId === 'string') return { kind: 'workflow', readFrom: base.workflowId } + if (typeof base.workspaceId === 'string') { + return { kind: 'workspace', readFrom: base.workspaceId } + } + const starter = await readStarterWorkspace(dir) + if (starter !== null || (await readSourceText(dir, SOURCE_ENTRY_FILE)) !== null) { + return { kind: 'workflow', readFrom: null } + } + return null +} + +/** + * Refuses a `kind` command on a directory of the other kind, whatever `--force` says: a workflow + * get into a workspace directory (or the reverse) would replace its base, so staleness could no + * longer be detected and its unapplied edits would sit in a directory neither command reads. + */ +export async function assertSourceDirKind(dir: string, kind: SourceDirKind): Promise { + const held = await sourceDirKind(dir) + if (held === null || held.kind === kind) return + const message = + held.kind === 'workspace' + ? `${dir} is a workspace source directory${held.readFrom ? ` (workspace ${held.readFrom})` : ''}, so a \`sim workflows source\` command would replace its base. Edit a workflow of it in ${dir}/${KIND_DIRS.workflow}/ with \`sim workspaces source\` (check, status, apply) --dir ${dir}, or get the workflow into another directory.` + : `${dir} is a workflow source directory${held.readFrom ? ` (workflow ${held.readFrom})` : ''}, so a \`sim workspaces source\` command would replace its base. Use \`sim workflows source\` (get, check, status, apply) --dir ${dir}, or get the workspace into another directory.` + throw new SourceDirError(message, { code: 'SOURCE_DIR_KIND_MISMATCH' }) +} + +/** The workspace {@link SOURCE_STARTER_FILE} names, or `null` when the directory holds no starter. */ +export async function readStarterWorkspace(dir: string): Promise { + const workspaceId = toRecord(await readSourceJson(dir, SOURCE_STARTER_FILE)).workspaceId + return typeof workspaceId === 'string' ? workspaceId : null +} + +/** + * A `workflows/` folder of a workspace directory: `root` holds a workspace `.sim/base.json`. + * The workspace's `.sim/written.json` and base are the folder's records too, so a single-workflow + * write keeps them instead of writing a record of its own the workspace commands never read. + */ +export interface WorkspaceFolder { + root: string + workspaceId: string + /** The folder's key under `workflows/`. */ + key: string +} + +/** The workspace directory `dir` is a workflow folder of, or `null`. */ +export async function workspaceFolderOf(dir: string): Promise { + const folder = resolve(dir) + if (basename(dirname(folder)) !== KIND_DIRS.workflow) return null + const root = dirname(dirname(folder)) + const base = toRecord(await readSourceJson(root, SOURCE_BASE_FILE)) + if (typeof base.workspaceId !== 'string' || base.workflowId !== undefined) return null + return { root, workspaceId: base.workspaceId, key: basename(folder) } +} diff --git a/packages/sim-cli/src/commands/protocol/source-dir/files.ts b/packages/sim-cli/src/commands/protocol/source-dir/files.ts new file mode 100644 index 00000000000..f0aa3531bd6 --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-dir/files.ts @@ -0,0 +1,324 @@ +/** + * Reading and writing the source directory's files. Every path is relative to `realpath(--dir)` + * and no component below it may be a symbolic link, so nothing is read from or written to outside + * the directory, whatever a link in it points at. Writes are staged in the directory first and + * moved into place, so a failed write leaves the files it did not reach as they were. An embedded + * run reads and writes through its host instead. + */ + +import type { Dirent } from 'node:fs' +import { + lstat, + mkdir, + mkdtemp, + readdir, + readFile, + realpath, + rename, + rm, + writeFile, +} from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { dirname, join, posix } from 'node:path' +import { mapWithConcurrency } from '@sim/utils/concurrency' +import { isRecordLike } from '@sim/utils/object' +import { + SOURCE_BASE_FILE, + SOURCE_ENTRY_FILE, + SOURCE_CODE_DIR as SOURCE_FUNCTIONS_DIR, + SOURCE_LOCK_FILE, +} from '@sim/workflow-code/limits' +import { embedStore } from '../../../embed-context' +import { embeddedFileContent } from '../../../transfer/local-file' +import { removeStagingOnSignal, saveToFile } from '../files-get' +import { SourceDirError } from '../local-errors' +import { + SOURCE_STAGING_PREFIX, + SOURCE_STORED_EDGES_FILE, + SOURCE_WRITTEN_FILE, + sourceRelativePath, +} from './layout' + +const GITATTRIBUTES_LINES = [ + `${SOURCE_ENTRY_FILE} text eol=lf`, + `${SOURCE_FUNCTIONS_DIR}/** text eol=lf`, +] +/** Files read or written at once on a local disk; a catalog's declarations run to hundreds of small files. */ +export const FILE_CONCURRENCY = 32 + +/** + * The lock an earlier get, apply or create left, read leniently (missing or invalid is `null`): + * it names the function files the CLI wrote, which is all a prune needs from it. + */ +export function readPreviousLock(dir: string): Promise { + return readSourceJson(dir, SOURCE_LOCK_FILE) +} + +/** A JSON file in the source directory, read leniently: `null` when it is missing or invalid. */ +export async function readSourceJson(dir: string, path: string): Promise { + const text = await readSourceText(dir, path) + if (text === null) return null + try { + return JSON.parse(text) + } catch { + return null + } +} + +/** + * A line file (`.gitattributes`, `.gitignore`) with `lines` appended where missing, or `null` + * when it has them all. + */ +export function withMissingLines( + existing: string | null, + lines: readonly string[] = GITATTRIBUTES_LINES +): string | null { + const present = new Set((existing ?? '').split('\n').map((line) => line.trim())) + const missing = lines.filter((line) => !present.has(line)) + if (missing.length === 0) return null + const prefix = existing && !existing.endsWith('\n') ? `${existing}\n` : (existing ?? '') + return `${prefix}${missing.join('\n')}\n` +} + +/** Whether a file-system error means the path does not exist (`ENOENT`, or a file as a folder). */ +export function isMissing(error: unknown): boolean { + const code = errorCode(error) + return code === 'ENOENT' || code === 'ENOTDIR' +} + +/** + * The absolute path of `path` under `root` (already a realpath), refusing a symbolic link at any + * component that exists. `verified` caches directories already checked across one operation. + */ +export async function checkedPath( + root: string, + path: string, + verified: Set +): Promise { + const segments = path.split('/') + let current = root + for (const [index, segment] of segments.entries()) { + current = join(current, segment) + if (verified.has(current)) continue + let stats: Awaited> + try { + stats = await lstat(current) + } catch (error) { + if (isMissing(error)) return join(root, ...segments) + throw error + } + if (stats.isSymbolicLink()) { + throw new SourceDirError( + `Refusing ${segments.slice(0, index + 1).join('/')}: it is a symbolic link, and the source directory only holds plain files.`, + { code: 'SOURCE_SYMLINK' } + ) + } + if (index < segments.length - 1) verified.add(current) + } + return current +} + +async function realRoot(dir: string): Promise { + try { + return await realpath(dir) + } catch (error) { + if (isMissing(error)) return null + throw error + } +} + +/** + * Source text as the CLI reads and compares it: no leading byte-order mark and LF line endings, so + * a Windows editor or checkout never turns an unchanged source into an edit, or hides its + * `// @sim-sdk` header. + */ +export function normalizeSourceText(text: string): string { + return text.replace(/^\uFEFF/, '').replace(/\r\n/g, '\n') +} + +/** + * The text of `path` in the source directory, normalized by `normalizeSourceText`, or `null` when + * it does not exist. A symbolic link anywhere below the directory is refused rather than followed. + */ +export async function readSourceText(dir: string, path: string): Promise { + const embedded = embedStore.getStore() + let text: string + if (embedded) { + try { + const content = await embeddedFileContent(embedded, posix.join(dir, path)) + text = typeof content === 'string' ? content : new TextDecoder().decode(content) + } catch (error) { + // The host's reader names no missing-file error, so a failure reads as a missing file; a + // stopped run does not. + if (embedded.identity.signal?.aborted) throw error + return null + } + } else { + const root = await realRoot(dir) + if (root === null) return null + const target = await checkedPath(root, path, new Set()) + try { + text = await readFile(target, 'utf8') + } catch (error) { + if (isMissing(error)) return null + throw new SourceDirError(`Cannot read ${join(dir, path)}: ${(error as Error).message}`) + } + } + return normalizeSourceText(text) +} + +/** What one write does to the source directory: files to remove first, then files to write. */ +export interface SourceDirChange { + write: ReadonlyArray + /** + * Files whose bytes `stage` writes into the staged file it is handed (a download), before + * anything is moved into place: one that throws leaves the directory as it was. + */ + fetch?: ReadonlyArray Promise]> + /** Files or directories (removed recursively); an embedded run cannot remove, so skips these. */ + remove?: readonly string[] +} + +/** + * The files that record what the directory was read from: each workflow's lock, base, + * `.sim/written.json` and stored edges, and the workspace base. They move into place only after every other file, + * so a write cut short leaves the old records beside whatever text did land: the next apply is + * refused as stale, never sent as if the old text were current. + */ +const RECORD_FILES = [ + SOURCE_BASE_FILE, + SOURCE_LOCK_FILE, + SOURCE_WRITTEN_FILE, + SOURCE_STORED_EDGES_FILE, +] + +function isRecordFile(path: string): boolean { + return RECORD_FILES.some((file) => path === file || path.endsWith(`/${file}`)) +} + +/** + * Applies a change to the source directory, creating it when absent. Every target is checked + * before anything is touched; the files are written to a staging directory inside it, then the + * removals run and each file is renamed into place, never through a link, the record files last. + */ +export async function updateSourceDir(dir: string, change: SourceDirChange): Promise { + const fetches = change.fetch ?? [] + const writes = [...change.write.map(([path]) => path), ...fetches.map(([path]) => path)] + for (const path of [...writes, ...(change.remove ?? [])]) { + if (sourceRelativePath(path) !== path) { + throw new SourceDirError(`Refusing to write ${path} outside ${dir}`) + } + } + if (embedStore.getStore()) { + const fetched: Array]> = [] + if (fetches.length > 0) { + const scratch = await mkdtemp(join(tmpdir(), 'sim-source-fetch-')) + try { + for (const [path, stage] of fetches) { + const file = join(scratch, String(fetched.length)) + await stage(file) + fetched.push([path, new Uint8Array(await readFile(file))]) + } + } finally { + await rm(scratch, { recursive: true, force: true }) + } + } + const all = [...change.write, ...fetched] + // The host has no rename, so the records go last here too: after every file and blob. + for (const [path, content] of [ + ...all.filter(([path]) => !isRecordFile(path)), + ...all.filter(([path]) => isRecordFile(path)), + ]) { + await saveToFile(new Blob([content]).stream(), posix.join(dir, path), true) + } + return + } + + await mkdir(dir, { recursive: true }) + const root = await realpath(dir) + const verified = new Set() + const targets: string[] = [] + for (const path of writes) targets.push(await checkedPath(root, path, verified)) + const removals: string[] = [] + for (const path of change.remove ?? []) removals.push(await checkedPath(root, path, verified)) + + let staging: string | null = null + const disposeSignalCleanup = removeStagingOnSignal(() => staging) + try { + const stagingDir = await mkdtemp(join(root, SOURCE_STAGING_PREFIX)) + staging = stagingDir + const indexes = writes.map((_, index) => index) + const staged = (index: number) => join(stagingDir, String(index)) + await mapWithConcurrency(indexes, FILE_CONCURRENCY, (index) => + index < change.write.length + ? writeFile(staged(index), change.write[index][1], { flag: 'wx' }) + : fetches[index - change.write.length][1](staged(index)) + ) + for (const target of removals) await rm(target, { recursive: true, force: true }) + const parents = new Map>() + const moveIntoPlace = async (index: number) => { + const parent = dirname(targets[index]) + let created = parents.get(parent) + if (!created) { + created = mkdir(parent, { recursive: true }) + parents.set(parent, created) + } + await created + await rename(staged(index), targets[index]) + } + await mapWithConcurrency( + indexes.filter((index) => !isRecordFile(writes[index])), + FILE_CONCURRENCY, + moveIntoPlace + ) + await mapWithConcurrency( + indexes.filter((index) => isRecordFile(writes[index])), + FILE_CONCURRENCY, + moveIntoPlace + ) + } finally { + if (staging) await rm(staging, { recursive: true, force: true }) + disposeSignalCleanup() + } +} + +/** + * Every plain file under `folder` in the source directory, as source-relative paths. A symbolic + * link anywhere is refused rather than followed; a missing folder lists nothing. + */ +export async function listSourceFiles(dir: string, folder: string): Promise { + if (embedStore.getStore()) { + throw new SourceDirError('Workspace source directories need a local disk.') + } + const root = await realRoot(dir) + if (root === null) return [] + const start = await checkedPath(root, folder, new Set()) + const files: string[] = [] + const walk = async (absolute: string, relative: string): Promise => { + let entries: Dirent[] + try { + entries = await readdir(absolute, { withFileTypes: true, encoding: 'utf8' }) + } catch (error) { + if (isMissing(error)) return + throw error + } + for (const entry of entries) { + const path = `${relative}/${entry.name}` + if (entry.isSymbolicLink()) { + throw new SourceDirError( + `Refusing ${path}: it is a symbolic link, and the source directory only holds plain files.`, + { code: 'SOURCE_SYMLINK' } + ) + } + if (entry.isDirectory()) await walk(join(absolute, entry.name), path) + else if (entry.isFile()) files.push(path) + } + } + await walk(start, folder) + return files +} + +/** A file-system or process error's code (`EEXIST`, `ESRCH`), if it has one. */ +export function errorCode(error: unknown): string | undefined { + return isRecordLike(error) && typeof error.code === 'string' ? error.code : undefined +} diff --git a/packages/sim-cli/src/commands/protocol/source-dir/index.ts b/packages/sim-cli/src/commands/protocol/source-dir/index.ts new file mode 100644 index 00000000000..b71036d9cee --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-dir/index.ts @@ -0,0 +1,66 @@ +/** + * The source directory on disk, for `workflows source` (one workflow's directory) and + * `workspaces source` (whose `workflows//` folders are each one): its layout beside the + * printed files, and the only code that reads or writes it. One definition, so the commands + * cannot drift apart. + */ + +export { + SOURCE_DIR_BUSY_HELP, + SOURCE_DIR_LOCK, + takeOverStaleLock, + withSourceCommandLock, + withSourceDirLock, +} from './command-lock' +export { + absoluteDirOption, + assertSourceDirKind, + directoryWorkspace, + readStarterWorkspace, + workspaceFolderOf, +} from './dir-kind' +export { + checkedPath, + FILE_CONCURRENCY, + isMissing, + listSourceFiles, + readPreviousLock, + readSourceJson, + readSourceText, + type SourceDirChange, + updateSourceDir, + withMissingLines, +} from './files' +export { + assertPrintedSourcePaths, + assertSendableCodeFiles, + lockCodeFiles, + orphanedCodeFiles, + SOURCE_AGENTS_FILE, + SOURCE_DOWNLOADS_DIR, + SOURCE_EXAMPLES_DIR, + SOURCE_GITATTRIBUTES_FILE, + SOURCE_GITIGNORE_FILE, + SOURCE_SDK_DIR, + SOURCE_STAGING_PREFIX, + SOURCE_STARTER_FILE, + SOURCE_STORED_EDGES_FILE, + SOURCE_WRITTEN_FILE, + sourceRelativePath, +} from './layout' +export { writeBack } from './write-back' +export { + digest, + editedFiles, + type FileStat, + fileStatOf, + locallyEditedFiles, + readFolderWritten, + readWritten, + recordedStats, + type StatDigest, + sameFileStat, + type Written, + writtenFiles, + writtenRecord, +} from './written' diff --git a/packages/sim-cli/src/commands/protocol/source-dir/layout.ts b/packages/sim-cli/src/commands/protocol/source-dir/layout.ts new file mode 100644 index 00000000000..82966b52d6e --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-dir/layout.ts @@ -0,0 +1,155 @@ +/** + * The source directory's layout beside the printed files: the CLI's own record and cache paths, + * which paths a server answer may write, and the function files a lock names. + */ + +import { posix } from 'node:path' +import { isRecordLike, toRecord } from '@sim/utils/object' +import { + isCodeFilePath, + SOURCE_ENTRY_FILE, + SOURCE_CODE_DIR as SOURCE_FUNCTIONS_DIR, +} from '@sim/workflow-code/limits' +import { printError } from '#sim-cli/output/io' +import { SourceDirError } from '../local-errors' + +/** Generated SDK declarations: the blocks the source uses (every block's are cached), plus `workspace.d.ts`. */ +export const SOURCE_SDK_DIR = '.sim/sdk' +/** A short guide for an agent working in the directory; never a top-level `AGENTS.md`. */ +export const SOURCE_AGENTS_FILE = '.sim/AGENTS.md' +/** + * Example workflows (`cookbook.ts`, `integrations.ts`, `agents.ts`): together one correct example + * of each common block, rewritten on every get like the agent guide. They name example resources + * the workspace may lack, so the written `tsconfig.json` never includes them. + */ +export const SOURCE_EXAMPLES_DIR = '.sim/examples' +/** Pins the source files to LF, merged into whatever the directory already holds. */ +export const SOURCE_GITATTRIBUTES_FILE = '.gitattributes' +/** Keeps a command's transient files out of a repository, merged like `.gitattributes`. */ +export const SOURCE_GITIGNORE_FILE = '.gitignore' +/** + * A digest of each source file as the CLI last wrote it, so a later get can tell an unapplied + * edit from a file it may replace. The CLI's own record: never sent to the server. + */ +export const SOURCE_WRITTEN_FILE = '.sim/written.json' +/** + * The workspace a starter written by `workflows source create` was typed for, so the create that + * follows targets that workspace. The CLI's own record: never sent, and removed once the workflow + * exists. + */ +export const SOURCE_STARTER_FILE = '.sim/starter.json' +/** + * The edges of the workflow a get or apply printed, by lock key, with the base fingerprint they + * belong to, so an offline check tells an edge the stored workflow holds from one the source + * adds. The CLI's own record: never sent to the server. + */ +export const SOURCE_STORED_EDGES_FILE = '.sim/stored-edges.json' +/** + * Where a workspace get keeps the blobs it verified until they move into place. A get that fails + * leaves them there, so the next one downloads only the rest; the one that completes removes the + * folder. + */ +export const SOURCE_DOWNLOADS_DIR = '.sim/downloads' +/** The folder a write stages its files in before moving them into place; removed when it ends. */ +export const SOURCE_STAGING_PREFIX = '.sim-staging-' + +/** + * Whether a server answer may write `path` into the source directory: the entry file, or a code + * file as {@link isCodeFilePath} (the rule the compiler applies to `codeFile()`) names one. + * Anything else (`.git/hooks/*`, `conftest.py`, a dotfile) is refused, whatever base URL answered. + */ +function isPrintedSourcePath(path: string): boolean { + return path === SOURCE_ENTRY_FILE || isCodeFilePath(path) +} + +/** The paths of `files` that {@link isPrintedSourcePath} refuses, in their order. */ +function refusedSourcePaths(files: Record): string[] { + return Object.keys(files).filter((path) => !isPrintedSourcePath(path)) +} + +/** Refuses a server answer naming a file the printer never writes, before anything is written. */ +export function assertPrintedSourcePaths(files: Record): void { + const [refused] = refusedSourcePaths(files) + if (refused !== undefined) { + throw new SourceDirError(`The workflow source named an unsafe file path: ${refused}`) + } +} + +/** + * Refuses, before a write is sent, a code file the server would store but the CLI could never + * write back: one rule ({@link isCodeFilePath}) for both, so an apply is never stored and then + * reported as failed. + */ +export function assertSendableCodeFiles(files: Record): void { + const refused = refusedSourcePaths(files) + if (refused.length === 0) return + throw new SourceDirError( + `${refused.join(', ')} ${refused.length === 1 ? 'is not a code file path' : 'are not code file paths'} a workflow directory holds, so nothing was sent. Move the code to ${SOURCE_FUNCTIONS_DIR}/.js (or .py, .sh: letters, digits, _, $, - and dots, not starting with a dot) and update its codeFile().`, + { code: 'E_CODE_FILE_PATH' } + ) +} + +/** + * A path inside the source directory, normalized (`./functions/x.js` → `functions/x.js`), or + * `null` for an absolute path, a drive letter or a `..` escape. + */ +export function sourceRelativePath(path: string): string | null { + const normalized = posix.normalize(path.replace(/\\/g, '/')) + if ( + normalized.startsWith('/') || + /^[A-Za-z]:/.test(normalized) || + normalized === '.' || + normalized === '..' || + normalized.startsWith('../') + ) { + return null + } + return normalized.startsWith('./') ? normalized.slice(2) : normalized +} + +/** Every `codeFile` a lock records, as source-relative paths. */ +export function lockCodeFiles(lock: unknown): Set { + const paths = new Set() + const blocks = toRecord(toRecord(lock).blocks) + for (const block of Object.values(blocks)) { + const path = + isRecordLike(block) && typeof block.codeFile === 'string' + ? sourceRelativePath(block.codeFile) + : null + if (path) paths.add(path) + } + return paths +} + +/** + * The function files a write leaves behind: named by the lock the directory held (so the CLI + * wrote them), no longer named by the new lock, and not being written now. A file the person + * added and no lock ever named is never pruned, nor is an entry that is no code file (such as + * `functions/`), whose removal would take the whole folder. + */ +export function orphanedCodeFiles( + previousLock: unknown, + nextLock: unknown, + written: Iterable +): string[] { + const kept = lockCodeFiles(nextLock) + for (const path of written) kept.add(path) + return [...lockCodeFiles(previousLock)].filter( + (path) => path.startsWith(`${SOURCE_FUNCTIONS_DIR}/`) && isCodeFilePath(path) && !kept.has(path) + ) +} + +/** + * The files of a stored write's answer the CLI may write: every one {@link isPrintedSourcePath} + * accepts. The write is already stored, so a path refused here is left as it is on disk (it is + * the author's own file, which the server echoed) with a warning, never thrown: the lock and base + * are still written, so the directory matches the stored workflow. + */ +export function writableFiles(files: Record): Record { + const refused = refusedSourcePaths(files) + if (refused.length === 0) return files + printError( + `warning: the answer named ${refused.join(', ')}, which the CLI never writes; the write was saved and the lock and base were written, but ${refused.length === 1 ? 'that file was' : 'those files were'} left as ${refused.length === 1 ? 'it is' : 'they are'} on disk.` + ) + return Object.fromEntries(Object.entries(files).filter(([path]) => isPrintedSourcePath(path))) +} diff --git a/packages/sim-cli/src/commands/protocol/source-dir/write-back.ts b/packages/sim-cli/src/commands/protocol/source-dir/write-back.ts new file mode 100644 index 00000000000..53c21b97b4b --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-dir/write-back.ts @@ -0,0 +1,141 @@ +/** + * Saving a server answer (a get's, or a stored write's) into a workflow source directory: its + * files, lock and base in one staged update, and the records that say the CLI wrote them. + */ + +import { toRecord } from '@sim/utils/object' +import { compareStrings } from '@sim/utils/string' +import { codeFileReferences } from '@sim/workflow-code/code-files/references' +import { + isCodeFilePath, + SOURCE_BASE_FILE, + SOURCE_ENTRY_FILE, + SOURCE_LOCK_FILE, +} from '@sim/workflow-code/limits' +import { fingerprintOf } from '@sim/workspace-code' +import { printError } from '#sim-cli/output/io' +import { type WorkspaceFolder, workspaceFolderOf } from './dir-kind' +import { readSourceJson, readSourceText, type SourceDirChange, updateSourceDir } from './files' +import { + lockCodeFiles, + orphanedCodeFiles, + SOURCE_WRITTEN_FILE, + sourceRelativePath, + writableFiles, +} from './layout' +import { + digest, + inWorkspace, + type PrintedSourceAnswer, + readWritten, + writtenFiles, + writtenRecord, +} from './written' + +/** + * Records a single-workflow write into workflow folder `folder` in the workspace records: each + * written file's digest (`written`, folder-relative path → text) replaces the one before, a + * `removed` file's is dropped, and the workspace base's entry for the workflow takes the + * fingerprint of the folder's new base, as a workspace get would record it. Written after the + * folder's files, so a write cut short leaves the old records. + */ +async function recordFolderWrite( + folder: WorkspaceFolder, + written: Record, + removed: readonly string[], + base: unknown +): Promise { + const record = await readWritten(folder.root) + const dropped = new Set(removed.map((path) => inWorkspace(folder.key, path))) + const digests = { + ...Object.fromEntries(Object.entries(record.digests).filter(([path]) => !dropped.has(path))), + ...Object.fromEntries( + Object.entries(written).map(([path, text]) => [inWorkspace(folder.key, path), digest(text)]) + ), + } + const write: Array = [ + ...writtenFiles({ digests, stats: record.stats }), + ] + const workspaceBase = toRecord(await readSourceJson(folder.root, SOURCE_BASE_FILE)) + const resources = toRecord(workspaceBase.resources) + const address = `workflow:${folder.key}` + const entry = toRecord(resources[address]) + const { workflowId, fingerprint } = toRecord(base) + if (entry.id === workflowId && typeof fingerprint === 'string') { + const next = { + ...workspaceBase, + resources: { + ...resources, + [address]: { ...entry, fingerprint: fingerprintOf({ source: fingerprint }) }, + }, + } + write.push([SOURCE_BASE_FILE, `${JSON.stringify(next, null, 2)}\n`]) + } + await updateSourceDir(folder.root, { write }) +} + +/** + * Writes a server answer into the source directory in one staged update: its files, lock and base, + * plus `extra`, pruning the function files the previous lock named and the new one no longer does. + * Returns how many files it wrote. A get refuses an unsafe path first + * ({@link assertPrintedSourcePaths}); after a stored write one is skipped ({@link writableFiles}). + */ +export async function writeBack( + dir: string, + answer: PrintedSourceAnswer, + previousLock: unknown, + extra: SourceDirChange = { write: [] } +): Promise { + const files = writableFiles(answer.files) + const folder = await workspaceFolderOf(dir) + const printed: Array = [ + ...Object.entries(files), + [SOURCE_LOCK_FILE, `${JSON.stringify(answer.lock, null, 2)}\n`], + [SOURCE_BASE_FILE, `${JSON.stringify(answer.base, null, 2)}\n`], + ] + const write = [...printed, ...(folder ? [] : writtenFiles(writtenRecord(files))), ...extra.write] + const orphaned = orphanedCodeFiles(previousLock, answer.lock, Object.keys(answer.files)) + const unnamed = await unnamedCodeFiles(dir, previousLock, answer, orphaned) + await updateSourceDir(dir, { + write, + remove: [ + ...orphaned, + ...(extra.remove ?? []), + // A record of the folder's own, from before, would shadow the workspace's. + ...(folder ? [SOURCE_WRITTEN_FILE] : []), + ], + }) + if (folder) await recordFolderWrite(folder, Object.fromEntries(printed), orphaned, answer.base) + if (unnamed.length > 0) { + printError( + `warning: ${unnamed.join(', ')} ${unnamed.length === 1 ? 'is' : 'are'} no longer named by ${SOURCE_ENTRY_FILE} (the printed source names no code file there, such as code now printed as stored()), so ${unnamed.length === 1 ? 'it was' : 'they were'} left on disk as written: delete ${unnamed.length === 1 ? 'it' : 'them'}, or name ${unnamed.length === 1 ? 'it' : 'them'} again with codeFile(). Never commit one that holds a credential.` + ) + } + return write.length +} + +/** + * The code files a write leaves on disk although the printed source no longer names them: named + * by the entry file on disk before the write or by the previous lock, neither written nor pruned + * now, and still present. A file the person added is never deleted, so the caller says where it + * is instead: one that held a literal secret now printed as `stored()` must not be committed. + */ +async function unnamedCodeFiles( + dir: string, + previousLock: unknown, + answer: PrintedSourceAnswer, + pruned: readonly string[] +): Promise { + const entry = await readSourceText(dir, SOURCE_ENTRY_FILE) + const named = lockCodeFiles(previousLock) + for (const literal of entry === null ? [] : codeFileReferences(entry)) { + const path = sourceRelativePath(literal) + if (path !== null) named.add(path) + } + const kept = new Set([...Object.keys(answer.files), ...lockCodeFiles(answer.lock), ...pruned]) + const candidates = [...named].filter((path) => isCodeFilePath(path) && !kept.has(path)) + const present = await Promise.all( + candidates.map(async (path) => (await readSourceText(dir, path)) !== null) + ) + return candidates.filter((_, index) => present[index]).sort(compareStrings) +} diff --git a/packages/sim-cli/src/commands/protocol/source-dir/written.ts b/packages/sim-cli/src/commands/protocol/source-dir/written.ts new file mode 100644 index 00000000000..76b8de80b0a --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-dir/written.ts @@ -0,0 +1,263 @@ +/** + * The CLI's record of what it last wrote into a source directory (`.sim/written.json`, and the + * machine-local stat cache beside it), and the edits on disk that record tells apart from its own + * writes. + */ + +import { createHash } from 'node:crypto' +import { mapWithConcurrency } from '@sim/utils/concurrency' +import { toRecord } from '@sim/utils/object' +import { compareStrings } from '@sim/utils/string' +import { SOURCE_BASE_FILE } from '@sim/workflow-code/limits' +import { KIND_DIRS } from '@sim/workspace-code' +import { workspaceFolderOf } from './dir-kind' +import { FILE_CONCURRENCY, normalizeSourceText, readSourceJson, readSourceText } from './files' +import { orphanedCodeFiles, SOURCE_WRITTEN_FILE } from './layout' + +/** + * Machine-local cache the CLI keeps beside its records: the stat each blob digest was taken at, + * which names this machine's files (device, inode, times) and so is never committed. The folder + * ignores itself (`.sim/cache/.gitignore`), whatever the repository's own `.gitignore` says. + */ +const SOURCE_CACHE_DIR = '.sim/cache' +const SOURCE_STATS_FILE = `${SOURCE_CACHE_DIR}/stats.json` +const SOURCE_CACHE_IGNORE_FILE = `${SOURCE_CACHE_DIR}/.gitignore` + +/** + * What identifies a file's bytes without reading them, taken when its digest was: the file (device + * and inode), its size, and its modification and change times. The change time is the one a user + * tool cannot set back (`cp -p`, `rsync -t` and `touch -d` restore only the modification time), so + * any write since the digest moves it even when the size and modification time match. + */ +export interface FileStat { + dev: number + ino: number + size: number + mtimeMs: number + ctimeMs: number +} + +/** The {@link FileStat} of a `stat` result. */ +export function fileStatOf(stats: FileStat): FileStat { + const { dev, ino, size, mtimeMs, ctimeMs } = stats + return { dev, ino, size, mtimeMs, ctimeMs } +} + +/** Whether two stats name the same, unchanged bytes. */ +export function sameFileStat(a: FileStat, b: FileStat): boolean { + return ( + a.dev === b.dev && + a.ino === b.ino && + a.size === b.size && + a.mtimeMs === b.mtimeMs && + a.ctimeMs === b.ctimeMs + ) +} + +/** A blob's SHA-256 with the stat of the file it was read from. */ +export interface StatDigest extends FileStat { + sha256: string +} + +/** + * `.sim/written.json`: each source file's digest as the CLI last wrote it, normalized, sorted by + * path. It holds nothing about this machine, so a clone reads it the same and a write that changes + * no file leaves it byte for byte. + */ +interface WrittenRecord { + version: 1 + files: Record +} + +/** + * `.sim/cache/stats.json`: for a blob, the stat its digest was taken at, beside that digest, so a + * later command on this machine reuses the digest of an untouched blob instead of reading it + * again. A stat whose digest is not the one `.sim/written.json` records is ignored. + */ +interface StatsRecord { + version: 1 + stats: Record +} + +/** + * What the CLI last wrote, as one command reads it, once: the digests from `.sim/written.json` + * and the stats from `.sim/cache/stats.json` that still match them. + */ +export interface Written { + digests: Record + stats: Record +} + +/** The digest of `text` as `readSourceText` would read it back. */ +export function digest(text: string): string { + return createHash('sha256').update(normalizeSourceText(text), 'utf8').digest('hex') +} + +/** + * What `.sim/written.json` records for `files` as they are about to be written, keeping the `kept` + * digests of files this write leaves as they are. A blob's stat is recorded only beside the digest + * it was taken with. + */ +export function writtenRecord( + files: Record, + kept: Record = {}, + stats: ReadonlyMap = new Map() +): Written { + const digests: Record = { + ...kept, + ...Object.fromEntries(Object.entries(files).map(([path, text]) => [path, digest(text)])), + } + const recorded = [...stats].flatMap(([path, { sha256, ...stat }]) => + digests[path] === sha256 ? [[path, fileStatOf(stat)] as const] : [] + ) + return { digests, stats: Object.fromEntries(recorded) } +} + +/** `record` with its keys in path order, so the same entries always print the same text. */ +function sortedByPath(record: Record): Record { + return Object.fromEntries(Object.entries(record).sort(([a], [b]) => compareStrings(a, b))) +} + +/** + * The files that record `written`: `.sim/written.json` (committed) and, when a blob stat is + * recorded, the machine-local `.sim/cache/stats.json` with the cache folder's own `.gitignore`. + */ +export function writtenFiles(written: Written): Array { + const record: WrittenRecord = { version: 1, files: sortedByPath(written.digests) } + const files: Array = [ + [SOURCE_WRITTEN_FILE, `${JSON.stringify(record, null, 2)}\n`], + ] + const stats = Object.entries(written.stats).flatMap(([path, stat]) => + Object.hasOwn(written.digests, path) + ? [[path, { sha256: written.digests[path], ...fileStatOf(stat) }] as const] + : [] + ) + if (stats.length === 0) return files + const cache: StatsRecord = { version: 1, stats: sortedByPath(Object.fromEntries(stats)) } + return [ + ...files, + [SOURCE_STATS_FILE, `${JSON.stringify(cache, null, 2)}\n`], + [SOURCE_CACHE_IGNORE_FILE, '*\n'], + ] +} + +/** What the CLI recorded for its last write, empty when it recorded nothing. */ +export async function readWritten(dir: string): Promise { + const record = toRecord(await readSourceJson(dir, SOURCE_WRITTEN_FILE)) + const digests = Object.fromEntries( + Object.entries(toRecord(record.files)).filter( + (entry): entry is [string, string] => typeof entry[1] === 'string' + ) + ) + const cache = toRecord(await readSourceJson(dir, SOURCE_STATS_FILE)) + const stats: Record = {} + for (const [path, stat] of Object.entries(toRecord(cache.stats))) { + const { sha256, dev, ino, size, mtimeMs, ctimeMs } = toRecord(stat) + if ( + typeof sha256 === 'string' && + Object.hasOwn(digests, path) && + digests[path] === sha256 && + typeof dev === 'number' && + typeof ino === 'number' && + typeof size === 'number' && + typeof mtimeMs === 'number' && + typeof ctimeMs === 'number' + ) { + stats[path] = { dev, ino, size, mtimeMs, ctimeMs } + } + } + return { digests, stats } +} + +/** The blob digests `written` holds with their stats, to carry into the next record. */ +export function recordedStats(written: Written): Map { + return new Map( + Object.entries(written.stats).flatMap(([path, stat]) => + Object.hasOwn(written.digests, path) + ? [[path, { ...stat, sha256: written.digests[path] }] as const] + : [] + ) + ) +} + +/** + * The files among `touched` (path → the text about to be written, or `null` for a removal) whose + * text on disk is neither what the CLI last wrote there nor what it is about to write: a write + * would destroy an edit no apply has saved. A file the CLI has no record of counts as edited, so a + * hand-written file is never replaced silently either; so does one the CLI wrote that is missing + * now and would be written again, a delete no apply has sent. + */ +export async function editedFiles( + dir: string, + touched: ReadonlyMap, + written: Written['digests'] +): Promise { + const entries = [...touched] + const texts = await mapWithConcurrency(entries, FILE_CONCURRENCY, ([path]) => + readSourceText(dir, path) + ) + return entries.flatMap(([path, next], index) => { + const current = texts[index] + if (current === null) return next !== null && Object.hasOwn(written, path) ? [path] : [] + if (next !== null && current === normalizeSourceText(next)) return [] + return written[path] !== digest(current) ? [path] : [] + }) +} + +/** + * The source files a write of `answer` would destroy edits in: a file it replaces or prunes whose + * text on disk is neither what the CLI last wrote there nor what it is about to write. + */ +export async function locallyEditedFiles( + dir: string, + answer: Pick, + previousLock: unknown +): Promise { + const touched = new Map(Object.entries(answer.files)) + for (const path of orphanedCodeFiles(previousLock, answer.lock, touched.keys())) { + touched.set(path, null) + } + return editedFiles(dir, touched, (await readFolderWritten(dir)).digests) +} + +/** `path` of the workflow folder `key`, relative to its workspace directory. */ +export function inWorkspace(key: string, path: string): string { + return `${KIND_DIRS.workflow}/${key}/${path}` +} + +/** + * What the CLI last wrote into the workflow directory `dir`: its own `.sim/written.json`, or for a + * workflow folder of a workspace directory, the workspace record's entries for the folder (its + * `.sim/` records left out). A write into such a folder records there and never in the folder, but + * a CLI from before that left a record of the folder's own. That record wins only when it is the + * later one: when the workspace record's digest of the folder's `.sim/base.json` is not the base on + * disk, a single-workflow write replaced the base after the workspace record was taken. Otherwise + * the folder's own record is stale and ignored. + */ +export async function readFolderWritten(dir: string): Promise { + const own = await readWritten(dir) + const folder = await workspaceFolderOf(dir) + if (!folder) return own + const record = await readWritten(folder.root) + if (Object.keys(own.digests).length > 0) { + const base = await readSourceText(dir, SOURCE_BASE_FILE) + const recorded = record.digests[inWorkspace(folder.key, SOURCE_BASE_FILE)] + if (base !== null && recorded !== digest(base)) return own + } + const prefix = inWorkspace(folder.key, '') + const inFolder = (entries: Record) => + Object.fromEntries( + Object.entries(entries).flatMap(([path, value]) => { + const relative = path.startsWith(prefix) ? path.slice(prefix.length) : null + return relative !== null && !relative.startsWith('.sim/') ? [[relative, value]] : [] + }) + ) + return { digests: inFolder(record.digests), stats: inFolder(record.stats) } +} + +/** What the server answers a get, apply or create with: the printed files, lock and base. */ +export interface PrintedSourceAnswer { + files: Record + lock: unknown + base: unknown +} diff --git a/packages/sim-cli/src/commands/protocol/source-exit-codes.test.ts b/packages/sim-cli/src/commands/protocol/source-exit-codes.test.ts index e2d3979b4e8..06cf8d8e4c9 100644 --- a/packages/sim-cli/src/commands/protocol/source-exit-codes.test.ts +++ b/packages/sim-cli/src/commands/protocol/source-exit-codes.test.ts @@ -2,7 +2,8 @@ import { readFileSync } from 'node:fs' import { join } from 'node:path' import { describe, expect, it } from 'vitest' import * as exitCodes from '../../exit-codes' -import { WORKFLOW_TREE } from './workflow-source-get' +import { sourceExitCodesHelp } from './source-exit-codes' +import { WORKFLOW_TREE } from './workflow-dir' import { WORKSPACE_TREE } from './workspace-source-dir' /** @@ -33,3 +34,14 @@ describe('the exit-code tables an agent or a reader relies on', () => { expect(listed).toEqual(SOURCE_EXIT_CODES) }) }) + +describe('the exit codes a get names in its --help', () => { + it.each(['workflows source get', 'workspaces source get'] as const)( + '%s names exit 13 for a missing or refused login', + (command) => { + expect(sourceExitCodesHelp(command).replace(/\s+/g, ' ')).toContain( + `${exitCodes.AUTH_REQUIRED_EXIT_CODE} not logged in` + ) + } + ) +}) diff --git a/packages/sim-cli/src/commands/protocol/source-exit-codes.ts b/packages/sim-cli/src/commands/protocol/source-exit-codes.ts index 9a4fed61347..0f7643a9edb 100644 --- a/packages/sim-cli/src/commands/protocol/source-exit-codes.ts +++ b/packages/sim-cli/src/commands/protocol/source-exit-codes.ts @@ -18,84 +18,243 @@ const INTERRUPTED_EXIT_CODE = 130 /** How long an interrupted write waits for its answer before it stops without it. */ const DEFERRED_SECONDS = DEFERRED_INTERRUPT_BOUND_MS / 1000 +/** The command group a directory guide's exit-code table is for. */ +type SourceCommandGroup = 'sim workflows source' | 'sim workspaces source' + +/** A source command whose `--help` lists its exit codes from {@link sourceExitCodeRows}. */ +export type SourceHelpCommand = + | 'workflows source apply' + | 'workflows source create' + | 'workflows source get' + | 'workspaces source apply' + | 'workspaces source get' + +/** One exit code of the source commands: what it means, what to do, and each command's wording. */ +interface SourceExitCodeRow { + code: number + /** Its meaning in the directory guides' table. */ + meaning: string + /** What to do next, in that table. */ + next: string + /** Its meaning in the `--help` of each command that exits with it. */ + help: Partial> +} + +/** A get's exit 13, which a write names in its shared section instead. */ +const GET_AUTH_REQUIRED = + 'not logged in, the login expired, or the server refused the credential (AUTH_REQUIRED): nothing was read or written, run `sim login` and the get again' + +/** The busy clause of a single-workflow command, which locks --dir or its workspace directory. */ +const WORKFLOW_DIR_BUSY = + '(busy, retry) another sim command is using --dir, or the workspace directory it is a workflow folder of (SOURCE_DIR_BUSY): nothing was read or written, run it again once the other finished' + /** - * Every exit code of the `workflows source` and `workspaces source` commands, as the Markdown - * table both directory guides (`.sim/AGENTS.md`) print, built from the codes the CLI exits with so - * the guides cannot drift from them. `prefix` is the command group the guide is for. + * Every exit code of the `workflows source` and `workspaces source` commands, in code order: the + * one table the directory guides and each command's `--help` are built from, so neither can + * drift from the codes the CLI exits with. `prefix` is the command group a guide is for. */ -export function sourceExitCodesTable( - prefix: 'sim workflows source' | 'sim workspaces source' -): string { - const rows: Array<[number, string, string]> = [ - [0, 'Applied or created, a dry run or check with no errors, or a clean status', 'Go on'], - [ - 1, - 'The source has errors (each printed as `file:line:column` with a hint), a locked block (`E_APPLY_LOCKED_BLOCK`), a name another workflow holds (`SOURCE_NAME_TAKEN`), a cancelled prompt, or the command failed', - 'Fix what it printed and run it again', - ], - [ - SOURCE_STALE_EXIT_CODE, - 'Stale (`SOURCE_STALE`, or `E_APPLY_STALE` on a dry run): the server changed what this directory was read from, and nothing was written', - `\`${prefix} get --dir \`, reapply your edit there, dry-run, apply`, - ], - [ - SOURCE_LOCKED_EXIT_CODE, - 'A workflow or its folder is locked (`E_APPLY_LOCKED`); a locked block exits 1 instead', - 'Ask someone to unlock it in the editor', - ], - [ - SOURCE_TOO_LARGE_EXIT_CODE, - 'The source is over the size limits (on a get, status or diff too)', - 'Split function code, or remove blocks or resources', - ], - [ - SOURCE_PARTIAL_EXIT_CODE, - '`sim workspaces source apply` stopped part-way: what was written is saved into the directory', - 'Apply again until it exits 0', - ], - [ - RATE_LIMITED_EXIT_CODE, - "Still rate limited (HTTP 429) after the CLI's retries (up to 6, at most a minute apart); nothing more was written", - 'Wait a minute and run the same command again', - ], - [ - SOURCE_BUSY_EXIT_CODE, - 'Busy: another apply is writing the workspace (`SOURCE_BUSY`), or another sim command is using this directory (`SOURCE_DIR_BUSY`); nothing was written', - 'Run it again once the other finished', - ], - [ - SOURCE_LOCAL_CHANGES_EXIT_CODE, - '`status`: only your edits, which applying writes', - 'Dry-run, then apply', - ], - [ - SOURCE_DRIFT_EXIT_CODE, - '`status`: the server changed since the get (drift, or a conflict with your edits)', - 'Follow the `Next:` line it prints', - ], - [ - AUTH_REQUIRED_EXIT_CODE, - 'Not logged in, the login expired, or the server refused the credential (`AUTH_REQUIRED`); nothing was sent', - '`sim login`, or fix or unset `SIM_API_KEY` when the key comes from it; then the same command again', - ], - [ - SOURCE_PLANNED_EXIT_CODE, - '`apply` or `create` without `--yes` and no terminal to ask: it printed the plan, found no errors, and wrote nothing', - 'Run the same command with `--yes`', - ], - [ - INTERRUPTED_EXIT_CODE, - `Interrupted (Ctrl-C). An apply or create already sent is finished and saved into the directory first, waiting up to ${DEFERRED_SECONDS}s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written`, - `\`${prefix} status --dir .\``, - ], +function sourceExitCodeRows(prefix: SourceCommandGroup): SourceExitCodeRow[] { + return [ + { + code: 0, + meaning: 'Applied or created, a dry run or check with no errors, or a clean status', + next: 'Go on', + help: { + 'workflows source apply': 'applied (or a clean dry run)', + 'workflows source create': 'created (or a clean dry run, or a starter written)', + 'workflows source get': 'written', + 'workspaces source apply': 'applied (or a clean dry run)', + 'workspaces source get': 'written', + }, + }, + { + code: 1, + meaning: + 'The source has errors (each printed as `file:line:column` with a hint), a locked block (`E_APPLY_LOCKED_BLOCK`), a name another workflow holds (`SOURCE_NAME_TAKEN`), a cancelled prompt, or the command failed', + next: 'Fix what it printed and run it again', + help: { + 'workflows source apply': + 'the source has errors (each printed as file:line:column with its frame and hint) or the command failed', + 'workflows source create': + 'the source has errors (each printed as file:line:column with its frame and hint), the workflow name is taken (SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the workspace its types were written for unless --workspace names another)', + 'workflows source get': + "the command failed, the directory is a workspace source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or it holds another workflow's source (SOURCE_WORKFLOW_MISMATCH) or edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force)", + 'workspaces source apply': + 'the source has errors (each printed as file:line:column with its frame and hint), or changes were held back because they use a declared secret the workspace has no value for or a declared credential key nothing binds to (the rest was applied: set the secret or connect the credential, then apply again), the directory was read from another workspace than the one --workspace names (SOURCE_OTHER_WORKSPACE: drop --workspace, or promote it with --to; without --workspace an apply targets the workspace the directory was read from), its base records binary files no get downloaded (SOURCE_BLOBS_NOT_FETCHED: get again first), or the command failed', + 'workspaces source get': + 'the command failed, the directory holds edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force), --workspace names another workspace than the one the directory was read from (SOURCE_OTHER_WORKSPACE), the directory is a workflow source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or a downloaded binary file does not match its listing (SOURCE_BLOB_MISMATCH: nothing was written)', + }, + }, + { + code: SOURCE_STALE_EXIT_CODE, + meaning: + 'Stale (`SOURCE_STALE`, or `E_APPLY_STALE` on a dry run): the server changed what this directory was read from, and nothing was written', + next: `\`${prefix} get --dir \`, reapply your edit there, dry-run, apply`, + help: { + 'workflows source apply': + 'the workflow changed since the directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry)', + 'workspaces source apply': + 'a resource changed or was deleted on the server since the directory was read (SOURCE_STALE: get it again into a fresh directory, reapply the edit, retry)', + }, + }, + { + code: SOURCE_LOCKED_EXIT_CODE, + meaning: + 'A workflow or its folder is locked (`E_APPLY_LOCKED`); a locked block exits 1 instead', + next: 'Ask someone to unlock it in the editor', + help: { + 'workflows source apply': + 'the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1, E_APPLY_LOCKED_BLOCK)', + 'workflows source create': 'the folder is locked', + 'workspaces source apply': + 'a resource (a workflow or its folder, a table) is locked (a locked block exits 1, E_APPLY_LOCKED_BLOCK)', + }, + }, + { + code: SOURCE_TOO_LARGE_EXIT_CODE, + meaning: 'The source is over the size limits (on a get, status or diff too)', + next: 'Split function code, or remove blocks or resources', + help: { + 'workflows source apply': 'the source is too large', + 'workflows source create': 'the source is too large', + 'workflows source get': 'the workflow is too large to print as source', + 'workspaces source apply': 'the directory is too large', + 'workspaces source get': + 'the workspace has more resources of a kind than one directory holds', + }, + }, + { + code: SOURCE_PARTIAL_EXIT_CODE, + meaning: + '`sim workspaces source apply` stopped part-way: what was written is saved into the directory', + next: 'Apply again until it exits 0', + help: { + 'workspaces source apply': `the apply stopped part-way on a server failure: what was written is saved into the directory, and applying again continues. An apply a resource refused part-way (stale, locked, forbidden) saves what was written too, and exits as that refusal: ${SOURCE_STALE_EXIT_CODE}, ${SOURCE_LOCKED_EXIT_CODE} or 1`, + }, + }, + { + code: RATE_LIMITED_EXIT_CODE, + meaning: + "Still rate limited (HTTP 429) after the CLI's retries (up to 6, at most a minute apart); nothing more was written", + next: 'Wait a minute and run the same command again', + help: { + 'workflows source apply': + "rate limited (HTTP 429) after the CLI's retries, so nothing was written: wait a minute and run it again", + 'workflows source create': + "rate limited (HTTP 429) after the CLI's retries, so nothing was created: wait a minute and run it again", + 'workflows source get': + "rate limited (HTTP 429) after the CLI's retries: wait a minute and run it again", + 'workspaces source apply': + "rate limited (HTTP 429) after the CLI's retries, before anything was written: wait a minute and apply again (binary files already uploaded are not sent again)", + 'workspaces source get': + "rate limited (HTTP 429) after the CLI's retries: nothing was written, the binary files already downloaded are kept in .sim/downloads, and running the get again downloads only the rest", + }, + }, + { + code: SOURCE_BUSY_EXIT_CODE, + meaning: + 'Busy: another apply is writing the workspace (`SOURCE_BUSY`), or another sim command is using this directory (`SOURCE_DIR_BUSY`); nothing was written', + next: 'Run it again once the other finished', + help: { + 'workflows source apply': WORKFLOW_DIR_BUSY, + 'workflows source create': WORKFLOW_DIR_BUSY, + 'workflows source get': WORKFLOW_DIR_BUSY, + 'workspaces source apply': + 'another apply is writing the workspace (SOURCE_BUSY), or another command is using the directory (SOURCE_DIR_BUSY): nothing was written, run the command again once the other finished', + 'workspaces source get': + '(busy, retry) another sim workspaces source command is using this directory (SOURCE_DIR_BUSY): nothing was read or written, run it again once it finished', + }, + }, + { + code: SOURCE_LOCAL_CHANGES_EXIT_CODE, + meaning: '`status`: only your edits, which applying writes', + next: 'Dry-run, then apply', + help: {}, + }, + { + code: SOURCE_DRIFT_EXIT_CODE, + meaning: '`status`: the server changed since the get (drift, or a conflict with your edits)', + next: 'Follow the `Next:` line it prints', + help: {}, + }, + { + code: AUTH_REQUIRED_EXIT_CODE, + meaning: + 'Not logged in, the login expired, or the server refused the credential (`AUTH_REQUIRED`); nothing was sent', + next: '`sim login`, or fix or unset `SIM_API_KEY` when the key comes from it; then the same command again', + help: { + 'workflows source get': GET_AUTH_REQUIRED, + 'workspaces source get': GET_AUTH_REQUIRED, + }, + }, + { + code: SOURCE_PLANNED_EXIT_CODE, + meaning: + '`apply` or `create` without `--yes` and no terminal to ask: it printed the plan, found no errors, and wrote nothing', + next: 'Run the same command with `--yes`', + help: { + 'workflows source apply': + 'without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it)', + 'workflows source create': + 'without --yes and with no terminal to ask: the plan was printed with no errors and nothing was created (run the same command with --yes)', + 'workspaces source apply': + 'without --yes and with no terminal to ask: the plan was printed with no errors and nothing was written (run the same command with --yes to apply it)', + }, + }, + { + code: INTERRUPTED_EXIT_CODE, + meaning: `Interrupted (Ctrl-C). An apply or create already sent is finished and saved into the directory first, waiting up to ${DEFERRED_SECONDS}s for its answer; a second Ctrl-C, or no answer by then, stops at once and prints that it may have been written`, + next: `\`${prefix} status --dir .\``, + help: {}, + }, ] +} + +/** + * Every exit code of the `workflows source` and `workspaces source` commands, as the Markdown + * table both directory guides (`.sim/AGENTS.md`) print. `prefix` is the command group the guide + * is for. + */ +export function sourceExitCodesTable(prefix: SourceCommandGroup): string { return [ '| Code | Meaning | Do |', '|---|---|---|', - ...rows.map(([code, meaning, next]) => `| ${code} | ${meaning} | ${next} |`), + ...sourceExitCodeRows(prefix).map( + ({ code, meaning, next }) => `| ${code} | ${meaning} | ${next} |` + ), ].join('\n') } +/** The width `--help` prose wraps at, as the hand-written sections around it do. */ +const HELP_WIDTH = 100 + +/** `text` broken into lines of at most {@link HELP_WIDTH} characters, between words. */ +function wrapHelp(text: string): string { + const lines: string[] = [] + let line = '' + for (const word of text.split(' ')) { + if (line && line.length + 1 + word.length > HELP_WIDTH) { + lines.push(line) + line = word + } else line = line ? `${line} ${word}` : word + } + return [...lines, line].join('\n') +} + +/** + * The exit-code section of `command`'s `--help`: the codes it exits with, in code order, each in + * the command's own words, from the same rows as {@link sourceExitCodesTable}. `lead` names the + * section (a get's codes apply only with `--dir`). + */ +export function sourceExitCodesHelp(command: SourceHelpCommand, lead = 'Exit codes'): string { + const prefix = command.startsWith('workflows') ? 'sim workflows source' : 'sim workspaces source' + const clauses = sourceExitCodeRows(prefix).flatMap(({ code, help }) => { + const text = help[command] + return text === undefined ? [] : [`${code} ${text}`] + }) + return `\n${wrapHelp(`${lead}: ${clauses.join('; ')}.`)}` +} + /** The exit codes every source write (apply, create) shares, appended to its help. */ export const SOURCE_WRITE_EXIT_CODES_HELP = ` Exit ${AUTH_REQUIRED_EXIT_CODE}: not logged in, or the server refused the credential (AUTH_REQUIRED); nothing was sent. diff --git a/packages/sim-cli/src/commands/protocol/source-status.ts b/packages/sim-cli/src/commands/protocol/source-status.ts index 85e6aa13f1a..efc6083e713 100644 --- a/packages/sim-cli/src/commands/protocol/source-status.ts +++ b/packages/sim-cli/src/commands/protocol/source-status.ts @@ -17,12 +17,17 @@ import { SOURCE_LOCAL_CHANGES_EXIT_CODE, SOURCE_TOO_LARGE_EXIT_CODE, } from '../../exit-codes' +import type { ApplyWorkspaceSourceResponse } from '../../generated/v2-api' import { noAnswer, SimApiError } from '../../http/client' import { printLine } from '../../output/io' import { styles } from '../../output/presentation' import { bytes, printDocument, sanitize } from '../../output/render' +import { CliUsageError, LocalRefusal } from './local-errors' import { type Diagnostic, findingsOutput } from './source-ui' +/** One change of a workspace apply's plan. */ +type WorkspacePlanChange = ApplyWorkspaceSourceResponse['data']['plan'][number] + /** Whether the server was asked: `checked`, `unreachable` (degraded to local), or `offline`. */ export type ServerCheck = 'checked' | 'unreachable' | 'offline' @@ -111,7 +116,7 @@ export function credentialRefused(error: unknown): boolean { * cancelled request included), which a status reports rather than hides. */ export function unreachableReason(error: unknown): string | null { - if (!(error instanceof SimApiError)) return null + if (!(error instanceof SimApiError) || error instanceof LocalRefusal) return null if (!noAnswer(error, UNREACHABLE_STATUSES) && !credentialRefused(error)) return null return /[.!?]$/.test(error.message) ? error.message : `${error.message}.` } @@ -382,10 +387,9 @@ export function pathFilter( } if (unmatched.size === 0) return } - throw new SimApiError( + throw new CliUsageError( `${[...unmatched].map((filter) => `"${sanitize(filter)}"`).join(', ')} selects no file in ${sanitize(dir)}; name a file or folder relative to --dir, or a path from the current directory inside it.`, - 0, - 'SOURCE_PATH_UNMATCHED' + { code: 'SOURCE_PATH_UNMATCHED' } ) }, } @@ -480,3 +484,11 @@ export async function readPlan(plan: () => Promise, dryRun: () => Promise< return dryRun() } } + +/** The changes that delete something (a resource, rows, documents, columns, a deployment). */ +export function destructiveChanges( + changes: ReadonlyArray> +): number { + return changes.filter((change) => change.action === 'delete' || change.destructive !== undefined) + .length +} diff --git a/packages/sim-cli/src/commands/protocol/source-tree.ts b/packages/sim-cli/src/commands/protocol/source-tree.ts index 4fae7aecd3b..7ef40906bd4 100644 --- a/packages/sim-cli/src/commands/protocol/source-tree.ts +++ b/packages/sim-cli/src/commands/protocol/source-tree.ts @@ -39,6 +39,7 @@ import type { GetSourceResourcesResponse, GetWorkflowSourceResponse } from '../. import { SimApiError } from '../../http/client' import type { OperationClient } from '../../runtime/called-operations' import { cliVersion } from '../../version' +import { SourceDirError } from './local-errors' import { printProtocolResult } from './result' import { FILE_CONCURRENCY, @@ -145,7 +146,7 @@ export async function cachedSourceCatalog(dir: string): Promise SDK_MAJOR ? `This workspace uses workflow SDK ${catalog.sdkMajor}, but this CLI writes SDK ${SDK_MAJOR} declarations. Upgrade the CLI (npm install -g sim@latest) and run the command again.` : `This workspace uses workflow SDK ${catalog.sdkMajor}, but this CLI writes SDK ${SDK_MAJOR} declarations. Install a CLI release for SDK ${catalog.sdkMajor} to work with this server.`, - 0, - 'SDK_MAJOR_MISMATCH' + { code: 'SDK_MAJOR_MISMATCH' } ) } diff --git a/packages/sim-cli/src/commands/protocol/source-ui.ts b/packages/sim-cli/src/commands/protocol/source-ui.ts index c39e7eb14f2..11f0b60d55c 100644 --- a/packages/sim-cli/src/commands/protocol/source-ui.ts +++ b/packages/sim-cli/src/commands/protocol/source-ui.ts @@ -4,8 +4,9 @@ * write nobody confirmed, and an API refusal re-thrown with the exit status a script branches on. */ import { isRecordLike, toRecord } from '@sim/utils/object' +import { boundFindingText, type RefusalClass } from '@sim/workflow-code/diagnostics' import { SOURCE_LOCK_FILE } from '@sim/workflow-code/limits' -import { type FlatDiagnostic, type RefusalClass, refusalClass } from '@sim/workspace-code' +import { type FlatDiagnostic, refusalClass } from '@sim/workspace-code' import type { Command } from 'commander' import { printError } from '#sim-cli/output/io' import { askLine } from '#sim-cli/terminal/line-input' @@ -23,6 +24,7 @@ import type { ApplyWorkspaceSourceResponse, } from '../../generated/v2-api' import { noAnswer, REQUEST_CANCELLED, SimApiError } from '../../http/client' +import { LocalRefusal } from './local-errors' /** One finding of a source write, as the workflow and workspace answers both carry it. */ type WorkflowDiagnostic = ApplyWorkflowSourceResponse['data']['diagnostics'][number] @@ -192,7 +194,11 @@ export interface FindingsOutput { result(diagnostics: readonly Diagnostic[]): { diagnostics?: ResultDiagnostic[] } } -/** The {@link FindingsOutput} of `format`; `verbose` keeps info findings in a machine result. */ +/** + * The {@link FindingsOutput} of `format`; `verbose` keeps info findings in a machine result. It is + * the CLI's output edge, where each finding's message and hint are cut (`boundFindingText`): a + * finding an offline check made is as long as its source made it until here. + */ export function findingsOutput(format: OutputFormat, verbose = false): FindingsOutput { const printed = new Set() const machine = isMachine(format) @@ -200,9 +206,14 @@ export function findingsOutput(format: OutputFormat, verbose = false): FindingsO machine, print(diagnostics) { const unseen = diagnostics.filter((diagnostic) => fresh(printed, findingKey(diagnostic))) - if (!machine) printDiagnostics(unseen) + if (!machine) printDiagnostics(unseen.map((diagnostic) => boundFindingText(diagnostic))) }, - result: (diagnostics) => resultDiagnostics(format, diagnostics, verbose), + result: (diagnostics) => + resultDiagnostics( + format, + diagnostics.map((diagnostic) => boundFindingText(diagnostic)), + verbose + ), } } @@ -277,6 +288,7 @@ const LOST_ANSWER_STATUSES = new Set([502, 504]) * that gave up, a cancellation), so it may or may not have been stored. */ function answerLost(error: SimApiError): boolean { + if (error instanceof LocalRefusal) return false return noAnswer(error, LOST_ANSWER_STATUSES) || error.code === REQUEST_CANCELLED } diff --git a/packages/sim-cli/src/commands/protocol/source-write.ts b/packages/sim-cli/src/commands/protocol/source-write.ts new file mode 100644 index 00000000000..f4bee3ac949 --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/source-write.ts @@ -0,0 +1,109 @@ +/** + * The one consent-and-write flow of every source write (`workflows source apply` and `create`, + * `workspaces source apply`): preview, refuse on errors, stop at the plan or ask, then send and + * save the write with interrupts held, so the commands cannot drift in what they print, ask or + * exit with. + */ + +import { DEFERRED_INTERRUPT_BOUND_MS, deferInterrupts } from '#sim-cli/terminal/interrupts' +import { setSoftExitCode } from '../../embed-context' +import { SourceDirError } from './local-errors' +import { + confirm, + type Diagnostic, + planNext, + refusedExitCode, + stopAtPlan, + type WriteMode, +} from './source-ui' + +/** The next step a result names: a command and why to run it. */ +export interface NextStep { + command: string + reason: string +} + +/** What a write answers with, as far as the flow reads it: its findings. */ +interface WriteAnswer { + diagnostics: ReadonlyArray> +} + +/** One source write as {@link runSourceWrite} drives it. */ +export interface SourceWriteFlow { + mode: WriteMode + /** The dry run that previews the write, and is the whole of a `--dry-run`. */ + preview(): Promise + /** Prints a preview: its findings (and a workspace's plan). */ + show(preview: D): void + /** Whether a preview's errors refuse the write, which is then reported and goes no further. */ + refuses(preview: D): boolean + /** Prints what a preview that refuses nothing would write, before the plan stop or the question. */ + explain?(preview: D): void + /** Runs once the write may go ahead, before it is sent (the preview, or `null` with `--yes`). */ + beforeCommit?(preview: D | null): Promise + /** Sends the write and prints its findings. */ + commit(): Promise + /** Saves a committed write's answer into the directory, when it stored anything. */ + save(answer: D): Promise + /** Prints `answer` as the result, with the next step a plan names. */ + report(answer: D, next?: NextStep): void + /** The confirmation question a person at a terminal answers. */ + question: string + /** This write as a command, without `--yes`: what a `plan` run names next with it. */ + command: string + /** What an interrupt during the send and save prints. */ + interrupt: { + verb: string + dir: string + noun: 'workflow' | 'workspace' + /** The next step when the write's answer never came back, so it may have been stored. */ + ifAnswerLost: string + } +} + +/** + * Runs `flow`: in every mode but `write`, previews it and prints the preview, exiting with the + * refusal's status on errors; a `dry-run` or a refused preview is reported there, a `plan` stops + * with {@link stopAtPlan}, and `ask` asks. Then sends the write and saves its answer with Ctrl-C + * held ({@link deferInterrupts}), so an interrupt never strands a stored write the directory did + * not record. Returns the committed answer for the caller to finish and report, or `null` when + * the flow ended at the preview, already reported. + */ +export async function runSourceWrite( + flow: SourceWriteFlow +): Promise { + let preview: D | null = null + if (flow.mode !== 'write') { + preview = await flow.preview() + flow.show(preview) + if (preview.diagnostics.some((finding) => finding.severity === 'error')) { + setSoftExitCode(refusedExitCode(preview.diagnostics)) + } + if (flow.mode === 'dry-run' || flow.refuses(preview)) { + flow.report(preview) + return null + } + flow.explain?.(preview) + if (flow.mode === 'plan') { + stopAtPlan() + flow.report(preview, planNext(flow.command)) + return null + } + if (!(await confirm(flow.question))) { + throw new SourceDirError('Nothing was applied.', { code: 'CANCELLED' }) + } + } + await flow.beforeCommit?.(preview) + const { verb, dir, noun, ifAnswerLost } = flow.interrupt + return deferInterrupts( + async () => { + const answer = await flow.commit() + await flow.save(answer) + return answer + }, + { + waiting: `Interrupted: the ${verb} was already sent. Waiting up to ${DEFERRED_INTERRUPT_BOUND_MS / 1000}s for its answer and saving it into ${dir}, so the directory matches the ${noun}; press Ctrl-C again to stop now.`, + abandoned: `Stopped before the ${verb} answered. ${ifAnswerLost}`, + } + ) +} diff --git a/packages/sim-cli/src/commands/protocol/workflow-dir.ts b/packages/sim-cli/src/commands/protocol/workflow-dir.ts new file mode 100644 index 00000000000..edfe121dbe4 --- /dev/null +++ b/packages/sim-cli/src/commands/protocol/workflow-dir.ts @@ -0,0 +1,385 @@ +/** + * A workflow source directory (`workflow.ts`, its function files, `.sim/lock.json` and + * `.sim/base.json`): what an apply, status or check reads from it, and what a get or create writes + * into it, so no command reads another's directory through that command's file. + */ + +import { mapWithConcurrency } from '@sim/utils/concurrency' +import type { BlockCatalog } from '@sim/workflow-code' +import { parseSourceBase, parseWorkflowLock, type WorkflowLockV1 } from '@sim/workflow-code' +import { codeFileReferences } from '@sim/workflow-code/code-files/references' +import { + CODE_FILE_EXTENSIONS, + SOURCE_BASE_FILE, + SOURCE_ENTRY_FILE, + SOURCE_LOCK_FILE, +} from '@sim/workflow-code/limits' +import { + SDK_GUIDE_FILE, + SOURCE_COMPILER_OPTIONS, + typecheckCommand, +} from '@sim/workflow-sdk/codegen' +import { SOURCE_PLANNED_EXIT_CODE, SOURCE_STALE_EXIT_CODE } from '../../exit-codes' +import type { ApplyWorkflowSourceBody, GetWorkflowSourceResponse } from '../../generated/v2-api' +import type { OperationClient } from '../../runtime/called-operations' +import { SourceDirError } from './local-errors' +import { + assertSourceDirKind, + FILE_CONCURRENCY, + lockCodeFiles, + readPreviousLock, + readSourceText, + readStarterWorkspace, + SOURCE_SDK_DIR as SDK_DIR, + SOURCE_EXAMPLES_DIR, + SOURCE_STARTER_FILE, + sourceRelativePath, + updateSourceDir, + writeBack, +} from './source-dir' +import { sourceExitCodesTable } from './source-exit-codes' +import { + assertSdkMajor, + finishTree, + loadCatalog, + loadWorkspaceTypes, + refreshTsconfigs, + type SourceTree, + type SourceTreeProfile, + sourceTreeChange, + TSCONFIG_FULL_FILE, + TSCONFIG_MARKER, + WORKSPACE_TYPES_FILE, + type WorkspaceTypes, +} from './source-tree' +import { storedEdgesWrite } from './stored-edges' + +type SourceBase = ApplyWorkflowSourceBody['base'] + +/** The only files an apply sends beside the entry: function code in a language a block runs. */ +const CODE_FILE_EXTENSION = new RegExp(`\\.(?:${Object.values(CODE_FILE_EXTENSIONS).join('|')})$`) + +/** + * The text of a file the directory must hold, or a refusal that says how to write it: `hint`, or + * for a starter no workflow was created from yet, the create that makes it one. + */ +async function requireText(directory: string, path: string, hint: string): Promise { + const text = await readSourceText(directory, path) + if (text !== null) return text + const starterWorkspace = await readStarterWorkspace(directory) + throw new SourceDirError( + `${directory} has no ${path}. ${ + starterWorkspace === null + ? hint + : `It holds a starter no workflow was created from yet: run \`${createCommand(directory, starterWorkspace)} --dry-run\`, then the same with --yes.` + }` + ) +} + +async function readJson(directory: string, path: string, hint: string): Promise { + const text = await requireText(directory, path, hint) + try { + return JSON.parse(text) + } catch { + throw new SourceDirError(`${path} in ${directory} is not valid JSON. ${hint}`) + } +} + +/** + * The function-code files the source can reference: every `codeFile(…)` call the entry file makes, + * found by parsing it as compile does (an aliased import or spacing before the parenthesis still + * counts; a comment or string that spells one does not), plus every `codeFile` the lock records, + * limited to `.js`, `.py` and `.sh` files inside the directory. Only those are sent, so a + * reference to any other file never uploads it; the server reports a referenced file that is + * missing or of another kind. + */ +function codeFilePaths(entry: string, lock: WorkflowLockV1 | null): string[] { + const candidates = lockCodeFiles(lock) + for (const literal of codeFileReferences(entry)) { + const path = sourceRelativePath(literal) + if (path) candidates.add(path) + } + return [...candidates].filter((path) => CODE_FILE_EXTENSION.test(path)) +} + +const GET_HINT = 'Write the directory with `sim workflows source get` first.' + +/** + * The base an apply sends, refused when it is not the named workflow's: a directory written for + * one workflow and applied to another would replace that workflow's draft with a stranger's + * source. Without a `workflowId` the apply targets the workflow the base names. + */ +export async function readBase( + directory: string, + workflowId: string | undefined +): Promise { + await assertSourceDirKind(directory, 'workflow') + const parsed = parseSourceBase(await readJson(directory, SOURCE_BASE_FILE, GET_HINT)) + if ('problem' in parsed) { + throw new SourceDirError( + `${SOURCE_BASE_FILE} in ${directory} is not a base: ${parsed.problem} ${GET_HINT}` + ) + } + const { base } = parsed + if (workflowId !== undefined && base.workflowId !== workflowId) { + throw new SourceDirError( + `${directory} holds the source of workflow ${base.workflowId}, not ${workflowId}. Apply it to ${base.workflowId}, or write this workflow's source with \`sim workflows source get ${workflowId} --dir \`.`, + { code: 'SOURCE_WORKFLOW_MISMATCH' } + ) + } + return base +} + +/** The directory's lock, refused with every problem the lock parser finds. */ +export async function readLock(directory: string): Promise { + const parsed = parseWorkflowLock(await readJson(directory, SOURCE_LOCK_FILE, GET_HINT)) + if (parsed.lock) return parsed.lock + const problems = parsed.diagnostics.map( + (finding) => `${finding.path === '' ? '' : `${finding.path}: `}${finding.message}` + ) + throw new SourceDirError( + `${SOURCE_LOCK_FILE} in ${directory} is not a valid lock: ${problems.join('; ')}` + ) +} + +/** The directory's `workflow.ts`, or a refusal naming the command that writes one. */ +export function requireEntry(directory: string): Promise { + return requireText(directory, SOURCE_ENTRY_FILE, GET_HINT) +} + +/** + * The files a write sends: the entry file and the code files it or the lock references. A + * referenced file that is missing is left out, and the server reports it with its source line. + */ +export async function readSourceFiles( + directory: string, + lock: WorkflowLockV1 | null +): Promise> { + const entry = await requireEntry(directory) + const files: Record = { [SOURCE_ENTRY_FILE]: entry } + const paths = codeFilePaths(entry, lock) + const texts = await mapWithConcurrency(paths, FILE_CONCURRENCY, (path) => + readSourceText(directory, path) + ) + paths.forEach((path, index) => { + const text = texts[index] + if (text !== null) files[path] = text + }) + return files +} + +/** The create command for `dir` in `workspaceId`, as the next step a starter prints. */ +export function createCommand(dir: string, workspaceId: string): string { + return `sim workflows source create --dir ${dir} --workspace ${workspaceId}` +} + +/** The public guide to workflow source, which AGENTS.md points an agent at. */ +const SOURCE_DOCS_URL = 'https://docs.sim.ai/workflows/code' + +type WorkflowSource = GetWorkflowSourceResponse['data'] + +/** + * A `tsconfig.json` that lets an editor or `tsc -p` resolve `@sim/workflow-sdk` to the written + * declarations and type-checks `workflow.ts` with the SDK files `sdkFiles`. + */ +function workflowTsconfig(sdkFiles: readonly string[]): string { + return `${TSCONFIG_MARKER}\n${JSON.stringify( + { + compilerOptions: { + ...SOURCE_COMPILER_OPTIONS, + skipLibCheck: true, + paths: { '@sim/workflow-sdk': [`./${SDK_DIR}/core.d.ts`] }, + }, + include: [SOURCE_ENTRY_FILE, ...sdkFiles], + }, + null, + 2 + )}\n` +} + +/** + * The reference below the primer in `.sim/AGENTS.md`, by task: the edit loop in detail, a new or + * forked workflow, the file shape, lookups, the rules and every exit code. The docs page has the + * rest. + */ +const AGENTS_GUIDE = `# Reference + +The primer above is enough to edit; read a section here when a step needs it. + +## Edit loop in detail + +1. Edit \`${SOURCE_ENTRY_FILE}\` (and \`functions/*\`). \`${SOURCE_EXAMPLES_DIR}/\` holds correct examples to copy from: \`cookbook.ts\` (agent with structured output, condition, function, table query and insert, knowledge search, Slack, human in the loop, loop, response), \`integrations.ts\` (webhook trigger, weekday cron schedule, HTTP request with headers, table update by row id, Gmail, an error path) and \`agents.ts\` (router, agent tools: custom tool, MCP tool, integration operation, child workflow; run a child workflow; parallel). Their resource keys are examples; this workspace's are in \`${WORKSPACE_TYPES_FILE}\`. +2. \`sim workflows source check --dir .\`: compiles offline with the server's compiler, in milliseconds. It catches what \`tsc\` cannot (duplicate names, cycles, literal secrets), and fails (exit 1) on a missing required field, an unknown option or output path, a wrong value kind, an out-of-range value, an unwired condition or router branch or a literal secret in a block you added (\`--strict\`: in every block). It also adds the blocks the source now uses to \`tsconfig.json\`. +3. \`${typecheckCommand()}\`: about a second; catches what step 2 and the dry run do not (such as a reference to a start input the trigger does not declare). It runs the TypeScript release the declarations are built for (the directory has no \`package.json\`; plain \`bunx tsc\` would run whatever version it finds). \`tsconfig.json\` includes only the blocks the source used at the last check, get or apply: a builder from another block fails with \`has no exported member\` until you run step 2 (or use \`${typecheckCommand(TSCONFIG_FULL_FILE)}\`, every current block). The dry run is the authority: a printed line you did not edit that \`tsc\` flags and the dry run accepts is fine. +4. \`sim workflows source status --dir .\`: the files you changed (\`sim workflows source diff --dir .\` shows the edits), whether the workflow changed on the server since the get, and the command to run next (\`--offline\`: skip the server). Then \`sim workflows source apply --dir . --dry-run\` until it exits 0. +5. \`sim workflows source apply --dir . --yes\`: saves the draft, refusing errors, block deletions without \`--allow-delete\` and a stale directory, so it needs no separate dry run (without \`--yes\` and a terminal it prints the plan and exits ${SOURCE_PLANNED_EXIT_CODE}). \`--deploy\` also deploys it; without it, a deployed workflow's next \`sim workflows deploy \` is printed. It then rewrites \`${SOURCE_ENTRY_FILE}\` in canonical form (order, quoting, defaults left out). +6. Exit ${SOURCE_STALE_EXIT_CODE} (\`SOURCE_STALE\`, HTTP 409, or \`E_APPLY_STALE\` on a dry run): the workflow changed and nothing was saved. Run \`sim workflows source get --dir \`, reapply your edit there, dry-run, apply. A get into this directory refuses to overwrite your unapplied edits unless you pass \`--force\`. Exit ${SOURCE_STALE_EXIT_CODE} right after an apply that ended without an answer (a timeout, a dropped connection) means the draft changed, by that apply or by someone else: get into a fresh directory and compare. Only when it holds your edit, run \`sim workflows source get --dir . --force\`; otherwise reapply your edit in the fresh directory. + +Editing the \`workflow('…')\` name or description renames or redescribes the workflow; a name another workflow in its folder holds is refused with \`E_NAME_TAKEN\`. Lines starting \`note:\` explain something the write did differently than the source says; read them. + +## A new workflow, or a fork + +\`sim workflows source create --dir --workspace \` (without \`--dry-run\` or \`--yes\`) on a directory without \`${SOURCE_ENTRY_FILE}\` writes a starter \`${SOURCE_ENTRY_FILE}\`, the types of that workspace, \`tsconfig.json\` and this guide, records the workspace in \`${SOURCE_STARTER_FILE}\`, and creates nothing. Until the workflow exists there is no \`${SOURCE_BASE_FILE}\`, so \`status\`, \`diff\` and \`apply\` refuse the directory: edit the starter, check and type-check it, then \`sim workflows source create --dir --workspace --dry-run\` and the same with \`--yes\` (the starter prints both with its workspace filled in): it prints the new workflow id and writes the lock files. A create goes to the starter's workspace whatever the profile's default, unless \`--workspace\` names another. Exit 1 with \`SOURCE_NAME_TAKEN\` means another workflow has that name: pass \`--name\`. To fork a workflow, copy its directory, delete the copy's \`.sim/\` and run \`sim workflows source create --dir --workspace --name --yes\`. + +## File shape + +\`\`\`ts +// @sim-sdk 1 +import { response, trigger, workflow } from '@sim/workflow-sdk' + +export default workflow('Workflow name', {}, () => { + const start = trigger.start('Start', {}) + const done = response('Done', { body: { ok: true } }) + + start.to(done) +}) +\`\`\` + +The \`// @sim-sdk 1\` header must be the first line, and the file has exactly one \`export default workflow(name, options, () => { ... })\`; \`options\` holds \`description\` and \`variables\` and may be \`{}\`. + +## Finding a builder and its types + +- \`sim workflows source params \`: one builder's params (required first, enums, defaults) and outputs; a namespace (\`gmail\`) lists its builders; a miss names the closest. Builders are named for the service, not the block type: \`gmail_v2\` is \`gmail.*\`, \`workflow_input\` is \`runWorkflow\`, the HTTP request block is \`api\`, and \`legacy.*\` builders are old versions not to write. +- \`${SDK_DIR}/core.d.ts\`: the SDK (\`workflow\`, \`t\`, \`env\`, \`credential\`, \`table\`, \`loop\`, \`parallel\`, \`BlockMeta\`). +- \`${SDK_DIR}/blocks/.d.ts\`: the builders of a block the source uses, with their params and outputs (check adds the file of a block an edit starts using). Do not read them, or \`${SDK_DIR}/models.d.ts\`, whole: \`params\` prints the part you need, and \`grep\` finds a model id. +- \`${WORKSPACE_TYPES_FILE}\`: this workspace's tables, knowledge bases, credentials, workflows and env names. + +## Rules + +- Beside the header and the imports from \`@sim/workflow-sdk\`, the file holds only the default-exported workflow. Its callback holds block declarations (\`const x = builder('Name', params)\`) and wiring (\`x.to(y)\`). Helpers, loops, conditionals and other imports have no stored form, so the compiler refuses them: use \`loop.forEach\`, \`condition\` and \`runWorkflow\` blocks instead. +- Logic that does not fit a block goes in a function block with \`code: codeFile('./functions/name.js')\` (references as the primer says). +- Hosted Sim supplies an agent's model key. On self-hosted Sim the agent needs \`apiKey: env.NAME\` (a secret the workspace holds); without it \`check\` reports \`W_REQUIRED_MISSING: apiKey is required\`. +- A child workflow's output is \`c.result\`, untyped: its \`response\` block's \`{ data, status, headers }\`, so a field of the child's response body is \`c.result.data.field\` (\`${SDK_DIR}/${SDK_GUIDE_FILE}\` "How it runs" has the rest). +- A table block's row is \`{ id, data }\`: read a cell as \`row.data.column\` (in a loop over \`query.rows\`, \`p.currentItem.data.column\`). +- Keep \`${SOURCE_LOCK_FILE}\` and \`${SOURCE_BASE_FILE}\` as written: they bind each block to its stored id and make the next apply safe. + +## Diagnostics and exit codes + +Findings print as \`file:line:column severity CODE: message\`, then the code frame and a hint. \`E_*\` refuses the whole apply; \`W_*\` is saved as written and names a real problem. \`check\` prints the \`W_*\` codes it fails on as \`error\` on a block you added; the apply still saves them as warnings. A path no output has (\`W_UNKNOWN_OUTPUT_PATH\`), in a block you added or changed, is the error \`E_APPLY_UNRESOLVED_REF\` in both, whatever \`--strict\` says; a path the run may still resolve (\`W_OUTPUT_PATH_UNDECLARED\`: an input the start does not declare, \`x.error\` with no error edge) stays a warning. Exit codes of every \`workflows source\` command: + +${sourceExitCodesTable('sim workflows source')} + +The full guide: ${SOURCE_DOCS_URL} +` + +/** The printed source a tree write saves: a get's answer, or an applied create's. */ +export type PrintedSource = Pick< + WorkflowSource, + 'workflowId' | 'workspaceId' | 'files' | 'lock' | 'base' +> + +/** + * Writes the source tree in one staged update. Declarations are regenerated only when the index + * on disk names another catalog or another generator, and replaced whole so a block the catalog + * dropped loses its file. A refused manifest removes the `workspace.d.ts` an earlier get wrote, + * function files the new lock no longer names are pruned, `.gitattributes` gains the LF lines it + * lacks, and a `tsconfig.json` the CLI did not write is left as it is. + */ +async function writeSourceTree( + dir: string, + source: PrintedSource, + catalog: BlockCatalog, + workspaceTypes: WorkspaceTypes | null, + blockTypes: readonly string[] +): Promise { + const [previousLock, tree] = await Promise.all([ + readPreviousLock(dir), + sourceTreeChange(dir, catalog, workspaceTypes, WORKFLOW_TREE, { + sources: [source.files[SOURCE_ENTRY_FILE] ?? ''], + blockTypes, + }), + ]) + const edges = storedEdgesWrite(source, catalog) + const written = await writeBack( + dir, + source, + previousLock, + edges ? { ...tree.change, write: [...tree.change.write, edges] } : tree.change + ) + return finishTree(dir, written, workspaceTypes, tree) +} + +/** The workflow directory's tree beside the printed source. */ +export const WORKFLOW_TREE: SourceTreeProfile = { + kind: 'workflow', + agents: AGENTS_GUIDE, + tsconfig: workflowTsconfig, +} + +/** The `workflow.ts` a new directory starts from: the smallest workflow that compiles. */ +const STARTER_SOURCE = `// @sim-sdk 1 +import { response, trigger, workflow } from '@sim/workflow-sdk' + +export default workflow('New workflow', { description: 'Describe what this workflow does.' }, () => { + const start = trigger.start('Start', { inputFormat: { message: 'string' } }) + const done = response('Done', { body: { received: start.message } }) + + start.to(done) +}) +` + +/** + * Starts a new source directory: a starter `workflow.ts` plus the tree a get writes (the + * declarations for the workspace catalog and its resources, `tsconfig.json`, the agent guide), + * with no lock or base, since no workflow exists yet, and {@link SOURCE_STARTER_FILE} naming the + * workspace those declarations are for. Only for a directory with no `workflow.ts`. + */ +export async function writeSourceScaffold( + client: OperationClient<'getSourceCatalog' | 'getSourceResources'>, + dir: string, + workspaceId: string +): Promise { + const [catalog, workspaceTypes] = await Promise.all([ + loadCatalog(client, { workspaceId, blockTypes: [] }), + loadWorkspaceTypes(client, workspaceId), + ]) + assertSdkMajor(catalog) + const tree = await sourceTreeChange(dir, catalog, workspaceTypes, WORKFLOW_TREE, { + sources: [STARTER_SOURCE], + }) + const write = [ + [SOURCE_ENTRY_FILE, STARTER_SOURCE] as const, + [SOURCE_STARTER_FILE, `${JSON.stringify({ workspaceId }, null, 2)}\n`] as const, + ...tree.change.write, + ] + await updateSourceDir(dir, { ...tree.change, write }) + return finishTree(dir, write.length, workspaceTypes, tree) +} + +/** + * Writes a printed source with everything that makes the directory editable: the declarations + * for the workspace catalog and its resources, `tsconfig.json`, the agent guide and the LF lines. + * Shared by `get --dir` and `create --dir`, so a created workflow's directory is the one a get + * would have written. + */ +export async function writeSourceDirectory( + client: OperationClient<'getSourceCatalog' | 'getSourceResources'>, + dir: string, + source: PrintedSource, + blockTypes: readonly string[] +): Promise { + const [catalog, workspaceTypes] = await Promise.all([ + loadCatalog(client, { + workspaceId: source.workspaceId, + workflowId: source.workflowId, + catalogHash: source.base.catalogHash, + blockTypes: [...blockTypes], + }), + loadWorkspaceTypes(client, source.workspaceId), + ]) + assertSdkMajor(catalog) + return writeSourceTree(dir, source, catalog, workspaceTypes, blockTypes) +} + +/** + * After an apply, points the CLI's tsconfig files at the blocks the applied source uses, so a + * block the edit added is type-checked by `tsconfig.json` from then on. + */ +export async function refreshWorkflowTsconfigs( + dir: string, + source: Pick +): Promise { + await refreshTsconfigs(dir, source.workspaceId, WORKFLOW_TREE, { + sources: [source.files[SOURCE_ENTRY_FILE] ?? ''], + }) +} diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-apply.test.ts b/packages/sim-cli/src/commands/protocol/workflow-source-apply.test.ts index 2a287718dce..c34537eecdd 100644 --- a/packages/sim-cli/src/commands/protocol/workflow-source-apply.test.ts +++ b/packages/sim-cli/src/commands/protocol/workflow-source-apply.test.ts @@ -1,4 +1,12 @@ -import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from 'node:fs' import { hostname, tmpdir } from 'node:os' import { join } from 'node:path' import { toRecord } from '@sim/utils/object' @@ -344,6 +352,24 @@ describe('sim workflows source apply', () => { ) }) + it('neither reads nor rewrites the workspace types on a plan', async () => { + setTerminal(false) + mkdirSync(join(dir, '.sim/sdk'), { recursive: true }) + writeFileSync(join(dir, WORKSPACE_TYPES_FILE), '// types the last get wrote\n') + writeFileSync(join(dir, '.sim/sdk/workspace.manifest.json'), '{}\n') + request.mockResolvedValue(outcome({ dryRun: true })) + + await apply(WORKFLOW_ID, '--dir', dir) + + expect(dryRunFlags()).toEqual([true]) + expect(request.mock.calls.map((call) => call[0])).not.toContainEqual( + expect.stringContaining('/source/resources') + ) + expect(readFileSync(join(dir, WORKSPACE_TYPES_FILE), 'utf8')).toBe( + '// types the last get wrote\n' + ) + }) + it('at a terminal, shows the dry run and applies once the person confirms', async () => { setTerminal(true) request.mockImplementation(async (_path: string, options: { query?: { dryRun?: boolean } }) => @@ -1060,6 +1086,47 @@ describe('sim workflows source apply in a workflow folder of a workspace directo }) }) +describe('a standalone workflow directory', () => { + it('refuses a second apply while the first holds the directory, sending nothing', async () => { + setTerminal(false) + let second: unknown = null + request.mockResolvedValue(outcome()) + request.mockImplementationOnce(async () => { + second = await apply('--dir', dir, '--yes').then( + () => null, + (error: unknown) => error + ) + return outcome() + }) + + await apply('--dir', dir, '--yes') + + expect(second).toMatchObject({ code: 'SOURCE_DIR_BUSY', exitCode: 10 }) + expect(request).toHaveBeenCalledTimes(1) + expect(existsSync(join(dir, '.sim/command.lock'))).toBe(false) + }) + + it('dry-runs and plans a directory no one may write, and refuses to apply it, sending nothing', async () => { + setTerminal(false) + request.mockResolvedValue(outcome({ dryRun: true })) + for (const path of ['', '.sim']) chmodSync(join(dir, path), 0o555) + try { + await apply('--dir', dir, '--dry-run') + await apply('--dir', dir) + expect(dryRunFlags()).toEqual([true, true]) + + const refused = await apply('--dir', dir, '--yes').then( + () => null, + (error: unknown) => error + ) + expect(refused).toMatchObject({ code: 'SOURCE_DIR_UNWRITABLE' }) + expect(request).toHaveBeenCalledTimes(2) + } finally { + for (const path of ['', '.sim']) chmodSync(join(dir, path), 0o755) + } + }) +}) + describe('the code files an apply sends', () => { function writeFunctions(...names: string[]) { mkdirSync(join(dir, 'functions'), { recursive: true }) @@ -1277,6 +1344,22 @@ describe('sim workflows source create on a directory with no workflow.ts', () => .filter((path) => path.endsWith('/workflows/source')) } + it('plans a starter without reading or rewriting its workspace types', async () => { + setTerminal(false) + vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config')) + const empty = join(dir, 'new') + answerStarterAndCreate() + await create('--dir', empty) + request.mockClear() + + await create('--dir', empty) + + expect(createCalls()).toHaveLength(1) + expect(request.mock.calls.map((call) => call[0])).not.toContainEqual( + expect.stringContaining('/source/resources') + ) + }) + it('records the workspace the starter was typed for and names it in the next step', async () => { setTerminal(false) vi.stubEnv('SIM_CONFIG_DIR', join(dir, 'config')) diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-apply.ts b/packages/sim-cli/src/commands/protocol/workflow-source-apply.ts index 2e68fbf376e..0e35fbabf7f 100644 --- a/packages/sim-cli/src/commands/protocol/workflow-source-apply.ts +++ b/packages/sim-cli/src/commands/protocol/workflow-source-apply.ts @@ -1,16 +1,7 @@ -import { mapWithConcurrency } from '@sim/utils/concurrency' import { toRecord } from '@sim/utils/object' -import { parseSourceBase, parseWorkflowLock, type WorkflowLockV1 } from '@sim/workflow-code' -import { codeFileReferences } from '@sim/workflow-code/code-files/references' -import { - SOURCE_BASE_FILE as BASE_FILE, - CODE_FILE_EXTENSIONS, - SOURCE_ENTRY_FILE as ENTRY_FILE, - SOURCE_LOCK_FILE as LOCK_FILE, -} from '@sim/workflow-code/limits' +import { SOURCE_ENTRY_FILE as ENTRY_FILE } from '@sim/workflow-code/limits' import type { Command } from 'commander' import { printError } from '#sim-cli/output/io' -import { DEFERRED_INTERRUPT_BOUND_MS, deferInterrupts } from '#sim-cli/terminal/interrupts' import { setSoftExitCode } from '../../embed-context' import type { ApplyWorkflowSourceBody, @@ -22,73 +13,49 @@ import { SimApiError } from '../../http/client' import { apiCommand, type OperationClient } from '../../runtime/called-operations' import { searchTasks } from '../search' import { gitProvenance } from './git-provenance' +import { CliUsageError, SourceDirError } from './local-errors' import { printProtocolResult } from './result' import { absoluteDirOption, assertSendableCodeFiles, assertSourceDirKind, - FILE_CONCURRENCY, - lockCodeFiles, readSourceText, readStarterWorkspace, SOURCE_AGENTS_FILE, SOURCE_STARTER_FILE, - sourceRelativePath, updateSourceDir, - WORKSPACE_FOLDER_BUSY_HELP, - withWorkspaceFolderLock, + withSourceCommandLock, writeBack, } from './source-dir' -import { SOURCE_WRITE_EXIT_CODES_HELP } from './source-exit-codes' +import { SOURCE_WRITE_EXIT_CODES_HELP, sourceExitCodesHelp } from './source-exit-codes' import { cachedCatalogByHash, printTreeResult, refreshWorkflowWorkspaceTypes } from './source-tree' import { type CreatedResource, - confirm, createdText, type FindingsOutput, findingsOutput, fresh, - planNext, printRefusal, refusedExitCode, shellWord, - stopAtPlan, verboseFlag, + type WriteMode, writeMode, } from './source-ui' +import { type NextStep, runSourceWrite } from './source-write' import { storedEdgesWrite } from './stored-edges' import { + createCommand, + readBase, + readLock, + readSourceFiles, refreshWorkflowTsconfigs, writeSourceDirectory, writeSourceScaffold, -} from './workflow-source-get' - -/** The only files an apply sends beside the entry: function code in a language a block runs. */ -const CODE_FILE_EXTENSION = new RegExp(`\\.(?:${Object.values(CODE_FILE_EXTENSIONS).join('|')})$`) - -const APPLY_EXIT_CODES_HELP = ` -Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as -file:line:column with its frame and hint) or the command failed; 5 the workflow changed since the -directory was read (SOURCE_STALE, on a dry run too: get it again, reapply the edit, retry); -6 the workflow or its folder is locked (E_APPLY_LOCKED; a locked block exits 1, -E_APPLY_LOCKED_BLOCK); 7 the source is too large; 9 rate limited (HTTP 429) after the CLI's -retries, so nothing was written: wait a minute and run it again; 14 without --yes and with no -terminal to ask: the plan was printed with no errors and nothing was written (run the same -command with --yes to apply it).` - -const CREATE_EXIT_CODES_HELP = ` -Exit codes: 0 created (or a clean dry run, or a starter written); 1 the source has errors (each -printed as file:line:column with its frame and hint), the workflow name is taken -(SOURCE_NAME_TAKEN: pass --name), or the command failed (a starter is created in the -workspace its types were written for unless --workspace names another); 6 the folder is locked; 7 the -source is too large; 9 rate limited (HTTP 429) after the CLI's retries, so nothing was created: -wait a minute and run it again; 14 without --yes and with no terminal to ask: the plan was -printed with no errors and nothing was created (run the same command with --yes).` +} from './workflow-dir' type SourceApplyData = ApplyWorkflowSourceResponse['data'] -type SourceBase = ApplyWorkflowSourceBody['base'] - interface SourceWriteOptions { dir: string dryRun?: boolean @@ -98,50 +65,6 @@ interface SourceWriteOptions { deploy?: boolean } -/** - * The text of a file the directory must hold, or a refusal that says how to write it: `hint`, or - * for a starter no workflow was created from yet, the create that makes it one. - */ -async function requireText(directory: string, path: string, hint: string): Promise { - const text = await readSourceText(directory, path) - if (text !== null) return text - const starterWorkspace = await readStarterWorkspace(directory) - throw new SimApiError( - `${directory} has no ${path}. ${ - starterWorkspace === null - ? hint - : `It holds a starter no workflow was created from yet: run \`${createCommand(directory, starterWorkspace)} --dry-run\`, then the same with --yes.` - }`, - 0 - ) -} - -async function readJson(directory: string, path: string, hint: string): Promise { - const text = await requireText(directory, path, hint) - try { - return JSON.parse(text) - } catch { - throw new SimApiError(`${path} in ${directory} is not valid JSON. ${hint}`, 0) - } -} - -/** - * The function-code files the source can reference: every `codeFile(…)` call the entry file makes, - * found by parsing it as compile does (an aliased import or spacing before the parenthesis still - * counts; a comment or string that spells one does not), plus every `codeFile` the lock records, - * limited to `.js`, `.py` and `.sh` files inside the directory. Only those are sent, so a - * reference to any other file never uploads it; the server reports a referenced file that is - * missing or of another kind. - */ -function codeFilePaths(entry: string, lock: WorkflowLockV1 | null): string[] { - const candidates = lockCodeFiles(lock) - for (const literal of codeFileReferences(entry)) { - const path = sourceRelativePath(literal) - if (path) candidates.add(path) - } - return [...candidates].filter((path) => CODE_FILE_EXTENSION.test(path)) -} - /** * Prints a write's diagnostics and, in a display format, its notes (such as a custom tool the * save could not store; a machine result lists them), each once: after a terminal preview, the @@ -181,86 +104,22 @@ function diffSummary(data: SourceApplyData): string { }${variableChanges.length > 0 ? `\nVariables: ${variableChanges.join('; ')}` : ''}` } -const GET_HINT = 'Write the directory with `sim workflows source get` first.' - -/** - * The base an apply sends, refused when it is not the named workflow's: a directory written for - * one workflow and applied to another would replace that workflow's draft with a stranger's - * source. Without a `workflowId` the apply targets the workflow the base names. - */ -export async function readBase( - directory: string, - workflowId: string | undefined -): Promise { - await assertSourceDirKind(directory, 'workflow') - const parsed = parseSourceBase(await readJson(directory, BASE_FILE, GET_HINT)) - if ('problem' in parsed) { - throw new SimApiError( - `${BASE_FILE} in ${directory} is not a base: ${parsed.problem} ${GET_HINT}`, - 0 - ) - } - const { base } = parsed - if (workflowId !== undefined && base.workflowId !== workflowId) { - throw new SimApiError( - `${directory} holds the source of workflow ${base.workflowId}, not ${workflowId}. Apply it to ${base.workflowId}, or write this workflow's source with \`sim workflows source get ${workflowId} --dir \`.`, - 0, - 'SOURCE_WORKFLOW_MISMATCH' - ) - } - return base -} - -/** The directory's lock, refused with every problem the lock parser finds. */ -export async function readLock(directory: string): Promise { - const parsed = parseWorkflowLock(await readJson(directory, LOCK_FILE, GET_HINT)) - if (parsed.lock) return parsed.lock - const problems = parsed.diagnostics.map( - (finding) => `${finding.path === '' ? '' : `${finding.path}: `}${finding.message}` - ) - throw new SimApiError( - `${LOCK_FILE} in ${directory} is not a valid lock: ${problems.join('; ')}`, - 0 - ) -} - -/** The directory's `workflow.ts`, or a refusal naming the command that writes one. */ -export function requireEntry(directory: string): Promise { - return requireText(directory, ENTRY_FILE, GET_HINT) -} - -/** - * The files a write sends: the entry file and the code files it or the lock references. A - * referenced file that is missing is left out, and the server reports it with its source line. - */ -export async function readSourceFiles( - directory: string, - lock: WorkflowLockV1 | null -): Promise> { - const entry = await requireEntry(directory) - const files: Record = { [ENTRY_FILE]: entry } - const paths = codeFilePaths(entry, lock) - const texts = await mapWithConcurrency(paths, FILE_CONCURRENCY, (path) => - readSourceText(directory, path) - ) - paths.forEach((path, index) => { - const text = texts[index] - if (text !== null) files[path] = text - }) - return files +/** Whether a write in `mode` only reads the directory: a dry run, or a plan that stops before writing. */ +function isReadOnly(mode: WriteMode): boolean { + return mode === 'dry-run' || mode === 'plan' } type SourceWriteOperation = 'applyWorkflowSource' | 'createWorkflowFromSource' /** - * The shared write flow of `apply` and `create`: refuse a write without consent when no terminal - * can ask, preview and ask on a terminal, send, print the diagnostics, and after a successful - * write save it into the directory with the caller's `save`. From the send through the save an - * interrupt waits for them ({@link deferInterrupts}), so Ctrl-C never strands a stored write the - * directory did not record. + * A workflow `apply` or `create` through {@link runSourceWrite}: refuse a code file the directory + * could never hold, then preview, ask or stop at the plan, send, print the diagnostics, and after + * a successful write save it into the directory with the caller's `save`. The deploy step runs + * after, on a stored write only, and the result is reported last. */ -async function runSourceWrite(args: { +async function writeWorkflowSource(args: { options: SourceWriteOptions + mode: WriteMode creating: boolean question: string client: OperationClient @@ -314,47 +173,38 @@ async function runSourceWrite(args: { } ) } - const mode = writeMode(options) - if (mode === 'ask' || mode === 'plan') { - const preview = await send(true) - printFindings(preview, findings, notes) - if (preview.diagnostics.some((diagnostic) => diagnostic.severity === 'error')) { - setSoftExitCode(refusedExitCode(preview.diagnostics)) - report(preview) - return - } - printError(diffSummary(preview)) - if (mode === 'plan') { - stopAtPlan() - report(preview, { next: planNext(args.command) }) - return - } - if (!(await confirm(args.question))) { - throw new SimApiError('Nothing was applied.', 0, 'CANCELLED') - } - } - + const hasErrors = (data: SourceApplyData) => + data.diagnostics.some((diagnostic) => diagnostic.severity === 'error') + /** Whether a committed answer stored the write: not refused, and a create that applied. */ + const stored = (data: SourceApplyData) => + !data.dryRun && !hasErrors(data) && (data.applied || !creating) + const { mode } = args const verb = creating ? 'create' : 'apply' - let refused = false - const sendAndSave = async (): Promise => { - const data = await send(options.dryRun === true) - printFindings(data, findings, notes) - refused = data.diagnostics.some((diagnostic) => diagnostic.severity === 'error') - if (refused) setSoftExitCode(refusedExitCode(data.diagnostics)) - if (!data.dryRun && !refused && (data.applied || !creating)) await args.save(data) - return data - } - const data = options.dryRun - ? await sendAndSave() - : await deferInterrupts(sendAndSave, { - waiting: `Interrupted: the ${verb} was already sent. Waiting up to ${DEFERRED_INTERRUPT_BOUND_MS / 1000}s for its answer and saving it into ${options.dir}, so the directory matches the workflow; press Ctrl-C again to stop now.`, - abandoned: `Stopped before the ${verb} answered. ${args.ifAnswerLost}`, - }) + const data = await runSourceWrite({ + mode, + preview: () => send(true), + show: (preview) => printFindings(preview, findings, notes), + refuses: hasErrors, + explain: (preview) => printError(diffSummary(preview)), + commit: async () => { + const answer = await send(false) + printFindings(answer, findings, notes) + if (hasErrors(answer)) setSoftExitCode(refusedExitCode(answer.diagnostics)) + return answer + }, + save: async (answer) => { + if (stored(answer)) await args.save(answer) + }, + report: (answer, next) => report(answer, next ? { next } : {}), + question: args.question, + command: args.command, + interrupt: { verb, dir: options.dir, noun: 'workflow', ifAnswerLost: args.ifAnswerLost }, + }) + if (!data) return - const stored = !data.dryRun && !refused && (data.applied || !creating) - if (mode === 'write' && stored) printError(diffSummary(data)) + if (mode === 'write' && stored(data)) printError(diffSummary(data)) let deployment: DeployOutcome = {} - if (stored) { + if (stored(data)) { try { deployment = await deployStep(args.client, data, { ...options, @@ -374,7 +224,7 @@ async function runSourceWrite(args: { interface DeployOutcome { deployed?: { version?: number; isDeployed: boolean; warnings?: string[] } needsRedeployment?: boolean - next?: { command: string; reason: string } + next?: NextStep } /** @@ -443,9 +293,8 @@ function deployFailure( /** Refuses `--deploy` beside `--dry-run`: a dry run stores nothing to deploy. */ function assertDeployable(options: SourceWriteOptions): void { if (options.deploy && options.dryRun) { - throw new SimApiError( - '--deploy deploys what the write saves, and --dry-run saves nothing: drop one of them.', - 0 + throw new CliUsageError( + '--deploy deploys what the write saves, and --dry-run saves nothing: drop one of them.' ) } } @@ -491,47 +340,53 @@ export function attachWorkflowSourceApply(source: Command): void { .description('Apply a workflow source directory to a workflow draft') .addHelpText( 'after', - `${APPLY_EXIT_CODES_HELP}${WORKSPACE_FOLDER_BUSY_HELP}${SOURCE_WRITE_EXIT_CODES_HELP}` - ) - .action(async (named: string | undefined, options: SourceWriteOptions) => - withWorkspaceFolderLock(options.dir, async () => { - assertDeployable(options) - await assertSourceDirKind(options.dir, 'workflow') - const { client, profile } = connectApply() - const lock = await readLock(options.dir) - const base = await readBase(options.dir, named) - const workflowId = base.workflowId - const files = await readSourceFiles(options.dir, lock) - if (!options.dryRun) { - await refreshWorkflowWorkspaceTypes(client, options.dir, base.workspaceId) - } - await runSourceWrite<'applyWorkflowSource'>({ - options, - creating: false, - question: `Apply these changes to workflow ${workflowId}?`, - command: `sim workflows source apply --dir ${shellWord(options.dir)}${options.allowDelete ? ' --allow-delete' : ''}${options.deploy ? ' --deploy' : ''}`, - output: profile.output, - client, - operation: 'applyWorkflowSource', - params: { workflowId }, - body: { - files, - lock, - base, - provenance: { client: 'cli', ...(await gitProvenance(options.dir)) }, - }, - save: async (data) => { - const edges = storedEdgesWrite( - data, - cachedCatalogByHash(toRecord(data.base).catalogHash) - ) - await writeBack(options.dir, data, lock, { write: edges ? [edges] : [] }) - await refreshWorkflowTsconfigs(options.dir, data) - }, - ifAnswerLost: `It may have been saved. Run \`sim workflows source apply ${workflowId} --dir ${options.dir} --dry-run\`: exit 5 (stale) means the draft changed since this directory was read, by this apply or by someone else. Run \`sim workflows source get ${workflowId} --dir \` and compare it with ${options.dir}: if it holds your edit, \`sim workflows source get ${workflowId} --dir ${options.dir} --force\` brings this directory up to date; if not, reapply your edit in the fresh directory. Any other outcome means it was not saved: apply again.`, - }) - }) + `${sourceExitCodesHelp('workflows source apply')}${SOURCE_WRITE_EXIT_CODES_HELP}` ) + .action(async (named: string | undefined, options: SourceWriteOptions) => { + const mode = writeMode(options) + await withSourceCommandLock( + options.dir, + async () => { + assertDeployable(options) + await assertSourceDirKind(options.dir, 'workflow') + const { client, profile } = connectApply() + const lock = await readLock(options.dir) + const base = await readBase(options.dir, named) + const workflowId = base.workflowId + const files = await readSourceFiles(options.dir, lock) + if (!isReadOnly(mode)) { + await refreshWorkflowWorkspaceTypes(client, options.dir, base.workspaceId) + } + await writeWorkflowSource<'applyWorkflowSource'>({ + options, + mode, + creating: false, + question: `Apply these changes to workflow ${workflowId}?`, + command: `sim workflows source apply --dir ${shellWord(options.dir)}${options.allowDelete ? ' --allow-delete' : ''}${options.deploy ? ' --deploy' : ''}`, + output: profile.output, + client, + operation: 'applyWorkflowSource', + params: { workflowId }, + body: { + files, + lock, + base, + provenance: { client: 'cli', ...(await gitProvenance(options.dir)) }, + }, + save: async (data) => { + const edges = storedEdgesWrite( + data, + cachedCatalogByHash(toRecord(data.base).catalogHash) + ) + await writeBack(options.dir, data, lock, { write: edges ? [edges] : [] }) + await refreshWorkflowTsconfigs(options.dir, data) + }, + ifAnswerLost: `It may have been saved. Run \`sim workflows source apply ${workflowId} --dir ${options.dir} --dry-run\`: exit 5 (stale) means the draft changed since this directory was read, by this apply or by someone else. Run \`sim workflows source get ${workflowId} --dir \` and compare it with ${options.dir}: if it holds your edit, \`sim workflows source get ${workflowId} --dir ${options.dir} --force\` brings this directory up to date; if not, reapply your edit in the fresh directory. Any other outcome means it was not saved: apply again.`, + }) + }, + { readOnly: isReadOnly(mode) } + ) + }) } /** @@ -572,59 +427,66 @@ export function attachWorkflowSourceCreate(source: Command): void { .description('Create a workflow from a workflow source directory') .addHelpText( 'after', - `${CREATE_EXIT_CODES_HELP}${WORKSPACE_FOLDER_BUSY_HELP}${SOURCE_WRITE_EXIT_CODES_HELP}` + `${sourceExitCodesHelp('workflows source create')}${SOURCE_WRITE_EXIT_CODES_HELP}` ) - .action(async (options: SourceWriteOptions & { name?: string; folder?: string }) => - withWorkspaceFolderLock(options.dir, async () => { - await assertSourceDirKind(options.dir, 'workflow') - if ((await readSourceText(options.dir, ENTRY_FILE)) === null) { - if (options.dryRun || options.yes) { - throw new SimApiError( - `${options.dir} has no ${ENTRY_FILE}, so there is nothing to ${options.dryRun ? 'check' : 'create'}. Write ${ENTRY_FILE} there, or run \`sim workflows source create --dir ${options.dir}\` without --dry-run and --yes to write a starter.`, - 0 + .action(async (options: SourceWriteOptions & { name?: string; folder?: string }) => { + const mode = writeMode(options) + await withSourceCommandLock( + options.dir, + async () => { + await assertSourceDirKind(options.dir, 'workflow') + if ((await readSourceText(options.dir, ENTRY_FILE)) === null) { + if (options.dryRun || options.yes) { + throw new SourceDirError( + `${options.dir} has no ${ENTRY_FILE}, so there is nothing to ${options.dryRun ? 'check' : 'create'}. Write ${ENTRY_FILE} there, or run \`sim workflows source create --dir ${options.dir}\` without --dry-run and --yes to write a starter.` + ) + } + const { client, profile } = connectCreate() + const workspaceId = client.requireWorkspace() + const tree = await writeSourceScaffold(client, options.dir, workspaceId) + printError( + `${options.dir} had no ${ENTRY_FILE}, so a starter, the SDK types for workspace ${workspaceId}, tsconfig.json, .sim/AGENTS.md and the examples in .sim/examples/ were written and nothing was created. Edit ${ENTRY_FILE}, then run \`${createCommand(options.dir, workspaceId)} --yes\`.` ) + printTreeResult(profile.output, { created: false, scaffolded: true, ...tree }) + return } + assertDeployable(options) const { client, profile } = connectCreate() - const workspaceId = client.requireWorkspace() - const tree = await writeSourceScaffold(client, options.dir, workspaceId) - printError( - `${options.dir} had no ${ENTRY_FILE}, so a starter, the SDK types for workspace ${workspaceId}, tsconfig.json, .sim/AGENTS.md and the examples in .sim/examples/ were written and nothing was created. Edit ${ENTRY_FILE}, then run \`${createCommand(options.dir, workspaceId)} --yes\`.` - ) - printTreeResult(profile.output, { created: false, scaffolded: true, ...tree }) - return - } - assertDeployable(options) - const { client, profile } = connectCreate() - // A starter is bound to the workspace its types were written for, as a get directory - // is to the one it was read from; --workspace still chooses another. - const starterWorkspace = - profile.sources.workspaceId === 'flag' ? null : await readStarterWorkspace(options.dir) - const workspaceId = client.requireWorkspace(starterWorkspace ?? undefined) - const files = await readSourceFiles(options.dir, null) - if (!options.dryRun) await refreshWorkflowWorkspaceTypes(client, options.dir, workspaceId) - await runSourceWrite<'createWorkflowFromSource'>({ - options, - creating: true, - question: 'Create this workflow?', - command: `${createCommand(shellWord(options.dir), workspaceId)}${options.name !== undefined ? ` --name ${shellWord(options.name)}` : ''}${options.folder !== undefined ? ` --folder ${shellWord(options.folder)}` : ''}${options.deploy ? ' --deploy' : ''}`, - output: profile.output, - client, - operation: 'createWorkflowFromSource', - params: { workspaceId }, - ifAnswerLost: `It may have been created. Run this command again: a SOURCE_NAME_TAKEN refusal means it was, so find its id with \`sim workflows list\` and run \`sim workflows source get --dir ${options.dir} --force\`.`, - save: async (data) => { - await writeSourceDirectory(client, options.dir, data, []) - await updateSourceDir(options.dir, { write: [], remove: [SOURCE_STARTER_FILE] }) - }, - body: { - files, - ...(options.name !== undefined ? { name: options.name } : {}), - ...(options.folder !== undefined ? { folderPath: options.folder } : {}), - provenance: { client: 'cli', ...(await gitProvenance(options.dir)) }, - }, - }) - }) - ) + // A starter is bound to the workspace its types were written for, as a get directory + // is to the one it was read from; --workspace still chooses another. + const starterWorkspace = + profile.sources.workspaceId === 'flag' ? null : await readStarterWorkspace(options.dir) + const workspaceId = client.requireWorkspace(starterWorkspace ?? undefined) + const files = await readSourceFiles(options.dir, null) + if (!isReadOnly(mode)) { + await refreshWorkflowWorkspaceTypes(client, options.dir, workspaceId) + } + await writeWorkflowSource<'createWorkflowFromSource'>({ + options, + mode, + creating: true, + question: 'Create this workflow?', + command: `${createCommand(shellWord(options.dir), workspaceId)}${options.name !== undefined ? ` --name ${shellWord(options.name)}` : ''}${options.folder !== undefined ? ` --folder ${shellWord(options.folder)}` : ''}${options.deploy ? ' --deploy' : ''}`, + output: profile.output, + client, + operation: 'createWorkflowFromSource', + params: { workspaceId }, + ifAnswerLost: `It may have been created. Run this command again: a SOURCE_NAME_TAKEN refusal means it was, so find its id with \`sim workflows list\` and run \`sim workflows source get --dir ${options.dir} --force\`.`, + save: async (data) => { + await writeSourceDirectory(client, options.dir, data, []) + await updateSourceDir(options.dir, { write: [], remove: [SOURCE_STARTER_FILE] }) + }, + body: { + files, + ...(options.name !== undefined ? { name: options.name } : {}), + ...(options.folder !== undefined ? { folderPath: options.folder } : {}), + provenance: { client: 'cli', ...(await gitProvenance(options.dir)) }, + }, + }) + }, + { readOnly: isReadOnly(mode) } + ) + }) } /** The variables a diff renames, none when the answer lists none. */ @@ -644,11 +506,6 @@ function variablesChanged(variables: SourceApplyData['diff']['variables']): bool ) } -/** The create command for `dir` in `workspaceId`, as the next step a starter prints. */ -function createCommand(dir: string, workspaceId: string): string { - return `sim workflows source create --dir ${dir} --workspace ${workspaceId}` -} - /** * What a write created (on a dry run, would create; ids are only known once it is written): the * workflow of a create, then each block added. diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-check.test.ts b/packages/sim-cli/src/commands/protocol/workflow-source-check.test.ts index 0c4adca3dbc..b26d38e116f 100644 --- a/packages/sim-cli/src/commands/protocol/workflow-source-check.test.ts +++ b/packages/sim-cli/src/commands/protocol/workflow-source-check.test.ts @@ -433,6 +433,26 @@ describe('sim workflows source check', () => { expect(findingLines()).toContain('W_ENV_UNKNOWN') }) + it('lists at most 20 warnings of one code, as a workflow answer does, and counts every one', async () => { + cacheCatalog() + writeWorkspace() + const unknown = Array.from({ length: 25 }, (_, index) => `extra${index}: 'x'`).join(', ') + const folder = writeWorkflowFolder( + 'flood', + workspaceWorkflow((source) => + source.replace("data: { email: start.email, status: 'pending' }", `data: { ${unknown} }`) + ) + ) + + await checkIn(folder, '--no-strict') + + const answer = result() + expect(answer).toMatchObject({ errors: 0, warnings: 25 }) + const codes = (answer.diagnostics as Array<{ code: string }>).map((finding) => finding.code) + expect(codes.filter((code) => code === 'W_TABLE_UNKNOWN_COLUMN')).toHaveLength(20) + expect(codes.filter((code) => code === 'I_FINDINGS_TRUNCATED')).toHaveLength(1) + }) + it('fails a row write naming a column or select option the table does not declare', async () => { cacheCatalog() writeWorkspace() diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-check.ts b/packages/sim-cli/src/commands/protocol/workflow-source-check.ts index 6b8b24da9e7..df197ef165d 100644 --- a/packages/sim-cli/src/commands/protocol/workflow-source-check.ts +++ b/packages/sim-cli/src/commands/protocol/workflow-source-check.ts @@ -1,159 +1,37 @@ -import { join } from 'node:path' -import { isRecordLike, toRecord } from '@sim/utils/object' -import { - type BlockCatalog, - type CanonicalWorkflow, - codeFrame, - codePointLabel, - compileWorkflow, - didYouMean, - EDIT_REFUSALS, - encodeAnchor, - getOwn, - parseJsonField, - parseWorkflowLock, - raisedMessage, - refuseEditedFindings, - scriptName, - startInputs, - type Value, - type WorkflowInput, -} from '@sim/workflow-code' -import { SOURCE_BASE_FILE, SOURCE_ENTRY_FILE, SOURCE_LOCK_FILE } from '@sim/workflow-code/limits' -import { KIND_DIRS } from '@sim/workspace-code' -import { KIND_LAYOUTS } from '@sim/workspace-code/kinds/registry' +import { toRecord } from '@sim/utils/object' +import { SOURCE_BASE_FILE } from '@sim/workflow-code/limits' import type { Command } from 'commander' import { profileFrom } from '../../context' -import { setSoftExitCode } from '../../embed-context' -import { SimApiError } from '../../http/client' -import { printProtocolResult } from './result' +import { CliUsageError } from './local-errors' +import { + cachedOrRefuse, + checkWorkflowDir, + refreshWorkspaceTsconfigs, + reportCheck, + STRICT_HELP, + type StrictOptions, + strictFlags, + strictMode, + withWorkflowInputs, + workspaceWorkflowInputs, +} from './source-check' import { absoluteDirOption, assertSourceDirKind, - digest, directoryWorkspace, - readFolderWritten, readSourceJson, - readSourceText, readStarterWorkspace, workspaceFolderOf, } from './source-dir' import { cachedCatalog, cachedSourceCatalog, - type LocalDeclarations, refreshTsconfigs, refreshWorkspaceTypes, } from './source-tree' -import { type Diagnostic, findingsOutput, verboseFlag } from './source-ui' -import { blockDigest, readStoredBlockDigests, readStoredEdges } from './stored-edges' -import { readSourceFiles, requireEntry } from './workflow-source-apply' -import { WORKFLOW_TREE } from './workflow-source-get' -import { - kindFolders, - readWorkspaceDeclarations, - WORKSPACE_TREE, - type WorkspaceDeclarations, -} from './workspace-source-dir' - -/** - * A finding only the server can decide: `stored()` keeps a secret the stored workflow holds, and - * a local compile has no stored workflow to read it from. - */ -const SERVER_ONLY_CODES: ReadonlySet = new Set(['E_STORED_VALUE_MISSING']) - -/** - * Warnings an apply saves as written although the block they name is broken: a strict check - * fails on them (see {@link StrictMode}). - */ -const STRICT_CODES: ReadonlySet = new Set([ - 'W_REQUIRED_MISSING', - 'W_UNKNOWN_OPTION', - 'W_UNKNOWN_OUTPUT_PATH', - 'W_VALUE_KIND', - 'W_OUT_OF_RANGE', - 'W_LITERAL_SECRET', - 'W_TABLE_UNKNOWN_COLUMN', - 'W_CHILD_INPUT_UNKNOWN', - 'W_BRANCH_UNWIRED', - 'W_UNRESOLVED_REF', - 'W_REF_OUT_OF_SCOPE', -]) - -/** - * {@link STRICT_CODES} warnings the stored workflow exempts, which a check never fails on a block - * the lock binds, `--strict` included: the server stays quiet when the stored workflow holds the - * same state, and an offline check cannot read it. `W_BRANCH_UNWIRED`: a branch the stored - * workflow leaves unwired. - */ -const STORED_EXEMPT_CODES: ReadonlySet = new Set(['W_BRANCH_UNWIRED']) - -/** - * Warnings a check reports only on a block the lock does not bind (one added since the get), and - * on every block with `--strict`: printed source holds them on blocks nobody edited, and the - * server, which compares with the stored workflow, already reports them only for an edited field. - * `W_JSON_REF_UNTYPED`: a bare reference nothing types, in a JSON field. - */ -const EDIT_SCOPED_CODES: ReadonlySet = new Set(['W_JSON_REF_UNTYPED']) - -/** - * Which {@link STRICT_CODES} warnings fail a check: `new` those on a block the lock does not bind - * (one added since the get, or every block of a workflow that has no lock yet), `all` every one, - * `off` none. Printed source can hold such a warning on a block nobody edited (a field left empty - * in the editor), so the default fails only on blocks the edit wrote. - */ -type StrictMode = 'new' | 'all' | 'off' - -/** One finding of a check, as the write commands print theirs. */ -export type CheckFinding = Diagnostic - -/** The `--strict` / `--no-strict` pair: absent selects the default, {@link StrictMode} `new`. */ -export interface StrictOptions { - strict?: boolean -} - -/** The strict mode the `--strict` / `--no-strict` pair selects. */ -export function strictMode(options: StrictOptions): StrictMode { - if (options.strict === undefined) return 'new' - return options.strict ? 'all' : 'off' -} - -/** Where each kind of edit refusal applies, as the help says it. */ -const REFUSAL_WHERE: Record<(typeof EDIT_REFUSALS)[number]['where'], string> = { - 'changed block': 'in any file of a block the edit added or changed', - 'workflow.ts': 'in workflow.ts on a block the edit added or changed', - 'edited code file': 'in a function file changed since the get', -} - -/** Help shared by both check commands: what fails, and the strict flags. */ -export const STRICT_HELP = `Exit codes: 0 no errors; 1 errors (each printed as file:line:column with its frame and hint), or -no cached catalog (run the get once). With --output json, stdout is one document that also lists -every finding under diagnostics. - -These warnings, which an apply saves as written, fail the check as errors on every block the lock -does not bind (a block added since the get, or every block of a new workflow): -${[...STRICT_CODES].map((code) => ` ${code}`).join(',\n')}. ---strict fails on them in every block, except ${[...STORED_EXEMPT_CODES].join(', ')} on a block the -lock binds (the stored workflow may hold it, and the server then stays quiet); --no-strict fails -on none of them. In every mode, as the server refuses them: ${EDIT_REFUSALS.map( - ({ warning, refusal, where }) => `${warning} is the error ${refusal} ${REFUSAL_WHERE[where]}` -).join('; ')}. A block counts as changed when the lock does not bind it, when the source moved it -into another container, or when it compiles differently than in the source the last get or -apply printed (.sim/stored-edges.json records a digest of each block). That last comparison -needs the catalog the source was printed with in the cache: without it, a bound block that did -not move counts as unchanged, so the server may refuse what the check passes. A $ref in a -function file to a block the lock binds that the source no longer declares stays a warning, unless -a warning cap left findings out: the server may infer that block was renamed and rewrite the -reference, and refuses it otherwise. -${[...EDIT_SCOPED_CODES].join(', ')} is reported only on blocks the lock does not bind, and with ---strict on every block.` - -/** Adds the `--strict` / `--no-strict` pair to a check command. */ -export function strictFlags(command: Command): Command { - return command - .option('--strict', 'Fail on the warnings listed below in every block, not only new ones') - .option('--no-strict', 'Never fail on a warning an apply saves') -} +import { verboseFlag } from './source-ui' +import { requireEntry, WORKFLOW_TREE } from './workflow-dir' +import { readWorkspaceDeclarations } from './workspace-source-dir' const CHECK_HELP = ` Compiles workflow.ts (and the function files it names) with the compiler the server runs, against @@ -168,483 +46,6 @@ regenerates .sim/sdk/workspace.d.ts, so tsc types each workflow's inputs as its ${STRICT_HELP}` -/** Every workflow key of the workspace directory `root` that has a `workflow.ts`. */ -export async function workspaceWorkflowKeys(root: string): Promise { - const { paths } = await kindFolders(root, ['workflow']) - return [...KIND_LAYOUTS.workflow.group(paths).groups] - .filter(([key, folderPaths]) => folderPaths.includes(workflowEntry(key))) - .map(([key]) => key) -} - -/** A workflow's `workflow.ts`, relative to the workspace directory. */ -function workflowEntry(key: string): string { - return `${KIND_DIRS.workflow}/${key}/${SOURCE_ENTRY_FILE}` -} - -/** - * Points the workspace directory's tsconfig files at the blocks its workflows now use, keeping the - * blocks they include. - */ -export async function refreshWorkspaceTsconfigs(root: string, workspaceId: string): Promise { - const keys = await workspaceWorkflowKeys(root) - const sources = await Promise.all(keys.map((key) => readSourceText(root, workflowEntry(key)))) - await refreshTsconfigs( - root, - workspaceId, - WORKSPACE_TREE, - { sources: sources.filter((text) => text !== null) }, - { incremental: true } - ) -} - -/** A canonical resource node: what `table('k')`, `credential('k')` and the like compile to. */ -function resourceNode(value: unknown): { kind: string; key: string } | null { - if (!isRecordLike(value) || value.$sim !== 'resource') return null - const { kind, key } = value - return typeof kind === 'string' && typeof key === 'string' ? { kind, key } : null -} - -/** Every resource node inside a value, depth first. */ -function resourceNodes(value: unknown): Array<{ kind: string; key: string }> { - const node = resourceNode(value) - if (node) return [node] - if (Array.isArray(value)) return value.flatMap(resourceNodes) - if (isRecordLike(value)) return Object.values(value).flatMap(resourceNodes) - return [] -} - -/** - * Why a key can never match: ` "triаge" holds U+0430, a Cyrillic letter; …` for a key with a - * character outside lowercase ASCII (a resource key is `[a-z][a-z0-9_-]*`), which reads the same - * as the declared key it imitates; empty for a key in the key format. - */ -function keyCharacterNote(key: string): string { - const odd = [...key].find((character) => !/[a-z0-9_-]/.test(character)) - if (odd === undefined) return '' - const script = scriptName(odd) - return ` "${key}" holds ${codePointLabel(odd)}${script ? `, a ${script} letter` : ''}: a key is lowercase ASCII letters, digits, - and _, so retype it.` -} - -/** How many keys a hint lists. */ -const LISTED_KEYS = 20 - -function listed(keys: Iterable): string { - const all = [...keys] - const more = all.length > LISTED_KEYS ? `, and ${all.length - LISTED_KEYS} more` : '' - return `${all.slice(0, LISTED_KEYS).join(', ')}${more}` -} - -/** - * A row-write param's value as a JSON tree: compiled row data is JSON text whose non-string - * references sit bare (`"n": `), so it parses as a JSON field, references as leaves. - */ -function rowJson(value: Value): unknown { - return typeof value === 'string' ? parseJsonField(value) : value -} - -/** The params of a table block that hold row data: one row, or a list of rows. */ -const ROW_PARAMS = ['data', 'rows'] as const - -/** - * A finding about one block param, located by the caller: at the param, or at the param whose - * source names the resource key `names` (a source param name can differ from its canonical id). - */ -type ParamFinding = Pick & { - block: string - param: string - names?: string -} - -/** The `inputMapping` of a child workflow call, as a record of input name to value. */ -function inputMappingOf(canonical: CanonicalWorkflow['blocks'][string]): Record { - const mapping = canonical.params.inputMapping - if (mapping?.kind !== 'value') return {} - const value = rowJson(mapping.value) - return toRecord(value) -} - -/** - * A child workflow call checked against the child's own `workflow.ts` in this directory: a child - * without a Start trigger (`W_CHILD_NO_START`), and an `inputMapping` key its Start trigger does - * not declare (`W_CHILD_INPUT_UNKNOWN`). A child that did not compile is not checked. - */ -function childInputFindings( - block: string, - canonical: CanonicalWorkflow['blocks'][string], - declarations: WorkspaceDeclarations -): ParamFinding[] { - const ref = canonical.params.workflowId - const child = ref?.kind === 'value' ? resourceNode(ref.value) : null - if (child?.kind !== 'workflow') return [] - const inputs = declarations.workflowInputs?.get(child.key) - if (inputs === undefined) return [] - const entry = `${KIND_DIRS.workflow}/${child.key}/${SOURCE_ENTRY_FILE}` - if (inputs === null) { - return [ - { - code: 'W_CHILD_NO_START', - severity: 'warning', - message: `Workflow "${child.key}" has no Start trigger, so it takes no inputs and a call cannot run it.`, - hint: `Add trigger.start('Start', { inputFormat: { … } }) to ${entry}, or call a workflow that has one.`, - block, - param: 'workflowId', - names: child.key, - }, - ] - } - const names = inputs.map((input) => input.name) - return Object.keys(inputMappingOf(canonical)) - .filter((field) => !names.includes(field)) - .map((field) => ({ - code: 'W_CHILD_INPUT_UNKNOWN', - severity: 'warning', - message: `inputMapping sets ${field}, which is not an input of workflow "${child.key}"; the child never receives it.`, - hint: - names.length > 0 - ? `${didYouMean(field, names) ?? ''} Inputs (the Start trigger's inputFormat in ${entry}): ${listed(names)}.`.trimStart() - : `The Start trigger of ${entry} declares no inputs: add ${field} to its inputFormat.`, - block, - param: 'inputMapping', - })) -} - -/** - * The findings a compile cannot make without the workspace's declarations: a resource key the - * directory does not declare (an apply fails on it), and a table row write naming a column, or a - * select option, its table does not declare. - */ -function declarationFindings( - workflow: CanonicalWorkflow, - declarations: WorkspaceDeclarations -): ParamFinding[] { - const findings: ParamFinding[] = [] - for (const [block, canonical] of Object.entries(workflow.blocks)) { - const tables = new Set() - for (const [param, value] of Object.entries(canonical.params)) { - for (const node of resourceNodes(value)) { - if (node.kind === 'table') tables.add(node.key) - const declared = declarations.keys.get(node.kind) - if (!declared || declared.has(node.key)) continue - const near = didYouMean(node.key, declared) - findings.push({ - code: 'E_MAT_UNKNOWN_RESOURCE', - severity: 'error', - message: `No ${node.kind} "${node.key}" is declared in this workspace directory.${keyCharacterNote(node.key)}`, - hint: - declared.size > 0 - ? `${near ? `${near} ` : ''}Keys: ${listed(declared)}.` - : `This directory declares no ${node.kind}: declare it before naming it.`, - block, - param, - names: node.key, - }) - } - } - findings.push(...childInputFindings(block, canonical, declarations)) - if (tables.size !== 1) continue - const [table] = tables - const columns = declarations.tables.get(table) - if (!columns) continue - for (const param of ROW_PARAMS) { - const value = canonical.params[param] - if (value?.kind !== 'value') continue - const json = rowJson(value.value) - for (const row of Array.isArray(json) ? json : [json]) { - if (!isRecordLike(row)) continue - for (const [field, cell] of Object.entries(row)) { - const column = columns.get(field) - if (!column) { - findings.push({ - code: 'W_TABLE_UNKNOWN_COLUMN', - severity: 'warning', - message: `${param} writes ${field}, which is not a column of table "${table}"; the run drops it.`, - hint: `${didYouMean(field, columns.keys()) ?? ''} Columns: ${listed(columns.keys())}.`.trimStart(), - block, - param, - }) - continue - } - const { options } = column - if (!options || column.multiple || typeof cell !== 'string' || cell.includes('<')) - continue - if (options.includes(cell)) continue - findings.push({ - code: 'W_UNKNOWN_OPTION', - severity: 'warning', - message: `${field} of table "${table}" has no option ${cell}; it is kept as written.`, - hint: `Options: ${options.join(', ')}.`, - block, - param, - }) - } - } - } - } - return findings -} - -/** - * Compiles the workflow source directory `dir` against the catalog its base names when that one is - * cached (a workflow holding a type the workspace catalog lacks was printed with its own), else - * `catalog`, checking env names against the names `secrets.ts` declares when `declarations` hold - * them. - */ -async function compileWorkflowDir( - dir: string, - workspaceCatalog: BlockCatalog, - declarations?: WorkspaceDeclarations -) { - const { lock } = parseWorkflowLock(await readSourceJson(dir, SOURCE_LOCK_FILE)) - const [files, storedEdges, storedBlocks, printedWith] = await Promise.all([ - readSourceFiles(dir, lock ?? null), - readStoredEdges(dir), - readStoredBlockDigests(dir), - cachedSourceCatalog(dir), - ]) - const catalog = printedWith ?? workspaceCatalog - const envNames = declarations?.envNames - const compiled = compileWorkflow({ - entry: { path: SOURCE_ENTRY_FILE, text: files[SOURCE_ENTRY_FILE] }, - files: new Map(Object.entries(files).filter(([path]) => path !== SOURCE_ENTRY_FILE)), - catalog, - sdkMajor: catalog.sdkMajor, - knownKeys: new Set(Object.keys(lock?.blocks ?? {})), - storedUnread: lock ?? undefined, - ...(storedEdges ? { storedUnreadEdges: storedEdges } : {}), - ...(envNames ? { envNames, envDeclaredIn: KIND_DIRS.secret } : {}), - }) - return { lock, files, compiled, storedBlocks, printedWith } -} - -/** - * The inputs of each workflow of the workspace directory `root` that compiles (`startInputs`, - * `null` for one without a Start trigger), by key: what a child workflow call is checked against - * and what `workspace.d.ts` types its `inputMapping` with. - */ -export async function workspaceWorkflowInputs( - root: string, - catalog: BlockCatalog -): Promise> { - const inputs = new Map() - for (const key of await workspaceWorkflowKeys(root)) { - const { compiled } = await compileWorkflowDir(join(root, KIND_DIRS.workflow, key), catalog) - if (compiled.workflow) inputs.set(key, startInputs(compiled.workflow)) - } - return inputs -} - -/** - * The workflows of `local` typed with the inputs each declares (`inputs`), for `workspace.d.ts`. - */ -export function withWorkflowInputs( - local: LocalDeclarations, - inputs: ReadonlyMap -): LocalDeclarations { - const workflows = { ...local.workflows } - for (const [key, fields] of inputs) { - workflows[key] = { - inputs: Object.fromEntries((fields ?? []).map((field) => [field.name, field.type])), - } - } - return { ...local, workflows } -} - -/** - * Compiles one workflow source directory offline against `catalog`, with the workspace's - * declarations when the directory sits in a workspace directory, and returns its findings with - * the {@link STRICT_CODES} the strict mode fails on raised to errors, and the `workflow.ts` text. - * `prefix` goes before each file path (`workflows//` in a workspace check). - */ -export async function checkWorkflowDir( - dir: string, - catalog: BlockCatalog, - options: { declarations?: WorkspaceDeclarations; strict: StrictMode; prefix?: string } -): Promise<{ findings: CheckFinding[]; entry: string }> { - const { lock, files, compiled, storedBlocks, printedWith } = await compileWorkflowDir( - dir, - catalog, - options.declarations - ) - const entryText = files[SOURCE_ENTRY_FILE] - const locate = (block: string, param: string, names: string | undefined) => { - const prefix = encodeAnchor({ kind: 'param', block, param: '' }) - const naming = - names === undefined - ? undefined - : Object.entries(compiled.spans).find( - ([anchor, span]) => - anchor.startsWith(prefix) && - span !== undefined && - span.file === SOURCE_ENTRY_FILE && - ['"', "'", '`'].some((quote) => - entryText.slice(span.start, span.end).includes(`${quote}${names}${quote}`) - ) - )?.[1] - return ( - naming ?? - compiled.spans[encodeAnchor({ kind: 'param', block, param })] ?? - compiled.spans[encodeAnchor({ kind: 'block', block })] - ) - } - const declared = - compiled.workflow && options.declarations - ? declarationFindings(compiled.workflow, options.declarations).map( - ({ block, param, names, ...finding }) => { - const span = locate(block, param, names) - const text = span ? files[span.file] : undefined - return { - ...finding, - blockKey: block, - ...(span ? { span } : {}), - ...(span && text !== undefined ? { frame: codeFrame(text, span) } : {}), - } - } - ) - : [] - const bound = (blockKey: string | undefined) => - lock !== undefined && blockKey !== undefined && Object.hasOwn(lock.blocks, blockKey) - const written = await readFolderWritten(dir) - // A file only moved from the path the lock bound its block to is unedited while its text matches that path's digest. - const edited = (file: string, blockKey: string | undefined) => { - const text = files[file] - if (text === undefined) return true - const textDigest = digest(text) - if (written.digests[file] === textDigest) return false - const previousFile = - lock && blockKey !== undefined && Object.hasOwn(lock.blocks, blockKey) - ? lock.blocks[blockKey].codeFile - : undefined - return previousFile === undefined || written.digests[previousFile] !== textDigest - } - const moved = (blockKey: string) => { - const held = lock ? getOwn(lock.blocks, blockKey) : undefined - const block = compiled.workflow ? getOwn(compiled.workflow.blocks, blockKey) : undefined - return held !== undefined && block !== undefined && held.parent !== block.parent - } - /** - * Whether the edit added, moved or changed the block. A block's digest is compared only when - * the catalog it was printed with is cached: against another catalog every block may compile - * differently, and a check prefers a warning to a refusal the server would not make. - */ - const changed = (blockKey: string) => { - if (!bound(blockKey) || moved(blockKey)) return true - const block = compiled.workflow?.blocks[blockKey] - const printed = printedWith ? storedBlocks?.get(blockKey) : undefined - return block !== undefined && printed !== undefined && blockDigest(block) !== printed - } - const fails = (code: string, blockKey: string | undefined) => - STRICT_CODES.has(code) && - !(STORED_EXEMPT_CODES.has(code) && bound(blockKey)) && - (options.strict === 'all' || (options.strict === 'new' && !bound(blockKey))) - const prefix = options.prefix ?? '' - const reported = (code: string, blockKey: string | undefined) => - !SERVER_ONLY_CODES.has(code) && - !(EDIT_SCOPED_CODES.has(code) && options.strict !== 'all' && bound(blockKey)) - /** - * Whether the code `blockKey` runs from `file` changed, as the server compares it: the file's - * text, or a bound block now pointed at another file whose text differs from what it ran. - */ - const codeEdited = (blockKey: string, file: string) => { - if (edited(file, blockKey)) return true - const before = lock ? getOwn(lock.blocks, blockKey)?.codeFile : undefined - const text = files[file] - return ( - before !== undefined && - before !== file && - text !== undefined && - written.digests[before] !== digest(text) - ) - } - const removedKeys = new Set( - Object.keys(lock?.blocks ?? {}).filter( - (key) => !compiled.workflow || getOwn(compiled.workflow.blocks, key) === undefined - ) - ) - /** - * A refused `$ref` in a code file naming a block the lock binds that the source no longer - * declares: the server may infer that block's rename (name and const both changed) and then - * rewrite the reference, which only it can tell, so the check keeps the warning. - */ - // Past a warning cap the compile keeps one finding per code, block and file, which could stand - // for a reference no rename settles, so then nothing is exempt. - const listedEvery = !compiled.diagnostics.some((d) => d.code === 'W_TOO_MANY_WARNINGS') - const mayBeRenamed = (finding: (typeof compiled.diagnostics)[number]) => { - if (!listedEvery || finding.code !== 'W_UNRESOLVED_REF') return false - if (finding.span.file === SOURCE_ENTRY_FILE) return false - const text = files[finding.span.file]?.slice(finding.span.start, finding.span.end) ?? '' - const head = /^(?:\$ref\.|<)([A-Za-z0-9_]+)/.exec(text)?.[1] - return head !== undefined && removedKeys.has(head) - } - const refusable = (finding: (typeof compiled.diagnostics)[number]) => !mayBeRenamed(finding) - const refused = [ - ...refuseEditedFindings( - { - diagnostics: compiled.diagnostics.filter(refusable), - withheld: compiled.withheld.filter(refusable), - }, - { changed, codeEdited } - ), - ...compiled.diagnostics.filter(mayBeRenamed), - ] - const findings = [ - ...refused.filter((diagnostic) => reported(diagnostic.code, diagnostic.blockKey)), - ...declared, - ].map(({ code, severity, message, hint, frame, span, blockKey }): CheckFinding => { - const raised = severity === 'warning' && fails(code, blockKey) - return { - code, - severity: raised ? 'error' : severity, - message: raised ? raisedMessage(message) : message, - ...(hint !== undefined ? { hint } : {}), - ...(frame !== undefined ? { frame } : {}), - ...(span ? { file: `${prefix}${span.file}`, line: span.line, column: span.column } : {}), - } - }) - return { findings, entry: entryText } -} - -/** - * Prints a check's findings and result, and exits 1 on any error. `verbose` keeps info findings in - * a machine result. - */ -export function reportCheck( - output: Parameters[0], - findings: CheckFinding[], - result: Record, - verbose = false -): void { - const shown = findingsOutput(output, verbose) - shown.print(findings) - const errors = findings.filter((finding) => finding.severity === 'error').length - if (errors > 0) setSoftExitCode(1) - printProtocolResult(output, { - checked: 'compile', - errors, - warnings: findings.filter((finding) => finding.severity === 'warning').length, - ...result, - ...shown.result(findings), - }) -} - -/** The catalog a cache read found, or a refusal naming `get`, the command that caches one. */ -function cachedOrRefuse( - catalog: BlockCatalog | null, - workspaceId: string, - get: string -): BlockCatalog { - if (catalog) return catalog - throw new SimApiError( - `No block catalog is cached for workspace ${workspaceId}. Run ${get} once, then check again.`, - 0, - 'CATALOG_NOT_CACHED' - ) -} - -/** The cached catalog of `workspaceId`, or a refusal naming `get`, the command that caches one. */ -export function requireCachedCatalog(workspaceId: string, get: string): BlockCatalog { - return cachedOrRefuse(cachedCatalog(workspaceId), workspaceId, get) -} - /** * `sim workflows source check [--dir ]`: compiles a source directory locally with the bundled * compiler and the cached catalog, so an agent's edit loop gets syntax and builder findings in @@ -684,9 +85,8 @@ export function attachWorkflowSourceCheck(source: Command): void { (from) => `Run it without --workspace to check it against the catalog of ${from}.` ) if (!workspaceId) { - throw new SimApiError( - `${options.dir} has no ${SOURCE_BASE_FILE} and no workspace is selected: pass --workspace .`, - 0 + throw new CliUsageError( + `${options.dir} has no ${SOURCE_BASE_FILE} and no workspace is selected: pass --workspace .` ) } const catalog = cachedOrRefuse( @@ -721,7 +121,14 @@ export function attachWorkflowSourceCheck(source: Command): void { { incremental: true } ) } - reportCheck(profile.output, findings, { catalogHash: catalog.hash }, options.verbose) + reportCheck( + profile.output, + findings, + { catalogHash: catalog.hash }, + { + verbose: options.verbose === true, + } + ) } ) } diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-get.ts b/packages/sim-cli/src/commands/protocol/workflow-source-get.ts index 1a8d25ac105..4623eb3e402 100644 --- a/packages/sim-cli/src/commands/protocol/workflow-source-get.ts +++ b/packages/sim-cli/src/commands/protocol/workflow-source-get.ts @@ -1,16 +1,10 @@ import { toRecord } from '@sim/utils/object' -import type { BlockCatalog } from '@sim/workflow-code' -import { SOURCE_BASE_FILE, SOURCE_ENTRY_FILE, SOURCE_LOCK_FILE } from '@sim/workflow-code/limits' -import { - SDK_GUIDE_FILE, - SOURCE_COMPILER_OPTIONS, - typecheckCommand, -} from '@sim/workflow-sdk/codegen' +import { SOURCE_BASE_FILE } from '@sim/workflow-code/limits' import type { Command } from 'commander' import type { GetWorkflowSourceResponse } from '../../generated/v2-api' -import { SimApiError } from '../../http/client' -import { callsOperations, type OperationClient } from '../../runtime/called-operations' +import { callsOperations } from '../../runtime/called-operations' import { buildRequest } from '../../runtime/request' +import { SourceDirError } from './local-errors' import { absoluteDirOption, assertPrintedSourcePaths, @@ -18,243 +12,15 @@ import { locallyEditedFiles, readPreviousLock, readSourceJson, - SOURCE_SDK_DIR as SDK_DIR, - SOURCE_EXAMPLES_DIR, - SOURCE_STARTER_FILE, - updateSourceDir, - WORKSPACE_FOLDER_BUSY_HELP, - withWorkspaceFolderLock, - writeBack, + withSourceCommandLock, } from './source-dir' -import { sourceExitCodesTable } from './source-exit-codes' -import { - assertSdkMajor, - finishTree, - loadCatalog, - loadWorkspaceTypes, - printTreeResult, - refreshTsconfigs, - type SourceTree, - type SourceTreeProfile, - sourceTreeChange, - TSCONFIG_FULL_FILE, - TSCONFIG_MARKER, - WORKSPACE_TYPES_FILE, - type WorkspaceTypes, - withSourceExitCode, -} from './source-tree' +import { sourceExitCodesHelp } from './source-exit-codes' +import { printTreeResult, withSourceExitCode } from './source-tree' import { findingsOutput } from './source-ui' -import { storedEdgesWrite } from './stored-edges' +import { type PrintedSource, writeSourceDirectory } from './workflow-dir' const DIR_FLAG = 'dir' const FORCE_FLAG = 'force' -/** The public guide to workflow source, which AGENTS.md points an agent at. */ -const SOURCE_DOCS_URL = 'https://docs.sim.ai/workflows/code' - -type WorkflowSource = GetWorkflowSourceResponse['data'] - -/** - * A `tsconfig.json` that lets an editor or `tsc -p` resolve `@sim/workflow-sdk` to the written - * declarations and type-checks `workflow.ts` with the SDK files `sdkFiles`. - */ -function workflowTsconfig(sdkFiles: readonly string[]): string { - return `${TSCONFIG_MARKER}\n${JSON.stringify( - { - compilerOptions: { - ...SOURCE_COMPILER_OPTIONS, - skipLibCheck: true, - paths: { '@sim/workflow-sdk': [`./${SDK_DIR}/core.d.ts`] }, - }, - include: [SOURCE_ENTRY_FILE, ...sdkFiles], - }, - null, - 2 - )}\n` -} - -/** - * The reference below the primer in `.sim/AGENTS.md`, by task: the edit loop in detail, a new or - * forked workflow, the file shape, lookups, the rules and every exit code. The docs page has the - * rest. - */ -const AGENTS_GUIDE = `# Reference - -The primer above is enough to edit; read a section here when a step needs it. - -## Edit loop in detail - -1. Edit \`${SOURCE_ENTRY_FILE}\` (and \`functions/*\`). \`${SOURCE_EXAMPLES_DIR}/\` holds correct examples to copy from: \`cookbook.ts\` (agent with structured output, condition, function, table query and insert, knowledge search, Slack, human in the loop, loop, response), \`integrations.ts\` (webhook trigger, weekday cron schedule, HTTP request with headers, table update by row id, Gmail, an error path) and \`agents.ts\` (router, agent tools: custom tool, MCP tool, integration operation, child workflow; run a child workflow; parallel). Their resource keys are examples; this workspace's are in \`${WORKSPACE_TYPES_FILE}\`. -2. \`sim workflows source check --dir .\`: compiles offline with the server's compiler, in milliseconds. It catches what \`tsc\` cannot (duplicate names, cycles, literal secrets), and fails (exit 1) on a missing required field, an unknown option or output path, a wrong value kind, an out-of-range value, an unwired condition or router branch or a literal secret in a block you added (\`--strict\`: in every block). It also adds the blocks the source now uses to \`tsconfig.json\`. -3. \`${typecheckCommand()}\`: about a second; catches what step 2 and the dry run do not (such as a reference to a start input the trigger does not declare). It runs the TypeScript release the declarations are built for (the directory has no \`package.json\`; plain \`bunx tsc\` would run whatever version it finds). \`tsconfig.json\` includes only the blocks the source used at the last check, get or apply: a builder from another block fails with \`has no exported member\` until you run step 2 (or use \`${typecheckCommand(TSCONFIG_FULL_FILE)}\`, every current block). The dry run is the authority: a printed line you did not edit that \`tsc\` flags and the dry run accepts is fine. -4. \`sim workflows source status --dir .\`: the files you changed (\`sim workflows source diff --dir .\` shows the edits), whether the workflow changed on the server since the get, and the command to run next (\`--offline\`: skip the server). Then \`sim workflows source apply --dir . --dry-run\` until it exits 0. -5. \`sim workflows source apply --dir . --yes\`: saves the draft, refusing errors, block deletions without \`--allow-delete\` and a stale directory, so it needs no separate dry run (without \`--yes\` and a terminal it prints the plan and exits 14). \`--deploy\` also deploys it; without it, a deployed workflow's next \`sim workflows deploy \` is printed. It then rewrites \`${SOURCE_ENTRY_FILE}\` in canonical form (order, quoting, defaults left out). -6. Exit 5 (\`SOURCE_STALE\`, HTTP 409, or \`E_APPLY_STALE\` on a dry run): the workflow changed and nothing was saved. Run \`sim workflows source get --dir \`, reapply your edit there, dry-run, apply. A get into this directory refuses to overwrite your unapplied edits unless you pass \`--force\`. Exit 5 right after an apply that ended without an answer (a timeout, a dropped connection) means the draft changed, by that apply or by someone else: get into a fresh directory and compare. Only when it holds your edit, run \`sim workflows source get --dir . --force\`; otherwise reapply your edit in the fresh directory. - -Editing the \`workflow('…')\` name or description renames or redescribes the workflow; a name another workflow in its folder holds is refused with \`E_NAME_TAKEN\`. Lines starting \`note:\` explain something the write did differently than the source says; read them. - -## A new workflow, or a fork - -\`sim workflows source create --dir --workspace \` (without \`--dry-run\` or \`--yes\`) on a directory without \`${SOURCE_ENTRY_FILE}\` writes a starter \`${SOURCE_ENTRY_FILE}\`, the types of that workspace, \`tsconfig.json\` and this guide, records the workspace in \`${SOURCE_STARTER_FILE}\`, and creates nothing. Until the workflow exists there is no \`${SOURCE_BASE_FILE}\`, so \`status\`, \`diff\` and \`apply\` refuse the directory: edit the starter, check and type-check it, then \`sim workflows source create --dir --workspace --dry-run\` and the same with \`--yes\` (the starter prints both with its workspace filled in): it prints the new workflow id and writes the lock files. A create goes to the starter's workspace whatever the profile's default, unless \`--workspace\` names another. Exit 1 with \`SOURCE_NAME_TAKEN\` means another workflow has that name: pass \`--name\`. To fork a workflow, copy its directory, delete the copy's \`.sim/\` and run \`sim workflows source create --dir --workspace --name --yes\`. - -## File shape - -\`\`\`ts -// @sim-sdk 1 -import { response, trigger, workflow } from '@sim/workflow-sdk' - -export default workflow('Workflow name', {}, () => { - const start = trigger.start('Start', {}) - const done = response('Done', { body: { ok: true } }) - - start.to(done) -}) -\`\`\` - -The \`// @sim-sdk 1\` header must be the first line, and the file has exactly one \`export default workflow(name, options, () => { ... })\`; \`options\` holds \`description\` and \`variables\` and may be \`{}\`. - -## Finding a builder and its types - -- \`sim workflows source params \`: one builder's params (required first, enums, defaults) and outputs; a namespace (\`gmail\`) lists its builders; a miss names the closest. Builders are named for the service, not the block type: \`gmail_v2\` is \`gmail.*\`, \`workflow_input\` is \`runWorkflow\`, the HTTP request block is \`api\`, and \`legacy.*\` builders are old versions not to write. -- \`${SDK_DIR}/core.d.ts\`: the SDK (\`workflow\`, \`t\`, \`env\`, \`credential\`, \`table\`, \`loop\`, \`parallel\`, \`BlockMeta\`). -- \`${SDK_DIR}/blocks/.d.ts\`: the builders of a block the source uses, with their params and outputs (check adds the file of a block an edit starts using). Do not read them, or \`${SDK_DIR}/models.d.ts\`, whole: \`params\` prints the part you need, and \`grep\` finds a model id. -- \`${WORKSPACE_TYPES_FILE}\`: this workspace's tables, knowledge bases, credentials, workflows and env names. - -## Rules - -- Beside the header and the imports from \`@sim/workflow-sdk\`, the file holds only the default-exported workflow. Its callback holds block declarations (\`const x = builder('Name', params)\`) and wiring (\`x.to(y)\`). Helpers, loops, conditionals and other imports have no stored form, so the compiler refuses them: use \`loop.forEach\`, \`condition\` and \`runWorkflow\` blocks instead. -- Logic that does not fit a block goes in a function block with \`code: codeFile('./functions/name.js')\` (references as the primer says). -- Hosted Sim supplies an agent's model key. On self-hosted Sim the agent needs \`apiKey: env.NAME\` (a secret the workspace holds); without it \`check\` reports \`W_REQUIRED_MISSING: apiKey is required\`. -- A child workflow's output is \`c.result\`, untyped: its \`response\` block's \`{ data, status, headers }\`, so a field of the child's response body is \`c.result.data.field\` (\`${SDK_DIR}/${SDK_GUIDE_FILE}\` "How it runs" has the rest). -- A table block's row is \`{ id, data }\`: read a cell as \`row.data.column\` (in a loop over \`query.rows\`, \`p.currentItem.data.column\`). -- Keep \`${SOURCE_LOCK_FILE}\` and \`${SOURCE_BASE_FILE}\` as written: they bind each block to its stored id and make the next apply safe. - -## Diagnostics and exit codes - -Findings print as \`file:line:column severity CODE: message\`, then the code frame and a hint. \`E_*\` refuses the whole apply; \`W_*\` is saved as written and names a real problem. \`check\` prints the \`W_*\` codes it fails on as \`error\` on a block you added; the apply still saves them as warnings. A path no output has (\`W_UNKNOWN_OUTPUT_PATH\`), in a block you added or changed, is the error \`E_APPLY_UNRESOLVED_REF\` in both, whatever \`--strict\` says; a path the run may still resolve (\`W_OUTPUT_PATH_UNDECLARED\`: an input the start does not declare, \`x.error\` with no error edge) stays a warning. Exit codes of every \`workflows source\` command: - -${sourceExitCodesTable('sim workflows source')} - -The full guide: ${SOURCE_DOCS_URL} -` - -/** The printed source a tree write saves: a get's answer, or an applied create's. */ -type PrintedSource = Pick - -/** - * Writes the source tree in one staged update. Declarations are regenerated only when the index - * on disk names another catalog or another generator, and replaced whole so a block the catalog - * dropped loses its file. A refused manifest removes the `workspace.d.ts` an earlier get wrote, - * function files the new lock no longer names are pruned, `.gitattributes` gains the LF lines it - * lacks, and a `tsconfig.json` the CLI did not write is left as it is. - */ -async function writeSourceTree( - dir: string, - source: PrintedSource, - catalog: BlockCatalog, - workspaceTypes: WorkspaceTypes | null, - blockTypes: readonly string[] -): Promise { - const [previousLock, tree] = await Promise.all([ - readPreviousLock(dir), - sourceTreeChange(dir, catalog, workspaceTypes, WORKFLOW_TREE, { - sources: [source.files[SOURCE_ENTRY_FILE] ?? ''], - blockTypes, - }), - ]) - const edges = storedEdgesWrite(source, catalog) - const written = await writeBack( - dir, - source, - previousLock, - edges ? { ...tree.change, write: [...tree.change.write, edges] } : tree.change - ) - return finishTree(dir, written, workspaceTypes, tree) -} - -/** The workflow directory's tree beside the printed source. */ -export const WORKFLOW_TREE: SourceTreeProfile = { - kind: 'workflow', - agents: AGENTS_GUIDE, - tsconfig: workflowTsconfig, -} - -/** The `workflow.ts` a new directory starts from: the smallest workflow that compiles. */ -const STARTER_SOURCE = `// @sim-sdk 1 -import { response, trigger, workflow } from '@sim/workflow-sdk' - -export default workflow('New workflow', { description: 'Describe what this workflow does.' }, () => { - const start = trigger.start('Start', { inputFormat: { message: 'string' } }) - const done = response('Done', { body: { received: start.message } }) - - start.to(done) -}) -` - -/** - * Starts a new source directory: a starter `workflow.ts` plus the tree a get writes (the - * declarations for the workspace catalog and its resources, `tsconfig.json`, the agent guide), - * with no lock or base, since no workflow exists yet, and {@link SOURCE_STARTER_FILE} naming the - * workspace those declarations are for. Only for a directory with no `workflow.ts`. - */ -export async function writeSourceScaffold( - client: OperationClient<'getSourceCatalog' | 'getSourceResources'>, - dir: string, - workspaceId: string -): Promise { - const [catalog, workspaceTypes] = await Promise.all([ - loadCatalog(client, { workspaceId, blockTypes: [] }), - loadWorkspaceTypes(client, workspaceId), - ]) - assertSdkMajor(catalog) - const tree = await sourceTreeChange(dir, catalog, workspaceTypes, WORKFLOW_TREE, { - sources: [STARTER_SOURCE], - }) - const write = [ - [SOURCE_ENTRY_FILE, STARTER_SOURCE] as const, - [SOURCE_STARTER_FILE, `${JSON.stringify({ workspaceId }, null, 2)}\n`] as const, - ...tree.change.write, - ] - await updateSourceDir(dir, { ...tree.change, write }) - return finishTree(dir, write.length, workspaceTypes, tree) -} - -/** - * Writes a printed source with everything that makes the directory editable: the declarations - * for the workspace catalog and its resources, `tsconfig.json`, the agent guide and the LF lines. - * Shared by `get --dir` and `create --dir`, so a created workflow's directory is the one a get - * would have written. - */ -export async function writeSourceDirectory( - client: OperationClient<'getSourceCatalog' | 'getSourceResources'>, - dir: string, - source: PrintedSource, - blockTypes: readonly string[] -): Promise { - const [catalog, workspaceTypes] = await Promise.all([ - loadCatalog(client, { - workspaceId: source.workspaceId, - workflowId: source.workflowId, - catalogHash: source.base.catalogHash, - blockTypes: [...blockTypes], - }), - loadWorkspaceTypes(client, source.workspaceId), - ]) - assertSdkMajor(catalog) - return writeSourceTree(dir, source, catalog, workspaceTypes, blockTypes) -} - -/** - * After an apply, points the CLI's tsconfig files at the blocks the applied source uses, so a - * block the edit added is type-checked by `tsconfig.json` from then on. - */ -export async function refreshWorkflowTsconfigs( - dir: string, - source: Pick -): Promise { - await refreshTsconfigs(dir, source.workspaceId, WORKFLOW_TREE, { - sources: [source.files[SOURCE_ENTRY_FILE] ?? ''], - }) -} /** The workflow `.sim/base.json` says the directory holds, or `null` when it names none. */ async function heldWorkflowId(dir: string): Promise { @@ -269,10 +35,9 @@ async function heldWorkflowId(dir: string): Promise { async function assertSameWorkflow(dir: string, workflowId: string): Promise { const held = await heldWorkflowId(dir) if (held === null || held === workflowId) return - throw new SimApiError( + throw new SourceDirError( `${dir} holds the source of workflow ${held}, not ${workflowId}. Get it into another directory, or pass --force to replace it.`, - 0, - 'SOURCE_WORKFLOW_MISMATCH' + { code: 'SOURCE_WORKFLOW_MISMATCH' } ) } @@ -283,20 +48,12 @@ async function assertSameWorkflow(dir: string, workflowId: string): Promise { const edited = await locallyEditedFiles(dir, source, await readPreviousLock(dir)) if (edited.length === 0) return - throw new SimApiError( + throw new SourceDirError( `${dir} has edits no apply has saved (${edited.join(', ')}). Apply them, get into another directory, or pass --force to discard them.`, - 0, - 'SOURCE_LOCAL_EDITS' + { code: 'SOURCE_LOCAL_EDITS' } ) } -const GET_EXIT_CODES_HELP = ` -Exit codes with --dir: 0 written; 1 the command failed, the directory is a workspace source -directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or it holds another workflow's source -(SOURCE_WORKFLOW_MISMATCH) or edits no apply has saved (SOURCE_LOCAL_EDITS: apply them, get into -another directory, or pass --force); 7 the workflow is too large to print as source; -9 rate limited (HTTP 429) after the CLI's retries: wait a minute and run it again.` - /** * Teaches the generated `workflows source get` leaf `--dir `: instead of printing the * response, it writes `workflow.ts`, `functions/*`, `.sim/lock.json`, `.sim/base.json`, SDK @@ -329,7 +86,7 @@ export function attachWorkflowSourceGet(source: Command): void { "With --dir: replace another workflow's source or edits no apply has saved" ) absoluteDirOption(get) - get.addHelpText('after', `${GET_EXIT_CODES_HELP}${WORKSPACE_FOLDER_BUSY_HELP}`) + get.addHelpText('after', sourceExitCodesHelp('workflows source get', 'Exit codes with --dir')) get.action(async (workflowId: string, _options: unknown, command: Command): Promise => { const flags = command.optsWithGlobals() as Record const dir = flags[DIR_FLAG] @@ -341,7 +98,7 @@ export function attachWorkflowSourceGet(source: Command): void { } return } - await withWorkspaceFolderLock(dir, async () => { + await withSourceCommandLock(dir, async () => { const force = flags[FORCE_FLAG] === true await assertSourceDirKind(dir, 'workflow') if (!force) await assertSameWorkflow(dir, workflowId) diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-params.ts b/packages/sim-cli/src/commands/protocol/workflow-source-params.ts index 632867ed1c2..03262398508 100644 --- a/packages/sim-cli/src/commands/protocol/workflow-source-params.ts +++ b/packages/sim-cli/src/commands/protocol/workflow-source-params.ts @@ -5,7 +5,7 @@ import { lookupBuilder, renderBuilderReference } from '@sim/workflow-sdk/codegen import type { Command } from 'commander' import { printLine } from '#sim-cli/output/io' import { profileFrom } from '../../context' -import { SimApiError } from '../../http/client' +import { CliUsageError, SourceDirError } from './local-errors' import { printProtocolResult } from './result' import { readSourceJson, workspaceFolderOf } from './source-dir' import { cachedCatalog, cachedSourceCatalog } from './source-tree' @@ -36,12 +36,11 @@ async function lookupCatalog(dir: string, profileWorkspace: string | null): Prom : ((await workspaceFolderOf(dir))?.workspaceId ?? profileWorkspace) const cached = workspaceId ? cachedCatalog(workspaceId) : null if (cached) return cached - throw new SimApiError( + throw new SourceDirError( workspaceId ? `No block catalog is cached for workspace ${workspaceId}. Run \`sim workflows source get --dir \` (or \`sim workspaces source get --dir \`) once, then look it up again.` : `${dir} is not a source directory and no workspace is selected: pass --dir or --workspace .`, - 0, - 'CATALOG_NOT_CACHED' + { code: 'CATALOG_NOT_CACHED' } ) } @@ -80,10 +79,9 @@ export function attachWorkflowSourceParams(source: Command): void { else printLine(`${found.name} is declared in .sim/sdk/core.d.ts:\n\n${found.declaration}`) return case 'missing': - throw new SimApiError( + throw new CliUsageError( `No builder or namespace is named ${query}.${found.hint ? ` ${found.hint}` : ''}`, - 0, - 'BUILDER_UNKNOWN' + { code: 'BUILDER_UNKNOWN' } ) } }) diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-status.test.ts b/packages/sim-cli/src/commands/protocol/workflow-source-status.test.ts index d52c9873ef8..e491378fbfa 100644 --- a/packages/sim-cli/src/commands/protocol/workflow-source-status.test.ts +++ b/packages/sim-cli/src/commands/protocol/workflow-source-status.test.ts @@ -370,6 +370,16 @@ describe('workflows source status', () => { expect(process.exitCode).toBeUndefined() }) + it('fails on a refusal of its own directory instead of reporting the server unreachable', async () => { + write(root, 'workflow.ts', `${ENTRY}// edited\n`) + mkdirSync(join(root, '.sim/sdk/workspace.d.ts'), { recursive: true }) + write(root, '.sim/sdk/workspace.manifest.json', '{}\n') + mockRequest.mockResolvedValue(dryRun()) + + await expect(run(root, 'status')).rejects.toMatchObject({ name: 'SourceDirError' }) + expect(process.exitCode).toBeUndefined() + }) + it('exits 7 when the server refuses the workflow as too large, as a get does', async () => { mockRequest.mockRejectedValue(new SimApiError('Too large', 413, 'PAYLOAD_TOO_LARGE')) diff --git a/packages/sim-cli/src/commands/protocol/workflow-source-status.ts b/packages/sim-cli/src/commands/protocol/workflow-source-status.ts index 5bcda703e6b..1656c0f2dba 100644 --- a/packages/sim-cli/src/commands/protocol/workflow-source-status.ts +++ b/packages/sim-cli/src/commands/protocol/workflow-source-status.ts @@ -51,7 +51,7 @@ import { } from './source-status' import { refreshWorkflowWorkspaceTypes, withSourceExitCode } from './source-tree' import type { Diagnostic } from './source-ui' -import { readBase, readLock, readSourceFiles } from './workflow-source-apply' +import { readBase, readLock, readSourceFiles } from './workflow-dir' type PlanData = PlanWorkflowSourceResponse['data'] diff --git a/packages/sim-cli/src/commands/protocol/workspace-source-apply.test.ts b/packages/sim-cli/src/commands/protocol/workspace-source-apply.test.ts index cd685259b45..9a4705c5b00 100644 --- a/packages/sim-cli/src/commands/protocol/workspace-source-apply.test.ts +++ b/packages/sim-cli/src/commands/protocol/workspace-source-apply.test.ts @@ -714,6 +714,30 @@ describe('workspaces source apply', () => { expect(Object.keys(written)).not.toContain('workflows/ingest/functions/old.js') }) + it('never prunes a lock entry that names the functions folder rather than a function file', async () => { + const lock = (codeFiles: string[]) => + `${JSON.stringify({ version: 1, blocks: Object.fromEntries(codeFiles.map((file, index) => [`b${index}`, { id: `blk_${index}`, codeFile: file }])), variables: {} })}\n` + write('workflows/ingest/.sim/lock.json', lock(['functions/', 'functions/score.js'])) + write('workflows/ingest/functions/score.js', 'return 1\n') + write('workflows/ingest/functions/mine.js', 'return 3\n') + request.mockResolvedValue( + answer({ + applied: [{ kind: 'workflow', key: 'ingest', action: 'update', id: 'wf_1' }], + files: { + 'workflows/ingest/workflow.ts': INGEST, + 'workflows/ingest/functions/score.js': 'return 1\n', + 'workflows/ingest/.sim/lock.json': lock(['functions/score.js']), + 'workflows/ingest/.sim/base.json': '{}\n', + }, + }) + ) + + await apply('--yes') + + expect(readFileSync(join(dir, 'workflows/ingest/functions/mine.js'), 'utf8')).toBe('return 3\n') + expect(existsSync(join(dir, 'workflows/ingest/functions/score.js'))).toBe(true) + }) + it('keeps every function file of a workflow whose new lock does not read as one', async () => { write( 'workflows/ingest/.sim/lock.json', diff --git a/packages/sim-cli/src/commands/protocol/workspace-source-apply.ts b/packages/sim-cli/src/commands/protocol/workspace-source-apply.ts index 20aa7b7e2de..66ca8109d33 100644 --- a/packages/sim-cli/src/commands/protocol/workspace-source-apply.ts +++ b/packages/sim-cli/src/commands/protocol/workspace-source-apply.ts @@ -22,7 +22,6 @@ import { import { serverResourceChanges } from '@sim/workspace-code/status' import type { Command } from 'commander' import { printError } from '#sim-cli/output/io' -import { DEFERRED_INTERRUPT_BOUND_MS, deferInterrupts } from '#sim-cli/terminal/interrupts' import { setSoftExitCode } from '../../embed-context' import { SOURCE_PARTIAL_EXIT_CODE } from '../../exit-codes' import type { @@ -34,6 +33,7 @@ import { SimApiError } from '../../http/client' import { apiCommand, type OperationClient } from '../../runtime/called-operations' import { searchTasks } from '../search' import { gitProvenance } from './git-provenance' +import { SourceDirError } from './local-errors' import { printProtocolResult } from './result' import { absoluteDirOption, @@ -49,20 +49,19 @@ import { writtenFiles, writtenRecord, } from './source-dir' -import { SOURCE_WRITE_EXIT_CODES_HELP } from './source-exit-codes' +import { SOURCE_WRITE_EXIT_CODES_HELP, sourceExitCodesHelp } from './source-exit-codes' +import { destructiveChanges } from './source-status' import { cachedCatalog } from './source-tree' import { - confirm, createdText, findingsOutput, - planNext, printRefusal, refusedExitCode, shellWord, - stopAtPlan, verboseFlag, writeMode, } from './source-ui' +import { type NextStep, runSourceWrite } from './source-write' import { removedWorkflowRecords, unrecordedFoldersWarning, @@ -78,10 +77,11 @@ import { import { assertWorkspaceDir, assertWorkspacePaths, + ONLY_KINDS_HELP, + parseKinds, readWorkspaceDir, type WorkspaceSubmission, } from './workspace-source-dir' -import { ONLY_KINDS_HELP, parseKinds } from './workspace-source-get' type ApplyData = ApplyWorkspaceSourceResponse['data'] type Change = ApplyData['plan'][number] @@ -96,27 +96,6 @@ interface ApplyOptions { to?: string } -const EXIT_CODES_HELP = ` -Exit codes: 0 applied (or a clean dry run); 1 the source has errors (each printed as -file:line:column with its frame and hint), or changes were held back because they use a declared -secret the workspace has no value for or a declared credential key nothing binds to (the rest was -applied: set the secret or connect the credential, then apply again), the directory was read from -another workspace than the one --workspace names (SOURCE_OTHER_WORKSPACE: drop --workspace, or -promote it with --to; without --workspace an apply targets the workspace the directory was read -from), its -base records binary files no get downloaded (SOURCE_BLOBS_NOT_FETCHED: get again first), or the -command failed; 5 a resource changed or was deleted on the server since the directory was read -(SOURCE_STALE: get it again into a fresh directory, reapply the edit, retry); 6 a resource (a -workflow or its folder, a table) is locked (a locked block exits 1, E_APPLY_LOCKED_BLOCK); 7 the directory is too large; 8 the apply stopped part-way on a server failure: what was -written is saved into the directory, and applying again continues. An apply a resource refused -part-way (stale, locked, forbidden) saves what was written too, and exits as that refusal: 5, 6 -or 1; 9 rate limited (HTTP 429) after the CLI's retries, before anything was written: wait a -minute and apply again (binary files already uploaded are not sent again); 10 another apply is -writing the workspace (SOURCE_BUSY), or another command is using the directory -(SOURCE_DIR_BUSY): nothing was written, run the command again once the other finished; 14 -without --yes and with no terminal to ask: the plan was printed with no errors and nothing was -written (run the same command with --yes to apply it).` - const SYMBOLS: Record = { create: '+', update: '~', @@ -242,12 +221,6 @@ function printPlan( } } -/** The changes that delete something (a resource, rows, documents, columns, a deployment). */ -export function destructiveChanges(changes: readonly Change[]): number { - return changes.filter((change) => change.action === 'delete' || change.destructive !== undefined) - .length -} - /** * The binary paths an apply refusal names as not staged, when those are its only errors: the CLI * stages them and applies once more. `null` for any other refusal. @@ -453,11 +426,9 @@ async function promoted( allowDelete: boolean ): Promise { if (baseWorkspaceOf(submission.base) === target) { - throw new SimApiError( - `${dir} was read from workspace ${target}; apply it without --to.`, - 0, - 'SOURCE_PROMOTE_SAME_WORKSPACE' - ) + throw new SourceDirError(`${dir} was read from workspace ${target}; apply it without --to.`, { + code: 'SOURCE_PROMOTE_SAME_WORKSPACE', + }) } const recordPath = promotionRecordPath(target) const record = recordPath @@ -469,7 +440,7 @@ async function promoted( }) const targetBase = parseWorkspaceBase(data.base).base if (!targetBase) { - throw new SimApiError(`Workspace ${target} answered with a base this CLI cannot read.`, 0) + throw new SourceDirError(`Workspace ${target} answered with a base this CLI cannot read.`) } const composed = promotionSubmission({ files: submission.files, @@ -664,7 +635,10 @@ export function attachWorkspaceSourceApply(source: Command): void { ) .allowExcessArguments(false) .description('Apply a workspace source directory') - .addHelpText('after', `${EXIT_CODES_HELP}${SOURCE_WRITE_EXIT_CODES_HELP}`) + .addHelpText( + 'after', + `${sourceExitCodesHelp('workspaces source apply')}${SOURCE_WRITE_EXIT_CODES_HELP}` + ) .action(async (options: ApplyOptions) => { const mode = writeMode(options) const only = parseKinds(options.only) @@ -775,7 +749,7 @@ export function attachWorkspaceSourceApply(source: Command): void { const notSent = notSentFindings(submission.ignored) findings.print(notSent) /** Prints `data` as the result, with the findings about files the CLI did not send. */ - const report = (data: ApplyData, next?: { command: string; reason: string }) => { + const report = (data: ApplyData, next?: NextStep) => { const diagnostics = [...notSent, ...data.diagnostics] const result = { ...summarize({ ...data, diagnostics }), ...(next ? { next } : {}) } printProtocolResult( @@ -784,53 +758,48 @@ export function attachWorkspaceSourceApply(source: Command): void { { created: createdText(result.created), ...(next ? { next: next.command } : {}) } ) } - let preview: ApplyData | null = null - if (mode !== 'write') { - preview = reported(await send(submission, true)) - printPlan(preview.plan, preview.diagnostics) - findings.print(preview.diagnostics) - const errors = preview.diagnostics.some((finding) => finding.severity === 'error') - if (errors) setSoftExitCode(refusedExitCode(preview.diagnostics)) - if (mode === 'dry-run' || refusesApply(preview)) { - report(preview) - return - } - if (mode === 'plan') { - stopAtPlan() - report(preview, planNext(applyCommand(options))) - return - } - if (!(await confirm(`Apply these changes to workspace ${workspaceId}?`))) { - throw new SimApiError('Nothing was applied.', 0, 'CANCELLED') - } - } - - // Without a preview, stage the blobs the base does not hold as they are. - await stage(submission, preview ? new Set(preview.needBlobs) : blobsToStage(submission)) - const data = await deferInterrupts( - async () => { + const data = await runSourceWrite({ + mode, + preview: async () => reported(await send(submission, true)), + show: (preview) => { + printPlan(preview.plan, preview.diagnostics) + findings.print(preview.diagnostics) + }, + refuses: refusesApply, + // Without a preview, stage the blobs the base does not hold as they are. + beforeCommit: (preview) => + stage(submission, preview ? new Set(preview.needBlobs) : blobsToStage(submission)), + commit: async () => { const answer = reported(await send(submission, false)) findings.print(answer.diagnostics) + return answer + }, + save: async (answer) => { // A promotion leaves the directory bound to the workspace it was read from, and // records what it left in the target for the next promotion to compare with. if (promotion) { await savePromotion(options.dir, promotion.target, promotion, answer) - } else { - const workspaceCatalog = cachedCatalog(workspaceId) - const { records, unrecorded } = workspaceCatalog - ? await workflowFolderRecords(client, workspaceId, workspaceCatalog, answer.files) - : { records: [], unrecorded: [] } - await saveApplied(options.dir, answer, submission, written, records) - const unrecordedWarning = unrecordedFoldersWarning(unrecorded) - if (unrecordedWarning) printError(unrecordedWarning) + return } - return answer + const workspaceCatalog = cachedCatalog(workspaceId) + const { records, unrecorded } = workspaceCatalog + ? await workflowFolderRecords(client, workspaceId, workspaceCatalog, answer.files) + : { records: [], unrecorded: [] } + await saveApplied(options.dir, answer, submission, written, records) + const unrecordedWarning = unrecordedFoldersWarning(unrecorded) + if (unrecordedWarning) printError(unrecordedWarning) + }, + report, + question: `Apply these changes to workspace ${workspaceId}?`, + command: applyCommand(options), + interrupt: { + verb: 'apply', + dir: options.dir, + noun: 'workspace', + ifAnswerLost: answerLostHint(options.dir), }, - { - waiting: `Interrupted: the apply was already sent. Waiting up to ${DEFERRED_INTERRUPT_BOUND_MS / 1000}s for its answer and saving it into ${options.dir}, so the directory matches the workspace; press Ctrl-C again to stop now.`, - abandoned: `Stopped before the apply answered. ${answerLostHint(options.dir)}`, - } - ) + }) + if (!data) return if (data.failed) { printError('The apply stopped part-way; each planned change:') printPlan(data.plan, data.diagnostics, data) diff --git a/packages/sim-cli/src/commands/protocol/workspace-source-blobs.ts b/packages/sim-cli/src/commands/protocol/workspace-source-blobs.ts index ed1a1d94e12..f985c1c0c3e 100644 --- a/packages/sim-cli/src/commands/protocol/workspace-source-blobs.ts +++ b/packages/sim-cli/src/commands/protocol/workspace-source-blobs.ts @@ -42,6 +42,7 @@ import type { } from '../../generated/v2-api' import { SimApiError } from '../../http/client' import type { OperationClient } from '../../runtime/called-operations' +import { SourceDirError } from './local-errors' import { checkedPath, FILE_CONCURRENCY, @@ -181,16 +182,15 @@ export function editedBlobs( export function assertListedBlobs(blobs: readonly ListedBlob[]): void { for (const blob of blobs) { if (!isWorkspaceSourcePath(blob.path) || !isBlobPath(blob.path)) { - throw new SimApiError(`The workspace source named an unsafe file path: ${blob.path}`, 0) + throw new SourceDirError(`The workspace source named an unsafe file path: ${blob.path}`) } if (blob.size > WORKSPACE_SOURCE_LIMITS.maxBlobBytes) { - throw new SimApiError( - `${blob.path} is ${blob.size} bytes, over the ${WORKSPACE_SOURCE_LIMITS.maxBlobBytes}-byte limit a directory holds.`, - 0 + throw new SourceDirError( + `${blob.path} is ${blob.size} bytes, over the ${WORKSPACE_SOURCE_LIMITS.maxBlobBytes}-byte limit a directory holds.` ) } if (!blob.href.startsWith('/') || !downloadOf(blob.href)) { - throw new SimApiError(`${blob.path} names a download this CLI cannot fetch: ${blob.href}`, 0) + throw new SourceDirError(`${blob.path} names a download this CLI cannot fetch: ${blob.href}`) } } } @@ -207,7 +207,7 @@ async function downloadBlob( ): Promise { const download = downloadOf(blob.href) if (!download) { - throw new SimApiError(`${blob.path} names a download this CLI cannot fetch: ${blob.href}`, 0) + throw new SourceDirError(`${blob.path} names a download this CLI cannot fetch: ${blob.href}`) } const response = download.operation === 'downloadFile' @@ -219,14 +219,13 @@ async function downloadBlob( params: download.params, query: download.query, }) - if (!response.body) throw new SimApiError(`The download of ${blob.path} had no body.`, 0) + if (!response.body) throw new SourceDirError(`The download of ${blob.path} had no body.`) const hash = createHash('sha256') let size = 0 const mismatch = () => - new SimApiError( + new SourceDirError( `The download of ${blob.path} does not match the workspace source (expected ${blob.size} bytes${blob.sha256 ? `, SHA-256 ${blob.sha256}` : ''}); nothing was written.`, - 0, - 'SOURCE_BLOB_MISMATCH' + { code: 'SOURCE_BLOB_MISMATCH' } ) const check = new Transform({ transform(chunk: Buffer, _encoding, done) { @@ -455,10 +454,9 @@ export function assertBlobsFetched( ) if (missing.length === 0) return const shown = missing.slice(0, 5).join(', ') - throw new SimApiError( + throw new SourceDirError( `${dir} never downloaded ${missing.length} binary file(s) its base records (${shown}${missing.length > 5 ? ', …' : ''}), so applying it would delete them. Run \`sim workspaces source get --dir ${dir}\` to download them, then apply again.`, - 0, - 'SOURCE_BLOBS_NOT_FETCHED' + { code: 'SOURCE_BLOBS_NOT_FETCHED' } ) } @@ -512,10 +510,9 @@ export async function stageBlobs( content.byteLength !== blob.size || createHash('sha256').update(content).digest('hex') !== blob.sha256 ) { - throw new SimApiError( + throw new SourceDirError( `${blob.path} changed while the apply read it; nothing was applied. Apply again.`, - 0, - 'SOURCE_BLOB_CHANGED' + { code: 'SOURCE_BLOB_CHANGED' } ) } form.append(blob.sha256, new Blob([new Uint8Array(content)]), blob.sha256) diff --git a/packages/sim-cli/src/commands/protocol/workspace-source-check.test.ts b/packages/sim-cli/src/commands/protocol/workspace-source-check.test.ts index bcc98d3d476..85fb144101a 100644 --- a/packages/sim-cli/src/commands/protocol/workspace-source-check.test.ts +++ b/packages/sim-cli/src/commands/protocol/workspace-source-check.test.ts @@ -341,6 +341,39 @@ describe('sim workspaces source check', () => { expect(process.exitCode).toBe(1) }) + it('lists at most the 1000 findings a server answer would, ending in one I_FINDINGS_TRUNCATED, and counts every error', async () => { + const tables = 11 + for (let index = 0; index < tables; index++) { + writeFile( + `tables/plans${index}.ts`, + [ + '// @sim-sdk 1', + "import { column, defineTable } from '@sim/workflow-sdk/workspace'", + 'export default defineTable({', + ` name: 'plans${index}',`, + ' columns: { code: column.string({ required: true, unique: true }) },', + " rows: { mode: 'managed', key: 'code' },", + '})', + '', + ].join('\n') + ) + writeFile( + `tables/plans${index}.rows.jsonl`, + `${Array.from({ length: 150 }, () => '{').join('\n')}\n` + ) + } + + await check() + + const answer = stdoutJson() + const found = answer.diagnostics as Array<{ code: string; severity: string }> + expect(found).toHaveLength(1_001) + expect(found.filter((finding) => finding.code === 'I_FINDINGS_TRUNCATED')).toHaveLength(1) + expect(found.at(-1)).toMatchObject({ code: 'I_FINDINGS_TRUNCATED', severity: 'info' }) + expect(answer.errors).toBe(tables * 100) + expect(process.exitCode).toBe(1) + }) + it('accepts seed rows the columns hold, coercing a numeric string as the write does', async () => { writeFile( 'tables/seeds.ts', diff --git a/packages/sim-cli/src/commands/protocol/workspace-source-check.ts b/packages/sim-cli/src/commands/protocol/workspace-source-check.ts index c16c3caa9ca..a07d8deae35 100644 --- a/packages/sim-cli/src/commands/protocol/workspace-source-check.ts +++ b/packages/sim-cli/src/commands/protocol/workspace-source-check.ts @@ -1,21 +1,11 @@ import { join } from 'node:path' import { toRecord } from '@sim/utils/object' -import { KIND_DIRS } from '@sim/workspace-code' +import { KIND_DIRS, WORKSPACE_ANSWER_BOUND } from '@sim/workspace-code' import { WORKSPACE_BASE_FILE } from '@sim/workspace-code/layout' import type { Command } from 'commander' import { profileFrom } from '../../context' -import { SimApiError } from '../../http/client' import { apiCommand, type Connection } from '../../runtime/called-operations' -import { - absoluteDirOption, - directoryWorkspace, - readSourceJson, - SOURCE_DIR_BUSY_HELP, - withSourceDirLock, -} from './source-dir' -import { credentialRefused, unreachableReason } from './source-status' -import { cachedCatalog, loadCatalog, refreshWorkspaceTypes } from './source-tree' -import { verboseFlag } from './source-ui' +import { CliUsageError, SourceDirError } from './local-errors' import { type CheckFinding, checkWorkflowDir, @@ -29,7 +19,17 @@ import { withWorkflowInputs, workspaceWorkflowInputs, workspaceWorkflowKeys, -} from './workflow-source-check' +} from './source-check' +import { + absoluteDirOption, + directoryWorkspace, + readSourceJson, + SOURCE_DIR_BUSY_HELP, + withSourceDirLock, +} from './source-dir' +import { credentialRefused, unreachableReason } from './source-status' +import { cachedCatalog, loadCatalog, refreshWorkspaceTypes } from './source-tree' +import { verboseFlag } from './source-ui' import { assertWorkspaceDir, readWorkspaceDeclarations } from './workspace-source-dir' const CHECK_HELP = ` @@ -77,10 +77,9 @@ async function fetchedCatalog( // leaves the catalog uncached, which the get that caches it fixes once the server answers. const reason = credentialRefused(error) ? null : unreachableReason(error) if (reason === null) throw error - throw new SimApiError( + throw new SourceDirError( `No block catalog is cached for workspace ${workspaceId}, and downloading it failed: ${reason} Run \`sim workspaces source get --dir ${dir}\` once the server can be reached, then check again.`, - 0, - 'CATALOG_NOT_CACHED' + { code: 'CATALOG_NOT_CACHED' } ) } } @@ -119,9 +118,8 @@ export function attachWorkspaceSourceCheck(source: Command): void { (from) => `Run it without --workspace to check it against the catalog of ${from}.` ) if (!workspaceId) { - throw new SimApiError( - `${options.dir} has no ${WORKSPACE_BASE_FILE} and no workspace is selected: pass --workspace .`, - 0 + throw new CliUsageError( + `${options.dir} has no ${WORKSPACE_BASE_FILE} and no workspace is selected: pass --workspace .` ) } await withSourceDirLock(options.dir, async () => { @@ -174,7 +172,7 @@ export function attachWorkspaceSourceCheck(source: Command): void { profile.output, findings, { workflows: keys.length, catalogHash: catalog.hash }, - options.verbose + { verbose: options.verbose === true, bound: WORKSPACE_ANSWER_BOUND } ) }) } diff --git a/packages/sim-cli/src/commands/protocol/workspace-source-dir.ts b/packages/sim-cli/src/commands/protocol/workspace-source-dir.ts index 31166010e36..5c44370f2d8 100644 --- a/packages/sim-cli/src/commands/protocol/workspace-source-dir.ts +++ b/packages/sim-cli/src/commands/protocol/workspace-source-dir.ts @@ -2,7 +2,7 @@ * The workspace source directory `workspaces source get --dir` writes and `workspaces source * apply --dir` reads back: one folder per kind (`tables/`, `workflows//`, …), the base in * `.sim/base.json`, and the SDK declarations once at the root. Path safety, staged writes and the - * `.sim/written.json` record are the source directory's (`source-dir.ts`). + * `.sim/written.json` record are the source directory's (`source-dir/`). */ import { realpath, stat } from 'node:fs/promises' import { mapWithConcurrency } from '@sim/utils/concurrency' @@ -18,6 +18,7 @@ import { type BlobEntry, type FlatDiagnostic, isFileKind, + isWorkspaceKind, KIND_DIRS, WORKSPACE_KIND_SPECS, type WorkspaceDiagnosticCode, @@ -38,13 +39,22 @@ import { DECLARATION_EXTENSION, DECLARATION_MODULES, isWorkspaceSourcePath, + KIND_BY_DIR, kindOfPath, WORKSPACE_BASE_FILE, } from '@sim/workspace-code/layout' import { WORKSPACE_SOURCE_LIMITS } from '@sim/workspace-code/limits' -import { SOURCE_TOO_LARGE_EXIT_CODE } from '../../exit-codes' +import { + RATE_LIMITED_EXIT_CODE, + SOURCE_BUSY_EXIT_CODE, + SOURCE_DRIFT_EXIT_CODE, + SOURCE_LOCAL_CHANGES_EXIT_CODE, + SOURCE_PLANNED_EXIT_CODE, + SOURCE_STALE_EXIT_CODE, + SOURCE_TOO_LARGE_EXIT_CODE, +} from '../../exit-codes' import type { ApplyWorkspaceSourceBody } from '../../generated/v2-api' -import { SimApiError } from '../../http/client' +import { CliUsageError, SourceDirError } from './local-errors' import { assertSourceDirKind, checkedPath, @@ -131,14 +141,14 @@ Secrets never appear in source: a workflow reads an environment variable as \`en 1. Edit the files. 2. \`sim workspaces source check --dir .\`: compiles every workflow offline with the server's compiler, against what this directory declares: an unknown resource key (skills included) or table column, a child workflow's inputs, and an env name \`${KIND_DIRS.secret}\` lacks (a warning); and checks each \`${KIND_DIRS.table}/.rows.jsonl\` line against its table as the apply would. It fails (exit 1) on errors and on a missing required field, an unknown option or output path, a wrong value kind, an out-of-range value, an unwired condition or router branch or a literal secret in a block you added (\`--strict\`: in every block), regenerates \`.sim/sdk/workspace.d.ts\` and adds the blocks the workflows now use to \`tsconfig.json\`. In a workflow folder, \`sim workflows source check --dir .\` checks that one workflow the same way. Fix and rerun until it exits 0. 3. Run \`${typecheckCommand()}\` after every edit, before any dry run or apply, and fix what it reports: it takes about a second and catches what step 2 and the dry run do not (such as a reference to a start input the trigger does not declare). It type-checks the files with the TypeScript release the declarations are built for. \`tsconfig.json\` includes only the blocks the workflows used at the last check or get, which keeps it fast: a builder from another block fails with \`has no exported member\` until you run step 2 (or check with \`${typecheckCommand(TSCONFIG_FULL_FILE)}\`, every current block). \`sim workflows source params \` prints one builder's params and outputs; do not read the declarations whole. The dry run is the authority: a printed file you did not edit that \`tsc\` flags and the dry run accepts is fine, so leave it as it is. -4. \`sim workspaces source status --dir .\` before applying: your changes by resource (\`sim workspaces source diff --dir .\` shows the edits), what changed on the server since the get (a workflow edited in the editor included), the conflicts, and the command to run next. Exit 0 nothing changed, 11 only your edits, 12 the server changed since (\`--offline\`: skip the server). Then \`sim workspaces source apply --dir . --dry-run\`: shows the plan (\`+\` create, \`~\` update, \`=\` adopt an existing resource of the same name, \`-\` delete) and every error; fix and rerun until it exits 0. -5. \`sim workspaces source apply --dir . --yes\`: writes it and rewrites this directory from what was stored; it prints the plan it applied (without \`--yes\` and a terminal it prints the plan and exits 14, writing nothing). +4. \`sim workspaces source status --dir .\` before applying: your changes by resource (\`sim workspaces source diff --dir .\` shows the edits), what changed on the server since the get (a workflow edited in the editor included), the conflicts, and the command to run next. Exit 0 nothing changed, ${SOURCE_LOCAL_CHANGES_EXIT_CODE} only your edits, ${SOURCE_DRIFT_EXIT_CODE} the server changed since (\`--offline\`: skip the server). Then \`sim workspaces source apply --dir . --dry-run\`: shows the plan (\`+\` create, \`~\` update, \`=\` adopt an existing resource of the same name, \`-\` delete) and every error; fix and rerun until it exits 0. +5. \`sim workspaces source apply --dir . --yes\`: writes it and rewrites this directory from what was stored; it prints the plan it applied (without \`--yes\` and a terminal it prints the plan and exits ${SOURCE_PLANNED_EXIT_CODE}, writing nothing). ## Good to know - A column's object key stays bound to its stored column: to rename one keep the key and set \`name\` (\`email: column.string({ name: 'email_address' })\`). Changing the key drops the column and its cells. - Renaming a skill (its \`name\`) does not rewrite the workflows that use it: each agent keeps the name it stored until its \`skills\` list is next edited. The key is what binds them. -- A workflow folder you did not edit is sent with only its base, and the server does not compare it with the stored workflow: unlike other kinds, an apply does not report that a workflow changed on the server since the get (\`status\`, which sends every folder whole, does, and the next get shows it). An edit of the folder is checked against its own \`.sim/base.json\` and refused as stale (exit 5) when the workflow changed since. +- A workflow folder you did not edit is sent with only its base, and the server does not compare it with the stored workflow: unlike other kinds, an apply does not report that a workflow changed on the server since the get (\`status\`, which sends every folder whole, does, and the next get shows it). An edit of the folder is checked against its own \`.sim/base.json\` and refused as stale (exit ${SOURCE_STALE_EXIT_CODE}) when the workflow changed since. - A knowledge base's \`embeddingModel\` is fixed when it is created: an apply that changes it is refused. - A document or workspace file over 10 MB, or whose name no directory can hold, is left out of a get and listed in \`.sim/skipped.json\`; an apply never writes or deletes it. - A hidden file (a name starting with a dot) under a kind folder is never sent: an apply names it in a warning. @@ -147,7 +157,7 @@ Secrets never appear in source: a workflow reads an environment variable as \`en ${sourceExitCodesTable('sim workspaces source')} -Exit 5 right after an apply that ended without an answer (a timeout, a dropped connection) means that apply or someone else changed the workspace: get into a fresh directory and compare, and only when it holds your edits run \`sim workspaces source get --dir . --force\`. A get that exits 9 keeps the binary files it downloaded and fetches only the rest. +Exit ${SOURCE_STALE_EXIT_CODE} right after an apply that ended without an answer (a timeout, a dropped connection) means that apply or someone else changed the workspace: get into a fresh directory and compare, and only when it holds your edits run \`sim workspaces source get --dir . --force\`. A get that exits ${RATE_LIMITED_EXIT_CODE} keeps the binary files it downloaded and fetches only the rest. Keep \`${WORKSPACE_BASE_FILE}\` and each workflow's \`.sim/\` files as written. ` @@ -170,13 +180,13 @@ const ADVANCED_GUIDE = `# Sim workspace source: advanced - Commit the whole directory, \`.sim/\` included: \`${WORKSPACE_BASE_FILE}\`, each workflow's \`.sim/\` and \`.sim/written.json\` are what a later apply compares with, from any clone. \`.sim/cache/\` holds this machine's file stats and ignores itself. The generated \`.gitignore\` leaves out only what a command keeps while it runs (\`.sim/downloads/\`, \`.sim/command.lock/\`, \`.sim-staging-*\`) and \`${TSCONFIG_FULL_FILE}\`, which names the CLI's cache on this machine (\`check\` writes it again), and \`.gitattributes\` keeps workspace files and documents byte for byte. - A fresh clone runs \`check\` (it downloads the block catalog once when none is cached), \`apply --dry-run\` and \`apply --yes\` with only the committed files. A workspace API key may apply tables, rows, knowledge bases, files, custom tools, MCP servers and credential bindings; workflows, skills, secrets and deployments need a member's personal API key (deployments an admin's). -- Plan on every pull request (\`apply --dry-run\`), apply on merge, and commit the files the apply rewrote. Two branches that each applied: merge them, then run the dry run. No changes means the merged files match the workspace. Exit 5 means a resource changed since the base this branch holds: \`sim workspaces source get --dir \`, carry the edit over, and apply from there. +- Plan on every pull request (\`apply --dry-run\`), apply on merge, and commit the files the apply rewrote. Two branches that each applied: merge them, then run the dry run. No changes means the merged files match the workspace. Exit ${SOURCE_STALE_EXIT_CODE} means a resource changed since the base this branch holds: \`sim workspaces source get --dir \`, carry the edit over, and apply from there. - \`sim workspaces source get --dir .\` refreshes the workspace \`${WORKSPACE_BASE_FILE}\` names, whatever the profile's default; a \`--workspace\` naming another is refused. -- One command at a time per directory: a second \`get\`, \`apply\` or \`check\` on the same directory exits 10 while the first runs. +- One command at a time per directory: a second \`get\`, \`apply\` or \`check\` on the same directory exits ${SOURCE_BUSY_EXIT_CODE} while the first runs. ## Promote to another workspace -\`sim workspaces source apply --dir . --to --dry-run\`, then \`--yes\`: applies this directory to another workspace (dev to prod). Each resource binds by key to the target's own: one the target holds under the same key is updated in place (a workflow keeps its id and block ids, so its webhook URLs), a key it lacks is created (or adopts a resource of the same name). Each promote records what it left in the target in \`.sim/targets/.json\` (commit it), and the next promote compares with that: a resource changed in the target since is kept when this directory left it alone, and refused as stale (exit 5) when this directory changed it too. A resource with no record (the first promote, or one created in the target since) is updated to match this directory; delete the record to take the target as it stands again. Secrets and credentials bind by name and key to the target's own; no value is copied. The target's resources this directory does not declare are kept with a warning, or deleted with \`--allow-delete\`. Before the first promote, set the target's secrets (\`sim secrets set NAME --scope workspace -w \`) and connect its credentials, or the workflows using them are held back. This directory stays bound to the workspace it was read from. Without \`--to\`, \`check\`, \`status\`, \`diff\` and \`apply\` act on the workspace this directory was read from, whatever the profile's default; a \`--workspace\` naming another is refused (\`SOURCE_OTHER_WORKSPACE\`). +\`sim workspaces source apply --dir . --to --dry-run\`, then \`--yes\`: applies this directory to another workspace (dev to prod). Each resource binds by key to the target's own: one the target holds under the same key is updated in place (a workflow keeps its id and block ids, so its webhook URLs), a key it lacks is created (or adopts a resource of the same name). Each promote records what it left in the target in \`.sim/targets/.json\` (commit it), and the next promote compares with that: a resource changed in the target since is kept when this directory left it alone, and refused as stale (exit ${SOURCE_STALE_EXIT_CODE}) when this directory changed it too. A resource with no record (the first promote, or one created in the target since) is updated to match this directory; delete the record to take the target as it stands again. Secrets and credentials bind by name and key to the target's own; no value is copied. The target's resources this directory does not declare are kept with a warning, or deleted with \`--allow-delete\`. Before the first promote, set the target's secrets (\`sim secrets set NAME --scope workspace -w \`) and connect its credentials, or the workflows using them are held back. This directory stays bound to the workspace it was read from. Without \`--to\`, \`check\`, \`status\`, \`diff\` and \`apply\` act on the workspace this directory was read from, whatever the profile's default; a \`--workspace\` naming another is refused (\`SOURCE_OTHER_WORKSPACE\`). ` /** The workspace directory's tree beside the printed source. */ @@ -473,7 +483,7 @@ function declaredColumns(decl: TableDecl): ReadonlyMap { export function assertWorkspacePaths(paths: Iterable): void { for (const path of paths) { if (!isWorkspaceSourcePath(path)) { - throw new SimApiError(`The workspace source named an unsafe file path: ${path}`, 0) + throw new SourceDirError(`The workspace source named an unsafe file path: ${path}`) } } } @@ -521,9 +531,8 @@ export async function assertWorkspaceDir(dir: string): Promise { await assertSourceDirKind(dir, 'workspace') const { present } = await kindFolders(dir, APPLY_ORDER) if (present.length > 0) return - throw new SimApiError( - `${dir} holds no workspace source (no kind folder such as ${KIND_DIRS.workflow}/ or ${KIND_DIRS.table}/). Point --dir at a directory \`sim workspaces source get\` wrote, or write one with \`sim workspaces source get --dir ${dir}\`.`, - 0 + throw new SourceDirError( + `${dir} holds no workspace source (no kind folder such as ${KIND_DIRS.workflow}/ or ${KIND_DIRS.table}/). Point --dir at a directory \`sim workspaces source get\` wrote, or write one with \`sim workspaces source get --dir ${dir}\`.` ) } @@ -624,12 +633,9 @@ export async function readWorkspaceDir( ) const blobPaths = paths.filter(isBlobPath) if (blobPaths.length > maxBlobs) { - throw new SimApiError( + throw new SourceDirError( `The directory lists ${blobPaths.length} binary files (files/ and knowledge documents), over the ${maxBlobs}-file limit; nothing was sent.`, - 0, - 'SOURCE_TOO_LARGE', - undefined, - SOURCE_TOO_LARGE_EXIT_CODE + { code: 'SOURCE_TOO_LARGE', exitCode: SOURCE_TOO_LARGE_EXIT_CODE } ) } const textPaths = paths.filter((path) => !isBlobPath(path)) @@ -653,13 +659,10 @@ export async function readWorkspaceDir( const current = await stat(file) const { size } = current if (size > maxBlobBytes) { - throw new SimApiError( - `${path} is ${size} bytes, over the ${maxBlobBytes}-byte limit.`, - 0, - 'SOURCE_TOO_LARGE', - undefined, - SOURCE_TOO_LARGE_EXIT_CODE - ) + throw new SourceDirError(`${path} is ${size} bytes, over the ${maxBlobBytes}-byte limit.`, { + code: 'SOURCE_TOO_LARGE', + exitCode: SOURCE_TOO_LARGE_EXIT_CODE, + }) } const digested = await statDigest(file, path, current, written) hashed.set(path, digested) @@ -672,10 +675,9 @@ export async function readWorkspaceDir( try { base = JSON.parse(baseText) } catch { - throw new SimApiError( + throw new SourceDirError( `${WORKSPACE_BASE_FILE} in ${dir} is not valid JSON (a merge conflict?). Restore it from version control, or get the workspace into a fresh directory with \`sim workspaces source get --dir \` and carry your edits over.`, - 0, - 'SOURCE_BASE_INVALID' + { code: 'SOURCE_BASE_INVALID' } ) } } @@ -686,3 +688,28 @@ export async function readWorkspaceDir( export function skippedFile(skipped: readonly unknown[]): string | null { return skipped.length > 0 ? `${JSON.stringify(skipped, null, 2)}\n` : null } + +/** `--only`'s help: every kind by its folder. */ +export const ONLY_KINDS_HELP = `comma-separated folders: ${APPLY_ORDER.map((kind) => KIND_DIRS[kind]).join(', ')}` + +/** `--only tables,workflows` (folders) or `--only table,workflow` (kinds) as kinds. */ +export function parseKinds(value: string | undefined): WorkspaceKind[] | undefined { + if (value === undefined) return undefined + const names = value + .split(',') + .map((name) => name.trim()) + .filter((name) => name.length > 0) + if (names.length === 0) { + throw new CliUsageError( + `--only names no kind; name a folder (${[...KIND_BY_DIR.keys()].join(', ')}) or a kind (${APPLY_ORDER.join(', ')}).` + ) + } + const kinds = names.map((name) => (isWorkspaceKind(name) ? name : KIND_BY_DIR.get(name))) + const unknown = names.filter((_, index) => kinds[index] === undefined) + if (unknown.length > 0) { + throw new CliUsageError( + `Unknown kind ${unknown.join(', ')}; name a folder (${[...KIND_BY_DIR.keys()].join(', ')}) or a kind (${APPLY_ORDER.join(', ')}).` + ) + } + return [...new Set(kinds.filter((kind) => kind !== undefined))] +} diff --git a/packages/sim-cli/src/commands/protocol/workspace-source-get.ts b/packages/sim-cli/src/commands/protocol/workspace-source-get.ts index aefbe831b6c..554edda745a 100644 --- a/packages/sim-cli/src/commands/protocol/workspace-source-get.ts +++ b/packages/sim-cli/src/commands/protocol/workspace-source-get.ts @@ -2,19 +2,9 @@ import { mkdir, mkdtemp, realpath, rename, rm, rmdir } from 'node:fs/promises' import { join } from 'node:path' import { mapWithConcurrency } from '@sim/utils/concurrency' import { isRecordLike, toRecord } from '@sim/utils/object' -import { - APPLY_ORDER, - isWorkspaceKind, - KIND_DIRS, - parseWorkspaceBase, - type WorkspaceKind, -} from '@sim/workspace-code' +import { isWorkspaceKind, parseWorkspaceBase } from '@sim/workspace-code' import { isBlobPath } from '@sim/workspace-code/kinds/registry' -import { - KIND_BY_DIR, - WORKSPACE_BASE_FILE, - WORKSPACE_SKIPPED_FILE, -} from '@sim/workspace-code/layout' +import { WORKSPACE_BASE_FILE, WORKSPACE_SKIPPED_FILE } from '@sim/workspace-code/layout' import type { Command } from 'commander' import { printError } from '#sim-cli/output/io' import type { GetWorkspaceSourceResponse } from '../../generated/v2-api' @@ -22,6 +12,7 @@ import { SimApiError } from '../../http/client' import { apiCommand } from '../../runtime/called-operations' import { searchTasks } from '../search' import { removeStagingOnSignal } from './files-get' +import { SourceDirError } from './local-errors' import { absoluteDirOption, assertSourceDirKind, @@ -33,13 +24,13 @@ import { readSourceText, readWritten, recordedStats, - SOURCE_DIR_BUSY_HELP, SOURCE_DOWNLOADS_DIR, updateSourceDir, withSourceDirLock, writtenFiles, writtenRecord, } from './source-dir' +import { sourceExitCodesHelp } from './source-exit-codes' import { assertSdkMajor, finishTree, @@ -68,6 +59,8 @@ import { keepFile, kindFolders, localDeclarations, + ONLY_KINDS_HELP, + parseKinds, seedRowsFiles, skippedFile, WORKSPACE_TREE, @@ -75,45 +68,10 @@ import { type WorkspaceSource = GetWorkspaceSourceResponse['data'] -const GET_EXIT_CODES_HELP = ` +const GET_HELP = ` A directory that already holds a .sim/base.json is refreshed from the workspace it was read from, whatever the profile's default workspace. - -Exit codes: 0 written; 1 the command failed, the directory holds edits no apply has saved -(SOURCE_LOCAL_EDITS: apply them, get into another directory, or pass --force), --workspace names -another workspace than the one the directory was read from (SOURCE_OTHER_WORKSPACE), the directory -is a workflow source directory (SOURCE_DIR_KIND_MISMATCH, even with --force), or a downloaded -binary file does not match its listing (SOURCE_BLOB_MISMATCH: nothing was written); 7 the workspace -has more resources of a kind than one directory holds; 9 rate limited (HTTP 429) after the CLI's -retries: nothing was written, the binary files already downloaded are kept in .sim/downloads, and -running the get again downloads only the rest.` - -/** `--only`'s help: every kind by its folder. */ -export const ONLY_KINDS_HELP = `comma-separated folders: ${APPLY_ORDER.map((kind) => KIND_DIRS[kind]).join(', ')}` - -/** `--only tables,workflows` (folders) or `--only table,workflow` (kinds) as kinds. */ -export function parseKinds(value: string | undefined): WorkspaceKind[] | undefined { - if (value === undefined) return undefined - const names = value - .split(',') - .map((name) => name.trim()) - .filter((name) => name.length > 0) - if (names.length === 0) { - throw new SimApiError( - `--only names no kind; name a folder (${[...KIND_BY_DIR.keys()].join(', ')}) or a kind (${APPLY_ORDER.join(', ')}).`, - 0 - ) - } - const kinds = names.map((name) => (isWorkspaceKind(name) ? name : KIND_BY_DIR.get(name))) - const unknown = names.filter((_, index) => kinds[index] === undefined) - if (unknown.length > 0) { - throw new SimApiError( - `Unknown kind ${unknown.join(', ')}; name a folder (${[...KIND_BY_DIR.keys()].join(', ')}) or a kind (${APPLY_ORDER.join(', ')}).`, - 0 - ) - } - return [...new Set(kinds.filter((kind) => kind !== undefined))] -} +${sourceExitCodesHelp('workspaces source get')}` /** * The base a get writes: the answer's, plus the directory's entries for the kinds this get did not @@ -208,10 +166,9 @@ async function writeWorkspaceSource( ...editedBlobs(touchedBlobs, digests, written.digests), ] if (edited.length > 0) { - throw new SimApiError( + throw new SourceDirError( `${dir} has edits no apply has saved (${edited.join(', ')}). Apply them, get into another directory, or pass --force to discard them.`, - 0, - 'SOURCE_LOCAL_EDITS' + { code: 'SOURCE_LOCAL_EDITS' } ) } } @@ -387,7 +344,7 @@ export function attachWorkspaceSourceGet(source: Command): void { .option('--force', 'Replace files with edits no apply has saved') .allowExcessArguments(false) .description('Write a workspace as a source directory') - .addHelpText('after', `${GET_EXIT_CODES_HELP}\n${SOURCE_DIR_BUSY_HELP}`) + .addHelpText('after', GET_HELP) .action(async (options: { dir: string; only?: string; force?: boolean }) => { const kinds = parseKinds(options.only) const { client, profile } = connect() diff --git a/packages/sim-cli/src/commands/protocol/workspace-source-status.ts b/packages/sim-cli/src/commands/protocol/workspace-source-status.ts index 15c1875646c..37067e8a0e8 100644 --- a/packages/sim-cli/src/commands/protocol/workspace-source-status.ts +++ b/packages/sim-cli/src/commands/protocol/workspace-source-status.ts @@ -31,8 +31,8 @@ import type { PlanWorkspaceSourceBody, PlanWorkspaceSourceResponse, } from '../../generated/v2-api' -import { SimApiError } from '../../http/client' import { apiCommand, type Connection } from '../../runtime/called-operations' +import { SourceDirError } from './local-errors' import { absoluteDirOption, digest, @@ -49,6 +49,7 @@ import { baseUnavailable, DIFF_HELP, type DiffReport, + destructiveChanges, type FileDiff, loginCommand, type NextStep, @@ -67,14 +68,14 @@ import { } from './source-status' import { withSourceExitCode } from './source-tree' import type { Diagnostic } from './source-ui' -import { destructiveChanges } from './workspace-source-apply' import { assertWorkspaceDir, + ONLY_KINDS_HELP, + parseKinds, readWorkspaceDir, type WorkspaceSubmission, withUnchangedWorkflowsPruned, } from './workspace-source-dir' -import { ONLY_KINDS_HELP, parseKinds } from './workspace-source-get' type PlanData = PlanWorkspaceSourceResponse['data'] type ServerSource = GetWorkspaceSourceResponse['data'] @@ -135,10 +136,9 @@ async function assertWrittenReadable(dir: string): Promise { record = JSON.parse(text) } catch {} if (isRecordLike(record) && isRecordLike(record.files)) return - throw new SimApiError( + throw new SourceDirError( `${SOURCE_WRITTEN_FILE} in ${dir} is not a record the CLI wrote (a merge conflict?), so local changes cannot be told from what the last get or apply wrote. Restore it from version control, or run \`${PREFIX} get --dir ${dir}\`: it rewrites it, and refuses to replace a file that differs from the server.`, - 0, - 'SOURCE_WRITTEN_INVALID' + { code: 'SOURCE_WRITTEN_INVALID' } ) } diff --git a/packages/sim-cli/src/exit-codes.ts b/packages/sim-cli/src/exit-codes.ts index ff1a53caad9..2841dac4ccd 100644 --- a/packages/sim-cli/src/exit-codes.ts +++ b/packages/sim-cli/src/exit-codes.ts @@ -42,8 +42,8 @@ export const RATE_LIMITED_EXIT_CODE = 9 /** * Busy, retry. A `workspaces source apply` refused because another apply is writing the workspace - * (`409` with `SOURCE_BUSY`), or any `workspaces source` command refused because another one holds - * the same directory (`SOURCE_DIR_BUSY`): nothing of this one was read or written. Wait for the + * (`409` with `SOURCE_BUSY`), or a source command refused because another one holds the same + * directory (`SOURCE_DIR_BUSY`): nothing of this one was read or written. Wait for the * other to finish, then run it again; a write the other apply made to a resource this directory * changes then exits 5. */ diff --git a/packages/sim-cli/src/generated/v2-api.ts b/packages/sim-cli/src/generated/v2-api.ts index e0c4cb5b9dd..74a90c48863 100644 --- a/packages/sim-cli/src/generated/v2-api.ts +++ b/packages/sim-cli/src/generated/v2-api.ts @@ -826,7 +826,7 @@ export type ApplyWorkflowSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' + | 'I_FINDINGS_TRUNCATED' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' @@ -1227,7 +1227,6 @@ export type ApplyWorkspaceSourceResponse = { | 'W_PATH_NOT_SENT' | 'E_BLOB_MANIFEST' | 'E_BLOB_MISSING' - | 'E_LIMIT_EXCEEDED' | 'E_KEY_INVALID' | 'W_DECL_SUSPICIOUS_CHARACTER' | 'E_BASE_JSON' @@ -1290,7 +1289,6 @@ export type ApplyWorkspaceSourceResponse = { | 'W_DEPLOY_DRAFT_NOT_SOURCE' | 'E_DEPLOY_DRAFT_NOT_SOURCE_CHAT' | 'I_DEPLOY_DRAFT_NOT_LIVE' - | 'E_APPLY_STALE' | 'E_APPLY_NO_BASE' | 'E_APPLY_FORBIDDEN' | 'E_APPLY_DELETE_NOT_ALLOWED' @@ -1302,7 +1300,6 @@ export type ApplyWorkspaceSourceResponse = { | 'E_PLAN_REF_HELD' | 'W_RESOURCE_IN_USE' | 'W_TABLE_COLUMN_RENAMED_IN_USE' - | 'E_NAME_TAKEN' | 'E_WORKFLOW_FILES_MISSING' | 'E_WORKFLOW_FOLDER_COPIED' | 'W_UNDECLARED' @@ -1314,6 +1311,9 @@ export type ApplyWorkspaceSourceResponse = { | 'W_SERVER_MOVED' | 'I_KEY_REBOUND' | 'E_KEY_RENAMED' + | 'E_APPLY_STALE' + | 'E_LIMIT_EXCEEDED' + | 'E_NAME_TAKEN' | 'I_FINDINGS_TRUNCATED' | 'E_SYNTAX' | 'E_SDK_HEADER' @@ -1325,7 +1325,6 @@ export type ApplyWorkspaceSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' @@ -1519,7 +1518,6 @@ export type ApplyWorkspaceSourceResponse = { | 'W_PATH_NOT_SENT' | 'E_BLOB_MANIFEST' | 'E_BLOB_MISSING' - | 'E_LIMIT_EXCEEDED' | 'E_KEY_INVALID' | 'W_DECL_SUSPICIOUS_CHARACTER' | 'E_BASE_JSON' @@ -1582,7 +1580,6 @@ export type ApplyWorkspaceSourceResponse = { | 'W_DEPLOY_DRAFT_NOT_SOURCE' | 'E_DEPLOY_DRAFT_NOT_SOURCE_CHAT' | 'I_DEPLOY_DRAFT_NOT_LIVE' - | 'E_APPLY_STALE' | 'E_APPLY_NO_BASE' | 'E_APPLY_FORBIDDEN' | 'E_APPLY_DELETE_NOT_ALLOWED' @@ -1594,7 +1591,6 @@ export type ApplyWorkspaceSourceResponse = { | 'E_PLAN_REF_HELD' | 'W_RESOURCE_IN_USE' | 'W_TABLE_COLUMN_RENAMED_IN_USE' - | 'E_NAME_TAKEN' | 'E_WORKFLOW_FILES_MISSING' | 'E_WORKFLOW_FOLDER_COPIED' | 'W_UNDECLARED' @@ -1606,6 +1602,9 @@ export type ApplyWorkspaceSourceResponse = { | 'W_SERVER_MOVED' | 'I_KEY_REBOUND' | 'E_KEY_RENAMED' + | 'E_APPLY_STALE' + | 'E_LIMIT_EXCEEDED' + | 'E_NAME_TAKEN' | 'I_FINDINGS_TRUNCATED' | 'E_SYNTAX' | 'E_SDK_HEADER' @@ -1617,7 +1616,6 @@ export type ApplyWorkspaceSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' @@ -5288,7 +5286,7 @@ export type CreateWorkflowFromSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' + | 'I_FINDINGS_TRUNCATED' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' @@ -10391,7 +10389,7 @@ export type GetWorkflowSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' + | 'I_FINDINGS_TRUNCATED' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' @@ -10986,7 +10984,6 @@ export type GetWorkspaceSourceResponse = { | 'W_PATH_NOT_SENT' | 'E_BLOB_MANIFEST' | 'E_BLOB_MISSING' - | 'E_LIMIT_EXCEEDED' | 'E_KEY_INVALID' | 'W_DECL_SUSPICIOUS_CHARACTER' | 'E_BASE_JSON' @@ -11049,7 +11046,6 @@ export type GetWorkspaceSourceResponse = { | 'W_DEPLOY_DRAFT_NOT_SOURCE' | 'E_DEPLOY_DRAFT_NOT_SOURCE_CHAT' | 'I_DEPLOY_DRAFT_NOT_LIVE' - | 'E_APPLY_STALE' | 'E_APPLY_NO_BASE' | 'E_APPLY_FORBIDDEN' | 'E_APPLY_DELETE_NOT_ALLOWED' @@ -11061,7 +11057,6 @@ export type GetWorkspaceSourceResponse = { | 'E_PLAN_REF_HELD' | 'W_RESOURCE_IN_USE' | 'W_TABLE_COLUMN_RENAMED_IN_USE' - | 'E_NAME_TAKEN' | 'E_WORKFLOW_FILES_MISSING' | 'E_WORKFLOW_FOLDER_COPIED' | 'W_UNDECLARED' @@ -11073,6 +11068,9 @@ export type GetWorkspaceSourceResponse = { | 'W_SERVER_MOVED' | 'I_KEY_REBOUND' | 'E_KEY_RENAMED' + | 'E_APPLY_STALE' + | 'E_LIMIT_EXCEEDED' + | 'E_NAME_TAKEN' | 'I_FINDINGS_TRUNCATED' | 'E_SYNTAX' | 'E_SDK_HEADER' @@ -11084,7 +11082,6 @@ export type GetWorkspaceSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' @@ -14411,7 +14408,7 @@ export type PlanWorkflowSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' + | 'I_FINDINGS_TRUNCATED' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' @@ -14767,7 +14764,6 @@ export type PlanWorkspaceSourceResponse = { | 'W_PATH_NOT_SENT' | 'E_BLOB_MANIFEST' | 'E_BLOB_MISSING' - | 'E_LIMIT_EXCEEDED' | 'E_KEY_INVALID' | 'W_DECL_SUSPICIOUS_CHARACTER' | 'E_BASE_JSON' @@ -14830,7 +14826,6 @@ export type PlanWorkspaceSourceResponse = { | 'W_DEPLOY_DRAFT_NOT_SOURCE' | 'E_DEPLOY_DRAFT_NOT_SOURCE_CHAT' | 'I_DEPLOY_DRAFT_NOT_LIVE' - | 'E_APPLY_STALE' | 'E_APPLY_NO_BASE' | 'E_APPLY_FORBIDDEN' | 'E_APPLY_DELETE_NOT_ALLOWED' @@ -14842,7 +14837,6 @@ export type PlanWorkspaceSourceResponse = { | 'E_PLAN_REF_HELD' | 'W_RESOURCE_IN_USE' | 'W_TABLE_COLUMN_RENAMED_IN_USE' - | 'E_NAME_TAKEN' | 'E_WORKFLOW_FILES_MISSING' | 'E_WORKFLOW_FOLDER_COPIED' | 'W_UNDECLARED' @@ -14854,6 +14848,9 @@ export type PlanWorkspaceSourceResponse = { | 'W_SERVER_MOVED' | 'I_KEY_REBOUND' | 'E_KEY_RENAMED' + | 'E_APPLY_STALE' + | 'E_LIMIT_EXCEEDED' + | 'E_NAME_TAKEN' | 'I_FINDINGS_TRUNCATED' | 'E_SYNTAX' | 'E_SDK_HEADER' @@ -14865,7 +14862,6 @@ export type PlanWorkspaceSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' @@ -15059,7 +15055,6 @@ export type PlanWorkspaceSourceResponse = { | 'W_PATH_NOT_SENT' | 'E_BLOB_MANIFEST' | 'E_BLOB_MISSING' - | 'E_LIMIT_EXCEEDED' | 'E_KEY_INVALID' | 'W_DECL_SUSPICIOUS_CHARACTER' | 'E_BASE_JSON' @@ -15122,7 +15117,6 @@ export type PlanWorkspaceSourceResponse = { | 'W_DEPLOY_DRAFT_NOT_SOURCE' | 'E_DEPLOY_DRAFT_NOT_SOURCE_CHAT' | 'I_DEPLOY_DRAFT_NOT_LIVE' - | 'E_APPLY_STALE' | 'E_APPLY_NO_BASE' | 'E_APPLY_FORBIDDEN' | 'E_APPLY_DELETE_NOT_ALLOWED' @@ -15134,7 +15128,6 @@ export type PlanWorkspaceSourceResponse = { | 'E_PLAN_REF_HELD' | 'W_RESOURCE_IN_USE' | 'W_TABLE_COLUMN_RENAMED_IN_USE' - | 'E_NAME_TAKEN' | 'E_WORKFLOW_FILES_MISSING' | 'E_WORKFLOW_FOLDER_COPIED' | 'W_UNDECLARED' @@ -15146,6 +15139,9 @@ export type PlanWorkspaceSourceResponse = { | 'W_SERVER_MOVED' | 'I_KEY_REBOUND' | 'E_KEY_RENAMED' + | 'E_APPLY_STALE' + | 'E_LIMIT_EXCEEDED' + | 'E_NAME_TAKEN' | 'I_FINDINGS_TRUNCATED' | 'E_SYNTAX' | 'E_SDK_HEADER' @@ -15157,7 +15153,6 @@ export type PlanWorkspaceSourceResponse = { | 'E_NUMBER_NOT_FINITE' | 'W_NUMBER_PRECISION' | 'E_TOO_MANY_ERRORS' - | 'W_TOO_MANY_WARNINGS' | 'E_DUPLICATE_NAME' | 'E_RESERVED_NAME' | 'E_KEY_OVERRIDE' diff --git a/packages/table-cells/package.json b/packages/table-cells/package.json new file mode 100644 index 00000000000..8dafd9b5dbb --- /dev/null +++ b/packages/table-cells/package.json @@ -0,0 +1,60 @@ +{ + "name": "@sim/table-cells", + "version": "0.1.0", + "private": true, + "sideEffects": false, + "type": "module", + "license": "Apache-2.0", + "engines": { + "bun": ">=1.4.4", + "node": ">=20.0.0" + }, + "exports": { + "./currency": { + "types": "./src/currency.ts", + "default": "./src/currency.ts" + }, + "./dates": { + "types": "./src/dates.ts", + "default": "./src/dates.ts" + }, + "./rows": { + "types": "./src/rows.ts", + "default": "./src/rows.ts" + }, + "./rules": { + "types": "./src/rules.ts", + "default": "./src/rules.ts" + }, + "./select": { + "types": "./src/select.ts", + "default": "./src/select.ts" + }, + "./ttl": { + "types": "./src/ttl.ts", + "default": "./src/ttl.ts" + }, + "./types": { + "types": "./src/types.ts", + "default": "./src/types.ts" + } + }, + "scripts": { + "type-check": "tsc --noEmit", + "lint": "biome check --write --unsafe .", + "lint:check": "biome check .", + "format": "biome format --write .", + "format:check": "biome format .", + "test": "vitest run", + "test:watch": "vitest" + }, + "dependencies": { + "@sim/utils": "workspace:*", + "zod": "4.3.6" + }, + "devDependencies": { + "@sim/tsconfig": "workspace:*", + "typescript": "^7.0.2", + "vitest": "^5.1.0" + } +} diff --git a/packages/workspace-code/src/table-cells/currency.test.ts b/packages/table-cells/src/currency.test.ts similarity index 92% rename from packages/workspace-code/src/table-cells/currency.test.ts rename to packages/table-cells/src/currency.test.ts index e551039e377..b07816d9d16 100644 --- a/packages/workspace-code/src/table-cells/currency.test.ts +++ b/packages/table-cells/src/currency.test.ts @@ -4,7 +4,7 @@ * through unparsed would break every query against the column. */ import { describe, expect, it } from 'vitest' -import { formatCurrencyDisplay, formatCurrencyForInput, parseCurrencyInput } from './currency' +import { parseCurrencyInput } from './currency' describe('parseCurrencyInput', () => { it('strips symbols, ISO codes, and every flavor of space', () => { @@ -246,25 +246,4 @@ describe('parseCurrencyInput', () => { expect(parseCurrencyInput('1,234,567.89')).toBe(1234567.89) expect(parseCurrencyInput('12,34,567')).toBe(1234567) }) - - it('round-trips its own display output', () => { - for (const amount of [0, 12, -12.5, 1234.56, 1234567.89]) { - const rendered = formatCurrencyDisplay(amount, 'USD', 'en-US') - expect(parseCurrencyInput(rendered)).toBe(amount) - } - }) -}) - -describe('formatCurrencyDisplay', () => { - it('renders an unreadable value verbatim rather than blanking it', () => { - // What a string → currency conversion can leave behind. - expect(formatCurrencyDisplay('pending', 'USD', 'en-US')).toBe('pending') - expect(formatCurrencyDisplay(null, 'USD', 'en-US')).toBe('') - }) -}) - -describe('formatCurrencyForInput', () => { - it('keeps unparseable text so an edit does not silently erase it', () => { - expect(formatCurrencyForInput('pending')).toBe('pending') - }) }) diff --git a/packages/workspace-code/src/table-cells/currency.ts b/packages/table-cells/src/currency.ts similarity index 80% rename from packages/workspace-code/src/table-cells/currency.ts rename to packages/table-cells/src/currency.ts index af08390c841..395b62de603 100644 --- a/packages/workspace-code/src/table-cells/currency.ts +++ b/packages/table-cells/src/currency.ts @@ -133,24 +133,6 @@ const INTERIOR_EXPONENT = /\d\s*[eE][+-]?\s*\d/ /** ISO 4217 alphabetic code: exactly three letters. */ const CURRENCY_CODE_PATTERN = /^[A-Za-z]{3}$/ -/** - * Codes offered first in the picker. The rest of ICU's set is still selectable - * (and any valid code is accepted over the API) — these are just the ones worth - * reaching without typing. - */ -const PINNED_CURRENCY_CODES = [ - 'USD', - 'EUR', - 'GBP', - 'JPY', - 'CAD', - 'AUD', - 'CHF', - 'CNY', - 'INR', - 'BRL', -] as const - /** * Every ISO 4217 code the runtime knows, or `null` when the runtime predates * `Intl.supportedValuesOf` — in which case validation falls back to the shape @@ -182,43 +164,12 @@ export function resolveCurrencyCode(currencyCode: string | undefined): string { return currencyCode ? currencyCode.toUpperCase() : DEFAULT_CURRENCY_CODE } -export interface CurrencyOption { - code: string - /** Localized currency name, e.g. `US Dollar`. Falls back to the code. */ - name: string -} - /** - * Codes for the column-config picker: the pinned set first, then every other - * code the runtime supports, alphabetically. - * - * Built on first call, not at module load: constructing `Intl.DisplayNames` and - * naming ~160 currencies costs several milliseconds of ICU work, and the only - * caller is the column-config sidebar — every table API route imports this - * module and would otherwise pay for a list it never reads. + * Every ISO 4217 code this runtime can format, in the runtime's order; empty when the runtime + * cannot enumerate them. */ -let currencyOptions: readonly CurrencyOption[] | null = null - -function currencyDisplayNames(): Intl.DisplayNames | null { - try { - return new Intl.DisplayNames(['en'], { type: 'currency' }) - } catch { - return null - } -} - -export function getCurrencyOptions(): readonly CurrencyOption[] { - if (currencyOptions) return currencyOptions - const pinned = new Set(PINNED_CURRENCY_CODES) - const rest = supportedCurrencyCodes - ? [...supportedCurrencyCodes].filter((code) => !pinned.has(code)).sort() - : [] - const displayNames = currencyDisplayNames() - currencyOptions = [...PINNED_CURRENCY_CODES, ...rest].map((code) => ({ - code, - name: displayNames?.of(code) ?? code, - })) - return currencyOptions +export function listSupportedCurrencyCodes(): string[] { + return supportedCurrencyCodes ? [...supportedCurrencyCodes] : [] } /** @@ -391,66 +342,27 @@ function loneSeparatorIsGrouping( return separator === ',' ? fractionDigits !== 3 : fractionDigits === 0 && hadMarker } +/** + * Decimal places by upper-cased code. Only codes {@link isSupportedCurrencyCode} accepts are + * cached, which bounds the map by the runtime's currency set (or by three-letter shapes on a + * runtime that cannot enumerate it); an import parses thousands of cells against one code. + */ +const fractionDigitsCache = new Map() + /** A currency's conventional decimal places, defaulting to 2 when unknown. */ function currencyFractionDigits(currencyCode: string | undefined): number { if (!currencyCode) return 2 - const formatter = currencyFormatter(resolveCurrencyCode(currencyCode), undefined) - return formatter?.resolvedOptions().maximumFractionDigits ?? 2 -} - -/** - * Formatters are cached by locale + code: a grid paints thousands of currency - * cells per scroll, and constructing an `Intl.NumberFormat` per cell is orders - * of magnitude more expensive than the format call itself. - */ -const formatterCache = new Map() - -function currencyFormatter( - currencyCode: string, - locale: string | undefined -): Intl.NumberFormat | null { - const key = `${locale ?? ''}:${currencyCode}` - const cached = formatterCache.get(key) + const code = resolveCurrencyCode(currencyCode) + const cached = fractionDigitsCache.get(code) if (cached !== undefined) return cached - let formatter: Intl.NumberFormat | null + let fractionDigits: number try { - formatter = new Intl.NumberFormat(locale, { style: 'currency', currency: currencyCode }) + fractionDigits = + new Intl.NumberFormat(undefined, { style: 'currency', currency: code }).resolvedOptions() + .maximumFractionDigits ?? 2 } catch { - formatter = null + fractionDigits = 2 } - formatterCache.set(key, formatter) - return formatter -} - -/** - * Formats a stored cell for display — symbol placement and fraction digits come - * from the currency itself, so `JPY` renders `¥1,235` while `USD` renders - * `$1,234.56`. Values that carry no readable amount (a string left behind by a - * `string` → `currency` conversion, say) render verbatim rather than blanking, - * and an unformattable code degrades to `CODE amount`. - * - * `locale` is left to the caller's runtime by default, which means the viewer's - * own grouping/decimal conventions in the browser. - */ -export function formatCurrencyDisplay( - value: unknown, - currencyCode: string | undefined, - locale?: string -): string { - const amount = parseCurrencyInput(value) - if (amount === null) return typeof value === 'string' ? value : '' - const code = resolveCurrencyCode(currencyCode) - const formatter = currencyFormatter(code, locale) - return formatter ? formatter.format(amount) : `${code} ${amount}` -} - -/** - * Renders a stored cell for a text input: the bare amount, with no symbol or - * grouping, so editing round-trips through {@link parseCurrencyInput} exactly. - */ -export function formatCurrencyForInput(value: unknown): string { - if (value === null || value === undefined) return '' - const amount = parseCurrencyInput(value) - if (amount !== null) return String(amount) - return typeof value === 'string' ? value : '' + if (isSupportedCurrencyCode(code)) fractionDigitsCache.set(code, fractionDigits) + return fractionDigits } diff --git a/packages/workspace-code/src/table-cells/dates.test.ts b/packages/table-cells/src/dates.test.ts similarity index 100% rename from packages/workspace-code/src/table-cells/dates.test.ts rename to packages/table-cells/src/dates.test.ts diff --git a/packages/workspace-code/src/table-cells/dates.ts b/packages/table-cells/src/dates.ts similarity index 100% rename from packages/workspace-code/src/table-cells/dates.ts rename to packages/table-cells/src/dates.ts diff --git a/packages/workspace-code/src/table-cells/rows.ts b/packages/table-cells/src/rows.ts similarity index 100% rename from packages/workspace-code/src/table-cells/rows.ts rename to packages/table-cells/src/rows.ts diff --git a/packages/workspace-code/src/table-cells/rules.ts b/packages/table-cells/src/rules.ts similarity index 100% rename from packages/workspace-code/src/table-cells/rules.ts rename to packages/table-cells/src/rules.ts diff --git a/packages/workspace-code/src/table-cells/select.ts b/packages/table-cells/src/select.ts similarity index 100% rename from packages/workspace-code/src/table-cells/select.ts rename to packages/table-cells/src/select.ts diff --git a/packages/workspace-code/src/table-cells/ttl.ts b/packages/table-cells/src/ttl.ts similarity index 67% rename from packages/workspace-code/src/table-cells/ttl.ts rename to packages/table-cells/src/ttl.ts index 96896ab95f0..969642d7091 100644 --- a/packages/workspace-code/src/table-cells/ttl.ts +++ b/packages/table-cells/src/ttl.ts @@ -61,28 +61,3 @@ export function ttlInstantForComparison(value: unknown): string | null { export function isTtlTimestamp(value: unknown): value is string { return normalizeTtlTimestamp(value) !== null } - -/** Serializes picker fields in their existing offset. A day alone means midnight in that offset. */ -export function ttlValueFromPicker(day: string, time: string | null, offset = '-00:00'): string { - const seconds = time ? (time.length === 5 ? `${time}:00` : time) : '00:00:00' - return `${day}T${seconds}${offset}` -} - -/** Reads the stored clock and offset for the picker without converting the instant. */ -export function ttlValueToPickerParts(value: string): { - day: string | null - time: string | null - offset: string -} { - const normalized = normalizeTtlTimestamp(value) - return normalized - ? { day: normalized.slice(0, 10), time: normalized.slice(11, -6), offset: normalized.slice(-6) } - : { day: null, time: null, offset: '-00:00' } -} - -/** Today's calendar date in a fixed numeric offset, independent of profile timezone settings. */ -export function todayAtTtlOffset(offset: string): string { - const minutes = Number(offset.slice(1, 3)) * 60 + Number(offset.slice(4, 6)) - const signedMinutes = offset.startsWith('-') ? -minutes : minutes - return new Date(Date.now() + signedMinutes * 60_000).toISOString().slice(0, 10) -} diff --git a/packages/workspace-code/src/table-cells/types.ts b/packages/table-cells/src/types.ts similarity index 100% rename from packages/workspace-code/src/table-cells/types.ts rename to packages/table-cells/src/types.ts diff --git a/packages/table-cells/tsconfig.json b/packages/table-cells/tsconfig.json new file mode 100644 index 00000000000..dd8fe6e74c5 --- /dev/null +++ b/packages/table-cells/tsconfig.json @@ -0,0 +1,8 @@ +{ + "extends": "@sim/tsconfig/library.json", + "compilerOptions": { + "types": [] + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist"] +} diff --git a/packages/table-cells/vitest.config.ts b/packages/table-cells/vitest.config.ts new file mode 100644 index 00000000000..6f729af4c84 --- /dev/null +++ b/packages/table-cells/vitest.config.ts @@ -0,0 +1,11 @@ +import { defineConfig, mergeConfig } from 'vitest/config' +import sharedConfig from '../../vitest.shared' + +export default mergeConfig( + sharedConfig, + defineConfig({ + test: { + include: ['src/**/*.test.ts'], + }, + }) +) diff --git a/packages/workflow-code/src/scaling.test-helpers.ts b/packages/testing/src/helpers/scaling.ts similarity index 64% rename from packages/workflow-code/src/scaling.test-helpers.ts rename to packages/testing/src/helpers/scaling.ts index 6da5d3ede2a..6dec9128dd7 100644 --- a/packages/workflow-code/src/scaling.test-helpers.ts +++ b/packages/testing/src/helpers/scaling.ts @@ -1,8 +1,9 @@ /** - * Scaling checks for the engines. A case times the same step at `n` and `8n` and bounds the ratio: - * linear work grows about 8x (n log n about 10x), quadratic work 64x. A ratio, not a wall-clock budget, so a slow - * runner cannot fail it; the best of several runs absorbs a stray pause or GC. + * Scaling checks. A case times the same step at `n` and `8n` and bounds the ratio: linear work + * grows about 8x (n log n about 10x), quadratic work 64x. A ratio, not a wall-clock budget, so a + * slow runner cannot fail it; the best of several runs absorbs a stray pause or GC. */ +import { expect } from 'vitest' /** How many times larger the second input is than the first. */ const SCALE = 8 @@ -11,7 +12,7 @@ const SCALE = 8 * Linear growth is 8x; 20x leaves 2.5x room for allocation and GC noise on a slow runner and still * fails anything quadratic (64x) by a factor of three. */ -export const MAX_GROWTH = 20 +const MAX_GROWTH = 20 const RUNS = 5 /** Repeats `run` for at least `MIN_SAMPLE_MS`, so a millisecond clock still times a fast step. */ @@ -35,7 +36,7 @@ function perRunMs(run: () => void): number { * of the ratio instead of one; each side keeps its best run. A linear step stops as soon as its * ratio is under `MAX_GROWTH`, and a quadratic one never gets there. */ -export function growth(make: (n: number) => () => void, n: number): number { +function growth(make: (n: number) => () => void, n: number): number { make(n)() make(SCALE * n)() let small = Number.POSITIVE_INFINITY @@ -47,3 +48,11 @@ export function growth(make: (n: number) => () => void, n: number): number { } return large / small } + +/** + * Asserts the step `make(n)` returns grows linearly (or n log n) from `n` to `8n`, failing + * anything quadratic. `make` builds the input outside the timed step. + */ +export function expectLinearGrowth(make: (n: number) => () => void, n: number): void { + expect(growth(make, n)).toBeLessThan(MAX_GROWTH) +} diff --git a/packages/workflow-code/DESIGN.md b/packages/workflow-code/DESIGN.md index 1007c3d7a15..c4d1e892b43 100644 --- a/packages/workflow-code/DESIGN.md +++ b/packages/workflow-code/DESIGN.md @@ -446,7 +446,10 @@ start line, and clamps bad input (NaN to 1, ±Infinity to a bound). ## Apply `applyWorkflowSource` and `createWorkflowFromSource` (`apps/sim/lib/workflows/application/`) share -one pipeline. Create uses an empty graph as the stored draft, so every block is new and no +one pipeline (`source-pipeline/`): `planSource` loads the target and evaluates, then a dry run +previews the plan (`previewSource`) and a write commits it (`commitSource`). A workspace apply +plans and commits each workflow through the same steps (`workflowSourceUpdate`, +`workflowSourceCreate`), the commit refusing a plan made for another workflow or principal. Create uses an empty graph as the stored draft, so every block is new and no fingerprint is checked. Both are gated on the `workflow-code` flag (see Rollout) and change only the draft; deploy is separate. @@ -457,7 +460,7 @@ deploy is separate. with lines the row lock ``` -**Evaluate** (`source-evaluate.ts`, writes nothing): +**Evaluate** (`source-pipeline/evaluate/`, one file per phase, writes nothing): 1. parse the lock; canonicalize the stored draft under it (what `stored()` reads and the diff compares against); diff --git a/packages/workflow-code/package.json b/packages/workflow-code/package.json index 4fa05111dfb..f70af53c35c 100644 --- a/packages/workflow-code/package.json +++ b/packages/workflow-code/package.json @@ -14,9 +14,9 @@ "types": "./src/index.ts", "default": "./src/index.ts" }, - "./scaling.test-helpers": { - "types": "./src/scaling.test-helpers.ts", - "default": "./src/scaling.test-helpers.ts" + "./diagnostics": { + "types": "./src/diagnostics/index.ts", + "default": "./src/diagnostics/index.ts" }, "./diagnostics/codes": { "types": "./src/diagnostics/codes.ts", @@ -84,6 +84,7 @@ }, "devDependencies": { "@babel/types": "7.29.8", + "@sim/testing": "workspace:*", "@sim/tsconfig": "workspace:*", "@xyflow/react": "12.12.1", "fast-check": "3.23.2", diff --git a/packages/workflow-code/src/canonicalize/branches.ts b/packages/workflow-code/src/canonicalize/branches.ts index 7a1a0663df7..b6f1ba8302e 100644 --- a/packages/workflow-code/src/canonicalize/branches.ts +++ b/packages/workflow-code/src/canonicalize/branches.ts @@ -1,5 +1,6 @@ import { isRecordLike } from '@sim/utils/object' import { isIdentifierName } from '../code-files/ref-path' +import { diagnostic } from '../diagnostics/factory' import type { BlockKey, BranchKey, CanonicalBranches, CanonicalBranchRow, Json } from '../model' import { parseStoredList, setOwn } from '../records' import type { CanonContext } from './context' @@ -155,24 +156,30 @@ export function liftBranches( const storedRows = parseStoredList(stored) if (!storedRows) { if (typeof stored !== 'string' || stored.trim() !== '') { - context.diagnostics.push({ - code: 'W_CANON_UNDECODABLE_JSON', - severity: 'warning', - message: `Stored ${kind} branches are not a JSON list; kept verbatim`, - blockKey, - param, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_UNDECODABLE_JSON', + `Stored ${kind} branches are not a JSON list; kept verbatim`, + { + blockKey, + param, + } + ) + ) } return unlifted(kind, blockId, stored) } if (!storedRows.every(isStoredRow)) { - context.diagnostics.push({ - code: 'W_CANON_UNDECODABLE_JSON', - severity: 'warning', - message: `Stored ${kind} branches are not a list of { id, title, value } rows; kept verbatim`, - blockKey, - param, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_UNDECODABLE_JSON', + `Stored ${kind} branches are not a list of { id, title, value } rows; kept verbatim`, + { + blockKey, + param, + } + ) + ) return unlifted(kind, blockId, stored) } if (storedRows.length === 0) return withImplicit(kind, blockId, []) @@ -186,13 +193,16 @@ export function liftBranches( return false }) if (sharedId !== undefined) { - context.diagnostics.push({ - code: 'W_CANON_UNDECODABLE_JSON', - severity: 'warning', - message: `Stored ${kind} branches use row id "${sharedId}" twice; kept verbatim`, - blockKey, - param, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_UNDECODABLE_JSON', + `Stored ${kind} branches use row id "${sharedId}" twice; kept verbatim`, + { + blockKey, + param, + } + ) + ) return unlifted(kind, blockId, stored) } const keys: Array = new Array(count) @@ -233,13 +243,16 @@ export function liftBranches( } else if (stored.title !== positional) { row.titleOverride = stored.title if (kind === 'condition') { - context.diagnostics.push({ - code: 'W_CANON_TITLE_OVERRIDE', - severity: 'warning', - message: `Branch ${index + 1} is titled "${stored.title}" but shows as "${positional}"; the executor matches else by stored title`, - blockKey, - param, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_TITLE_OVERRIDE', + `Branch ${index + 1} is titled "${stored.title}" but shows as "${positional}"; the executor matches else by stored title`, + { + blockKey, + param, + } + ) + ) } } const extra: Record = {} diff --git a/packages/workflow-code/src/canonicalize/edges.ts b/packages/workflow-code/src/canonicalize/edges.ts index 7165ae5f139..02c9059f448 100644 --- a/packages/workflow-code/src/canonicalize/edges.ts +++ b/packages/workflow-code/src/canonicalize/edges.ts @@ -8,6 +8,7 @@ import { } from '@sim/workflow-types/workflow' import type { Edge } from '@xyflow/react' import { edgeTupleKey, portKey } from '../anchors' +import { diagnostic } from '../diagnostics/factory' import type { BranchKey, CanonicalEdge, Port } from '../model' import type { CanonContext } from './context' @@ -64,12 +65,15 @@ export function canonicalEdges( const from = context.keyById.get(edge.source) const to = context.keyById.get(edge.target) if (!source || !target || from === undefined || to === undefined) { - context.diagnostics.push({ - code: 'W_CANON_DANGLING_EDGE', - severity: 'warning', - message: `Edge ${edge.source} -> ${edge.target} references a missing block; dropped`, - ...(from !== undefined ? { blockKey: from } : {}), - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_DANGLING_EDGE', + `Edge ${edge.source} -> ${edge.target} references a missing block; dropped`, + { + ...(from !== undefined ? { blockKey: from } : {}), + } + ) + ) continue } @@ -80,21 +84,23 @@ export function canonicalEdges( seen.add(identity) if (port.kind === 'error') errorSourceIds.add(edge.source) if (port.kind === 'raw') { - context.diagnostics.push({ - code: 'W_CANON_RAW_HANDLE', - severity: 'warning', - message: `Edge from "${from}" leaves unknown handle "${port.handle}"; kept as a raw port`, - blockKey: from, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_RAW_HANDLE', + `Edge from "${from}" leaves unknown handle "${port.handle}"; kept as a raw port`, + { + blockKey: from, + } + ) + ) } const scopeProblem = getWorkflowEdgeScopeDropReason(edge, inputs.blocks) if (scopeProblem) { - context.diagnostics.push({ - code: 'E_CANON_EDGE_SCOPE', - severity: 'error', - message: `Edge ${from} -> ${to}: ${scopeProblem}`, - blockKey: from, - }) + context.diagnostics.push( + diagnostic('E_CANON_EDGE_SCOPE', `Edge ${from} -> ${to}: ${scopeProblem}`, { + blockKey: from, + }) + ) } result.push({ from, port, to }) diff --git a/packages/workflow-code/src/canonicalize/index.ts b/packages/workflow-code/src/canonicalize/index.ts index 26084024568..6a7dc2f3c6f 100644 --- a/packages/workflow-code/src/canonicalize/index.ts +++ b/packages/workflow-code/src/canonicalize/index.ts @@ -11,6 +11,7 @@ import { import type { BlockCatalog } from '../catalog/types' import { catalogBlock } from '../catalog/view' import type { CanonicalizeDiagnosticCode } from '../diagnostics/codes' +import { diagnostic } from '../diagnostics/factory' import type { Diagnostic } from '../diagnostics/frame' import { BLOCK_LAYOUT_DATA_FIELDS, layoutOf } from '../layout' import type { LockIdentity, LockIdentityBlock, LockLayout } from '../lock' @@ -153,12 +154,15 @@ export function canonicalize( const catalog = catalogBlock(context.catalog, block.type) const isContainer = isSubflowType(block.type) if (!catalog && !isContainer) { - context.diagnostics.push({ - code: 'W_CANON_UNKNOWN_BLOCK_TYPE', - severity: 'warning', - message: `Block type "${block.type}" is not in the catalog; params kept verbatim`, - blockKey: key, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_UNKNOWN_BLOCK_TYPE', + `Block type "${block.type}" is not in the catalog; params kept verbatim`, + { + blockKey: key, + } + ) + ) } const extraFields: Record = {} @@ -166,12 +170,15 @@ export function canonicalize( const parent = parentId !== undefined ? (keyById.get(parentId) ?? null) : null if (data.parentId != null && (parentId === undefined || parent === null)) { extraFields.parentId = toJson(data.parentId) - context.diagnostics.push({ - code: 'W_CANON_DANGLING_PARENT', - severity: 'warning', - message: `Block "${key}" names parent "${String(data.parentId)}", which is not a block; placed at the root`, - blockKey: key, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_DANGLING_PARENT', + `Block "${key}" names parent "${String(data.parentId)}", which is not a block; placed at the root`, + { + blockKey: key, + } + ) + ) } const overrides = parseModeMap(data.canonicalModes) diff --git a/packages/workflow-code/src/canonicalize/keys.ts b/packages/workflow-code/src/canonicalize/keys.ts index 55fb69fc814..d2b8af08258 100644 --- a/packages/workflow-code/src/canonicalize/keys.ts +++ b/packages/workflow-code/src/canonicalize/keys.ts @@ -5,6 +5,7 @@ import { type Variable, } from '@sim/workflow-types/workflow' import type { CanonicalizeDiagnosticCode } from '../diagnostics/codes' +import { diagnostic } from '../diagnostics/factory' import type { Diagnostic } from '../diagnostics/frame' import type { BlockKey, VarKey } from '../model' import type { VariableEntry } from './context' @@ -49,24 +50,30 @@ export function assignBlockKeys( if (winnerByName.get(normalized) === id) { keyById.set(id, normalized) if (isReservedWorkflowBlockName(normalized)) { - diagnostics.push({ - code: 'W_CANON_RESERVED_NAME', - severity: 'warning', - message: `Block name "${block.name}" normalizes to the reserved reference prefix "${normalized}"`, - blockKey: normalized, - }) + diagnostics.push( + diagnostic( + 'W_CANON_RESERVED_NAME', + `Block name "${block.name}" normalizes to the reserved reference prefix "${normalized}"`, + { + blockKey: normalized, + } + ) + ) } continue } const key = smallestFreeSuffix(normalized, taken) taken.add(key) keyById.set(id, key) - diagnostics.push({ - code: 'E_CANON_LEGACY_NAME_COLLISION', - severity: 'error', - message: `Block name "${block.name}" normalizes to "${normalized}", which "${blocks[winnerByName.get(normalized) ?? id].name}" already resolves to; keyed "${key}" until renamed`, - blockKey: key, - }) + diagnostics.push( + diagnostic( + 'E_CANON_LEGACY_NAME_COLLISION', + `Block name "${block.name}" normalizes to "${normalized}", which "${blocks[winnerByName.get(normalized) ?? id].name}" already resolves to; keyed "${key}" until renamed`, + { + blockKey: key, + } + ) + ) } return { keyById, referenceableKeys: new Set(winnerByName.keys()) } @@ -93,11 +100,12 @@ export function assignVariableKeys( if (claimed.has(normalizedName)) { key = smallestFreeSuffix(normalizedName, taken) taken.add(key) - diagnostics.push({ - code: 'E_CANON_LEGACY_NAME_COLLISION', - severity: 'error', - message: `Variable name "${variable.name}" normalizes to "${normalizedName}", which another variable already uses; keyed "${key}" until renamed`, - }) + diagnostics.push( + diagnostic( + 'E_CANON_LEGACY_NAME_COLLISION', + `Variable name "${variable.name}" normalizes to "${normalizedName}", which another variable already uses; keyed "${key}" until renamed` + ) + ) } claimed.add(normalizedName) entries.push({ id: variable.id, key, normalizedName }) diff --git a/packages/workflow-code/src/canonicalize/params.ts b/packages/workflow-code/src/canonicalize/params.ts index daf3fd3be78..990f9893d97 100644 --- a/packages/workflow-code/src/canonicalize/params.ts +++ b/packages/workflow-code/src/canonicalize/params.ts @@ -4,6 +4,7 @@ import type { CanonicalModeOverrides } from '../catalog/canonical-index' import { isConditionClause } from '../catalog/conditions' import type { CatalogBlock, CatalogCondition, CatalogSubBlock } from '../catalog/types' import { declarationsOf } from '../catalog/view' +import { diagnostic } from '../diagnostics/factory' import type { BlockKey, CanonicalParam, SubBlockId, Value } from '../model' import { parseJsonContainer, setOwn } from '../records' import { structuredCodec } from '../structured' @@ -76,13 +77,16 @@ function decodeStructured( if (typeof raw !== 'string' || raw.trim() === '') return raw const parsed = parseJsonContainer(raw) if (parsed) return parsed - context.diagnostics.push({ - code: 'W_CANON_UNDECODABLE_JSON', - severity: 'warning', - message: `Stored ${param} is not a JSON list or object; kept as text`, - blockKey, - param, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_UNDECODABLE_JSON', + `Stored ${param} is not a JSON list or object; kept as text`, + { + blockKey, + param, + } + ) + ) return raw } @@ -154,13 +158,16 @@ export function canonicalParams(inputs: ParamInputs, context: CanonContext): Par const declaration = surfaceDeclaration(catalog, id, triggerMode) if (typeof subBlock.id === 'string' && subBlock.id !== id) { - context.diagnostics.push({ - code: 'W_CANON_SUBBLOCK_TYPE_DRIFT', - severity: 'warning', - message: `Stored subBlock "${id}" carries id "${subBlock.id}"; the map key wins`, - blockKey, - param: id, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_SUBBLOCK_TYPE_DRIFT', + `Stored subBlock "${id}" carries id "${subBlock.id}"; the map key wins`, + { + blockKey, + param: id, + } + ) + ) } if ( catalog && @@ -168,13 +175,16 @@ export function canonicalParams(inputs: ParamInputs, context: CanonContext): Par declarationsOf(catalog, id).length > 0 && !declarationsOf(catalog, id).some((candidate) => candidate.type === subBlock.type) ) { - context.diagnostics.push({ - code: 'W_CANON_SUBBLOCK_TYPE_DRIFT', - severity: 'warning', - message: `Stored subBlock "${id}" has type "${subBlock.type}"; the catalog type "${declaration?.type}" wins`, - blockKey, - param: id, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_SUBBLOCK_TYPE_DRIFT', + `Stored subBlock "${id}" has type "${subBlock.type}"; the catalog type "${declaration?.type}" wins`, + { + blockKey, + param: id, + } + ) + ) } const decoded = shouldDecode(declaration) diff --git a/packages/workflow-code/src/canonicalize/subflows.ts b/packages/workflow-code/src/canonicalize/subflows.ts index 77f88100cb4..a181df6ca64 100644 --- a/packages/workflow-code/src/canonicalize/subflows.ts +++ b/packages/workflow-code/src/canonicalize/subflows.ts @@ -1,4 +1,5 @@ import type { BlockState, Loop, Parallel } from '@sim/workflow-types/workflow' +import { diagnostic } from '../diagnostics/factory' import { type BlockKey, isLoopType, @@ -78,12 +79,7 @@ function jsonField(value: unknown, context: CanonContext): Json | undefined { } function repaired(context: CanonContext, blockKey: BlockKey, message: string) { - context.diagnostics.push({ - code: 'W_CANON_SUBFLOW_REPAIRED', - severity: 'warning', - message, - blockKey, - }) + context.diagnostics.push(diagnostic('W_CANON_SUBFLOW_REPAIRED', message, { blockKey })) } /** diff --git a/packages/workflow-code/src/canonicalize/values.ts b/packages/workflow-code/src/canonicalize/values.ts index 5823857e263..06cfd27f754 100644 --- a/packages/workflow-code/src/canonicalize/values.ts +++ b/packages/workflow-code/src/canonicalize/values.ts @@ -7,6 +7,7 @@ import { isReservedWorkflowBlockName, normalizeWorkflowBlockName, } from '@sim/workflow-types/workflow' +import { diagnostic } from '../diagnostics/factory' import type { Json, ResourceKind, ResourceNode, UnresolvedNode, Value } from '../model' import { setOwn } from '../records' import type { CanonContext } from './context' @@ -110,13 +111,12 @@ export function liftResourceId( const key = context.resources?.toKey(kind, id) ?? null if (key !== null) return { $sim: 'resource', kind, key } if (context.resources) { - context.diagnostics.push({ - code: 'W_CANON_UNRESOLVED_ID', - severity: 'warning', - message: `No ${kind} key for stored id "${id}"; kept verbatim`, - blockKey: location.blockKey, - param: location.param, - }) + context.diagnostics.push( + diagnostic('W_CANON_UNRESOLVED_ID', `No ${kind} key for stored id "${id}"; kept verbatim`, { + blockKey: location.blockKey, + param: location.param, + }) + ) } return { $sim: 'unresolved', kind, id } } diff --git a/packages/workflow-code/src/catalog/scaling.test.ts b/packages/workflow-code/src/catalog/scaling.test.ts index 8211634beb4..9991cdad6c7 100644 --- a/packages/workflow-code/src/catalog/scaling.test.ts +++ b/packages/workflow-code/src/catalog/scaling.test.ts @@ -1,9 +1,10 @@ /** * Catalog lookups grow linearly with the block (see `growth` for how a case is measured). */ + +import { expectLinearGrowth } from '@sim/testing/helpers/scaling' import { describe, expect, it } from 'vitest' import { TEST_CATALOG } from '../compile/catalog.test-helpers' -import { growth, MAX_GROWTH } from '../scaling.test-helpers' import { buildFieldIndex } from './surface' import type { CatalogBlock, CatalogSubBlock } from './types' import { catalogBlock } from './view' @@ -32,7 +33,6 @@ function pairedBlock(pairs: number): CatalogBlock { describe('catalog scaling', () => { it('indexes the fields of a block with many basic/advanced pairs in linear time', () => { - const ratio = growth((n) => () => buildFieldIndex(pairedBlock(n), false, new Set()), 1_000) - expect(ratio).toBeLessThan(MAX_GROWTH) + expectLinearGrowth((n) => () => buildFieldIndex(pairedBlock(n), false, new Set()), 1_000) }) }) diff --git a/packages/workflow-code/src/compile/SYNTAX.md b/packages/workflow-code/src/compile/SYNTAX.md index 285331073d7..56ed953e179 100644 --- a/packages/workflow-code/src/compile/SYNTAX.md +++ b/packages/workflow-code/src/compile/SYNTAX.md @@ -385,7 +385,7 @@ Errors refuse the whole apply. Warnings are saved as written; info is informatio | `E_WORKFLOW_EXPORT` | Not exactly one `export default workflow(…)`. | | `E_LIMIT_EXCEEDED` | Over the block, edge, total-byte, depth or `codeFile()` reference cap (`SOURCE_LIMITS`). Apply answers 413. | | `E_TOO_MANY_ERRORS` | The compile stopped after 100 errors; fix those and compile again to see the rest. | -| `W_TOO_MANY_WARNINGS` | 100 warnings, or 20 of one code, were listed and the rest omitted, so a flood of one finding never hides another. A finding repeated at the same place is listed once. Infos are capped apart (100, the rest dropped), so they never displace a warning, and errors are never displaced by either. | +| `I_FINDINGS_TRUNCATED` | 100 warnings, or 20 of one code, were listed and the rest omitted, so a flood of one finding never hides another. A finding repeated at the same place is listed once. Infos are capped apart (100, the rest dropped), so they never displace a warning, and errors are never displaced by either. An answer that merges findings from several stages is capped the same way as a whole; when that cuts it, it ends in one `I_FINDINGS_TRUNCATED` counting what it left out, in place of the stages' own. | | `E_DUPLICATE_NAME` | Two blocks normalize to one key. | | `E_RESERVED_NAME` | A block name normalizes to `loop`, `parallel` or `variable`. | | `E_KEY_OVERRIDE` | A `meta.key` that is not a legacy key the stored workflow already has. | diff --git a/packages/workflow-code/src/compile/grammar.test.ts b/packages/workflow-code/src/compile/grammar.test.ts index fcc21519559..10fa4003e98 100644 --- a/packages/workflow-code/src/compile/grammar.test.ts +++ b/packages/workflow-code/src/compile/grammar.test.ts @@ -1993,7 +1993,7 @@ describe('reference-shaped text that floods the warnings', () => { const later = "const later = api('Later', { url: t`${a0.missing}` })" const result = run(source([START, ...flood, later].join('\n'))) expect(result.diagnostics.filter((d) => d.code === 'W_UNKNOWN_OUTPUT_PATH')).toHaveLength(1) - expect(result.diagnostics.filter((d) => d.code === 'W_TOO_MANY_WARNINGS')).toHaveLength(1) + expect(result.diagnostics.filter((d) => d.code === 'I_FINDINGS_TRUNCATED')).toHaveLength(1) }) }) diff --git a/packages/workflow-code/src/compile/hostile.test.ts b/packages/workflow-code/src/compile/hostile.test.ts index 12ba5f9ab6d..80ed340b098 100644 --- a/packages/workflow-code/src/compile/hostile.test.ts +++ b/packages/workflow-code/src/compile/hostile.test.ts @@ -1,12 +1,13 @@ +import { expectLinearGrowth } from '@sim/testing/helpers/scaling' import { afterEach, describe, expect, it } from 'vitest' import { codeFileReferences } from '../code-files/references' +import { FINDING_LIMITS } from '../diagnostics/bound' import { MAX_ERRORS } from '../diagnostics/codes' -import { growth, MAX_GROWTH } from '../scaling.test-helpers' import { TEST_CATALOG } from './catalog.test-helpers' import type { CompileResult } from './index' import { compileWorkflow } from './index' import { type CompileLimits, sourceDepthOverflow } from './parse' -import { MAX_INFOS, MAX_WARNINGS, MAX_WARNINGS_PER_CODE, Reporter } from './report' +import { Reporter } from './report' /** * Hostile sources: compile never executes anything and never throws. Each input must come @@ -158,7 +159,7 @@ describe('hostile sources produce diagnostics only', () => { const text = source(repeats) return () => run(text) } - expect(growth(scan, 12_500)).toBeLessThan(MAX_GROWTH) + expectLinearGrowth(scan, 12_500) const result = run(source(100_000)) expect(result.workflow).not.toBeNull() expect(codes(result)).not.toContain('W_LITERAL_SECRET') @@ -195,16 +196,42 @@ describe('hostile sources produce diagnostics only', () => { 'W_COMMENT_DROPPED', 'W_FIELD_NOT_APPLICABLE', ] as const - expect(fillers.length * MAX_WARNINGS_PER_CODE).toBeGreaterThanOrEqual(MAX_WARNINGS) - for (let i = 0; i < MAX_WARNINGS; i++) { - reporter.report(fillers[Math.floor(i / MAX_WARNINGS_PER_CODE)], `x${i}`, at) + expect(fillers.length * FINDING_LIMITS.warningsPerCode).toBeGreaterThanOrEqual( + FINDING_LIMITS.perSeverity + ) + for (let i = 0; i < FINDING_LIMITS.perSeverity; i++) { + reporter.report(fillers[Math.floor(i / FINDING_LIMITS.warningsPerCode)], `x${i}`, at) } reporter.report('W_LITERAL_SECRET', 'body holds a literal secret', at) const capped = reporter.diagnostics.at(-1) - expect(capped?.code).toBe('W_TOO_MANY_WARNINGS') + expect(capped?.code).toBe('I_FINDINGS_TRUNCATED') expect(capped?.frame).toBeUndefined() }) + it('records one I_FINDINGS_TRUNCATED however many warnings come past the total cap', () => { + const reporter = new Reporter() + reporter.addFile('w.ts', 'x') + const at = { file: 'w.ts', start: 0, end: 1 } + // Spread over enough codes that none reaches its own cap before the total cap. + const spread = [ + 'W_SUBFLOW_COUNT', + 'W_FORWARD_REF', + 'W_VALUE_KIND', + 'W_OUT_OF_RANGE', + 'W_UNKNOWN_OPTION', + 'W_REQUIRED_MISSING', + 'W_ENV_UNKNOWN', + 'W_REF_NOT_NORMALIZED', + ] as const + expect(FINDING_LIMITS.perSeverity / spread.length).toBeLessThan(FINDING_LIMITS.warningsPerCode) + for (let i = 0; i < FINDING_LIMITS.perSeverity + 50; i++) { + reporter.report(spread[i % spread.length], `x${i}`, at) + } + + const markers = reporter.diagnostics.filter((d) => d.code === 'I_FINDINGS_TRUNCATED') + expect(markers).toHaveLength(1) + }) + it('checks the precision of a 100 KB numeric literal in linear time', () => { const source = (zeros: number) => wrap(`const a = api('A', { url: 'x', timeout: 0.1${'0'.repeat(zeros)}1 })`) @@ -212,7 +239,7 @@ describe('hostile sources produce diagnostics only', () => { const text = source(zeros) return () => run(text) } - expect(growth(check, 12_500)).toBeLessThan(MAX_GROWTH) + expectLinearGrowth(check, 12_500) expect(codes(run(source(100_000)))).toContain('W_NUMBER_PRECISION') }) @@ -231,8 +258,8 @@ describe('hostile sources produce diagnostics only', () => { body += `const a150 = api('A150', { url: 'x' })` const result = run(wrap(body)) expect(result.workflow).not.toBeNull() - expect(result.diagnostics).toHaveLength(MAX_WARNINGS_PER_CODE + 1) - expect(result.diagnostics.at(-1)?.code).toBe('W_TOO_MANY_WARNINGS') + expect(result.diagnostics).toHaveLength(FINDING_LIMITS.warningsPerCode + 1) + expect(result.diagnostics.at(-1)?.code).toBe('I_FINDINGS_TRUNCATED') }) it('never computes the did-you-mean of a warning past the cap', () => { @@ -252,7 +279,7 @@ describe('hostile sources produce diagnostics only', () => { } ) } - expect(computed).toBe(MAX_WARNINGS_PER_CODE) + expect(computed).toBe(FINDING_LIMITS.warningsPerCode) expect(reporter.diagnostics[0]?.hint).toBe('Did you mean y?') }) @@ -274,22 +301,25 @@ describe('hostile sources produce diagnostics only', () => { } ) } - expect(reporter.withheld).toHaveLength(1_000 - MAX_WARNINGS_PER_CODE) - expect(computed).toBe(MAX_WARNINGS_PER_CODE) + expect(reporter.withheld).toHaveLength(1_000 - FINDING_LIMITS.warningsPerCode) + expect(computed).toBe(FINDING_LIMITS.warningsPerCode) expect({ ...reporter.withheld[0] }.hint).toBe('Did you mean y?') - expect(computed).toBe(MAX_WARNINGS_PER_CODE + 1) + expect(computed).toBe(FINDING_LIMITS.warningsPerCode + 1) }) it('keeps infos from crowding out a warning', () => { const reporter = new Reporter() reporter.addFile('w.ts', 'x') const at = { file: 'w.ts', start: 0, end: 1 } - for (let i = 0; i < MAX_WARNINGS + 50; i++) reporter.report('I_SCHEDULE_UTC', `x${i}`, at) + for (let i = 0; i < FINDING_LIMITS.perSeverity + 50; i++) + reporter.report('I_SCHEDULE_UTC', `x${i}`, at) reporter.report('W_LITERAL_SECRET', 'apiKey holds a literal secret', at) const listed = reporter.diagnostics.map((d) => d.code) expect(listed).toContain('W_LITERAL_SECRET') - expect(listed).not.toContain('W_TOO_MANY_WARNINGS') - expect(listed.filter((code) => code === 'I_SCHEDULE_UTC')).toHaveLength(MAX_INFOS) + expect(listed).not.toContain('I_FINDINGS_TRUNCATED') + expect(listed.filter((code) => code === 'I_SCHEDULE_UTC')).toHaveLength( + FINDING_LIMITS.perSeverity + ) }) it('caps a diagnostic message that quotes a huge name', () => { diff --git a/packages/workflow-code/src/compile/report.ts b/packages/workflow-code/src/compile/report.ts index ea645adf345..30e33e930ad 100644 --- a/packages/workflow-code/src/compile/report.ts +++ b/packages/workflow-code/src/compile/report.ts @@ -1,4 +1,5 @@ import { truncateAtCodePoint } from '@sim/utils/string' +import { FINDING_LIMITS } from '../diagnostics/bound' import { type CompileDiagnosticCode, framesSource, MAX_ERRORS } from '../diagnostics/codes' import { codeFrame, @@ -8,6 +9,7 @@ import { splitLines, } from '../diagnostics/frame' import { REFUSABLE_WARNING_CODES } from '../diagnostics/refusals' +import { severityOf } from '../diagnostics/severity' import type { BlockKey, SubBlockId } from '../model' /** Babel's line terminators, so lines agree with Babel `loc`. */ @@ -54,7 +56,7 @@ export interface Range { interface ReportExtras { /** * A function is called only when the diagnostic is recorded, so a costly did-you-mean is never - * computed for a warning past {@link MAX_WARNINGS}; a withheld warning computes it when its + * computed for a warning past {@link MAX_LISTED}; a withheld warning computes it when its * `hint` is first read. */ hint?: string | (() => string | undefined) @@ -63,25 +65,18 @@ interface ReportExtras { related?: Array<{ at: Range; message: string }> } -type SeverityOfCode = LocatedDiagnostic['severity'] - -function severityOf(code: CompileDiagnosticCode): SeverityOfCode { - if (code.startsWith('E_')) return 'error' - return code.startsWith('I_') ? 'info' : 'warning' -} - -/** Warnings one compile lists before the rest are omitted with `W_TOO_MANY_WARNINGS`. */ -export const MAX_WARNINGS = 100 /** - * Warnings of one code one compile lists before the rest of that code are omitted with - * `W_TOO_MANY_WARNINGS`, so a flood of one finding leaves room under {@link MAX_WARNINGS} for the - * others. + * How many warnings and infos one compile builds, and how long their text runs: the defaults of + * the bound every answer applies (`FINDING_LIMITS`). These stop work, not only trim a list: a + * warning past them computes no did-you-mean unless read, and no code frame unless it is a + * refusable one kept back (one per code, block and file); a message never holds more of the + * source it quotes than an answer could show. */ -export const MAX_WARNINGS_PER_CODE = 20 -/** Infos one compile lists before the rest are dropped; counted apart so none displaces a warning. */ -export const MAX_INFOS = 100 -/** Code units of a message or hint kept; longer text, which quotes the source, is cut. */ -const MAX_MESSAGE_LENGTH = 1_000 +const { + perSeverity: MAX_LISTED, + warningsPerCode: MAX_WARNINGS_PER_CODE, + textLength, +} = FINDING_LIMITS /** * Thrown by {@link Reporter.report} once {@link MAX_ERRORS} errors are listed, after it records @@ -96,14 +91,14 @@ export class ErrorLimitReached extends Error { } function capLength(text: string): string { - return truncateAtCodePoint(text, MAX_MESSAGE_LENGTH, '…') + return truncateAtCodePoint(text, textLength, '…') } /** * Collects located diagnostics for every file a compile reads. Each diagnostic gets its span (from - * offsets) and code frame here, so no other module computes line numbers. Output is bounded: at - * most {@link MAX_ERRORS} errors, {@link MAX_WARNINGS} warnings and {@link MAX_INFOS} infos, each with a bounded frame and - * message. + * offsets) and code frame here, so no other module computes line numbers. The work is bounded: at + * most {@link MAX_ERRORS} errors, and {@link MAX_LISTED} warnings and as many infos, each with a + * bounded frame and message. */ export class Reporter { readonly diagnostics: LocatedDiagnostic[] = [] @@ -150,10 +145,11 @@ export class Reporter { /** * Records a diagnostic, once: a finding with the same code, message and range as one already * recorded is dropped. The error after the {@link MAX_ERRORS}th is recorded as - * `E_TOO_MANY_ERRORS` and throws {@link ErrorLimitReached}; the warning after the - * {@link MAX_WARNINGS}th, or after the {@link MAX_WARNINGS_PER_CODE}th of its code, is recorded as - * `W_TOO_MANY_WARNINGS` and later ones are dropped. Each severity has its own cap, so a warning - * never crowds out an error, nor an info a warning. + * `E_TOO_MANY_ERRORS` and throws {@link ErrorLimitReached}; where the warning after the + * {@link MAX_LISTED}th, or after the {@link MAX_WARNINGS_PER_CODE}th of its code, was found one + * `I_FINDINGS_TRUNCATED` is recorded (taking that warning's place in the count) and later ones + * are dropped. Each severity has its own cap, so a warning never crowds out an error, nor an + * info a warning. */ report(code: CompileDiagnosticCode, message: string, at: Range, extras: ReportExtras = {}): void { const key = `${code}\u0000${at.file}\u0000${at.start}\u0000${at.end}\u0000${message}` @@ -172,15 +168,13 @@ export class Reporter { throw new ErrorLimitReached() } } else if (severity === 'info') { - if (this.infoCount >= MAX_INFOS) return - } else if (this.warningCount >= MAX_WARNINGS) { + if (this.infoCount >= MAX_LISTED) return + } else if (this.warningCount >= MAX_LISTED) { this.withhold(code, message, at, extras) - if (this.warningCount === MAX_WARNINGS) { - this.record( - 'W_TOO_MANY_WARNINGS', - `Listed ${MAX_WARNINGS} warnings; the rest are omitted. The next: ${message}`, + if (this.warningCount === MAX_LISTED) { + this.truncated( + `Listed ${MAX_LISTED} warnings; the rest are omitted. The next: ${message}`, at, - {}, framesSource(code) ) } @@ -191,11 +185,9 @@ export class Reporter { if (ofCode >= MAX_WARNINGS_PER_CODE) { this.withhold(code, message, at, extras) if (ofCode === MAX_WARNINGS_PER_CODE) { - this.record( - 'W_TOO_MANY_WARNINGS', + this.truncated( `Listed ${MAX_WARNINGS_PER_CODE} ${code} warnings; the rest of them are omitted. The next: ${message}`, at, - {}, framesSource(code) ) } @@ -239,6 +231,16 @@ export class Reporter { this.withheld.push(diagnostic) } + /** + * Records the `I_FINDINGS_TRUNCATED` that stands where a warning cap left a warning out. It + * takes that warning's place in the warning count, so the caps list as many findings as when + * the marker was a warning. + */ + private truncated(message: string, at: Range, framed: boolean): void { + this.warningCount++ + this.diagnostics.push(this.build('I_FINDINGS_TRUNCATED', message, at, {}, framed)) + } + /** Records one diagnostic; `framed` is false when the finding at `at` is about a literal secret. */ private record( code: CompileDiagnosticCode, diff --git a/packages/workflow-code/src/compile/scaling.test.ts b/packages/workflow-code/src/compile/scaling.test.ts index 3784e65b8bb..409ec37c78f 100644 --- a/packages/workflow-code/src/compile/scaling.test.ts +++ b/packages/workflow-code/src/compile/scaling.test.ts @@ -1,17 +1,17 @@ /** * Compile steps grow linearly with the workflow (see `growth` for how a case is measured). */ + +import { expectLinearGrowth } from '@sim/testing/helpers/scaling' import { describe, expect, it } from 'vitest' -import { growth, MAX_GROWTH } from '../scaling.test-helpers' import { stronglyConnected } from './edges' describe('compile scaling', () => { it('finds the cycles of a block wired to many targets in linear time', () => { - const ratio = growth((n) => { + expectLinearGrowth((n) => { const targets = Array.from({ length: n }, (_, i) => `t${i}`) const keys = ['hub', ...targets] return () => stronglyConnected(keys, (key) => (key === 'hub' ? [...targets] : [])) }, 5_000) - expect(ratio).toBeLessThan(MAX_GROWTH) }) }) diff --git a/packages/workflow-code/src/diagnostics/bound.ts b/packages/workflow-code/src/diagnostics/bound.ts new file mode 100644 index 00000000000..be3f53d706c --- /dev/null +++ b/packages/workflow-code/src/diagnostics/bound.ts @@ -0,0 +1,156 @@ +/** + * How many findings one answer lists, and how long their text runs: bounded once, where findings + * leave the process (a route's answer, a CLI command's output), never per stage, so a flood from + * any stage (compile, canonicalize, print, materialize, lint) is capped the same way and a list + * merged from several stages is capped as a whole. A cut list ends in one `I_FINDINGS_TRUNCATED`. + * + * A limit that stops work stays where the work is (the compile's `E_TOO_MANY_ERRORS`, its warning + * caps that spare hint and frame work, a declaration reader's per-file stop); this module only + * trims lists that already exist. + */ +import { truncateAtCodePoint } from '@sim/utils/string' +import { diagnostic } from './factory' +import { type RefusalClass, refusalClass } from './refusals' +import type { DiagnosticSeverity } from './severity' + +/** The limits {@link boundAnswerFindings} applies. */ +export interface FindingLimits { + /** Findings of one severity. */ + perSeverity: number + /** Warnings of one code, so a flood of one finding leaves room for the others. */ + warningsPerCode: number + /** Findings in all, kept errors first, then warnings, then infos. */ + total: number + /** Code units of a message or hint; longer text, which can quote the source, is cut. */ + textLength: number +} + +/** The default limits: what one compile lists. */ +export const FINDING_LIMITS: Readonly = { + perSeverity: 100, + warningsPerCode: 20, + total: Number.POSITIVE_INFINITY, + textLength: 1_000, +} + +/** The fields {@link boundAnswerFindings} reads. */ +interface BoundedFinding { + code: string + severity: DiagnosticSeverity + message: string + hint?: string +} + +/** What a bound left out of a list, by severity, beside how long the list was. */ +interface DroppedFindings { + /** Findings in the list, the `I_FINDINGS_TRUNCATED` it already carried aside. */ + total: number + error: number + warning: number + info: number + /** `I_FINDINGS_TRUNCATED` the list carried, where a stage stopped listing its own findings. */ + stopped: number +} + +/** How one answer's findings are bounded. */ +export interface AnswerBound { + /** Limits beside the defaults ({@link FINDING_LIMITS}). */ + limits?: Partial + /** + * The vocabulary's refusal classifier: errors it classes (stale, locked, too large) are kept + * ahead of the other errors when errors are cut, so a cut list still shows why the refusal is + * classed as it is. The workflow vocabulary's by default. + */ + classify?: (codes: Iterable) => RefusalClass +} + +const SEVERITY_RANK: Readonly> = { + error: 0, + warning: 1, + info: 2, +} + +/** `finding` with its message and hint cut to `textLength` code units; itself when both fit. */ +export function boundFindingText( + finding: D, + textLength: number = FINDING_LIMITS.textLength +): D { + const { message, hint } = finding + if (message.length <= textLength && (hint === undefined || hint.length <= textLength)) { + return finding + } + return { + ...finding, + message: truncateAtCodePoint(message, textLength, '…'), + ...(hint !== undefined ? { hint: truncateAtCodePoint(hint, textLength, '…') } : {}), + } +} + +/** The `I_FINDINGS_TRUNCATED` that ends a list one answer cut, counting what it left out. */ +function findingsTruncated(dropped: DroppedFindings) { + const left = dropped.error + dropped.warning + dropped.info + const stopped = + dropped.stopped > 0 ? ' Some checks also stopped listing their own findings early.' : '' + return diagnostic( + 'I_FINDINGS_TRUNCATED', + `There are ${dropped.total} findings; ${left} more${dropped.error > 0 ? ` (${dropped.error} of them errors)` : ''} are left out.${stopped}`, + { hint: 'Fix the findings shown, then check again to see the rest.' } + ) +} + +/** The `I_FINDINGS_TRUNCATED` a cut list ends in. */ +type TruncationFinding = ReturnType + +/** + * `list` as one answer lists it: within the limits, in its own order, each finding's text cut + * (see {@link boundFindingText}). Which findings stay: per severity the first ones, errors before + * warnings before infos when `total` cuts (and among errors those `classify` classes before the + * rest), and at most `warningsPerCode` warnings of one code. A list it cuts ends in one + * `I_FINDINGS_TRUNCATED` counting what it left out, in place of any the list carried (where a + * stage stopped listing), so a cut list never holds a second; a list it does not cut keeps those + * as they are, one per stage that stopped. The marker is an info, so cutting a list never refuses it nor changes its class. + */ +export function boundAnswerFindings( + list: readonly D[], + bound: AnswerBound = {} +): Array { + const limits = { ...FINDING_LIMITS, ...bound.limits } + const classify = bound.classify ?? refusalClass + const rank = (finding: D) => + SEVERITY_RANK[finding.severity] * 2 + + (finding.severity === 'error' && classify([finding.code]) === 'validation' ? 1 : 0) + const order = list + .map((finding, index) => ({ finding, index })) + .filter(({ finding }) => finding.code !== 'I_FINDINGS_TRUNCATED') + .sort((a, b) => rank(a.finding) - rank(b.finding) || a.index - b.index) + + const kept = new Set() + const bySeverity: Record = { error: 0, warning: 0, info: 0 } + const warningsByCode = new Map() + const dropped: DroppedFindings = { + total: order.length, + error: 0, + warning: 0, + info: 0, + stopped: list.length - order.length, + } + for (const { finding, index } of order) { + const ofCode = finding.severity === 'warning' ? (warningsByCode.get(finding.code) ?? 0) : 0 + if ( + kept.size >= limits.total || + bySeverity[finding.severity] >= limits.perSeverity || + ofCode >= limits.warningsPerCode + ) { + dropped[finding.severity]++ + continue + } + kept.add(index) + bySeverity[finding.severity]++ + if (finding.severity === 'warning') warningsByCode.set(finding.code, ofCode + 1) + } + + const bounded = (finding: D) => boundFindingText(finding, limits.textLength) + if (kept.size === order.length) return list.map(bounded) + const listed = list.flatMap((finding, index) => (kept.has(index) ? [bounded(finding)] : [])) + return [...listed, boundFindingText(findingsTruncated(dropped), limits.textLength)] +} diff --git a/packages/workflow-code/src/diagnostics/codes.ts b/packages/workflow-code/src/diagnostics/codes.ts index a80c749a0da..de3caaa0826 100644 --- a/packages/workflow-code/src/diagnostics/codes.ts +++ b/packages/workflow-code/src/diagnostics/codes.ts @@ -2,13 +2,18 @@ * Every diagnostic code `@sim/workflow-code` and the source apply path can emit. CLI output, the * copilot and tests match on them, so they freeze at release. * - * The first letter fixes the severity (`E_` error, `W_` warning, `I_` info; see - * {@link SeverityOf}). The segment after it names the stage that owns the code: + * The first letter fixes the severity (`E_` error, `W_` warning, `I_` info; see `SeverityOf` in + * `./severity`). The segment after it usually names the stage that owns the code: * - no stage segment: compile and print, which share one vocabulary; * - `LOCK_`: `parseWorkflowLock`; + * - `SOURCE_META_`: `parseSourceMeta`; * - `CANON_`: canonicalize; * - `MAT_`: materialize and identity resolution; - * - `APPLY_`: the server apply pipeline and its gates. + * - `APPLY_`: the server apply pipeline and its gates. That array also holds the apply's checks + * whose codes name what they check instead (`W_TABLE_UNKNOWN_COLUMN`, `E_NAME_TAKEN`, + * `E_WORKFLOW_META_INVALID`). + * + * Every code is declared in exactly one array; {@link DiagnosticCode} is `never` if one is not. */ /** Syntax, header and file-shape errors from compile. */ @@ -41,13 +46,17 @@ const COMPILE_SYNTAX_CODES = [ * Fixing the errors listed and compiling again shows the rest. */ 'E_TOO_MANY_ERRORS', - /** - * `MAX_WARNINGS` warnings, or `MAX_WARNINGS_PER_CODE` of one code, were listed; this one sits - * where the next was found, and the rest (of that code) are omitted. - */ - 'W_TOO_MANY_WARNINGS', ] as const +/** + * The one truncation marker, in every vocabulary: a list of findings was cut, and this info ends + * it. The compile records it where its warning caps (`FINDING_LIMITS`) leave the next warning (of + * that code) out; an answer's list it cuts (`boundAnswerFindings`) ends in one that counts what it + * left out, in place of the compile's. + * Informational: the findings it follows decide whether a write is refused. + */ +const TRUNCATION_CODES = ['I_FINDINGS_TRUNCATED'] as const + /** Names, identifiers and scope. */ const COMPILE_SCOPE_CODES = [ /** Two block names normalize to the same key. */ @@ -287,6 +296,7 @@ const REFERENCE_CODES = [ const COMPILE_DIAGNOSTIC_CODES = [ ...COMPILE_SYNTAX_CODES, + ...TRUNCATION_CODES, ...COMPILE_SCOPE_CODES, ...COMPILE_EDGE_CODES, ...COMPILE_CATALOG_CODES, @@ -299,16 +309,13 @@ export const MAX_ERRORS = 100 export type CompileDiagnosticCode = (typeof COMPILE_DIAGNOSTIC_CODES)[number] /** - * Print emits only reference, comment and code-file fallbacks and literal secrets; every print - * compiles. + * Print emits only compile's reference, comment and code-file fallbacks and literal secrets, so + * that whatever it prints compiles back with the same findings. */ -const PRINT_DIAGNOSTIC_CODES = [ - ...REFERENCE_CODES, - 'W_CODE_FILE_INLINED', - 'W_RAW_HANDLE', - 'W_LITERAL_SECRET', -] as const -export type PrintDiagnosticCode = (typeof PRINT_DIAGNOSTIC_CODES)[number] +export type PrintDiagnosticCode = Extract< + CompileDiagnosticCode, + (typeof REFERENCE_CODES)[number] | 'W_CODE_FILE_INLINED' | 'W_RAW_HANDLE' | 'W_LITERAL_SECRET' +> /** `parseWorkflowLock` (lock.ts). */ const LOCK_DIAGNOSTIC_CODES = [ @@ -532,18 +539,34 @@ export type ApplyDiagnosticCode = (typeof APPLY_DIAGNOSTIC_CODES)[number] /** * Every code, as a value: the wire contracts' enum, so a client matching on a code is checked - * against the vocabulary. Print's reference codes are compile's too, so a few appear twice. + * against the vocabulary. */ export const DIAGNOSTIC_CODES = [ ...COMPILE_DIAGNOSTIC_CODES, - ...PRINT_DIAGNOSTIC_CODES, ...LOCK_DIAGNOSTIC_CODES, ...SOURCE_META_DIAGNOSTIC_CODES, ...CANONICALIZE_DIAGNOSTIC_CODES, ...MATERIALIZE_DIAGNOSTIC_CODES, ...APPLY_DIAGNOSTIC_CODES, ] as const -export type DiagnosticCode = (typeof DIAGNOSTIC_CODES)[number] + +/** Whether a tuple of codes lists one of them twice. */ +export type HasDuplicates = T extends readonly [ + infer Head, + ...infer Tail, +] + ? Head extends Tail[number] + ? true + : HasDuplicates + : false + +/** + * One of {@link DIAGNOSTIC_CODES}. It is `never`, which fails every use, if a code is declared in + * two arrays: each code has one owner and one place in the wire enum. + */ +export type DiagnosticCode = HasDuplicates extends false + ? (typeof DIAGNOSTIC_CODES)[number] + : never /** * Whether a finding with `code` may carry a code frame. A finding about a literal secret in @@ -553,12 +576,3 @@ export type DiagnosticCode = (typeof DIAGNOSTIC_CODES)[number] export function framesSource(code: DiagnosticCode): boolean { return code !== 'W_LITERAL_SECRET' } - -export type DiagnosticSeverity = 'error' | 'warning' | 'info' - -/** The severity a code's prefix fixes. */ -export type SeverityOf = C extends `E_${string}` - ? 'error' - : C extends `W_${string}` - ? 'warning' - : 'info' diff --git a/packages/workflow-code/src/diagnostics/factory.ts b/packages/workflow-code/src/diagnostics/factory.ts new file mode 100644 index 00000000000..831340fff27 --- /dev/null +++ b/packages/workflow-code/src/diagnostics/factory.ts @@ -0,0 +1,15 @@ +import { type SeverityOf, severityOf } from './severity' + +/** + * A finding of `code` with its severity filled in from the code's prefix, so no caller writes a + * severity by hand. `extras` (a hint, a span, a block key, …) follow the message. Nothing is + * truncated here: lists and text are bounded once, where findings leave the process + * (`boundAnswerFindings`). + */ +export function diagnostic>( + code: C, + message: string, + extras?: E +): { code: C; severity: SeverityOf; message: string } & E { + return { code, severity: severityOf(code), message, ...(extras as E) } +} diff --git a/packages/workflow-code/src/diagnostics/frame.ts b/packages/workflow-code/src/diagnostics/frame.ts index 3d08e4ebeae..24e3cc8e196 100644 --- a/packages/workflow-code/src/diagnostics/frame.ts +++ b/packages/workflow-code/src/diagnostics/frame.ts @@ -1,5 +1,6 @@ import type { BlockKey, SubBlockId } from '../model' -import type { DiagnosticCode, SeverityOf } from './codes' +import type { DiagnosticCode } from './codes' +import type { SeverityOf } from './severity' /** * A source range. Lines and columns are 1-based and columns count UTF-16 code units, which is diff --git a/packages/workflow-code/src/diagnostics/index.ts b/packages/workflow-code/src/diagnostics/index.ts new file mode 100644 index 00000000000..148943b79d8 --- /dev/null +++ b/packages/workflow-code/src/diagnostics/index.ts @@ -0,0 +1,24 @@ +export { type AnswerBound, boundAnswerFindings, boundFindingText } from './bound' +export { + type ApplyDiagnosticCode, + type DiagnosticCode, + framesSource, + type HasDuplicates, +} from './codes' +export { diagnostic } from './factory' +export { + codeFrame, + type Diagnostic, + type DiagnosticFields, + type LocatedDiagnostic, + type SourceSpan, + splitLines, +} from './frame' +export { + EDIT_REFUSALS, + type RefusalClass, + raisedMessage, + refusalClassifier, + refuseEditedFindings, +} from './refusals' +export type { DiagnosticSeverity, SeverityOf } from './severity' diff --git a/packages/workflow-code/src/diagnostics/refusals.test.ts b/packages/workflow-code/src/diagnostics/refusals.test.ts index 5e1dabd5007..e5f0d2f95bb 100644 --- a/packages/workflow-code/src/diagnostics/refusals.test.ts +++ b/packages/workflow-code/src/diagnostics/refusals.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from 'vitest' import { TEST_CATALOG } from '../compile/catalog.test-helpers' import { compileWorkflow } from '../compile/index' -import { MAX_WARNINGS_PER_CODE } from '../compile/report' +import { FINDING_LIMITS } from './bound' import { MAX_ERRORS } from './codes' import { type EditedBlocks, refuseEditedFindings } from './refusals' @@ -33,7 +33,7 @@ const START = "const start = trigger.start('Start')" describe('refuseEditedFindings', () => { it('refuses an edited block’s warning that the per-code cap left out of the listing', () => { const flood = Array.from( - { length: MAX_WARNINGS_PER_CODE }, + { length: FINDING_LIMITS.warningsPerCode }, (_, i) => `q${i} = $ref.start.q${i}` ) const compiled = compile( diff --git a/packages/workflow-code/src/diagnostics/refusals.ts b/packages/workflow-code/src/diagnostics/refusals.ts index 45de916b849..90c6256d488 100644 --- a/packages/workflow-code/src/diagnostics/refusals.ts +++ b/packages/workflow-code/src/diagnostics/refusals.ts @@ -1,14 +1,65 @@ /** + * Refusals: which findings refuse a write, and how a refused write is classed. + * * Which compile warnings an apply refuses on the blocks an edit wrote. A warning stays a warning * on what the stored workflow already holds, so a workflow saved with one still applies * unchanged; on a block the edit added or changed it fails at run time, so the apply refuses it * under an `E_APPLY_*` code. The server's source evaluation and the CLI's offline check both call * {@link refuseEditedFindings} with what each knows the edit changed, so they cannot disagree. + * + * How a refusal is classed ({@link refusalClass}): its HTTP status on the server and its exit + * code in the CLI, the same for workflow and workspace source. */ import { SOURCE_ENTRY_FILE } from '../limits' import { type CompileDiagnosticCode, MAX_ERRORS } from './codes' +import { diagnostic } from './factory' import type { LocatedDiagnostic } from './frame' +/** How a refused write is classed: its HTTP status on the server, its exit code in the CLI. */ +export type RefusalClass = 'conflict' | 'locked' | 'payload_too_large' | 'validation' + +/** The error codes of each class but `validation`, which is every other error. */ +export interface RefusalCodes { + /** What the write was based on changed since it was read (`409`). */ + conflict: readonly string[] + /** The target is locked against the change (`423`). */ + locked: readonly string[] + /** A size, count or nesting limit (`413`). */ + payload_too_large: readonly string[] +} + +/** The workflow vocabulary's classed codes, which every vocabulary shares. */ +const REFUSAL_CODES: RefusalCodes = { + conflict: ['E_APPLY_STALE'], + locked: [], + payload_too_large: ['E_LIMIT_EXCEEDED'], +} + +/** + * The classifier of a vocabulary that adds `extra` classed codes to the workflow vocabulary's: the + * class of a refusal with these error codes, the most specific first (a conflict, then a lock, + * then a size limit), else `validation`. + */ +export function refusalClassifier( + extra: Partial = {} +): (codes: Iterable) => RefusalClass { + const classed = (refusal: keyof RefusalCodes) => + new Set([...REFUSAL_CODES[refusal], ...(extra[refusal] ?? [])]) + const order = (['conflict', 'locked', 'payload_too_large'] as const).map( + (refusal) => [refusal, classed(refusal)] as const + ) + return (codes) => { + const present = new Set(codes) + for (const [refusal, members] of order) { + for (const code of members) if (present.has(code)) return refusal + } + return 'validation' + } +} + +/** The class of a workflow source refusal with these error codes (see {@link refusalClassifier}). */ +export const refusalClass = refusalClassifier() + /** The error codes an edited block's warning is refused under. */ export type EditRefusalCode = | 'E_APPLY_UNRESOLVED_REF' @@ -131,7 +182,7 @@ export function refuseEditedFindings( const code = refusalOf(finding, edit) return code === undefined ? undefined - : { ...finding, code, severity: 'error', message: raisedMessage(finding.message) } + : { ...finding, ...diagnostic(code, raisedMessage(finding.message)) } } const listed: Array = compiled.diagnostics.map( (finding) => refused(finding) ?? finding @@ -145,14 +196,15 @@ export function refuseEditedFindings( if (extra.length <= budget) return [...listed, ...raised] const [first] = extra.slice(budget) const omitted = extra.length - budget - const summary: LocatedDiagnostic<'E_TOO_MANY_ERRORS'> = { - code: 'E_TOO_MANY_ERRORS', - severity: 'error', - message: `${omitted} more edited ${omitted === 1 ? 'finding is' : 'findings are'} refused, as the ones listed are. The next: ${raisedMessage(first.message)}`, - hint: 'Fix the errors listed and check again to see the rest.', - span: first.span, - ...(first.frame === undefined ? {} : { frame: first.frame }), - ...(first.blockKey === undefined ? {} : { blockKey: first.blockKey }), - } + const summary: LocatedDiagnostic<'E_TOO_MANY_ERRORS'> = diagnostic( + 'E_TOO_MANY_ERRORS', + `${omitted} more edited ${omitted === 1 ? 'finding is' : 'findings are'} refused, as the ones listed are. The next: ${raisedMessage(first.message)}`, + { + hint: 'Fix the errors listed and check again to see the rest.', + span: first.span, + ...(first.frame === undefined ? {} : { frame: first.frame }), + ...(first.blockKey === undefined ? {} : { blockKey: first.blockKey }), + } + ) return [...listed, ...raised, summary] } diff --git a/packages/workflow-code/src/diagnostics/severity.ts b/packages/workflow-code/src/diagnostics/severity.ts new file mode 100644 index 00000000000..b08bad63048 --- /dev/null +++ b/packages/workflow-code/src/diagnostics/severity.ts @@ -0,0 +1,20 @@ +/** + * How a finding weighs, in every vocabulary (workflow and workspace source): an error refuses, a + * warning and an info only report. A code's first letter fixes it (`E_`, `W_`, `I_`), so a + * finding never carries a severity its code disagrees with. + */ +export type DiagnosticSeverity = 'error' | 'warning' | 'info' + +/** The severity a code's prefix fixes. */ +export type SeverityOf = C extends `E_${string}` + ? 'error' + : C extends `W_${string}` + ? 'warning' + : 'info' + +/** The severity of any code, by its prefix. */ +export function severityOf(code: C): SeverityOf { + return ( + code.startsWith('E_') ? 'error' : code.startsWith('W_') ? 'warning' : 'info' + ) as SeverityOf +} diff --git a/packages/workflow-code/src/index.ts b/packages/workflow-code/src/index.ts index b9bd87b7774..275d1a751bc 100644 --- a/packages/workflow-code/src/index.ts +++ b/packages/workflow-code/src/index.ts @@ -48,23 +48,18 @@ export { export { LineIndex } from './compile/report' export { type ApplyDiagnosticCode, - type DiagnosticCode, - type DiagnosticSeverity, - framesSource, -} from './diagnostics/codes' -export { codeFrame, type Diagnostic, + type DiagnosticCode, type DiagnosticFields, - type LocatedDiagnostic, - type SourceSpan, - splitLines, -} from './diagnostics/frame' -export { EDIT_REFUSALS, + framesSource, + type LocatedDiagnostic, raisedMessage, refuseEditedFindings, -} from './diagnostics/refusals' + type SourceSpan, + splitLines, +} from './diagnostics' export { type CanonicalDiff, diffCanonical, renamesBetween, replacedBetween } from './diff' export { canonicalDigests, @@ -115,6 +110,7 @@ export type { ResourceKind, Value, } from './model' +export { isResourceKind } from './model' export { assignIdentifiers, printWorkflow } from './print' export { hasSuspiciousCharacter, printKey, quoteString } from './print/strings' export { getOwn, invertRecord, parseJsonContainer, setOrDelete, setOwn } from './records' diff --git a/packages/workflow-code/src/lock.ts b/packages/workflow-code/src/lock.ts index 72db73437ee..52b7d0ba015 100644 --- a/packages/workflow-code/src/lock.ts +++ b/packages/workflow-code/src/lock.ts @@ -20,6 +20,7 @@ import { isRecordLike } from '@sim/utils/object' import { normalizeWorkflowBlockName } from '@sim/workflow-types/workflow' import { isIdentifierName } from './code-files/ref-path' import type { LockDiagnosticCode } from './diagnostics/codes' +import { diagnostic } from './diagnostics/factory' import type { Diagnostic } from './diagnostics/frame' import type { BlockKey, BranchKey, Json, SubBlockId, VarKey } from './model' import { SDK_RESERVED_IDENTIFIERS } from './sdk' @@ -260,7 +261,7 @@ class LockReader { readonly diagnostics: LockDiagnostic[] = [] report(code: LockDiagnostic['code'], segments: readonly string[], message: string): void { - this.diagnostics.push({ code, severity: 'error', message, path: pointer(segments) }) + this.diagnostics.push(diagnostic(code, message, { path: pointer(segments) })) } /** diff --git a/packages/workflow-code/src/materialize/identity.ts b/packages/workflow-code/src/materialize/identity.ts index 0feb02e8e76..b8a9528b49f 100644 --- a/packages/workflow-code/src/materialize/identity.ts +++ b/packages/workflow-code/src/materialize/identity.ts @@ -5,6 +5,7 @@ import type { BlockCatalog } from '../catalog/types' import { catalogBlock } from '../catalog/view' import { parseStoredPath } from '../code-files/ref-path' import type { MaterializeDiagnosticCode } from '../diagnostics/codes' +import { diagnostic } from '../diagnostics/factory' import type { Diagnostic } from '../diagnostics/frame' import { pointer, type WorkflowLockBlock, type WorkflowLockV1 } from '../lock' import type { BlockKey, CanonicalBlock, CanonicalWorkflow } from '../model' @@ -108,12 +109,15 @@ function duplicateLockKeys( for (const [id, keys] of keysById) { if (keys.length < 2) continue for (const key of keys) duplicates.add(key) - diagnostics.push({ - code: 'E_MAT_DUPLICATE_LOCK_ID', - severity: 'error', - message: `Lock keys ${keys.map((key) => `"${key}"`).join(', ')} all claim block id "${id}"`, - path: pointer(['blocks', keys[1], 'id']), - }) + diagnostics.push( + diagnostic( + 'E_MAT_DUPLICATE_LOCK_ID', + `Lock keys ${keys.map((key) => `"${key}"`).join(', ')} all claim block id "${id}"`, + { + path: pointer(['blocks', keys[1], 'id']), + } + ) + ) } return duplicates } @@ -163,12 +167,15 @@ export function resolveIdentity(inputs: IdentityInputs): IdentityResolution { if (!stored || prevKey === undefined || claimed.has(prevId)) return false const relation = typeRelation(catalog, stored.type, target.type) if (relation === 'different') { - diagnostics.push({ - code: 'W_MAT_RETYPE_RECREATED', - severity: 'warning', - message: `Block "${key}" changed type from "${stored.type}" to "${target.type}"; it is deleted and re-created, so its webhook URL and schedules are lost`, - blockKey: key, - }) + diagnostics.push( + diagnostic( + 'W_MAT_RETYPE_RECREATED', + `Block "${key}" changed type from "${stored.type}" to "${target.type}"; it is deleted and re-created, so its webhook URL and schedules are lost`, + { + blockKey: key, + } + ) + ) settled.set(key, { retypedPrevId: prevId }) return true } @@ -267,12 +274,15 @@ export function resolveIdentity(inputs: IdentityInputs): IdentityResolution { !Object.hasOwn(prevBlocks, lockId) && !resolvedIds.has(lockId) if (lockId !== undefined && !reusable) { - diagnostics.push({ - code: 'W_MAT_LOCK_ID_CLAIMED', - severity: 'warning', - message: `Lock id "${lockId}" for block "${key}" already belongs to another block; a new id was minted`, - blockKey: key, - }) + diagnostics.push( + diagnostic( + 'W_MAT_LOCK_ID_CLAIMED', + `Lock id "${lockId}" for block "${key}" already belongs to another block; a new id was minted`, + { + blockKey: key, + } + ) + ) } const id = reusable ? lockId : hooks.newId() resolve(key, { @@ -284,12 +294,15 @@ export function resolveIdentity(inputs: IdentityInputs): IdentityResolution { } for (const [key, lockId] of deletedSincePull) { if (resolved.get(key)?.entry.status !== 'created') continue - diagnostics.push({ - code: 'W_MAT_DELETED_SINCE_PULL', - severity: 'warning', - message: `Block "${key}" (${lockId}) was deleted after the lock was pulled; it is re-created`, - blockKey: key, - }) + diagnostics.push( + diagnostic( + 'W_MAT_DELETED_SINCE_PULL', + `Block "${key}" (${lockId}) was deleted after the lock was pulled; it is re-created`, + { + blockKey: key, + } + ) + ) } for (const [key, block] of resolved) { @@ -496,12 +509,15 @@ function inferRenames(state: InferenceState): void { const pair = (key: BlockKey, prevKey: BlockKey, basis: InferenceBasis): boolean => { if (!state.match(key, prevIdentity[prevKey].id, 'inferred')) return false - state.diagnostics.push({ - code: 'W_MAT_INFERRED_RENAME', - severity: 'warning', - message: `Block "${key}" is taken as a rename of "${prevKey}" (${INFERENCE_REASON[basis]}), so it keeps that block's id and run history. To make it a new block instead, delete "${prevKey}" in one apply and add "${key}" in the next`, - blockKey: key, - }) + state.diagnostics.push( + diagnostic( + 'W_MAT_INFERRED_RENAME', + `Block "${key}" is taken as a rename of "${prevKey}" (${INFERENCE_REASON[basis]}), so it keeps that block's id and run history. To make it a new block instead, delete "${prevKey}" in one apply and add "${key}" in the next`, + { + blockKey: key, + } + ) + ) return true } diff --git a/packages/workflow-code/src/materialize/index.ts b/packages/workflow-code/src/materialize/index.ts index d06ab4b50e2..8f1d8884a6a 100644 --- a/packages/workflow-code/src/materialize/index.ts +++ b/packages/workflow-code/src/materialize/index.ts @@ -11,6 +11,7 @@ import { type CanonicalizeResult, canonicalize } from '../canonicalize' import type { BlockCatalog } from '../catalog/types' import { catalogBlock } from '../catalog/view' import type { MaterializeDiagnosticCode } from '../diagnostics/codes' +import { diagnostic } from '../diagnostics/factory' import type { Diagnostic } from '../diagnostics/frame' import type { LockIdentity, LockIdentityBlock, WorkflowLockBlock, WorkflowLockV1 } from '../lock' import type { @@ -308,12 +309,15 @@ export function materialize( built.set(key, result) if (result.needsLayout) { needsLayout.push(key) - context.diagnostics.push({ - code: 'I_MAT_NEEDS_LAYOUT', - severity: 'info', - message: `Block "${key}" has no stored or lock position; it is placed by autolayout`, - blockKey: key, - }) + context.diagnostics.push( + diagnostic( + 'I_MAT_NEEDS_LAYOUT', + `Block "${key}" has no stored or lock position; it is placed by autolayout`, + { + blockKey: key, + } + ) + ) } } diff --git a/packages/workflow-code/src/materialize/values.ts b/packages/workflow-code/src/materialize/values.ts index 26f840f7833..93b581a5090 100644 --- a/packages/workflow-code/src/materialize/values.ts +++ b/packages/workflow-code/src/materialize/values.ts @@ -1,6 +1,7 @@ import { isRecordLike } from '@sim/utils/object' import { escapeRegExp } from '@sim/utils/string' import { closest } from '../compile/hints' +import { diagnostic } from '../diagnostics/factory' import { type BlockKey, type CanonicalBlock, @@ -116,26 +117,28 @@ function unliftNode(node: SimNode, context: MaterializeContext, location: ValueL const id = context.variableIdByKey.get(node.key) if (id !== undefined) return id const hint = variableHint(node.key, [...context.variableIdByKey.keys()]) - context.diagnostics.push({ - code: 'E_MAT_UNKNOWN_VARIABLE', - severity: 'error', - message: `Variable "${node.key}" does not exist in this workflow`, - hint, - ...location, - }) + context.diagnostics.push( + diagnostic( + 'E_MAT_UNKNOWN_VARIABLE', + `Variable "${node.key}" does not exist in this workflow`, + { + hint, + ...location, + } + ) + ) return node.key } case 'resource': { const id = context.resources?.toId(node.kind, node.key) ?? null if (id !== null) return id const hint = availableKeysHint(node.kind, context.resources?.keysOf?.(node.kind), node.key) - context.diagnostics.push({ - code: 'E_MAT_UNKNOWN_RESOURCE', - severity: 'error', - message: `No ${node.kind} "${node.key}" in this workspace`, - ...(hint !== undefined ? { hint } : {}), - ...location, - }) + context.diagnostics.push( + diagnostic('E_MAT_UNKNOWN_RESOURCE', `No ${node.kind} "${node.key}" in this workspace`, { + ...(hint !== undefined ? { hint } : {}), + ...location, + }) + ) return node.key } } diff --git a/packages/workflow-code/src/print/comments.ts b/packages/workflow-code/src/print/comments.ts index 952ca8b4cd9..ed440247c9b 100644 --- a/packages/workflow-code/src/print/comments.ts +++ b/packages/workflow-code/src/print/comments.ts @@ -6,6 +6,7 @@ import { decodeAnchor, encodeAnchor } from '../anchors' import type { PrintDiagnosticCode } from '../diagnostics/codes' +import { diagnostic } from '../diagnostics/factory' import { type Diagnostic, splitLines } from '../diagnostics/frame' import type { AnchorKey, BlockKey, CodeComment, PortKey } from '../model' import { breakParent, type Doc, hardline, join, lineSuffix, literalText } from './doc' @@ -86,12 +87,15 @@ export class CommentPlacer { if (!list) continue this.pending.delete(anchor) moved.push(...list) - this.diagnostics.push({ - code: 'W_COMMENT_MOVED', - severity: 'warning', - message: 'A comment has no printed anchor; moved to its block declaration.', - blockKey: block, - }) + this.diagnostics.push( + diagnostic( + 'W_COMMENT_MOVED', + 'A comment has no printed anchor; moved to its block declaration.', + { + blockKey: block, + } + ) + ) } return split(moved) } @@ -100,12 +104,11 @@ export class CommentPlacer { dropRemaining(): void { for (const anchor of this.pending.keys()) { const block = anchorBlock(anchor) - this.diagnostics.push({ - code: 'W_COMMENT_DROPPED', - severity: 'warning', - message: 'A comment was dropped: its anchor no longer exists.', - ...(block === null ? {} : { blockKey: block }), - }) + this.diagnostics.push( + diagnostic('W_COMMENT_DROPPED', 'A comment was dropped: its anchor no longer exists.', { + ...(block === null ? {} : { blockKey: block }), + }) + ) } this.pending.clear() } diff --git a/packages/workflow-code/src/print/edges.ts b/packages/workflow-code/src/print/edges.ts index 5adc9e66907..e5d9724c882 100644 --- a/packages/workflow-code/src/print/edges.ts +++ b/packages/workflow-code/src/print/edges.ts @@ -7,6 +7,7 @@ import { compareStrings } from '@sim/utils/string' import { getWorkflowSubflowHandleIds } from '@sim/workflow-types/workflow' import { portKey } from '../anchors' import { catalogBlock } from '../catalog/view' +import { diagnostic } from '../diagnostics/factory' import type { AnchorKey, BlockKey, CanonicalEdge, PortKey } from '../model' import { anchors } from './comments' import type { PrintContext } from './context' @@ -115,12 +116,11 @@ export function wiringStatements( [...targets].sort(byTarget).map((edge) => identExpr(ctx, edge.to)) ) const raw = (source: BlockKey, handle: string, target: BlockKey): Expr => { - ctx.diagnostics.push({ - code: 'W_RAW_HANDLE', - severity: 'warning', - message: `Edge handle ${handle} has no SDK form; printed as edge().`, - blockKey: source, - }) + ctx.diagnostics.push( + diagnostic('W_RAW_HANDLE', `Edge handle ${handle} has no SDK form; printed as edge().`, { + blockKey: source, + }) + ) return ctx.values.call('edge', [ identExpr(ctx, source), ctx.values.string(handle), diff --git a/packages/workflow-code/src/print/params.ts b/packages/workflow-code/src/print/params.ts index 598a46a2791..3542e0e8554 100644 --- a/packages/workflow-code/src/print/params.ts +++ b/packages/workflow-code/src/print/params.ts @@ -25,6 +25,7 @@ import { } from '../catalog/surface' import type { CatalogBlock, CatalogSubBlock } from '../catalog/types' import { type CodeFileLanguage, checkCodeFileBijection } from '../code-files' +import { diagnostic } from '../diagnostics/factory' import type { ResponseForm } from '../forms/response' import { JsonRef, type JsonTree, jsonFieldText, parseJsonField, refValueKind } from '../json-fields' import { CODE_FILE_EXTENSIONS, isCodeFilePath, SOURCE_CODE_DIR } from '../limits' @@ -216,12 +217,15 @@ function printCodeFile( ): PrintedValue | null { const language: CodeFileLanguage = tag === 'js' ? 'javascript' : tag === 'py' ? 'python' : 'shell' const inline = (reason: string): null => { - ctx.diagnostics.push({ - code: 'W_CODE_FILE_INLINED', - severity: 'warning', - message: `Function code of ${block.name} printed inline: ${reason}.`, - blockKey: block.key, - }) + ctx.diagnostics.push( + diagnostic( + 'W_CODE_FILE_INLINED', + `Function code of ${block.name} printed inline: ${reason}.`, + { + blockKey: block.key, + } + ) + ) return null } const stored = stringParam(block, 'language') ?? 'javascript' diff --git a/packages/workflow-code/src/print/refs.ts b/packages/workflow-code/src/print/refs.ts index 6d41422d1c3..5280110becf 100644 --- a/packages/workflow-code/src/print/refs.ts +++ b/packages/workflow-code/src/print/refs.ts @@ -15,6 +15,7 @@ import { parseStoredPath, } from '../code-files/ref-path' import type { PrintDiagnosticCode } from '../diagnostics/codes' +import { diagnostic } from '../diagnostics/factory' import type { Diagnostic } from '../diagnostics/frame' import type { BlockKey } from '../model' @@ -106,15 +107,17 @@ export class RefPrinter { from: BlockKey | null, code: 'W_UNRESOLVED_REF' | 'W_FORWARD_REF_LITERALIZED' ): null { - this.diagnostics.push({ - code, - severity: 'warning', - message: + this.diagnostics.push( + diagnostic( + code, code === 'W_FORWARD_REF_LITERALIZED' ? `Reference ${token} would point at a block declared later; printed as literal text.` : `Reference ${token} does not resolve in scope; printed as literal text.`, - ...(from === null ? {} : { blockKey: from }), - }) + { + ...(from === null ? {} : { blockKey: from }), + } + ) + ) return null } } diff --git a/packages/workflow-code/src/print/scaling.test.ts b/packages/workflow-code/src/print/scaling.test.ts index 2ef2fc30725..92be222febf 100644 --- a/packages/workflow-code/src/print/scaling.test.ts +++ b/packages/workflow-code/src/print/scaling.test.ts @@ -1,12 +1,13 @@ /** * Printing grows linearly with the workflow (see `growth` for how a case is measured). */ + +import { expectLinearGrowth } from '@sim/testing/helpers/scaling' import { describe, expect, it } from 'vitest' import { compareEdges } from '../canonicalize/edges' import { compileWorkflow } from '../compile' import { TEST_CATALOG } from '../compile/catalog.test-helpers' import type { CanonicalBlock, CanonicalEdge, CanonicalWorkflow } from '../model' -import { growth, MAX_GROWTH } from '../scaling.test-helpers' import { type Doc, fill, group, indent, line, printDoc, softline } from './doc' import { printWorkflow } from './printer' @@ -57,19 +58,17 @@ function siblingScopes(n: number): CanonicalWorkflow { describe('print scaling', () => { it('fills a long number-only array in linear time', () => { - const ratio = growth((n) => { + expectLinearGrowth((n) => { const doc = numberArray(n) return () => printDoc(doc, 80) }, 5_000) - expect(ratio).toBeLessThan(MAX_GROWTH) }) it('wires and orders many sibling scopes in linear time', () => { - const ratio = growth((n) => { + expectLinearGrowth((n) => { const workflow = siblingScopes(n) return () => printWorkflow({ workflow, catalog: TEST_CATALOG, options: { functionCode: 'inline' } }) }, 500) - expect(ratio).toBeLessThan(MAX_GROWTH) }) }) diff --git a/packages/workflow-code/src/print/tools.ts b/packages/workflow-code/src/print/tools.ts index 7bddd1aa607..460872038f8 100644 --- a/packages/workflow-code/src/print/tools.ts +++ b/packages/workflow-code/src/print/tools.ts @@ -14,6 +14,7 @@ import { toolModeOverrides } from '../canonicalize/modes' import { builderPins, buildFieldIndex, paramAliases, typedForBuilder } from '../catalog/surface' import type { BlockCatalog, CatalogBlock, CatalogBuilder } from '../catalog/types' import { catalogBlock, catalogBuildersOfType, declarationsOf } from '../catalog/view' +import { diagnostic } from '../diagnostics/factory' import { isJson } from '../lock' import { type CanonicalBlock, @@ -171,13 +172,16 @@ export function printTools( const elements = tools.map((tool) => { const code = tool.customTool?.snapshot?.code if (code !== undefined && holdsDetectedSecret(code)) { - ctx.diagnostics.push({ - code: 'W_LITERAL_SECRET', - severity: 'warning', - message: `A custom tool snapshot in ${block.key} holds what looks like a literal credential in its code, which this file carries.`, - hint: 'Move the credential into a workspace env variable and read it from the custom tool code, then save the custom tool again.', - blockKey: block.key, - }) + ctx.diagnostics.push( + diagnostic( + 'W_LITERAL_SECRET', + `A custom tool snapshot in ${block.key} holds what looks like a literal credential in its code, which this file carries.`, + { + hint: 'Move the credential into a workspace env variable and read it from the custom tool code, then save the custom tool again.', + blockKey: block.key, + } + ) + ) } const mcp = readMcpTool(ctx.catalog, tool) if (mcp) return mcpToolCall(ctx, block, mcp) @@ -343,12 +347,9 @@ export function storedSecret( name: string, password = true ): Expr { - ctx.diagnostics.push({ - code: 'W_LITERAL_SECRET', - severity: 'warning', - ...literalSecretFinding(name, { password }), - blockKey: block.key, - param: name, - }) + const { message, hint } = literalSecretFinding(name, { password }) + ctx.diagnostics.push( + diagnostic('W_LITERAL_SECRET', message, { hint, blockKey: block.key, param: name }) + ) return ctx.values.call('stored', []) } diff --git a/packages/workflow-code/src/source-meta.ts b/packages/workflow-code/src/source-meta.ts index 2d7bc3465dd..e6b26ccb23b 100644 --- a/packages/workflow-code/src/source-meta.ts +++ b/packages/workflow-code/src/source-meta.ts @@ -18,6 +18,7 @@ import { isRecordLike } from '@sim/utils/object' import { decodeAnchor, encodeAnchor } from './anchors' import type { SourceMetaDiagnosticCode } from './diagnostics/codes' +import { diagnostic } from './diagnostics/factory' import type { Diagnostic } from './diagnostics/frame' import { fieldNames, @@ -62,7 +63,7 @@ class SourceMetaReader { readonly diagnostics: SourceMetaDiagnostic[] = [] report(code: SourceMetaDiagnosticCode, segments: readonly string[], message: string): void { - this.diagnostics.push({ code, severity: 'error', message, path: pointer(segments) }) + this.diagnostics.push(diagnostic(code, message, { path: pointer(segments) })) } extensions(value: Record): Record { diff --git a/packages/workflow-code/src/structured/variables.ts b/packages/workflow-code/src/structured/variables.ts index 67253abf54f..803704576e3 100644 --- a/packages/workflow-code/src/structured/variables.ts +++ b/packages/workflow-code/src/structured/variables.ts @@ -6,6 +6,7 @@ import { isRecordLike } from '@sim/utils/object' import type { CanonContext } from '../canonicalize/context' import { toValue } from '../canonicalize/values' +import { diagnostic } from '../diagnostics/factory' import type { BlockKey, Value } from '../model' import { setOwn } from '../records' import { compilePlain, encodePlain, noLockItems, printPlain, type StructuredCodec } from './codec' @@ -33,13 +34,16 @@ function liftVariableAssignments( if (variableKey !== undefined) { setOwn(result, key, { $sim: 'var', key: variableKey }) } else { - context.diagnostics.push({ - code: 'W_CANON_UNRESOLVED_ID', - severity: 'warning', - message: `Variable assignment names unknown variable id "${member}"; kept verbatim`, - blockKey, - param, - }) + context.diagnostics.push( + diagnostic( + 'W_CANON_UNRESOLVED_ID', + `Variable assignment names unknown variable id "${member}"; kept verbatim`, + { + blockKey, + param, + } + ) + ) setOwn(result, key, { $sim: 'unresolved', kind: 'var', id: member }) } continue diff --git a/packages/workspace-code/package.json b/packages/workspace-code/package.json index 545f9d0dc56..76722f68ace 100644 --- a/packages/workspace-code/package.json +++ b/packages/workspace-code/package.json @@ -41,10 +41,6 @@ "./kinds/*": { "types": "./src/kinds/*.ts", "default": "./src/kinds/*.ts" - }, - "./table-cells/*": { - "types": "./src/table-cells/*.ts", - "default": "./src/table-cells/*.ts" } }, "scripts": { @@ -58,12 +54,13 @@ }, "dependencies": { "@babel/parser": "7.29.9", + "@sim/table-cells": "workspace:*", "@sim/utils": "workspace:*", - "@sim/workflow-code": "workspace:*", - "zod": "4.3.6" + "@sim/workflow-code": "workspace:*" }, "devDependencies": { "@babel/types": "7.29.8", + "@sim/testing": "workspace:*", "@sim/tsconfig": "workspace:*", "@types/js-yaml": "4.0.9", "fast-check": "3.23.2", diff --git a/packages/workspace-code/src/diagnostics/codes.ts b/packages/workspace-code/src/diagnostics/codes.ts index 4e3ebcfc4a1..a3b1ca034d5 100644 --- a/packages/workspace-code/src/diagnostics/codes.ts +++ b/packages/workspace-code/src/diagnostics/codes.ts @@ -2,9 +2,18 @@ * Every diagnostic code workspace source can emit beside the workflow pipeline's own (which keep * their codes, with files prefixed `workflows//`). The CLI and tests match on them, so they * freeze at release. The first letter fixes the severity, as in `@sim/workflow-code`. + * + * A code means one thing across both vocabularies: the few workspace findings that say what a + * workflow code already says ({@link SHARED_CODES}) use that code, and every other workspace code + * is one the workflow vocabulary does not hold ({@link WorkspaceDiagnosticCode} is `never` if it + * does). */ -import type { DiagnosticCode } from '@sim/workflow-code' +import { + type DiagnosticCode, + type HasDuplicates, + refusalClassifier, +} from '@sim/workflow-code/diagnostics' /** Declaration files and the directory layout. */ const DECLARATION_CODES = [ @@ -39,8 +48,6 @@ const DECLARATION_CODES = [ * apply writes nothing. Stage it, then apply again. */ 'E_BLOB_MISSING', - /** A size, count or nesting limit exceeded (a file, a kind, the staging area). */ - 'E_LIMIT_EXCEEDED', /** A resource key outside `^[a-z][a-z0-9_-]{0,63}$`. */ 'E_KEY_INVALID', /** @@ -230,8 +237,6 @@ const DEPLOYMENT_CODES = [ /** Plan and apply. */ const PLAN_CODES = [ - /** The resource changed on the server since the directory was read (HTTP 409). */ - 'E_APPLY_STALE', /** The server holds the resource but the directory has no base for it: get it first. */ 'E_APPLY_NO_BASE', /** The caller may not make this change. */ @@ -257,8 +262,6 @@ const PLAN_CODES = [ * stored workflow's Table block, which the run then drops. */ 'W_TABLE_COLUMN_RENAMED_IN_USE', - /** Another resource of the kind holds this name or key. */ - 'E_NAME_TAKEN', /** A workflow folder sent without its files whose base names no workflow of this workspace. */ 'E_WORKFLOW_FILES_MISSING', /** A workflow folder holding the base of another declared folder's workflow: a copy. */ @@ -290,15 +293,23 @@ const PLAN_CODES = [ 'I_KEY_REBOUND', /** A declaration under a new key for a resource the base read under another: keys never move. */ 'E_KEY_RENAMED', - /** - * More findings than one answer (or one file's list) reports; this one says where the list - * stopped. Informational: the findings it follows decide whether the apply is refused. - */ - 'I_FINDINGS_TRUNCATED', ] as const -/** Every code a workspace source finding can carry, as a value: the wire contracts' enum. */ -export const WORKSPACE_DIAGNOSTIC_CODES = [ +/** + * Workflow codes workspace findings carry with the same meaning: a resource changed on the server + * since the directory was read (`409`), a size, count or nesting limit exceeded (a file, a kind, + * the staging area), a name or key another resource of the kind holds, and the marker that ends a + * cut list (more findings than one answer, or one file's list, reports). + */ +const SHARED_CODES = [ + 'E_APPLY_STALE', + 'E_LIMIT_EXCEEDED', + 'E_NAME_TAKEN', + 'I_FINDINGS_TRUNCATED', +] as const satisfies readonly DiagnosticCode[] + +/** The codes only workspace source emits. */ +const OWN_CODES = [ ...DECLARATION_CODES, ...BASE_CODES, ...SECRET_CODES, @@ -312,44 +323,28 @@ export const WORKSPACE_DIAGNOSTIC_CODES = [ ...DEPLOYMENT_CODES, ...PLAN_CODES, ] as const -export type WorkspaceDiagnosticCode = (typeof WORKSPACE_DIAGNOSTIC_CODES)[number] -/** How a finding weighs: an error refuses, a warning and an info only report. */ -export type DiagnosticSeverity = 'error' | 'warning' | 'info' +type OwnCode = (typeof OWN_CODES)[number] -/** The severity a code's prefix fixes. */ -export type SeverityOfCode = C extends `E_${string}` - ? 'error' - : C extends `W_${string}` - ? 'warning' - : 'info' - -/** The severity of any code, by its prefix. */ -export function severityOf(code: string): DiagnosticSeverity { - return code.startsWith('E_') ? 'error' : code.startsWith('W_') ? 'warning' : 'info' -} +/** Every code a workspace source finding can carry, as a value: the wire contracts' enum. */ +export const WORKSPACE_DIAGNOSTIC_CODES = [...OWN_CODES, ...SHARED_CODES] as const -/** How a refused apply is classed: its HTTP status on the server, its exit code in the CLI. */ -export type RefusalClass = 'conflict' | 'locked' | 'payload_too_large' | 'validation' +/** + * One of {@link WORKSPACE_DIAGNOSTIC_CODES}. It is `never`, which fails every use, if a workspace + * code is declared twice or is also a workflow code outside {@link SHARED_CODES}. + */ +export type WorkspaceDiagnosticCode = HasDuplicates extends false + ? [Extract] extends [never] + ? (typeof WORKSPACE_DIAGNOSTIC_CODES)[number] + : never + : never /** - * The class of a refusal with these error codes, the most specific first: a resource changed - * since the directory was read (`conflict`), then a locked one, then a size limit, else - * `validation`. + * The class of a workspace refusal with these error codes (see `refusalClassifier`): beside the + * workflow vocabulary's, a locked resource is `locked`, and a plan, file or table over its size + * limit is `payload_too_large`. */ -export function refusalClass( - codes: Iterable -): RefusalClass { - const present = new Set(codes) - if (present.has('E_APPLY_STALE')) return 'conflict' - if (present.has('E_APPLY_LOCKED')) return 'locked' - if ( - present.has('E_LIMIT_EXCEEDED') || - present.has('E_PLAN_TOO_LARGE') || - present.has('E_FILE_TOO_LARGE') || - present.has('E_TABLE_TOO_MANY_COLUMNS') - ) { - return 'payload_too_large' - } - return 'validation' -} +export const refusalClass = refusalClassifier({ + locked: ['E_APPLY_LOCKED'], + payload_too_large: ['E_PLAN_TOO_LARGE', 'E_FILE_TOO_LARGE', 'E_TABLE_TOO_MANY_COLUMNS'], +} satisfies Record<'locked' | 'payload_too_large', readonly OwnCode[]>) diff --git a/packages/workspace-code/src/diagnostics/finding.ts b/packages/workspace-code/src/diagnostics/finding.ts index b6f866905c8..2c8e669c7b5 100644 --- a/packages/workspace-code/src/diagnostics/finding.ts +++ b/packages/workspace-code/src/diagnostics/finding.ts @@ -1,4 +1,3 @@ -import { truncateAtCodePoint } from '@sim/utils/string' import { codeFrame, type DiagnosticCode, @@ -6,13 +5,15 @@ import { LineIndex, type SourceSpan, } from '@sim/workflow-code' -import type { WorkspaceKind } from '../kind' import { + type AnswerBound, + boundFindingText, type DiagnosticSeverity, - type SeverityOfCode, - severityOf, - type WorkspaceDiagnosticCode, -} from './codes' + type SeverityOf, + diagnostic as sharedDiagnostic, +} from '@sim/workflow-code/diagnostics' +import type { WorkspaceKind } from '../kind' +import { refusalClass, type WorkspaceDiagnosticCode } from './codes' /** The resource a finding is about: the CLI joins a finding to its planned change by both. */ export interface FindingResource { @@ -34,7 +35,7 @@ type ResourceFields = Partial */ export type WorkspaceDiagnostic = C extends WorkspaceDiagnosticCode - ? DiagnosticFields & ResourceFields & { code: C; severity: SeverityOfCode } + ? DiagnosticFields & ResourceFields & { code: C; severity: SeverityOf } : never /** The optional fields of a finding. */ @@ -45,8 +46,6 @@ type DiagnosticExtras = Omit & ResourceFields * are dropped, so a file of bad values costs no more than its first hundred. */ export const MAX_FINDINGS_PER_FILE = 100 -/** Code units of a message or hint kept; longer text, which quotes the source, is cut. */ -const MAX_MESSAGE_LENGTH = 1_000 /** The finding that ends a file's list at {@link MAX_FINDINGS_PER_FILE}. */ export function findingLimitReached(extras: DiagnosticExtras = {}): WorkspaceDiagnostic { @@ -57,20 +56,32 @@ export function findingLimitReached(extras: DiagnosticExtras = {}): WorkspaceDia ) } -/** A finding with its severity filled in from the code, and its message and hint length-capped. */ +/** + * A workspace finding with its severity filled in from the code. Its message and hint are cut as + * the compile cuts its own (`boundFindingText`): a limit that stops work, so a finding never holds + * more of the source it quotes (a 50 KB key, a hundred times in one file) than an answer could show. + */ export function diagnostic( code: C, message: string, extras: DiagnosticExtras = {} ): WorkspaceDiagnostic { - const { hint, ...rest } = extras - return { - ...rest, - ...(hint !== undefined ? { hint: truncateAtCodePoint(hint, MAX_MESSAGE_LENGTH, '…') } : {}), - code, - message: truncateAtCodePoint(message, MAX_MESSAGE_LENGTH, '…'), - severity: severityOf(code), - } as WorkspaceDiagnostic + return boundFindingText(sharedDiagnostic(code, message, extras)) as WorkspaceDiagnostic +} + +/** + * How one workspace answer bounds its findings (`boundAnswerFindings`): at most 1,000, errors + * first and those of a classed refusal (stale, locked, too large) first of all, then warnings, + * then infos. No per-severity or per-code cap: a status reads every server-change finding of the + * resources it lists. + */ +export const WORKSPACE_ANSWER_BOUND: AnswerBound = { + limits: { + total: 1_000, + perSeverity: Number.POSITIVE_INFINITY, + warningsPerCode: Number.POSITIVE_INFINITY, + }, + classify: refusalClass, } /** Spans and code frames in one file, splitting its lines once however many it locates. */ diff --git a/packages/workspace-code/src/diagnostics/index.ts b/packages/workspace-code/src/diagnostics/index.ts index 9782bccd98e..f4a9ac0aeb4 100644 --- a/packages/workspace-code/src/diagnostics/index.ts +++ b/packages/workspace-code/src/diagnostics/index.ts @@ -1,9 +1,4 @@ -export { - type RefusalClass, - refusalClass, - severityOf, - type WorkspaceDiagnosticCode, -} from './codes' +export { refusalClass, type WorkspaceDiagnosticCode } from './codes' export { diagnostic, type FindingResource, @@ -13,6 +8,7 @@ export { MAX_FINDINGS_PER_FILE, resourceOf, sourceLocator, + WORKSPACE_ANSWER_BOUND, type WorkspaceDiagnostic, wholeFile, } from './finding' diff --git a/packages/workspace-code/src/index.ts b/packages/workspace-code/src/index.ts index 6bcc816973f..cc7d8eaa381 100644 --- a/packages/workspace-code/src/index.ts +++ b/packages/workspace-code/src/index.ts @@ -12,9 +12,8 @@ export { export { type FlatDiagnostic, flattenDiagnostic, - type RefusalClass, refusalClass, - severityOf, + WORKSPACE_ANSWER_BOUND, type WorkspaceDiagnosticCode, } from './diagnostics' export { fingerprintOf } from './fingerprint' diff --git a/packages/workspace-code/src/kinds/table.test.ts b/packages/workspace-code/src/kinds/table.test.ts index 8084e0b7e12..24055354e80 100644 --- a/packages/workspace-code/src/kinds/table.test.ts +++ b/packages/workspace-code/src/kinds/table.test.ts @@ -1,4 +1,4 @@ -import { growth, MAX_GROWTH } from '@sim/workflow-code/scaling.test-helpers' +import { expectLinearGrowth } from '@sim/testing/helpers/scaling' import fc from 'fast-check' import { describe, expect, it } from 'vitest' import { refusalClass } from '../diagnostics' @@ -188,7 +188,7 @@ describe('createTableCodec', () => { expect(parse(20_000)().diagnostics).toMatchObject([ { code: 'E_TABLE_OPTIONS', message: expect.stringContaining('20000 options') }, ]) - expect(growth(parse, 5_000)).toBeLessThan(MAX_GROWTH) + expectLinearGrowth(parse, 5_000) }) it('pins a duplicate column name to the second column, not the whole table', () => { diff --git a/packages/workspace-code/src/kinds/table.ts b/packages/workspace-code/src/kinds/table.ts index a4b2f2b99cc..5dfdf8bbcf9 100644 --- a/packages/workspace-code/src/kinds/table.ts +++ b/packages/workspace-code/src/kinds/table.ts @@ -18,6 +18,10 @@ * column-type registry and makes the codec with {@link createTableCodec}, so the types, what * metadata each owns, which may be unique and how many one table holds all come from the registry. */ + +import { isSupportedCurrencyCode } from '@sim/table-cells/currency' +import { type BatchRowErrors, coerceRowCells, refusedRows } from '@sim/table-cells/rows' +import type { CellColumn, CellRow } from '@sim/table-cells/types' import type { Json } from '@sim/workflow-code' import { asDeclared, @@ -46,8 +50,6 @@ import { diagnostic, resourceOf, type WorkspaceDiagnostic, wholeFile } from '../ import { KIND_DIRS, type KindCodec, type KindLayout } from '../kind' import { DECLARATION_EXTENSION } from '../layout' import { WORKSPACE_SOURCE_LIMITS } from '../limits' -import { type BatchRowErrors, coerceRowCells, refusedRows } from '../table-cells/rows' -import type { CellColumn, CellRow } from '../table-cells/types' import { groupByFile } from './group' import { parseTableRows, @@ -127,22 +129,6 @@ export const TABLE_TEXT_RULES = { maxColumnName: 50, maxDescription: 500, } as const -const CURRENCY_CODE = /^[A-Za-z]{3}$/ - -/** - * Whether `code` is an ISO 4217 code this runtime knows, in any letter case, as the server's - * currency column checks it (`isSupportedCurrencyCode`); a runtime that cannot list currencies - * checks the shape alone. - */ -function isSupportedCurrencyCode(code: string): boolean { - if (!CURRENCY_CODE.test(code)) return false - try { - return Intl.supportedValuesOf('currency').includes(code.toUpperCase()) - } catch { - return true - } -} - const CALLEE = 'defineTable' /** diff --git a/packages/workspace-code/src/status.test.ts b/packages/workspace-code/src/status.test.ts index e15181d6964..c87d0141455 100644 --- a/packages/workspace-code/src/status.test.ts +++ b/packages/workspace-code/src/status.test.ts @@ -1,4 +1,4 @@ -import { growth, MAX_GROWTH } from '@sim/workflow-code/scaling.test-helpers' +import { expectLinearGrowth } from '@sim/testing/helpers/scaling' import { describe, expect, it } from 'vitest' import { localResourceChanges } from './status' @@ -16,6 +16,6 @@ describe('localResourceChanges', () => { const files = changedFiles(count) return () => localResourceChanges(files) } - expect(growth(grouped, 10_000)).toBeLessThan(MAX_GROWTH) + expectLinearGrowth(grouped, 10_000) }) }) diff --git a/scripts/check-monorepo-boundaries.ts b/scripts/check-monorepo-boundaries.ts index 62617ed1504..9637cc82346 100644 --- a/scripts/check-monorepo-boundaries.ts +++ b/scripts/check-monorepo-boundaries.ts @@ -19,11 +19,12 @@ * context type. * Fix by taking the surface fact (a cursor's route identity, a wire shape) as input from * the adapter. - * 3. `@sim/workflow-code` and `@sim/workspace-code` (every non-test file under their `src`) - * compile untrusted source on the server, in the browser and in the CLI, so they must stay pure - * and must never execute code: no Node built-in (`vm`, `child_process`, `fs`, any `node:*`, - * even type-only), no dynamic `import()`, `import.meta` or `require`, no `.constructor` read - * (a function's constructor is `Function`), and no value reference to `eval`, `Function`, + * 3. `@sim/workflow-code`, `@sim/workspace-code` and the `@sim/table-cells` rules they check rows + * with (every non-test file under their `src`) compile untrusted source on the server, in the + * browser and in the CLI, so they must stay pure and must never execute code: no Node built-in + * (`vm`, `child_process`, `fs`, any `node:*`, even type-only), no dynamic `import()`, + * `import.meta` or `require`, no `.constructor` read (a function's constructor is + * `Function`), and no value reference to `eval`, `Function`, * `Reflect` or the global object under any name (`globalThis`, `window`, `self`, `global`), * which also bans computed lookups such as `globalThis[name]`. Fix by keeping I/O in * the caller (the compiler takes file text as input) and interpreting the AST instead of @@ -183,7 +184,7 @@ async function scanImports( return offenders } -const PURE_PACKAGES = ['workflow-code', 'workspace-code'].map((name) => ({ +const PURE_PACKAGES = ['workflow-code', 'workspace-code', 'table-cells'].map((name) => ({ name: `@sim/${name}`, src: path.join(PACKAGES_DIR, name, 'src'), })) @@ -372,7 +373,7 @@ async function main() { if (offenders.length === 0) { console.log( - '✅ Monorepo boundaries OK: no package imports from apps/*, application code is surface-neutral, and @sim/workflow-code and @sim/workspace-code are pure' + '✅ Monorepo boundaries OK: no package imports from apps/*, application code is surface-neutral, and @sim/workflow-code, @sim/workspace-code and @sim/table-cells are pure' ) return }