Skip to content

fix(workflow-code): correctness, safety and speed for workflows and workspace as code - #10237

Merged
waleedlatif1 merged 105 commits into
stagingfrom
fix/workflow-code-wave1
Nov 20, 2026
Merged

waleedlatif1 merged 105 commits into
stagingfrom
fix/workflow-code-wave1

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Wave 1 of fixes from an 8-dimension evaluation of workflows/workspace as code on staging (fidelity, workspace source, agent UX, human DX, code quality ×2, soundness/security, integration/performance), followed by a 5-lens audit of this branch and a re-audit of those fixes.

Deploy & drift safety

  • live: 'draft' no longer publishes drafts edited outside source: only a draft this apply wrote, or one equal to the folder's base, is deployed (W_DEPLOY_DRAFT_NOT_SOURCE). Deploys and chat writes carry an expected draft fingerprint and refuse a draft that moved (409 / E_APPLY_STALE).
  • Server-side drift is a warning by default (W_SERVER_CHANGED|DELETED|MOVED), incl. managed-row drift; no more "unchanged" for drifted resources.
  • .sim/written.json holds digests only; machine-local stats moved to a self-ignoring .sim/cache/. CLI sends gitSha + sanitized repo; server refuses credentialed repo URLs.

Adopt / promote / partial apply

  • Adopting an unread same-named resource may only add to it; anything else is E_APPLY_NO_BASE (all kinds).
  • Promote warns about target-deleted resources and target-only deletes; promotion-specific hints.
  • MCP URLs validated at plan time (after role checks); a part-way stop marks what was written. 10k-row table creates work (inserts chunked under Postgres's parameter limit). Cross-workspace raw ids refused at apply.

One refusal decision

  • refuseEditedFindings (workflow-code) is the single rule the server and offline check both use; refusals are decided on findings the output caps withheld. Iteration-scope checked once (compile); new W_REF_OUT_OF_SCOPE/E_APPLY_REF_OUT_OF_SCOPE, W_CODE_FILE_REF_SYNTAX, W_OUTPUT_PATH_UNDECLARED, E_APPLY_VARIABLE_DELETE, E_CODE_FILE_PATH.

Compiler & CLI

  • Truthful diff summaries for name swaps and retypes; missing-import diagnostic; notes not scanned; per-code warning cap; identity kept for branch keys/inputs/variables (with evidence); stripe stays stripe; tags in JS strings checked; one MCP auth-scheme rule.
  • Directory-kind guard; one code-file path rule (never fail after a commit); workspace.d.ts refreshed; get prints server findings and warns about orphaned code files; actionable unknown-resource hints; next: never truncated; --dir defaults to . for readers; primer covers run/debug/credentials/promote; cli search finds source for natural tasks.

Server

  • workflow-code feature gate declared on operations and checked after authorization (FEATURE_NOT_ENABLED; no existence leak).
  • New read-cost plan routes (POST …/source/plan) for status, with their own rate bucket, slot lane and per-folder cost; plans write nothing (readDraftFingerprint, no key allocation).
  • Realtime subblock and variable flushes take the workflow row lock and treat 0 rows as retryable (lost-update races proven and fixed).

Performance

  • Migration 0467 restores/declares the workflow_edges single-column block FK indexes (missing on some databases; schema never declared them). Normalized save is diff-based. 700-block one-field edit: ~10.5 s → ~10 ms in benchmark.

Every behaviour fix has a test that fails without it (integration tests on real Postgres where it matters). Generated OpenAPI, CLI API types, MCP operations, route table and CLI docs regenerated.

@greptile

🤖 Generated with Claude Code

Deleting a workflow_blocks row runs two ON DELETE CASCADE lookups on
workflow_edges (WHERE source_block_id = $1 / target_block_id = $1). Only the
composite (workflow_id, *_block_id) indexes are declared, and they cannot serve
that lookup, so a database without 0043's single-column fk_idx pair seq-scans
the whole edges table twice per deleted block inside the workflow row lock.
Schema-pushed databases never get the pair, and a drifted one can lose it: a
700-block delete against 164k edges takes 7.5 s locally without the indexes
and 8 ms with them.

Declare both indexes in schema.ts and add a concurrent, replay-safe migration
that is a no-op where they already exist.
saveWorkflowToNormalizedTables deleted every block, edge and subflow of the
workflow and re-inserted them all, so a one-field edit paid a delete (and its
foreign-key cascade checks) per stored block inside the workflow row lock that
source apply, operations apply and Copilot edits hold.

Read the stored rows FOR UPDATE (the rows the delete locked), diff them by id
against the new state, and write only the difference: unchanged rows are left
alone with their timestamps, changed rows are rewritten in one confined upsert
per table, new ids are plain inserts (so an id another workflow owns still
fails with the unique violation callers map to a conflict), and removed rows
are deleted after edges are retargeted. The resulting rows are the ones the
full replace left, and the signature is unchanged for every caller.

The integration suite saves random edit sequences both ways and compares the
rows, and proves a one-field edit rewrites one block. bench-normalized-save
measures edit, shrink and grow at 5/50/300/700 blocks: a one-field edit at 700
blocks drops from 40 ms to 10 ms with the edge FK indexes, and from 10.5 s to
10 ms without them.
…FEATURE_NOT_ENABLED

The workflow-code flag check was copied into ten use cases, and three ran it in
resolveContext before authorizeWorkspaceOperation: with the flag off, a caller
with no access got 403 for a real workflow or workspace id and 404 for a made-up
one, and the 403 named no cause.

An operation now declares the rollout flag it ships behind (feature on
defineWorkspaceOperation); the shared use-case wrapper checks it last in the
authorize phase, so execute, authorize and HEAD answer the same, an unreachable
workspace stays concealed, and a member is refused with the new
FEATURE_NOT_ENABLED 403 detail code. The per-use-case copies are gone.
…ost to a graph replace

The flush read-modified-wrote a block's subBlocks with no workflow row lock and
confirmed without checking the update hit a row. A keystroke landing between a
source apply's locked re-read and its delete-and-reinsert was confirmed, then
overwritten; one landing after the delete updated nothing and was still
confirmed.

The flush now takes the workflow row lock (no key update, so a blind replace's
foreign-key checks cannot deadlock with it) before reading, which serializes it
with every read-then-rewrite writer, and reports a retryable failure when its
update affected no row, as after a blind replace.
… a source apply

The variable flush read the workflow's variables without the row lock and
updated them afterwards. A source apply that rewrote the variables under the
lock in between was reverted by the flush's stale copy, though both reported
success. The flush now takes the same workflow row lock as the subblock flush
before its read.
…s a workflow does

The MCP declaration kept its own auth-scheme regex, without digest and key, so
'Digest {{TOKEN}}' was refused as a literal secret there while a workflow
accepted it. Both now strip the scheme through withoutAuthScheme in secrets.ts.
…refused

lp.index after the loop, $ref.loop.index or <loop.index> in a block in no loop,
and a block moved out of its container passed tsc, check and apply, then failed
(or read nothing) at run time. Compile now reports W_REF_OUT_OF_SCOPE for every
such read (handle, code file $ref, tag text); the apply refuses it as
E_APPLY_REF_OUT_OF_SCOPE on a block the write adds, changes or moves, and an
offline check does the same on a new block, an edited function file and a block
the source moved into another container. A stored workflow that already holds
one keeps the warning, so it still applies. The workflow lint reports the same
reads for the operations path.

The SDK keeps index/currentItem/items on a container handle: a block in a
nested container reads an outer loop's index through it, and the printer emits
that form.
A <key.path> tag in a JavaScript string, or in a Python or shell file, is
resolved at run time and rewritten on a rename, but compile never checked it:
'<nosuchblock.x>' or <variable.ghost> passed with no finding. Compile now finds
each tag with the runtime's scanner and checks it as it checks a $ref chain
(W_UNRESOLVED_REF, W_UNKNOWN_OUTPUT_PATH, W_REF_OUT_OF_SCOPE), written as the
file writes it, so the apply refuses a dangling tag in code the edit changed.
A token with no path (<b> in HTML text) is not a reference.
Only JavaScript code files are rewritten, so $ref.key.path or $env.NAME in a
.py or .sh file was saved as written and broke the code at run time, while the
primer told agents to write $ref in any function code file. Compile now reports
W_CODE_FILE_REF_SYNTAX with the form those files take (<key.path>, {{NAME}});
the apply refuses it as E_APPLY_CODE_FILE_REF_SYNTAX on code the edit changed,
and an offline check in a file changed since the get. The code-file refusals
the server and the check apply are one shared map, EDITED_CODE_FILE_REFUSALS.
The primer, the docs and SYNTAX.md state the per-language form.
…a retype

diffCanonical dropped every rename pair whose base key the target still held,
so two blocks that swapped display names fell back to pairing by key: the dry
run and apply reported both as code/language changes with 3/3 edges moved
where only two names changed. The same drop broke a chain rename after a name
collision (Fetch Data renamed while Fetch.Data took its key), reporting a
false 'both name and const changed' E_APPLY_BLOCK_DELETE. Rename pairs now win
over key pairing, and an unpaired target block's edges never match a base edge.

A retype mints a new id under the same key; it was reported as one modified
block with one edge each way. replacedBetween names such keys from the two
identities, and diffCanonical reports them as a removed plus an added block
with all their edges. The delete gate still treats a retype as no delete,
since W_MAT_RETYPE_RECREATED covers it.
A loop, a template tag or env used without importing it got a wrong lead:
'loop is not an SDK namespace. Did you mean loops.*?', 'This template tag is
not supported... use js``' on a js`` tag, 'env is not defined'. Any SDK core
export or catalog namespace the file uses unimported is now one finding,
'<name> is not imported from @sim/workflow-sdk', with the import to add.

HTML in a note gave 99 identical '<b> names no block' warnings on every check,
and the 100-warning cap could hide a real warning behind them. A note never
resolves references, so its text is not scanned; the reporter lists a finding
repeated at the same place once, and caps each warning code at 20 so a flood
of one code leaves room for the rest. Errors keep their own cap.
loop.doWhile('X', { condition, count: 7 }) passed check and the dry run with no
finding while the count changed nothing: the executor ignores the count when a
condition is set, and the printer never writes one. Compile now reports
W_FIELD_NOT_APPLICABLE on it, from the form table's printedWhen, with the fix:
remove count, or remove condition to run count times.
…eps its name

const stripe = trigger.stripe.webhook(...) came back as const stripe2 from
create and every later get: the printer reserved every catalog namespace as a
block ident, imported or not. It now reserves only the namespaces of the
builders of the block and agent tool types the workflow holds, the ones the
printed file may import, so a const shadows no import it uses.
…eps its id

Block renames keep their ids, but renaming a condition branch key (same
expression), a start input field or a variable key minted a new id: the edge
from the branch was removed and re-added, and anything keyed on the row,
field or variable id lost continuity. Materialize now infers these renames as
it does a block's: when exactly one entry's key is new and exactly one known
id's key is gone, and the former entry agrees (the same branch expression, the
same field or variable type), the entry keeps that id. Anything else still
gets a new id.
…leting a block does

An apply that dropped a variable from workflow(name, { variables }) deleted it
with no --allow-delete, even when code read it, while a block delete is
refused. The apply now refuses it as E_APPLY_VARIABLE_DELETE, naming the
variables, unless allowDelete is passed; a variable whose key was renamed (same
type) keeps its id and is no delete. The contract, the CLI help, the primer and
the docs say --allow-delete covers variables.
A workflow get into a workspace source directory (or a workspace get into a
workflow directory) replaced its base and exited 0, so staleness could no
longer be detected and unapplied edits sat in a directory neither command
reads. One guard, assertSourceDirKind, now refuses the other kind in both gets,
create, check and every command that reads a workflow base, --force included.
… and CLI

A function file named functions/My_Sink.js (or for_body.py) passed check and
the dry run, was stored, and then the CLI exited 1 "unsafe file path" without
writing the base or lock, because the CLI only accepted the printer's kebab
names while the compiler accepted any relative path. isCodeFilePath in
@sim/workflow-code/limits is now the one rule: the compiler refuses anything
else as E_CODE_FILE_PATH (so check and the server refuse it before a write),
the printer only reuses a hinted path it accepts, and the CLI refuses it before
sending. After a stored write the CLI never throws on an answer path: it skips
one it would not write, warns, and still writes the lock and base.
…as one finding

An output-reference typo in an existing block passed check with a warning, while
the dry run refused it as E_APPLY_LINT, printed beside W_UNKNOWN_OUTPUT_PATH and
W_JSON_REF_UNTYPED for the same reference. Now:
- compile drops W_JSON_REF_UNTYPED at a reference whose path is already unknown;
- the server drops the lint's block-output finding on a block compile reported a
  dangling entry reference for, and where the lint would refuse it (a block the
  apply adds or changes) refuses compile's finding as E_APPLY_UNRESOLVED_REF at
  the reference;
- check refuses W_UNKNOWN_OUTPUT_PATH and W_UNRESOLVED_REF in workflow.ts as
  E_APPLY_UNRESOLVED_REF on a block the edit added or changed, telling a changed
  block from the per-block digests .sim/stored-edges.json now records.
…ce types

workspace.d.ts in a workflow source directory was written only by get, so tsc
refused a child workflow (or table, credential) created since, and the edit
loop's type-check step failed until an undocumented get. Every workflow source
command that already talks to the server now re-reads the resources into
workspace.d.ts and the manifest beside it (apply, its dry run and plan, create;
status follows when it becomes a read), rewriting only what changed and never
failing the command over a listing it could not get.
…ed code files

workflows source get dropped the diagnostics the server returns with the printed
source, so a function whose literal secret became code: stored() changed with no
explanation, and the old function file holding the secret stayed on disk
unmentioned. Both gets now print and return the server's findings, and every
write back (get, apply, create) warns with the path of a code file the entry or
the previous lock named that the printed source no longer does, leaving it on
disk rather than deleting a file the person may have written.
…reates it

"This workspace has no credential a source can name yet." was a dead end: an
agent had to search for sim credentials connect. E_MAT_UNKNOWN_RESOURCE now ends
with the sim command that creates a resource of that kind (credentials connect
or create, tables, knowledge, skills, custom-tools, mcp-servers create, or
workflows source create) and where the keys are listed; a did-you-mean hint
keeps pointing at the near key instead.
With --output table the record renderer clamped every field to 160 columns, so
the next: line, the one instruction a write stopped at its plan (exit 14) leaves,
was cut at the very flag the person needed. Result fields holding a command are
now printed whole in every display format.
…other readers

check, status, diff and params defaulted --dir to . while workflow and workspace
apply required it, so apply failed when run from inside the directory the primer
says to run commands from. One rule now: a command that reads an existing source
directory (check, status, diff, params, apply) defaults --dir to the current
directory; get and create, which write one, still require it.
…search finds them

The source primer stopped at apply, so running a workflow, reading why a run
failed, adding a credential and promoting a workspace each cost an agent turns
of discovery, and cli search missed source for natural phrasings. The primer
(still within its byte budget) now names sim workflows run, logs get --trace,
credentials connect with the key a name slugs to, and apply --to in the
workspace primer; a test keeps every command and flag it names real. cli search
indexes the tasks a command answers (searchTasks), so "build a workflow",
"change the model in every agent block" and "promote dev to prod" rank the
source commands first.
The CLI resent a request answered 429 after a silent wait of up to a minute,
six times, so a workspace status hitting the source budget looked like a
three-minute hang. Each wait now prints how long it is and which retry.
…ly dry run

workflows and workspaces source status posted an apply dry run, so every status
spent the source-write budget (one unit per workflow folder for a workspace):
a clean 27-workflow workspace status took 171-178 s through 429 waits, and the
status, dry run, apply loop allowed about one call per two minutes.

The server gains POST /api/v2/workflows/{id}/source/plan and
POST /api/v2/workspaces/{id}/source/plan (workflows.source.plan and
workspaces.source.plan, read role, api:read): the same pipeline as a dry run,
recording no dry-run analytics, on the generic and the source-read budget (one
unit whatever the directory holds). Both status commands read them, falling
back to the apply dry run only on a server without the route, and the workflow
status also refreshes the directory's workspace types.
…ectory never read

A declaration naming an unbound resource of the same name adopted it and planned the full diff
against it with no base: a skill or tool was overwritten, and a table lost its columns and rows
under --allow-delete. An adopt may now only add to the resource (bind its key when the declaration
matches, add a column, rows into an empty table, a knowledge base document), which keeps a
stopped create converging; any other adopt is E_APPLY_NO_BASE naming what it would change, for
tables, skills, custom tools, knowledge bases and MCP servers.
Creating a managed table with 10,000 rows (the documented maximum) inserted every row in one
statement, 9 parameters a row, past Postgres's 65,535-parameter limit, and the apply stopped with
an opaque E_APPLY_FAILED. Batch inserts now write 1,000 rows a statement. An unexpected write
failure now names its cause by database error code (a timeout, a deadlock, a statement too large)
and the apply id to report, never the error text.
… MCP URL

The dry run checked an MCP server's URL only by role, so a host that does not resolve (or one the
egress policy refuses) passed and then stopped the real apply after earlier resources were
written. The plan now checks every URL a create, replace or URL change writes the way the write
does (allowlist, egress policy, DNS), refusing before anything is written. After any part-way
stop the CLI reprints the plan with each change marked written, failed or not written, and the
JSON result lists notWritten.
…lan display

A rename-only apply recorded zero variable changes: the audit's variablesChanged, the
analytics op count and the workspace plan's variables field summed added, removed and
modified, never renamed. A renamed variable whose type or value also changed is now also
listed under modified by its new key.
…f its kind

The foreign-id check skipped any id an enclosing apply creates or adopts, whatever field
named it, so a knowledge base being adopted could pass as a table id. Pending ids are
now matched by kind.
…art input

W_JSON_REF_UNTYPED was suppressed wherever W_OUTPUT_PATH_UNDECLARED fired, which covers
both x.error with no error edge (moot, nothing is read) and an input the start does not
declare, which a caller may still pass as text that breaks the JSON. Only the x.error
reads are suppressed now.
…ery block that runs it

A code file read by several blocks was parsed and checked once, for the first block, so
a second block outside the loop could run return $ref.each.index without
W_REF_OUT_OF_SCOPE, and a source apply (which turns the lint's scope walk off) saved it.
Each later reader's iteration-field references are now checked for its own scope.
A Python or shell file (or a JavaScript string) writes an env variable as {{NAME}}, which
the run substitutes, but the code-file scan kept only <block.path> tags, so a misspelled
name compiled without W_ENV_UNKNOWN. Tokens holding an env-variable name are now checked
like $env.NAME.
… read

withhold() built each refusable warning past the cap with its hint resolved, so the lazy
did-you-mean ran for every withheld warning although few are ever listed. The hint is now
a getter computed on first read, and the refusal raises only the withheld findings it
lists, so it reads at most the error budget's worth.
…y reads a lock

A workflow apply in a workspace source directory failed on the missing .sim/lock.json and
told the user to run a get, which would replace the workspace base, instead of the
directory-kind refusal that names the workspace commands.
…ock tag hint

For a <key.path> tag in a code file the fallback hint read <<its new key>>, a tag users
might copy. It now writes <newkey> (or $ref.newkey) and says what newkey is.
…get-only deletes

A promotion that stopped part-way warned of every planned delete of a resource only the
target holds, including the ones it never reached, so the result claimed deletes that did
not happen. Once an apply has failed, only the deletes it wrote are named.
diffCanonical listed removed blocks in the base's block order, which is the order the
stored rows load in, so the delete refusal and the diff summary named them in an order
that said nothing and changed with the rows' ids. They are now sorted by key.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

Ordering edges by created_at then id put edges saved together (one timestamp) in id order,
and the executor assigns a handle-less condition or router edge its branch by position, so
a reload could switch branches. A save now rewrites a workflow's edges whole, in the state's
order and a microsecond apart, whenever any edge or their order changes, and the loader
orders edges by created_at then ctid, so rows an older save wrote with one timestamp keep
their insertion order.
…left findings out

Past the cap the compile keeps one W_UNRESOLVED_REF per code, block and file, so the
listed references to a removed block could stand for a genuinely missing one after them,
and exempting them let the check pass what the server refuses.
A module-level order read workflowEdges at import, which broke every test that mocks
@sim/db without it and imports the loader.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

Each rewritten edge binds seven parameters with its timestamp offset, so a workflow at the
10,000-edge source limit sent 70,000 in one insert, past Postgres's 65,535, and the save
rolled back. Edges now go in batches of 5,000, keeping their offsets, and the old ones are
deleted by workflow rather than by a list of ids.
@waleedlatif1
waleedlatif1 merged commit 02fb744 into staging Nov 20, 2026
84 of 85 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/workflow-code-wave1 branch November 20, 2026 18:43

This branch was previously deployed

1 inactive deployment
Preview — a2b132dd Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant