Skip to content

Add chained invoke to the local runner - #731

Merged
yaythomas merged 1 commit into
mainfrom
feat/chained-invoke
Sep 17, 2026
Merged

yaythomas merged 1 commit into
mainfrom
feat/chained-invoke

Conversation

@yaythomas

@yaythomas yaythomas commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Closes #436
Closes #735

The local runner supports context.invoke.

In-process

with DurableFunctionTestRunner(handler=place_order) as runner:
    runner.register_durable_function("process-payment", process_payment)
    runner.register_function("lookup-price", lookup_price)
    result = runner.run(input='{"sku": "book-1"}')

register_durable_function registers a durable function; the runner
runs it as a child execution. register_function registers a plain
Lambda function; the runner runs it as one invocation.

Web runner

--function-configs gives the functions a durable function may invoke:
a JSON object mapping each function name to its configuration, in the
shape of the Lambda function configuration, or file://<path> to a
file holding it. A durable function carries a DurableConfig; a plain
function has none.

dex-local-runner start-server --lambda-endpoint http://127.0.0.1:3001 \
    --function-configs file://function-configs.json
{
  "process-payment": {"DurableConfig": {"ExecutionTimeout": 60, "RetentionPeriodInDays": 7}},
  "lookup-price": {}
}

The runner invokes targets at the Lambda endpoint and marks every
handler invocation with the header X-Dex-Handler-Invoke: true, so a
Lambda-compatible endpoint runs the handler once instead of starting an
execution.

Behavior

  • Validation, outcomes, payload limits, error codes and history events
    follow the service. A target the runner cannot resolve fails in the
    checkpoint response, so the handler raises without suspending.
  • A qualified target (process-payment:prod) resolves to the
    registration under that key, else under the bare name.
  • The runner emulates one region, fixed at startup (default
    us-west-2). PUT /lambda-endpoint moves the endpoint only and
    rejects another RegionName. Whether that route should exist is Testing: investigate removing PUT /lambda-endpoint from the web runner #736.
  • GetDurableExecution and ListDurableExecutions report the function
    ARN qualified with the executed version, and Version, from the
    execution's own region. A numeric qualifier is the version; anything
    else runs $LATEST.
  • A child that completes after a runner restart still completes its
    parent's operation: the link is rebuilt from the stored parent ARN.
  • Handler invocations and dispatch run on bounded pools of daemon
    threads. Closing the runner during a blocked Invoke no longer holds
    the process; a result that lands after shutdown is dropped.
  • run() takes tenant_id.

Behavior changes

  • The in-process Lambda context's tenant_id was the synthetic value
    test-tenant-789. It is now None unless run(tenant_id=...) or the
    chained invoke's TenantId gives one, as in Lambda. A handler that
    read the synthetic value will see None.
  • The ChainedInvokeStarted history event omits TenantId, as the
    service's history does.

Testing

Unit and e2e suites at 96% coverage; mypy and ruff clean; the JS
examples conformance suite against the web runner, all six
chained-invoke suites passing. A subprocess test proves the process
exits while an Invoke is blocked.

sam

No sam change is needed to release this. sam gains chained invokes once
it passes a reachable Lambda endpoint, generates the function
configurations from the template, and runs handler invocations directly
when the header is present.

@yaythomas
yaythomas force-pushed the feat/chained-invoke branch 4 times, most recently from a48ee11 to 3d17858 Compare September 17, 2026 00:39
@yaythomas
yaythomas marked this pull request as ready for review September 17, 2026 01:02
@yaythomas
yaythomas deployed to ai-pr-review-runtime September 17, 2026 01:02 — with GitHub Actions Active
@yaythomas
yaythomas deployed to ai-pr-review-runtime September 17, 2026 01:02 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@yaythomas
yaythomas deployed to ai-pr-review-runtime September 17, 2026 08:05 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@yaythomas
yaythomas deployed to ai-pr-review-runtime September 17, 2026 08:32 — with GitHub Actions Active
@yaythomas
yaythomas deployed to ai-pr-review-runtime September 17, 2026 08:43 — with GitHub Actions Active
@github-actions

This comment has been minimized.

The local runner rejected CHAINED_INVOKE checkpoints, so a durable
function that calls context.invoke could not be tested locally.

- A START checkpoint validates the options as the service does (Lambda
  function name grammar, same account and region, TenantId constraint,
  input at most 1 MiB) and resolves the target before anything runs: a
  target the runner cannot resolve comes back FAILED in the checkpoint
  response, so the handler raises without suspending, as at the
  service; otherwise the operation is recorded STARTED and dispatched.
  The child's terminal transition completes the parent's operation
  and re-invokes the parent; no thread waits on a child.
- A qualified target (child:prod) is invoked as written; its
  registration or configuration is the one under that key, else under
  the bare name. A PENDING response is valid when an operation
  completed after the invocation's input was built.
- Outcomes follow the service: SUCCEEDED, FAILED with the child's error,
  STOPPED with the stop error, or TIMED_OUT with ChainedInvoke.Timeout
  and "CHAINED_INVOKE timed out after N seconds". A plain target is
  bounded by the invocation timeout; a child result over 1 MiB fails.
  A target that cannot be invoked fails with the Lambda API error code.
- In-process runner: targets are registered with
  register_durable_function or register_function.
- Web runner: --function-configs gives the functions a durable
  function may invoke, as a JSON object mapping each name to its
  configuration in the shape of the Lambda function configuration
  ({"ProcessPayment": {"DurableConfig": {...}}, "LookupPrice": {}}), or
  file://<path>. A durable target runs as a child execution the runner
  drives; a plain target is one synchronous Invoke. Without the option
  every chained invoke fails with a message naming it.
- Every handler invocation carries the header X-Dex-Handler-Invoke:
  true, so a Lambda-compatible endpoint runs the handler once instead
  of starting an execution as it does for a caller's Invoke of a
  durable function, and the execution's TenantId, so handlers see
  their tenant.
- Client read timeouts follow --invocation-timeout plus 60 s.
- Both runners emulate one region (default us-west-2), fixed at
  startup: executions record it, targets in another region are
  rejected, and PUT /lambda-endpoint moves the endpoint only. Lambda
  contexts report the run's account and tenant, and the target's
  function name, version and ARN as Lambda fills them; run() takes
  tenant_id.
- GetDurableExecution and the list report the function ARN qualified
  with the executed version, and Version, from the execution's own
  region and account. A numeric qualifier and $LATEST.PUBLISHED are
  reported as given; anything else runs $LATEST, as the runner keeps
  no versions or aliases.
- History records ChainedInvokeStarted, Succeeded, Failed, TimedOut,
  and Stopped, redacted unless IncludeExecutionData is set. The started
  event omits TenantId, as the service's history does. A target the
  runner could not resolve is recorded without its input, as at the
  service.
- UpdatedOperationIds lists the operations changed since the state the
  handler last observed, so a target that completes during an
  invocation is reported on the next one.
- The child-to-parent link is rebuilt from the stored parent ARN and
  child map when the in-memory link is gone, so a child that completes
  after a runner restart still completes its parent's operation.
- Handler invocations and dispatch run on bounded pools of daemon
  threads. Python cannot interrupt a blocked Invoke, so closing the
  runner leaves it to its read timeout without holding the process;
  a result that lands after shutdown is dropped.

Verified with the unit and e2e suites at 95% coverage and the JS
examples conformance suite through the web runner.

Closes #436
Closes #735
@yaythomas
yaythomas deployed to ai-pr-review-runtime September 17, 2026 09:14 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

Found five actionable issues: two crash-recovery gaps can permanently strand chained invokes, plus shutdown, timeout-isolation, and observer compatibility regressions.

Reviewed commit 016ccb33c00e7b8840da85de747444f9ec131aec. Workflow run

@wangyb-A wangyb-A left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The design mirrors the service closely. The threads with Codex are well argued.

One optional follow-up, not blocking: the while self._child_starts_in_flight > 0 loop in Executor.shutdown() has no overall bound. It's fine when the parent lane is healthy, but a lane stuck on a store write would hang shutdown. A total timeout plus a warning log would make that failure visible without changing semantics.

@yaythomas
yaythomas merged commit d3230f3 into main Sep 17, 2026
102 of 109 checks passed
@yaythomas
yaythomas deleted the feat/chained-invoke branch September 17, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Testing: GetDurableExecution ignores the function qualifier in FunctionArn and Version [Testing] [feature]: chained invoke

2 participants