Add chained invoke to the local runner - #731
Merged
Merged
Conversation
yaythomas
force-pushed
the
feat/chained-invoke
branch
4 times, most recently
from
September 17, 2026 00:39
a48ee11 to
3d17858
Compare
yaythomas
marked this pull request as ready for review
September 17, 2026 01:02
This comment has been minimized.
This comment has been minimized.
This was referenced Sep 17, 2026
yaythomas
force-pushed
the
feat/chained-invoke
branch
from
September 17, 2026 07:37
3d17858 to
3b15943
Compare
This comment has been minimized.
This comment has been minimized.
yaythomas
force-pushed
the
feat/chained-invoke
branch
from
September 17, 2026 08:04
3b15943 to
7f5a127
Compare
This comment has been minimized.
This comment has been minimized.
yaythomas
force-pushed
the
feat/chained-invoke
branch
from
September 17, 2026 08:32
7f5a127 to
9c943ce
Compare
yaythomas
force-pushed
the
feat/chained-invoke
branch
from
September 17, 2026 08:43
9c943ce to
75207a5
Compare
This comment has been minimized.
This comment has been minimized.
The local runner rejected CHAINED_INVOKE checkpoints, so a durable
function that calls context.invoke could not be tested locally.
- A START checkpoint validates the options as the service does (Lambda
function name grammar, same account and region, TenantId constraint,
input at most 1 MiB) and resolves the target before anything runs: a
target the runner cannot resolve comes back FAILED in the checkpoint
response, so the handler raises without suspending, as at the
service; otherwise the operation is recorded STARTED and dispatched.
The child's terminal transition completes the parent's operation
and re-invokes the parent; no thread waits on a child.
- A qualified target (child:prod) is invoked as written; its
registration or configuration is the one under that key, else under
the bare name. A PENDING response is valid when an operation
completed after the invocation's input was built.
- Outcomes follow the service: SUCCEEDED, FAILED with the child's error,
STOPPED with the stop error, or TIMED_OUT with ChainedInvoke.Timeout
and "CHAINED_INVOKE timed out after N seconds". A plain target is
bounded by the invocation timeout; a child result over 1 MiB fails.
A target that cannot be invoked fails with the Lambda API error code.
- In-process runner: targets are registered with
register_durable_function or register_function.
- Web runner: --function-configs gives the functions a durable
function may invoke, as a JSON object mapping each name to its
configuration in the shape of the Lambda function configuration
({"ProcessPayment": {"DurableConfig": {...}}, "LookupPrice": {}}), or
file://<path>. A durable target runs as a child execution the runner
drives; a plain target is one synchronous Invoke. Without the option
every chained invoke fails with a message naming it.
- Every handler invocation carries the header X-Dex-Handler-Invoke:
true, so a Lambda-compatible endpoint runs the handler once instead
of starting an execution as it does for a caller's Invoke of a
durable function, and the execution's TenantId, so handlers see
their tenant.
- Client read timeouts follow --invocation-timeout plus 60 s.
- Both runners emulate one region (default us-west-2), fixed at
startup: executions record it, targets in another region are
rejected, and PUT /lambda-endpoint moves the endpoint only. Lambda
contexts report the run's account and tenant, and the target's
function name, version and ARN as Lambda fills them; run() takes
tenant_id.
- GetDurableExecution and the list report the function ARN qualified
with the executed version, and Version, from the execution's own
region and account. A numeric qualifier and $LATEST.PUBLISHED are
reported as given; anything else runs $LATEST, as the runner keeps
no versions or aliases.
- History records ChainedInvokeStarted, Succeeded, Failed, TimedOut,
and Stopped, redacted unless IncludeExecutionData is set. The started
event omits TenantId, as the service's history does. A target the
runner could not resolve is recorded without its input, as at the
service.
- UpdatedOperationIds lists the operations changed since the state the
handler last observed, so a target that completes during an
invocation is reported on the next one.
- The child-to-parent link is rebuilt from the stored parent ARN and
child map when the in-memory link is gone, so a child that completes
after a runner restart still completes its parent's operation.
- Handler invocations and dispatch run on bounded pools of daemon
threads. Python cannot interrupt a blocked Invoke, so closing the
runner leaves it to its read timeout without holding the process;
a result that lands after shutdown is dropped.
Verified with the unit and e2e suites at 95% coverage and the JS
examples conformance suite through the web runner.
Closes #436
Closes #735
yaythomas
force-pushed
the
feat/chained-invoke
branch
from
September 17, 2026 09:14
75207a5 to
016ccb3
Compare
Contributor
Codex AI reviewFound five actionable issues: two crash-recovery gaps can permanently strand chained invokes, plus shutdown, timeout-isolation, and observer compatibility regressions. Reviewed commit |
wangyb-A
approved these changes
Sep 17, 2026
wangyb-A
left a comment
Contributor
There was a problem hiding this comment.
LGTM. The design mirrors the service closely. The threads with Codex are well argued.
One optional follow-up, not blocking: the while self._child_starts_in_flight > 0 loop in Executor.shutdown() has no overall bound. It's fine when the parent lane is healthy, but a lane stuck on a store write would hang shutdown. A total timeout plus a warning log would make that failure visible without changing semantics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #436
Closes #735
The local runner supports
context.invoke.In-process
register_durable_functionregisters a durable function; the runnerruns it as a child execution.
register_functionregisters a plainLambda function; the runner runs it as one invocation.
Web runner
--function-configsgives the functions a durable function may invoke:a JSON object mapping each function name to its configuration, in the
shape of the Lambda function configuration, or
file://<path>to afile holding it. A durable function carries a
DurableConfig; a plainfunction has none.
{ "process-payment": {"DurableConfig": {"ExecutionTimeout": 60, "RetentionPeriodInDays": 7}}, "lookup-price": {} }The runner invokes targets at the Lambda endpoint and marks every
handler invocation with the header
X-Dex-Handler-Invoke: true, so aLambda-compatible endpoint runs the handler once instead of starting an
execution.
Behavior
follow the service. A target the runner cannot resolve fails in the
checkpoint response, so the handler raises without suspending.
process-payment:prod) resolves to theregistration under that key, else under the bare name.
us-west-2).PUT /lambda-endpointmoves the endpoint only andrejects another
RegionName. Whether that route should exist is Testing: investigate removing PUT /lambda-endpoint from the web runner #736.GetDurableExecutionandListDurableExecutionsreport the functionARN qualified with the executed version, and
Version, from theexecution's own region. A numeric qualifier is the version; anything
else runs
$LATEST.parent's operation: the link is rebuilt from the stored parent ARN.
threads. Closing the runner during a blocked Invoke no longer holds
the process; a result that lands after shutdown is dropped.
run()takestenant_id.Behavior changes
tenant_idwas the synthetic valuetest-tenant-789. It is nowNoneunlessrun(tenant_id=...)or thechained invoke's
TenantIdgives one, as in Lambda. A handler thatread the synthetic value will see
None.ChainedInvokeStartedhistory event omitsTenantId, as theservice's history does.
Testing
Unit and e2e suites at 96% coverage; mypy and ruff clean; the JS
examples conformance suite against the web runner, all six
chained-invoke suites passing. A subprocess test proves the process
exits while an Invoke is blocked.
sam
No sam change is needed to release this. sam gains chained invokes once
it passes a reachable Lambda endpoint, generates the function
configurations from the template, and runs handler invocations directly
when the header is present.