From adb57764dd7b0d927e52f756ecb9bca4eb0f5a91 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Wed, 16 Sep 2026 17:15:10 -0700 Subject: [PATCH 01/19] fix(insight): key export scheduling by execution ARN, serialize flush MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One plugin instance serves every execution its environment hosts, and Lambda Managed Instances makes concurrent executions in one environment routine. ExportScheduler held one pending record for the whole plugin and overwrote it regardless of which execution the record belonged to. Measured before the change, 20 trials per case: 2 concurrent executions lost a terminal record in 20 of 20 trials, 10 concurrent lost 5 to 8 per trial, and with 5 concurrent executions and a 200 ms exporter 2 of 5 records were exported while drain() returned successfully anyway. Pending records are now keyed by execution ARN in insertion order, each execution has its own completion signal, and drain(executionArn) waits for that signal rather than for the whole queue. Coalescing happens only within one execution. One pump is unchanged, so no exporter sees two records exported at once. flush() is now served by that same pump, between records, so an exporter never sees flush() overlap export() — including an export belonging to a different execution in the same environment. Requests queued together share one flush, and the pump exports the records an invocation is waiting on before spending a flush fan-out, which keeps a burst of simultaneous invocation ends from paying N times for one flush (measured: 8 ends with a 60 ms flush, 514 ms and 8 flushes before, 60 to 138 ms and 1 to 2 flushes after). Also fixed, each found while reviewing the change above: - Orphan detection inferred "nothing outstanding" from the pending map alone, which is also true of a record already taken and being exported, so an exiting pump could complete another execution's drain signal mid-export and that invocation returned before its record was delivered. - drainAll() gave up as soon as an already-seen execution queued new work, silently weakening every assertion made after it returned. - The drain failure path left a record queued for an unrelated execution's pump to export later, out of order and after the invocation had returned. - flush() and drain() called from the pump thread parked forever, because the only thread able to serve the wait was the one waiting. Both now report an IllegalStateException through the failure handler and return. - A null execution ARN threw NullPointerException out of onInvocationEnd into the SDK, which the hook's own javadoc forbids. - InsightExporter.flush() had no documented contract. It now states the cadence, the exclusivity guarantee, that a flush may cover other executions' records, and how failures are handled. No public type changed: ExportScheduler stays package-private and ExecutionState stays a private nested class. Record fields, emit modes, sampling, truncation, per-record exporter fan-out and the default exporter are unchanged. --- .../durable/insight/ExportScheduler.java | 569 ++++++++++++++++-- .../durable/insight/InsightExporter.java | 24 +- .../durable/insight/WorkflowInsight.java | 62 +- .../ConcurrentExecutionsExportTest.java | 392 ++++++++++++ .../ExportSchedulerFlushCoalescingTest.java | 308 ++++++++++ ...ExportSchedulerFlushSerializationTest.java | 429 +++++++++++++ .../ExportSchedulerReentrantFlushTest.java | 129 ++++ .../durable/insight/ExportSchedulerTest.java | 75 +-- .../PluginThrowableContainmentTest.java | 29 + .../WorkflowInsightFlushCadenceTest.java | 222 +++++++ 10 files changed, 2148 insertions(+), 91 deletions(-) create mode 100644 insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ConcurrentExecutionsExportTest.java create mode 100644 insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushCoalescingTest.java create mode 100644 insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushSerializationTest.java create mode 100644 insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerReentrantFlushTest.java create mode 100644 insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightFlushCadenceTest.java diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java index e161df5b3..69ba94b21 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java @@ -2,32 +2,63 @@ // SPDX-License-Identifier: Apache-2.0 package software.amazon.lambda.durable.insight; +import java.util.ArrayDeque; import java.util.ArrayList; +import java.util.Deque; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Iterator; +import java.util.LinkedHashMap; import java.util.List; +import java.util.Map; +import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.concurrent.Executor; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; import java.util.function.BiConsumer; import java.util.function.Consumer; /** - * Serializes record exports so that, at most, one export runs at a time. + * Serializes record exports so that, at most, one export runs at a time, while keeping the records of concurrently + * running executions independent. * - *

Each {@link WorkflowInsightRecord} is a complete snapshot of the execution, so a newer record fully supersedes any - * record still waiting to be exported. While an export is in flight, additional updates are coalesced into a single - * "pending" slot — intermediate records are dropped because the latest one already contains all of their information. - * This prevents overlapping {@code export()} calls when updates arrive faster than the exporters can keep up, and it - * keeps exporter I/O off the SDK threads that deliver plugin hooks. + *

One plugin instance — and therefore one scheduler — serves the whole execution environment, and an environment can + * host several durable executions at the same time (Lambda Managed Instances makes that routine). So the pending work + * is keyed by execution ARN: each execution has its own latest-record slot, and coalescing happens only within + * one execution. * - *

Exports are otherwise fire-and-forget; {@link #drain()} is called before the invocation returns to guarantee the - * final record is delivered. + *

Each {@link WorkflowInsightRecord} is a complete snapshot of its execution, so a newer record for the same + * execution fully supersedes one still waiting to be exported. While an export is in flight, additional updates for + * that execution are coalesced into its slot — intermediate records are dropped because the latest one already contains + * all of their information. A record for a different execution never displaces another execution's record. + * + *

A single pump exports the queued records one at a time, in the order the executions first queued work, so + * exporters still never see two exports at once and each record keeps its per-exporter fan-out. {@link #flush()} + * requests are served by that same pump, between records, so an exporter never sees a {@code flush()} overlap an + * {@code export()} either. Requests are served as a batch — the cadence is at most one flush per requesting invocation + * end, not exactly one — and a flush is preceded by the queued records a {@link #drain(String)} is waiting for, so a + * burst of invocation ends is covered by one flush rather than one each. A request made while a flush runs waits for + * the next turn. Exports are otherwise fire-and-forget; {@link #drain(String)} is called before an invocation returns + * and waits for that execution's own latest record to reach every exporter. + * + *

Because there is one pump, that wait can also cover records other executions had already queued ahead of this one: + * a drain is not isolated from the queue's head-of-line cost. What the per-execution keying guarantees is that another + * execution's record can never displace this one — records coalesce only within their own execution, and a + * drain cannot return until this execution's own latest record has reached every exporter. */ final class ExportScheduler { private static final AtomicInteger THREAD_NUMBER = new AtomicInteger(); + /** + * Upper bound on the passes {@link #drainAll()} makes over the outstanding executions. Only reached if new work + * keeps arriving for as long as the drain runs; a normal drain settles in two passes. + */ + private static final int MAX_DRAIN_ALL_PASSES = 1_000; + /** Shared for the process lifetime; idle daemon workers are reclaimed, so nothing keeps the runtime alive. */ private static final ExecutorService WORKERS = Executors.newCachedThreadPool(runnable -> { var thread = new Thread(runnable, "workflow-insight-export-" + THREAD_NUMBER.incrementAndGet()); @@ -43,8 +74,65 @@ final class ExportScheduler { /** Completes when the current pump finishes; {@code null} while idle. Guarded by {@code this}. */ private CompletableFuture inFlight; - /** The latest record not yet picked up by the pump. Guarded by {@code this}. */ - private WorkflowInsightRecord pending; + /** + * The thread serving the pump right now, or {@code null} while no pump is running. Deliberately not + * guarded by {@code this}: it is read by {@link #flush()} and {@link #drain(String)} before they touch anything + * else, and a lock acquisition there would put the pump's own monitor on the path of every invocation end. + * + *

Written only by the thread that enters {@link #pump} — a worker, or the caller's own thread on the + * rejected-worker fallback, which is a case where the calling thread genuinely is the pump — and cleared + * by that same thread on the way out, only if it is still the recorded one. A compare-and-set on the way out rather + * than a blind clear: if some anomaly ever did leave two pumps running, the one that finishes first must not clear + * the other, and must not leave a stale thread behind that a later, legitimate {@code flush()} from that same + * thread would be mistaken for. + */ + private final AtomicReference pumpThread = new AtomicReference<>(); + + /** + * The latest record per execution ARN that the pump has not picked up yet, in the order the executions first queued + * work. Guarded by {@code this}. + */ + private final Map pending = new LinkedHashMap<>(); + + /** + * Per-execution completion signal, present while that execution has work outstanding (pending or being exported + * right now) and completed once its latest record has been handed to every exporter. Guarded by {@code this}. + */ + private final Map> settled = new HashMap<>(); + + /** + * The execution ARNs whose record a pump has taken out of {@link #pending} and is handing to the exporters right + * now. Guarded by {@code this}. + * + *

Without this, "no record in {@code pending}" is indistinguishable from "record already taken and mid-export", + * and a pump running only its own {@code finally} would treat the second case as orphaned work and complete that + * execution's drain signal while the record was still inside the exporters — exactly the early return + * {@link #drain(String)} exists to prevent. + */ + private final Set exporting = new HashSet<>(); + + /** + * One entry per outstanding {@link #flush()} request, in request order, completed when a {@code flush()} that + * started after that request was enqueued has reached every exporter. Guarded by {@code this}. + * + *

A queue of requests rather than a single flag: the pump takes the requests that are queued when its turn + * begins and satisfies all of them with one flush, so concurrent invocation ends share a flush; a request enqueued + * while that flush runs stays in the queue for the next turn, because a flush already in progress cannot be shown + * to have seen the new requester's records. + */ + private final Deque> flushRequests = new ArrayDeque<>(); + + /** + * How many {@link #drain(String)} calls are waiting for each execution right now. Guarded by {@code this}; an entry + * is removed when its last waiter returns. + * + *

A record with a waiter is the last record of an invocation that cannot return until it is exported, so the + * pump exports those records before it spends a flush fan-out. Without that, a burst of invocation ends is + * serialized by the pump itself — one record exported per turn, a flush in between — and each end ends up paying + * for its own flush, which is what coalescing is meant to prevent. Records nobody is waiting for (an + * {@code ON_CHANGE} stream, say) are not front-loaded, so they cannot push a flush back either. + */ + private final Map drainWaiters = new HashMap<>(); ExportScheduler( List exporters, @@ -65,13 +153,15 @@ final class ExportScheduler { } /** - * Queues the latest record for export. If an export is already running, the record is held in the pending slot - * (replacing any earlier pending record) and exported once the in-flight export completes. + * Queues the latest record of one execution for export. If an export is already running, the record is held in that + * execution's own slot (replacing only an earlier record of the same execution) and exported once the pump + * reaches it. */ - void schedule(WorkflowInsightRecord record) { + void schedule(String executionArn, WorkflowInsightRecord record) { CompletableFuture handle; synchronized (this) { - pending = record; + pending.put(executionArn, record); + settled.computeIfAbsent(executionArn, arn -> new CompletableFuture<>()); if (inFlight != null) { return; } @@ -95,20 +185,55 @@ void schedule(WorkflowInsightRecord record) { } /** - * Waits for any in-flight and pending exports to complete. Safe to call when idle. Used before the invocation - * returns to guarantee the final record is delivered. + * Waits until the latest record of one execution has been handed to every exporter. Safe to call when that + * execution has nothing outstanding. Used before the invocation returns to guarantee the final record is delivered. + * + *

The wait is for this execution's own latest record. Another execution's record can never displace it, so this + * always returns having delivered this execution's latest snapshot; but since one pump exports serially, the wait + * can also cover records other executions had already queued ahead of it. + * + *

While this waits, the execution is registered in {@link #drainWaiters}: it tells the pump that this record + * gates an invocation return, so the pump exports it before spending a flush fan-out. See + * {@link #exportRecordsADrainIsWaitingFor}. + * + *

Called from the pump thread itself, the wait is refused and reported instead of made: see + * {@link #refuseWaitFromThePumpThread}. */ - void drain() { + void drain(String executionArn) { + // Re-entered from the pump: this thread is the one that would settle the signal it is about to wait for. Refuse + // and return; the record stays queued and this same pump exports it when it resumes its loop. + if (refuseWaitFromThePumpThread("drain(executionArn)")) { + return; + } + synchronized (this) { + if (!settled.containsKey(executionArn)) { + return; + } + drainWaiters.merge(executionArn, 1, Integer::sum); + } + try { + drainUntilSettled(executionArn); + } finally { + synchronized (this) { + drainWaiters.compute( + executionArn, (arn, waiting) -> waiting == null || waiting <= 1 ? null : waiting - 1); + } + } + } + + private void drainUntilSettled(String executionArn) { while (true) { + CompletableFuture signal; CompletableFuture handle; boolean runInline = false; synchronized (this) { + signal = settled.get(executionArn); + if (signal == null) { + return; + } handle = inFlight; if (handle == null) { - if (pending == null) { - return; - } - // A record is pending with no pump running (a worker could not be started): export it here. + // A record is outstanding with no pump running (a worker could not be started): export it here. handle = new CompletableFuture<>(); inFlight = handle; runInline = true; @@ -116,44 +241,382 @@ void drain() { } if (runInline) { pump(handle); - } else { - handle.join(); + continue; + } + // Wake either when this execution's record has been exported or when the current pump ends — the pump may + // have ended without taking this record (a rejected worker), in which case the loop re-evaluates and + // exports it inline. + try { + CompletableFuture.anyOf(signal, handle).join(); + } catch (Throwable t) { + // Never spin on an unexpected wait failure, and never let it escape into the execution. Abandon this + // execution's outstanding record instead of leaving it queued: WORKERS is a static, process-wide pool, + // so a record left in pending here would be exported later by some unrelated execution's pump — out of + // order, and after this invocation has already returned. Completing the signal also releases any other + // drain waiting on the same execution rather than stranding it behind work nobody will do. + abandon(executionArn); + reportFailure(t); + return; } } } + /** + * Flushes every exporter, serialized against exports: the request is queued and served by the pump between records, + * so an exporter never sees {@code flush()} overlap {@code export()} — not even an export belonging to a different + * execution running in the same environment. Returns once a flush that started after this request was enqueued has + * reached every exporter. + * + *

Requests are coalesced: the pump takes every request queued at the start of its turn, exports any queued + * record a {@link #drain(String)} is still waiting for, re-takes the requests those ends make as they are released, + * and satisfies them all with one flush. Invocation ends that overlap therefore share a flush instead of paying for + * one fan-out each. That is sound because a caller drains its own record before asking, so a flush that + * starts after the request was enqueued has that record in the exporter's buffer. A request enqueued while + * a flush is already running is never satisfied by it — it waits for the next turn. + * + *

A queue that never runs dry cannot starve a request either: the pump alternates one record and one batch of + * requests, so a flush waits at most one export fan-out. + * + *

Called from the pump thread itself — which only something the pump invokes synchronously can do — the request + * is refused and reported instead of made: see {@link #refuseWaitFromThePumpThread}. + */ + void flush() { + // Re-entered from the pump: this thread is the only one that could serve the request it is about to make, so it + // must not make it. Refuse and return rather than enqueue a request nobody can serve. + if (refuseWaitFromThePumpThread("flush()")) { + return; + } + CompletableFuture request = new CompletableFuture<>(); + synchronized (this) { + flushRequests.add(request); + } + while (true) { + CompletableFuture handle; + boolean startPump = false; + synchronized (this) { + if (request.isDone()) { + return; + } + handle = inFlight; + if (handle == null) { + if (!flushRequests.contains(request)) { + // Liveness backstop: a pump took this request and unwound without serving it, which its + // `finally` is there to prevent. The request is no longer in the queue, so no future pump can + // find it — release the caller here instead of spinning up pumps that have nothing to do. + break; + } + // No pump is running (a worker could not be started earlier, or the pump went idle between the add + // above and this check): start one. + handle = new CompletableFuture<>(); + inFlight = handle; + startPump = true; + } + } + if (startPump) { + CompletableFuture started = handle; + try { + executor.execute(() -> pump(started)); + } catch (Throwable t) { + // No worker could be started. Serve the request on the calling thread, exactly as drain() exports a + // pending record inline: this pump owns `inFlight`, so no export can run beside it. + reportFailure(t); + pump(started); + continue; + } + } + // Wake either when this request has been served or when the current pump ends — a pump can end without + // serving it (a rejected worker), in which case the loop starts another one. + try { + CompletableFuture.anyOf(request, handle).join(); + } catch (Throwable t) { + // Never spin on an unexpected wait failure, and never let it escape into the execution. Drop the + // request rather than leaving it queued for some later, unrelated invocation's pump to serve. + synchronized (this) { + flushRequests.remove(request); + } + reportFailure(t); + break; + } + } + request.complete(null); + } + + /** + * Waits for every execution's outstanding record. Test seam for a plugin-wide drain; the per-invocation path uses + * {@link #drain(String)}. + * + *

Bounded by the number of passes, not by the set of ARNs seen: an execution that queues new work after it was + * already drained must still be waited for (dropping it would silently weaken every assertion made after this + * returns), while a producer that never stops cannot keep this spinning forever. + */ + void drainAll() { + // Every pass below is a drain(), and each one would be refused; without this the loop spends all of its passes + // reporting the same refusal. + if (refuseWaitFromThePumpThread("drainAll()")) { + return; + } + for (int pass = 0; pass < MAX_DRAIN_ALL_PASSES; pass++) { + List outstanding; + synchronized (this) { + if (settled.isEmpty()) { + return; + } + outstanding = new ArrayList<>(settled.keySet()); + } + for (String executionArn : outstanding) { + drain(executionArn); + } + } + } + + /** Gives up one execution's outstanding work: drops its queued record and releases every drain waiting on it. */ + private void abandon(String executionArn) { + CompletableFuture signal; + synchronized (this) { + pending.remove(executionArn); + signal = settled.remove(executionArn); + } + if (signal != null) { + signal.complete(null); + } + } + private void pump(CompletableFuture handle) { + // Recorded for as long as this thread serves the pump — a worker, or a caller pumping inline after a rejected + // worker — so that a flush() or drain() re-entered from anything the fan-out calls synchronously can tell that + // it is asking itself. One atomic write per pump, and no lock: see the field. + Thread self = Thread.currentThread(); + pumpThread.set(self); + // The ARN this pump has taken and not settled yet. Only this pump may release it, so an abnormal unwind cannot + // strand a drain, and no other pump can mistake it for orphaned work. + String taken = null; + // Likewise for the flush requests this pump has taken out of the queue and not completed yet. + List> takenFlushes = null; try { - // Drain the pending slot until no newer record has arrived. Taking the record and returning to idle both - // happen under the lock, so an update scheduled at any point is either exported by this pump or starts - // the next one — never lost. + // One record, then every flush request queued at that moment, alternating. Taking the record and returning + // to idle both happen under the lock, so a record scheduled at any point is either exported by this pump or + // starts the next one — never lost, and never displaced by another execution's record. A flush therefore + // waits at most one fan-out (it cannot be starved by a queue that never runs dry) and still never overlaps + // an export, because this loop runs them one after the other. + // + // A loop, deliberately, not a pump that re-enters itself to pick up the next item: written that way, one + // frame per queued item accumulates until the stack overflows, and the rest of the queue is dropped. while (true) { - WorkflowInsightRecord record; + String executionArn = null; + WorkflowInsightRecord record = null; synchronized (this) { - record = pending; - pending = null; - if (record == null) { - inFlight = null; + Iterator> queued = + pending.entrySet().iterator(); + if (!queued.hasNext() && flushRequests.isEmpty()) { + if (inFlight == handle) { + inFlight = null; + } return; } + if (queued.hasNext()) { + Map.Entry next = queued.next(); + queued.remove(); + executionArn = next.getKey(); + record = next.getValue(); + // Marked under the same lock that removes the record, so the execution is never momentarily + // invisible to another pump's orphan check. + exporting.add(executionArn); + } + } + if (executionArn != null) { + taken = executionArn; + try { + exportToAll(record); + } finally { + signalSettled(executionArn); + taken = null; + } + } + // Taken only now that the fan-out above has settled, and taken as a batch: every request queued at this + // instant is satisfied by the single flush below, so invocation ends that ask together cost one flush + // rather than one each. Sound because each requester drained its own record before asking, so a flush + // that starts after the request was enqueued already has that record in the exporter's buffer. + // + // Emptying the queue here — rather than after the flush — is what keeps a request that arrives while + // that flush runs out of this batch: it lands in the now-empty queue and is served by the next turn, + // never credited to a flush that was already in progress when it was made. + synchronized (this) { + if (!flushRequests.isEmpty()) { + takenFlushes = new ArrayList<>(flushRequests); + flushRequests.clear(); + } + } + if (takenFlushes != null) { + // Before spending the fan-out: export the queued records other invocations are still waiting on. + // Those ends cannot have asked for their flush yet — they are inside drain() — so without this the + // pump staggers them one record per turn, with a whole flush in between, and each pays for its own + // flush however aggressively the queue is coalesced. + exportRecordsADrainIsWaitingFor(); + // Re-take: the ends released above ask for their flush now, and one flush covers all of them since + // it starts after every one of those records reached the exporters. + synchronized (this) { + if (!flushRequests.isEmpty()) { + takenFlushes.addAll(flushRequests); + flushRequests.clear(); + } + } + try { + flushEveryExporter(); + } finally { + // In `finally`: a Throwable from a customer's flush() — an Error, not just an exception — must + // never leave the invocations waiting on these requests parked forever. + completeAll(takenFlushes); + takenFlushes = null; + } } - exportToAll(record); } } finally { + // Before anything else, and before the handle below: whoever waits on these requests must be released even + // if this pump is unwinding for a reason none of the guards above anticipated. + if (takenFlushes != null) { + completeAll(takenFlushes); + } + List> orphaned; synchronized (this) { if (inFlight == handle) { inFlight = null; } + if (taken != null) { + // Unwinding with a record still marked as being exported: this pump will never settle it, so + // release it here and let the orphan sweep below complete its drain. + exporting.remove(taken); + } + orphaned = takeSignalsWithNothingOutstanding(); + } + // Defensive backstop: an execution with nothing outstanding — no record in the queue and none inside the + // exporters — whose signal nevertheless survived must not leave a drain() waiting forever. No ordinary path + // is known to produce that; it covers the unwinds that are hard to enumerate exhaustively rather than one + // specific failure. + for (CompletableFuture signal : orphaned) { + signal.complete(null); } handle.complete(null); + // Last, because everything above is still this pump's work and a flush() re-entered from any of it would + // still have nobody to serve it. Conditional: a pump that recorded itself since must not be cleared here. + pumpThread.compareAndSet(self, null); } } + /** + * Exports the queued records that a {@link #drain(String)} is waiting for, one at a time, and returns once they + * have all reached the exporters. Called by the pump immediately before a flush. + * + *

Those records are the last records of invocations that cannot return until they are exported, and their ends + * cannot ask for their flush until then. Exporting them first is therefore what lets one flush serve a whole burst + * of invocation ends: without it the pump interleaves one record and one flush fan-out, and each end pays for a + * flush of its own even though every request is coalesced. + * + *

Bounded by the snapshot taken under the lock, so a producer that keeps scheduling for an execution someone is + * draining cannot hold a flush back indefinitely — and records nobody waits for are not exported here at all, so a + * stream of {@code ON_CHANGE} snapshots still cannot starve a flush: it waits at most one ordinary fan-out plus + * this pass over the executions whose invocation return is already blocked on their own record. + */ + private void exportRecordsADrainIsWaitingFor() { + List awaited; + synchronized (this) { + if (pending.isEmpty() || drainWaiters.isEmpty()) { + return; + } + awaited = new ArrayList<>(); + for (String executionArn : pending.keySet()) { + if (drainWaiters.containsKey(executionArn)) { + awaited.add(executionArn); + } + } + } + for (String executionArn : awaited) { + WorkflowInsightRecord record; + synchronized (this) { + record = pending.remove(executionArn); + if (record != null) { + // Marked under the same lock that removes the record, exactly as the pump's own record step does, + // so the execution is never momentarily invisible to another pump's orphan check. + exporting.add(executionArn); + } + } + if (record == null) { + continue; + } + try { + exportToAll(record); + } finally { + try { + signalSettled(executionArn); + } catch (Throwable t) { + // Nothing here is expected to throw, but a record left marked as being exported would strand the + // drain that is waiting for it, so release it rather than leave the invocation parked. + synchronized (this) { + exporting.remove(executionArn); + } + reportFailure(t); + } + } + } + } + + /** Completes every taken flush request; one that cannot be completed must not stop the rest from being. */ + private void completeAll(List> requests) { + for (CompletableFuture request : requests) { + try { + request.complete(null); + } catch (Throwable t) { + reportFailure(t); + } + } + } + + /** + * Completes one execution's drain signal now that its record has been exported, unless a newer record for the same + * execution arrived meanwhile — that one settles the signal instead, so drain() always waits for the latest. + */ + private void signalSettled(String executionArn) { + CompletableFuture signal; + synchronized (this) { + if (pending.containsKey(executionArn)) { + // A newer record is queued for the same execution. Leave the ARN marked as being exported: it is still + // outstanding, and the export of that newer record settles the signal. + return; + } + exporting.remove(executionArn); + signal = settled.remove(executionArn); + } + if (signal != null) { + signal.complete(null); + } + } + + /** + * Removes and returns the signals of executions with nothing outstanding: no record queued and none being + * handed to the exporters right now. Caller holds the lock. + */ + private List> takeSignalsWithNothingOutstanding() { + List> taken = new ArrayList<>(); + Iterator>> signals = + settled.entrySet().iterator(); + while (signals.hasNext()) { + Map.Entry> signal = signals.next(); + if (!pending.containsKey(signal.getKey()) && !exporting.contains(signal.getKey())) { + taken.add(signal.getValue()); + signals.remove(); + } + } + return taken; + } + /** * Flushes every exporter, each on its own worker, and waits for all of them to settle. A slow or failing flush on - * one exporter never delays or fails the others. + * one exporter never delays or fails the others. Plugin-wide, like the exporters themselves. + * + *

Private and called only from the pump: routing every flush through the pump is what keeps a {@code flush()} + * from overlapping an {@code export()}, so this must not be reachable from outside. The per-exporter fan-out below + * is parallelism within one flush, not concurrency with an export. */ - void flushAll() { + private void flushEveryExporter() { forEachExporterSettled(InsightExporter::flush); } @@ -171,17 +634,17 @@ private void forEachExporterSettled(Consumer action) { runSafely(() -> action.accept(exporters.get(0))); return; } - List> settled = new ArrayList<>(exporters.size()); + List> settledExporters = new ArrayList<>(exporters.size()); for (InsightExporter exporter : exporters) { Runnable task = () -> runSafely(() -> action.accept(exporter)); try { - settled.add(CompletableFuture.runAsync(task, executor)); + settledExporters.add(CompletableFuture.runAsync(task, executor)); } catch (Throwable t) { reportFailure(t); task.run(); } } - for (CompletableFuture task : settled) { + for (CompletableFuture task : settledExporters) { runSafely(task::join); } } @@ -201,4 +664,36 @@ private void reportFailure(Throwable t) { // A scheduler diagnostic must never disrupt durable execution. } } + + /** + * Reports and refuses a wait for the pump that was issued from the pump. Returns whether the caller is the + * pump thread; when it is, the failure has already been reported and the caller must return without waiting. + * + *

Invariant: the thread that waits for the pump is never the thread that serves it. {@link #flush()} waits for a + * request only a pump can complete, and {@link #drain(String)} waits for a signal only a pump can complete or for + * the running pump's own handle. All three are satisfied by the pump between records. + * + *

Without this, a wait issued from the pump is a wait-for cycle one thread wide: the pump parks on the future it + * would itself have completed, so it never reaches the point in its loop that completes it, and no other thread may + * take over because {@code inFlight} is this pump's. The invocation never returns, and nothing reports it — a + * {@link CompletableFuture} park cycle is not a monitor deadlock, so the JVM's deadlock detection cannot see it. + * Reachable through anything the pump calls synchronously: with a single exporter the fan-out runs on the pump + * thread, so a customer exporter's {@code export()} that asks for a flush, or a non-conforming {@code exportOne}, + * is enough. A conforming production {@code exportOne} does not re-enter the scheduler, so this is hardening. + * + *

So the call fails fast instead: the plugin's failure handler is told — it logs — and the caller returns as it + * would from any other flush or drain, with nothing propagating into the execution. The queued work itself is not + * dropped by refusing a {@code drain}: the record stays in {@code pending} and the pump asking the question is the + * one that will export it. Callers that are not the pump — every SDK hook thread — never enter this branch and + * behave exactly as before, and the check is a single volatile read, so no lock is added to that path. + */ + private boolean refuseWaitFromThePumpThread(String call) { + if (pumpThread.get() != Thread.currentThread()) { + return false; + } + reportFailure(new IllegalStateException(call + + " was called from the export pump thread, the only thread able to serve it; the call was refused" + + " rather than deadlocking the invocation")); + return true; + } } diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightExporter.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightExporter.java index 69ff39739..a6acb8c88 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightExporter.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightExporter.java @@ -11,7 +11,29 @@ public interface InsightExporter { /** Emits one record to the destination. */ void export(WorkflowInsightRecord record); - /** Flushes any buffered records; no-op by default. */ + /** + * Flushes any records this exporter has buffered. The default is a no-op; override it only if + * {@link #export(WorkflowInsightRecord)} buffers rather than emitting immediately. + * + *

Called at most once per sampled-in invocation end, after that invocation's own record — if it emitted one — + * has been handed to every exporter. An end that emits no record still flushes (a non-terminal suspend under + * {@code ON_COMPLETE}, a success under {@code ON_FAILURE}), so records buffered by that execution's earlier + * emissions are never left behind. Invocation ends that overlap may share a single flush: one flush is enough for + * all of them, because it starts only after each of their records has been handed to every exporter. An execution + * that is sampled out neither exports nor flushes. + * + *

Never called concurrently with {@link #export(WorkflowInsightRecord)} on the same plugin instance. + * + *

May cover records belonging to other executions running in the same environment, so it is not a per-execution + * barrier. + * + *

Must return promptly. No invocation whose end is waiting on this flush can return until it returns, and since + * overlapping ends may share one flush, a slow flush is billed to every one of those invocations — not only to the + * one that asked for it. + * + *

Failures are isolated: a {@link Throwable} thrown here is reported through the plugin's failure handler, never + * retried, never propagated into the execution, and never prevents another exporter from flushing. + */ default void flush() {} /** diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/WorkflowInsight.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/WorkflowInsight.java index 531bbc475..5634e3c19 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/WorkflowInsight.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/WorkflowInsight.java @@ -56,6 +56,7 @@ public static DurableExecutionPlugin workflowInsight(WorkflowInsightConfig confi /** Per-execution state, keyed by execution ARN, to prevent warm-container bleed and handle resume. */ private static final class ExecutionState { + final String executionArn; final Instant startTime; final ArnParser arn; final boolean sampledIn; @@ -68,7 +69,8 @@ private static final class ExecutionState { */ boolean closed; - ExecutionState(Instant startTime, ArnParser arn, boolean sampledIn) { + ExecutionState(String executionArn, Instant startTime, ArnParser arn, boolean sampledIn) { + this.executionArn = executionArn; this.startTime = startTime; this.arn = arn; this.sampledIn = sampledIn; @@ -80,7 +82,7 @@ boolean scheduleIfOpen(ExportScheduler scheduler, WorkflowInsightRecord record) if (closed) { return false; } - scheduler.schedule(record); + scheduler.schedule(executionArn, record); return true; } } @@ -90,7 +92,7 @@ void closeAndSchedule(ExportScheduler scheduler, WorkflowInsightRecord finalReco synchronized (this) { closed = true; if (finalRecord != null) { - scheduler.schedule(finalRecord); + scheduler.schedule(executionArn, finalRecord); } } } @@ -115,7 +117,7 @@ int retainedStateCount() { /** Test seam: waits until every scheduled record has been handed to the exporters. */ void drainExports() { - scheduler.drain(); + scheduler.drainAll(); } InsightPlugin(WorkflowInsightConfig config) { @@ -137,7 +139,7 @@ void drainExports() { private ExecutionState getState(String arn, Instant startTime) { return byArn.computeIfAbsent( - arn, a -> new ExecutionState(startTime, ArnParser.parse(a), shouldSample(a, samplingRate))); + arn, a -> new ExecutionState(a, startTime, ArnParser.parse(a), shouldSample(a, samplingRate))); } @Override @@ -159,15 +161,17 @@ public void onInvocationStart(InvocationInfo info) { state.cachedInput = null; } if (emitMode == WorkflowInsightConfig.EmitMode.ON_CHANGE) { - scheduler.schedule(buildRecord( - state, + scheduler.schedule( info.durableExecutionArn(), - "RUNNING", - info.operations(), - null, - state.cachedInput, - null, - null)); + buildRecord( + state, + info.durableExecutionArn(), + "RUNNING", + info.operations(), + null, + state.cachedInput, + null, + null)); } } catch (Throwable t) { logSafely("onInvocationStart failed", t); @@ -253,7 +257,7 @@ public void onInvocationEnd(InvocationEndInfo info) { // Sampled-out executions never schedule a record, so there is nothing to drain or flush. If the state // lookup itself failed, drain anyway: it is a no-op when idle and otherwise delivers what is pending. if (state == null || state.sampledIn) { - drainAndFlush(); + drainAndFlush(info.durableExecutionArn()); } // Remove per-execution state on EVERY invocation end, including non-terminal PENDING/RETRYING suspends, // once any emission work above is done. Nothing durable is lost: the next invocation's onInvocation @@ -263,19 +267,39 @@ public void onInvocationEnd(InvocationEndInfo info) { // InvocationInfo.executionInput(). Retaining state instead leaked one entry per suspended execution for // the lifetime of the warm container. This runs even if emission above threw, so a plugin failure can // never turn into a state leak. - byArn.remove(info.durableExecutionArn()); + // + // Contained like every other step of this hook: the removal itself can fail — a null execution ARN + // makes ConcurrentHashMap.remove throw — and this is the last statement of onInvocationEnd, so an + // uncaught Throwable here would escape into the SDK and disrupt durable execution, which this method's + // contract forbids. + try { + byArn.remove(info.durableExecutionArn()); + } catch (Throwable t) { + logSafely("failed to remove per-execution state", t); + } } } - /** Waits for every scheduled record to reach the exporters, then flushes each exporter once, concurrently. */ - private void drainAndFlush() { + /** + * Waits for this execution's scheduled record to reach the exporters, then flushes each exporter once. The wait + * is per execution: another execution running in the same environment can never displace this execution's + * record, so this always returns having delivered this execution's latest snapshot. It is not insulated from + * the queue, though — one pump exports serially, so records another execution had already queued ahead of this + * one are exported first and this drain waits for them too. + * + *

The flush goes through the scheduler's queue and is served by that same pump, between records, so no + * exporter ever sees this execution's {@code flush()} overlap another execution's {@code export()}. Invocation + * ends that overlap share one flush: the cadence the exporter contract promises is at most one flush per + * sampled-in invocation end, not exactly one. + */ + private void drainAndFlush(String executionArn) { try { - scheduler.drain(); + scheduler.drain(executionArn); } catch (Throwable t) { logSafely("failed to drain export scheduler", t); } try { - scheduler.flushAll(); + scheduler.flush(); } catch (Throwable t) { logSafely("exporter flush failed", t); } diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ConcurrentExecutionsExportTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ConcurrentExecutionsExportTest.java new file mode 100644 index 000000000..fec4ea396 --- /dev/null +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ConcurrentExecutionsExportTest.java @@ -0,0 +1,392 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Deque; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.Executor; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.TimeUnit; +import org.junit.jupiter.api.Test; +import software.amazon.awssdk.services.lambda.model.OperationStatus; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationInfo; +import software.amazon.lambda.durable.plugin.InvocationStatus; +import software.amazon.lambda.durable.plugin.OperationChangeInfo; +import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; + +/** + * One plugin instance — and therefore one {@link ExportScheduler} — serves a whole execution environment, and an + * environment can host several durable executions at once (routine under Lambda Managed Instances). These tests pin the + * per-execution guarantees that concurrency demands: one execution's record never displaces another's, and each + * execution's drain returns only after its own record reached the exporters. + */ +class ConcurrentExecutionsExportTest { + + private static final Instant START = Instant.parse("2026-08-05T00:00:00Z"); + + private static String arn(int index) { + return "arn:aws:lambda:us-west-2:111122223333:function:f:$LATEST/durable-execution/exec-" + index + "/inv-1"; + } + + private static class CapturingExporter implements InsightExporter { + final List records = new CopyOnWriteArrayList<>(); + + @Override + public void export(WorkflowInsightRecord record) { + records.add(record); + } + + /** Identity, not equality: these tests track the exact record instance an execution scheduled. */ + boolean exported(WorkflowInsightRecord record) { + for (WorkflowInsightRecord seen : records) { + if (seen == record) { + return true; + } + } + return false; + } + } + + private static WorkflowInsightRecord record(String executionArn, String status) { + var r = new WorkflowInsightRecord(); + r.executionArn = executionArn; + r.status = status; + return r; + } + + private static ExportScheduler scheduler(List failures, InsightExporter... exporters) { + return new ExportScheduler(List.of(exporters), (rec, exp) -> exp.export(rec), failures::add, workers()); + } + + private static Executor workers() { + return command -> new Thread(command, "test-export-worker").start(); + } + + /** + * Parks the pump task and then reports rejection, leaving the scheduler idle with the record still queued. Nothing + * about the scheduler is faked: the parked task is its own {@code () -> pump(handle)} lambda, run later verbatim. + */ + private static final class ParkingExecutor implements Executor { + final Deque parked = new ArrayDeque<>(); + + @Override + public void execute(Runnable command) { + parked.add(command); + throw new RejectedExecutionException("test: parked, reported as rejected"); + } + } + + /** Terminal records, by execution ARN, in the order the exporter received them. */ + private static List terminalArns(CapturingExporter exporter) { + List out = new ArrayList<>(); + for (WorkflowInsightRecord r : exporter.records) { + if ("SUCCEEDED".equals(r.status())) { + out.add(r.executionArn()); + } + } + return out; + } + + @Test + void everyConcurrentExecutionDeliversItsTerminalRecordExactlyOnce() throws Exception { + int executions = 10; + int changesEach = 3; + var failures = new CopyOnWriteArrayList(); + var exporter = new CapturingExporter(); + var scheduler = scheduler(failures, exporter); + + var barrier = new CyclicBarrier(executions); + // Executions whose drain returned before their own terminal record had reached the exporter. Each thread checks + // its own postcondition the instant its drain returns; inspecting the exporter only after joining every thread + // would also pass if a drain returned early and the export landed a moment later. + var returnedBeforeExport = Collections.synchronizedList(new ArrayList()); + var threads = new ArrayList(); + for (int i = 0; i < executions; i++) { + String executionArn = arn(i); + var thread = new Thread( + () -> { + awaitBarrier(barrier); + for (int c = 0; c < changesEach; c++) { + scheduler.schedule(executionArn, record(executionArn, "RUNNING")); + } + WorkflowInsightRecord terminal = record(executionArn, "SUCCEEDED"); + scheduler.schedule(executionArn, terminal); + scheduler.drain(executionArn); + // This thread is the only one scheduling for this ARN, so no later record can supersede the + // terminal one: once drain returns, it must already have reached the exporter. + if (!exporter.exported(terminal)) { + returnedBeforeExport.add(executionArn); + } + }, + "execution-" + i); + threads.add(thread); + thread.start(); + } + for (Thread thread : threads) { + thread.join(30_000); + assertFalse(thread.isAlive(), "every execution's drain returned"); + } + + assertEquals( + List.of(), + returnedBeforeExport, + "drain returned before this execution's own terminal record reached the exporter"); + List delivered = terminalArns(exporter); + Set expected = new HashSet<>(); + for (int i = 0; i < executions; i++) { + expected.add(arn(i)); + } + assertEquals(expected, new HashSet<>(delivered), "no execution lost its terminal record"); + assertEquals(executions, delivered.size(), "and none was exported twice"); + assertTrue(failures.isEmpty(), "no scheduler failure was reported: " + failures); + } + + /** + * Regression: a pump that is exiting must not complete the drain signal of an execution whose record it does not + * own. Such a record has already left the queue — it is inside the exporters — so "no record queued for this ARN" + * is not enough to call the signal orphaned. If it were, the exiting pump would release {@code drain(arn)} mid + * export and the invocation could return before its final record was delivered. + * + *

The state is built through the {@link Executor} seam rather than by racing threads: the executor parks the + * pump task and reports rejection, which is the same shape the scheduler produces on its own in the window between + * the pump loop's return to idle and its {@code finally} — a live pump whose handle is no longer the installed one. + */ + @Test + void anExitingPumpDoesNotReleaseADrainWhoseRecordIsStillInsideTheExporter() throws Exception { + var executor = new ParkingExecutor(); + var exporting = new CountDownLatch(1); + var release = new CountDownLatch(1); + Set exported = ConcurrentHashMap.newKeySet(); + var scheduler = new ExportScheduler( + List.of(record -> {}), + (rec, exp) -> { + exporting.countDown(); + await(release, 10); + exported.add(rec); + }, + new CopyOnWriteArrayList()::add, + executor); + + String executionArn = arn(1); + WorkflowInsightRecord terminal = record(executionArn, "SUCCEEDED"); + scheduler.schedule(executionArn, terminal); + assertEquals(1, executor.parked.size(), "the pump task was parked, so the record is still queued"); + + // A drainer picks the record up on the inline path and is now inside the exporter. + var inlineDrained = new CountDownLatch(1); + var inlineDrainer = new Thread( + () -> { + scheduler.drain(executionArn); + inlineDrained.countDown(); + }, + "inline-drainer"); + inlineDrainer.setDaemon(true); + inlineDrainer.start(); + assertTrue(exporting.await(5, TimeUnit.SECONDS), "the terminal record is inside the exporter"); + + // Now let the parked pump run to completion. It finds nothing queued and exits; its cleanup must leave the + // record that is mid-export alone. + executor.parked.poll().run(); + + var secondDrained = new CountDownLatch(1); + var secondDrainer = new Thread( + () -> { + scheduler.drain(executionArn); + secondDrained.countDown(); + }, + "second-drainer"); + secondDrainer.setDaemon(true); + secondDrainer.start(); + assertFalse( + secondDrained.await(500, TimeUnit.MILLISECONDS), + "drain returned while the execution's record was still inside the exporter"); + assertTrue(exported.isEmpty(), "the exporter has not finished with the record yet"); + + release.countDown(); + assertTrue(secondDrained.await(5, TimeUnit.SECONDS), "the drain returns once the export completes"); + assertTrue(inlineDrained.await(5, TimeUnit.SECONDS), "so does the drain that ran the export"); + assertEquals(Set.of(terminal), exported, "the terminal record was exported exactly once"); + } + + @Test + void aRecordForAnotherExecutionNeverDisplacesAPendingTerminalRecord() throws Exception { + String slowExecution = arn(1); + String otherExecution = arn(2); + var exporting = new CountDownLatch(1); + var release = new CountDownLatch(1); + var exporter = new CapturingExporter() { + @Override + public void export(WorkflowInsightRecord record) { + super.export(record); + if (records.size() == 1) { + exporting.countDown(); + await(release); + } + } + }; + var scheduler = scheduler(new CopyOnWriteArrayList<>(), exporter); + + // One execution's export is in flight and blocked... + scheduler.schedule(slowExecution, record(slowExecution, "RUNNING")); + assertTrue(exporting.await(5, TimeUnit.SECONDS), "the first export is in flight"); + // ...while a second execution's terminal record is queued, followed by an update for the first execution. + // The first execution's own update must coalesce only with its own slot, never over the second execution's. + scheduler.schedule(otherExecution, record(otherExecution, "SUCCEEDED")); + scheduler.schedule(slowExecution, record(slowExecution, "SUCCEEDED")); + + var seenBySlowDrain = Collections.synchronizedList(new ArrayList()); + var seenByOtherDrain = Collections.synchronizedList(new ArrayList()); + var slowDrained = new CountDownLatch(1); + var otherDrained = new CountDownLatch(1); + var slowDrainer = new Thread( + () -> { + scheduler.drain(slowExecution); + seenBySlowDrain.addAll(terminalArns(exporter)); + slowDrained.countDown(); + }, + "slow-drainer"); + var otherDrainer = new Thread( + () -> { + scheduler.drain(otherExecution); + seenByOtherDrain.addAll(terminalArns(exporter)); + otherDrained.countDown(); + }, + "other-drainer"); + slowDrainer.start(); + otherDrainer.start(); + + assertFalse(slowDrained.await(200, TimeUnit.MILLISECONDS), "a drain cannot return while its record is pending"); + assertFalse(otherDrained.await(50, TimeUnit.MILLISECONDS), "nor can the other execution's drain"); + + release.countDown(); + assertTrue(slowDrained.await(5, TimeUnit.SECONDS), "the blocked execution's drain completes"); + assertTrue(otherDrained.await(5, TimeUnit.SECONDS), "the other execution's drain completes"); + + assertTrue( + seenByOtherDrain.contains(otherExecution), + "drain returned only after this execution's own terminal record was exported"); + assertTrue( + seenBySlowDrain.contains(slowExecution), + "drain returned only after this execution's own terminal record was exported"); + List delivered = terminalArns(exporter); + assertEquals( + Set.of(slowExecution, otherExecution), + new HashSet<>(delivered), + "neither execution's terminal record was lost"); + assertEquals(2, delivered.size(), "and neither was exported twice"); + } + + @Test + void concurrentExecutionsDrivenThroughThePluginHooksAllDeliverTheirTerminalRecord() throws Exception { + int executions = 5; + var exporter = new CapturingExporter(); + var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .addExporter(exporter) + .build()); + + var barrier = new CyclicBarrier(executions); + var threads = new ArrayList(); + for (int i = 0; i < executions; i++) { + String executionArn = arn(i); + var thread = new Thread( + () -> { + awaitBarrier(barrier); + plugin.onInvocationStart(start(executionArn)); + for (int c = 0; c < 3; c++) { + plugin.onOperationChange(new OperationChangeInfo( + "req", + executionArn, + ops(OperationStatus.SUCCEEDED), + ops(OperationStatus.SUCCEEDED))); + } + plugin.onInvocationEnd(end(executionArn, InvocationStatus.SUCCEEDED)); + }, + "execution-" + i); + threads.add(thread); + thread.start(); + } + for (Thread thread : threads) { + thread.join(30_000); + assertFalse(thread.isAlive(), "every invocation-end hook returned"); + } + + List delivered = terminalArns(exporter); + Set expected = new HashSet<>(); + for (int i = 0; i < executions; i++) { + expected.add(arn(i)); + } + assertEquals(expected, new HashSet<>(delivered), "every execution's terminal record arrived"); + assertEquals(executions, delivered.size(), "and none arrived twice"); + assertEquals(0, plugin.retainedStateCount(), "no per-execution state retained after invocation end"); + } + + private static Map ops(OperationStatus status) { + Map operations = new LinkedHashMap<>(); + operations.put( + "op-1", + new OperationChangeItemInfo( + "op-1", + "greet", + "STEP", + "Step", + null, + START, + START.plusMillis(5), + status, + 1, + false, + null, + null)); + return operations; + } + + private static InvocationInfo start(String executionArn) { + return new InvocationInfo("req", executionArn, true, START, "in", ops(OperationStatus.STARTED), Map.of()); + } + + private static InvocationEndInfo end(String executionArn, InvocationStatus status) { + return new InvocationEndInfo( + "req", executionArn, true, START, ops(OperationStatus.SUCCEEDED), status, null, "in", "out"); + } + + private static void awaitBarrier(CyclicBarrier barrier) { + try { + barrier.await(30, TimeUnit.SECONDS); + } catch (Exception e) { + throw new AssertionError(e); + } + } + + private static void await(CountDownLatch latch) { + await(latch, 5); + } + + private static void await(CountDownLatch latch, long timeoutSeconds) { + try { + if (!latch.await(timeoutSeconds, TimeUnit.SECONDS)) { + throw new AssertionError("latch not released"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + } +} diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushCoalescingTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushCoalescingTest.java new file mode 100644 index 000000000..014732f76 --- /dev/null +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushCoalescingTest.java @@ -0,0 +1,308 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.Collections; +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.Executor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.Test; + +/** + * Contract tests for the flush cadence: at most one flush per invocation end that asks for one, invocation ends that + * overlap may share a flush, and a request made while a flush is already running is never satisfied by that flush. + * + *

Coalescing is sound because every requester drains its own record before asking, so a flush that starts + * after the request was made has that record in the buffer. It is what stops N ends that ask together from paying for N + * serialized flush fan-outs — a 60 ms exporter flush cost the slowest of 8 ends ~520 ms before this change. + */ +class ExportSchedulerFlushCoalescingTest { + + /** A slow flush must not cost the caller more than a small multiple of the one flush it asked for. */ + private static final int PROMPTNESS_FACTOR = 3; + + private static String arn(int index) { + return "arn:aws:lambda:us-west-2:111122223333:function:f:$LATEST/durable-execution/exec-" + index + "/inv-1"; + } + + private static WorkflowInsightRecord record(String executionArn, String status) { + var r = new WorkflowInsightRecord(); + r.executionArn = executionArn; + r.status = status; + return r; + } + + private static ExportScheduler scheduler(Executor executor, InsightExporter... exporters) { + return new ExportScheduler(List.of(exporters), (rec, exp) -> exp.export(rec), t -> {}, executor); + } + + /** Unbounded thread-per-task executor, like the cached pool the plugin injects in production. */ + private static Executor sharedWorkers() { + return command -> { + var thread = new Thread(command, "coalescing-worker"); + thread.setDaemon(true); + thread.start(); + }; + } + + private static final class SlowFlushExporter implements InsightExporter { + private final long flushMillis; + final AtomicInteger flushes = new AtomicInteger(); + + SlowFlushExporter(long flushMillis) { + this.flushMillis = flushMillis; + } + + @Override + public void export(WorkflowInsightRecord record) {} + + @Override + public void flush() { + flushes.incrementAndGet(); + sleep(flushMillis); + } + } + + /** + * N invocation ends whose records have already been delivered ask for their flush together: they share one flush, + * so the slowest pays a small multiple of one flush rather than N times one. + * + *

The records are drained before the requests are made on purpose: this is coalescing on its own, with every + * request already queued when the pump reaches its flush step. The harder case — ends that are still inside + * {@code drain()} when the first request is served, and so cannot have asked yet — is + * {@link #simultaneousDrainAndFlushEndsShareAFlushRatherThanOneEach()}. + */ + @Test + void invocationEndsAskingForAFlushTogetherShareOneFlush() { + long flushMillis = 60; + int executions = 8; + var exporter = new SlowFlushExporter(flushMillis); + var scheduler = scheduler(sharedWorkers(), exporter); + + var durations = Collections.synchronizedList(new java.util.ArrayList()); + var recordsDelivered = new CyclicBarrier(executions); + var done = new CountDownLatch(executions); + for (int i = 0; i < executions; i++) { + String executionArn = arn(i); + start("end-" + i, () -> { + scheduler.schedule(executionArn, record(executionArn, "SUCCEEDED")); + scheduler.drain(executionArn); + awaitBarrier(recordsDelivered); + long began = System.nanoTime(); + scheduler.flush(); + durations.add((System.nanoTime() - began) / 1_000_000L); + done.countDown(); + }); + } + assertTrue(await(done, 60_000), "an invocation end never returned"); + + long slowest = Collections.max(durations); + int flushes = exporter.flushes.get(); + System.out.printf( + "COALESCING: %d invocation ends flushing together, %d ms exporter flush | slowest end returned after" + + " %d ms | flushes run: %d (one per end, %d, before coalescing)%n", + executions, flushMillis, slowest, flushes, executions); + + assertTrue(flushes >= 1, "every invocation end must be covered by a flush"); + assertTrue(flushes <= executions, "at most one flush per invocation end: " + flushes + " for " + executions); + assertTrue( + slowest <= flushMillis * PROMPTNESS_FACTOR, + "the slowest invocation end waited " + slowest + " ms for a " + flushMillis + " ms flush: ends that ask" + + " together must share a flush rather than serialize one fan-out each"); + } + + /** + * The realistic shape: schedule, drain, flush, all landing at once. An end cannot ask for its flush until its own + * record has been exported, so the pump exports the records a drain is waiting for before it spends a flush + * fan-out; without that the ends are staggered one record per flush and each pays for a flush of its own. + */ + @Test + void simultaneousDrainAndFlushEndsShareAFlushRatherThanOneEach() { + long flushMillis = 40; + int executions = 8; + var exporter = new SlowFlushExporter(flushMillis); + var scheduler = scheduler(sharedWorkers(), exporter); + + var durations = Collections.synchronizedList(new java.util.ArrayList()); + var barrier = new CyclicBarrier(executions); + var done = new CountDownLatch(executions); + for (int i = 0; i < executions; i++) { + String executionArn = arn(i); + start("drain-and-flush-" + i, () -> { + awaitBarrier(barrier); + long began = System.nanoTime(); + scheduler.schedule(executionArn, record(executionArn, "SUCCEEDED")); + scheduler.drain(executionArn); + scheduler.flush(); + durations.add((System.nanoTime() - began) / 1_000_000L); + done.countDown(); + }); + } + assertTrue(await(done, 60_000), "an invocation end never returned"); + + int flushes = exporter.flushes.get(); + long slowest = Collections.max(durations); + System.out.printf( + "COALESCING (drain then flush): %d simultaneous ends, %d ms exporter flush | slowest end after %d ms |" + + " flushes run: %d (one per end, %d, before coalescing)%n", + executions, flushMillis, slowest, flushes, executions); + assertTrue(flushes >= 1, "every invocation end must be covered by a flush"); + assertTrue(flushes <= executions, "at most one flush per invocation end: " + flushes + " for " + executions); + assertTrue( + slowest <= flushMillis * (PROMPTNESS_FACTOR + 1), + "the slowest of " + executions + " simultaneous ends waited " + slowest + " ms for a " + flushMillis + + " ms flush: an end must not pay for one flush per end"); + } + + /** + * The exact counts, deterministically: five requests made while a flush is running are not satisfied by it — that + * flush cannot have seen their records — and they share the single flush that follows it. + */ + @Test + void requestsMadeWhileAFlushRunsShareTheNextFlushAndAreNeverSatisfiedByTheRunningOne() { + var insideFirstFlush = new CountDownLatch(1); + var releaseFirstFlush = new CountDownLatch(1); + var flushStarts = new AtomicInteger(); + var flushCompletions = new AtomicInteger(); + var exporter = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) {} + + @Override + public void flush() { + if (flushStarts.incrementAndGet() == 1) { + insideFirstFlush.countDown(); + await(releaseFirstFlush, 30_000); + } + flushCompletions.incrementAndGet(); + } + }; + var scheduler = scheduler(sharedWorkers(), exporter); + + start("first-flusher", scheduler::flush); + assertTrue(await(insideFirstFlush, 5_000), "the pump never entered the first flush"); + + int latecomers = 5; + var returned = new CountDownLatch(latecomers); + var startsSeenOnReturn = new CopyOnWriteArrayList(); + for (int i = 0; i < latecomers; i++) { + start("latecomer-" + i, () -> { + scheduler.flush(); + startsSeenOnReturn.add(flushStarts.get()); + returned.countDown(); + }); + } + sleep(300); // every latecomer is queued while the first flush is still inside the exporter + + assertFalse( + await(returned, 200), + "a request made while a flush was already running was satisfied by that flush, which cannot have seen" + + " the requester's record"); + releaseFirstFlush.countDown(); + assertTrue(await(returned, 10_000), "a queued request was never served"); + + assertEquals( + 2, + flushStarts.get(), + "the five latecomers must share exactly one flush, taken as a batch after the first one ended"); + assertTrue( + startsSeenOnReturn.stream().allMatch(starts -> starts >= 2), + "each latecomer must be served by a flush that started after it was enqueued: " + startsSeenOnReturn); + assertEquals(2, flushCompletions.get(), "no flush ran twice for the same batch"); + } + + /** + * The same property under load, measured per request: when a request returns, a flush that started after the + * request was made must already have completed. Flushes are serialized by the pump, so counting completions is + * enough — a request satisfied by a flush that was already running would return with the completion count still at + * or below the value observed before it asked. + */ + @Test + void everyRequestIsSatisfiedByAFlushThatStartedAfterItWasMade() { + var flushStarts = new AtomicInteger(); + var flushCompletions = new AtomicInteger(); + var exporter = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) {} + + @Override + public void flush() { + flushStarts.incrementAndGet(); + sleep(1); + flushCompletions.incrementAndGet(); + } + }; + var scheduler = scheduler(sharedWorkers(), exporter); + + int requests = 120; + var done = new CountDownLatch(requests); + var violations = new CopyOnWriteArrayList(); + for (int i = 0; i < requests; i++) { + String executionArn = arn(i); + start("load-flusher-" + i, () -> { + scheduler.schedule(executionArn, record(executionArn, "SUCCEEDED")); + scheduler.drain(executionArn); + int startsBefore = flushStarts.get(); + scheduler.flush(); + if (flushCompletions.get() <= startsBefore) { + violations.add("returned with completions=" + flushCompletions.get() + " after observing starts=" + + startsBefore); + } + done.countDown(); + }); + } + assertTrue(await(done, 60_000), "a request was never served"); + scheduler.drainAll(); + + System.out.printf("COALESCING under load: %d requests satisfied by %d flushes%n", requests, flushStarts.get()); + assertEquals(List.of(), violations, "a request was credited to a flush that was already running"); + assertTrue(flushStarts.get() >= 1); + assertTrue( + flushStarts.get() <= requests, + "at most one flush per request: " + flushStarts.get() + " for " + requests); + } + + private static Thread start(String name, Runnable body) { + var thread = new Thread(body, name); + thread.setDaemon(true); + thread.start(); + return thread; + } + + private static boolean await(CountDownLatch latch, long timeoutMillis) { + try { + return latch.await(timeoutMillis, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + return false; + } + } + + private static void awaitBarrier(CyclicBarrier barrier) { + try { + barrier.await(60, TimeUnit.SECONDS); + } catch (Exception e) { + throw new AssertionError(e); + } + } + + private static void sleep(long millis) { + if (millis <= 0) { + return; + } + try { + Thread.sleep(millis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + } +} diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushSerializationTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushSerializationTest.java new file mode 100644 index 000000000..b37677d2a --- /dev/null +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushSerializationTest.java @@ -0,0 +1,429 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.Executor; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.Test; + +/** + * Contract tests for the exporter-facing flush guarantee: {@code flush()} is served by the export pump, so it is never + * called concurrently with {@code export()} on the same plugin instance, and cannot be starved by a queue that keeps + * receiving records. + * + *

The cadence itself — at most one flush per request, requests that overlap sharing one flush — is covered by + * {@link ExportSchedulerFlushCoalescingTest}. + */ +class ExportSchedulerFlushSerializationTest { + + /** Longest a flush may take to be served before the property under test is considered broken. */ + private static final long FLUSH_DEADLINE_MILLIS = 2_000; + + private static String arn(int index) { + return "arn:aws:lambda:us-west-2:111122223333:function:f:$LATEST/durable-execution/exec-" + index + "/inv-1"; + } + + private static WorkflowInsightRecord record(String executionArn, String status) { + var r = new WorkflowInsightRecord(); + r.executionArn = executionArn; + r.status = status; + return r; + } + + private static ExportScheduler scheduler( + Executor executor, List failures, InsightExporter... exporters) { + return new ExportScheduler(List.of(exporters), (rec, exp) -> exp.export(rec), failures::add, executor); + } + + /** Unbounded thread-per-task executor, like the cached pool the plugin injects in production. */ + private static Executor sharedWorkers() { + return command -> { + var thread = new Thread(command, "test-export-worker"); + thread.setDaemon(true); + thread.start(); + }; + } + + /** + * Records, per exporter instance, whether an {@code export()} and a {@code flush()} were ever inside the exporter + * at the same time, and how many of each ran concurrently. + */ + private static final class OverlapProbeExporter implements InsightExporter { + private final long exportMillis; + private final long flushMillis; + final AtomicInteger inExport = new AtomicInteger(); + final AtomicInteger inFlush = new AtomicInteger(); + final AtomicInteger maxConcurrentExports = new AtomicInteger(); + final AtomicInteger maxConcurrentFlushes = new AtomicInteger(); + final AtomicInteger flushes = new AtomicInteger(); + final List exported = new CopyOnWriteArrayList<>(); + final AtomicBoolean overlapped = new AtomicBoolean(); + + OverlapProbeExporter(long exportMillis, long flushMillis) { + this.exportMillis = exportMillis; + this.flushMillis = flushMillis; + } + + @Override + public void export(WorkflowInsightRecord record) { + trackMax(maxConcurrentExports, inExport.incrementAndGet()); + try { + checkOverlap(); + sleep(exportMillis); + checkOverlap(); + exported.add(record.status() + "@" + record.executionArn()); + } finally { + inExport.decrementAndGet(); + } + } + + @Override + public void flush() { + trackMax(maxConcurrentFlushes, inFlush.incrementAndGet()); + try { + checkOverlap(); + sleep(flushMillis); + checkOverlap(); + flushes.incrementAndGet(); + } finally { + inFlush.decrementAndGet(); + } + } + + private void checkOverlap() { + if (inExport.get() > 0 && inFlush.get() > 0) { + overlapped.set(true); + } + } + + private static void trackMax(AtomicInteger max, int observed) { + max.accumulateAndGet(observed, Math::max); + } + } + + @Test + void aFlushNeverOverlapsAnExportEvenWithManyExecutionsEndingAtOnce() throws Exception { + var first = new OverlapProbeExporter(15, 5); + var second = new OverlapProbeExporter(15, 5); + var failures = new CopyOnWriteArrayList(); + var scheduler = scheduler(sharedWorkers(), failures, first, second); + + int executions = 6; + var barrier = new CyclicBarrier(executions); + var threads = new ArrayList(); + for (int i = 0; i < executions; i++) { + String executionArn = arn(i); + var thread = new Thread( + () -> { + awaitBarrier(barrier); + // What an invocation does: a few RUNNING snapshots, the terminal record, then drain + flush. + for (int change = 0; change < 3; change++) { + scheduler.schedule(executionArn, record(executionArn, "RUNNING")); + } + scheduler.schedule(executionArn, record(executionArn, "SUCCEEDED")); + scheduler.drain(executionArn); + scheduler.flush(); + }, + "invocation-" + i); + thread.setDaemon(true); + threads.add(thread); + thread.start(); + } + for (Thread thread : threads) { + thread.join(30_000); + assertFalse(thread.isAlive(), thread.getName() + " never returned from drain/flush"); + } + + for (OverlapProbeExporter exporter : List.of(first, second)) { + assertFalse(exporter.overlapped.get(), "flush() ran while an export was in flight on the same exporter"); + assertEquals(1, exporter.maxConcurrentExports.get(), "exports must stay serialized"); + assertEquals(1, exporter.maxConcurrentFlushes.get(), "one flush at a time, one fan-out per batch"); + // At most one flush per invocation end, and at least one: ends that land together share a flush, so the + // count is bounded by the number of ends rather than equal to it. + assertTrue(exporter.flushes.get() >= 1, "every invocation end must be covered by a flush"); + assertTrue( + exporter.flushes.get() <= executions, + "at most one flush per invocation end: " + exporter.flushes.get() + " for " + executions); + for (int i = 0; i < executions; i++) { + assertTrue( + exporter.exported.contains("SUCCEEDED@" + arn(i)), + "terminal record of " + arn(i) + " never reached the exporter"); + } + } + assertTrue(failures.isEmpty(), "no failure should be reported: " + failures); + } + + @Test + void twoFlushRequestsQueuedTogetherShareOneFlush() throws Exception { + var exporting = new CountDownLatch(1); + var release = new CountDownLatch(1); + var flushes = new AtomicInteger(); + var exporter = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) { + exporting.countDown(); + await(release); + } + + @Override + public void flush() { + flushes.incrementAndGet(); + } + }; + var scheduler = scheduler(sharedWorkers(), new CopyOnWriteArrayList<>(), exporter); + + scheduler.schedule(arn(0), record(arn(0), "SUCCEEDED")); + assertTrue(exporting.await(5, TimeUnit.SECONDS), "the pump is inside the exporter"); + + var flushed = new CountDownLatch(2); + for (int i = 0; i < 2; i++) { + var flusher = new Thread( + () -> { + scheduler.flush(); + flushed.countDown(); + }, + "flusher-" + i); + flusher.setDaemon(true); + flusher.start(); + } + Thread.sleep(200); // let both requests queue up behind the in-flight export + + release.countDown(); + assertTrue(flushed.await(5, TimeUnit.SECONDS), "both flush requests must be served"); + assertEquals( + 1, + flushes.get(), + "two requests queued together are taken as one batch and share a single flush: both drained their own" + + " record before asking, so one flush covers both"); + } + + /** Distinct {@link Error} type so the test asserts on this exact failure rather than any Error. */ + private static final class FlushError extends Error { + FlushError() { + super("flush blew up with an Error"); + } + } + + @Test + void aFlushThatThrowsStillReleasesTheInvocationAndLetsTheOtherExportersFlush() throws Exception { + var flushed = new AtomicInteger(); + var throwsException = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) {} + + @Override + public void flush() { + throw new IllegalStateException("flush blew up"); + } + }; + var throwsError = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) {} + + @Override + public void flush() { + throw new FlushError(); + } + }; + var healthy = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) {} + + @Override + public void flush() { + flushed.incrementAndGet(); + } + }; + var failures = new CopyOnWriteArrayList(); + var scheduler = scheduler(sharedWorkers(), failures, throwsException, throwsError, healthy); + + assertTrue(returnsWithin(scheduler::flush, FLUSH_DEADLINE_MILLIS), "a throwing flush stranded the invocation"); + + assertEquals(1, flushed.get(), "the healthy exporter still flushed"); + assertEquals(2, failures.size(), "both failures are reported, neither escapes: " + failures); + assertTrue( + failures.stream().anyMatch(t -> t instanceof IllegalStateException), + "the thrown exception is reported"); + assertTrue(failures.stream().anyMatch(t -> t instanceof FlushError), "the thrown Error is reported"); + } + + @Test + void anErrorFromTheOnlyExportersFlushStillReleasesTheInvocation() throws Exception { + var failures = new CopyOnWriteArrayList(); + var onlyExporter = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) {} + + @Override + public void flush() { + throw new FlushError(); + } + }; + var scheduler = scheduler(sharedWorkers(), failures, onlyExporter); + + assertTrue(returnsWithin(scheduler::flush, FLUSH_DEADLINE_MILLIS), "an Error from flush() stranded the caller"); + assertEquals(1, failures.size(), "the Error is reported, not propagated: " + failures); + assertTrue(failures.get(0) instanceof FlushError); + } + + @Test + void aFlushIsServedWhileTheQueueKeepsReceivingRecords() throws Exception { + var exports = new AtomicInteger(); + var flushes = new AtomicInteger(); + var exporter = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) { + sleep(2); + exports.incrementAndGet(); + } + + @Override + public void flush() { + flushes.incrementAndGet(); + } + }; + var scheduler = scheduler(sharedWorkers(), new CopyOnWriteArrayList<>(), exporter); + + // A producer that never lets the queue run dry: it keeps re-scheduling a fixed, rotating set of executions, so + // `pending` stays non-empty (and bounded, since records coalesce per execution) for as long as it runs. + var stop = new AtomicBoolean(); + var scheduled = new AtomicInteger(); + var producing = new CountDownLatch(1); + var producer = new Thread( + () -> { + int index = 0; + while (!stop.get()) { + String executionArn = arn(index++ % 50); + scheduler.schedule(executionArn, record(executionArn, "RUNNING")); + scheduled.incrementAndGet(); + producing.countDown(); + } + }, + "record-producer"); + producer.setDaemon(true); + producer.start(); + int exportsBefore; + int scheduledBefore; + boolean served; + var exportsWhenServed = new AtomicInteger(); + var scheduledWhenServed = new AtomicInteger(); + var flushReturned = new CountDownLatch(1); + try { + assertTrue(producing.await(5, TimeUnit.SECONDS), "the producer never started scheduling"); + exportsBefore = exports.get(); + scheduledBefore = scheduled.get(); + + // On its own thread with a deadline: a starved flush must fail this test, not hang it. + var flusher = new Thread( + () -> { + scheduler.flush(); + exportsWhenServed.set(exports.get()); + scheduledWhenServed.set(scheduled.get()); + flushReturned.countDown(); + }, + "flusher"); + flusher.setDaemon(true); + flusher.start(); + served = flushReturned.await(FLUSH_DEADLINE_MILLIS, TimeUnit.MILLISECONDS); + } finally { + // Stop the producer before asserting, so a starved flush is released and its thread does not leak. + stop.set(true); + producer.join(10_000); + } + + assertTrue( + served, + "the flush was not served within " + FLUSH_DEADLINE_MILLIS + " ms while the queue kept receiving" + + " records; it must be served between records rather than after the queue drains"); + assertTrue(flushReturned.await(5, TimeUnit.SECONDS)); + assertEquals(1, flushes.get(), "the flush was served exactly once"); + assertTrue( + exportsWhenServed.get() > exportsBefore, "the pump kept exporting: the flush did not stall the queue"); + assertTrue( + scheduledWhenServed.get() > scheduledBefore, + "the queue was still receiving records when the flush was served"); + } + + @Test + void theFlushHappensOnTheCallingThreadWhenNoWorkerCouldBeStarted() { + Executor rejecting = command -> { + throw new RejectedExecutionException("no worker"); + }; + var flushThreads = new CopyOnWriteArrayList(); + var exporter = new InsightExporter() { + @Override + public void export(WorkflowInsightRecord record) {} + + @Override + public void flush() { + flushThreads.add(Thread.currentThread()); + } + }; + var failures = new CopyOnWriteArrayList(); + var scheduler = scheduler(rejecting, failures, exporter); + + scheduler.flush(); + + assertEquals(1, flushThreads.size(), "the flush must still happen when no worker can be started"); + assertSame(Thread.currentThread(), flushThreads.get(0), "the invocation boundary flushes inline"); + assertFalse(failures.isEmpty(), "the rejected worker is reported"); + } + + /** Runs {@code action} on its own thread and reports whether it returned within the deadline. */ + private static boolean returnsWithin(Runnable action, long timeoutMillis) throws InterruptedException { + var returned = new CountDownLatch(1); + var thread = new Thread( + () -> { + action.run(); + returned.countDown(); + }, + "deadline-runner"); + thread.setDaemon(true); + thread.start(); + return returned.await(timeoutMillis, TimeUnit.MILLISECONDS); + } + + private static void awaitBarrier(CyclicBarrier barrier) { + try { + barrier.await(10, TimeUnit.SECONDS); + } catch (Exception e) { + throw new AssertionError(e); + } + } + + private static void await(CountDownLatch latch) { + try { + if (!latch.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("latch not released"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + } + + private static void sleep(long millis) { + if (millis <= 0) { + return; + } + try { + Thread.sleep(millis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + } +} diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerReentrantFlushTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerReentrantFlushTest.java new file mode 100644 index 000000000..42da4c005 --- /dev/null +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerReentrantFlushTest.java @@ -0,0 +1,129 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.Executor; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import org.junit.jupiter.api.Test; + +/** + * A {@code flush()} issued from the thread currently serving the export pump is refused and reported, not waited on. + * + *

That thread is the only one able to serve the request it would be making — flush requests are served by the pump, + * between records — so waiting for it is a wait-for cycle one thread wide, and the invocation never returns. With a + * single exporter the fan-out runs on the pump thread, so anything an exporter's {@code export()} does synchronously is + * enough to reach it. + */ +class ExportSchedulerReentrantFlushTest { + + /** Longest a call that must return promptly may take before the property under test is considered broken. */ + private static final long DEADLINE_MILLIS = 5_000; + + private static String arn(int index) { + return "arn:aws:lambda:us-west-2:111122223333:function:f:$LATEST/durable-execution/exec-" + index + "/inv-1"; + } + + private static WorkflowInsightRecord record(String executionArn, String status) { + var r = new WorkflowInsightRecord(); + r.executionArn = executionArn; + r.status = status; + return r; + } + + /** Unbounded thread-per-task executor, like the cached pool the plugin injects in production. */ + private static Executor sharedWorkers() { + return command -> { + var thread = new Thread(command, "reentrant-flush-worker"); + thread.setDaemon(true); + thread.start(); + }; + } + + private static final class CountingExporter implements InsightExporter { + final AtomicInteger exports = new AtomicInteger(); + final AtomicInteger flushes = new AtomicInteger(); + + @Override + public void export(WorkflowInsightRecord record) { + exports.incrementAndGet(); + } + + @Override + public void flush() { + flushes.incrementAndGet(); + } + } + + @Test + void flushReenteredFromThePumpThreadIsRefusedReportedAndLeavesTheSchedulerUsable() throws Exception { + var exporter = new CountingExporter(); + var failures = new CopyOnWriteArrayList(); + var holder = new AtomicReference(); + var reentrantFlushReturned = new CountDownLatch(1); + var flushesSeenByTheRefusedCall = new AtomicInteger(-1); + + var scheduler = new ExportScheduler( + List.of(exporter), + (rec, exp) -> { + exp.export(rec); + if (reentrantFlushReturned.getCount() > 0 && "SUCCEEDED".equals(rec.status())) { + // Re-entering the scheduler from inside the fan-out: this is the pump's own thread. + holder.get().flush(); + flushesSeenByTheRefusedCall.set(exporter.flushes.get()); + reentrantFlushReturned.countDown(); + } + }, + failures::add, + sharedWorkers()); + holder.set(scheduler); + + var drainReturned = new CountDownLatch(1); + var invocation = new Thread( + () -> { + scheduler.schedule(arn(0), record(arn(0), "SUCCEEDED")); + scheduler.drain(arn(0)); + drainReturned.countDown(); + }, + "reentrant-flush-invocation"); + invocation.setDaemon(true); + invocation.start(); + + assertTrue( + reentrantFlushReturned.await(DEADLINE_MILLIS, MILLISECONDS), + "flush() re-entered from the pump thread never returned: the only thread that can serve the request is" + + " the one waiting for it"); + assertTrue( + drainReturned.await(DEADLINE_MILLIS, MILLISECONDS), + "drain() never returned after the re-entrant flush"); + invocation.join(DEADLINE_MILLIS); + + // Reported, not silently swallowed, and nothing thrown into the caller. + assertEquals(1, failures.size(), "exactly one failure reported: " + failures); + assertTrue( + failures.get(0) instanceof IllegalStateException, + "the refusal is reported as an IllegalStateException: " + failures.get(0)); + assertTrue( + failures.get(0).getMessage().contains("flush()"), + "the report names the refused call: " + failures.get(0).getMessage()); + assertEquals(0, flushesSeenByTheRefusedCall.get(), "the refused request must not have reached an exporter"); + + // Still usable: the next invocation's record is exported and its flush — from a thread that is not the pump — + // is + // served exactly as before. + scheduler.schedule(arn(1), record(arn(1), "SUCCEEDED")); + scheduler.drain(arn(1)); + scheduler.flush(); + + assertEquals(2, exporter.exports.get(), "both records reached the exporter"); + assertEquals(1, exporter.flushes.get(), "the next invocation's flush is served normally"); + assertEquals(1, failures.size(), "no further failure after the refusal: " + failures); + } +} diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerTest.java index 8b38dec5b..adca36bdf 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerTest.java @@ -22,6 +22,13 @@ /** Contract tests for {@link ExportScheduler}: serial exports, latest-wins coalescing, drain, and exporter fan-out. */ class ExportSchedulerTest { + /** All single-execution cases below drive one execution ARN through the scheduler. */ + private static final String ARN = arn(0); + + private static String arn(int index) { + return "arn:aws:lambda:us-west-2:111122223333:function:f:$LATEST/durable-execution/exec-" + index + "/inv-1"; + } + /** Runs submitted tasks only when the test asks, so pump timing is fully controlled. */ private static final class ManualExecutor implements Executor { final Deque tasks = new ArrayDeque<>(); @@ -73,7 +80,7 @@ void scheduleHandsTheRecordToAWorkerRatherThanExportingOnTheCallingThread() { var exporter = new CapturingExporter(); var scheduler = scheduler(executor, new ArrayList<>(), exporter); - scheduler.schedule(record("RUNNING")); + scheduler.schedule(ARN, record("RUNNING")); assertTrue(exporter.records.isEmpty(), "nothing exported until a worker runs"); executor.runAll(); @@ -87,9 +94,9 @@ void updatesScheduledBeforeTheWorkerRunsCollapseIntoTheLatestRecord() { var exporter = new CapturingExporter(); var scheduler = scheduler(executor, new ArrayList<>(), exporter); - scheduler.schedule(record("r1")); - scheduler.schedule(record("r2")); - scheduler.schedule(record("r3")); + scheduler.schedule(ARN, record("r1")); + scheduler.schedule(ARN, record("r2")); + scheduler.schedule(ARN, record("r3")); executor.runAll(); assertEquals(1, exporter.records.size(), "one pump, one latest record"); @@ -103,9 +110,9 @@ void recordScheduledAfterAPumpFinishesStartsANewPump() { var exporter = new CapturingExporter(); var scheduler = scheduler(executor, new ArrayList<>(), exporter); - scheduler.schedule(record("first")); + scheduler.schedule(ARN, record("first")); executor.runAll(); - scheduler.schedule(record("second")); + scheduler.schedule(ARN, record("second")); executor.runAll(); assertEquals(List.of("first", "second"), statuses(exporter)); @@ -127,13 +134,13 @@ public void export(WorkflowInsightRecord record) { }; var scheduler = scheduler(sharedWorkers(), new ArrayList<>(), exporter); - scheduler.schedule(record("first")); + scheduler.schedule(ARN, record("first")); assertTrue(entered.await(5, TimeUnit.SECONDS), "first export is in flight"); - scheduler.schedule(record("dropped-1")); - scheduler.schedule(record("dropped-2")); - scheduler.schedule(record("final")); + scheduler.schedule(ARN, record("dropped-1")); + scheduler.schedule(ARN, record("dropped-2")); + scheduler.schedule(ARN, record("final")); release.countDown(); - scheduler.drain(); + scheduler.drain(ARN); assertEquals(List.of("first", "final"), statuses(exporter)); } @@ -141,8 +148,8 @@ public void export(WorkflowInsightRecord record) { @Test void drainReturnsImmediatelyWhenIdle() { var scheduler = scheduler(new ManualExecutor(), new ArrayList<>(), new CapturingExporter()); - scheduler.drain(); - scheduler.drain(); + scheduler.drain(ARN); + scheduler.drain(ARN); } @Test @@ -161,13 +168,13 @@ public void export(WorkflowInsightRecord record) { }; var scheduler = scheduler(sharedWorkers(), new ArrayList<>(), exporter); - scheduler.schedule(record("slow")); + scheduler.schedule(ARN, record("slow")); assertTrue(entered.await(5, TimeUnit.SECONDS)); - scheduler.schedule(record("final")); + scheduler.schedule(ARN, record("final")); var drained = new CountDownLatch(1); var drainer = new Thread(() -> { - scheduler.drain(); + scheduler.drain(ARN); drained.countDown(); }); drainer.start(); @@ -183,8 +190,8 @@ void exportersRunOffTheSchedulingThread() { var exporter = new CapturingExporter(); var scheduler = scheduler(sharedWorkers(), new ArrayList<>(), exporter); - scheduler.schedule(record("RUNNING")); - scheduler.drain(); + scheduler.schedule(ARN, record("RUNNING")); + scheduler.drain(ARN); assertEquals(1, exporter.threads.size()); assertNotSame(Thread.currentThread(), exporter.threads.get(0)); @@ -199,8 +206,8 @@ void aFailingExporterNeverBlocksTheOthersForTheSameRecord() { }; var scheduler = scheduler(sharedWorkers(), failures, bad, good); - scheduler.schedule(record("RUNNING")); - scheduler.drain(); + scheduler.schedule(ARN, record("RUNNING")); + scheduler.drain(ARN); assertEquals(1, good.records.size()); assertEquals(1, failures.size()); @@ -214,7 +221,7 @@ void exportersForOneRecordRunConcurrentlySoASlowExporterDoesNotDelayTheOthers() InsightExporter slow = record -> await(release); var scheduler = scheduler(sharedWorkers(), new ArrayList<>(), slow, fast); - scheduler.schedule(record("RUNNING")); + scheduler.schedule(ARN, record("RUNNING")); long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); while (fast.records.isEmpty() && System.nanoTime() < deadline) { Thread.sleep(5); @@ -222,7 +229,7 @@ void exportersForOneRecordRunConcurrentlySoASlowExporterDoesNotDelayTheOthers() assertEquals(1, fast.records.size(), "fast exporter received the record while the slow one is still blocked"); release.countDown(); - scheduler.drain(); + scheduler.drain(ARN); } @Test @@ -233,11 +240,11 @@ void drainExportsThePendingRecordInlineWhenNoWorkerCouldBeStarted() { var exporter = new CapturingExporter(); var scheduler = scheduler(executor, failures, exporter); - scheduler.schedule(record("final")); + scheduler.schedule(ARN, record("final")); assertTrue(exporter.records.isEmpty(), "the hook thread does not export"); assertEquals(1, failures.size(), "the worker failure is reported"); - scheduler.drain(); + scheduler.drain(ARN); assertEquals(List.of("final"), statuses(exporter)); assertSame(Thread.currentThread(), exporter.threads.get(0), "the invocation boundary delivers it"); @@ -250,13 +257,13 @@ void aLaterScheduleRetriesTheWorkerAfterARejection() { var exporter = new CapturingExporter(); var scheduler = scheduler(executor, new ArrayList<>(), exporter); - scheduler.schedule(record("older")); + scheduler.schedule(ARN, record("older")); executor.reject = false; - scheduler.schedule(record("newer")); + scheduler.schedule(ARN, record("newer")); executor.runAll(); assertEquals(List.of("newer"), statuses(exporter), "the retry exports the latest record"); - scheduler.drain(); + scheduler.drain(ARN); assertEquals(1, exporter.records.size()); } @@ -273,14 +280,14 @@ void aDrainThatObservedTheHandleBeforeTheWorkerWasRejectedStillCompletesInline() var exporter = new CapturingExporter(); var scheduler = scheduler(blockingRejector, failures, exporter); - var scheduling = new Thread(() -> scheduler.schedule(record("final")), "scheduling"); + var scheduling = new Thread(() -> scheduler.schedule(ARN, record("final")), "scheduling"); scheduling.start(); assertTrue(submitted.await(5, TimeUnit.SECONDS), "the pump handle is published before execute rejects"); var drained = new CountDownLatch(1); var drainer = new Thread( () -> { - scheduler.drain(); + scheduler.drain(ARN); drained.countDown(); }, "drainer"); @@ -296,7 +303,7 @@ void aDrainThatObservedTheHandleBeforeTheWorkerWasRejectedStillCompletesInline() } @Test - void flushAllRunsExporterFlushesConcurrentlySoASlowFlushDoesNotDelayTheOthers() throws Exception { + void flushRunsExporterFlushesConcurrentlySoASlowFlushDoesNotDelayTheOthers() throws Exception { var release = new CountDownLatch(1); var fastFlushed = new CountDownLatch(1); var slow = new InsightExporter() { @@ -321,19 +328,19 @@ public void flush() { var flushed = new CountDownLatch(1); new Thread(() -> { - scheduler.flushAll(); + scheduler.flush(); flushed.countDown(); }) .start(); assertTrue(fastFlushed.await(5, TimeUnit.SECONDS), "fast exporter flushed while the slow one is blocked"); - assertFalse(flushed.await(100, TimeUnit.MILLISECONDS), "flushAll waits for every exporter"); + assertFalse(flushed.await(100, TimeUnit.MILLISECONDS), "flush waits for every exporter"); release.countDown(); assertTrue(flushed.await(5, TimeUnit.SECONDS)); } @Test - void flushAllIsolatesAFailingFlush() { + void flushIsolatesAFailingFlush() { var failures = new CopyOnWriteArrayList(); var flushed = new CountDownLatch(1); var bad = new InsightExporter() { @@ -356,7 +363,7 @@ public void flush() { }; var scheduler = scheduler(sharedWorkers(), failures, bad, good); - scheduler.flushAll(); + scheduler.flush(); assertEquals(0, flushed.getCount(), "the healthy exporter still flushed"); assertEquals(1, failures.size()); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/PluginThrowableContainmentTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/PluginThrowableContainmentTest.java index 4a13268be..721d40d5a 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/PluginThrowableContainmentTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/PluginThrowableContainmentTest.java @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 package software.amazon.lambda.durable.insight; +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -83,6 +84,34 @@ private InvocationEndInfo end(Object input) { "req", ARN, true, START, ops("compute"), InvocationStatus.SUCCEEDED, null, input, "out"); } + @Test + void aNullExecutionArnEscapesNoHook() { + // The SDK's contract for these hooks is that a plugin fault never disrupts durable execution, so an input the + // plugin cannot key its per-execution state by must be contained rather than thrown back. onInvocationEnd is + // the case that matters: its state removal runs last, in a `finally`, and a ConcurrentHashMap cannot remove a + // null key. + var exporter = new CapturingExporter(); + DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()); + + InvocationInfo nullStart = new InvocationInfo("req", null, true, START, "in", ops("compute"), Map.of()); + InvocationEndInfo nullEnd = new InvocationEndInfo( + "req", null, true, START, ops("compute"), InvocationStatus.SUCCEEDED, null, "in", "out"); + + assertDoesNotThrow(() -> plugin.onInvocationStart(nullStart), "onInvocationStart must contain a null ARN"); + assertDoesNotThrow( + () -> plugin.onOperationChange(new software.amazon.lambda.durable.plugin.OperationChangeInfo( + "req", null, ops("compute"), ops("compute"))), + "onOperationChange must contain a null ARN"); + assertDoesNotThrow(() -> plugin.onInvocationEnd(nullEnd), "onInvocationEnd must contain a null ARN"); + + // The plugin is still usable afterwards: a well-formed execution on the same instance still emits and flushes. + plugin.onInvocationStart(start("in")); + plugin.onInvocationEnd(end("in")); + assertEquals(1, exporter.records.size(), "the plugin still works after a null-ARN invocation"); + assertTrue(exporter.flushes > 0); + } + @Test void exporterThrowingErrorIsIsolatedAndLaterExportersStillReceiveAndFlush() { var throwing = new InsightExporter() { diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightFlushCadenceTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightFlushCadenceTest.java new file mode 100644 index 000000000..72bdcc0c4 --- /dev/null +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightFlushCadenceTest.java @@ -0,0 +1,222 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.time.Instant; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.Test; +import software.amazon.awssdk.services.lambda.model.OperationStatus; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationInfo; +import software.amazon.lambda.durable.plugin.InvocationStatus; +import software.amazon.lambda.durable.plugin.OperationChangeInfo; +import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; + +/** + * Pins the flush cadence at the plugin boundary: every invocation end that reaches the exporters flushes them — at most + * once, and exactly once when ends do not overlap — including the ends that emit no record, while a sampled-out + * execution flushes not at all. Ends that overlap may share one flush; no end's record is ever left unflushed. + */ +class WorkflowInsightFlushCadenceTest { + + private static final Instant START = Instant.parse("2026-08-05T00:00:00Z"); + + private static String arn(int index) { + return "arn:aws:lambda:us-west-2:111122223333:function:f:$LATEST/durable-execution/exec-" + index + "/inv-1"; + } + + /** Counts exports and flushes, and remembers how many exports had happened when each flush ran. */ + private static final class CountingExporter implements InsightExporter { + final List exported = new CopyOnWriteArrayList<>(); + final AtomicInteger flushes = new AtomicInteger(); + final List exportsAtFlush = new CopyOnWriteArrayList<>(); + + @Override + public void export(WorkflowInsightRecord record) { + exported.add(record.status() + "@" + record.executionArn()); + } + + @Override + public void flush() { + flushes.incrementAndGet(); + exportsAtFlush.add(exported.size()); + } + } + + private static WorkflowInsight.InsightPlugin plugin( + WorkflowInsightConfig.EmitMode mode, Double samplingRate, CountingExporter... exporters) { + var builder = WorkflowInsightConfig.builder().emitMode(mode); + for (CountingExporter exporter : exporters) { + builder = builder.addExporter(exporter); + } + if (samplingRate != null) { + builder = builder.samplingRate(samplingRate); + } + return (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(builder.build()); + } + + @Test + void anInvocationEndThatEmitsARecordFlushesEveryExporterExactlyOnce() { + var first = new CountingExporter(); + var second = new CountingExporter(); + var plugin = plugin(WorkflowInsightConfig.EmitMode.ON_COMPLETE, null, first, second); + + plugin.onInvocationStart(start(arn(0))); + plugin.onInvocationEnd(end(arn(0), InvocationStatus.SUCCEEDED)); + + for (CountingExporter exporter : List.of(first, second)) { + assertEquals(List.of("SUCCEEDED@" + arn(0)), exporter.exported); + assertEquals(1, exporter.flushes.get(), "exactly one flush per invocation end"); + assertEquals(List.of(1), exporter.exportsAtFlush, "the flush follows the record it is meant to flush"); + } + } + + @Test + void anInvocationEndThatEmitsNothingStillFlushesEveryExporterExactlyOnce() { + // ON_COMPLETE + a non-terminal suspend, and ON_FAILURE + a success: both are sampled in, both emit no record, + // and both must still flush — a buffering exporter's earlier records depend on it. + record Case(String name, WorkflowInsightConfig.EmitMode mode, InvocationStatus status) {} + List cases = List.of( + new Case("ON_COMPLETE + PENDING", WorkflowInsightConfig.EmitMode.ON_COMPLETE, InvocationStatus.PENDING), + new Case( + "ON_COMPLETE + RETRYING", + WorkflowInsightConfig.EmitMode.ON_COMPLETE, + InvocationStatus.RETRYING), + new Case( + "ON_FAILURE + SUCCEEDED", + WorkflowInsightConfig.EmitMode.ON_FAILURE, + InvocationStatus.SUCCEEDED)); + + for (Case scenario : cases) { + var exporter = new CountingExporter(); + var plugin = plugin(scenario.mode(), null, exporter); + plugin.onInvocationStart(start(arn(1))); + plugin.onInvocationEnd(end(arn(1), scenario.status())); + + assertEquals(List.of(), exporter.exported, scenario.name() + ": no record should be emitted"); + assertEquals(1, exporter.flushes.get(), scenario.name() + ": the flush must happen anyway"); + } + } + + @Test + void everyInvocationEndOfAWarmEnvironmentFlushesExactlyOnce() { + var exporter = new CountingExporter(); + var plugin = plugin(WorkflowInsightConfig.EmitMode.ON_CHANGE, null, exporter); + + int invocations = 5; + for (int i = 0; i < invocations; i++) { + plugin.onInvocationStart(start(arn(i))); + plugin.onOperationChange(change(arn(i))); + plugin.onInvocationEnd(end(arn(i), InvocationStatus.SUCCEEDED)); + // Sequential ends have nothing to share a flush with, so the cadence bound is tight here. + assertEquals(i + 1, exporter.flushes.get(), "one flush per invocation end, never skipped"); + } + assertEquals(invocations, exporter.flushes.get()); + assertTrue(exporter.exported.size() >= invocations, "each execution's terminal record was exported"); + } + + @Test + void invocationEndsThatOverlapMayShareAFlushButNoneIsLeftUnflushed() { + var exporter = new CountingExporter(); + var plugin = plugin(WorkflowInsightConfig.EmitMode.ON_COMPLETE, null, exporter); + + int executions = 8; + var barrier = new CyclicBarrier(executions); + var done = new CountDownLatch(executions); + for (int i = 0; i < executions; i++) { + String executionArn = arn(100 + i); + var thread = new Thread( + () -> { + plugin.onInvocationStart(start(executionArn)); + try { + barrier.await(60, TimeUnit.SECONDS); + } catch (Exception e) { + throw new AssertionError(e); + } + plugin.onInvocationEnd(end(executionArn, InvocationStatus.SUCCEEDED)); + done.countDown(); + }, + "overlapping-end-" + i); + thread.setDaemon(true); + thread.start(); + } + try { + assertTrue(done.await(60, TimeUnit.SECONDS), "an invocation end never returned"); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + + assertEquals(executions, exporter.exported.size(), "every execution's terminal record must be exported"); + assertTrue(exporter.flushes.get() >= 1, "the ends must be covered by at least one flush"); + assertTrue( + exporter.flushes.get() <= executions, + "at most one flush per invocation end: " + exporter.flushes.get() + " for " + executions); + // Every record must be followed by a flush: each end's own request is served by a flush that starts after its + // record was exported, so the last flush cannot precede the last export. + assertEquals( + executions, + exporter.exportsAtFlush.get(exporter.exportsAtFlush.size() - 1), + "the last flush ran after every terminal record: " + exporter.exportsAtFlush); + } + + @Test + void aSampledOutExecutionFlushesNothing() { + // Unchanged by the move of flush onto the export pump: a sampled-out end never schedules a record, so it + // neither drains nor flushes. + var exporter = new CountingExporter(); + var plugin = plugin(WorkflowInsightConfig.EmitMode.ON_CHANGE, 0.0, exporter); + + for (int i = 0; i < 10; i++) { + plugin.onInvocationStart(start(arn(i))); + plugin.onOperationChange(change(arn(i))); + plugin.onInvocationEnd(end(arn(i), InvocationStatus.SUCCEEDED)); + } + + assertEquals(List.of(), exporter.exported); + assertEquals(0, exporter.flushes.get(), "a sampled-out invocation end neither drains nor flushes"); + assertEquals(0, plugin.retainedStateCount()); + } + + private static Map ops() { + Map operations = new LinkedHashMap<>(); + operations.put( + "op-1", + new OperationChangeItemInfo( + "op-1", + "greet", + "STEP", + "Step", + null, + START, + START.plusMillis(5), + OperationStatus.SUCCEEDED, + 1, + false, + null, + null)); + return operations; + } + + private static InvocationInfo start(String executionArn) { + return new InvocationInfo("req", executionArn, true, START, "in", ops(), Map.of()); + } + + private static OperationChangeInfo change(String executionArn) { + return new OperationChangeInfo("req", executionArn, ops(), ops()); + } + + private static InvocationEndInfo end(String executionArn, InvocationStatus status) { + return new InvocationEndInfo("req", executionArn, true, START, ops(), status, null, "in", "out"); + } +} From 6cca5434eabe54d85f0c08613b2938245c70d29e Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Thu, 17 Sep 2026 06:45:15 -0700 Subject: [PATCH 02/19] refactor(plugin)!: create one plugin instance per invocation A plugin instance used to live as long as the execution environment while serving every execution that landed on it. Under Lambda Managed Instances several executions run concurrently in one environment, so every plugin had to key its own state by execution ARN and clean that map up itself. Getting that wrong loses telemetry silently, and both bundled plugins had got it wrong. The SDK now builds one plugin per invocation and drops it when the invocation ends. `withPlugins` takes factories, not instances: .withPlugins(info -> new MyPlugin(sharedExporter, info.durableExecutionArn())) `DurableExecutionPluginFactory` is a functional interface, so a lambda or a method reference works directly. Environment-lifetime state stays in the enclosing object; per-invocation state becomes plain instance fields. This deletes the provider path rather than adding a parallel one. While an instance path exists a plugin cannot delete its ARN-keyed map, which is the entire point of the change. `DurableExecutionPluginProvider` now extends the factory and keeps only `getName()`; `getApiVersion()`, `getPluginType()`, and the no-arg `createPlugin()` are removed. A provider is therefore itself the per-invocation factory, so `ServiceLoader` discovery works unchanged. Workflow Insight now holds no ARN-keyed structure at all: `WorkflowInsight` becomes a thin factory over the new `InsightPlugin` and `InsightSettings`, and `ExecutionState` is deleted. The OTel plugin resolves its tracer provider once per environment in the new `OtelPluginEnvironment` and makes every per-invocation field `final`, assigned in the constructor. Constructor assignment is what makes dropping `volatile` safe: `PluginRunner` publishes the plugin list with a volatile write before firing `onInvocationStart`, so a field written in that hook is not guaranteed visible to a checkpoint or operation thread that already existed, whereas a field written in the constructor is. BREAKING CHANGE: `withPlugins` accepts `DurableExecutionPluginFactory` instead of `DurableExecutionPlugin`, and the provider interface loses `getApiVersion()`, `getPluginType()`, and the no-arg `createPlugin()`. Pass `info -> new MyPlugin()` where you passed `new MyPlugin()`, and implement `createPlugin(InvocationInfo)` in a provider. --- .../otel/OtelConformanceHandler.java | 10 +- .../java/plugin/PluginAttemptHooksRetry.java | 2 +- .../java/plugin/PluginErrorIsolation.java | 2 +- .../plugin/PluginExternalUpdateOnInvoke.java | 4 +- .../java/plugin/PluginFaultyAndHealthy.java | 2 +- .../plugin/PluginFirstInvocationFlag.java | 2 +- .../plugin/PluginInvocationLifecycle.java | 2 +- .../java/plugin/PluginMultiplePlugins.java | 4 +- .../plugin/PluginNestedParentLinkage.java | 4 +- .../java/plugin/PluginOperationChange.java | 2 +- .../java/plugin/PluginOperationLifecycle.java | 2 +- .../plugin/PluginParallelBranchHooks.java | 4 +- .../main/java/plugin/PluginReplayFlags.java | 4 +- .../java/plugin/PluginRetryExhaustion.java | 2 +- .../plugin/PluginSuspensionInvocationEnd.java | 4 +- .../java/plugin/PluginTerminalFailure.java | 2 +- .../java/plugin/PluginTerminalPayloads.java | 2 +- .../java/plugin/PluginWaitOperationHooks.java | 4 +- .../java/plugin/PluginWaitReplayFlag.java | 4 +- docs/advanced/configuration.md | 18 +- .../examples/general/PluginExample.java | 2 +- .../durable/insight/ExportScheduler.java | 632 ++++++++++-------- .../lambda/durable/insight/InsightPlugin.java | 382 +++++++++++ .../durable/insight/InsightSettings.java | 59 ++ .../durable/insight/WorkflowInsight.java | 450 ++----------- .../ConcurrentExecutionsExportTest.java | 53 +- .../durable/insight/ErrorPrivacyGateTest.java | 14 +- .../lambda/durable/insight/Executions.java | 77 +++ .../ExportSchedulerFlushCoalescingTest.java | 15 +- ...ExportSchedulerFlushSerializationTest.java | 17 +- .../ExportSchedulerReentrantFlushTest.java | 10 +- .../durable/insight/ExportSchedulerTest.java | 72 +- .../insight/ExporterIsolationTest.java | 19 +- .../durable/insight/InputSnapshotTest.java | 25 +- .../durable/insight/JsonJavaTimeTest.java | 11 +- .../insight/MutableNumberIsolationTest.java | 13 +- .../insight/OperationErrorIdentityTest.java | 15 +- .../insight/OperationOrderingTest.java | 11 +- .../PluginThrowableContainmentTest.java | 96 +-- .../insight/StateCleanupLifecycleTest.java | 99 ++- .../insight/TransformContractTest.java | 32 +- .../insight/UnrecoverableErrorUnwrapTest.java | 26 +- .../WorkflowInsightFlushCadenceTest.java | 34 +- .../insight/WorkflowInsightHookTest.java | 79 +-- .../insight/WorkflowInsightPluginTest.java | 4 +- otel-plugin/README.md | 49 +- .../durable/otel/ExecutionOtelPlugin.java | 319 +++++---- .../otel/ExecutionOtelPluginProvider.java | 21 +- .../durable/otel/InvocationOtelPlugin.java | 317 +++++---- .../otel/InvocationOtelPluginProvider.java | 21 +- .../lambda/durable/otel/OtelPluginConfig.java | 6 +- .../durable/otel/OtelPluginEnvironment.java | 92 +++ .../durable/otel/OtelPluginSupport.java | 4 +- .../durable/otel/DurableSamplerTest.java | 9 +- .../ExecutionOtelPluginIntegrationTest.java | 10 +- .../durable/otel/ExecutionOtelPluginTest.java | 159 +++-- .../InvocationOtelPluginIntegrationTest.java | 23 +- .../otel/InvocationOtelPluginTest.java | 194 +++--- .../lambda/durable/otel/Invocations.java | 27 + .../durable/otel/MdcSpanEnricherTest.java | 44 +- .../lambda/durable/PluginIntegrationTest.java | 102 +-- .../testing/LocalDurableTestRunner.java | 4 +- .../testing/LocalDurableTestRunnerTest.java | 3 +- .../amazon/lambda/durable/DurableConfig.java | 50 +- .../lambda/durable/DynamicPluginLoader.java | 78 +-- .../durable/execution/DurableExecutor.java | 4 +- .../durable/execution/ExecutionManager.java | 29 +- .../operation/BaseDurableOperation.java | 9 +- .../plugin/DurableExecutionPluginFactory.java | 35 + .../DurableExecutionPluginProvider.java | 36 +- .../lambda/durable/plugin/PluginRunner.java | 74 +- .../lambda/durable/DurableConfigTest.java | 60 +- .../durable/DynamicPluginLoaderTest.java | 180 ++--- .../BaseDurableOperationPluginTest.java | 21 +- .../durable/plugin/PluginRunnerTest.java | 186 +++++- 75 files changed, 2649 insertions(+), 1844 deletions(-) create mode 100644 insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightPlugin.java create mode 100644 insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightSettings.java create mode 100644 insight-plugin/src/test/java/software/amazon/lambda/durable/insight/Executions.java create mode 100644 otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java create mode 100644 otel-plugin/src/test/java/software/amazon/lambda/durable/otel/Invocations.java create mode 100644 sdk/src/main/java/software/amazon/lambda/durable/plugin/DurableExecutionPluginFactory.java diff --git a/conformance-tests-otel/src/main/java/software/amazon/lambda/durable/conformance/otel/OtelConformanceHandler.java b/conformance-tests-otel/src/main/java/software/amazon/lambda/durable/conformance/otel/OtelConformanceHandler.java index 9c4bc01e3..0151d0d14 100644 --- a/conformance-tests-otel/src/main/java/software/amazon/lambda/durable/conformance/otel/OtelConformanceHandler.java +++ b/conformance-tests-otel/src/main/java/software/amazon/lambda/durable/conformance/otel/OtelConformanceHandler.java @@ -9,7 +9,7 @@ import software.amazon.lambda.durable.TypeToken; import software.amazon.lambda.durable.otel.ExecutionOtelPlugin; import software.amazon.lambda.durable.otel.InvocationOtelPlugin; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; /** * Shared base for the OTel conformance suite's handlers. Ported from the otel-invocation/otel-execution examples in @@ -26,13 +26,13 @@ protected OtelConformanceHandler() { @Override protected final DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(createPlugin()).build(); + return DurableConfig.builder().withPlugins(createPluginFactory()).build(); } - private DurableExecutionPlugin createPlugin() { + private DurableExecutionPluginFactory createPluginFactory() { return "execution".equals(System.getenv("OTEL_PLUGIN_MODE")) - ? new ExecutionOtelPlugin() - : new InvocationOtelPlugin(); + ? ExecutionOtelPlugin.factory() + : InvocationOtelPlugin.factory(); } protected final void requireScenario(Map event, String expected) { diff --git a/conformance-tests/src/main/java/plugin/PluginAttemptHooksRetry.java b/conformance-tests/src/main/java/plugin/PluginAttemptHooksRetry.java index 7a15a5957..1b304429d 100644 --- a/conformance-tests/src/main/java/plugin/PluginAttemptHooksRetry.java +++ b/conformance-tests/src/main/java/plugin/PluginAttemptHooksRetry.java @@ -22,7 +22,7 @@ public class PluginAttemptHooksRetry extends DurableHandler { @Override protected DurableConfig createConfiguration() { return DurableConfig.builder() - .withPlugins(new ConformanceLoggingPlugin("CONFPLUGIN")) + .withPlugins(info -> new ConformanceLoggingPlugin("CONFPLUGIN")) .build(); } diff --git a/conformance-tests/src/main/java/plugin/PluginErrorIsolation.java b/conformance-tests/src/main/java/plugin/PluginErrorIsolation.java index 2ea0fce4c..9df4b7f8e 100644 --- a/conformance-tests/src/main/java/plugin/PluginErrorIsolation.java +++ b/conformance-tests/src/main/java/plugin/PluginErrorIsolation.java @@ -18,7 +18,7 @@ public class PluginErrorIsolation extends DurableHandler { @Override protected DurableConfig createConfiguration() { return DurableConfig.builder() - .withPlugins(new FaultyConformancePlugin()) + .withPlugins(info -> new FaultyConformancePlugin()) .build(); } diff --git a/conformance-tests/src/main/java/plugin/PluginExternalUpdateOnInvoke.java b/conformance-tests/src/main/java/plugin/PluginExternalUpdateOnInvoke.java index 6a5669108..4bbf21c8f 100644 --- a/conformance-tests/src/main/java/plugin/PluginExternalUpdateOnInvoke.java +++ b/conformance-tests/src/main/java/plugin/PluginExternalUpdateOnInvoke.java @@ -23,7 +23,9 @@ public class PluginExternalUpdateOnInvoke extends DurableHandler @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new UpdatedOnInvokePlugin()).build(); + return DurableConfig.builder() + .withPlugins(info -> new UpdatedOnInvokePlugin()) + .build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginFaultyAndHealthy.java b/conformance-tests/src/main/java/plugin/PluginFaultyAndHealthy.java index 9393f5527..0823a8183 100644 --- a/conformance-tests/src/main/java/plugin/PluginFaultyAndHealthy.java +++ b/conformance-tests/src/main/java/plugin/PluginFaultyAndHealthy.java @@ -29,7 +29,7 @@ public class PluginFaultyAndHealthy extends DurableHandler { @Override protected DurableConfig createConfiguration() { return DurableConfig.builder() - .withPlugins(new FaultyPlugin(), new HealthyPlugin()) + .withPlugins(info -> new FaultyPlugin(), info -> new HealthyPlugin()) .build(); } diff --git a/conformance-tests/src/main/java/plugin/PluginFirstInvocationFlag.java b/conformance-tests/src/main/java/plugin/PluginFirstInvocationFlag.java index 016f7e3a4..bdb74936e 100644 --- a/conformance-tests/src/main/java/plugin/PluginFirstInvocationFlag.java +++ b/conformance-tests/src/main/java/plugin/PluginFirstInvocationFlag.java @@ -19,7 +19,7 @@ public class PluginFirstInvocationFlag extends DurableHandler { @Override protected DurableConfig createConfiguration() { return DurableConfig.builder() - .withPlugins(new ConformanceLoggingPlugin("CONFPLUGIN")) + .withPlugins(info -> new ConformanceLoggingPlugin("CONFPLUGIN")) .build(); } diff --git a/conformance-tests/src/main/java/plugin/PluginInvocationLifecycle.java b/conformance-tests/src/main/java/plugin/PluginInvocationLifecycle.java index d75f84683..bef2dc5c3 100644 --- a/conformance-tests/src/main/java/plugin/PluginInvocationLifecycle.java +++ b/conformance-tests/src/main/java/plugin/PluginInvocationLifecycle.java @@ -18,7 +18,7 @@ public class PluginInvocationLifecycle extends DurableHandler { @Override protected DurableConfig createConfiguration() { return DurableConfig.builder() - .withPlugins(new ConformanceLoggingPlugin("CONFPLUGIN")) + .withPlugins(info -> new ConformanceLoggingPlugin("CONFPLUGIN")) .build(); } diff --git a/conformance-tests/src/main/java/plugin/PluginMultiplePlugins.java b/conformance-tests/src/main/java/plugin/PluginMultiplePlugins.java index 6ecce36cc..c033152e7 100644 --- a/conformance-tests/src/main/java/plugin/PluginMultiplePlugins.java +++ b/conformance-tests/src/main/java/plugin/PluginMultiplePlugins.java @@ -52,7 +52,9 @@ public void onInvocationEnd(InvocationEndInfo info) { @Override protected DurableConfig createConfiguration() { return DurableConfig.builder() - .withPlugins(new InvocationLoggingPlugin("CONFPLUGIN-A"), new InvocationLoggingPlugin("CONFPLUGIN-B")) + .withPlugins( + info -> new InvocationLoggingPlugin("CONFPLUGIN-A"), + info -> new InvocationLoggingPlugin("CONFPLUGIN-B")) .build(); } diff --git a/conformance-tests/src/main/java/plugin/PluginNestedParentLinkage.java b/conformance-tests/src/main/java/plugin/PluginNestedParentLinkage.java index c8dc2e0b1..dcb28823d 100644 --- a/conformance-tests/src/main/java/plugin/PluginNestedParentLinkage.java +++ b/conformance-tests/src/main/java/plugin/PluginNestedParentLinkage.java @@ -20,7 +20,9 @@ public class PluginNestedParentLinkage extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new ParentLinkagePlugin()).build(); + return DurableConfig.builder() + .withPlugins(info -> new ParentLinkagePlugin()) + .build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginOperationChange.java b/conformance-tests/src/main/java/plugin/PluginOperationChange.java index 05c7d1fa3..790ef9ffc 100644 --- a/conformance-tests/src/main/java/plugin/PluginOperationChange.java +++ b/conformance-tests/src/main/java/plugin/PluginOperationChange.java @@ -21,7 +21,7 @@ public class PluginOperationChange extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new ChangePlugin()).build(); + return DurableConfig.builder().withPlugins(info -> new ChangePlugin()).build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginOperationLifecycle.java b/conformance-tests/src/main/java/plugin/PluginOperationLifecycle.java index 8a51e8296..a1b4cf047 100644 --- a/conformance-tests/src/main/java/plugin/PluginOperationLifecycle.java +++ b/conformance-tests/src/main/java/plugin/PluginOperationLifecycle.java @@ -18,7 +18,7 @@ public class PluginOperationLifecycle extends DurableHandler { @Override protected DurableConfig createConfiguration() { return DurableConfig.builder() - .withPlugins(new ConformanceLoggingPlugin("CONFPLUGIN")) + .withPlugins(info -> new ConformanceLoggingPlugin("CONFPLUGIN")) .build(); } diff --git a/conformance-tests/src/main/java/plugin/PluginParallelBranchHooks.java b/conformance-tests/src/main/java/plugin/PluginParallelBranchHooks.java index ec1e949c7..d44016243 100644 --- a/conformance-tests/src/main/java/plugin/PluginParallelBranchHooks.java +++ b/conformance-tests/src/main/java/plugin/PluginParallelBranchHooks.java @@ -27,7 +27,9 @@ public class PluginParallelBranchHooks extends DurableHandler new BranchHooksPlugin()) + .build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginReplayFlags.java b/conformance-tests/src/main/java/plugin/PluginReplayFlags.java index 69b3bf9d5..031c7a1dd 100644 --- a/conformance-tests/src/main/java/plugin/PluginReplayFlags.java +++ b/conformance-tests/src/main/java/plugin/PluginReplayFlags.java @@ -26,7 +26,9 @@ public class PluginReplayFlags extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new ReplayFlagPlugin()).build(); + return DurableConfig.builder() + .withPlugins(info -> new ReplayFlagPlugin()) + .build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginRetryExhaustion.java b/conformance-tests/src/main/java/plugin/PluginRetryExhaustion.java index 2f334f06c..3f311200e 100644 --- a/conformance-tests/src/main/java/plugin/PluginRetryExhaustion.java +++ b/conformance-tests/src/main/java/plugin/PluginRetryExhaustion.java @@ -27,7 +27,7 @@ public class PluginRetryExhaustion extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new AttemptPlugin()).build(); + return DurableConfig.builder().withPlugins(info -> new AttemptPlugin()).build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginSuspensionInvocationEnd.java b/conformance-tests/src/main/java/plugin/PluginSuspensionInvocationEnd.java index 08d7d0797..67b964fbc 100644 --- a/conformance-tests/src/main/java/plugin/PluginSuspensionInvocationEnd.java +++ b/conformance-tests/src/main/java/plugin/PluginSuspensionInvocationEnd.java @@ -23,7 +23,9 @@ public class PluginSuspensionInvocationEnd extends DurableHandler new InvocationEndPlugin()) + .build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginTerminalFailure.java b/conformance-tests/src/main/java/plugin/PluginTerminalFailure.java index f2c4cb454..bd0cf9bb2 100644 --- a/conformance-tests/src/main/java/plugin/PluginTerminalFailure.java +++ b/conformance-tests/src/main/java/plugin/PluginTerminalFailure.java @@ -20,7 +20,7 @@ public class PluginTerminalFailure extends DurableHandler { @Override protected DurableConfig createConfiguration() { return DurableConfig.builder() - .withPlugins(new ConformanceLoggingPlugin("CONFPLUGIN")) + .withPlugins(info -> new ConformanceLoggingPlugin("CONFPLUGIN")) .build(); } diff --git a/conformance-tests/src/main/java/plugin/PluginTerminalPayloads.java b/conformance-tests/src/main/java/plugin/PluginTerminalPayloads.java index 4ed57f200..0b1fc8290 100644 --- a/conformance-tests/src/main/java/plugin/PluginTerminalPayloads.java +++ b/conformance-tests/src/main/java/plugin/PluginTerminalPayloads.java @@ -28,7 +28,7 @@ public class PluginTerminalPayloads extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new PayloadPlugin()).build(); + return DurableConfig.builder().withPlugins(info -> new PayloadPlugin()).build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginWaitOperationHooks.java b/conformance-tests/src/main/java/plugin/PluginWaitOperationHooks.java index 95f5613b1..ec4c7a98c 100644 --- a/conformance-tests/src/main/java/plugin/PluginWaitOperationHooks.java +++ b/conformance-tests/src/main/java/plugin/PluginWaitOperationHooks.java @@ -23,7 +23,9 @@ public class PluginWaitOperationHooks extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new WaitHooksPlugin()).build(); + return DurableConfig.builder() + .withPlugins(info -> new WaitHooksPlugin()) + .build(); } @Override diff --git a/conformance-tests/src/main/java/plugin/PluginWaitReplayFlag.java b/conformance-tests/src/main/java/plugin/PluginWaitReplayFlag.java index 18cc01613..09864daec 100644 --- a/conformance-tests/src/main/java/plugin/PluginWaitReplayFlag.java +++ b/conformance-tests/src/main/java/plugin/PluginWaitReplayFlag.java @@ -35,7 +35,9 @@ public class PluginWaitReplayFlag extends DurableHandler> { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new WaitReplayFlagPlugin()).build(); + return DurableConfig.builder() + .withPlugins(info -> new WaitReplayFlagPlugin()) + .build(); } @Override diff --git a/docs/advanced/configuration.md b/docs/advanced/configuration.md index bc8bf89d0..eaa936e8c 100644 --- a/docs/advanced/configuration.md +++ b/docs/advanced/configuration.md @@ -50,7 +50,7 @@ DURABLE_EXECUTION_PLUGINS=otel-invocation,com.example.audit When the variable is unset or blank, the SDK does not perform provider discovery. During `DurableConfig` construction, the SDK uses `ServiceLoader` and the thread context class loader to find `DurableExecutionPluginProvider` implementations. Only named providers create plugins. -Dynamically loaded plugins run first in the order listed in `DURABLE_EXECUTION_PLUGINS`. Plugins registered through `withPlugins(...)` follow in configuration order. Both sources are additive: if the same plugin type is selected dynamically and registered explicitly, both instances are registered and receive lifecycle hooks. Duplicate configured provider names, duplicate discovered provider names, missing providers, incompatible provider API versions, invalid plugin types, and provider construction failures stop configuration with an `IllegalStateException`. +Dynamically loaded plugins run first in the order listed in `DURABLE_EXECUTION_PLUGINS`. Plugins registered through `withPlugins(...)` follow in configuration order. Both sources are additive: if the same plugin is selected dynamically and registered explicitly, both factories are registered and each produces an instance that receives lifecycle hooks. Duplicate configured provider names, duplicate discovered provider names, missing providers, and provider discovery failures stop configuration with an `IllegalStateException`. To distribute a provider in a Lambda layer, package its JAR under `java/lib`: @@ -67,7 +67,7 @@ The provider JAR must contain: META-INF/services/software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider ``` -The service file contains the provider implementation class name. A minimal provider looks like: +The service file contains the provider implementation class name. A provider is itself the per-invocation plugin factory: the SDK calls `createPlugin(InvocationInfo)` once per Lambda invocation and drops the returned instance when that invocation returns, so the instance can hold its invocation's state in plain fields. A minimal provider looks like: ```java public final class AuditPluginProvider implements DurableExecutionPluginProvider { @@ -77,18 +77,8 @@ public final class AuditPluginProvider implements DurableExecutionPluginProvider } @Override - public int getApiVersion() { - return API_VERSION; - } - - @Override - public Class getPluginType() { - return AuditPlugin.class; - } - - @Override - public DurableExecutionPlugin createPlugin() { - return new AuditPlugin(); + public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { + return new AuditPlugin(invocationInfo.durableExecutionArn()); } } ``` diff --git a/examples/src/main/java/software/amazon/lambda/durable/examples/general/PluginExample.java b/examples/src/main/java/software/amazon/lambda/durable/examples/general/PluginExample.java index 28c6675d4..e02fbcf66 100644 --- a/examples/src/main/java/software/amazon/lambda/durable/examples/general/PluginExample.java +++ b/examples/src/main/java/software/amazon/lambda/durable/examples/general/PluginExample.java @@ -33,7 +33,7 @@ public class PluginExample extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new LoggingPlugin()).build(); + return DurableConfig.builder().withPlugins(info -> new LoggingPlugin()).build(); } @Override diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java index 69ba94b21..3cc3562a3 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java @@ -5,17 +5,15 @@ import java.util.ArrayDeque; import java.util.ArrayList; import java.util.Deque; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Iterator; -import java.util.LinkedHashMap; +import java.util.LinkedHashSet; import java.util.List; -import java.util.Map; import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.concurrent.Executor; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import java.util.function.BiConsumer; @@ -25,40 +23,54 @@ * Serializes record exports so that, at most, one export runs at a time, while keeping the records of concurrently * running executions independent. * - *

One plugin instance — and therefore one scheduler — serves the whole execution environment, and an environment can - * host several durable executions at the same time (Lambda Managed Instances makes that routine). So the pending work - * is keyed by execution ARN: each execution has its own latest-record slot, and coalescing happens only within - * one execution. + *

One scheduler serves the whole execution environment — the exporters it fans out to belong to the environment, and + * an environment can host several durable executions at the same time (Lambda Managed Instances makes that routine). + * The work, by contrast, is held on the invocation it belongs to: each {@link InsightPlugin} instance is one + * invocation's plugin and owns that invocation's latest-record slot, drain signal, mid-export marker and drain-waiter + * count. Coalescing happens only within one invocation, because the slot belongs to it. * - *

Each {@link WorkflowInsightRecord} is a complete snapshot of its execution, so a newer record for the same - * execution fully supersedes one still waiting to be exported. While an export is in flight, additional updates for - * that execution are coalesced into its slot — intermediate records are dropped because the latest one already contains - * all of their information. A record for a different execution never displaces another execution's record. + *

That shape is not a tidying-up. The same facts once lived in five structures keyed by execution ARN — the plugin's + * state map plus this class's {@code pending}, {@code settled}, {@code exporting} and {@code drainWaiters} — and two of + * them could disagree about one execution. They did: "nothing queued for this ARN" was read as "nothing outstanding", + * which is equally true of a record already taken and being exported, so an exiting pump completed another execution's + * drain signal mid-export and that invocation returned before its record was delivered. Every fact now exists exactly + * once, as a field of the one object the SDK gave that invocation, so the disagreement has no place to happen — and + * since the SDK creates that object and drops it, the scheduler has no execution registry to keep in step with + * anything. * - *

A single pump exports the queued records one at a time, in the order the executions first queued work, so - * exporters still never see two exports at once and each record keeps its per-exporter fan-out. {@link #flush()} - * requests are served by that same pump, between records, so an exporter never sees a {@code flush()} overlap an - * {@code export()} either. Requests are served as a batch — the cadence is at most one flush per requesting invocation - * end, not exactly one — and a flush is preceded by the queued records a {@link #drain(String)} is waiting for, so a - * burst of invocation ends is covered by one flush rather than one each. A request made while a flush runs waits for - * the next turn. Exports are otherwise fire-and-forget; {@link #drain(String)} is called before an invocation returns - * and waits for that execution's own latest record to reach every exporter. + *

Each {@link WorkflowInsightRecord} is a complete snapshot of its execution, so a newer record from the same + * invocation fully supersedes one still waiting to be exported. While an export is in flight, additional updates from + * that invocation are coalesced into its slot — intermediate records are dropped because the latest one already + * contains all of their information. A record from a different invocation never displaces another's record. * - *

Because there is one pump, that wait can also cover records other executions had already queued ahead of this one: - * a drain is not isolated from the queue's head-of-line cost. What the per-execution keying guarantees is that another - * execution's record can never displace this one — records coalesce only within their own execution, and a - * drain cannot return until this execution's own latest record has reached every exporter. + *

A single pump exports the queued records one at a time, in the order the invocations first queued work + * ({@link #queue}, which is ordering only — membership in it is the same fact as "this invocation has a record", + * written in one place), so exporters still never see two exports at once and each record keeps its per-exporter + * fan-out. {@link #flush()} requests are served by that same pump, between records, so an exporter never sees a + * {@code flush()} overlap an {@code export()} either. Requests are served as a batch — the cadence is at most one flush + * per requesting invocation end, not exactly one — and a flush is preceded by the queued records a drain is waiting + * for, so a burst of invocation ends is covered by one flush rather than one each. A request made while a flush runs + * waits for the next turn. Exports are otherwise fire-and-forget; {@link #drain(InsightPlugin)} is called before an + * invocation returns and waits for that invocation's own latest record to reach every exporter. + * + *

Because there is one pump, that wait can also cover records other invocations had already queued ahead of this + * one: a drain is not isolated from the queue's head-of-line cost. What per-invocation ownership guarantees is that + * another invocation's record can never displace this one — records coalesce only within their own invocation, + * and a drain cannot return until its own latest record has reached every exporter. */ final class ExportScheduler { private static final AtomicInteger THREAD_NUMBER = new AtomicInteger(); /** - * Upper bound on the passes {@link #drainAll()} makes over the outstanding executions. Only reached if new work + * Upper bound on the passes {@link #drainAll()} makes over the outstanding invocations. Only reached if new work * keeps arriving for as long as the drain runs; a normal drain settles in two passes. */ private static final int MAX_DRAIN_ALL_PASSES = 1_000; + /** How long one {@link #drainAll()} pass waits for a running pump before taking another pass. */ + private static final long PUMP_WAIT_MILLIS = 50; + /** Shared for the process lifetime; idle daemon workers are reclaimed, so nothing keeps the runtime alive. */ private static final ExecutorService WORKERS = Executors.newCachedThreadPool(runnable -> { var thread = new Thread(runnable, "workflow-insight-export-" + THREAD_NUMBER.incrementAndGet()); @@ -76,8 +88,8 @@ final class ExportScheduler { /** * The thread serving the pump right now, or {@code null} while no pump is running. Deliberately not - * guarded by {@code this}: it is read by {@link #flush()} and {@link #drain(String)} before they touch anything - * else, and a lock acquisition there would put the pump's own monitor on the path of every invocation end. + * guarded by {@code this}: it is read by {@link #flush()} and the drains before they touch anything else, and a + * lock acquisition there would put the pump's own monitor on the path of every invocation end. * *

Written only by the thread that enters {@link #pump} — a worker, or the caller's own thread on the * rejected-worker fallback, which is a case where the calling thread genuinely is the pump — and cleared @@ -89,27 +101,18 @@ final class ExportScheduler { private final AtomicReference pumpThread = new AtomicReference<>(); /** - * The latest record per execution ARN that the pump has not picked up yet, in the order the executions first queued - * work. Guarded by {@code this}. - */ - private final Map pending = new LinkedHashMap<>(); - - /** - * Per-execution completion signal, present while that execution has work outstanding (pending or being exported - * right now) and completed once its latest record has been handed to every exporter. Guarded by {@code this}. - */ - private final Map> settled = new HashMap<>(); - - /** - * The execution ARNs whose record a pump has taken out of {@link #pending} and is handing to the exporters right - * now. Guarded by {@code this}. + * The invocations with a record no pump has picked up yet, in the order they first queued work. Ordering only — the + * record itself lives on the invocation's plugin instance. Guarded by {@code this}. + * + *

This is the only collection of per-invocation objects the scheduler has, and it holds an instance for exactly + * as long as that instance has a record waiting: nothing here has to be cleaned up at an invocation boundary, and + * an instance the SDK has dropped is unreachable from the scheduler the moment its last record is taken. * - *

Without this, "no record in {@code pending}" is indistinguishable from "record already taken and mid-export", - * and a pump running only its own {@code finally} would treat the second case as orphaned work and complete that - * execution's drain signal while the record was still inside the exporters — exactly the early return - * {@link #drain(String)} exists to prevent. + *

Invariant, and the only thing that could still be said twice: an invocation is in here exactly while its + * {@link InsightPlugin#record} is non-null. Every record moves through {@link #queueRecord}, {@link #takeRecord} or + * {@link #dropRecord}, which write both halves together, and a set makes a double entry impossible by construction. */ - private final Set exporting = new HashSet<>(); + private final Set queue = new LinkedHashSet<>(); /** * One entry per outstanding {@link #flush()} request, in request order, completed when a {@code flush()} that @@ -122,18 +125,6 @@ final class ExportScheduler { */ private final Deque> flushRequests = new ArrayDeque<>(); - /** - * How many {@link #drain(String)} calls are waiting for each execution right now. Guarded by {@code this}; an entry - * is removed when its last waiter returns. - * - *

A record with a waiter is the last record of an invocation that cannot return until it is exported, so the - * pump exports those records before it spends a flush fan-out. Without that, a burst of invocation ends is - * serialized by the pump itself — one record exported per turn, a flush in between — and each end ends up paying - * for its own flush, which is what coalescing is meant to prevent. Records nobody is waiting for (an - * {@code ON_CHANGE} stream, say) are not front-loaded, so they cannot push a flush back either. - */ - private final Map drainWaiters = new HashMap<>(); - ExportScheduler( List exporters, BiConsumer exportOne, @@ -152,28 +143,81 @@ final class ExportScheduler { this.executor = executor; } + // --- Scheduling. --- + /** - * Queues the latest record of one execution for export. If an export is already running, the record is held in that - * execution's own slot (replacing only an earlier record of the same execution) and exported once the pump - * reaches it. + * Queues the latest record of one invocation for export. If an export is already running, the record is held in + * that invocation's own slot (replacing only an earlier record of the same invocation) and exported once + * the pump reaches it. */ - void schedule(String executionArn, WorkflowInsightRecord record) { + void schedule(InsightPlugin execution, WorkflowInsightRecord record) { CompletableFuture handle; synchronized (this) { - pending.put(executionArn, record); - settled.computeIfAbsent(executionArn, arn -> new CompletableFuture<>()); - if (inFlight != null) { - return; + queueRecord(execution, record); + handle = claimPumpIfIdle(); + } + startPump(handle); + } + + /** + * Schedules the record unless this invocation has already ended; the check and the hand-off are one critical + * section, on the same monitor that owns the {@code closed} flag. + */ + boolean scheduleIfOpen(InsightPlugin execution, WorkflowInsightRecord record) { + CompletableFuture handle; + synchronized (this) { + if (execution.closed) { + return false; } - handle = new CompletableFuture<>(); - inFlight = handle; + queueRecord(execution, record); + handle = claimPumpIfIdle(); + } + startPump(handle); + return true; + } + + /** Marks the invocation ended and, when given a record, schedules it as the last one for that invocation. */ + void closeAndSchedule(InsightPlugin execution, WorkflowInsightRecord finalRecord) { + if (finalRecord == null && execution.closed) { + // The idempotent second call from the hook's `finally`. A volatile read, so the common case of an + // invocation end that already scheduled its final record does not take the lock again. + return; + } + CompletableFuture handle = null; + synchronized (this) { + execution.closed = true; + if (finalRecord != null) { + queueRecord(execution, finalRecord); + handle = claimPumpIfIdle(); + } + } + startPump(handle); + } + + /** + * Claims the pump for the caller when none is running, returning the handle to run with, or {@code null} when a + * pump already owns the scheduler and will pick the work up. Caller holds the lock. + */ + private CompletableFuture claimPumpIfIdle() { + if (inFlight != null) { + return null; + } + CompletableFuture handle = new CompletableFuture<>(); + inFlight = handle; + return handle; + } + + /** Starts a claimed pump on a worker; a no-op when the caller claimed nothing. */ + private void startPump(CompletableFuture handle) { + if (handle == null) { + return; } try { executor.execute(() -> pump(handle)); } catch (Throwable t) { - // No worker could be started. Keep the pending record and return to idle so a later schedule() retries, - // and drain() runs whatever is still pending on the calling thread before the invocation returns. Complete - // the handle too: a drain() that already observed it must wake up and take that inline path. + // No worker could be started. Keep the queued record and return to idle so a later schedule() retries, and + // a drain runs whatever is still queued on the calling thread before the invocation returns. Complete the + // handle too: a drain that already observed it must wake up and take that inline path. synchronized (this) { if (inFlight == handle) { inFlight = null; @@ -184,50 +228,85 @@ void schedule(String executionArn, WorkflowInsightRecord record) { } } + // --- The record slot: the two halves of "this invocation has a record queued", always written together. --- + + /** Puts this invocation's latest record in its slot and makes sure it has a drain signal. Caller holds the lock. */ + private void queueRecord(InsightPlugin execution, WorkflowInsightRecord record) { + execution.record = record; + queue.add(execution); + if (execution.settled == null) { + execution.settled = new CompletableFuture<>(); + } + } + /** - * Waits until the latest record of one execution has been handed to every exporter. Safe to call when that - * execution has nothing outstanding. Used before the invocation returns to guarantee the final record is delivered. + * Takes this invocation's queued record for export and marks it mid-export. Caller holds the lock and has checked + * that a record is there. * - *

The wait is for this execution's own latest record. Another execution's record can never displace it, so this - * always returns having delivered this execution's latest snapshot; but since one pump exports serially, the wait - * can also cover records other executions had already queued ahead of it. + *

The marking is not a separate step in a separate structure: leaving the slot and becoming "inside the + * exporters" are one write of one object, so no reader can see the invocation between the two and conclude it has + * nothing outstanding. + */ + private WorkflowInsightRecord takeRecord(InsightPlugin execution) { + WorkflowInsightRecord record = execution.record; + execution.record = null; + queue.remove(execution); + execution.exporting = true; + return record; + } + + /** Drops this invocation's queued record without exporting it. Caller holds the lock. */ + private void dropRecord(InsightPlugin execution) { + execution.record = null; + queue.remove(execution); + } + + // --- Draining. --- + + /** + * Waits until the latest record of one invocation has been handed to every exporter. Safe to call when that + * invocation has nothing outstanding. Used before the invocation returns to guarantee the final record is + * delivered. * - *

While this waits, the execution is registered in {@link #drainWaiters}: it tells the pump that this record - * gates an invocation return, so the pump exports it before spending a flush fan-out. See - * {@link #exportRecordsADrainIsWaitingFor}. + *

The wait is for that invocation's own latest record. Another invocation's record can never displace it, so + * this always returns having delivered this invocation's latest snapshot; but since one pump exports serially, the + * wait can also cover records other invocations had already queued ahead of it. + * + *

While this waits, the invocation counts a drain waiter — on the instance itself, so the count cannot come to + * describe a different one. It tells the pump that this record gates an invocation return, so the pump exports it + * before spending a flush fan-out. See {@link #exportRecordsADrainIsWaitingFor}. * *

Called from the pump thread itself, the wait is refused and reported instead of made: see * {@link #refuseWaitFromThePumpThread}. */ - void drain(String executionArn) { + void drain(InsightPlugin execution) { // Re-entered from the pump: this thread is the one that would settle the signal it is about to wait for. Refuse // and return; the record stays queued and this same pump exports it when it resumes its loop. - if (refuseWaitFromThePumpThread("drain(executionArn)")) { + if (refuseWaitFromThePumpThread("drain(execution)")) { return; } synchronized (this) { - if (!settled.containsKey(executionArn)) { + if (execution.settled == null) { return; } - drainWaiters.merge(executionArn, 1, Integer::sum); + execution.drainWaiters++; } try { - drainUntilSettled(executionArn); + drainUntilSettled(execution); } finally { synchronized (this) { - drainWaiters.compute( - executionArn, (arn, waiting) -> waiting == null || waiting <= 1 ? null : waiting - 1); + execution.drainWaiters--; } } } - private void drainUntilSettled(String executionArn) { + private void drainUntilSettled(InsightPlugin execution) { while (true) { CompletableFuture signal; CompletableFuture handle; boolean runInline = false; synchronized (this) { - signal = settled.get(executionArn); + signal = execution.settled; if (signal == null) { return; } @@ -241,20 +320,32 @@ private void drainUntilSettled(String executionArn) { } if (runInline) { pump(handle); + if (nothingCanSettle(execution, signal)) { + // The pump this thread just ran found no record for this invocation and left none inside the + // exporters, yet the signal survives: no later step can complete it, so waiting again would only + // start empty pumps forever. Release it here and report — the pump's own exit does not sweep for + // this any more, because it has no registry of invocations to sweep and does not need one: the + // thread that would be stranded is this one, and it holds the instance. + abandon(execution); + reportFailure(new IllegalStateException( + "a drain signal survived a pump that had nothing to export for it; the drain was released" + + " rather than waiting for work nobody will do")); + return; + } continue; } - // Wake either when this execution's record has been exported or when the current pump ends — the pump may + // Wake either when this invocation's record has been exported or when the current pump ends — the pump may // have ended without taking this record (a rejected worker), in which case the loop re-evaluates and // exports it inline. try { CompletableFuture.anyOf(signal, handle).join(); } catch (Throwable t) { // Never spin on an unexpected wait failure, and never let it escape into the execution. Abandon this - // execution's outstanding record instead of leaving it queued: WORKERS is a static, process-wide pool, - // so a record left in pending here would be exported later by some unrelated execution's pump — out of + // invocation's outstanding record instead of leaving it queued: WORKERS is a static, process-wide pool, + // so a record left queued here would be exported later by some unrelated execution's pump — out of // order, and after this invocation has already returned. Completing the signal also releases any other - // drain waiting on the same execution rather than stranding it behind work nobody will do. - abandon(executionArn); + // drain waiting on the same invocation rather than stranding it behind work nobody will do. + abandon(execution); reportFailure(t); return; } @@ -262,173 +353,124 @@ private void drainUntilSettled(String executionArn) { } /** - * Flushes every exporter, serialized against exports: the request is queued and served by the pump between records, - * so an exporter never sees {@code flush()} overlap {@code export()} — not even an export belonging to a different - * execution running in the same environment. Returns once a flush that started after this request was enqueued has - * reached every exporter. - * - *

Requests are coalesced: the pump takes every request queued at the start of its turn, exports any queued - * record a {@link #drain(String)} is still waiting for, re-takes the requests those ends make as they are released, - * and satisfies them all with one flush. Invocation ends that overlap therefore share a flush instead of paying for - * one fan-out each. That is sound because a caller drains its own record before asking, so a flush that - * starts after the request was enqueued has that record in the exporter's buffer. A request enqueued while - * a flush is already running is never satisfied by it — it waits for the next turn. - * - *

A queue that never runs dry cannot starve a request either: the pump alternates one record and one batch of - * requests, so a flush waits at most one export fan-out. - * - *

Called from the pump thread itself — which only something the pump invokes synchronously can do — the request - * is refused and reported instead of made: see {@link #refuseWaitFromThePumpThread}. + * True when this invocation still holds the same drain signal but has no queued record and none inside the + * exporters, so nothing that could complete the signal is left. No ordinary path produces that; this is the + * liveness backstop for the unwinds that are hard to enumerate exhaustively. */ - void flush() { - // Re-entered from the pump: this thread is the only one that could serve the request it is about to make, so it - // must not make it. Refuse and return rather than enqueue a request nobody can serve. - if (refuseWaitFromThePumpThread("flush()")) { - return; - } - CompletableFuture request = new CompletableFuture<>(); - synchronized (this) { - flushRequests.add(request); - } - while (true) { - CompletableFuture handle; - boolean startPump = false; - synchronized (this) { - if (request.isDone()) { - return; - } - handle = inFlight; - if (handle == null) { - if (!flushRequests.contains(request)) { - // Liveness backstop: a pump took this request and unwound without serving it, which its - // `finally` is there to prevent. The request is no longer in the queue, so no future pump can - // find it — release the caller here instead of spinning up pumps that have nothing to do. - break; - } - // No pump is running (a worker could not be started earlier, or the pump went idle between the add - // above and this check): start one. - handle = new CompletableFuture<>(); - inFlight = handle; - startPump = true; - } - } - if (startPump) { - CompletableFuture started = handle; - try { - executor.execute(() -> pump(started)); - } catch (Throwable t) { - // No worker could be started. Serve the request on the calling thread, exactly as drain() exports a - // pending record inline: this pump owns `inFlight`, so no export can run beside it. - reportFailure(t); - pump(started); - continue; - } - } - // Wake either when this request has been served or when the current pump ends — a pump can end without - // serving it (a rejected worker), in which case the loop starts another one. - try { - CompletableFuture.anyOf(request, handle).join(); - } catch (Throwable t) { - // Never spin on an unexpected wait failure, and never let it escape into the execution. Drop the - // request rather than leaving it queued for some later, unrelated invocation's pump to serve. - synchronized (this) { - flushRequests.remove(request); - } - reportFailure(t); - break; - } - } - request.complete(null); + private synchronized boolean nothingCanSettle(InsightPlugin execution, CompletableFuture signal) { + return execution.settled == signal && execution.record == null && !execution.exporting; } /** - * Waits for every execution's outstanding record. Test seam for a plugin-wide drain; the per-invocation path uses - * {@link #drain(String)}. + * Waits for every outstanding record. Test seam for an environment-wide drain; the per-invocation path uses + * {@link #drain(InsightPlugin)}. + * + *

Returns once nothing is queued and no pump owns the scheduler, which is exactly "every record scheduled so far + * has reached the exporters": a pump only returns to idle with its queue empty and the record it took settled. * - *

Bounded by the number of passes, not by the set of ARNs seen: an execution that queues new work after it was - * already drained must still be waited for (dropping it would silently weaken every assertion made after this - * returns), while a producer that never stops cannot keep this spinning forever. + *

Bounded by the number of passes, not by the set of invocations seen: an invocation that queues new work after + * it was already drained must still be waited for (dropping it would silently weaken every assertion made after + * this returns), while a producer that never stops cannot keep this spinning forever. */ void drainAll() { - // Every pass below is a drain(), and each one would be refused; without this the loop spends all of its passes + // Every pass below is a drain, and each one would be refused; without this the loop spends all of its passes // reporting the same refusal. if (refuseWaitFromThePumpThread("drainAll()")) { return; } for (int pass = 0; pass < MAX_DRAIN_ALL_PASSES; pass++) { - List outstanding; + List outstanding; + CompletableFuture handle; synchronized (this) { - if (settled.isEmpty()) { + outstanding = new ArrayList<>(queue); + handle = inFlight; + if (outstanding.isEmpty() && handle == null) { return; } - outstanding = new ArrayList<>(settled.keySet()); } - for (String executionArn : outstanding) { - drain(executionArn); + for (InsightPlugin execution : outstanding) { + drain(execution); + } + if (outstanding.isEmpty()) { + // Nothing is queued for anyone, but a pump still owns the scheduler: it may be inside an exporter with + // a + // record whose only reference is its own local variable, and with no registry of invocations there is + // no + // way to name that record and drain it. Waiting for the pump itself covers it — in bounded steps, so a + // producer that keeps the pump permanently busy cannot make this unbounded, and so the wait is a real + // wait rather than a re-poll. + try { + handle.get(PUMP_WAIT_MILLIS, TimeUnit.MILLISECONDS); + } catch (TimeoutException e) { + // Still running; take another pass. + } catch (Throwable t) { + reportFailure(t); + return; + } } } } - /** Gives up one execution's outstanding work: drops its queued record and releases every drain waiting on it. */ - private void abandon(String executionArn) { + /** Gives up one invocation's outstanding work: drops its queued record and releases every drain waiting on it. */ + private void abandon(InsightPlugin execution) { CompletableFuture signal; synchronized (this) { - pending.remove(executionArn); - signal = settled.remove(executionArn); + dropRecord(execution); + execution.exporting = false; + signal = execution.settled; + execution.settled = null; } if (signal != null) { signal.complete(null); } } + // --- The pump. --- + private void pump(CompletableFuture handle) { // Recorded for as long as this thread serves the pump — a worker, or a caller pumping inline after a rejected // worker — so that a flush() or drain() re-entered from anything the fan-out calls synchronously can tell that // it is asking itself. One atomic write per pump, and no lock: see the field. Thread self = Thread.currentThread(); pumpThread.set(self); - // The ARN this pump has taken and not settled yet. Only this pump may release it, so an abnormal unwind cannot - // strand a drain, and no other pump can mistake it for orphaned work. - String taken = null; + // The invocation this pump has taken a record from and not settled yet. Only this pump may release it, so an + // abnormal unwind cannot strand a drain, and no other pump can mistake it for orphaned work. + InsightPlugin taken = null; // Likewise for the flush requests this pump has taken out of the queue and not completed yet. List> takenFlushes = null; try { // One record, then every flush request queued at that moment, alternating. Taking the record and returning // to idle both happen under the lock, so a record scheduled at any point is either exported by this pump or - // starts the next one — never lost, and never displaced by another execution's record. A flush therefore + // starts the next one — never lost, and never displaced by another invocation's record. A flush therefore // waits at most one fan-out (it cannot be starved by a queue that never runs dry) and still never overlaps // an export, because this loop runs them one after the other. // // A loop, deliberately, not a pump that re-enters itself to pick up the next item: written that way, one // frame per queued item accumulates until the stack overflows, and the rest of the queue is dropped. while (true) { - String executionArn = null; + InsightPlugin next = null; WorkflowInsightRecord record = null; synchronized (this) { - Iterator> queued = - pending.entrySet().iterator(); - if (!queued.hasNext() && flushRequests.isEmpty()) { + if (queue.isEmpty() && flushRequests.isEmpty()) { if (inFlight == handle) { inFlight = null; } return; } - if (queued.hasNext()) { - Map.Entry next = queued.next(); - queued.remove(); - executionArn = next.getKey(); - record = next.getValue(); - // Marked under the same lock that removes the record, so the execution is never momentarily - // invisible to another pump's orphan check. - exporting.add(executionArn); + if (!queue.isEmpty()) { + next = queue.iterator().next(); + // Leaving the slot and being marked mid-export are one write of one object, so the invocation + // is + // never momentarily indistinguishable from one with nothing outstanding. + record = takeRecord(next); } } - if (executionArn != null) { - taken = executionArn; + if (next != null) { + taken = next; try { exportToAll(record); } finally { - signalSettled(executionArn); + signalSettled(next); taken = null; } } @@ -448,7 +490,7 @@ record = next.getValue(); } if (takenFlushes != null) { // Before spending the fan-out: export the queued records other invocations are still waiting on. - // Those ends cannot have asked for their flush yet — they are inside drain() — so without this the + // Those ends cannot have asked for their flush yet — they are inside a drain — so without this the // pump staggers them one record per turn, with a whole flush in between, and each pays for its own // flush however aggressively the queue is coalesced. exportRecordsADrainIsWaitingFor(); @@ -476,24 +518,25 @@ record = next.getValue(); if (takenFlushes != null) { completeAll(takenFlushes); } - List> orphaned; + CompletableFuture orphaned = null; synchronized (this) { if (inFlight == handle) { inFlight = null; } if (taken != null) { - // Unwinding with a record still marked as being exported: this pump will never settle it, so - // release it here and let the orphan sweep below complete its drain. - exporting.remove(taken); + // Unwinding with a record still marked as being exported: this pump will never settle it. Release + // the marker and, unless a newer record for the same invocation is queued for a later pump to + // export, complete the drain waiting on it — the instance is right here, so no sweep over other + // invocations is needed to find it. + taken.exporting = false; + if (taken.record == null) { + orphaned = taken.settled; + taken.settled = null; + } } - orphaned = takeSignalsWithNothingOutstanding(); } - // Defensive backstop: an execution with nothing outstanding — no record in the queue and none inside the - // exporters — whose signal nevertheless survived must not leave a drain() waiting forever. No ordinary path - // is known to produce that; it covers the unwinds that are hard to enumerate exhaustively rather than one - // specific failure. - for (CompletableFuture signal : orphaned) { - signal.complete(null); + if (orphaned != null) { + orphaned.complete(null); } handle.complete(null); // Last, because everything above is still this pump's work and a flush() re-entered from any of it would @@ -503,41 +546,38 @@ record = next.getValue(); } /** - * Exports the queued records that a {@link #drain(String)} is waiting for, one at a time, and returns once they - * have all reached the exporters. Called by the pump immediately before a flush. + * Exports the queued records that a drain is waiting for, one at a time, and returns once they have all reached the + * exporters. Called by the pump immediately before a flush. * *

Those records are the last records of invocations that cannot return until they are exported, and their ends * cannot ask for their flush until then. Exporting them first is therefore what lets one flush serve a whole burst * of invocation ends: without it the pump interleaves one record and one flush fan-out, and each end pays for a * flush of its own even though every request is coalesced. * - *

Bounded by the snapshot taken under the lock, so a producer that keeps scheduling for an execution someone is + *

Bounded by the snapshot taken under the lock, so a producer that keeps scheduling for an invocation someone is * draining cannot hold a flush back indefinitely — and records nobody waits for are not exported here at all, so a * stream of {@code ON_CHANGE} snapshots still cannot starve a flush: it waits at most one ordinary fan-out plus - * this pass over the executions whose invocation return is already blocked on their own record. + * this pass over the invocations whose return is already blocked on their own record. */ private void exportRecordsADrainIsWaitingFor() { - List awaited; + List awaited = null; synchronized (this) { - if (pending.isEmpty() || drainWaiters.isEmpty()) { - return; - } - awaited = new ArrayList<>(); - for (String executionArn : pending.keySet()) { - if (drainWaiters.containsKey(executionArn)) { - awaited.add(executionArn); + for (InsightPlugin execution : queue) { + if (execution.drainWaiters > 0) { + if (awaited == null) { + awaited = new ArrayList<>(); + } + awaited.add(execution); } } } - for (String executionArn : awaited) { + if (awaited == null) { + return; + } + for (InsightPlugin execution : awaited) { WorkflowInsightRecord record; synchronized (this) { - record = pending.remove(executionArn); - if (record != null) { - // Marked under the same lock that removes the record, exactly as the pump's own record step does, - // so the execution is never momentarily invisible to another pump's orphan check. - exporting.add(executionArn); - } + record = execution.record == null ? null : takeRecord(execution); } if (record == null) { continue; @@ -546,13 +586,12 @@ record = pending.remove(executionArn); exportToAll(record); } finally { try { - signalSettled(executionArn); + signalSettled(execution); } catch (Throwable t) { // Nothing here is expected to throw, but a record left marked as being exported would strand the - // drain that is waiting for it, so release it rather than leave the invocation parked. - synchronized (this) { - exporting.remove(executionArn); - } + // drain that is waiting for it, so release it and that drain rather than leave the invocation + // parked. + abandon(execution); reportFailure(t); } } @@ -571,46 +610,111 @@ private void completeAll(List> requests) { } /** - * Completes one execution's drain signal now that its record has been exported, unless a newer record for the same - * execution arrived meanwhile — that one settles the signal instead, so drain() always waits for the latest. + * Completes one invocation's drain signal now that its record has been exported, unless a newer record from the + * same invocation arrived meanwhile — that one settles the signal instead, so a drain always waits for the latest. */ - private void signalSettled(String executionArn) { + private void signalSettled(InsightPlugin execution) { CompletableFuture signal; synchronized (this) { - if (pending.containsKey(executionArn)) { - // A newer record is queued for the same execution. Leave the ARN marked as being exported: it is still + if (execution.record != null) { + // A newer record is queued for the same invocation. Leave it marked as being exported: it is still // outstanding, and the export of that newer record settles the signal. return; } - exporting.remove(executionArn); - signal = settled.remove(executionArn); + execution.exporting = false; + signal = execution.settled; + execution.settled = null; } if (signal != null) { signal.complete(null); } } + // --- Flushing. --- + /** - * Removes and returns the signals of executions with nothing outstanding: no record queued and none being - * handed to the exporters right now. Caller holds the lock. + * Flushes every exporter, serialized against exports: the request is queued and served by the pump between records, + * so an exporter never sees {@code flush()} overlap {@code export()} — not even an export belonging to a different + * execution running in the same environment. Returns once a flush that started after this request was enqueued has + * reached every exporter. + * + *

Requests are coalesced: the pump takes every request queued at the start of its turn, exports any queued + * record a drain is still waiting for, re-takes the requests those ends make as they are released, and satisfies + * them all with one flush. Invocation ends that overlap therefore share a flush instead of paying for one fan-out + * each. That is sound because a caller drains its own record before asking, so a flush that starts after + * the request was enqueued has that record in the exporter's buffer. A request enqueued while a flush is already + * running is never satisfied by it — it waits for the next turn. + * + *

A queue that never runs dry cannot starve a request either: the pump alternates one record and one batch of + * requests, so a flush waits at most one export fan-out. + * + *

Called from the pump thread itself — which only something the pump invokes synchronously can do — the request + * is refused and reported instead of made: see {@link #refuseWaitFromThePumpThread}. */ - private List> takeSignalsWithNothingOutstanding() { - List> taken = new ArrayList<>(); - Iterator>> signals = - settled.entrySet().iterator(); - while (signals.hasNext()) { - Map.Entry> signal = signals.next(); - if (!pending.containsKey(signal.getKey()) && !exporting.contains(signal.getKey())) { - taken.add(signal.getValue()); - signals.remove(); + void flush() { + // Re-entered from the pump: this thread is the only one that could serve the request it is about to make, so it + // must not make it. Refuse and return rather than enqueue a request nobody can serve. + if (refuseWaitFromThePumpThread("flush()")) { + return; + } + CompletableFuture request = new CompletableFuture<>(); + synchronized (this) { + flushRequests.add(request); + } + while (true) { + CompletableFuture handle; + boolean startPump = false; + synchronized (this) { + if (request.isDone()) { + return; + } + handle = inFlight; + if (handle == null) { + if (!flushRequests.contains(request)) { + // Liveness backstop: a pump took this request and unwound without serving it, which its + // `finally` is there to prevent. The request is no longer in the queue, so no future pump can + // find it — release the caller here instead of spinning up pumps that have nothing to do. + break; + } + // No pump is running (a worker could not be started earlier, or the pump went idle between the add + // above and this check): start one. + handle = new CompletableFuture<>(); + inFlight = handle; + startPump = true; + } + } + if (startPump) { + CompletableFuture started = handle; + try { + executor.execute(() -> pump(started)); + } catch (Throwable t) { + // No worker could be started. Serve the request on the calling thread, exactly as a drain exports a + // queued record inline: this pump owns `inFlight`, so no export can run beside it. + reportFailure(t); + pump(started); + continue; + } + } + // Wake either when this request has been served or when the current pump ends — a pump can end without + // serving it (a rejected worker), in which case the loop starts another one. + try { + CompletableFuture.anyOf(request, handle).join(); + } catch (Throwable t) { + // Never spin on an unexpected wait failure, and never let it escape into the execution. Drop the + // request rather than leaving it queued for some later, unrelated invocation's pump to serve. + synchronized (this) { + flushRequests.remove(request); + } + reportFailure(t); + break; } } - return taken; + request.complete(null); } /** * Flushes every exporter, each on its own worker, and waits for all of them to settle. A slow or failing flush on - * one exporter never delays or fails the others. Plugin-wide, like the exporters themselves. + * one exporter never delays or fails the others. Environment-wide, like the exporters themselves. * *

Private and called only from the pump: routing every flush through the pump is what keeps a {@code flush()} * from overlapping an {@code export()}, so this must not be reachable from outside. The per-exporter fan-out below @@ -620,6 +724,8 @@ private void flushEveryExporter() { forEachExporterSettled(InsightExporter::flush); } + // --- Exporting. --- + /** * Exports one record to every exporter, each on its own worker, and waits for all of them to settle. One failing or * slow exporter never blocks or fails the others, and an export error never propagates into the execution. @@ -670,8 +776,8 @@ private void reportFailure(Throwable t) { * pump thread; when it is, the failure has already been reported and the caller must return without waiting. * *

Invariant: the thread that waits for the pump is never the thread that serves it. {@link #flush()} waits for a - * request only a pump can complete, and {@link #drain(String)} waits for a signal only a pump can complete or for - * the running pump's own handle. All three are satisfied by the pump between records. + * request only a pump can complete, and a drain waits for a signal only a pump can complete or for the running + * pump's own handle. All three are satisfied by the pump between records. * *

Without this, a wait issued from the pump is a wait-for cycle one thread wide: the pump parks on the future it * would itself have completed, so it never reaches the point in its loop that completes it, and no other thread may @@ -683,7 +789,7 @@ private void reportFailure(Throwable t) { * *

So the call fails fast instead: the plugin's failure handler is told — it logs — and the caller returns as it * would from any other flush or drain, with nothing propagating into the execution. The queued work itself is not - * dropped by refusing a {@code drain}: the record stays in {@code pending} and the pump asking the question is the + * dropped by refusing a drain: the record stays in the invocation's slot and the pump asking the question is the * one that will export it. Callers that are not the pump — every SDK hook thread — never enter this branch and * behave exactly as before, and the check is a single volatile read, so no lock is added to that path. */ diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightPlugin.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightPlugin.java new file mode 100644 index 000000000..82b67ba1a --- /dev/null +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightPlugin.java @@ -0,0 +1,382 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import java.time.Instant; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationInfo; +import software.amazon.lambda.durable.plugin.OperationChangeInfo; +import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; + +/** + * The Workflow Insight plugin instance for one Lambda invocation: both the state that invocation's records are built + * from and the slot the {@link ExportScheduler} exports them through. + * + *

The SDK creates one of these per invocation, from the {@link InvocationInfo} it is about to hand the first hook, + * and drops it when the invocation returns. So everything about an execution is a plain field here — the parsed ARN, + * the stable start time, the one-time sampling decision, the detached input snapshot, the latest queued record, the + * drain signal, the mid-export marker and the drain-waiter count. There is nothing to key by execution ARN and nothing + * to register or release: an instance is the registration, and its lifetime is the invocation's. + * + *

Two objects outlive the invocation and are shared by every instance the factory creates: the resolved + * {@link InsightSettings} and the {@link ExportScheduler}. The scheduler is shared on purpose — serializing exports is + * a property of the exporters, which belong to the environment, not to one invocation. + * + *

Ownership. Three groups of fields: + * + *

+ * + *

{@link #closed} is additionally {@code volatile}: the hook threads read it without the lock as a fast pre-check. + * That read only ever skips work — the authoritative check is made under the monitor by + * {@link ExportScheduler#scheduleIfOpen}. It is written once, from false to true, and never back: an execution that + * suspends and resumes gets a new instance rather than a reset one. + */ +final class InsightPlugin implements DurableExecutionPlugin { + + /** Resolved configuration, shared by every instance of the environment. */ + private final InsightSettings settings; + + /** + * Shared with every other instance: exports are serialized across the whole environment. Package-private because it + * is the monitor this instance's scheduling fields are guarded by, which the tests in this package hold when they + * read them. + */ + final ExportScheduler scheduler; + + // --- Identity, from the InvocationInfo the factory was called with. --- + + /** The execution this instance observes. */ + final String executionArn; + + /** The parsed execution ARN, parsed once for every record this instance builds. */ + final ArnParser arn; + + /** Stable execution start time, from {@code InvocationInfo.executionStartTime()}. */ + final Instant startTime; + + /** The one-time sampling decision; deterministic in the ARN, so a resumed invocation decides the same way. */ + final boolean sampledIn; + + // --- The input snapshot. --- + + /** + * Detached snapshot of the execution input, the single source of truth for {@code input} on every emission of this + * invocation. Written by {@code onInvocationStart}, read by every later build. + */ + private volatile Object cachedInput; + + // --- Scheduling state: guarded by the scheduler's monitor. --- + + /** + * The latest record for this invocation that no pump has picked up yet, or {@code null} when none is queued. + * + *

A newer record replaces an older one here — each record is a complete snapshot, so the older one carries + * nothing the newer one lacks. That is the whole of coalescing: one slot, on the instance, which no other + * invocation can reach. + */ + WorkflowInsightRecord record; + + /** + * Completes once this invocation's latest record has been handed to every exporter; {@code null} when nothing is + * outstanding. + */ + CompletableFuture settled; + + /** + * Whether a pump has taken this invocation's record and is handing it to the exporters right now. + * + *

Set and cleared in the same critical sections that move {@link #record}, so "no queued record" is never + * mistaken for "nothing outstanding" while the record is inside the exporters. + */ + boolean exporting; + + /** + * How many {@code drain} calls are waiting for this invocation right now. + * + *

A record with a waiter gates an invocation return, so the pump exports it before it spends a flush fan-out. + */ + int drainWaiters; + + /** + * Set once invocation end begins; never cleared. Guarded by the scheduler's monitor — the same monitor that queues + * the record, so the check and the hand-off are one critical section — and {@code volatile} for the hook-side + * pre-check. + * + *

A checkpoint that completes while the end record is being drained still delivers an operation-change hook to + * this same instance, and that RUNNING snapshot must not supersede the final record. + */ + volatile boolean closed; + + /** + * Creates the instance that serves one invocation. Identity comes from {@code info} rather than from the first + * hook, so every field a record is keyed by exists before any hook can fire. + * + * @throws RuntimeException if the invocation has no usable execution ARN; the SDK contains that exactly as it + * contains a hook failure, by skipping this plugin for the invocation + */ + InsightPlugin(InsightSettings settings, ExportScheduler scheduler, InvocationInfo info) { + this.settings = settings; + this.scheduler = scheduler; + this.executionArn = info.durableExecutionArn(); + this.arn = ArnParser.parse(executionArn); + this.startTime = info.executionStartTime(); + this.sampledIn = WorkflowInsight.shouldSample(executionArn, settings.samplingRate); + } + + /** Test seam: waits until every scheduled record has been handed to the exporters. */ + void drainExports() { + scheduler.drainAll(); + } + + @Override + public void onInvocationStart(InvocationInfo info) { + try { + if (!sampledIn) { + return; + } + // Detach the execution input from the live handler value immediately, before the user handler or any + // content transform can mutate it. This raw, detached snapshot is the single source of truth for input + // on every emission (start / change / end); each build hands transforms a separate defensive copy so a + // mutating transform cannot corrupt it. Guard the snapshot: a Throwable here (e.g. a payload whose + // serialization overflows the stack) must omit the captured input, never fail the user handler. + try { + cachedInput = Json.deepCopyContent(info.executionInput()); + } catch (Throwable t) { + WorkflowInsight.logSafely("failed to snapshot execution input; omitting input", t); + cachedInput = null; + } + if (settings.emitMode == WorkflowInsightConfig.EmitMode.ON_CHANGE) { + scheduler.schedule(this, buildRecord("RUNNING", info.operations(), null, cachedInput, null, null)); + } + } catch (Throwable t) { + WorkflowInsight.logSafely("onInvocationStart failed", t); + } + } + + @Override + public void onOperationChange(OperationChangeInfo info) { + try { + if (settings.emitMode != WorkflowInsightConfig.EmitMode.ON_CHANGE || !sampledIn) { + return; + } + // Lock-free pre-check: this invocation's end may already have begun, in which case no RUNNING snapshot may + // follow the final record. The authoritative check is made again under the scheduler's lock below. + if (closed) { + return; + } + scheduler.scheduleIfOpen(this, buildRecord("RUNNING", info.operations(), null, cachedInput, null, null)); + } catch (Throwable t) { + WorkflowInsight.logSafely("onOperationChange failed", t); + } + } + + // onInvocationEnd is the hook the SDK awaits, so it is where the export queue is drained before the invocation + // returns; this guarantees the final record (scheduled above the drain) is delivered. The drain and flush run + // in finally so they also cover the paths where record construction fails. + @Override + public void onInvocationEnd(InvocationEndInfo info) { + try { + String status = WorkflowInsight.mapStatus(info.invocationStatus()); + boolean isTerminal = "SUCCEEDED".equals(status) || "FAILED".equals(status); + boolean isFailure = "FAILED".equals(status); + boolean shouldEmit; + switch (settings.emitMode) { + case ON_CHANGE: + shouldEmit = true; + break; + case ON_FAILURE: + shouldEmit = isFailure; + break; + case ON_COMPLETE: + default: + shouldEmit = isTerminal; + break; + } + + WorkflowInsightRecord finalRecord = null; + if (sampledIn && shouldEmit) { + finalRecord = buildRecord( + status, + info.operations(), + Instant.now(), + cachedInput, + info.executionResult(), + info.executionError()); + } + // Close before the drain below: an operation-change hook arriving from a checkpoint that completes + // during the drain is rejected, so no RUNNING snapshot can follow (or replace) the final record. + scheduler.closeAndSchedule(this, finalRecord); + } catch (Throwable t) { + // A plugin failure at end-of-invocation (record construction, transforms, truncation, export/flush, + // or optional exporter class linkage) must never disrupt durable execution. + WorkflowInsight.logSafely("onInvocationEnd failed", t); + } finally { + // If record construction failed above, this instance is still open: close it so a late change hook cannot + // schedule into the drain. Idempotent when already closed. + scheduler.closeAndSchedule(this, null); + // Sampled-out invocations never schedule a record, so there is nothing to drain or flush. The drain needs + // no lookup: this instance is the thing whose record it waits for. + if (sampledIn) { + drainAndFlush(); + } + // Nothing is released here. There is no per-execution entry to remove — this instance is the state, the SDK + // drops it when the invocation returns, and a suspended execution that resumes in the same container is + // served by a new instance built from the resume's own InvocationInfo (same stable start time, same + // deterministic sampling decision, its own input snapshot). A plugin failure therefore cannot turn into a + // state leak, because there is no place a leak could accumulate. + } + } + + /** + * Waits for this invocation's scheduled record to reach the exporters, then flushes each exporter once. The wait is + * per invocation: another execution running in the same environment can never displace this record, so this always + * returns having delivered this invocation's latest snapshot. It is not insulated from the queue, though — one pump + * exports serially, so records another execution had already queued ahead of this one are exported first and this + * drain waits for them too. + * + *

The flush goes through the scheduler's queue and is served by that same pump, between records, so no exporter + * ever sees this invocation's {@code flush()} overlap another's {@code export()}. Invocation ends that overlap + * share one flush: the cadence the exporter contract promises is at most one flush per sampled-in invocation end, + * not exactly one. + */ + private void drainAndFlush() { + try { + scheduler.drain(this); + } catch (Throwable t) { + WorkflowInsight.logSafely("failed to drain export scheduler", t); + } + try { + scheduler.flush(); + } catch (Throwable t) { + WorkflowInsight.logSafely("exporter flush failed", t); + } + } + + // --- Record building. --- + + private WorkflowInsightRecord buildRecord( + String status, + Map operations, + Instant endTime, + Object input, + Object output, + Throwable error) { + ContentConfig content = settings.content; + WorkflowInsightRecord record = new WorkflowInsightRecord(); + record.emittedAt = Instant.now().toString(); + record.executionArn = executionArn; + record.executionName = WorkflowInsight.emptyToNull(arn.executionName()); + record.functionName = arn.functionName(); + record.functionQualifier = arn.qualifier(); + record.region = arn.region(); + record.accountId = arn.accountId(); + record.status = status; + record.startTime = startTime != null ? startTime.toString() : null; + if (endTime != null) { + record.endTime = endTime.toString(); + if (startTime != null) { + record.durationMs = endTime.toEpochMilli() - startTime.toEpochMilli(); + } + } + record.input = WorkflowInsight.applyDataContent( + "input", + input, + content == null || content.includeInput(), + content == null ? null : content.inputTransform()); + record.output = WorkflowInsight.applyDataContent( + "output", + output, + content == null || content.includeOutput(), + content == null ? null : content.outputTransform()); + // Honor ContentConfig.includeErrors for the execution-level error exactly as for operation-level errors + // below: with includeErrors(false) no execution error is emitted, so a sensitive failure message never + // reaches a record. Without this gate the execution error leaked even when errors were disabled. + if (settings.includeErrors && error != null) { + record.error = WorkflowInsight.toErrorInfo(error); + } + record.operations = buildOperationRecords(operations); + return record; + } + + private List buildOperationRecords(Map operations) { + List out = new ArrayList<>(); + if (operations == null) { + return out; + } + // The hook contract supplies a map with no iteration-order guarantee (the core snapshot originates from a + // concurrent map). Sort by startTimestamp ascending (null timestamps last), then by a stable operation id + // tie-breaker, so the emitted operations array is deterministic and OperationsIndex's "latest occurrence" + // scalar fields reflect true chronological order rather than arbitrary map iteration order. + List items = new ArrayList<>(operations.values()); + items.sort(Comparator.comparing( + OperationChangeItemInfo::startTimestamp, Comparator.nullsLast(Comparator.naturalOrder())) + .thenComparing(OperationChangeItemInfo::id, Comparator.nullsLast(Comparator.naturalOrder()))); + for (OperationChangeItemInfo item : items) { + // The SDK core tracks the invocation/execution itself as a pseudo-entry of type EXECUTION; it is not a + // customer operation and the record already carries the execution status/timing at top level. + if ("EXECUTION".equals(item.type())) { + continue; + } + // Unnamed operations can't be targeted or keyed — excluded by default (matches JS `if (!op.name)`). + if (item.name() == null) { + continue; + } + // top-level detail drops anything nested under a context (parallel branches, map items, nested steps). + if (settings.topLevelOnly && item.parentId() != null) { + continue; + } + OperationOverride override = settings.overridesByName.get(item.name()); + if (override != null && override.isExclude()) { + continue; + } + OperationRecord rec = new OperationRecord() + .id(item.id()) + .name(item.name()) + .type(item.type()) + .subType(item.subType()) + .parentId(item.parentId()) + .status(item.status() != null ? item.status().toString() : "UNKNOWN") + .startTime( + item.startTimestamp() != null + ? item.startTimestamp().toString() + : null) + .endTime(item.endTimestamp() != null ? item.endTimestamp().toString() : null) + .attempt(item.attempt()); + if (item.startTimestamp() != null && item.endTimestamp() != null) { + rec.durationMs(item.endTimestamp().toEpochMilli() + - item.startTimestamp().toEpochMilli()); + } + if (settings.includeErrors && item.error() != null) { + rec.error(WorkflowInsight.toErrorInfo(item.error())); + } + // Results are omitted unless an override explicitly opts in via a transform (matches JS). + if (override != null && override.result() != null) { + rec.result(WorkflowInsight.applyResultOverride(override.result(), item.result())); + } + out.add(rec); + } + return out; + } + + @Override + public String toString() { + return "InsightPlugin[" + executionArn + "]"; + } +} diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightSettings.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightSettings.java new file mode 100644 index 000000000..8729b8bec --- /dev/null +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightSettings.java @@ -0,0 +1,59 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import software.amazon.lambda.durable.insight.exporters.LambdaLogExporter; + +/** + * The plugin's configuration, resolved once and then immutable: everything a record's shape depends on that does not + * depend on which invocation is being observed. + * + *

This belongs to the execution environment, not to an invocation. {@link WorkflowInsight#workflowInsight} resolves + * it once and the factory it returns hands the same instance to every {@link InsightPlugin} it creates, so resolving + * defaults, validating the sampling rate and indexing the operation overrides happen once per environment rather than + * once per invocation. + */ +final class InsightSettings { + + /** Sampling rate, clamped to [0, 1]; the per-invocation decision is derived from it and the execution ARN. */ + final double samplingRate; + + final WorkflowInsightConfig.EmitMode emitMode; + + /** True when nested operations (parallel branches, map items, nested steps) are dropped from the record. */ + final boolean topLevelOnly; + + final boolean includeErrors; + + /** May be null, which means "every default": include input, output and errors, with no transforms. */ + final ContentConfig content; + + /** Operation overrides indexed by operation name, in declaration order. */ + final Map overridesByName; + + /** The configured exporters, or the default single {@link LambdaLogExporter} when none were configured. */ + final List exporters; + + InsightSettings(WorkflowInsightConfig config) { + this.samplingRate = WorkflowInsight.resolveSamplingRate(config.samplingRate()); + this.emitMode = config.emitMode() != null ? config.emitMode() : WorkflowInsightConfig.EmitMode.ON_COMPLETE; + this.topLevelOnly = config.operationDetail() != WorkflowInsightConfig.OperationDetail.FULL_TREE; + this.content = config.content(); + this.includeErrors = content == null || content.includeErrors(); + Map overrides = new LinkedHashMap<>(); + if (content != null) { + for (OperationOverride override : content.overrides()) { + overrides.put(override.operationName(), override); + } + } + // Unmodifiable wrapper rather than Map.copyOf: declaration order is preserved and an override with a null + // operation name is tolerated exactly as the mutable map tolerated it. + this.overridesByName = Collections.unmodifiableMap(overrides); + this.exporters = + config.exporters().isEmpty() ? List.of(new LambdaLogExporter()) : List.copyOf(config.exporters()); + } +} diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/WorkflowInsight.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/WorkflowInsight.java index 5634e3c19..35e683e27 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/WorkflowInsight.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/WorkflowInsight.java @@ -2,13 +2,6 @@ // SPDX-License-Identifier: Apache-2.0 package software.amazon.lambda.durable.insight; -import java.time.Instant; -import java.util.ArrayList; -import java.util.Comparator; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.concurrent.ConcurrentHashMap; import java.util.function.Function; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -16,12 +9,10 @@ import software.amazon.lambda.durable.annotations.Experimental; import software.amazon.lambda.durable.exception.DurableOperationException; import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; -import software.amazon.lambda.durable.insight.exporters.LambdaLogExporter; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; -import software.amazon.lambda.durable.plugin.OperationChangeInfo; import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; /** @@ -35,12 +26,18 @@ * {@link OperationChangeItemInfo#result()}; these are the fields PR #618 surfaced on the hook records, so * {@code input}, {@code output}, and operation {@code result} are now populated exactly as in the JS plugin. * - *

Per-execution state (keyed by execution ARN) holds only the stable start time, the parsed ARN, the one-time - * sampling decision, and a detached snapshot of the execution input. State is removed on every {@code onInvocationEnd} - * — including non-terminal PENDING/RETRYING suspends — so a suspended execution never leaks a retained entry for the - * lifetime of a warm container. Nothing is lost across a resume: the next invocation recreates the same stable start - * time from {@link InvocationInfo#executionStartTime()}, the same sampling decision deterministically from the ARN, and - * the input snapshot from {@link InvocationInfo#executionInput()}. + *

{@link #workflowInsight} returns a {@link DurableExecutionPluginFactory}, so the SDK creates one + * {@link InsightPlugin} per Lambda invocation and drops it when the invocation returns. Everything about an execution — + * the stable start time, the parsed ARN, the one-time sampling decision, the detached input snapshot, the queued + * record, the drain signal — is therefore a plain field of that instance. Nothing is keyed by execution ARN, and there + * is no per-execution entry to remove at invocation end, so a suspended execution cannot leak one for the lifetime of a + * warm container. Nothing is lost across a resume either: the resume's own {@link InvocationInfo} carries the same + * stable start time, the sampling decision is deterministic in the ARN, and the input snapshot is taken again from + * {@link InvocationInfo#executionInput()}. + * + *

What belongs to the execution environment rather than to an invocation stays in the factory: the resolved + * {@link InsightSettings}, the exporters, and the {@link ExportScheduler} that serializes exports across every + * execution the environment hosts. */ @Experimental public final class WorkflowInsight { @@ -49,393 +46,44 @@ public final class WorkflowInsight { private WorkflowInsight() {} - /** Creates a Workflow Insight plugin from the given config. Mirrors the JS {@code workflowInsight(config)}. */ - public static DurableExecutionPlugin workflowInsight(WorkflowInsightConfig config) { - return new InsightPlugin(config); - } - - /** Per-execution state, keyed by execution ARN, to prevent warm-container bleed and handle resume. */ - private static final class ExecutionState { - final String executionArn; - final Instant startTime; - final ArnParser arn; - final boolean sampledIn; - volatile Object cachedInput; - - /** - * Set once invocation end begins; guarded by {@code this}. A checkpoint that completes while the end record is - * being drained still delivers an operation-change hook, and that RUNNING snapshot must not supersede the final - * record. - */ - boolean closed; - - ExecutionState(String executionArn, Instant startTime, ArnParser arn, boolean sampledIn) { - this.executionArn = executionArn; - this.startTime = startTime; - this.arn = arn; - this.sampledIn = sampledIn; - } - - /** Schedules the record unless the invocation has already ended; the check and the hand-off are atomic. */ - boolean scheduleIfOpen(ExportScheduler scheduler, WorkflowInsightRecord record) { - synchronized (this) { - if (closed) { - return false; - } - scheduler.schedule(executionArn, record); - return true; - } - } - - /** Marks the invocation ended and, when given a record, schedules it as the last one for this execution. */ - void closeAndSchedule(ExportScheduler scheduler, WorkflowInsightRecord finalRecord) { - synchronized (this) { - closed = true; - if (finalRecord != null) { - scheduler.schedule(executionArn, finalRecord); - } - } - } + /** + * Creates a Workflow Insight plugin factory from the given config. Mirrors the JS {@code workflowInsight(config)}. + * + *

The configuration is resolved once, here; the exporters and the scheduler that serializes exports across them + * are created once, here. The returned factory then builds one plugin instance per invocation, which is what lets + * that instance hold its execution's state in plain fields. + * + * @param config the plugin configuration + * @return a factory to hand to {@code DurableConfig.Builder.withPlugins} + */ + public static DurableExecutionPluginFactory workflowInsight(WorkflowInsightConfig config) { + InsightSettings settings = new InsightSettings(config); + ExportScheduler scheduler = new ExportScheduler( + settings.exporters, WorkflowInsight::exportRecord, t -> logSafely("export scheduling failed", t)); + return info -> new InsightPlugin(settings, scheduler, info); } - static final class InsightPlugin implements DurableExecutionPlugin { - private final double samplingRate; - private final WorkflowInsightConfig.EmitMode emitMode; - private final boolean topLevelOnly; - private final boolean includeErrors; - private final ContentConfig content; - private final Map overridesByName = new LinkedHashMap<>(); - private final List exporters; - private final ExportScheduler scheduler; - - private final Map byArn = new ConcurrentHashMap<>(); - - /** Test seam: number of live per-execution state entries retained across invocations. */ - int retainedStateCount() { - return byArn.size(); - } - - /** Test seam: waits until every scheduled record has been handed to the exporters. */ - void drainExports() { - scheduler.drainAll(); - } - - InsightPlugin(WorkflowInsightConfig config) { - this.samplingRate = resolveSamplingRate(config.samplingRate()); - this.emitMode = config.emitMode() != null ? config.emitMode() : WorkflowInsightConfig.EmitMode.ON_COMPLETE; - this.topLevelOnly = config.operationDetail() != WorkflowInsightConfig.OperationDetail.FULL_TREE; - this.content = config.content(); - this.includeErrors = content == null || content.includeErrors(); - if (content != null) { - for (OperationOverride o : content.overrides()) { - overridesByName.put(o.operationName(), o); - } - } - this.exporters = - config.exporters().isEmpty() ? List.of(new LambdaLogExporter()) : List.copyOf(config.exporters()); - this.scheduler = - new ExportScheduler(exporters, this::exportRecord, t -> logSafely("export scheduling failed", t)); - } - - private ExecutionState getState(String arn, Instant startTime) { - return byArn.computeIfAbsent( - arn, a -> new ExecutionState(a, startTime, ArnParser.parse(a), shouldSample(a, samplingRate))); - } - - @Override - public void onInvocationStart(InvocationInfo info) { - try { - ExecutionState state = getState(info.durableExecutionArn(), info.executionStartTime()); - if (!state.sampledIn) { - return; - } - // Detach the execution input from the live handler value immediately, before the user handler or any - // content transform can mutate it. This raw, detached snapshot is the single source of truth for input - // on every emission (start / change / end); each build hands transforms a separate defensive copy so a - // mutating transform cannot corrupt it. Guard the snapshot: a Throwable here (e.g. a payload whose - // serialization overflows the stack) must omit the captured input, never fail the user handler. - try { - state.cachedInput = Json.deepCopyContent(info.executionInput()); - } catch (Throwable t) { - logSafely("failed to snapshot execution input; omitting input", t); - state.cachedInput = null; - } - if (emitMode == WorkflowInsightConfig.EmitMode.ON_CHANGE) { - scheduler.schedule( - info.durableExecutionArn(), - buildRecord( - state, - info.durableExecutionArn(), - "RUNNING", - info.operations(), - null, - state.cachedInput, - null, - null)); - } - } catch (Throwable t) { - logSafely("onInvocationStart failed", t); - } - } - - @Override - public void onOperationChange(OperationChangeInfo info) { - try { - if (emitMode != WorkflowInsightConfig.EmitMode.ON_CHANGE) { - return; - } - ExecutionState state = byArn.get(info.durableExecutionArn()); - if (state == null || !state.sampledIn) { - return; - } - state.scheduleIfOpen( - scheduler, - buildRecord( - state, - info.durableExecutionArn(), - "RUNNING", - info.operations(), - null, - state.cachedInput, - null, - null)); - } catch (Throwable t) { - logSafely("onOperationChange failed", t); - } - } - - // onInvocationEnd is the hook the SDK awaits, so it is where the export queue is drained before the invocation - // returns; this guarantees the final record (scheduled above the drain) is delivered. The drain and flush run - // in finally so they also cover the paths where record construction fails. - @Override - public void onInvocationEnd(InvocationEndInfo info) { - ExecutionState state = null; - try { - state = getState(info.durableExecutionArn(), info.executionStartTime()); - String status = mapStatus(info.invocationStatus()); - boolean isTerminal = "SUCCEEDED".equals(status) || "FAILED".equals(status); - boolean isFailure = "FAILED".equals(status); - boolean shouldEmit; - switch (emitMode) { - case ON_CHANGE: - shouldEmit = true; - break; - case ON_FAILURE: - shouldEmit = isFailure; - break; - case ON_COMPLETE: - default: - shouldEmit = isTerminal; - break; - } - - WorkflowInsightRecord finalRecord = null; - if (state.sampledIn && shouldEmit) { - finalRecord = buildRecord( - state, - info.durableExecutionArn(), - status, - info.operations(), - Instant.now(), - state.cachedInput, - info.executionResult(), - info.executionError()); - } - // Close before the drain below: an operation-change hook arriving from a checkpoint that completes - // during the drain is rejected, so no RUNNING snapshot can follow (or replace) the final record. - state.closeAndSchedule(scheduler, finalRecord); - } catch (Throwable t) { - // A plugin failure at end-of-invocation (record construction, transforms, truncation, export/flush, - // or optional exporter class linkage) must never disrupt durable execution. - logSafely("onInvocationEnd failed", t); - } finally { - // If record construction failed above, the state is still open: close it so a late change hook cannot - // schedule into the drain. Idempotent when already closed. - if (state != null) { - state.closeAndSchedule(scheduler, null); - } - // Sampled-out executions never schedule a record, so there is nothing to drain or flush. If the state - // lookup itself failed, drain anyway: it is a no-op when idle and otherwise delivers what is pending. - if (state == null || state.sampledIn) { - drainAndFlush(info.durableExecutionArn()); - } - // Remove per-execution state on EVERY invocation end, including non-terminal PENDING/RETRYING suspends, - // once any emission work above is done. Nothing durable is lost: the next invocation's onInvocation - // start recreates the stable startTime from InvocationInfo.executionStartTime() (stable across - // resumes), - // the one-time sampling decision deterministically from the ARN, and the input snapshot from - // InvocationInfo.executionInput(). Retaining state instead leaked one entry per suspended execution for - // the lifetime of the warm container. This runs even if emission above threw, so a plugin failure can - // never turn into a state leak. - // - // Contained like every other step of this hook: the removal itself can fail — a null execution ARN - // makes ConcurrentHashMap.remove throw — and this is the last statement of onInvocationEnd, so an - // uncaught Throwable here would escape into the SDK and disrupt durable execution, which this method's - // contract forbids. - try { - byArn.remove(info.durableExecutionArn()); - } catch (Throwable t) { - logSafely("failed to remove per-execution state", t); - } - } - } - - /** - * Waits for this execution's scheduled record to reach the exporters, then flushes each exporter once. The wait - * is per execution: another execution running in the same environment can never displace this execution's - * record, so this always returns having delivered this execution's latest snapshot. It is not insulated from - * the queue, though — one pump exports serially, so records another execution had already queued ahead of this - * one are exported first and this drain waits for them too. - * - *

The flush goes through the scheduler's queue and is served by that same pump, between records, so no - * exporter ever sees this execution's {@code flush()} overlap another execution's {@code export()}. Invocation - * ends that overlap share one flush: the cadence the exporter contract promises is at most one flush per - * sampled-in invocation end, not exactly one. - */ - private void drainAndFlush(String executionArn) { - try { - scheduler.drain(executionArn); - } catch (Throwable t) { - logSafely("failed to drain export scheduler", t); - } - try { - scheduler.flush(); - } catch (Throwable t) { - logSafely("exporter flush failed", t); - } - } - - /** Shapes and exports one record to one exporter; runs on a scheduler worker, never on an SDK hook thread. */ - private void exportRecord(WorkflowInsightRecord record, InsightExporter exporter) { - try { - // Give each exporter its own deep copy: truncation returns the original record when it already fits, - // so without this a custom exporter that mutates operations or nested content would corrupt every - // other exporter's view of the same record. - WorkflowInsightRecord isolated = record.deepCopy(); - WorkflowInsightRecord shaped = - Truncation.truncateRecord(isolated, exporter.maxRecordSizeBytes(), exporter::render); - exporter.export(shaped); - } catch (Throwable t) { - // Catch Throwable, not just RuntimeException: deep copy, truncation, an exporter's render/export, or - // the linkage of an optional exporter class (a NoClassDefFoundError when the S3 / CloudWatch SDK is - // absent) can each fail with an Error. Isolating every Throwable here guarantees one failing exporter - // cannot affect the others, nor disrupt the execution. - logSafely("exporter failed", t); - } - } - - private WorkflowInsightRecord buildRecord( - ExecutionState state, - String arn, - String status, - Map operations, - Instant endTime, - Object input, - Object output, - Throwable error) { - WorkflowInsightRecord record = new WorkflowInsightRecord(); - ArnParser a = state.arn; - record.emittedAt = Instant.now().toString(); - record.executionArn = arn; - record.executionName = emptyToNull(a.executionName()); - record.functionName = a.functionName(); - record.functionQualifier = a.qualifier(); - record.region = a.region(); - record.accountId = a.accountId(); - record.status = status; - record.startTime = state.startTime != null ? state.startTime.toString() : null; - if (endTime != null) { - record.endTime = endTime.toString(); - if (state.startTime != null) { - record.durationMs = endTime.toEpochMilli() - state.startTime.toEpochMilli(); - } - } - record.input = applyDataContent( - "input", - input, - content == null || content.includeInput(), - content == null ? null : content.inputTransform()); - record.output = applyDataContent( - "output", - output, - content == null || content.includeOutput(), - content == null ? null : content.outputTransform()); - // Honor ContentConfig.includeErrors for the execution-level error exactly as for operation-level errors - // below: with includeErrors(false) no execution error is emitted, so a sensitive failure message never - // reaches a record. Without this gate the execution error leaked even when errors were disabled. - if (includeErrors && error != null) { - record.error = toErrorInfo(error); - } - record.operations = buildOperationRecords(operations); - return record; - } + // --- helpers --- - private List buildOperationRecords(Map operations) { - List out = new ArrayList<>(); - if (operations == null) { - return out; - } - // The hook contract supplies a map with no iteration-order guarantee (the core snapshot originates from a - // concurrent map). Sort by startTimestamp ascending (null timestamps last), then by a stable operation id - // tie-breaker, so the emitted operations array is deterministic and OperationsIndex's "latest occurrence" - // scalar fields reflect true chronological order rather than arbitrary map iteration order. - List items = new ArrayList<>(operations.values()); - items.sort(Comparator.comparing( - OperationChangeItemInfo::startTimestamp, Comparator.nullsLast(Comparator.naturalOrder())) - .thenComparing(OperationChangeItemInfo::id, Comparator.nullsLast(Comparator.naturalOrder()))); - for (OperationChangeItemInfo item : items) { - // The SDK core tracks the invocation/execution itself as a pseudo-entry of type EXECUTION; it is not a - // customer operation and the record already carries the execution status/timing at top level. - if ("EXECUTION".equals(item.type())) { - continue; - } - // Unnamed operations can't be targeted or keyed — excluded by default (matches JS `if (!op.name)`). - if (item.name() == null) { - continue; - } - // top-level detail drops anything nested under a context (parallel branches, map items, nested steps). - if (topLevelOnly && item.parentId() != null) { - continue; - } - OperationOverride override = overridesByName.get(item.name()); - if (override != null && override.isExclude()) { - continue; - } - OperationRecord rec = new OperationRecord() - .id(item.id()) - .name(item.name()) - .type(item.type()) - .subType(item.subType()) - .parentId(item.parentId()) - .status(item.status() != null ? item.status().toString() : "UNKNOWN") - .startTime( - item.startTimestamp() != null - ? item.startTimestamp().toString() - : null) - .endTime( - item.endTimestamp() != null - ? item.endTimestamp().toString() - : null) - .attempt(item.attempt()); - if (item.startTimestamp() != null && item.endTimestamp() != null) { - rec.durationMs(item.endTimestamp().toEpochMilli() - - item.startTimestamp().toEpochMilli()); - } - if (includeErrors && item.error() != null) { - rec.error(toErrorInfo(item.error())); - } - // Results are omitted unless an override explicitly opts in via a transform (matches JS). - if (override != null && override.result() != null) { - rec.result(applyResultOverride(override.result(), item.result())); - } - out.add(rec); - } - return out; + /** Shapes and exports one record to one exporter; runs on a scheduler worker, never on an SDK hook thread. */ + static void exportRecord(WorkflowInsightRecord record, InsightExporter exporter) { + try { + // Give each exporter its own deep copy: truncation returns the original record when it already fits, + // so without this a custom exporter that mutates operations or nested content would corrupt every + // other exporter's view of the same record. + WorkflowInsightRecord isolated = record.deepCopy(); + WorkflowInsightRecord shaped = + Truncation.truncateRecord(isolated, exporter.maxRecordSizeBytes(), exporter::render); + exporter.export(shaped); + } catch (Throwable t) { + // Catch Throwable, not just RuntimeException: deep copy, truncation, an exporter's render/export, or + // the linkage of an optional exporter class (a NoClassDefFoundError when the S3 / CloudWatch SDK is + // absent) can each fail with an Error. Isolating every Throwable here guarantees one failing exporter + // cannot affect the others, nor disrupt the execution. + logSafely("exporter failed", t); } } - // --- helpers --- - /** * Applies a user-supplied result transform to an operation's checkpointed (serialized JSON) result. Parses the JSON * before handing it to the transform, so the transform always receives a detached, JSON-compatible value @@ -493,7 +141,7 @@ static Object applyDataContent(String label, Object value, boolean include, Func } /** Logs a plugin failure without ever letting the logging itself disrupt durable execution. */ - private static void logSafely(String message, Throwable t) { + static void logSafely(String message, Throwable t) { try { logger.warn("[workflow-insight] {}", message, t); } catch (Throwable ignored) { @@ -501,7 +149,7 @@ private static void logSafely(String message, Throwable t) { } } - private static ErrorInfo toErrorInfo(Throwable t) { + static ErrorInfo toErrorInfo(Throwable t) { // Operation and execution snapshot errors are exposed wrapped: operation failures as DurableOperationException // and unrecoverable execution failures as UnrecoverableDurableExecutionException. The wrapper's own // class/message would lose the original checkpointed failure identity. When the checkpointed ErrorObject is @@ -533,11 +181,11 @@ private static ErrorObject extractErrorObject(Throwable t) { return null; } - private static String emptyToNull(String s) { + static String emptyToNull(String s) { return s == null || s.isEmpty() ? null : s; } - private static String mapStatus(InvocationStatus status) { + static String mapStatus(InvocationStatus status) { if (status == InvocationStatus.SUCCEEDED) { return "SUCCEEDED"; } diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ConcurrentExecutionsExportTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ConcurrentExecutionsExportTest.java index fec4ea396..538e39240 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ConcurrentExecutionsExportTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ConcurrentExecutionsExportTest.java @@ -32,10 +32,11 @@ import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; /** - * One plugin instance — and therefore one {@link ExportScheduler} — serves a whole execution environment, and an - * environment can host several durable executions at once (routine under Lambda Managed Instances). These tests pin the - * per-execution guarantees that concurrency demands: one execution's record never displaces another's, and each - * execution's drain returns only after its own record reached the exporters. + * One {@link ExportScheduler} — created once by {@code workflowInsight()} and shared by every plugin instance the + * factory makes — serves a whole execution environment, and an environment can host several durable executions at once + * (routine under Lambda Managed Instances). These tests pin the per-execution guarantees that concurrency demands: one + * execution's record never displaces another's, and each execution's drain returns only after its own record reached + * the exporters. */ class ConcurrentExecutionsExportTest { @@ -120,15 +121,18 @@ void everyConcurrentExecutionDeliversItsTerminalRecordExactlyOnce() throws Excep var threads = new ArrayList(); for (int i = 0; i < executions; i++) { String executionArn = arn(i); + // One plugin instance per execution, as the SDK creates one per invocation; the thread below holds it + // exactly as an invocation's hooks do. + InsightPlugin execution = Executions.plugin(scheduler, executionArn); var thread = new Thread( () -> { awaitBarrier(barrier); for (int c = 0; c < changesEach; c++) { - scheduler.schedule(executionArn, record(executionArn, "RUNNING")); + scheduler.schedule(execution, record(executionArn, "RUNNING")); } WorkflowInsightRecord terminal = record(executionArn, "SUCCEEDED"); - scheduler.schedule(executionArn, terminal); - scheduler.drain(executionArn); + scheduler.schedule(execution, terminal); + scheduler.drain(execution); // This thread is the only one scheduling for this ARN, so no later record can supersede the // terminal one: once drain returns, it must already have reached the exporter. if (!exporter.exported(terminal)) { @@ -185,15 +189,16 @@ void anExitingPumpDoesNotReleaseADrainWhoseRecordIsStillInsideTheExporter() thro executor); String executionArn = arn(1); + InsightPlugin execution = Executions.plugin(scheduler, executionArn); WorkflowInsightRecord terminal = record(executionArn, "SUCCEEDED"); - scheduler.schedule(executionArn, terminal); + scheduler.schedule(execution, terminal); assertEquals(1, executor.parked.size(), "the pump task was parked, so the record is still queued"); // A drainer picks the record up on the inline path and is now inside the exporter. var inlineDrained = new CountDownLatch(1); var inlineDrainer = new Thread( () -> { - scheduler.drain(executionArn); + scheduler.drain(execution); inlineDrained.countDown(); }, "inline-drainer"); @@ -208,7 +213,7 @@ void anExitingPumpDoesNotReleaseADrainWhoseRecordIsStillInsideTheExporter() thro var secondDrained = new CountDownLatch(1); var secondDrainer = new Thread( () -> { - scheduler.drain(executionArn); + scheduler.drain(execution); secondDrained.countDown(); }, "second-drainer"); @@ -242,14 +247,16 @@ public void export(WorkflowInsightRecord record) { } }; var scheduler = scheduler(new CopyOnWriteArrayList<>(), exporter); + InsightPlugin slow = Executions.plugin(scheduler, slowExecution); + InsightPlugin other = Executions.plugin(scheduler, otherExecution); // One execution's export is in flight and blocked... - scheduler.schedule(slowExecution, record(slowExecution, "RUNNING")); + scheduler.schedule(slow, record(slowExecution, "RUNNING")); assertTrue(exporting.await(5, TimeUnit.SECONDS), "the first export is in flight"); // ...while a second execution's terminal record is queued, followed by an update for the first execution. // The first execution's own update must coalesce only with its own slot, never over the second execution's. - scheduler.schedule(otherExecution, record(otherExecution, "SUCCEEDED")); - scheduler.schedule(slowExecution, record(slowExecution, "SUCCEEDED")); + scheduler.schedule(other, record(otherExecution, "SUCCEEDED")); + scheduler.schedule(slow, record(slowExecution, "SUCCEEDED")); var seenBySlowDrain = Collections.synchronizedList(new ArrayList()); var seenByOtherDrain = Collections.synchronizedList(new ArrayList()); @@ -257,14 +264,14 @@ public void export(WorkflowInsightRecord record) { var otherDrained = new CountDownLatch(1); var slowDrainer = new Thread( () -> { - scheduler.drain(slowExecution); + scheduler.drain(slow); seenBySlowDrain.addAll(terminalArns(exporter)); slowDrained.countDown(); }, "slow-drainer"); var otherDrainer = new Thread( () -> { - scheduler.drain(otherExecution); + scheduler.drain(other); seenByOtherDrain.addAll(terminalArns(exporter)); otherDrained.countDown(); }, @@ -297,19 +304,25 @@ public void export(WorkflowInsightRecord record) { void concurrentExecutionsDrivenThroughThePluginHooksAllDeliverTheirTerminalRecord() throws Exception { int executions = 5; var exporter = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + // One factory — one environment, one scheduler, one set of exporters — and one plugin instance per invocation, + // which is how the SDK drives several concurrent executions through the same exporters. + var factory = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) .addExporter(exporter) .build()); var barrier = new CyclicBarrier(executions); + var plugins = new ArrayList(); var threads = new ArrayList(); for (int i = 0; i < executions; i++) { String executionArn = arn(i); + InvocationInfo startInfo = start(executionArn); + var plugin = Executions.plugin(factory, startInfo); + plugins.add(plugin); var thread = new Thread( () -> { awaitBarrier(barrier); - plugin.onInvocationStart(start(executionArn)); + plugin.onInvocationStart(startInfo); for (int c = 0; c < 3; c++) { plugin.onOperationChange(new OperationChangeInfo( "req", @@ -335,7 +348,11 @@ void concurrentExecutionsDrivenThroughThePluginHooksAllDeliverTheirTerminalRecor } assertEquals(expected, new HashSet<>(delivered), "every execution's terminal record arrived"); assertEquals(executions, delivered.size(), "and none arrived twice"); - assertEquals(0, plugin.retainedStateCount(), "no per-execution state retained after invocation end"); + for (InsightPlugin plugin : plugins) { + assertFalse( + Executions.outstanding(plugin), + "the scheduler still owes this execution work after its invocation end: " + plugin); + } } private static Map ops(OperationStatus status) { diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ErrorPrivacyGateTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ErrorPrivacyGateTest.java index 0cd5ede12..2e0a47af9 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ErrorPrivacyGateTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ErrorPrivacyGateTest.java @@ -14,7 +14,6 @@ import java.util.Map; import org.junit.jupiter.api.Test; import software.amazon.awssdk.services.lambda.model.OperationStatus; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; @@ -64,10 +63,15 @@ private Map failingOp() { } private WorkflowInsightRecord runFailedExecution(boolean includeErrors, CapturingExporter exporter) { - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .content(ContentConfig.builder().includeErrors(includeErrors).build()) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .content(ContentConfig.builder() + .includeErrors(includeErrors) + .build()) + .addExporter(exporter) + .build()), + ARN, + START); plugin.onInvocationStart(new InvocationInfo("req", ARN, true, START, "in", failingOp(), Map.of())); plugin.onInvocationEnd(new InvocationEndInfo( "req", diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/Executions.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/Executions.java new file mode 100644 index 000000000..51d30e414 --- /dev/null +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/Executions.java @@ -0,0 +1,77 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import java.time.Instant; +import java.util.Map; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; +import software.amazon.lambda.durable.plugin.InvocationInfo; + +/** + * Test helper: builds the per-invocation plugin instances the {@link ExportScheduler} schedules for. + * + *

The scheduler no longer resolves an execution ARN to anything — an invocation's state is its plugin + * instance, created from the {@link InvocationInfo} the SDK is about to hand the first hook — so tests hold the + * instance exactly as the SDK does. Everything here goes through the same constructor and the same factory production + * uses; nothing is a test-only back door into the scheduler. + */ +final class Executions { + + private static final Instant START = Instant.parse("2026-08-05T00:00:00Z"); + + private Executions() {} + + /** The invocation description the SDK would build for one execution, with no payload or operation snapshot. */ + static InvocationInfo info(String executionArn) { + return new InvocationInfo("req", executionArn, true, START, null, Map.of(), Map.of()); + } + + /** + * One invocation's plugin instance, bound to this scheduler and configured with the plugin's defaults. For tests + * that drive the scheduler directly and do not care how records are shaped. + */ + static InsightPlugin plugin(ExportScheduler scheduler, String executionArn) { + return new InsightPlugin( + new InsightSettings(WorkflowInsightConfig.builder().build()), scheduler, info(executionArn)); + } + + /** + * One invocation's plugin instance from the factory, for the identity the SDK would have built it with. The + * invocation's own {@code InvocationInfo} still goes to {@code onInvocationStart}; this is the same pair of facts + * that info carries, which is all an instance's identity is. + */ + static InsightPlugin plugin(DurableExecutionPluginFactory factory, String executionArn, Instant startTime) { + return plugin(factory, new InvocationInfo("req", executionArn, true, startTime, null, Map.of(), Map.of())); + } + + /** + * One invocation's plugin instance, exactly as the SDK creates it: from the factory, with that invocation's info. + */ + static InsightPlugin plugin(DurableExecutionPluginFactory factory, InvocationInfo info) { + return (InsightPlugin) factory.createPlugin(info); + } + + /** + * Whether the scheduler still owes this invocation anything: a queued record, a record inside the exporters, an + * uncompleted drain signal, or a drain waiting on it. Read under the monitor those fields are guarded by. + * + *

This is the question the plugin's {@code retainedStateCount()} seam used to answer for a whole registry. There + * is no registry to count now — an invocation's state is its plugin instance, and the SDK drops it — so the + * property worth asserting is that nothing the environment outlives keeps hold of it. + */ + static boolean outstanding(InsightPlugin plugin) { + synchronized (plugin.scheduler) { + return plugin.record != null || plugin.exporting || plugin.settled != null || plugin.drainWaiters > 0; + } + } + + /** + * The instance plus its first hook, in the order the SDK dispatches them: the factory is called with the very + * {@link InvocationInfo} that {@code onInvocationStart} then receives. + */ + static InsightPlugin started(DurableExecutionPluginFactory factory, InvocationInfo info) { + InsightPlugin plugin = plugin(factory, info); + plugin.onInvocationStart(info); + return plugin; + } +} diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushCoalescingTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushCoalescingTest.java index 014732f76..fc162981f 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushCoalescingTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushCoalescingTest.java @@ -92,9 +92,10 @@ void invocationEndsAskingForAFlushTogetherShareOneFlush() { var done = new CountDownLatch(executions); for (int i = 0; i < executions; i++) { String executionArn = arn(i); + InsightPlugin execution = Executions.plugin(scheduler, executionArn); start("end-" + i, () -> { - scheduler.schedule(executionArn, record(executionArn, "SUCCEEDED")); - scheduler.drain(executionArn); + scheduler.schedule(execution, record(executionArn, "SUCCEEDED")); + scheduler.drain(execution); awaitBarrier(recordsDelivered); long began = System.nanoTime(); scheduler.flush(); @@ -136,11 +137,12 @@ void simultaneousDrainAndFlushEndsShareAFlushRatherThanOneEach() { var done = new CountDownLatch(executions); for (int i = 0; i < executions; i++) { String executionArn = arn(i); + InsightPlugin execution = Executions.plugin(scheduler, executionArn); start("drain-and-flush-" + i, () -> { awaitBarrier(barrier); long began = System.nanoTime(); - scheduler.schedule(executionArn, record(executionArn, "SUCCEEDED")); - scheduler.drain(executionArn); + scheduler.schedule(execution, record(executionArn, "SUCCEEDED")); + scheduler.drain(execution); scheduler.flush(); durations.add((System.nanoTime() - began) / 1_000_000L); done.countDown(); @@ -247,9 +249,10 @@ public void flush() { var violations = new CopyOnWriteArrayList(); for (int i = 0; i < requests; i++) { String executionArn = arn(i); + InsightPlugin execution = Executions.plugin(scheduler, executionArn); start("load-flusher-" + i, () -> { - scheduler.schedule(executionArn, record(executionArn, "SUCCEEDED")); - scheduler.drain(executionArn); + scheduler.schedule(execution, record(executionArn, "SUCCEEDED")); + scheduler.drain(execution); int startsBefore = flushStarts.get(); scheduler.flush(); if (flushCompletions.get() <= startsBefore) { diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushSerializationTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushSerializationTest.java index b37677d2a..ff6254539 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushSerializationTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFlushSerializationTest.java @@ -126,15 +126,16 @@ void aFlushNeverOverlapsAnExportEvenWithManyExecutionsEndingAtOnce() throws Exce var threads = new ArrayList(); for (int i = 0; i < executions; i++) { String executionArn = arn(i); + InsightPlugin execution = Executions.plugin(scheduler, executionArn); var thread = new Thread( () -> { awaitBarrier(barrier); // What an invocation does: a few RUNNING snapshots, the terminal record, then drain + flush. for (int change = 0; change < 3; change++) { - scheduler.schedule(executionArn, record(executionArn, "RUNNING")); + scheduler.schedule(execution, record(executionArn, "RUNNING")); } - scheduler.schedule(executionArn, record(executionArn, "SUCCEEDED")); - scheduler.drain(executionArn); + scheduler.schedule(execution, record(executionArn, "SUCCEEDED")); + scheduler.drain(execution); scheduler.flush(); }, "invocation-" + i); @@ -185,7 +186,7 @@ public void flush() { }; var scheduler = scheduler(sharedWorkers(), new CopyOnWriteArrayList<>(), exporter); - scheduler.schedule(arn(0), record(arn(0), "SUCCEEDED")); + scheduler.schedule(Executions.plugin(scheduler, arn(0)), record(arn(0), "SUCCEEDED")); assertTrue(exporting.await(5, TimeUnit.SECONDS), "the pump is inside the exporter"); var flushed = new CountDownLatch(2); @@ -299,6 +300,10 @@ public void flush() { // A producer that never lets the queue run dry: it keeps re-scheduling a fixed, rotating set of executions, so // `pending` stays non-empty (and bounded, since records coalesce per execution) for as long as it runs. + var rotation = new ArrayList(); + for (int i = 0; i < 50; i++) { + rotation.add(Executions.plugin(scheduler, arn(i))); + } var stop = new AtomicBoolean(); var scheduled = new AtomicInteger(); var producing = new CountDownLatch(1); @@ -306,8 +311,8 @@ public void flush() { () -> { int index = 0; while (!stop.get()) { - String executionArn = arn(index++ % 50); - scheduler.schedule(executionArn, record(executionArn, "RUNNING")); + InsightPlugin execution = rotation.get(index++ % rotation.size()); + scheduler.schedule(execution, record(execution.executionArn, "RUNNING")); scheduled.incrementAndGet(); producing.countDown(); } diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerReentrantFlushTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerReentrantFlushTest.java index 42da4c005..bb0577ca2 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerReentrantFlushTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerReentrantFlushTest.java @@ -86,10 +86,11 @@ void flushReenteredFromThePumpThreadIsRefusedReportedAndLeavesTheSchedulerUsable holder.set(scheduler); var drainReturned = new CountDownLatch(1); + var firstExecution = Executions.plugin(scheduler, arn(0)); var invocation = new Thread( () -> { - scheduler.schedule(arn(0), record(arn(0), "SUCCEEDED")); - scheduler.drain(arn(0)); + scheduler.schedule(firstExecution, record(arn(0), "SUCCEEDED")); + scheduler.drain(firstExecution); drainReturned.countDown(); }, "reentrant-flush-invocation"); @@ -118,8 +119,9 @@ void flushReenteredFromThePumpThreadIsRefusedReportedAndLeavesTheSchedulerUsable // Still usable: the next invocation's record is exported and its flush — from a thread that is not the pump — // is // served exactly as before. - scheduler.schedule(arn(1), record(arn(1), "SUCCEEDED")); - scheduler.drain(arn(1)); + var secondExecution = Executions.plugin(scheduler, arn(1)); + scheduler.schedule(secondExecution, record(arn(1), "SUCCEEDED")); + scheduler.drain(secondExecution); scheduler.flush(); assertEquals(2, exporter.exports.get(), "both records reached the exporter"); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerTest.java index adca36bdf..747681318 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerTest.java @@ -22,7 +22,7 @@ /** Contract tests for {@link ExportScheduler}: serial exports, latest-wins coalescing, drain, and exporter fan-out. */ class ExportSchedulerTest { - /** All single-execution cases below drive one execution ARN through the scheduler. */ + /** All single-execution cases below drive one invocation's plugin instance through the scheduler. */ private static final String ARN = arn(0); private static String arn(int index) { @@ -79,8 +79,9 @@ void scheduleHandsTheRecordToAWorkerRatherThanExportingOnTheCallingThread() { var executor = new ManualExecutor(); var exporter = new CapturingExporter(); var scheduler = scheduler(executor, new ArrayList<>(), exporter); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("RUNNING")); + scheduler.schedule(execution, record("RUNNING")); assertTrue(exporter.records.isEmpty(), "nothing exported until a worker runs"); executor.runAll(); @@ -93,10 +94,11 @@ void updatesScheduledBeforeTheWorkerRunsCollapseIntoTheLatestRecord() { var executor = new ManualExecutor(); var exporter = new CapturingExporter(); var scheduler = scheduler(executor, new ArrayList<>(), exporter); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("r1")); - scheduler.schedule(ARN, record("r2")); - scheduler.schedule(ARN, record("r3")); + scheduler.schedule(execution, record("r1")); + scheduler.schedule(execution, record("r2")); + scheduler.schedule(execution, record("r3")); executor.runAll(); assertEquals(1, exporter.records.size(), "one pump, one latest record"); @@ -109,10 +111,11 @@ void recordScheduledAfterAPumpFinishesStartsANewPump() { var executor = new ManualExecutor(); var exporter = new CapturingExporter(); var scheduler = scheduler(executor, new ArrayList<>(), exporter); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("first")); + scheduler.schedule(execution, record("first")); executor.runAll(); - scheduler.schedule(ARN, record("second")); + scheduler.schedule(execution, record("second")); executor.runAll(); assertEquals(List.of("first", "second"), statuses(exporter)); @@ -133,14 +136,15 @@ public void export(WorkflowInsightRecord record) { } }; var scheduler = scheduler(sharedWorkers(), new ArrayList<>(), exporter); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("first")); + scheduler.schedule(execution, record("first")); assertTrue(entered.await(5, TimeUnit.SECONDS), "first export is in flight"); - scheduler.schedule(ARN, record("dropped-1")); - scheduler.schedule(ARN, record("dropped-2")); - scheduler.schedule(ARN, record("final")); + scheduler.schedule(execution, record("dropped-1")); + scheduler.schedule(execution, record("dropped-2")); + scheduler.schedule(execution, record("final")); release.countDown(); - scheduler.drain(ARN); + scheduler.drain(execution); assertEquals(List.of("first", "final"), statuses(exporter)); } @@ -148,8 +152,9 @@ public void export(WorkflowInsightRecord record) { @Test void drainReturnsImmediatelyWhenIdle() { var scheduler = scheduler(new ManualExecutor(), new ArrayList<>(), new CapturingExporter()); - scheduler.drain(ARN); - scheduler.drain(ARN); + var execution = Executions.plugin(scheduler, ARN); + scheduler.drain(execution); + scheduler.drain(execution); } @Test @@ -167,14 +172,15 @@ public void export(WorkflowInsightRecord record) { } }; var scheduler = scheduler(sharedWorkers(), new ArrayList<>(), exporter); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("slow")); + scheduler.schedule(execution, record("slow")); assertTrue(entered.await(5, TimeUnit.SECONDS)); - scheduler.schedule(ARN, record("final")); + scheduler.schedule(execution, record("final")); var drained = new CountDownLatch(1); var drainer = new Thread(() -> { - scheduler.drain(ARN); + scheduler.drain(execution); drained.countDown(); }); drainer.start(); @@ -189,9 +195,10 @@ public void export(WorkflowInsightRecord record) { void exportersRunOffTheSchedulingThread() { var exporter = new CapturingExporter(); var scheduler = scheduler(sharedWorkers(), new ArrayList<>(), exporter); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("RUNNING")); - scheduler.drain(ARN); + scheduler.schedule(execution, record("RUNNING")); + scheduler.drain(execution); assertEquals(1, exporter.threads.size()); assertNotSame(Thread.currentThread(), exporter.threads.get(0)); @@ -205,9 +212,10 @@ void aFailingExporterNeverBlocksTheOthersForTheSameRecord() { throw new AssertionError("exporter blew up"); }; var scheduler = scheduler(sharedWorkers(), failures, bad, good); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("RUNNING")); - scheduler.drain(ARN); + scheduler.schedule(execution, record("RUNNING")); + scheduler.drain(execution); assertEquals(1, good.records.size()); assertEquals(1, failures.size()); @@ -220,8 +228,9 @@ void exportersForOneRecordRunConcurrentlySoASlowExporterDoesNotDelayTheOthers() var fast = new CapturingExporter(); InsightExporter slow = record -> await(release); var scheduler = scheduler(sharedWorkers(), new ArrayList<>(), slow, fast); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("RUNNING")); + scheduler.schedule(execution, record("RUNNING")); long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); while (fast.records.isEmpty() && System.nanoTime() < deadline) { Thread.sleep(5); @@ -229,7 +238,7 @@ void exportersForOneRecordRunConcurrentlySoASlowExporterDoesNotDelayTheOthers() assertEquals(1, fast.records.size(), "fast exporter received the record while the slow one is still blocked"); release.countDown(); - scheduler.drain(ARN); + scheduler.drain(execution); } @Test @@ -239,12 +248,13 @@ void drainExportsThePendingRecordInlineWhenNoWorkerCouldBeStarted() { var failures = new ArrayList(); var exporter = new CapturingExporter(); var scheduler = scheduler(executor, failures, exporter); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("final")); + scheduler.schedule(execution, record("final")); assertTrue(exporter.records.isEmpty(), "the hook thread does not export"); assertEquals(1, failures.size(), "the worker failure is reported"); - scheduler.drain(ARN); + scheduler.drain(execution); assertEquals(List.of("final"), statuses(exporter)); assertSame(Thread.currentThread(), exporter.threads.get(0), "the invocation boundary delivers it"); @@ -256,14 +266,15 @@ void aLaterScheduleRetriesTheWorkerAfterARejection() { executor.reject = true; var exporter = new CapturingExporter(); var scheduler = scheduler(executor, new ArrayList<>(), exporter); + var execution = Executions.plugin(scheduler, ARN); - scheduler.schedule(ARN, record("older")); + scheduler.schedule(execution, record("older")); executor.reject = false; - scheduler.schedule(ARN, record("newer")); + scheduler.schedule(execution, record("newer")); executor.runAll(); assertEquals(List.of("newer"), statuses(exporter), "the retry exports the latest record"); - scheduler.drain(ARN); + scheduler.drain(execution); assertEquals(1, exporter.records.size()); } @@ -279,15 +290,16 @@ void aDrainThatObservedTheHandleBeforeTheWorkerWasRejectedStillCompletesInline() var failures = new CopyOnWriteArrayList(); var exporter = new CapturingExporter(); var scheduler = scheduler(blockingRejector, failures, exporter); + var execution = Executions.plugin(scheduler, ARN); - var scheduling = new Thread(() -> scheduler.schedule(ARN, record("final")), "scheduling"); + var scheduling = new Thread(() -> scheduler.schedule(execution, record("final")), "scheduling"); scheduling.start(); assertTrue(submitted.await(5, TimeUnit.SECONDS), "the pump handle is published before execute rejects"); var drained = new CountDownLatch(1); var drainer = new Thread( () -> { - scheduler.drain(ARN); + scheduler.drain(execution); drained.countDown(); }, "drainer"); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExporterIsolationTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExporterIsolationTest.java index 61fe8ae68..4a297c100 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExporterIsolationTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExporterIsolationTest.java @@ -69,14 +69,17 @@ public void export(WorkflowInsightRecord record) { void firstExporterMutationsDoNotLeakIntoLaterExporter() { var mutating = new MutatingExporter(); var good = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .content(ContentConfig.builder() - .addOverride(OperationOverride.withResult("compute", r -> r)) - .build()) - .addExporter(mutating) - .addExporter(good) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .content(ContentConfig.builder() + .addOverride(OperationOverride.withResult("compute", r -> r)) + .build()) + .addExporter(mutating) + .addExporter(good) + .build()), + ARN, + START); Map input = new LinkedHashMap<>(); input.put("k", "v"); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/InputSnapshotTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/InputSnapshotTest.java index abd5f89f7..4af7d55fb 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/InputSnapshotTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/InputSnapshotTest.java @@ -13,7 +13,6 @@ import java.util.function.Function; import org.junit.jupiter.api.Test; import software.amazon.awssdk.services.lambda.model.OperationStatus; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; @@ -64,8 +63,11 @@ private Map ops() { @Test void handlerMutationAfterStartDoesNotCorruptCachedInputSnapshot() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( - WorkflowInsightConfig.builder().addExporter(exporter).build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()), + ARN, + START); // A mutable input whose nested list the handler mutates after the invocation has started. List items = new ArrayList<>(); @@ -99,13 +101,16 @@ void mutatingInputTransformDoesNotAccumulateAcrossEmissions() { return v; }; var exporter = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .content(ContentConfig.builder() - .inputTransform(mutatingTransform) - .build()) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .content(ContentConfig.builder() + .inputTransform(mutatingTransform) + .build()) + .addExporter(exporter) + .build()), + ARN, + START); List items = new ArrayList<>(); items.add("a"); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/JsonJavaTimeTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/JsonJavaTimeTest.java index e57366bba..91d5ec9da 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/JsonJavaTimeTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/JsonJavaTimeTest.java @@ -55,10 +55,13 @@ public void export(WorkflowInsightRecord record) { @Test void pluginOutputWithInstantInInputSerializesInsteadOfDropping() { var exporter = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .addExporter(exporter) + .build()), + ARN, + START); Map input = new LinkedHashMap<>(); input.put("startedAt", TS); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/MutableNumberIsolationTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/MutableNumberIsolationTest.java index 19f906209..e9191ee8f 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/MutableNumberIsolationTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/MutableNumberIsolationTest.java @@ -113,11 +113,14 @@ public void export(WorkflowInsightRecord record) { } }; var good = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .addExporter(mutating) - .addExporter(good) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .addExporter(mutating) + .addExporter(good) + .build()), + ARN, + START); AtomicInteger topLevel = new AtomicInteger(3); List list = new ArrayList<>(); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/OperationErrorIdentityTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/OperationErrorIdentityTest.java index 5e67cd554..116823363 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/OperationErrorIdentityTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/OperationErrorIdentityTest.java @@ -15,7 +15,6 @@ import software.amazon.awssdk.services.lambda.model.Operation; import software.amazon.awssdk.services.lambda.model.OperationStatus; import software.amazon.lambda.durable.exception.DurableOperationException; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; @@ -72,8 +71,11 @@ private Map failedOps(Throwable opError) { @Test void operationAndExecutionErrorUseCheckpointedErrorObjectIdentity() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( - WorkflowInsightConfig.builder().addExporter(exporter).build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()), + ARN, + START); Throwable opError = wrapped("CustomerValidationError", "invalid postal code"); Throwable execError = wrapped("OrchestrationFailure", "workflow aborted"); @@ -98,8 +100,11 @@ void operationAndExecutionErrorUseCheckpointedErrorObjectIdentity() { @Test void fallsBackToThrowableFieldsWhenErrorObjectFieldsMissing() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( - WorkflowInsightConfig.builder().addExporter(exporter).build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()), + ARN, + START); // ErrorObject present but errorType null: name falls back to the throwable's simple class name. ErrorObject partial = diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/OperationOrderingTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/OperationOrderingTest.java index d1ca11b03..fb5e9a94c 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/OperationOrderingTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/OperationOrderingTest.java @@ -44,10 +44,13 @@ private static OperationChangeItemInfo item( private WorkflowInsightRecord emitStart(Map ops) { var exporter = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .addExporter(exporter) + .build()), + ARN, + START); plugin.onInvocationStart(new InvocationInfo("req", ARN, true, START, "in", ops, Map.of())); plugin.drainExports(); return exporter.records.get(0); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/PluginThrowableContainmentTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/PluginThrowableContainmentTest.java index 721d40d5a..c0d073ac0 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/PluginThrowableContainmentTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/PluginThrowableContainmentTest.java @@ -14,11 +14,11 @@ import java.util.Map; import org.junit.jupiter.api.Test; import software.amazon.awssdk.services.lambda.model.OperationStatus; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; +import software.amazon.lambda.durable.plugin.PluginRunner; /** * Fix 2 — plugin {@link Throwable} containment. A plugin fault at any plugin-owned boundary (record construction, input @@ -86,29 +86,34 @@ private InvocationEndInfo end(Object input) { @Test void aNullExecutionArnEscapesNoHook() { - // The SDK's contract for these hooks is that a plugin fault never disrupts durable execution, so an input the - // plugin cannot key its per-execution state by must be contained rather than thrown back. onInvocationEnd is - // the case that matters: its state removal runs last, in a `finally`, and a ConcurrentHashMap cannot remove a - // null key. + // The SDK's contract is that a plugin fault never disrupts durable execution, so an invocation whose execution + // ARN the plugin cannot use must be contained rather than thrown back. It is contained one step earlier now: + // identity is taken when the instance is built, so the failure happens in the factory and no hook is ever + // dispatched. That containment belongs to the SDK, so it is asserted through the SDK's own runner — which is + // also what makes the old worst case ("the state removal runs last, in a finally, and a ConcurrentHashMap + // cannot remove a null key") unreachable: there is no map and no removal. var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( + var environment = WorkflowInsight.workflowInsight( WorkflowInsightConfig.builder().addExporter(exporter).build()); InvocationInfo nullStart = new InvocationInfo("req", null, true, START, "in", ops("compute"), Map.of()); InvocationEndInfo nullEnd = new InvocationEndInfo( "req", null, true, START, ops("compute"), InvocationStatus.SUCCEEDED, null, "in", "out"); - assertDoesNotThrow(() -> plugin.onInvocationStart(nullStart), "onInvocationStart must contain a null ARN"); + var runner = new PluginRunner(List.of(environment)); + assertDoesNotThrow(() -> runner.onInvocationStart(nullStart), "onInvocationStart must contain a null ARN"); assertDoesNotThrow( - () -> plugin.onOperationChange(new software.amazon.lambda.durable.plugin.OperationChangeInfo( + () -> runner.onOperationChange(new software.amazon.lambda.durable.plugin.OperationChangeInfo( "req", null, ops("compute"), ops("compute"))), "onOperationChange must contain a null ARN"); - assertDoesNotThrow(() -> plugin.onInvocationEnd(nullEnd), "onInvocationEnd must contain a null ARN"); + assertDoesNotThrow(() -> runner.onInvocationEnd(nullEnd), "onInvocationEnd must contain a null ARN"); + assertEquals(0, exporter.records.size(), "an invocation with no usable ARN emits nothing"); - // The plugin is still usable afterwards: a well-formed execution on the same instance still emits and flushes. + // The environment is still usable afterwards: a well-formed invocation still emits and flushes. + var plugin = Executions.plugin(environment, ARN, START); plugin.onInvocationStart(start("in")); plugin.onInvocationEnd(end("in")); - assertEquals(1, exporter.records.size(), "the plugin still works after a null-ARN invocation"); + assertEquals(1, exporter.records.size(), "the environment still works after a null-ARN invocation"); assertTrue(exporter.flushes > 0); } @@ -121,10 +126,13 @@ public void export(WorkflowInsightRecord record) { } }; var good = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .addExporter(throwing) - .addExporter(good) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .addExporter(throwing) + .addExporter(good) + .build()), + ARN, + START); plugin.onInvocationStart(start("in")); plugin.onInvocationEnd(end("in")); @@ -136,8 +144,11 @@ public void export(WorkflowInsightRecord record) { @Test void inputSnapshotErrorOmitsInputButDoesNotDisruptExecution() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( - WorkflowInsightConfig.builder().addExporter(exporter).build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()), + ARN, + START); // Snapshotting the input fails with an Error; the hook must not propagate it. plugin.onInvocationStart(start(new ExplodingPayload())); @@ -152,14 +163,17 @@ void inputSnapshotErrorOmitsInputButDoesNotDisruptExecution() { @Test void throwingInputTransformOmitsInputWithoutFailure() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .content(ContentConfig.builder() - .inputTransform(v -> { - throw new AssertionError("redactor blew up"); - }) - .build()) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .content(ContentConfig.builder() + .inputTransform(v -> { + throw new AssertionError("redactor blew up"); + }) + .build()) + .addExporter(exporter) + .build()), + ARN, + START); plugin.onInvocationStart(start("in")); plugin.onInvocationEnd(end("in")); @@ -172,14 +186,17 @@ void throwingInputTransformOmitsInputWithoutFailure() { @Test void throwingResultTransformOmitsResultWithoutFailure() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .content(ContentConfig.builder() - .addOverride(OperationOverride.withResult("compute", r -> { - throw new AssertionError("result redactor blew up"); - })) - .build()) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .content(ContentConfig.builder() + .addOverride(OperationOverride.withResult("compute", r -> { + throw new AssertionError("result redactor blew up"); + })) + .build()) + .addExporter(exporter) + .build()), + ARN, + START); plugin.onInvocationStart(start("in")); plugin.onInvocationEnd(end("in")); @@ -203,11 +220,14 @@ public void export(WorkflowInsightRecord record) { } }; var third = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .addExporter(first) - .addExporter(throwing) - .addExporter(third) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .addExporter(first) + .addExporter(throwing) + .addExporter(third) + .build()), + ARN, + START); plugin.onInvocationStart(start("in")); plugin.onInvocationEnd(end("in")); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/StateCleanupLifecycleTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/StateCleanupLifecycleTest.java index da2768602..3775cbf4e 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/StateCleanupLifecycleTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/StateCleanupLifecycleTest.java @@ -3,7 +3,10 @@ package software.amazon.lambda.durable.insight; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import java.lang.ref.WeakReference; import java.time.Instant; import java.util.ArrayList; import java.util.LinkedHashMap; @@ -11,15 +14,23 @@ import java.util.Map; import org.junit.jupiter.api.Test; import software.amazon.awssdk.services.lambda.model.OperationStatus; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; /** - * Finding {@code arf_v1_qh6xoafzze3z3ccgrbppucmunr} ([P2] remove retained suspended execution state): per-execution - * state must be removed on every {@code onInvocationEnd}, including non-terminal PENDING/RETRYING suspends, so a warm - * container never leaks one entry per suspended execution. A resume re-seeds identical stable start time and input. + * Finding {@code arf_v1_qh6xoafzze3z3ccgrbppucmunr} ([P2] remove retained suspended execution state): a warm container + * must never accumulate per-execution state, including for executions that suspend (PENDING/RETRYING) and never + * terminate in that container. A resume re-seeds identical stable start time and input. + * + *

The plugin used to keep that state in an ARN-keyed map and remove the entry at every invocation end, so the test + * counted the entries left behind. There is no map now — an invocation's state is its plugin instance, which + * the SDK creates per invocation and drops when it returns — so the two things worth proving are that the environment + * (the factory's scheduler, which does outlive invocations) owes a finished invocation nothing, and that it holds no + * reference to the instance once the invocation is over. A retained entry of any kind would fail the second assertion, + * which the old count could not make: it could only count the entries the plugin knew it had. */ class StateCleanupLifecycleTest { @@ -58,57 +69,93 @@ private static Map ops() { return m; } + private static InvocationInfo start(int i) { + return new InvocationInfo("req", arn(i), true, START, "in-" + i, ops(), Map.of()); + } + + private static InvocationEndInfo end(int i, InvocationStatus status) { + return new InvocationEndInfo("req", arn(i), true, START, ops(), status, null, "in-" + i, null); + } + + /** + * Runs one whole invocation in the given environment and returns a weak reference to the instance that served it, + * keeping no strong reference of its own — so whatever the reference still points at afterwards is retained by the + * environment, not by this test. + */ + private static WeakReference runInvocation( + DurableExecutionPluginFactory environment, int i, InvocationStatus status) { + InsightPlugin plugin = Executions.started(environment, start(i)); + plugin.onInvocationEnd(end(i, status)); + assertFalse(Executions.outstanding(plugin), "the scheduler still owes execution " + i + " work"); + return new WeakReference<>(plugin); + } + + /** True once every instance has been collected; polls, because a single GC request need not clear them. */ + private static boolean allCollected(List> instances) { + for (int attempt = 0; attempt < 50; attempt++) { + if (instances.stream().allMatch(reference -> reference.get() == null)) { + return true; + } + System.gc(); + try { + Thread.sleep(10); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + return false; + } + } + return instances.stream().allMatch(reference -> reference.get() == null); + } + @Test void nDistinctPendingExecutionsLeaveNoRetainedState() { - var plugin = (WorkflowInsight.InsightPlugin) + var environment = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder().build()); int n = 25; + var instances = new ArrayList>(); for (int i = 0; i < n; i++) { - String arn = arn(i); - plugin.onInvocationStart(new InvocationInfo("req", arn, true, START, "in-" + i, ops(), Map.of())); // Each execution suspends (PENDING) and never terminates in this container. - plugin.onInvocationEnd(new InvocationEndInfo( - "req", arn, true, START, ops(), InvocationStatus.PENDING, null, "in-" + i, null)); + instances.add(runInvocation(environment, i, InvocationStatus.PENDING)); } - assertEquals(0, plugin.retainedStateCount(), "no per-execution state retained for suspended executions"); + assertTrue( + allCollected(instances), + "the environment still holds the state of a suspended execution after its invocation ended"); } @Test void retryingSuspendAlsoLeavesNoRetainedState() { - var plugin = (WorkflowInsight.InsightPlugin) + var environment = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder().build()); - String arn = arn(0); - plugin.onInvocationStart(new InvocationInfo("req", arn, true, START, "in", ops(), Map.of())); - plugin.onInvocationEnd( - new InvocationEndInfo("req", arn, true, START, ops(), InvocationStatus.RETRYING, null, "in", null)); - assertEquals(0, plugin.retainedStateCount(), "RETRYING suspend also clears state"); + var instance = runInvocation(environment, 0, InvocationStatus.RETRYING); + assertTrue(allCollected(List.of(instance)), "a RETRYING suspend leaves nothing retained either"); } @Test void resumeReSeedsStableStartTimeAndInput() { var exporter = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + var environment = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) .addExporter(exporter) .build()); - String arn = arn(0); - // First invocation with input "alpha", then suspend (state removed). - plugin.onInvocationStart(new InvocationInfo("req", arn, true, START, "alpha", ops(), Map.of())); - plugin.onInvocationEnd( - new InvocationEndInfo("req", arn, true, START, ops(), InvocationStatus.PENDING, null, "alpha", null)); + // First invocation with input "alpha", then suspend. Its instance is dropped with it. + var first = Executions.started( + environment, new InvocationInfo("req", arn(0), true, START, "alpha", ops(), Map.of())); + first.onInvocationEnd(new InvocationEndInfo( + "req", arn(0), true, START, ops(), InvocationStatus.PENDING, null, "alpha", null)); - // Resume invocation: onInvocationStart re-seeds state from hook data (same START, same input). - plugin.onInvocationStart(new InvocationInfo("req", arn, false, START, "alpha", ops(), Map.of())); - plugin.onInvocationEnd(new InvocationEndInfo( - "req", arn, true, START, ops(), InvocationStatus.SUCCEEDED, null, "alpha", "out")); + // Resume invocation: a new instance, seeded from the resume's own hook data (same START, same input). + var resumed = Executions.started( + environment, new InvocationInfo("req", arn(0), false, START, "alpha", ops(), Map.of())); + resumed.onInvocationEnd(new InvocationEndInfo( + "req", arn(0), true, START, ops(), InvocationStatus.SUCCEEDED, null, "alpha", "out")); var terminal = exporter.records.get(exporter.records.size() - 1); assertEquals("SUCCEEDED", terminal.status()); assertEquals(START.toString(), terminal.startTime(), "stable start time recreated across the suspend boundary"); assertEquals("alpha", terminal.input, "input re-seeded from resume onInvocationStart"); - assertEquals(0, plugin.retainedStateCount(), "terminal end also clears state"); + assertFalse(Executions.outstanding(resumed), "the terminal end leaves the scheduler owing nothing"); } } diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/TransformContractTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/TransformContractTest.java index ee8219bc7..b4c426964 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/TransformContractTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/TransformContractTest.java @@ -17,7 +17,6 @@ import software.amazon.awssdk.services.lambda.model.OperationStatus; import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.model.ExecutionStatus; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; @@ -81,10 +80,15 @@ private Map ops() { private WorkflowInsightRecord runOnce(Object input, Function inputTransform) { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .content(ContentConfig.builder().inputTransform(inputTransform).build()) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .content(ContentConfig.builder() + .inputTransform(inputTransform) + .build()) + .addExporter(exporter) + .build()), + ARN, + START); plugin.onInvocationStart(new InvocationInfo("req", ARN, true, START, input, ops(), Map.of())); plugin.onInvocationEnd( new InvocationEndInfo("req", ARN, true, START, ops(), InvocationStatus.SUCCEEDED, null, input, "out")); @@ -126,11 +130,15 @@ void eachTransformInvocationReceivesAFreshDetachedCopy() { m.put("injected-" + m.size(), Boolean.TRUE); // mutate the argument in place return m; }; - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .content(ContentConfig.builder().inputTransform(mutating).build()) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .content( + ContentConfig.builder().inputTransform(mutating).build()) + .addExporter(exporter) + .build()), + ARN, + START); Map input = new LinkedHashMap<>(); input.put("a", 1); @@ -150,7 +158,7 @@ void eachTransformInvocationReceivesAFreshDetachedCopy() { @Test void throwingTransformOmitsInputWithoutFailingExecution() { var exporter = new CapturingExporter(); - var plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + var factory = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() .content(ContentConfig.builder() .inputTransform(v -> { throw new AssertionError("redactor blew up"); @@ -161,7 +169,7 @@ void throwingTransformOmitsInputWithoutFailingExecution() { var runner = LocalDurableTestRunner.create( String.class, (input, context) -> context.step("greet", String.class, sc -> "hi"), - DurableConfig.builder().withPlugins(plugin).build()); + DurableConfig.builder().withPlugins(factory).build()); var result = runner.runUntilComplete("World"); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/UnrecoverableErrorUnwrapTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/UnrecoverableErrorUnwrapTest.java index ccd32782f..972862170 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/UnrecoverableErrorUnwrapTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/UnrecoverableErrorUnwrapTest.java @@ -14,7 +14,6 @@ import software.amazon.awssdk.services.lambda.model.ErrorObject; import software.amazon.awssdk.services.lambda.model.OperationStatus; import software.amazon.lambda.durable.exception.UnrecoverableDurableExecutionException; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; @@ -71,8 +70,11 @@ private static Map ops(OperationStatus status) @Test void failedExecutionUnwrapsUnrecoverableErrorObject() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( - WorkflowInsightConfig.builder().addExporter(exporter).build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()), + ARN, + START); Throwable execError = unrecoverable("PoisonPayload", "cannot deserialize checkpoint"); plugin.onInvocationEnd(new InvocationEndInfo( @@ -88,10 +90,13 @@ void failedExecutionUnwrapsUnrecoverableErrorObject() { @Test void retryingExecutionUnwrapsUnrecoverableErrorObjectInOnChangeMode() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .addExporter(exporter) - .build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .addExporter(exporter) + .build()), + ARN, + START); Throwable execError = unrecoverable("TransientBackendError", "retry scheduled"); // RETRYING maps to a non-terminal RUNNING status but still emits in ON_CHANGE mode. @@ -117,8 +122,11 @@ void retryingExecutionUnwrapsUnrecoverableErrorObjectInOnChangeMode() { @Test void fallsBackToThrowableFieldsWhenUnrecoverableErrorTypeMissing() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( - WorkflowInsightConfig.builder().addExporter(exporter).build()); + var plugin = Executions.plugin( + WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()), + ARN, + START); ErrorObject partial = ErrorObject.builder().errorMessage("only a message").build(); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightFlushCadenceTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightFlushCadenceTest.java index 72bdcc0c4..43f5b7e57 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightFlushCadenceTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightFlushCadenceTest.java @@ -3,9 +3,11 @@ package software.amazon.lambda.durable.insight; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; import java.time.Instant; +import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -16,6 +18,7 @@ import java.util.concurrent.atomic.AtomicInteger; import org.junit.jupiter.api.Test; import software.amazon.awssdk.services.lambda.model.OperationStatus; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; @@ -53,7 +56,8 @@ public void flush() { } } - private static WorkflowInsight.InsightPlugin plugin( + /** The environment: one factory, one scheduler, one set of exporters, however many invocations follow. */ + private static DurableExecutionPluginFactory environment( WorkflowInsightConfig.EmitMode mode, Double samplingRate, CountingExporter... exporters) { var builder = WorkflowInsightConfig.builder().emitMode(mode); for (CountingExporter exporter : exporters) { @@ -62,16 +66,16 @@ private static WorkflowInsight.InsightPlugin plugin( if (samplingRate != null) { builder = builder.samplingRate(samplingRate); } - return (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(builder.build()); + return WorkflowInsight.workflowInsight(builder.build()); } @Test void anInvocationEndThatEmitsARecordFlushesEveryExporterExactlyOnce() { var first = new CountingExporter(); var second = new CountingExporter(); - var plugin = plugin(WorkflowInsightConfig.EmitMode.ON_COMPLETE, null, first, second); + var environment = environment(WorkflowInsightConfig.EmitMode.ON_COMPLETE, null, first, second); - plugin.onInvocationStart(start(arn(0))); + var plugin = Executions.started(environment, start(arn(0))); plugin.onInvocationEnd(end(arn(0), InvocationStatus.SUCCEEDED)); for (CountingExporter exporter : List.of(first, second)) { @@ -99,8 +103,7 @@ record Case(String name, WorkflowInsightConfig.EmitMode mode, InvocationStatus s for (Case scenario : cases) { var exporter = new CountingExporter(); - var plugin = plugin(scenario.mode(), null, exporter); - plugin.onInvocationStart(start(arn(1))); + var plugin = Executions.started(environment(scenario.mode(), null, exporter), start(arn(1))); plugin.onInvocationEnd(end(arn(1), scenario.status())); assertEquals(List.of(), exporter.exported, scenario.name() + ": no record should be emitted"); @@ -111,11 +114,12 @@ record Case(String name, WorkflowInsightConfig.EmitMode mode, InvocationStatus s @Test void everyInvocationEndOfAWarmEnvironmentFlushesExactlyOnce() { var exporter = new CountingExporter(); - var plugin = plugin(WorkflowInsightConfig.EmitMode.ON_CHANGE, null, exporter); + var environment = environment(WorkflowInsightConfig.EmitMode.ON_CHANGE, null, exporter); int invocations = 5; for (int i = 0; i < invocations; i++) { - plugin.onInvocationStart(start(arn(i))); + // A warm environment: each invocation is served by its own instance from the same factory. + var plugin = Executions.started(environment, start(arn(i))); plugin.onOperationChange(change(arn(i))); plugin.onInvocationEnd(end(arn(i), InvocationStatus.SUCCEEDED)); // Sequential ends have nothing to share a flush with, so the cadence bound is tight here. @@ -128,16 +132,16 @@ void everyInvocationEndOfAWarmEnvironmentFlushesExactlyOnce() { @Test void invocationEndsThatOverlapMayShareAFlushButNoneIsLeftUnflushed() { var exporter = new CountingExporter(); - var plugin = plugin(WorkflowInsightConfig.EmitMode.ON_COMPLETE, null, exporter); + var environment = environment(WorkflowInsightConfig.EmitMode.ON_COMPLETE, null, exporter); int executions = 8; var barrier = new CyclicBarrier(executions); var done = new CountDownLatch(executions); for (int i = 0; i < executions; i++) { String executionArn = arn(100 + i); + var plugin = Executions.started(environment, start(executionArn)); var thread = new Thread( () -> { - plugin.onInvocationStart(start(executionArn)); try { barrier.await(60, TimeUnit.SECONDS); } catch (Exception e) { @@ -175,17 +179,21 @@ void aSampledOutExecutionFlushesNothing() { // Unchanged by the move of flush onto the export pump: a sampled-out end never schedules a record, so it // neither drains nor flushes. var exporter = new CountingExporter(); - var plugin = plugin(WorkflowInsightConfig.EmitMode.ON_CHANGE, 0.0, exporter); + var environment = environment(WorkflowInsightConfig.EmitMode.ON_CHANGE, 0.0, exporter); + var plugins = new ArrayList(); for (int i = 0; i < 10; i++) { - plugin.onInvocationStart(start(arn(i))); + var plugin = Executions.started(environment, start(arn(i))); + plugins.add(plugin); plugin.onOperationChange(change(arn(i))); plugin.onInvocationEnd(end(arn(i), InvocationStatus.SUCCEEDED)); } assertEquals(List.of(), exporter.exported); assertEquals(0, exporter.flushes.get(), "a sampled-out invocation end neither drains nor flushes"); - assertEquals(0, plugin.retainedStateCount()); + for (InsightPlugin plugin : plugins) { + assertFalse(Executions.outstanding(plugin), "a sampled-out invocation leaves the scheduler owing nothing"); + } } private static Map ops() { diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightHookTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightHookTest.java index 390cab14b..ebaf6c606 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightHookTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightHookTest.java @@ -15,7 +15,7 @@ import java.util.concurrent.TimeUnit; import org.junit.jupiter.api.Test; import software.amazon.awssdk.services.lambda.model.OperationStatus; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.InvocationStatus; @@ -68,17 +68,21 @@ private InvocationEndInfo end(InvocationStatus status, Object result, Throwable "req", ARN, true, START, ops("greet", OperationStatus.SUCCEEDED), status, error, "in", result); } - @Test - void onChangeEmitsAtStartChangeAndEnd() { - var exporter = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + /** The environment one or more invocations are then served in: one factory, one scheduler, one exporter set. */ + private static DurableExecutionPluginFactory onChangeEnvironment(InsightExporter exporter) { + return WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) .addExporter(exporter) .build()); + } + + @Test + void onChangeEmitsAtStartChangeAndEnd() { + var exporter = new CapturingExporter(); + var plugin = Executions.started(onChangeEnvironment(exporter), start(true)); // Let each scheduled export land before the next hook so all three snapshots are observable; back-to-back // hooks may otherwise coalesce into the latest record (covered separately below). - plugin.onInvocationStart(start(true)); plugin.drainExports(); plugin.onOperationChange(new OperationChangeInfo( "req", ARN, ops("greet", OperationStatus.SUCCEEDED), ops("greet", OperationStatus.SUCCEEDED))); @@ -95,12 +99,8 @@ void onChangeEmitsAtStartChangeAndEnd() { @Test void onChangeExportsOffTheHookThreadAndCoalescesBurstsIntoTheLatestRecord() { var exporter = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .addExporter(exporter) - .build()); + var plugin = Executions.started(onChangeEnvironment(exporter), start(true)); - plugin.onInvocationStart(start(true)); for (int i = 0; i < 20; i++) { plugin.onOperationChange(new OperationChangeInfo( "req", ARN, ops("greet", OperationStatus.SUCCEEDED), ops("greet", OperationStatus.SUCCEEDED))); @@ -137,12 +137,7 @@ public void export(WorkflowInsightRecord record) { } } }; - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .addExporter(exporter) - .build()); - - plugin.onInvocationStart(start(true)); + var plugin = Executions.started(onChangeEnvironment(exporter), start(true)); plugin.drainExports(); // The end hook blocks in its drain while the final record is being exported; the change hook arrives then, @@ -165,10 +160,10 @@ public void export(WorkflowInsightRecord record) { @Test void invocationEndFlushesExportersEvenWhenNothingWasEmitted() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( - WorkflowInsightConfig.builder().addExporter(exporter).build()); - - plugin.onInvocationStart(start(true)); + var plugin = Executions.started( + WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()), + start(true)); plugin.onInvocationEnd(end(InvocationStatus.PENDING, null, null)); assertTrue(exporter.records.isEmpty(), "on-complete emits nothing for a suspend"); @@ -178,10 +173,10 @@ void invocationEndFlushesExportersEvenWhenNothingWasEmitted() { @Test void onCompleteSkipsNonTerminalAndEmitsTerminalOnly() { var exporter = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight( - WorkflowInsightConfig.builder().addExporter(exporter).build()); - - plugin.onInvocationStart(start(true)); + var plugin = Executions.started( + WorkflowInsight.workflowInsight( + WorkflowInsightConfig.builder().addExporter(exporter).build()), + start(true)); plugin.onOperationChange(new OperationChangeInfo( "req", ARN, ops("greet", OperationStatus.SUCCEEDED), ops("greet", OperationStatus.SUCCEEDED))); assertTrue(exporter.records.isEmpty(), "no record before terminal in on-complete mode"); @@ -193,17 +188,16 @@ void onCompleteSkipsNonTerminalAndEmitsTerminalOnly() { @Test void suspendResumeKeepsStableStartTimeAndLeavesNoRetainedState() { var exporter = new CapturingExporter(); - var plugin = (WorkflowInsight.InsightPlugin) WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .addExporter(exporter) - .build()); + var environment = onChangeEnvironment(exporter); - plugin.onInvocationStart(start(true)); // first invocation - plugin.drainExports(); - plugin.onInvocationEnd(end(InvocationStatus.PENDING, null, null)); // suspend -> state removed - plugin.onInvocationStart(start(false)); // resume invocation re-seeds state - plugin.drainExports(); - plugin.onInvocationEnd(end(InvocationStatus.SUCCEEDED, "out", null)); // resume + terminal + // The suspend and the resume are two invocations of the same execution in one warm environment, so the SDK + // serves them with two instances: nothing is carried over in the plugin, and nothing has to be cleaned up. + var first = Executions.started(environment, start(true)); + first.drainExports(); + first.onInvocationEnd(end(InvocationStatus.PENDING, null, null)); // suspend + var resumed = Executions.started(environment, start(false)); // resume re-seeds from its own InvocationInfo + resumed.drainExports(); + resumed.onInvocationEnd(end(InvocationStatus.SUCCEEDED, "out", null)); // resume + terminal // start(RUNNING) + pending(RUNNING) + resume-start(RUNNING) + terminal(SUCCEEDED); all share the stable // startTime recreated from InvocationInfo.executionStartTime() across the suspend boundary. @@ -211,7 +205,8 @@ void suspendResumeKeepsStableStartTimeAndLeavesNoRetainedState() { String startTime = exporter.records.get(0).startTime(); assertTrue(exporter.records.stream().allMatch(r -> startTime.equals(r.startTime()))); assertEquals(START.toString(), startTime); - assertEquals(0, plugin.retainedStateCount(), "no per-execution state retained after invocation end"); + assertFalse(Executions.outstanding(first), "the suspended invocation left the scheduler owing nothing"); + assertFalse(Executions.outstanding(resumed), "nor did the resumed one"); } @Test @@ -223,12 +218,12 @@ public void export(WorkflowInsightRecord record) { } }; var good = new CapturingExporter(); - DurableExecutionPlugin plugin = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .addExporter(throwing) - .addExporter(good) - .build()); - - plugin.onInvocationStart(start(true)); + var plugin = Executions.started( + WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .addExporter(throwing) + .addExporter(good) + .build()), + start(true)); plugin.onInvocationEnd(end(InvocationStatus.SUCCEEDED, "out", null)); assertEquals(1, good.records.size(), "failing exporter never blocks the others"); diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightPluginTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightPluginTest.java index 8d5cf0fff..98531b6e6 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightPluginTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/WorkflowInsightPluginTest.java @@ -44,8 +44,8 @@ public void export(WorkflowInsightRecord record) { 2, Duration.ofSeconds(1), Duration.ofSeconds(1), 2.0, JitterStrategy.NONE); private DurableConfig configWith(CapturingExporter exporter, WorkflowInsightConfig.Builder cfg) { - var plugin = WorkflowInsight.workflowInsight(cfg.addExporter(exporter).build()); - return DurableConfig.builder().withPlugins(plugin).build(); + var factory = WorkflowInsight.workflowInsight(cfg.addExporter(exporter).build()); + return DurableConfig.builder().withPlugins(factory).build(); } private OperationRecord op(WorkflowInsightRecord rec, String name) { diff --git a/otel-plugin/README.md b/otel-plugin/README.md index c51032aaa..7aa9c2467 100644 --- a/otel-plugin/README.md +++ b/otel-plugin/README.md @@ -10,7 +10,7 @@ OpenTelemetry instrumentation plugin for the AWS Lambda Durable Execution SDK fo - **Span-per-Operation**: Each durable operation (step, wait, map, etc.) gets its own span with accurate timing - **Attempt Spans**: Each user function execution (step attempt, child context run) gets a span, including retries - **Log Correlation**: Injects `traceId`, `spanId`, and `otelTraceSampled` into SLF4J MDC for end-to-end observability -- **ADOT Java Agent Integration**: `new InvocationOtelPlugin()` late-binds the ADOT Java agent's global provider with no handler-side OpenTelemetry initialization +- **ADOT Java Agent Integration**: `InvocationOtelPlugin.factory()` binds the ADOT Java agent's global provider on first use, with no handler-side OpenTelemetry initialization - **Lambda Layer Discovery**: `DURABLE_EXECUTION_PLUGINS` loads either OTel plugin from a JAR under a layer's `java/lib` directory ## Installation @@ -23,7 +23,7 @@ OpenTelemetry instrumentation plugin for the AWS Lambda Durable Execution SDK fo ``` -For the no-arg constructor (`new InvocationOtelPlugin()`), no additional OpenTelemetry dependencies are needed — the ADOT Java agent layer provides them. +For the agent path (`InvocationOtelPlugin.factory()`), no additional OpenTelemetry dependencies are needed — the ADOT Java agent layer provides them. If you configure your own `SdkTracerProviderBuilder`, add the OpenTelemetry SDK and an exporter: @@ -50,7 +50,7 @@ If you configure your own `SdkTracerProviderBuilder`, add the OpenTelemetry SDK ### 1. ADOT Lambda Layer -This plugin uses the [AWS Distro for OpenTelemetry (ADOT) Lambda layer](https://aws-otel.github.io/docs/getting-started/lambda) for trace export. The `new InvocationOtelPlugin()` constructor resolves the global provider initialized by the ADOT Java agent at invocation start, with deterministic span ID generation installed through the plugin's `AutoConfigurationCustomizerProvider` SPI. If the provider is not ready, the plugin emits no telemetry for that invocation and retries provider resolution on the next invocation. +This plugin uses the [AWS Distro for OpenTelemetry (ADOT) Lambda layer](https://aws-otel.github.io/docs/getting-started/lambda) for trace export. `InvocationOtelPlugin.factory()` resolves the global provider initialized by the ADOT Java agent when the first invocation's plugin instance is created, with deterministic span ID generation installed through the plugin's `AutoConfigurationCustomizerProvider` SPI. If the provider is not ready, that invocation's instance emits no telemetry and the next invocation's instance resolves the provider again. The layer ARN follows the format: @@ -130,7 +130,8 @@ public class MyHandler extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new InvocationOtelPlugin()).build(); + // A factory, not a plugin instance: the SDK creates one plugin instance per invocation from it. + return DurableConfig.builder().withPlugins(InvocationOtelPlugin.factory()).build(); } @Override @@ -193,7 +194,7 @@ The plugin decides sampling once per invocation and applies that single decision 1. **Backend decision** — `Sampled=1` / `Sampled=0` in the propagated header is authoritative and always preserved, regardless of the configured sampler. 2. **Same-trace ambient span** — when the header carries no usable `Sampled` value but a valid ambient span (for example an auto-instrumentation Lambda handler span) is already on the execution's trace, the plugin follows that span's decision: sampled → sampled; unsampled but still recording → `RECORD_ONLY`; unsampled and not recording → dropped. -3. **Configured sampler (application-owned provider)** — when you pass a `SdkTracerProvider` to the plugin, its sampler is read directly and evaluated once with the trace ID, span name, and attributes. A trace-ID-ratio sampler therefore produces a stable decision across reinvocations (the trace ID is stable). +3. **Configured sampler (application-owned provider)** — when you pass a `SdkTracerProviderBuilder` to `factory(...)`, the sampler of the provider it builds is read directly and evaluated once with the trace ID, span name, and attributes. A trace-ID-ratio sampler therefore produces a stable decision across reinvocations (the trace ID is stable). 4. **Installed sampler (Java-agent path)** — when the agent owns the provider, it is behind a classloader boundary and its *effective* sampler (which another agent extension may have wrapped or replaced) cannot be reliably read at decision time. Rather than guess, the plugin **defers**: it installs a delegating sampler through the agent's autoconfiguration and lets that wrapper consult the agent's real sampler. The delegate's decision is honored in full — if your configured policy is `always_off`, a rate limiter, or a remote sampler (`xray`, `jaeger_remote`) that returns drop, the durable spans are dropped; they are **not** force-sampled. To avoid consuming a stateful or quota-based sampler once per span, the wrapper consults the delegate once per execution (keyed by trace ID) and reuses that decision for the execution's remaining durable spans within the invocation. For precise, provider-independent control, set an explicit `Sampled` value upstream (for example by enabling X-Ray active tracing) — that backend decision takes precedence over everything else. @@ -255,21 +256,27 @@ With Lambda's `LoggingConfig: JSON` (required for durable functions), CloudWatch ## Configuration -Both plugins take a required `SdkTracerProviderBuilder` (your exporter/processor pipeline) plus an optional -`OtelPluginConfig` built with a named-field builder. This replaces the older telescoping constructors, giving readable, -type-safe call sites, and matches the `OtelPluginConfig` object in the JavaScript and Python SDKs. +Each plugin is registered as a `DurableExecutionPluginFactory` obtained from its static `factory(...)` methods, because +a plugin instance serves exactly one invocation: the SDK calls the factory once per invocation and drops the instance +when the invocation returns. The factory holds what belongs to the execution environment — your tracer provider (built +once) or the ADOT global provider binding, plus the deterministic ID generator — while each instance holds only its own +invocation's spans. + +The `factory(...)` overloads take an optional `SdkTracerProviderBuilder` (your exporter/processor pipeline) plus an +optional `OtelPluginConfig` built with a named-field builder, which matches the `OtelPluginConfig` object in the +JavaScript and Python SDKs. ### InvocationOtelPlugin ```java // Default: ADOT Java agent global provider, X-Ray context extraction, MDC enabled -new InvocationOtelPlugin(); +InvocationOtelPlugin.factory(); // Custom tracer provider pipeline, all other options defaulted -new InvocationOtelPlugin(tracerProviderBuilder); +InvocationOtelPlugin.factory(tracerProviderBuilder); // Full configuration via the builder -new InvocationOtelPlugin( +InvocationOtelPlugin.factory( tracerProviderBuilder, OtelPluginConfig.builder() .contextExtractor(new XRayContextExtractor()) @@ -282,18 +289,18 @@ new InvocationOtelPlugin( ### ExecutionOtelPlugin The `ExecutionOtelPlugin` renders the Workflow span as the durable trace root with operations beneath it. Invocation -spans remain in the ambient Lambda trace, and operations link to the Invocation that ran them. It takes the same -`(SdkTracerProviderBuilder, OtelPluginConfig)` constructor: +spans remain in the ambient Lambda trace, and operations link to the Invocation that ran them. It exposes the same +`factory(SdkTracerProviderBuilder, OtelPluginConfig)` methods: ```java // Default: ADOT Java agent global provider, X-Ray context extraction, MDC enabled -new ExecutionOtelPlugin(); +ExecutionOtelPlugin.factory(); // Custom tracer provider pipeline, all other options defaulted -new ExecutionOtelPlugin(tracerProviderBuilder); +ExecutionOtelPlugin.factory(tracerProviderBuilder); // Full configuration via the builder -new ExecutionOtelPlugin( +ExecutionOtelPlugin.factory( tracerProviderBuilder, OtelPluginConfig.builder() .enableMdc(false) @@ -310,9 +317,9 @@ new ExecutionOtelPlugin( | `workflowSpanName(...)` | Name for the Workflow span | `"Workflow"` | | `instrumentationName(...)` | Instrumentation scope name registered with the tracer | `"aws-durable-execution-sdk-java"` | -> The `tracerProviderBuilder` argument is not used by the no-arg `new InvocationOtelPlugin()` / -> `new ExecutionOtelPlugin()` constructors; those resolve the ADOT Java agent's global provider at invocation start. -> If it is not ready, all telemetry is disabled for that invocation and resolution is retried on the next invocation. +> The no-builder `InvocationOtelPlugin.factory()` / `ExecutionOtelPlugin.factory()` forms resolve the ADOT Java agent's +> global provider instead, when the first invocation's instance needs it. If it is not ready, all telemetry is disabled +> for that invocation and the next invocation's instance resolves it again. > A `null` passed to any `OtelPluginConfig` builder setter falls back to that option's default. ## Known Limitations @@ -357,7 +364,7 @@ For local testing, use a logging exporter to print spans to stdout: ```java import io.opentelemetry.exporter.logging.LoggingSpanExporter; -var otelPlugin = new InvocationOtelPlugin( +var otelPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder() .addSpanProcessor(SimpleSpanProcessor.create(LoggingSpanExporter.create()))); ``` @@ -367,7 +374,7 @@ var otelPlugin = new InvocationOtelPlugin( - Java 17+ - AWS Durable Execution SDK for Java 2.0.0+ - OpenTelemetry SDK 1.65.0+ (only for custom TracerProvider path) -- ADOT Lambda Layer `AWSOpenTelemetryDistroJava` (for the no-arg constructor path) +- ADOT Lambda Layer `AWSOpenTelemetryDistroJava` (for the agent path, `factory()` without a builder) ## License diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPlugin.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPlugin.java index 09214055c..010c48eca 100644 --- a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPlugin.java +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPlugin.java @@ -25,6 +25,7 @@ import org.slf4j.LoggerFactory; import org.slf4j.MDC; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.OperationEndInfo; @@ -62,10 +63,20 @@ * *

The Workflow and Invocation spans share one execution trace, anchored at the execution ancestor resolved at * invocation start: a valid propagated remote server span becomes that ancestor directly, otherwise a synthetic - * execution root anchors the trace. The trace ID is stable across invocations of the same execution. When using - * {@link #ExecutionOtelPlugin()}, the plugin resolves the global provider at invocation start. If the OpenTelemetry - * Java agent is not initialized yet, telemetry is disabled for that entire invocation and provider resolution is - * retried on the next invocation. + * execution root anchors the trace. The trace ID is stable across invocations of the same execution, because it is + * derived from the execution ARN and start time rather than carried in the plugin. + * + *

Lifetime. One instance serves exactly one Lambda invocation: {@link #factory()} and its overloads return a + * {@link DurableExecutionPluginFactory} that the SDK calls once per invocation, and the instance is dropped when the + * invocation returns. Everything about the invocation — the execution ARN, the resolved execution trace and ancestor, + * the sampling intent, the Invocation span, the deferred Workflow span context — is therefore a {@code final} field, + * resolved in the constructor from the {@link InvocationInfo} the factory receives. Nothing is reset between + * invocations because nothing is carried between them. + * + *

What belongs to the execution environment stays in the factory's {@link OtelPluginEnvironment}: the configuration, + * the ID generator, and either the application-owned tracer provider (built once) or the lazily resolved ADOT global + * provider. An invocation whose instance cannot resolve the global provider emits no telemetry at all, and the next + * invocation's instance resolves it again. * *

Status mapping (parity with the Python/JS references): * @@ -85,43 +96,61 @@ * current, so {@code Span.current()} enrichment is not recorded on the final operation span. The placeholder uses the * Invocation span's resolved sampling metadata when available. * - *

Thread-safe: uses {@link ConcurrentHashMap} for span/scope storage since the SDK runs user code on multiple - * threads. + *

Thread-safe within its invocation: the SDK runs user code on multiple threads, so the open-span registries are + * {@link ConcurrentHashMap}s. The invocation's identity needs no such protection — it is final state written before the + * SDK publishes the instance to those threads. */ -public class ExecutionOtelPlugin implements DurableExecutionPlugin { +public final class ExecutionOtelPlugin implements DurableExecutionPlugin { private static final Logger logger = LoggerFactory.getLogger(ExecutionOtelPlugin.class); - private volatile SdkTracerProvider sdkTracerProvider; - private volatile Tracer tracer; + // ─── Environment lifetime (shared with every other invocation's instance) ───────────── + private final DeterministicIdGenerator idGenerator; - private final ContextExtractor contextExtractor; private final boolean enableMdc; private final String workflowSpanName; - private final String instrumentationName; - - // Per-invocation state - private volatile boolean tracingEnabled; - private volatile Span invocationSpan; - private volatile String durableExecutionArn; - - // Trace ID and flags of the execution trace, published together as one snapshot so readers never pair a trace ID - // with mismatched flags. - private volatile ExecutionTrace executionTrace; - // The execution's single sampling intent for this invocation, computed once at onInvocationStart and attached to - // every durable span's parent context so DurableSampler applies it (a resolved decision verbatim, or a deferral to - // its own delegate) without re-invoking the configured sampler per span. - private volatile DurableSamplingDecision.Intent samplingIntent; - - /** Immutable snapshot of the resolved execution trace, read atomically through a single volatile reference. */ - private record ExecutionTrace(String traceId, TraceFlags flags) {} - // Between invocations the Workflow span exists only as a deterministic context that operations parent onto; the - // recording span is started and ended in a single call on the terminal invocation, so it is never left open. The - // execution ancestor and start time are retained so that span can be built at invocation end. - private volatile SpanContext workflowSpanContext; - private volatile SpanContext executionAncestor; - private volatile Instant executionStartTime; + // ─── This invocation, all resolved in the constructor from its InvocationInfo ───────── + + /** The provider used to flush before Lambda freezes; null when it is not visible to the application. */ + private final SdkTracerProvider sdkTracerProvider; + + /** Null when telemetry is disabled for this invocation, which makes every hook on this instance a no-op. */ + private final Tracer tracer; + + private final String durableExecutionArn; + private final Instant executionStartTime; + + /** Trace ID and flags of the execution trace, resolved together so they can never be paired mismatched. */ + private final ExecutionTrace executionTrace; + + /** + * The execution's single sampling intent for this invocation, resolved once and attached to every durable span's + * parent context so DurableSampler applies it (a resolved decision verbatim, or a deferral to its own delegate) + * without re-invoking the configured sampler per span. + */ + private final DurableSamplingDecision.Intent samplingIntent; + + private final Span invocationSpan; + + /** + * The Workflow span exists as a deterministic context that operations parent onto; the recording span is started + * and ended in a single call on the terminal invocation, so it is never left open. The execution ancestor and start + * time are held so that span can be built at invocation end. + */ + private final SpanContext workflowSpanContext; + + private final SpanContext executionAncestor; + + /** + * Set when this invocation ends; never cleared, because an instance is never reused. Read by the operation and user + * function hooks, which may run on other threads of this invocation, so that a straggler hook arriving after the + * spans have been ended does not open a new one — volatile for that publication. + */ + private volatile boolean ended; + + /** Immutable snapshot of the resolved execution trace. */ + private record ExecutionTrace(String traceId, TraceFlags flags) {} // Thread-safe storage for attempt spans/scopes (keyed by operationId + "-" + attempt) private final ConcurrentHashMap attemptSpans = new ConcurrentHashMap<>(); @@ -136,89 +165,108 @@ private record ExecutionTrace(String traceId, TraceFlags flags) {} private final ConcurrentHashMap operationStartTimes = new ConcurrentHashMap<>(); /** - * Creates a Workflow-rooted OTel plugin with default settings: X-Ray context extraction, MDC enabled, root span - * named {@code "Workflow"}. + * Returns a factory that creates one plugin instance per invocation against the ADOT Java agent's global provider, + * with default settings: X-Ray context extraction, MDC enabled, root span named {@code "Workflow"}. * - *

Uses the provided tracer provider builder. For ADOT Java agent usage, prefer {@link #ExecutionOtelPlugin()} - * with the plugin jar configured through {@code OTEL_JAVAAGENT_EXTENSIONS}. + * 

{@code
+     * DurableConfig.builder().withPlugins(ExecutionOtelPlugin.factory()).build();
+     * }
* - * @param tracerProviderBuilder the tracer provider builder (its ID generator will be wrapped) + * @return the per-invocation plugin factory to hand to {@code DurableConfig.Builder.withPlugins} */ - public ExecutionOtelPlugin(SdkTracerProviderBuilder tracerProviderBuilder) { - this(tracerProviderBuilder, OtelPluginConfig.defaults()); + public static DurableExecutionPluginFactory factory() { + return factory(OtelPluginConfig.defaults()); } /** - * Creates a Workflow-rooted OTel plugin with default settings: X-Ray context extraction and MDC enabled. + * Returns a factory that creates one plugin instance per invocation against the ADOT Java agent's global provider. + * + *

The global provider is resolved when the first invocation's instance needs it. If the agent has not + * initialized it yet, that invocation emits no telemetry and the next invocation's instance resolves it again. + * + * @param config the plugin configuration + * @return the per-invocation plugin factory to hand to {@code DurableConfig.Builder.withPlugins} + */ + public static DurableExecutionPluginFactory factory(OtelPluginConfig config) { + var environment = OtelPluginEnvironment.forGlobalProvider(config); + return info -> new ExecutionOtelPlugin(environment, info); + } + + /** + * Returns a factory that creates one plugin instance per invocation against an application-owned tracer provider, + * with default settings: X-Ray context extraction, MDC enabled, root span named {@code "Workflow"}. + * + *

Customers configure exporters and span processors on the builder — the plugin handles ID generation. The + * provider is built once, here, and shared by every invocation's instance. * - *

Resolves {@code GlobalOpenTelemetry} at invocation start. If the ADOT Java agent has not initialized it yet, - * telemetry is disabled for that invocation and resolution is retried on the next invocation. + * @param tracerProviderBuilder the tracer provider builder (its ID generator and sampler will be wrapped) + * @return the per-invocation plugin factory to hand to {@code DurableConfig.Builder.withPlugins} */ - public ExecutionOtelPlugin() { - this(OtelPluginConfig.defaults()); + public static DurableExecutionPluginFactory factory(SdkTracerProviderBuilder tracerProviderBuilder) { + return factory(tracerProviderBuilder, OtelPluginConfig.defaults()); } /** - * Creates a Workflow-rooted OTel plugin from the given tracer provider builder and configuration. + * Returns a factory that creates one plugin instance per invocation against an application-owned tracer provider. * *

Customers configure exporters and span processors on the builder; all other tunables (context extractor, MDC - * toggle, Workflow span name, instrumentation scope name) come from {@link OtelPluginConfig}. Use - * {@link OtelPluginConfig#builder()} for readable, named configuration: + * toggle, Workflow span name, instrumentation scope name) come from {@link OtelPluginConfig}: * * 

{@code
-     * var plugin = new ExecutionOtelPlugin(
+     * var factory = ExecutionOtelPlugin.factory(
      *     SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)),
      *     OtelPluginConfig.builder().enableMdc(false).workflowSpanName("Workflow").build());
      * }
* - * @param tracerProviderBuilder the tracer provider builder (its ID generator will be wrapped) + * @param tracerProviderBuilder the tracer provider builder (its ID generator and sampler will be wrapped) * @param config the plugin configuration + * @return the per-invocation plugin factory to hand to {@code DurableConfig.Builder.withPlugins} */ - public ExecutionOtelPlugin(SdkTracerProviderBuilder tracerProviderBuilder, OtelPluginConfig config) { - this.idGenerator = DeterministicIdGenerator.installOn(tracerProviderBuilder); - // Wrap the configured sampler so durable spans use the execution's single precomputed decision. - DurableSampler.installOn(tracerProviderBuilder); - - this.sdkTracerProvider = tracerProviderBuilder.build(); - this.tracer = sdkTracerProvider.get(config.instrumentationName()); - this.contextExtractor = config.contextExtractor(); - this.enableMdc = config.enableMdc(); - this.workflowSpanName = config.workflowSpanName(); - this.instrumentationName = config.instrumentationName(); + public static DurableExecutionPluginFactory factory( + SdkTracerProviderBuilder tracerProviderBuilder, OtelPluginConfig config) { + var environment = OtelPluginEnvironment.forProviderBuilder(tracerProviderBuilder, config); + return info -> new ExecutionOtelPlugin(environment, info); } /** - * Creates a Workflow-rooted OTel plugin from configuration alone (no caller-supplied tracer provider builder). + * Creates the instance that serves one invocation. * - *

The config-only constructor uses the ADOT/global provider. Supply a {@code SdkTracerProviderBuilder} via the - * two-arg constructor for an application-owned provider. + *

Everything this invocation's spans are keyed by is resolved here, from the {@code info} the factory received: + * the tracer binding, the extracted context, the canonical execution trace and its ancestor, the single sampling + * intent, the Invocation span, and the deferred Workflow span context. Resolving them in the constructor — before + * the SDK publishes this instance to the operation and user function threads — is what lets them be {@code final} + * rather than volatile per-invocation state. * - * @param config the plugin configuration + *

When the tracer cannot be bound, telemetry is disabled for this invocation: the span fields stay null and + * every hook returns immediately. The next invocation gets a new instance, which binds again. */ - public ExecutionOtelPlugin(OtelPluginConfig config) { - this.contextExtractor = config.contextExtractor(); + private ExecutionOtelPlugin(OtelPluginEnvironment environment, InvocationInfo info) { + var config = environment.config(); + this.idGenerator = environment.idGenerator(); this.enableMdc = config.enableMdc(); this.workflowSpanName = config.workflowSpanName(); - this.instrumentationName = config.instrumentationName(); - this.idGenerator = OtelPluginSupport.createDefaultIdGenerator(); - } - - // ─── Invocation hooks ──────────────────────────────────────────────── - - @Override - public void onInvocationStart(InvocationInfo info) { - tracingEnabled = false; - if (!bindTracer()) { + this.durableExecutionArn = info.durableExecutionArn(); + this.executionStartTime = info.executionStartTime(); + + var setup = environment.bind("ExecutionOtelPlugin"); + if (setup == null) { + this.sdkTracerProvider = null; + this.tracer = null; + this.samplingIntent = null; + this.executionTrace = null; + this.executionAncestor = null; + this.invocationSpan = null; + this.workflowSpanContext = null; return; } - - this.durableExecutionArn = info.durableExecutionArn(); + this.sdkTracerProvider = setup.sdkTracerProvider(); + this.tracer = setup.tracer(); // Resolve the one execution ancestor both spans parent onto, so they share a stable-per-execution trace and a // sampling decision. - var extracted = contextExtractor.extract(); + var extracted = config.contextExtractor().extract(); var canonicalTraceId = - ExecutionTraceContext.canonicalTraceId(extracted, arn(), info.executionStartTime(), idGenerator); + ExecutionTraceContext.canonicalTraceId(extracted, arn(), executionStartTime, idGenerator); // Resolve the execution's sampling decision once for this invocation as a full SamplingResult, then apply it to // every durable span via DurableSampler. The execution ancestor's trace flags are derived from the same // decision so a parent-based sampler stays consistent with it. @@ -231,14 +279,13 @@ public void onInvocationStart(InvocationInfo info) { Attributes.of(DURABLE_EXECUTION_ARN, arn())); // A null decision is unresolved on the agent path: defer to DurableSampler's own delegate (keyed by trace ID), // rather than fabricating a decision that would bypass an installed drop/rate-limit policy. - samplingIntent = decision != null + this.samplingIntent = decision != null ? DurableSamplingDecision.Intent.resolved(decision) : DurableSamplingDecision.Intent.deferred(canonicalTraceId); var sampled = OtelPluginSupport.isSampled(decision); var execCtx = ExecutionTraceContext.resolve(extracted, canonicalTraceId, arn(), idGenerator, () -> sampled); - executionTrace = new ExecutionTrace(canonicalTraceId, execCtx.traceFlags()); - executionAncestor = execCtx.executionAncestor(); - executionStartTime = info.executionStartTime(); + this.executionTrace = new ExecutionTrace(canonicalTraceId, execCtx.traceFlags()); + this.executionAncestor = execCtx.executionAncestor(); // Invocation span — child of the ambient Lambda span when it is on the execution trace, otherwise a child of // the execution ancestor so it stays within the same trace. @@ -246,69 +293,73 @@ public void onInvocationStart(InvocationInfo info) { var spanBuilder = tracer.spanBuilder("Invocation") .setSpanKind(SpanKind.INTERNAL) .setParent(invocationParent) - .setAttribute(DURABLE_EXECUTION_ARN, info.durableExecutionArn()) + .setAttribute(DURABLE_EXECUTION_ARN, durableExecutionArn) .setAttribute(DURABLE_FIRST_INVOCATION, info.isFirstInvocation()); if (info.requestId() != null) { spanBuilder.setAttribute(AttributeKey.stringKey("faas.invocation_id"), info.requestId()); } - invocationSpan = startDurableSpan(spanBuilder); + this.invocationSpan = startDurableSpan(spanBuilder); // Defer the recording Workflow span until terminal completion. The placeholder uses the Invocation span's // resolved sampling metadata so operation parents/links match the span that is eventually exported. - var workflowSpanId = idGenerator.generateWorkflowSpanId(info.durableExecutionArn()); var invocationContext = invocationSpan.getSpanContext(); - workflowSpanContext = SpanContext.create( - canonicalTraceId, workflowSpanId, invocationContext.getTraceFlags(), invocationContext.getTraceState()); + this.workflowSpanContext = SpanContext.create( + canonicalTraceId, + idGenerator.generateWorkflowSpanId(durableExecutionArn), + invocationContext.getTraceFlags(), + invocationContext.getTraceState()); + } - // Inject MDC on the handler thread so handler-level logs (between steps) have trace context. - if (enableMdc) { - MDC.put( - MdcSpanEnricher.MDC_TRACE_ID, - invocationSpan.getSpanContext().getTraceId()); + // ─── Invocation hooks ──────────────────────────────────────────────── + + @Override + public void onInvocationStart(InvocationInfo info) { + // This invocation's identity, its Invocation span and its Workflow span context were resolved in the + // constructor, from the very InvocationInfo this hook receives. What is left is the MDC injection, which + // belongs + // here because it must run on the handler thread so handler-level logs between steps carry trace context. + if (invocationSpan == null || !enableMdc) { + return; } - tracingEnabled = true; + MDC.put(MdcSpanEnricher.MDC_TRACE_ID, invocationSpan.getSpanContext().getTraceId()); } @Override public void onInvocationEnd(InvocationEndInfo info) { - if (!tracingEnabled) { + if (disabled()) { return; } - tracingEnabled = false; + // Set before the spans are ended, so a straggler hook from another thread of this invocation cannot open a span + // under one that is already closed. Never cleared: this instance serves no second invocation. + ended = true; // Clear invocation-level MDC if (enableMdc) { MdcSpanEnricher.clear(); } - // Drop placeholder state. Open operations have no recording span to abandon. - operationContexts.clear(); - operationStartTimes.clear(); - // Release OTel context on worker threads, then end any attempt spans still open so no recording span is // abandoned. Attempt spans normally start and end within one user-function call, so this is a safeguard. for (var scope : attemptScopes.values()) { scope.close(); } - attemptScopes.clear(); for (var span : attemptSpans.values()) { span.end(); } - attemptSpans.clear(); + // The placeholder and attempt registries are not emptied: an operation that never completed has no recording + // span to abandon, every attempt span above has been ended, and this instance is dropped when the invocation + // returns, so there is nothing to recycle them for. // End the invocation span every invocation. - if (invocationSpan != null) { - invocationSpan.setAttribute( - DURABLE_INVOCATION_STATUS, info.invocationStatus().name()); - applyInvocationStatus(invocationSpan, info); - invocationSpan.end(); - invocationSpan = null; - } + invocationSpan.setAttribute( + DURABLE_INVOCATION_STATUS, info.invocationStatus().name()); + applyInvocationStatus(invocationSpan, info); + invocationSpan.end(); // Materialize the Workflow span only on terminal status. - if (isTerminal(info) && workflowSpanContext != null && executionAncestor != null) { + if (isTerminal(info)) { var workflowSpanBuilder = tracer.spanBuilder(workflowSpanName) .setSpanKind(SpanKind.INTERNAL) .setParent(withDurableDecision(Context.root().with(Span.wrap(executionAncestor)))) @@ -332,10 +383,6 @@ public void onInvocationEnd(InvocationEndInfo info) { } workflowSpan.end(); } - workflowSpanContext = null; - executionAncestor = null; - executionStartTime = null; - samplingIntent = null; // Flush spans before Lambda freezes if (sdkTracerProvider != null) { @@ -350,7 +397,7 @@ public void onInvocationEnd(InvocationEndInfo info) { @Override public void onOperationStart(OperationInfo info) { - if (!tracingEnabled) return; + if (disabled()) return; if (info.id() == null) return; // Retain only a deterministic placeholder. Its flags/state come from the Invocation span's resolved sampling @@ -368,7 +415,7 @@ public void onOperationStart(OperationInfo info) { @Override public void onOperationEnd(OperationEndInfo info) { - if (!tracingEnabled) return; + if (disabled()) return; if (info.id() == null) return; // Start and end the operation's single span here, using its deterministic span ID and linking to the @@ -428,7 +475,7 @@ public void onOperationEnd(OperationEndInfo info) { @Override public void onUserFunctionStart(UserFunctionStartInfo info) { - if (!tracingEnabled) return; + if (disabled()) return; // Skip attempt spans for CONTEXT operations — they are a scoping construct, not a retriable unit of work. Still // make the operation's context current so auto-instrumented calls become children of the (deferred) operation @@ -486,7 +533,7 @@ public void onUserFunctionStart(UserFunctionStartInfo info) { @Override public void onUserFunctionEnd(UserFunctionEndInfo info) { - if (!tracingEnabled) return; + if (disabled()) return; var key = attemptKey(info.id(), info.attempt()); @@ -524,22 +571,12 @@ public void onUserFunctionEnd(UserFunctionEndInfo info) { // ─── Helpers ───────────────────────────────────────────────────────── - private boolean bindTracer() { - if (tracer != null) { - return true; - } - synchronized (this) { - if (tracer != null) { - return true; - } - var setup = OtelPluginSupport.tryResolveGlobalProvider(instrumentationName, "ExecutionOtelPlugin"); - if (setup == null) { - return false; - } - sdkTracerProvider = setup.sdkTracerProvider(); - tracer = setup.tracer(); - return true; - } + /** + * True when this instance emits no telemetry: either the tracer could not be bound for this invocation, or the + * invocation has already ended and its spans are closed. + */ + private boolean disabled() { + return invocationSpan == null || ended; } private void applyInvocationStatus(Span span, InvocationEndInfo info) { @@ -636,17 +673,11 @@ private Context withDurableDecision(Context context) { } private TraceFlags effectiveTraceFlags() { - var invocation = invocationSpan; - if (invocation != null) { - return invocation.getSpanContext().getTraceFlags(); - } - var trace = executionTrace; - return trace != null ? trace.flags() : TraceFlags.getDefault(); + return invocationSpan.getSpanContext().getTraceFlags(); } private TraceState effectiveTraceState() { - var invocation = invocationSpan; - return invocation != null ? invocation.getSpanContext().getTraceState() : TraceState.getDefault(); + return invocationSpan.getSpanContext().getTraceState(); } /** diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginProvider.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginProvider.java index 011e1dc02..1bf442d37 100644 --- a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginProvider.java +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginProvider.java @@ -3,30 +3,27 @@ package software.amazon.lambda.durable.otel; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; +import software.amazon.lambda.durable.plugin.InvocationInfo; /** * Dynamically loads {@link ExecutionOtelPlugin} when {@code DURABLE_EXECUTION_PLUGINS} contains {@code otel-execution}. + * + *

The provider is itself the per-invocation factory: it holds the environment-lifetime state (the ADOT global + * provider binding, the ID generator) once and creates one plugin instance per invocation from it. */ public final class ExecutionOtelPluginProvider implements DurableExecutionPluginProvider { + private final DurableExecutionPluginFactory factory = ExecutionOtelPlugin.factory(); + @Override public String getName() { return "otel-execution"; } @Override - public int getApiVersion() { - return API_VERSION; - } - - @Override - public Class getPluginType() { - return ExecutionOtelPlugin.class; - } - - @Override - public DurableExecutionPlugin createPlugin() { - return new ExecutionOtelPlugin(); + public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { + return factory.createPlugin(invocationInfo); } } diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPlugin.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPlugin.java index e89804bdc..389159965 100644 --- a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPlugin.java +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPlugin.java @@ -26,6 +26,7 @@ import org.slf4j.LoggerFactory; import org.slf4j.MDC; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.OperationEndInfo; @@ -61,9 +62,18 @@ *

  • Tracing: Active (to populate {@code _X_AMZN_TRACE_ID}) * * - *

    When using {@link #InvocationOtelPlugin()}, the plugin resolves the global provider at invocation start. If the - * OpenTelemetry Java agent is not initialized yet, telemetry is disabled for that entire invocation and provider - * resolution is retried on the next invocation. + *

    Lifetime. One instance serves exactly one Lambda invocation: {@link #factory()} and its overloads return a + * {@link DurableExecutionPluginFactory} that the SDK calls once per invocation, and the instance is dropped when the + * invocation returns. Everything about the invocation — the execution ARN, the resolved execution trace and ancestor, + * the sampling intent, the Invocation span, the deferred Workflow span context — is therefore a {@code final} field, + * resolved in the constructor from the {@link InvocationInfo} the factory receives (the same instance + * {@link #onInvocationStart(InvocationInfo)} then receives). Nothing is reset between invocations because nothing is + * carried between them. + * + *

    What belongs to the execution environment stays in the factory's {@link OtelPluginEnvironment}: the configuration, + * the ID generator, and either the application-owned tracer provider (built once) or the lazily resolved ADOT global + * provider. On the agent path, an invocation whose instance cannot resolve the global provider emits no telemetry at + * all, and the next invocation's instance resolves it again. * *

    X-Ray console limitation: In the X-Ray "Segments Timeline" ungrouped view, the plugin's spans (Invocation, * operation, attempt) do not appear as nested subsegments of the Lambda platform segment. This is a known limitation of @@ -72,39 +82,56 @@ * view to inspect parent-child relationships within the shared execution trace and the links between operation spans * and the Workflow span. * - *

    Thread-safe: uses {@link ConcurrentHashMap} for span/scope storage since the SDK runs user code on multiple - * threads. + *

    Thread-safe within its invocation: the SDK runs user code on multiple threads, so the open-span registries are + * {@link ConcurrentHashMap}s. The invocation's identity needs no such protection — it is final state written before the + * SDK publishes the instance to those threads. */ -public class InvocationOtelPlugin implements DurableExecutionPlugin { +public final class InvocationOtelPlugin implements DurableExecutionPlugin { private static final Logger logger = LoggerFactory.getLogger(InvocationOtelPlugin.class); - private volatile SdkTracerProvider sdkTracerProvider; - private volatile Tracer tracer; + // ─── Environment lifetime (shared with every other invocation's instance) ───────────── + private final DeterministicIdGenerator idGenerator; - private final ContextExtractor contextExtractor; private final boolean enableMdc; private final String workflowSpanName; - private final String instrumentationName; - - // Per-invocation state - private volatile boolean tracingEnabled; - private volatile Span invocationSpan; - private volatile String durableExecutionArn; - // Trace ID and flags of the execution trace, published together as one snapshot so readers never pair a trace ID - // with mismatched flags. - private volatile ExecutionTrace executionTrace; - // The execution's single sampling intent for this invocation, computed once at onInvocationStart and attached to - // every durable span's parent context so DurableSampler applies it (a resolved decision verbatim, or a deferral to - // its own delegate) without re-invoking the configured sampler per span. - private volatile DurableSamplingDecision.Intent samplingIntent; - - // Deferred Workflow placeholder; the recording span is emitted only on terminal invocation. - private volatile SpanContext workflowSpanContext; - private volatile SpanContext executionAncestor; - private volatile Instant executionStartTime; - - /** Immutable snapshot of the resolved execution trace, read atomically through a single volatile reference. */ + + // ─── This invocation, all resolved in the constructor from its InvocationInfo ───────── + + /** The provider used to flush before Lambda freezes; null when it is not visible to the application. */ + private final SdkTracerProvider sdkTracerProvider; + + /** Null when telemetry is disabled for this invocation, which makes every hook on this instance a no-op. */ + private final Tracer tracer; + + private final String durableExecutionArn; + private final Instant executionStartTime; + + /** Trace ID and flags of the execution trace, resolved together so they can never be paired mismatched. */ + private final ExecutionTrace executionTrace; + + /** + * The execution's single sampling intent for this invocation, resolved once and attached to every durable span's + * parent context so DurableSampler applies it (a resolved decision verbatim, or a deferral to its own delegate) + * without re-invoking the configured sampler per span. + */ + private final DurableSamplingDecision.Intent samplingIntent; + + private final SpanContext executionAncestor; + + private final Span invocationSpan; + + /** Deferred Workflow placeholder; the recording span is emitted only on terminal invocation. */ + private final SpanContext workflowSpanContext; + + /** + * Set when this invocation ends; never cleared, because an instance is never reused. Read by the operation and user + * function hooks, which may run on other threads of this invocation, so that a straggler hook arriving after the + * spans have been ended does not open a new one — volatile for that publication. + */ + private volatile boolean ended; + + /** Immutable snapshot of the resolved execution trace. */ private record ExecutionTrace(String traceId, TraceFlags flags) {} // Thread-safe storage for operation spans (keyed by operationId) — open spans that need ending @@ -121,97 +148,114 @@ private record ExecutionTrace(String traceId, TraceFlags flags) {} private final ConcurrentLinkedDeque operationStartOrder = new ConcurrentLinkedDeque<>(); /** - * Creates an OTel plugin with default settings: X-Ray context extraction, MDC enabled. - * - *

    Uses the provided tracer provider builder. Customers configure exporters and span processors on the builder — - * the plugin handles ID generation. - * - *

    For ADOT Java agent usage, prefer {@link #InvocationOtelPlugin()} with the plugin jar configured through - * {@code OTEL_JAVAAGENT_EXTENSIONS}. Use this builder constructor when you want to own the exporter pipeline: + * Returns a factory that creates one plugin instance per invocation against the ADOT Java agent's global provider, + * with default settings: X-Ray context extraction and MDC enabled. * * 

    {@code
-     * var exporter = LoggingSpanExporter.create();
-     * var plugin = new InvocationOtelPlugin(
-     *     SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)));
+     * DurableConfig.builder().withPlugins(InvocationOtelPlugin.factory()).build();
      * }
    * - * @param tracerProviderBuilder the tracer provider builder (its ID generator will be wrapped) + * @return the per-invocation plugin factory to hand to {@code DurableConfig.Builder.withPlugins} + */ + public static DurableExecutionPluginFactory factory() { + return factory(OtelPluginConfig.defaults()); + } + + /** + * Returns a factory that creates one plugin instance per invocation against the ADOT Java agent's global provider. + * + *

    The global provider is resolved when the first invocation's instance needs it. If the agent has not + * initialized it yet, that invocation emits no telemetry and the next invocation's instance resolves it again. + * + * @param config the plugin configuration + * @return the per-invocation plugin factory to hand to {@code DurableConfig.Builder.withPlugins} */ - public InvocationOtelPlugin(SdkTracerProviderBuilder tracerProviderBuilder) { - this(tracerProviderBuilder, OtelPluginConfig.defaults()); + public static DurableExecutionPluginFactory factory(OtelPluginConfig config) { + var environment = OtelPluginEnvironment.forGlobalProvider(config); + return info -> new InvocationOtelPlugin(environment, info); } /** - * Creates an OTel plugin with default settings: X-Ray context extraction and MDC enabled. + * Returns a factory that creates one plugin instance per invocation against an application-owned tracer provider, + * with default settings: X-Ray context extraction and MDC enabled. + * + *

    Customers configure exporters and span processors on the builder — the plugin handles ID generation. The + * provider is built once, here, and shared by every invocation's instance: + * + * 

    {@code
+     * var exporter = LoggingSpanExporter.create();
+     * var factory = InvocationOtelPlugin.factory(
+     *     SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)));
+     * }
    * - *

    Resolves {@code GlobalOpenTelemetry} at invocation start. If the ADOT Java agent has not initialized it yet, - * telemetry is disabled for that invocation and resolution is retried on the next invocation. + * @param tracerProviderBuilder the tracer provider builder (its ID generator and sampler will be wrapped) + * @return the per-invocation plugin factory to hand to {@code DurableConfig.Builder.withPlugins} */ - public InvocationOtelPlugin() { - this(OtelPluginConfig.defaults()); + public static DurableExecutionPluginFactory factory(SdkTracerProviderBuilder tracerProviderBuilder) { + return factory(tracerProviderBuilder, OtelPluginConfig.defaults()); } /** - * Creates an OTel plugin from the given tracer provider builder and configuration. + * Returns a factory that creates one plugin instance per invocation against an application-owned tracer provider. * *

    Customers configure exporters and span processors on the builder; all other tunables (context extractor, MDC - * toggle, Workflow span name, instrumentation scope name) come from {@link OtelPluginConfig}. Use - * {@link OtelPluginConfig#builder()} for readable, named configuration: + * toggle, Workflow span name, instrumentation scope name) come from {@link OtelPluginConfig}: * * 

    {@code
-     * var plugin = new InvocationOtelPlugin(
+     * var factory = InvocationOtelPlugin.factory(
      *     SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)),
      *     OtelPluginConfig.builder().enableMdc(false).workflowSpanName("Workflow").build());
      * }
    * - * @param tracerProviderBuilder the tracer provider builder (its ID generator will be wrapped) + * @param tracerProviderBuilder the tracer provider builder (its ID generator and sampler will be wrapped) * @param config the plugin configuration + * @return the per-invocation plugin factory to hand to {@code DurableConfig.Builder.withPlugins} */ - public InvocationOtelPlugin(SdkTracerProviderBuilder tracerProviderBuilder, OtelPluginConfig config) { - this.idGenerator = DeterministicIdGenerator.installOn(tracerProviderBuilder); - // Wrap the configured sampler so durable spans use the execution's single precomputed decision. - DurableSampler.installOn(tracerProviderBuilder); - - this.sdkTracerProvider = tracerProviderBuilder.build(); - this.tracer = sdkTracerProvider.get(config.instrumentationName()); - this.contextExtractor = config.contextExtractor(); - this.enableMdc = config.enableMdc(); - this.workflowSpanName = config.workflowSpanName(); - this.instrumentationName = config.instrumentationName(); + public static DurableExecutionPluginFactory factory( + SdkTracerProviderBuilder tracerProviderBuilder, OtelPluginConfig config) { + var environment = OtelPluginEnvironment.forProviderBuilder(tracerProviderBuilder, config); + return info -> new InvocationOtelPlugin(environment, info); } /** - * Creates an OTel plugin from configuration alone (no caller-supplied tracer provider builder). + * Creates the instance that serves one invocation. * - *

    The config-only constructor uses the ADOT/global provider. Supply a {@code SdkTracerProviderBuilder} via the - * two-arg constructor for an application-owned provider. + *

    Everything this invocation's spans are keyed by is resolved here, from the {@code info} the factory received: + * the tracer binding, the extracted context, the canonical execution trace and its ancestor, the single sampling + * intent, the Invocation span, and the deferred Workflow span context. Resolving them in the constructor — before + * the SDK publishes this instance to the operation and user function threads — is what lets them be {@code final} + * rather than volatile per-invocation state. * - * @param config the plugin configuration + *

    When the tracer cannot be bound, telemetry is disabled for this invocation: the span fields stay null and + * every hook returns immediately. The next invocation gets a new instance, which binds again. */ - public InvocationOtelPlugin(OtelPluginConfig config) { - this.contextExtractor = config.contextExtractor(); + private InvocationOtelPlugin(OtelPluginEnvironment environment, InvocationInfo info) { + var config = environment.config(); + this.idGenerator = environment.idGenerator(); this.enableMdc = config.enableMdc(); this.workflowSpanName = config.workflowSpanName(); - this.instrumentationName = config.instrumentationName(); - this.idGenerator = OtelPluginSupport.createDefaultIdGenerator(); - } - - // ─── Invocation hooks ──────────────────────────────────────────────── - - @Override - public void onInvocationStart(InvocationInfo info) { - tracingEnabled = false; - if (!bindTracer()) { + this.durableExecutionArn = info.durableExecutionArn(); + this.executionStartTime = info.executionStartTime(); + + var setup = environment.bind("InvocationOtelPlugin"); + if (setup == null) { + this.sdkTracerProvider = null; + this.tracer = null; + this.samplingIntent = null; + this.executionTrace = null; + this.executionAncestor = null; + this.invocationSpan = null; + this.workflowSpanContext = null; return; } + this.sdkTracerProvider = setup.sdkTracerProvider(); + this.tracer = setup.tracer(); - this.durableExecutionArn = info.durableExecutionArn(); - - var extracted = contextExtractor.extract(); + var extracted = config.contextExtractor().extract(); // Resolve the execution ancestor the Workflow span parents onto so it joins the stable-per-execution trace. - var canonicalTraceId = ExecutionTraceContext.canonicalTraceId( - extracted, info.durableExecutionArn(), info.executionStartTime(), idGenerator); + var canonicalTraceId = + ExecutionTraceContext.canonicalTraceId(extracted, durableExecutionArn, executionStartTime, idGenerator); // Resolve the execution's sampling decision once for this invocation as a full SamplingResult, then apply it to // every durable span via DurableSampler (see below). The execution ancestor's trace flags are derived from the // same decision so a parent-based sampler stays consistent with it. @@ -221,18 +265,17 @@ public void onInvocationStart(InvocationInfo info) { Span.current(), canonicalTraceId, workflowSpanName, - Attributes.of(DURABLE_EXECUTION_ARN, info.durableExecutionArn())); + Attributes.of(DURABLE_EXECUTION_ARN, durableExecutionArn)); // A null decision is unresolved on the agent path: defer to DurableSampler's own delegate (keyed by trace ID), // rather than fabricating a decision that would bypass an installed drop/rate-limit policy. - samplingIntent = decision != null + this.samplingIntent = decision != null ? DurableSamplingDecision.Intent.resolved(decision) : DurableSamplingDecision.Intent.deferred(canonicalTraceId); var sampled = OtelPluginSupport.isSampled(decision); var execCtx = ExecutionTraceContext.resolve( - extracted, canonicalTraceId, info.durableExecutionArn(), idGenerator, () -> sampled); - executionTrace = new ExecutionTrace(canonicalTraceId, execCtx.traceFlags()); - executionAncestor = execCtx.executionAncestor(); - executionStartTime = info.executionStartTime(); + extracted, canonicalTraceId, durableExecutionArn, idGenerator, () -> sampled); + this.executionTrace = new ExecutionTrace(canonicalTraceId, execCtx.traceFlags()); + this.executionAncestor = execCtx.executionAncestor(); // Invocation span parent — the same-trace ambient span when available, then the execution ancestor, so the // Invocation span stays on the execution trace. @@ -242,46 +285,53 @@ public void onInvocationStart(InvocationInfo info) { var spanBuilder = tracer.spanBuilder("Invocation") .setSpanKind(SpanKind.INTERNAL) .setParent(parentContext) - .setAttribute(DURABLE_EXECUTION_ARN, info.durableExecutionArn()) + .setAttribute(DURABLE_EXECUTION_ARN, durableExecutionArn) .setAttribute(DURABLE_FIRST_INVOCATION, info.isFirstInvocation()); if (info.requestId() != null) { spanBuilder.setAttribute(AttributeKey.stringKey("faas.invocation_id"), info.requestId()); } - invocationSpan = startDurableSpan(spanBuilder); + this.invocationSpan = startDurableSpan(spanBuilder); // Defer the recording Workflow span until terminal completion. The placeholder uses the Invocation span's // resolved sampling metadata so operation links match the span that is eventually exported. - var workflowSpanId = idGenerator.generateWorkflowSpanId(info.durableExecutionArn()); var invocationContext = invocationSpan.getSpanContext(); - workflowSpanContext = SpanContext.create( - canonicalTraceId, workflowSpanId, invocationContext.getTraceFlags(), invocationContext.getTraceState()); + this.workflowSpanContext = SpanContext.create( + canonicalTraceId, + idGenerator.generateWorkflowSpanId(durableExecutionArn), + invocationContext.getTraceFlags(), + invocationContext.getTraceState()); + } - // Inject MDC on the handler thread so handler-level logs (between steps) have trace context. - // This runs on the same thread as context.getLogger() calls in the handler. - if (enableMdc) { - MDC.put( - MdcSpanEnricher.MDC_TRACE_ID, - invocationSpan.getSpanContext().getTraceId()); + // ─── Invocation hooks ──────────────────────────────────────────────── + + @Override + public void onInvocationStart(InvocationInfo info) { + // This invocation's identity and its Invocation span were resolved in the constructor, from the very + // InvocationInfo this hook receives. What is left is the MDC injection, which belongs here because it must run + // on the handler thread — the same thread as the context.getLogger() calls in the handler — so handler-level + // logs between steps carry trace context. + if (invocationSpan == null || !enableMdc) { + return; } - tracingEnabled = true; + MDC.put(MdcSpanEnricher.MDC_TRACE_ID, invocationSpan.getSpanContext().getTraceId()); } @Override public void onInvocationEnd(InvocationEndInfo info) { - if (!tracingEnabled) { + if (disabled()) { return; } - tracingEnabled = false; + // Set before the spans are ended, so a straggler hook from another thread of this invocation cannot open a span + // under one that is already closed. Never cleared: this instance serves no second invocation. + ended = true; // Clear invocation-level MDC (set in onInvocationStart on the handler thread) if (enableMdc) { MdcSpanEnricher.clear(); } - if (invocationSpan == null) return; - endOpenSpansChildFirst(); // End invocation span @@ -311,10 +361,9 @@ public void onInvocationEnd(InvocationEndInfo info) { } invocationSpan.end(); - invocationSpan = null; // Materialize the Workflow span only on terminal status. - if (isTerminal(info) && workflowSpanContext != null && executionAncestor != null) { + if (isTerminal(info)) { var workflowSpanBuilder = tracer.spanBuilder(workflowSpanName) .setSpanKind(SpanKind.INTERNAL) .setParent(withDurableDecision(Context.root().with(Span.wrap(executionAncestor)))) @@ -338,10 +387,6 @@ public void onInvocationEnd(InvocationEndInfo info) { } workflowSpan.end(); } - workflowSpanContext = null; - executionAncestor = null; - executionStartTime = null; - samplingIntent = null; if (sdkTracerProvider != null) { // Flush spans before Lambda freezes @@ -356,7 +401,7 @@ public void onInvocationEnd(InvocationEndInfo info) { @Override public void onOperationStart(OperationInfo info) { - if (!tracingEnabled) return; + if (disabled()) return; if (info.id() == null) return; var parentContext = resolveParentContext(info.parentId()); @@ -397,7 +442,7 @@ public void onOperationStart(OperationInfo info) { @Override public void onOperationEnd(OperationEndInfo info) { - if (!tracingEnabled) return; + if (disabled()) return; if (info.id() == null) return; var span = operationSpans.remove(info.id()); @@ -468,7 +513,7 @@ public void onOperationEnd(OperationEndInfo info) { @Override public void onUserFunctionStart(UserFunctionStartInfo info) { - if (!tracingEnabled) return; + if (disabled()) return; // Skip attempt spans for CONTEXT operations — they are a scoping construct, not a // retriable unit of work, so attempt number/outcome attributes don't apply. @@ -528,7 +573,7 @@ public void onUserFunctionStart(UserFunctionStartInfo info) { @Override public void onUserFunctionEnd(UserFunctionEndInfo info) { - if (!tracingEnabled) return; + if (disabled()) return; var key = attemptKey(info.id(), info.attempt()); @@ -575,34 +620,22 @@ public void onUserFunctionEnd(UserFunctionEndInfo info) { // ─── Helpers ───────────────────────────────────────────────────────── - private boolean bindTracer() { - if (tracer != null) { - return true; - } - synchronized (this) { - if (tracer != null) { - return true; - } - var setup = OtelPluginSupport.tryResolveGlobalProvider(instrumentationName, "InvocationOtelPlugin"); - if (setup == null) { - return false; - } - sdkTracerProvider = setup.sdkTracerProvider(); - tracer = setup.tracer(); - return true; - } + /** + * True when this instance emits no telemetry: either the tracer could not be bound for this invocation, or the + * invocation has already ended and its spans are closed. + */ + private boolean disabled() { + return invocationSpan == null || ended; } private void endOpenSpansChildFirst() { - // Attempt spans are children of operation spans. + // Attempt spans are children of operation spans, so release their scopes and end them first. for (var scope : attemptScopes.values()) { scope.close(); } - attemptScopes.clear(); for (var span : attemptSpans.values()) { span.end(); } - attemptSpans.clear(); // End still-open operation spans with the STARTED status set in onOperationStart. // A later invocation's onOperationEnd emits a continuation span with the real terminal status. @@ -613,8 +646,8 @@ private void endOpenSpansChildFirst() { span.end(); } } - operationSpans.clear(); - operationContexts.clear(); + // The registries are not emptied afterwards: every span they held has been ended above, and this instance is + // dropped when the invocation returns, so there is nothing to recycle them for. } /** @@ -710,17 +743,11 @@ private void addInitialOperationLink(SpanBuilder spanBuilder, String operationId } private TraceFlags effectiveTraceFlags() { - var invocation = invocationSpan; - if (invocation != null) { - return invocation.getSpanContext().getTraceFlags(); - } - var trace = executionTrace; - return trace != null ? trace.flags() : TraceFlags.getDefault(); + return invocationSpan.getSpanContext().getTraceFlags(); } private TraceState effectiveTraceState() { - var invocation = invocationSpan; - return invocation != null ? invocation.getSpanContext().getTraceState() : TraceState.getDefault(); + return invocationSpan.getSpanContext().getTraceState(); } private static boolean isTerminal(InvocationEndInfo info) { diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPluginProvider.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPluginProvider.java index 165ed88fc..fd81b39d8 100644 --- a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPluginProvider.java +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPluginProvider.java @@ -3,31 +3,28 @@ package software.amazon.lambda.durable.otel; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; +import software.amazon.lambda.durable.plugin.InvocationInfo; /** * Dynamically loads {@link InvocationOtelPlugin} when {@code DURABLE_EXECUTION_PLUGINS} contains * {@code otel-invocation}. + * + *

    The provider is itself the per-invocation factory: it holds the environment-lifetime state (the ADOT global + * provider binding, the ID generator) once and creates one plugin instance per invocation from it. */ public final class InvocationOtelPluginProvider implements DurableExecutionPluginProvider { + private final DurableExecutionPluginFactory factory = InvocationOtelPlugin.factory(); + @Override public String getName() { return "otel-invocation"; } @Override - public int getApiVersion() { - return API_VERSION; - } - - @Override - public Class getPluginType() { - return InvocationOtelPlugin.class; - } - - @Override - public DurableExecutionPlugin createPlugin() { - return new InvocationOtelPlugin(); + public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { + return factory.createPlugin(invocationInfo); } } diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginConfig.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginConfig.java index ed4ac9be5..8fc5b136a 100644 --- a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginConfig.java +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginConfig.java @@ -10,8 +10,8 @@ * mirrors the {@code OtelPluginConfig} object in the JavaScript SDK and the {@code OtelPluginConfig} dataclass in the * Python SDK for cross-SDK parity. * - *

    Construct via {@link #builder()} and pass to a plugin's {@code (SdkTracerProviderBuilder, OtelPluginConfig)} - * constructor: + *

    Construct via {@link #builder()} and pass to a plugin's {@code factory(SdkTracerProviderBuilder, + * OtelPluginConfig)}: * * 

    {@code
  * var config = OtelPluginConfig.builder()
@@ -20,7 +20,7 @@
  *     .workflowSpanName("Workflow")
  *     .instrumentationName("my-scope")
  *     .build();
- * var plugin = new InvocationOtelPlugin(tracerProviderBuilder, config);
+ * var factory = InvocationOtelPlugin.factory(tracerProviderBuilder, config);
  * }
    * *

    Defaults: {@code contextExtractor = new XRayContextExtractor()}, {@code enableMdc = true}, {@code workflowSpanName diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java new file mode 100644 index 000000000..91caba1b2 --- /dev/null +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java @@ -0,0 +1,92 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.otel; + +import io.opentelemetry.sdk.trace.SdkTracerProviderBuilder; + +/** + * Everything the OTel plugins need that belongs to the execution environment rather than to one invocation. + * + *

    A plugin instance now serves exactly one Lambda invocation, so the objects that must exist once per environment + * live here: the resolved {@link OtelPluginConfig}, the {@link DeterministicIdGenerator}, and — for an + * application-owned tracer provider — the built provider and its tracer. {@code InvocationOtelPlugin.factory(...)} and + * {@code ExecutionOtelPlugin.factory(...)} create one of these and hand the same instance to every plugin instance they + * create, so the provider is built (and its ID generator and sampler installed) once per environment rather than once + * per invocation. + * + *

    On the ADOT Java agent path there is no provider to build here: the global provider is resolved on first use and + * then cached. An invocation that runs before the agent has finished initializing therefore disables telemetry for + * itself only, and the next invocation's instance resolves the provider again. + */ +final class OtelPluginEnvironment { + + private final OtelPluginConfig config; + private final DeterministicIdGenerator idGenerator; + + /** The application-owned provider and tracer, or null on the Java agent path. */ + private final OtelPluginSupport.ProviderSetup ownedSetup; + + /** + * The global provider and tracer, once resolved. Environment-lifetime state shared by every invocation's instance, + * hence volatile; a lost race only resolves the same global provider twice. + */ + private volatile OtelPluginSupport.ProviderSetup resolvedGlobalSetup; + + private OtelPluginEnvironment( + OtelPluginConfig config, DeterministicIdGenerator idGenerator, OtelPluginSupport.ProviderSetup ownedSetup) { + this.config = config; + this.idGenerator = idGenerator; + this.ownedSetup = ownedSetup; + } + + /** + * Builds the application-owned provider once: wraps the builder's ID generator and sampler, builds the provider and + * gets the tracer. Every invocation's plugin instance then shares them. + */ + static OtelPluginEnvironment forProviderBuilder( + SdkTracerProviderBuilder tracerProviderBuilder, OtelPluginConfig config) { + var idGenerator = DeterministicIdGenerator.installOn(tracerProviderBuilder); + // Wrap the configured sampler so durable spans use the execution's single precomputed decision. + DurableSampler.installOn(tracerProviderBuilder); + var sdkTracerProvider = tracerProviderBuilder.build(); + var setup = new OtelPluginSupport.ProviderSetup( + sdkTracerProvider, sdkTracerProvider.get(config.instrumentationName())); + return new OtelPluginEnvironment(config, idGenerator, setup); + } + + /** The Java agent path: the global provider is resolved lazily, when an invocation's instance first needs it. */ + static OtelPluginEnvironment forGlobalProvider(OtelPluginConfig config) { + return new OtelPluginEnvironment(config, OtelPluginSupport.createDefaultIdGenerator(), null); + } + + OtelPluginConfig config() { + return config; + } + + DeterministicIdGenerator idGenerator() { + return idGenerator; + } + + /** + * The provider and tracer one invocation's plugin instance should use, or {@code null} when telemetry must be + * disabled for that invocation because the agent's global provider is not available yet. + * + * @param pluginName the plugin name used in diagnostics + */ + OtelPluginSupport.ProviderSetup bind(String pluginName) { + if (ownedSetup != null) { + return ownedSetup; + } + var alreadyResolved = resolvedGlobalSetup; + if (alreadyResolved != null) { + return alreadyResolved; + } + var setup = OtelPluginSupport.tryResolveGlobalProvider(config.instrumentationName(), pluginName); + if (setup != null) { + // Resolution succeeded, so it holds for the rest of this environment's life: cache it so later invocations + // neither re-resolve nor re-log it. A failure is not cached — that is what makes the retry per invocation. + resolvedGlobalSetup = setup; + } + return setup; + } +} diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginSupport.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginSupport.java index 63fbec908..129c20e56 100644 --- a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginSupport.java +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginSupport.java @@ -17,7 +17,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -/** Shared utilities for OTel plugin default constructor support (ADOT Java agent SPI path). */ +/** Shared utilities for the OTel plugins' ADOT Java agent SPI path. */ final class OtelPluginSupport { private static final Logger logger = LoggerFactory.getLogger(OtelPluginSupport.class); @@ -47,7 +47,7 @@ static DeterministicIdGenerator createDefaultIdGenerator() { * sampled span yields {@code RECORD_AND_SAMPLE}; an unsampled but recording span yields {@code RECORD_ONLY} * (its spans still reach processors); only an unsampled, non-recording span yields {@code DROP}; *

  • Application-owned provider: configured sampler, once. When the tracer provider is reachable (the - * two-argument constructor path), its sampler is read directly and evaluated a single time with + * application-owned provider path), its sampler is read directly and evaluated a single time with * {@code ROOT_CONTEXT} (so a parent-based sampler applies its root policy), the canonical trace ID, span * name, and attributes, and its full result is returned; *
  • Java-agent path: defer to the installed sampler. When the provider is not visible diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/DurableSamplerTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/DurableSamplerTest.java index 6fea5784e..cdf4f7f45 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/DurableSamplerTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/DurableSamplerTest.java @@ -6,6 +6,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertSame; import static org.junit.jupiter.api.Assertions.assertTrue; +import static software.amazon.lambda.durable.otel.Invocations.started; import io.opentelemetry.api.GlobalOpenTelemetry; import io.opentelemetry.api.common.Attributes; @@ -172,7 +173,7 @@ private static Sampler captureInstalledSampler(Sampler effectiveSampler) { void configuredSampler_isEvaluatedAtMostOncePerInvocation() { var delegate = new CountingSampler(Sampler.alwaysOn()); var exporter = InMemorySpanExporter.create(); - var plugin = new InvocationOtelPlugin( + var pluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().setSampler(delegate).addSpanProcessor(SimpleSpanProcessor.create(exporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) @@ -180,7 +181,7 @@ void configuredSampler_isEvaluatedAtMostOncePerInvocation() { .build()); // A full invocation with a Workflow span, Invocation span, operation span, and attempt span. - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(pluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "step", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onUserFunctionStart( @@ -226,7 +227,7 @@ void explicitNotSampled_winsOverConfiguredAlwaysOn() { private InMemorySpanExporter exportedWith(Sampler configuredSampler, ExtractedContext.Sampling sampling) { var exporter = InMemorySpanExporter.create(); var extracted = new ExtractedContext(TRACE_ID, SPAN_ID, sampling); - var plugin = new InvocationOtelPlugin( + var pluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder() .setSampler(configuredSampler) .addSpanProcessor(SimpleSpanProcessor.create(exporter)), @@ -235,7 +236,7 @@ private InMemorySpanExporter exportedWith(Sampler configuredSampler, ExtractedCo .enableMdc(false) .build()); - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(pluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); return exporter; } diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginIntegrationTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginIntegrationTest.java index 129a1b045..471d3505e 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginIntegrationTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginIntegrationTest.java @@ -42,14 +42,15 @@ void setUp() { OtelPluginAutoConfigurationState.resetInstalledForTest(); spanExporter = InMemorySpanExporter.create(); - var plugin = new ExecutionOtelPlugin( + // One factory for the environment; the SDK creates one plugin instance per invocation from it. + var factory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) .enableMdc(false) .build()); - otelConfig = DurableConfig.builder().withPlugins(plugin).build(); + otelConfig = DurableConfig.builder().withPlugins(factory).build(); } @AfterEach @@ -170,8 +171,9 @@ public ContextPropagators getPropagators() { } }); - var defaultConfig = - DurableConfig.builder().withPlugins(new ExecutionOtelPlugin()).build(); + var defaultConfig = DurableConfig.builder() + .withPlugins(ExecutionOtelPlugin.factory()) + .build(); var runner = LocalDurableTestRunner.create( String.class, (input, ctx) -> ctx.step("wrapped-step", String.class, stepCtx -> "Hello " + input), diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginTest.java index 8b84efc10..7729739e3 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginTest.java @@ -3,6 +3,7 @@ package software.amazon.lambda.durable.otel; import static org.junit.jupiter.api.Assertions.*; +import static software.amazon.lambda.durable.otel.Invocations.started; import io.opentelemetry.api.GlobalOpenTelemetry; import io.opentelemetry.api.common.AttributeKey; @@ -37,7 +38,9 @@ class ExecutionOtelPluginTest { private static final String CONFIGURED_SERVICE_NAME = "durable-execution-conformance"; private InMemorySpanExporter spanExporter; - private ExecutionOtelPlugin plugin; + + /** The environment's plugin factory; each test creates one instance per invocation from it. */ + private DurableExecutionPluginFactory factory; @BeforeEach void setUp() { @@ -46,7 +49,7 @@ void setUp() { OtelPluginAutoConfigurationState.resetInstalledForTest(); spanExporter = InMemorySpanExporter.create(); var resource = Resource.create(Attributes.of(SERVICE_NAME, CONFIGURED_SERVICE_NAME)); - plugin = new ExecutionOtelPlugin( + factory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder() .setResource(resource) .addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), @@ -65,12 +68,12 @@ void tearDown() { OtelPluginAutoConfigurationState.resetInstalledForTest(); } - // ─── Default constructor ───────────────────────────────────────────── + // ─── Java agent path (global provider) ─────────────────────────────── @Test void customInstrumentationName_isUsedForTracerScope() { var exporter = InMemorySpanExporter.create(); - var customPlugin = new ExecutionOtelPlugin( + var customPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) @@ -78,7 +81,7 @@ void customInstrumentationName_isUsedForTracerScope() { .workflowSpanName("Workflow") .instrumentationName("my-custom-scope") .build()); - customPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var customPlugin = started(customPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); customPlugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); var spans = exporter.getFinishedSpanItems(); @@ -89,12 +92,13 @@ void customInstrumentationName_isUsedForTracerScope() { } @Test - void defaultConstructor_retriesGlobalProviderBindingOnNextInvocation() { + void agentPathFactory_bindsGlobalProviderOnALaterInvocationsInstance() { GlobalOpenTelemetry.resetForTest(); OtelPluginAutoConfigurationState.markInstalled(); - var defaultPlugin = new ExecutionOtelPlugin(); - defaultPlugin.onInvocationStart(new InvocationInfo("req-disabled", "arn:disabled", true, Instant.now())); + var defaultPluginFactory = ExecutionOtelPlugin.factory(); + var defaultPlugin = + started(defaultPluginFactory, new InvocationInfo("req-disabled", "arn:disabled", true, Instant.now())); defaultPlugin.onOperationStart(new OperationInfo( "op-disabled", "disabled-step", "STEP", "Step", null, Instant.now(), null, null, false)); defaultPlugin.onOperationEnd(new OperationEndInfo( @@ -121,7 +125,8 @@ void defaultConstructor_retriesGlobalProviderBindingOnNextInvocation() { .build(); OpenTelemetrySdk.builder().setTracerProvider(globalTracerProvider).buildAndRegisterGlobal(); - defaultPlugin.onInvocationStart(new InvocationInfo("req-enabled", "arn:enabled", true, Instant.now())); + defaultPlugin = + started(defaultPluginFactory, new InvocationInfo("req-enabled", "arn:enabled", true, Instant.now())); defaultPlugin.onOperationStart(new OperationInfo( "op-enabled", "enabled-step", "STEP", "Step", null, Instant.now(), null, null, false)); defaultPlugin.onOperationEnd(new OperationEndInfo( @@ -147,8 +152,8 @@ void defaultConstructor_retriesGlobalProviderBindingOnNextInvocation() { } @Test - void defaultConstructor_usesGlobalSdkTracerProviderDirectly() { - var defaultPlugin = new ExecutionOtelPlugin(); + void agentPathFactory_usesGlobalSdkTracerProviderDirectly() { + var defaultPluginFactory = ExecutionOtelPlugin.factory(); assertFalse(GlobalOpenTelemetry.isSet()); OtelPluginAutoConfigurationState.markInstalled(); @@ -158,7 +163,8 @@ void defaultConstructor_usesGlobalSdkTracerProviderDirectly() { .build(); OpenTelemetrySdk.builder().setTracerProvider(globalTracerProvider).buildAndRegisterGlobal(); - defaultPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var defaultPlugin = + started(defaultPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); defaultPlugin.onOperationStart( new OperationInfo("op-1", "step", "STEP", "Step", null, Instant.now(), null, null, false)); defaultPlugin.onOperationEnd(new OperationEndInfo( @@ -194,15 +200,22 @@ void executionOtelPluginProvider_isRegisteredAsServiceProvider() { .get(); assertEquals("otel-execution", provider.getName()); - assertEquals(DurableExecutionPluginProvider.API_VERSION, provider.getApiVersion()); - assertEquals(ExecutionOtelPlugin.class, provider.getPluginType()); + + // The provider is the per-invocation factory: it creates an ExecutionOtelPlugin for the invocation it is + // handed, + // and a distinct instance for the next one. + var first = provider.createPlugin(new InvocationInfo("req-1", ARN, true, Instant.now())); + var second = provider.createPlugin(new InvocationInfo("req-2", ARN, false, Instant.now())); + assertInstanceOf(ExecutionOtelPlugin.class, first); + assertInstanceOf(ExecutionOtelPlugin.class, second); + assertNotSame(first, second, "Each invocation gets its own plugin instance"); } // ─── Workflow root span lifecycle ──────────────────────────────────── @Test void terminalInvocation_exportsWorkflowAndInvocationSpans() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); var spans = spanExporter.getFinishedSpanItems(); @@ -217,7 +230,7 @@ void terminalInvocation_exportsWorkflowAndInvocationSpans() { @Test void spans_preserveConfiguredServiceName() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); for (var span : spanExporter.getFinishedSpanItems()) { @@ -231,7 +244,7 @@ void spans_preserveConfiguredServiceName() { @Test void workflowSpan_startsAtExecutionStartTime() { var start = Instant.parse("2026-01-15T08:00:00Z"); - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, start)); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, start)); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); var workflowSpan = spanByName(spanExporter.getFinishedSpanItems(), "Workflow"); @@ -243,7 +256,7 @@ void workflowSpan_startsAtExecutionStartTime() { @Test void workflowSpan_hasInternalKind() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); assertEquals( @@ -254,7 +267,7 @@ void workflowSpan_hasInternalKind() { @Test void workflowAndInvocationSpans_shareExecutionTrace_withoutAmbientContext() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); var spans = spanExporter.getFinishedSpanItems(); @@ -280,7 +293,7 @@ void workflowAndInvocationSpans_shareExecutionTrace_withoutAmbientContext() { void invocationStart_joinsAmbientTrace_whenAmbientIsOnExecutionTrace() { // Drive an invocation to learn the canonical execution trace ID, then start a fresh invocation with an ambient // span on that same trace: the Invocation span joins the ambient span directly. - plugin.onInvocationStart(new InvocationInfo("req-0", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-0", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-0", ARN, true, InvocationStatus.SUCCEEDED, null)); var canonicalTraceId = spanByName(spanExporter.getFinishedSpanItems(), "Workflow").getTraceId(); @@ -290,7 +303,7 @@ void invocationStart_joinsAmbientTrace_whenAmbientIsOnExecutionTrace() { var ambient = SpanContext.create(canonicalTraceId, ambientSpanId, TraceFlags.getSampled(), TraceState.getDefault()); try (var ignored = Span.wrap(ambient).makeCurrent()) { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, false, Instant.now())); + plugin = started(factory, new InvocationInfo("req-1", ARN, false, Instant.now())); } plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, false, InvocationStatus.SUCCEEDED, null)); @@ -309,8 +322,11 @@ void invocationStart_staysOnExecutionTrace_withoutLinkingAmbientSpan() { var ambientSpanId = "1111111111111111"; var ambient = SpanContext.create(ambientTraceId, ambientSpanId, TraceFlags.getSampled(), TraceState.getDefault()); + // The instance is created inside the ambient scope because the invocation's parent resolution happens when the + // factory creates it, not later. + DurableExecutionPlugin plugin; try (var ignored = Span.wrap(ambient).makeCurrent()) { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); } plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); @@ -330,7 +346,7 @@ void contextExtractor_isInvokedEveryInvocation_evenWithAmbientSpan_andBackendCon var backendParentId = "2222222222222222"; var extractCalls = new AtomicInteger(); var exporter = InMemorySpanExporter.create(); - var extractorPlugin = new ExecutionOtelPlugin( + var extractorPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)), OtelPluginConfig.builder() .contextExtractor(() -> { @@ -347,8 +363,9 @@ void contextExtractor_isInvokedEveryInvocation_evenWithAmbientSpan_andBackendCon "1111111111111111", TraceFlags.getSampled(), TraceState.getDefault()); + DurableExecutionPlugin extractorPlugin; try (var ignored = Span.wrap(ambient).makeCurrent()) { - extractorPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + extractorPlugin = started(extractorPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); } extractorPlugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); @@ -375,8 +392,10 @@ void executionTrace_isStableAcrossReinvocations_withDifferentAmbientTraces() { TraceState.getDefault()); var startTime = Instant.now(); + // Two invocations of the same execution, so two instances from the same factory. + DurableExecutionPlugin plugin; try (var ignored = Span.wrap(ambientA).makeCurrent()) { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, startTime)); + plugin = started(factory, new InvocationInfo("req-1", ARN, true, startTime)); } plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.PENDING, null)); var firstInvocationTrace = @@ -384,7 +403,7 @@ void executionTrace_isStableAcrossReinvocations_withDifferentAmbientTraces() { spanExporter.reset(); try (var ignored = Span.wrap(ambientB).makeCurrent()) { - plugin.onInvocationStart(new InvocationInfo("req-2", ARN, false, startTime)); + plugin = started(factory, new InvocationInfo("req-2", ARN, false, startTime)); } plugin.onInvocationEnd(new InvocationEndInfo("req-2", ARN, false, InvocationStatus.SUCCEEDED, null)); var secondInvocationTrace = @@ -398,7 +417,7 @@ void executionTrace_isStableAcrossReinvocations_withDifferentAmbientTraces() { @Test void nonTerminalInvocation_doesNotExportWorkflowSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.PENDING, null)); var spans = spanExporter.getFinishedSpanItems(); @@ -411,7 +430,7 @@ void nonTerminalInvocation_doesNotExportWorkflowSpan() { @Test void workflowSpan_exportedOnceAcrossInvocations_sameSpanId() { // Invocation 1: non-terminal → no Workflow span exported - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.PENDING, null)); assertTrue( spanExporter.getFinishedSpanItems().stream() @@ -420,7 +439,7 @@ void workflowSpan_exportedOnceAcrossInvocations_sameSpanId() { spanExporter.reset(); // Invocation 2: terminal → Workflow span exported with the deterministic ID - plugin.onInvocationStart(new InvocationInfo("req-2", ARN, false, Instant.now())); + plugin = started(factory, new InvocationInfo("req-2", ARN, false, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-2", ARN, false, InvocationStatus.SUCCEEDED, null)); var workflowSpan = spanByName(spanExporter.getFinishedSpanItems(), "Workflow"); @@ -432,7 +451,7 @@ void workflowSpan_exportedOnceAcrossInvocations_sameSpanId() { @Test void failedInvocation_setsErrorOnBothWorkflowAndInvocationSpans() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd( new InvocationEndInfo("req-1", ARN, true, InvocationStatus.FAILED, new RuntimeException("boom"))); @@ -444,7 +463,7 @@ void failedInvocation_setsErrorOnBothWorkflowAndInvocationSpans() { @Test void retryingInvocation_invocationSpanUnset_workflowNotExported() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo( "req-1", ARN, true, InvocationStatus.RETRYING, new RuntimeException("transient"))); @@ -462,7 +481,7 @@ void retryingInvocation_invocationSpanUnset_workflowNotExported() { @Test void operationSpan_carriesAttemptNumberAtEnd() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "flaky", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onOperationEnd(new OperationEndInfo( @@ -491,7 +510,7 @@ void operationSpan_carriesAttemptNumberAtEnd() { @Test void continuationOperationSpan_carriesAttemptNumber() { - plugin.onInvocationStart(new InvocationInfo("req-2", ARN, false, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-2", ARN, false, Instant.now())); // No matching onOperationStart in this invocation — continuation branch. plugin.onOperationEnd(new OperationEndInfo( "op-1", @@ -521,7 +540,7 @@ void continuationOperationSpan_carriesAttemptNumber() { void operationSpan_startsAtOperationStartTimestamp() { var opStart = Instant.parse("2026-02-01T10:00:00Z"); var opEnd = Instant.parse("2026-02-01T10:00:03Z"); - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart(new OperationInfo("op-1", "step-a", "STEP", "Step", null, opStart, null, null, false)); plugin.onOperationEnd(new OperationEndInfo( "op-1", "step-a", "STEP", "Step", null, opStart, opEnd, "SUCCEEDED", null, false, null, null)); @@ -536,7 +555,7 @@ void operationSpan_startsAtOperationStartTimestamp() { @Test void operationSpan_parentedToWorkflow_linkedToInvocation() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onOperationEnd(new OperationEndInfo( @@ -571,7 +590,7 @@ void operationSpan_parentedToWorkflow_linkedToInvocation() { @Test void attemptSpan_childOfOperation_linkedToInvocation() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "compute", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onUserFunctionStart( @@ -624,7 +643,7 @@ void attemptSpan_childOfOperation_linkedToInvocation() { @Test void attemptSpan_carriesOperationSubtype() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "process-order", "STEP", "Step", null, Instant.now(), false, 1)); plugin.onUserFunctionEnd(new UserFunctionEndInfo( @@ -653,7 +672,7 @@ void attemptSpan_carriesOperationSubtype() { @Test void childOperation_parentedToParentOperationSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart(new OperationInfo( "op-parent", "my-context", "CONTEXT", "RunInChildContext", null, Instant.now(), null, null, false)); plugin.onOperationStart(new OperationInfo( @@ -702,7 +721,7 @@ void contextOperation_currentContextCarriesResolvedFlags_withAlwaysOffSampler() // DurableSampler), so descendants inherit the resolved decision. With always_off the resolved decision is // unsampled, so the current context inside the context body must be unsampled — not a provisional sampled bit. var exporter = InMemorySpanExporter.create(); - var offPlugin = new ExecutionOtelPlugin( + var offPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder() .setSampler(Sampler.alwaysOff()) .addSpanProcessor(SimpleSpanProcessor.create(exporter)), @@ -712,7 +731,7 @@ void contextOperation_currentContextCarriesResolvedFlags_withAlwaysOffSampler() .workflowSpanName("Workflow") .build()); - offPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var offPlugin = started(offPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); offPlugin.onOperationStart( new OperationInfo("ctx-1", "my-ctx", "CONTEXT", "Context", null, Instant.now(), null, null, false)); offPlugin.onUserFunctionStart( @@ -757,7 +776,7 @@ void contextOperation_currentContextCarriesResolvedFlags_withAlwaysOffSampler() @Test void userFunctionFailure_setsErrorOnAttemptSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "failing", "STEP", "Step", null, Instant.now(), false, 1)); plugin.onUserFunctionEnd(new UserFunctionEndInfo( @@ -783,7 +802,7 @@ void userFunctionFailure_setsErrorOnAttemptSpan() { @Test void userFunctionSuccess_setsOkOnAttemptSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "compute", "STEP", "Step", null, Instant.now(), false, 1)); plugin.onUserFunctionEnd(new UserFunctionEndInfo( @@ -809,7 +828,7 @@ void userFunctionSuccess_setsOkOnAttemptSpan() { @Test void userFunctionIncomplete_leavesAttemptSpanUnset() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "waiting", "STEP", "Step", null, Instant.now(), false, 1)); plugin.onUserFunctionEnd(new UserFunctionEndInfo( @@ -834,7 +853,7 @@ void userFunctionIncomplete_leavesAttemptSpanUnset() { @Test void operationSuccess_setsOkOnOperationSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "step-ok", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onOperationEnd(new OperationEndInfo( @@ -861,7 +880,7 @@ void operationEnd_withNonSuccessStatusAndNoError_leavesOperationSpanUnset() { // onOperationEnd fires for every terminal status. A CANCELLED operation (or an error-less // FAILED/TIMED_OUT/STOPPED) carries a non-null, non-SUCCEEDED status with a null error. It must NOT be // stamped OK — the span status stays UNSET. - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-cancel", "step-cancel", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onOperationEnd(new OperationEndInfo( @@ -887,7 +906,7 @@ void operationEnd_withNonSuccessStatusAndNoError_leavesOperationSpanUnset() { void operationEnd_withoutStart_nonSuccessStatusAndNoError_leavesContinuationSpanUnset() { // Same guard on the continuation-span branch (operation completed between invocations): an error-less // TIMED_OUT terminal status must NOT be stamped OK. - plugin.onInvocationStart(new InvocationInfo("req-2", ARN, false, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-2", ARN, false, Instant.now())); plugin.onOperationEnd(new OperationEndInfo( "op-cb-timeout", "my-callback", @@ -911,7 +930,7 @@ void operationEnd_withoutStart_nonSuccessStatusAndNoError_leavesContinuationSpan void operationEnd_withNullStatusAndNoError_setsOkOnOperationSpan() { // A successful statusless virtual (FLAT CONTEXT) operation fires onOperationEnd with a null operation -> // null status and null error. This is genuine success and must be stamped OK. - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-ctx", "my-ctx", "CONTEXT", null, null, Instant.now(), null, null, false)); plugin.onOperationEnd(new OperationEndInfo( @@ -935,7 +954,7 @@ void operationEnd_withNullStatusAndNoError_setsOkOnOperationSpan() { @Test void operationNotCompleted_notEndedAtInvocationEnd() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "my-wait", "WAIT", "Wait", null, Instant.now(), null, null, false)); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.PENDING, null)); @@ -955,7 +974,7 @@ void operationNotCompleted_notEndedAtInvocationEnd() { void openAttemptSpan_isEndedAtInvocationEnd_notAbandoned() { // A user function that starts but never ends (e.g. the execution suspends mid-attempt) must not leave a // recording span abandoned: onInvocationEnd force-ends it so it is exported. - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "stuck", "STEP", "Step", null, Instant.now(), false, 1)); // No onUserFunctionEnd — the invocation suspends. @@ -971,7 +990,7 @@ void openAttemptSpan_isEndedAtInvocationEnd_notAbandoned() { @Test void everyRecordingSpanIsEnded_onNonTerminalInvocation() { // No recording span may be left un-ended when the execution returns a non-terminal status. - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onUserFunctionStart( @@ -1006,7 +1025,7 @@ void noRecordingSpanIsLeftOpen_onRetrying_trackedByLifecycleProcessor() { */ private void assertNoOpenSpansOnNonTerminal(InvocationStatus status) { var lifecycle = new LifecycleTrackingSpanProcessor(); - var trackingPlugin = new ExecutionOtelPlugin( + var trackingPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(lifecycle), OtelPluginConfig.builder() .contextExtractor(() -> null) @@ -1014,7 +1033,7 @@ private void assertNoOpenSpansOnNonTerminal(InvocationStatus status) { .workflowSpanName("Workflow") .build()); - trackingPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var trackingPlugin = started(trackingPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); trackingPlugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); trackingPlugin.onUserFunctionStart( @@ -1037,7 +1056,7 @@ private void assertNoOpenSpansOnNonTerminal(InvocationStatus status) { @Test void operationOpenedThenCompletedNextInvocation_exportedOnceOnOperationEnd() { // Invocation 1: operation opens but does not complete. - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "my-wait", "WAIT", "Wait", null, Instant.now(), null, null, false)); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.PENDING, null)); @@ -1048,7 +1067,7 @@ void operationOpenedThenCompletedNextInvocation_exportedOnceOnOperationEnd() { spanExporter.reset(); // Invocation 2: the operation completes → materialized once via onOperationEnd, linked to this invocation. - plugin.onInvocationStart(new InvocationInfo("req-2", ARN, false, Instant.now())); + plugin = started(factory, new InvocationInfo("req-2", ARN, false, Instant.now())); plugin.onOperationEnd(new OperationEndInfo( "op-1", "my-wait", @@ -1080,7 +1099,7 @@ void operationOpenedThenCompletedNextInvocation_exportedOnceOnOperationEnd() { @Test void executionTraceIsStableAcrossInvocations_andSharedByInvocationSpans() { var executionStartTime = Instant.parse("2026-08-15T00:00:00Z"); - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, executionStartTime)); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, executionStartTime)); plugin.onOperationStart( new OperationInfo("op-1", "step-1", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onOperationEnd(new OperationEndInfo( @@ -1102,7 +1121,7 @@ void executionTraceIsStableAcrossInvocations_andSharedByInvocationSpans() { var firstInvocationTraceId = spanByName(firstSpans, "Invocation").getTraceId(); spanExporter.reset(); - plugin.onInvocationStart(new InvocationInfo("req-2", ARN, false, executionStartTime)); + plugin = started(factory, new InvocationInfo("req-2", ARN, false, executionStartTime)); plugin.onInvocationEnd(new InvocationEndInfo("req-2", ARN, false, InvocationStatus.SUCCEEDED, null)); var secondSpans = spanExporter.getFinishedSpanItems(); var workflowSpan = spanByName(secondSpans, "Workflow"); @@ -1116,7 +1135,7 @@ void executionTraceIsStableAcrossInvocations_andSharedByInvocationSpans() { @Test void operationEnd_withoutStart_createsContinuationSpanWithLink() { - plugin.onInvocationStart(new InvocationInfo("req-2", ARN, false, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-2", ARN, false, Instant.now())); // Operation completed between invocations — no matching onOperationStart in this invocation. plugin.onOperationEnd(new OperationEndInfo( "op-wait-1", @@ -1145,7 +1164,7 @@ void operationEnd_withoutStart_createsContinuationSpanWithLink() { @Test void deterministicWorkflowSpanId_stableAcrossInvocations() { - plugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", ARN, true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); var firstWorkflowSpanId = spanByName(spanExporter.getFinishedSpanItems(), "Workflow").getSpanId(); @@ -1153,14 +1172,14 @@ void deterministicWorkflowSpanId_stableAcrossInvocations() { // A second (independent) plugin for the same execution ARN must derive the same Workflow span ID. var exporter2 = InMemorySpanExporter.create(); - var plugin2 = new ExecutionOtelPlugin( + var plugin2Factory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter2)), OtelPluginConfig.builder() .contextExtractor(() -> null) .enableMdc(false) .workflowSpanName("Workflow") .build()); - plugin2.onInvocationStart(new InvocationInfo("req-9", ARN, true, Instant.now())); + var plugin2 = started(plugin2Factory, new InvocationInfo("req-9", ARN, true, Instant.now())); plugin2.onInvocationEnd(new InvocationEndInfo("req-9", ARN, true, InvocationStatus.SUCCEEDED, null)); var secondWorkflowSpanId = spanByName(exporter2.getFinishedSpanItems(), "Workflow").getSpanId(); @@ -1176,7 +1195,7 @@ void deterministicWorkflowSpanId_stableAcrossInvocations() { @Test void sampling_disabled_producesNoSpans() { var exporter = InMemorySpanExporter.create(); - var sampledPlugin = new ExecutionOtelPlugin( + var sampledPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder() .setSampler(io.opentelemetry.sdk.trace.samplers.Sampler.alwaysOff()) .addSpanProcessor(SimpleSpanProcessor.create(exporter)), @@ -1185,7 +1204,7 @@ void sampling_disabled_producesNoSpans() { .enableMdc(false) .workflowSpanName("Workflow") .build()); - sampledPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var sampledPlugin = started(sampledPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); sampledPlugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); assertTrue(exporter.getFinishedSpanItems().isEmpty(), "No spans should be exported with 0% sampling"); } @@ -1200,7 +1219,7 @@ void xrayExtraction_undecidedSampling_remoteParentIsAncestor_flagUnset() { // Two-arg context → UNDECIDED sampling: the valid remote parent is still the authoritative ancestor. A // non-parent-based alwaysOn sampler exports the spans so the topology is observable (a plain parent-based // sampler would drop them, since the remote parent's sampled flag is left unset). - var xrayPlugin = new ExecutionOtelPlugin( + var xrayPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder() .setSampler(Sampler.alwaysOn()) .addSpanProcessor(SimpleSpanProcessor.create(exporter)), @@ -1209,7 +1228,7 @@ void xrayExtraction_undecidedSampling_remoteParentIsAncestor_flagUnset() { .enableMdc(false) .workflowSpanName("Workflow") .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); xrayPlugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); xrayPlugin.onOperationEnd(new OperationEndInfo( @@ -1249,7 +1268,7 @@ void xrayExtraction_undecidedSampling_parentBasedSampler_defersToSamplerAndExpor var xrayTraceId = "aabbccddee112233445566778899aabb"; var parentSpanId = "53995c3f42cd8ad8"; var exporter = InMemorySpanExporter.create(); - var xrayPlugin = new ExecutionOtelPlugin( + var xrayPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder() .setSampler(Sampler.parentBased(Sampler.alwaysOn())) .addSpanProcessor(SimpleSpanProcessor.create(exporter)), @@ -1258,7 +1277,7 @@ void xrayExtraction_undecidedSampling_parentBasedSampler_defersToSamplerAndExpor .enableMdc(false) .workflowSpanName("Workflow") .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); xrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); var spans = exporter.getFinishedSpanItems(); @@ -1279,7 +1298,7 @@ void xrayExtraction_undecidedSampling_parentBasedNeverSampler_dropsExecutionTrac var xrayTraceId = "aabbccddee112233445566778899aabb"; var parentSpanId = "53995c3f42cd8ad8"; var exporter = InMemorySpanExporter.create(); - var xrayPlugin = new ExecutionOtelPlugin( + var xrayPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder() .setSampler(Sampler.parentBased(Sampler.alwaysOff())) .addSpanProcessor(SimpleSpanProcessor.create(exporter)), @@ -1288,7 +1307,7 @@ void xrayExtraction_undecidedSampling_parentBasedNeverSampler_dropsExecutionTrac .enableMdc(false) .workflowSpanName("Workflow") .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); xrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); assertTrue( @@ -1302,7 +1321,7 @@ void xrayExtraction_explicitSampled_remoteParentIsExecutionAncestor() { var parentSpanId = "53995c3f42cd8ad8"; var exporter = InMemorySpanExporter.create(); // Explicit Sampled=1 with a complete parent → the remote context is the execution ancestor directly. - var xrayPlugin = new ExecutionOtelPlugin( + var xrayPluginFactory = ExecutionOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)), OtelPluginConfig.builder() .contextExtractor(() -> @@ -1311,7 +1330,7 @@ void xrayExtraction_explicitSampled_remoteParentIsExecutionAncestor() { .workflowSpanName("Workflow") .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", ARN, true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", ARN, true, Instant.now())); xrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", ARN, true, InvocationStatus.SUCCEEDED, null)); var spans = exporter.getFinishedSpanItems(); diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginIntegrationTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginIntegrationTest.java index f9ff81d20..fe54f3499 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginIntegrationTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginIntegrationTest.java @@ -45,14 +45,15 @@ void setUp() { OtelPluginAutoConfigurationState.resetInstalledForTest(); spanExporter = InMemorySpanExporter.create(); - var plugin = new InvocationOtelPlugin( + // One factory for the environment; the SDK creates one plugin instance per invocation from it. + var factory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) .enableMdc(false) .build()); - otelConfig = DurableConfig.builder().withPlugins(plugin).build(); + otelConfig = DurableConfig.builder().withPlugins(factory).build(); } @AfterEach @@ -337,7 +338,7 @@ void failedStep_producesErrorSpan() { void sampling_off_producesNoSpans() { var sampledExporter = InMemorySpanExporter.create(); - var noSamplePlugin = new InvocationOtelPlugin( + var noSampleFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder() .setSampler(Sampler.alwaysOff()) .addSpanProcessor(SimpleSpanProcessor.create(sampledExporter)), @@ -346,7 +347,8 @@ void sampling_off_producesNoSpans() { .enableMdc(false) .build()); - var noSampleConfig = DurableConfig.builder().withPlugins(noSamplePlugin).build(); + var noSampleConfig = + DurableConfig.builder().withPlugins(noSampleFactory).build(); var runner = LocalDurableTestRunner.create( String.class, (input, ctx) -> ctx.step("step", String.class, stepCtx -> "result"), noSampleConfig); @@ -545,8 +547,8 @@ void waitForCondition_producesSpansWithAttempts() { } @Test - void defaultConstructor_lateBindsGlobalSdkTracerProviderAtInvocationStart() { - var defaultPlugin = new InvocationOtelPlugin(); + void agentPathFactory_bindsGlobalSdkTracerProviderWhenTheInvocationsInstanceIsCreated() { + var defaultFactory = InvocationOtelPlugin.factory(); assertFalse(GlobalOpenTelemetry.isSet()); OtelPluginAutoConfigurationState.markInstalled(); @@ -556,7 +558,7 @@ void defaultConstructor_lateBindsGlobalSdkTracerProviderAtInvocationStart() { .build(); OpenTelemetrySdk.builder().setTracerProvider(globalTracerProvider).buildAndRegisterGlobal(); - var defaultConfig = DurableConfig.builder().withPlugins(defaultPlugin).build(); + var defaultConfig = DurableConfig.builder().withPlugins(defaultFactory).build(); var runner = LocalDurableTestRunner.create( String.class, (input, ctx) -> ctx.step("global-step", String.class, stepCtx -> "Hello " + input), @@ -573,7 +575,7 @@ void defaultConstructor_lateBindsGlobalSdkTracerProviderAtInvocationStart() { } @Test - void defaultConstructor_usesJavaAgentGlobalTracerProviderDirectly_withSeparateAutoConfiguredIdGenerator() { + void agentPathFactory_usesJavaAgentGlobalTracerProviderDirectly_withSeparateAutoConfiguredIdGenerator() { OtelPluginAutoConfigurationState.markInstalled(); GlobalOpenTelemetry.resetForTest(); var globalExporter = InMemorySpanExporter.create(); @@ -595,8 +597,9 @@ public ContextPropagators getPropagators() { } }); - var defaultConfig = - DurableConfig.builder().withPlugins(new InvocationOtelPlugin()).build(); + var defaultConfig = DurableConfig.builder() + .withPlugins(InvocationOtelPlugin.factory()) + .build(); var runner = LocalDurableTestRunner.create( String.class, (input, ctx) -> ctx.step("javaagent-step", String.class, stepCtx -> "Hello " + input), diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginTest.java index d8d7e7f43..7a91b14ae 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginTest.java @@ -7,6 +7,7 @@ import static org.mockito.Mockito.mock; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; +import static software.amazon.lambda.durable.otel.Invocations.started; import io.opentelemetry.api.GlobalOpenTelemetry; import io.opentelemetry.api.OpenTelemetry; @@ -43,7 +44,9 @@ class InvocationOtelPluginTest { private InMemorySpanExporter spanExporter; - private InvocationOtelPlugin plugin; + + /** The environment's plugin factory; each test creates one instance per invocation from it. */ + private DurableExecutionPluginFactory factory; @BeforeEach void setUp() { @@ -52,7 +55,7 @@ void setUp() { OtelPluginAutoConfigurationState.resetInstalledForTest(); spanExporter = InMemorySpanExporter.create(); - plugin = new InvocationOtelPlugin( + factory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) @@ -69,12 +72,13 @@ void tearDown() { } @Test - void defaultConstructor_retriesGlobalProviderBindingOnNextInvocation() { + void agentPathFactory_bindsGlobalProviderOnALaterInvocationsInstance() { GlobalOpenTelemetry.resetForTest(); OtelPluginAutoConfigurationState.markInstalled(); - var defaultPlugin = new InvocationOtelPlugin(); - defaultPlugin.onInvocationStart(new InvocationInfo("req-disabled", "arn:disabled", true, Instant.now())); + var defaultPluginFactory = InvocationOtelPlugin.factory(); + var defaultPlugin = + started(defaultPluginFactory, new InvocationInfo("req-disabled", "arn:disabled", true, Instant.now())); defaultPlugin.onOperationStart(new OperationInfo( "op-disabled", "disabled-step", "STEP", "Step", null, Instant.now(), null, null, false)); defaultPlugin.onOperationEnd(new OperationEndInfo( @@ -101,7 +105,8 @@ void defaultConstructor_retriesGlobalProviderBindingOnNextInvocation() { .build(); OpenTelemetrySdk.builder().setTracerProvider(globalTracerProvider).buildAndRegisterGlobal(); - defaultPlugin.onInvocationStart(new InvocationInfo("req-enabled", "arn:enabled", true, Instant.now())); + defaultPlugin = + started(defaultPluginFactory, new InvocationInfo("req-enabled", "arn:enabled", true, Instant.now())); defaultPlugin.onOperationStart(new OperationInfo( "op-enabled", "enabled-step", "STEP", "Step", null, Instant.now(), null, null, false)); defaultPlugin.onOperationEnd(new OperationEndInfo( @@ -127,8 +132,8 @@ void defaultConstructor_retriesGlobalProviderBindingOnNextInvocation() { } @Test - void defaultConstructor_usesGlobalSdkTracerProviderDirectly() { - var defaultPlugin = new InvocationOtelPlugin(); + void agentPathFactory_usesGlobalSdkTracerProviderDirectly() { + var defaultPluginFactory = InvocationOtelPlugin.factory(); assertFalse(GlobalOpenTelemetry.isSet()); OtelPluginAutoConfigurationState.markInstalled(); @@ -138,7 +143,8 @@ void defaultConstructor_usesGlobalSdkTracerProviderDirectly() { .build(); OpenTelemetrySdk.builder().setTracerProvider(globalTracerProvider).buildAndRegisterGlobal(); - defaultPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var defaultPlugin = + started(defaultPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); defaultPlugin.onOperationStart( new OperationInfo("op-1", "step", "STEP", "Step", null, Instant.now(), null, null, false)); defaultPlugin.onOperationEnd(new OperationEndInfo( @@ -164,7 +170,7 @@ void defaultConstructor_usesGlobalSdkTracerProviderDirectly() { } @Test - void defaultConstructor_usesJavaAgentGlobalTracerProviderDirectly_withSeparateAutoConfiguredIdGenerator() { + void agentPathFactory_usesJavaAgentGlobalTracerProviderDirectly_withSeparateAutoConfiguredIdGenerator() { OtelPluginAutoConfigurationState.markInstalled(); GlobalOpenTelemetry.resetForTest(); var globalExporter = InMemorySpanExporter.create(); @@ -186,8 +192,9 @@ public ContextPropagators getPropagators() { } }); - var defaultPlugin = new InvocationOtelPlugin(); - defaultPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var defaultPluginFactory = InvocationOtelPlugin.factory(); + var defaultPlugin = + started(defaultPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); defaultPlugin.onOperationStart( new OperationInfo("op-1", "step", "STEP", "Step", null, Instant.now(), null, null, false)); defaultPlugin.onOperationEnd(new OperationEndInfo( @@ -291,8 +298,14 @@ void invocationOtelPluginProvider_isRegisteredAsServiceProvider() { .get(); assertEquals("otel-invocation", provider.getName()); - assertEquals(DurableExecutionPluginProvider.API_VERSION, provider.getApiVersion()); - assertEquals(InvocationOtelPlugin.class, provider.getPluginType()); + + // The provider is the per-invocation factory: it creates an InvocationOtelPlugin for the invocation it is + // handed, and a distinct instance for the next one. + var first = provider.createPlugin(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var second = provider.createPlugin(new InvocationInfo("req-2", "arn:exec1", false, Instant.now())); + assertInstanceOf(InvocationOtelPlugin.class, first); + assertInstanceOf(InvocationOtelPlugin.class, second); + assertNotSame(first, second, "Each invocation gets its own plugin instance"); } @Test @@ -306,8 +319,11 @@ void invocationStart_staysOnExecutionTrace_withoutLinkingAmbientSpan() { var ambientSpanContext = SpanContext.create(ambientTraceId, ambientSpanId, TraceFlags.getSampled(), TraceState.getDefault()); + // The instance is created inside the ambient scope because the invocation's parent resolution happens when the + // factory creates it, not later. + DurableExecutionPlugin plugin; try (var ignored = Span.wrap(ambientSpanContext).makeCurrent()) { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); } plugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); @@ -320,8 +336,13 @@ void invocationStart_staysOnExecutionTrace_withoutLinkingAmbientSpan() { @Test void invocationStart_and_end_createsSpan() { - plugin.onInvocationStart(new InvocationInfo( - "req-123", "arn:aws:lambda:us-east-1:123:function:test:$LATEST/durable/exec1", true, Instant.now())); + var plugin = started( + factory, + new InvocationInfo( + "req-123", + "arn:aws:lambda:us-east-1:123:function:test:$LATEST/durable/exec1", + true, + Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo( "req-123", "arn:aws:lambda:us-east-1:123:function:test:$LATEST/durable/exec1", @@ -340,7 +361,7 @@ void invocationStart_and_end_createsSpan() { @Test void customInstrumentationName_isUsedForTracerScope() { var exporter = InMemorySpanExporter.create(); - var customPlugin = new InvocationOtelPlugin( + var customPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) @@ -348,7 +369,7 @@ void customInstrumentationName_isUsedForTracerScope() { .workflowSpanName("Workflow") .instrumentationName("my-custom-scope") .build()); - customPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var customPlugin = started(customPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); customPlugin.onInvocationEnd( new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); @@ -361,11 +382,14 @@ void customInstrumentationName_isUsedForTracerScope() { @Test void explicitProvider_unrelatedRootSpansKeepFreshTraceIds() { + // This invocation's instance and the unrelated library share the one provider the factory built. + var info = new InvocationInfo("req-1", "arn:exec1", true, Instant.now()); + var plugin = (InvocationOtelPlugin) factory.createPlugin(info); var provider = sdkTracerProvider(plugin); var unrelatedTracer = provider.get("unrelated-library"); var before = unrelatedTracer.spanBuilder("before").setNoParent().startSpan(); - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + plugin.onInvocationStart(info); var during = unrelatedTracer.spanBuilder("during").setNoParent().startSpan(); plugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); var after = unrelatedTracer.spanBuilder("after").setNoParent().startSpan(); @@ -394,8 +418,8 @@ void globalProvider_unrelatedRootSpansKeepFreshTraceIds() { var unrelatedTracer = provider.get("unrelated-library"); var before = unrelatedTracer.spanBuilder("before").setNoParent().startSpan(); - var globalPlugin = new InvocationOtelPlugin(); - globalPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var globalPluginFactory = InvocationOtelPlugin.factory(); + var globalPlugin = started(globalPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); var during = unrelatedTracer.spanBuilder("during").setNoParent().startSpan(); globalPlugin.onInvocationEnd( new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); @@ -415,7 +439,7 @@ void globalProvider_unrelatedRootSpansKeepFreshTraceIds() { @Test void invocationSpan_hasInternalKind() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); var span = spanExporter.getFinishedSpanItems().get(0); @@ -424,7 +448,7 @@ void invocationSpan_hasInternalKind() { @Test void operationSpanName_usesOperationName_withoutPrefix() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "create-greeting", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onOperationEnd(new OperationEndInfo( @@ -454,7 +478,7 @@ void operationSpanName_usesOperationName_withoutPrefix() { @Test void attemptSpanName_usesOperationNameWithAttemptNumber() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "process-order", "STEP", "Step", null, Instant.now(), false, 1)); plugin.onUserFunctionEnd(new UserFunctionEndInfo( @@ -483,7 +507,7 @@ void attemptSpanName_usesOperationNameWithAttemptNumber() { @Test void operationEnd_withAttempt_stampsAttemptNumberOnOperationSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "flaky", "STEP", "Step", null, Instant.now(), null, null, false)); @@ -514,7 +538,7 @@ void operationEnd_withAttempt_stampsAttemptNumberOnOperationSpan() { @Test void attemptSpan_carriesOperationSubtype() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "process-order", "STEP", "Step", null, Instant.now(), false, 1)); plugin.onUserFunctionEnd(new UserFunctionEndInfo( @@ -543,7 +567,7 @@ void attemptSpan_carriesOperationSubtype() { @Test void operationEnd_withoutMatchingStart_stampsAttemptNumberOnContinuationSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); // No onOperationStart in this invocation → onOperationEnd takes the continuation-span branch. plugin.onOperationEnd(new OperationEndInfo( @@ -577,7 +601,7 @@ void operationEnd_withoutMatchingStart_stampsAttemptNumberOnContinuationSpan() { @Test void invocationEnd_withFailure_setsErrorStatus() { - plugin.onInvocationStart(new InvocationInfo("req-123", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-123", "arn:exec1", true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo( "req-123", "arn:exec1", true, InvocationStatus.FAILED, new RuntimeException("boom"))); @@ -588,7 +612,7 @@ void invocationEnd_withFailure_setsErrorStatus() { @Test void invocationEnd_withRetrying_leavesStatusUnset() { - plugin.onInvocationStart(new InvocationInfo("req-123", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-123", "arn:exec1", true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo( "req-123", "arn:exec1", true, InvocationStatus.RETRYING, new RuntimeException("transient"))); @@ -602,7 +626,7 @@ void invocationEnd_withRetrying_leavesStatusUnset() { @Test void operationStart_createsSpan_operationEnd_endsIt() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); var start = Instant.parse("2026-06-01T10:00:00Z"); var end = Instant.parse("2026-06-01T10:00:05Z"); @@ -629,7 +653,7 @@ void operationStart_createsSpan_operationEnd_endsIt() { @Test void userFunctionStart_and_end_createsAttemptSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "compute", "STEP", "Step", null, Instant.now(), false, 1)); @@ -662,7 +686,7 @@ void userFunctionStart_and_end_createsAttemptSpan() { @Test void userFunctionEnd_withFailure_setsErrorOnAttemptSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "failing", "STEP", "Step", null, Instant.now(), false, 1)); @@ -691,7 +715,7 @@ void userFunctionEnd_withFailure_setsErrorOnAttemptSpan() { @Test void userFunctionEnd_withSuccess_setsOkOnAttemptSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "compute", "STEP", "Step", null, Instant.now(), false, 1)); @@ -719,7 +743,7 @@ void userFunctionEnd_withSuccess_setsOkOnAttemptSpan() { @Test void userFunctionEnd_withIncomplete_leavesAttemptSpanUnset() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "waiting", "STEP", "Step", null, Instant.now(), false, 1)); @@ -749,7 +773,7 @@ void userFunctionEnd_withIncomplete_leavesAttemptSpanUnset() { @Test void operationEnd_withSuccess_setsOkOnOperationSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "step-ok", "STEP", "Step", null, Instant.now(), null, null, false)); @@ -783,7 +807,7 @@ void operationEnd_withNonSuccessStatusAndNoError_leavesOperationSpanUnset() { // onOperationEnd fires for every terminal status. A CANCELLED operation (or an error-less // FAILED/TIMED_OUT/STOPPED) carries a non-null, non-SUCCEEDED status with a null error. It must NOT be // stamped OK — the span status stays UNSET. - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-cancel", "step-cancel", "STEP", "Step", null, Instant.now(), null, null, false)); @@ -814,7 +838,7 @@ void operationEnd_withNonSuccessStatusAndNoError_leavesOperationSpanUnset() { void operationEnd_withoutMatchingStart_nonSuccessStatusAndNoError_leavesContinuationSpanUnset() { // Same guard on the continuation-span branch (operation completed between invocations): an error-less // TIMED_OUT terminal status must NOT be stamped OK. - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onOperationEnd(new OperationEndInfo( "op-cb-timeout", @@ -843,7 +867,7 @@ void operationEnd_withoutMatchingStart_nonSuccessStatusAndNoError_leavesContinua void operationEnd_withNullStatusAndNoError_setsOkOnOperationSpan() { // A successful statusless virtual (FLAT CONTEXT) operation fires onOperationEnd with a null operation -> // null status and null error. This is genuine success and must be stamped OK. - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-ctx", "my-ctx", "CONTEXT", null, null, Instant.now(), null, null, false)); @@ -873,7 +897,7 @@ void operationEnd_withNullStatusAndNoError_setsOkOnOperationSpan() { @Test void fullLifecycle_producesCorrectSpanHierarchy() { var arn = "arn:aws:lambda:us-east-1:123:function:test:$LATEST/durable/exec1"; - plugin.onInvocationStart(new InvocationInfo("req-1", arn, true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", arn, true, Instant.now())); // Step 1: operation starts, user function runs, operation completes plugin.onOperationStart( @@ -956,14 +980,14 @@ void invocationRoots_sameExecutionShareExecutionTrace() { // Same execution start time across invocations so the ARN-derived canonical trace ID is reproducible. var startTime = Instant.now(); - plugin.onInvocationStart(new InvocationInfo("req-1", arn, true, startTime)); + var plugin = started(factory, new InvocationInfo("req-1", arn, true, startTime)); plugin.onInvocationEnd(new InvocationEndInfo("req-1", arn, true, InvocationStatus.PENDING, null)); var firstTraceId = spanByName("Invocation").getTraceId(); spanExporter.reset(); // Second invocation of same execution - plugin.onInvocationStart(new InvocationInfo("req-2", arn, false, startTime)); + plugin = started(factory, new InvocationInfo("req-2", arn, false, startTime)); plugin.onInvocationEnd(new InvocationEndInfo("req-2", arn, false, InvocationStatus.SUCCEEDED, null)); var secondTraceId = spanByName("Invocation").getTraceId(); @@ -977,7 +1001,7 @@ void invocationRoots_sameExecutionShareExecutionTrace() { @Test void operationNotCompleted_spanEndedAtInvocationEnd() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); // Operation starts but never completes (e.g., wait operation, invocation suspends) plugin.onOperationStart( @@ -1000,7 +1024,7 @@ void operationNotCompleted_spanEndedAtInvocationEnd() { @Test void operationStart_withStatus_preservesStatus() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", false, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", false, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "my-step", "STEP", "Step", null, Instant.now(), null, "PENDING", true)); @@ -1017,7 +1041,7 @@ void operationStart_withStatus_preservesStatus() { void invocationEnd_closesNestedSpansChildFirst() { var parentId = "op-parent"; var childId = "op-child"; - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onOperationStart(new OperationInfo( parentId, "parent-context", "CONTEXT", "RunInChildContext", null, Instant.now(), null, null, false)); plugin.onOperationStart( @@ -1048,7 +1072,7 @@ void invocationEnd_closesNestedSpansChildFirst() { @Test void sampling_disabled_producesNoSpans() { spanExporter = InMemorySpanExporter.create(); - var sampledPlugin = new InvocationOtelPlugin( + var sampledPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder() .setSampler(io.opentelemetry.sdk.trace.samplers.Sampler.alwaysOff()) .addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), @@ -1057,7 +1081,8 @@ void sampling_disabled_producesNoSpans() { .enableMdc(false) .build()); - sampledPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var sampledPlugin = + started(sampledPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); sampledPlugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "step", "STEP", "Step", null, Instant.now(), false, 1)); sampledPlugin.onUserFunctionEnd(new UserFunctionEndInfo( @@ -1099,14 +1124,14 @@ void xrayExtraction_withoutParentDoesNotForceTraceId() { var extractedContext = new ExtractedContext(xrayTraceId, null); spanExporter = InMemorySpanExporter.create(); - var xrayPlugin = new InvocationOtelPlugin( + var xrayPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> extractedContext) .enableMdc(false) .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); xrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); var spans = spanExporter.getFinishedSpanItems(); @@ -1131,14 +1156,14 @@ void xrayExtraction_invocationTreeUsesExtractedTraceId_workflowJoinsExecutionTra var extractedContext = new ExtractedContext(xrayTraceId, parentSpanId); spanExporter = InMemorySpanExporter.create(); - var xrayPlugin = new InvocationOtelPlugin( + var xrayPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> extractedContext) .enableMdc(false) .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); xrayPlugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); xrayPlugin.onUserFunctionStart( @@ -1185,14 +1210,14 @@ void xrayExtraction_withParentSpanId_invocationSpanHasCorrectParent() { var extractedContext = new ExtractedContext(xrayTraceId, parentSpanId, ExtractedContext.Sampling.SAMPLED); spanExporter = InMemorySpanExporter.create(); - var xrayPlugin = new InvocationOtelPlugin( + var xrayPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> extractedContext) .enableMdc(false) .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); xrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); var spans = spanExporter.getFinishedSpanItems(); @@ -1219,14 +1244,14 @@ void xrayExtraction_withoutParentSpanId_invocationSpanParentsOntoSyntheticRoot() var extractedContext = new ExtractedContext(xrayTraceId, null); spanExporter = InMemorySpanExporter.create(); - var xrayPlugin = new InvocationOtelPlugin( + var xrayPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> extractedContext) .enableMdc(false) .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); xrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); var spans = spanExporter.getFinishedSpanItems(); @@ -1248,7 +1273,7 @@ void xrayExtraction_multipleInvocations_sameTraceId_unifiedTrace() { var extractedContext = new ExtractedContext(xrayTraceId, "53995c3f42cd8ad8", ExtractedContext.Sampling.SAMPLED); spanExporter = InMemorySpanExporter.create(); - var xrayPlugin = new InvocationOtelPlugin( + var xrayPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> extractedContext) @@ -1256,7 +1281,7 @@ void xrayExtraction_multipleInvocations_sameTraceId_unifiedTrace() { .build()); // First invocation - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); xrayPlugin.onOperationStart( new OperationInfo("op-1", "step-1", "STEP", "Step", null, Instant.now(), null, null, false)); xrayPlugin.onOperationEnd(new OperationEndInfo( @@ -1275,7 +1300,7 @@ void xrayExtraction_multipleInvocations_sameTraceId_unifiedTrace() { xrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.PENDING, null)); // Second invocation (same execution, same X-Ray Root from backend) - xrayPlugin.onInvocationStart(new InvocationInfo("req-2", "arn:exec1", false, Instant.now())); + xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-2", "arn:exec1", false, Instant.now())); xrayPlugin.onOperationStart( new OperationInfo("op-2", "step-2", "STEP", "Step", null, Instant.now(), null, null, false)); xrayPlugin.onOperationEnd(new OperationEndInfo( @@ -1306,7 +1331,7 @@ void xrayExtraction_multipleInvocations_sameTraceId_unifiedTrace() { @Test void xrayExtraction_nullExtractor_sharesArnDerivedExecutionTrace() { spanExporter = InMemorySpanExporter.create(); - var noXrayPlugin = new InvocationOtelPlugin( + var noXrayPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) @@ -1314,7 +1339,7 @@ void xrayExtraction_nullExtractor_sharesArnDerivedExecutionTrace() { .build()); var arn = "arn:aws:lambda:us-east-1:123:function:test:$LATEST/durable/exec1"; - noXrayPlugin.onInvocationStart(new InvocationInfo("req-1", arn, true, Instant.now())); + var noXrayPlugin = started(noXrayPluginFactory, new InvocationInfo("req-1", arn, true, Instant.now())); noXrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", arn, true, InvocationStatus.SUCCEEDED, null)); var spans = spanExporter.getFinishedSpanItems(); @@ -1346,14 +1371,14 @@ void xrayExtraction_extractedTraceIdMatchesXrayConversion() { // the spans export. var extractedContext = new ExtractedContext(convertedId, "53995c3f42cd8ad8", ExtractedContext.Sampling.SAMPLED); spanExporter = InMemorySpanExporter.create(); - var xrayPlugin = new InvocationOtelPlugin( + var xrayPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> extractedContext) .enableMdc(false) .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); xrayPlugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); assertEquals(expectedOtelTraceId, spanByName("Invocation").getTraceId()); @@ -1365,7 +1390,7 @@ void xrayExtraction_extractedTraceIdMatchesXrayConversion() { @Test void operationEnd_withoutMatchingStart_createsContinuationSpanWithLink() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); // onOperationEnd without a prior onOperationStart — operation completed between invocations plugin.onOperationEnd(new OperationEndInfo( @@ -1397,7 +1422,7 @@ void operationEnd_withoutMatchingStart_createsContinuationSpanWithLink() { @Test void operationEnd_withoutMatchingStart_startsWithinCurrentInvocation() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); var operationStart = Instant.EPOCH; var operationEnd = operationStart.plusSeconds(60); @@ -1436,7 +1461,7 @@ void operationEnd_withoutMatchingStart_startsWithinCurrentInvocation() { @Test void operationEnd_withoutMatchingStart_withError_setsErrorStatus() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onOperationEnd(new OperationEndInfo( "op-cb-1", @@ -1466,7 +1491,7 @@ void operationEnd_withoutMatchingStart_withError_setsErrorStatus() { @Test void contextOperation_doesNotCreateAttemptSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); // Create operation span first so the CONTEXT user function has a parent plugin.onOperationStart(new OperationInfo( @@ -1501,7 +1526,7 @@ void contextOperation_doesNotCreateAttemptSpan() { @Test void attemptSpan_endedAtInvocationEnd_whenUserFunctionEndNotCalled() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); // Start attempt but never call onUserFunctionEnd (simulates crash before end hook) plugin.onUserFunctionStart( @@ -1522,7 +1547,7 @@ void attemptSpan_endedAtInvocationEnd_whenUserFunctionEndNotCalled() { @Test void childOperation_parentedToParentOperationSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); // Parent context operation plugin.onOperationStart(new OperationInfo( @@ -1588,7 +1613,7 @@ void multiInvocation_stepWaitStep_producesCorrectSpans() { var startTime = Instant.now(); // Invocation 1: step completes, wait starts - plugin.onInvocationStart(new InvocationInfo("req-1", arn, true, startTime)); + var plugin = started(factory, new InvocationInfo("req-1", arn, true, startTime)); plugin.onOperationStart( new OperationInfo("op-1", "step-A", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onUserFunctionStart( @@ -1639,7 +1664,7 @@ void multiInvocation_stepWaitStep_producesCorrectSpans() { spanExporter.reset(); // Invocation 2: wait completed between invocations, new step runs - plugin.onInvocationStart(new InvocationInfo("req-2", arn, false, startTime)); + plugin = started(factory, new InvocationInfo("req-2", arn, false, startTime)); plugin.onOperationEnd(new OperationEndInfo( "op-2", "pause", @@ -1741,7 +1766,7 @@ void crossInvocation_stepRetry_attemptsParentedToRespectiveInvocations() { var startTime = Instant.now(); // Invocation 1: step starts, attempt 1 fails, invocation suspended during retry poll - plugin.onInvocationStart(new InvocationInfo("req-1", arn, true, startTime)); + var plugin = started(factory, new InvocationInfo("req-1", arn, true, startTime)); plugin.onOperationStart( new OperationInfo("op-1", "process-payment", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onUserFunctionStart( @@ -1783,7 +1808,7 @@ void crossInvocation_stepRetry_attemptsParentedToRespectiveInvocations() { spanExporter.reset(); // Invocation 2: step is replayed (continuation), attempt 2 executes and succeeds - plugin.onInvocationStart(new InvocationInfo("req-2", arn, false, startTime)); + plugin = started(factory, new InvocationInfo("req-2", arn, false, startTime)); // isReplay=true: this operation already exists in the execution state plugin.onOperationStart( new OperationInfo("op-1", "process-payment", "STEP", "Step", null, Instant.now(), null, null, true)); @@ -1881,7 +1906,7 @@ void crossInvocation_stepRetry_attemptsParentedToRespectiveInvocations() { @Test void workflowSpan_exportedOnTerminal_internal_deterministicId() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec-wf", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec-wf", true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec-wf", true, InvocationStatus.SUCCEEDED, null)); var workflow = spanByName("Workflow"); @@ -1892,7 +1917,7 @@ void workflowSpan_exportedOnTerminal_internal_deterministicId() { @Test void workflowSpan_notExportedOnNonTerminal() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.PENDING, null)); assertTrue( @@ -1904,7 +1929,7 @@ void workflowSpan_notExportedOnNonTerminal() { @Test void workflowSpan_notExportedOnRetrying() { // RETRYING is non-terminal, so the deferred Workflow span is neither materialized nor abandoned. - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo( "req-1", "arn:exec1", true, InvocationStatus.RETRYING, new RuntimeException("transient"))); @@ -1918,7 +1943,7 @@ void workflowSpan_notExportedOnRetrying() { void deferredWorkflowSpan_whenExported_isEnded_andMatchesLinkedSpanId() { // The Workflow span is created only at the terminal invocation, but operations that ran earlier linked to its // deterministic context. When it is finally exported it must be ended and carry that same span ID. - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec-wf", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec-wf", true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onOperationEnd(new OperationEndInfo( @@ -1947,7 +1972,7 @@ void deferredWorkflowSpan_whenExported_isEnded_andMatchesLinkedSpanId() { @Test void operationAndAttemptSpans_linkToWorkflowSpan() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec-wf", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec-wf", true, Instant.now())); plugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); plugin.onUserFunctionStart( @@ -1994,7 +2019,7 @@ void operationAndAttemptSpans_linkToWorkflowSpan() { void operationLinksToWorkflow_withXRayContext() { // "Other case": invocation span is parented to the X-Ray segment, but operation spans still link to Workflow. var exporter = InMemorySpanExporter.create(); - var xrayPlugin = new InvocationOtelPlugin( + var xrayPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)), OtelPluginConfig.builder() .contextExtractor(() -> new ExtractedContext( @@ -2003,7 +2028,7 @@ void operationLinksToWorkflow_withXRayContext() { ExtractedContext.Sampling.SAMPLED)) .enableMdc(false) .build()); - xrayPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var xrayPlugin = started(xrayPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); xrayPlugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); xrayPlugin.onOperationEnd(new OperationEndInfo( @@ -2039,14 +2064,14 @@ void operationLinksToWorkflow_withXRayContext() { @Test void workflowSpanName_isConfigurable() { var exporter = InMemorySpanExporter.create(); - var customPlugin = new InvocationOtelPlugin( + var customPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(exporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) .enableMdc(false) .workflowSpanName("MyWorkflow") .build()); - customPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var customPlugin = started(customPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); customPlugin.onInvocationEnd( new InvocationEndInfo("req-1", "arn:exec1", true, InvocationStatus.SUCCEEDED, null)); @@ -2062,7 +2087,7 @@ void workflowSpanName_isConfigurable() { @Test void failedInvocation_setsErrorOnBothWorkflowAndInvocationSpans() { - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var plugin = started(factory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); plugin.onInvocationEnd(new InvocationEndInfo( "req-1", "arn:exec1", true, InvocationStatus.FAILED, new RuntimeException("boom"))); @@ -2163,14 +2188,15 @@ void noRecordingSpanIsLeftOpen_onRetrying_trackedByLifecycleProcessor() { */ private void assertNoOpenSpansOnNonTerminal(InvocationStatus status) { var lifecycle = new LifecycleTrackingSpanProcessor(); - var trackingPlugin = new InvocationOtelPlugin( + var trackingPluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(lifecycle), OtelPluginConfig.builder() .contextExtractor(() -> null) .enableMdc(false) .build()); - trackingPlugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); + var trackingPlugin = + started(trackingPluginFactory, new InvocationInfo("req-1", "arn:exec1", true, Instant.now())); trackingPlugin.onOperationStart( new OperationInfo("op-1", "step-a", "STEP", "Step", null, Instant.now(), null, null, false)); trackingPlugin.onUserFunctionStart( diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/Invocations.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/Invocations.java new file mode 100644 index 000000000..ed9a21e33 --- /dev/null +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/Invocations.java @@ -0,0 +1,27 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.otel; + +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; +import software.amazon.lambda.durable.plugin.InvocationInfo; + +/** + * Test helper: builds one invocation's plugin instance the way the SDK does. + * + *

    A plugin instance serves exactly one invocation, so a test that drives several invocations of an execution creates + * one instance per invocation from the same factory — the factory being what the environment owns. The factory is + * called with the very {@link InvocationInfo} that {@code onInvocationStart} then receives, exactly as + * {@code PluginRunner} does. + */ +final class Invocations { + + private Invocations() {} + + /** One invocation's plugin instance, created from the factory and started with the same info. */ + static DurableExecutionPlugin started(DurableExecutionPluginFactory factory, InvocationInfo info) { + var plugin = factory.createPlugin(info); + plugin.onInvocationStart(info); + return plugin; + } +} diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/MdcSpanEnricherTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/MdcSpanEnricherTest.java index 9c4399817..88c9a50c4 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/MdcSpanEnricherTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/MdcSpanEnricherTest.java @@ -3,11 +3,14 @@ package software.amazon.lambda.durable.otel; import static org.junit.jupiter.api.Assertions.*; +import static software.amazon.lambda.durable.otel.Invocations.started; import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter; import io.opentelemetry.sdk.trace.SdkTracerProvider; +import io.opentelemetry.sdk.trace.data.SpanData; import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor; import java.time.Instant; +import java.util.List; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Test; import org.slf4j.MDC; @@ -56,14 +59,14 @@ void inject_withNoActiveSpan_doesNotSetMdcFields() { void plugin_withMdcEnabled_setsFieldsInMdc() { var spanExporter = InMemorySpanExporter.create(); - var plugin = new InvocationOtelPlugin( + var pluginFactory = InvocationOtelPlugin.factory( SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), OtelPluginConfig.builder() .contextExtractor(() -> null) .enableMdc(true) .build()); - plugin.onInvocationStart(new InvocationInfo("req-1", "arn:exec-mdc-test", true, Instant.now())); + var plugin = started(pluginFactory, new InvocationInfo("req-1", "arn:exec-mdc-test", true, Instant.now())); plugin.onUserFunctionStart( new UserFunctionStartInfo("op-1", "step", "STEP", "Step", null, Instant.now(), false, 1)); @@ -99,4 +102,41 @@ void plugin_withMdcEnabled_setsFieldsInMdc() { assertNull(MDC.get(MdcSpanEnricher.MDC_SPAN_ID)); assertNull(MDC.get(MdcSpanEnricher.MDC_TRACE_SAMPLED)); } + + @Test + void logCorrelationFollowsEachInvocationsOwnInstance() { + // Regression guard taken from the Python port of this refactor: there a log filter installed by the first + // invocation's plugin outlived that plugin and kept querying the discarded instance, so log correlation + // silently stopped after the first invocation. Java correlates through the SLF4J MDC, written by the hooks of + // whichever instance is serving the invocation, so every invocation publishes its own execution trace. This + // test pins that down across two invocations served by two instances of one factory. + var spanExporter = InMemorySpanExporter.create(); + var pluginFactory = InvocationOtelPlugin.factory( + SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)), + OtelPluginConfig.builder() + .contextExtractor(() -> null) + .enableMdc(true) + .build()); + + var first = started(pluginFactory, new InvocationInfo("req-1", "arn:exec-a", true, Instant.now())); + var firstTraceId = MDC.get(MdcSpanEnricher.MDC_TRACE_ID); + first.onInvocationEnd(new InvocationEndInfo("req-1", "arn:exec-a", true, InvocationStatus.SUCCEEDED, null)); + assertNull(MDC.get(MdcSpanEnricher.MDC_TRACE_ID), "an invocation clears the correlation it set"); + + var second = started(pluginFactory, new InvocationInfo("req-2", "arn:exec-b", true, Instant.now())); + var secondTraceId = MDC.get(MdcSpanEnricher.MDC_TRACE_ID); + second.onInvocationEnd(new InvocationEndInfo("req-2", "arn:exec-b", true, InvocationStatus.SUCCEEDED, null)); + + assertNotNull(firstTraceId); + assertNotNull(secondTraceId, "log correlation must not stop after the first invocation"); + assertNotEquals(firstTraceId, secondTraceId, "each instance publishes its own execution trace"); + var invocationTraceIds = spanExporter.getFinishedSpanItems().stream() + .filter(span -> span.getName().equals("Invocation")) + .map(SpanData::getTraceId) + .toList(); + assertEquals( + List.of(firstTraceId, secondTraceId), + invocationTraceIds, + "the correlated trace ID is the one on that invocation's own Invocation span"); + } } diff --git a/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java b/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java index f8e3bceb6..01b1540d9 100644 --- a/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java +++ b/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java @@ -39,10 +39,10 @@ void pluginsFromConfigurationAndEnvironment_receiveLifecycleEvents() { var configuredPlugin = new RecordingPlugin(); var dynamicPlugin = new RecordingPlugin(); var provider = new RecordingPluginProvider(dynamicPlugin); - var plugins = - DynamicPluginLoader.loadConfiguredPlugins("recording", List.of(provider), List.of(configuredPlugin)); + var factories = DynamicPluginLoader.loadConfiguredPluginFactories( + "recording", List.of(provider), List.of(info -> configuredPlugin)); var config = DurableConfig.builder() - .withPlugins(plugins.toArray(DurableExecutionPlugin[]::new)) + .withPlugins(factories.toArray(DurableExecutionPluginFactory[]::new)) .build(); var runner = LocalDurableTestRunner.create( @@ -60,7 +60,7 @@ void pluginsFromConfigurationAndEnvironment_receiveLifecycleEvents() { @Test void plugin_receivesInvocationStartAndEnd_onSuccessfulExecution() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -84,7 +84,7 @@ void plugin_receivesInvocationStartAndEnd_onSuccessfulExecution() { @Test void plugin_receivesInvocationEnd_withPendingStatus_onSuspension() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -106,7 +106,7 @@ void plugin_receivesInvocationEnd_withPendingStatus_onSuspension() { @Test void plugin_invocationSnapshots_trackReplayAcrossSuspension() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -168,7 +168,7 @@ void plugin_invocationSnapshots_trackReplayAcrossSuspension() { @Test void plugin_receivesInvocationEnd_withFailedStatus_onError() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -195,7 +195,7 @@ void plugin_receivesInvocationEnd_withFailedStatus_onError() { @Test void plugin_invocationHooks_carryExecutionInputAndResult_onSuccess() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -217,7 +217,7 @@ void plugin_invocationHooks_carryExecutionInputAndResult_onSuccess() { @Test void plugin_invocationEnd_omitsExecutionResult_onFailure() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -244,7 +244,7 @@ void plugin_invocationEnd_omitsExecutionResult_onFailure() { @Test void plugin_invocationEnd_omitsExecutionResult_onSuspension() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -268,8 +268,10 @@ void plugin_invocationEnd_omitsExecutionResult_onSuspension() { void plugin_executionInput_isDeserializedOnce_andSharedWithHandler() { var serDes = new CountingSerDes(); var plugin = new RecordingPlugin(); - var config = - DurableConfig.builder().withPlugins(plugin).withSerDes(serDes).build(); + var config = DurableConfig.builder() + .withPlugins(info -> plugin) + .withSerDes(serDes) + .build(); var handlerInput = new AtomicReference(); var runner = LocalDurableTestRunner.create( @@ -318,7 +320,7 @@ int inputDeserializations(String value) { void plugin_hooksStayPaired_whenSerDesSneakyThrowsCheckedException() { var plugin = new RecordingPlugin(); var config = DurableConfig.builder() - .withPlugins(plugin) + .withPlugins(info -> plugin) .withSerDes(new SneakyThrowingSerDes()) .build(); @@ -358,7 +360,7 @@ public T deserialize(String data, TypeToken typeToken) { @Test void plugin_receivesOperationStartAndEnd_forStep() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, (input, context) -> context.step("my-step", String.class, stepCtx -> "result"), config); @@ -379,7 +381,7 @@ void plugin_receivesOperationStartAndEnd_forStep() { @Test void plugin_receivesOperationStart_forMultipleSteps() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -402,7 +404,7 @@ void plugin_receivesOperationStart_forMultipleSteps() { @Test void plugin_operationEnd_notFiredOnReplay() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -436,7 +438,7 @@ void plugin_operationEnd_notFiredOnReplay() { @Test void plugin_operationEnd_firedForOperationCompletedDuringSuspension() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -473,7 +475,7 @@ void plugin_operationEnd_firedForOperationCompletedDuringSuspension() { @Test void plugin_operationEnd_firedOnceForStepCompletingInCurrentInvocation() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -496,7 +498,7 @@ void plugin_operationEnd_firedOnceForStepCompletingInCurrentInvocation() { @Test void plugin_operationEnd_includesError_whenInvokeFailsDuringSuspension() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -541,7 +543,7 @@ void plugin_operationEnd_includesError_whenInvokeFailsDuringSuspension() { @Test void plugin_operationEnd_includesError_whenStepFailsViaCheckpoint() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -572,7 +574,7 @@ void plugin_operationEnd_includesError_whenStepFailsViaCheckpoint() { @Test void plugin_operationEnd_noError_whenOperationSucceeds() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, (input, context) -> context.step("ok-step", String.class, stepCtx -> "success"), config); @@ -590,7 +592,7 @@ void plugin_operationEnd_noError_whenOperationSucceeds() { @Test void plugin_operationEnd_includesResult_whenStepSucceeds() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, (input, context) -> context.step("my-step", String.class, stepCtx -> "task-a"), config); @@ -611,7 +613,7 @@ void plugin_operationStartAndEnd_balanced_forEmptyMap() { var plugin = new RecordingPlugin(); // withCheckpointEmptyMap is a temporary flag expected to be removed in a future major version. var config = DurableConfig.builder() - .withPlugins(plugin) + .withPlugins(info -> plugin) .withCheckpointEmptyMap(true) .build(); @@ -643,7 +645,7 @@ void plugin_operationStartAndEnd_balanced_forEmptyMap() { @Test void plugin_receivesOperationChange_forStep() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, (input, context) -> context.step("my-step", String.class, stepCtx -> "result"), config); @@ -666,7 +668,7 @@ void plugin_receivesOperationChange_forStep() { @Test void plugin_operationChange_includesErrorAndStatus_whenStepFails() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -700,7 +702,7 @@ void plugin_operationChange_includesErrorAndStatus_whenStepFails() { @Test void plugin_receivesUserFunctionStartAndEnd_forStep() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, (input, context) -> context.step("compute", String.class, stepCtx -> "42"), config); @@ -720,7 +722,7 @@ void plugin_receivesUserFunctionStartAndEnd_forStep() { @Test void plugin_userFunctionEnd_reportsFailed_whenStepFails() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); // When a step's user function throws, the exception propagates through the user-function hook // boundary, so onUserFunctionEnd reports FAILED with the error. Retry/checkpoint @@ -762,7 +764,7 @@ void plugin_userFunctionEnd_reportsFailed_whenStepFails() { void plugin_userFunctionStart_includesAttemptNumber_forRetries() { var attemptCounter = new AtomicInteger(0); var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -797,7 +799,9 @@ void plugin_userFunctionStart_includesAttemptNumber_forRetries() { void multiplePlugins_allReceiveHooks() { var plugin1 = new RecordingPlugin(); var plugin2 = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin1, plugin2).build(); + var config = DurableConfig.builder() + .withPlugins(info -> plugin1, info -> plugin2) + .build(); var runner = LocalDurableTestRunner.create( String.class, (input, context) -> context.step("step", String.class, stepCtx -> "result"), config); @@ -816,7 +820,7 @@ void throwingPlugin_doesNotDisruptExecution() { var throwingPlugin = new ThrowingPlugin(); var recordingPlugin = new RecordingPlugin(); var config = DurableConfig.builder() - .withPlugins(throwingPlugin, recordingPlugin) + .withPlugins(info -> throwingPlugin, info -> recordingPlugin) .build(); var runner = LocalDurableTestRunner.create( @@ -838,7 +842,7 @@ void throwingPlugin_doesNotDisruptExecution() { @Test void plugin_receivesHooks_forChildContextOperations() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -861,7 +865,7 @@ void plugin_receivesHooks_forChildContextOperations() { void plugin_receivesAttemptNumbers_forWaitForCondition() { var checkCount = new AtomicInteger(0); var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -892,7 +896,7 @@ void plugin_receivesAttemptNumbers_forWaitForCondition() { void plugin_reportsFailedThenSucceededAttempts_forRetriedStep() { var attempts = new AtomicInteger(0); var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -948,7 +952,7 @@ void plugin_reportsFailedThenSucceededAttempts_forRetriedStep() { @Test void plugin_userFunctionEnd_reportsSuspension_asIncomplete() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); // A child context whose body suspends (on a wait) throws SuspendExecutionException through the // user-function boundary, so onUserFunctionEnd fires with INCOMPLETE and the suspend exception. @@ -976,7 +980,7 @@ void plugin_userFunctionEnd_reportsSuspension_asIncomplete() { @Test void plugin_userFunctionEnd_unwrapsCompletionExceptionForSuspension() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var suspension = new SuspendExecutionException(); var runner = LocalDurableTestRunner.create( @@ -1000,7 +1004,7 @@ void plugin_userFunctionEnd_unwrapsCompletionExceptionForSuspension() { @Test void plugin_parallelBranches_emitUserFunctionHooks_butConsumerDoesNot() { var plugin = new RecordingPlugin(); - var config = DurableConfig.builder().withPlugins(plugin).build(); + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, @@ -1037,7 +1041,15 @@ void plugin_parallelBranches_emitUserFunctionHooks_butConsumerDoesNot() { // ─── Test helper classes ───────────────────────────────────────────── - /** Plugin that records all hook invocations for assertions. */ + /** + * Plugin that records all hook invocations for assertions. + * + *

    Registered as {@code withPlugins(info -> plugin)}, so every invocation of a test's execution is handed the + * same recorder. The SDK creates a plugin instance per invocation, and several tests here span two invocations (a + * suspension and its resume, or a retry with a delay); handing all of them one recorder is what lets those tests + * assert on what the whole execution observed, e.g. that {@code step1}'s operation-end fired exactly once across + * both invocations. Production plugins return a fresh instance instead. + */ private static class RecordingPlugin implements DurableExecutionPlugin { final List invocationStarts = Collections.synchronizedList(new ArrayList<>()); final List invocationEnds = Collections.synchronizedList(new ArrayList<>()); @@ -1089,18 +1101,12 @@ public String getName() { return "recording"; } + /** + * Hands every invocation the same recorder so the assertions can read what all of them observed; a real + * provider would build a fresh instance here. + */ @Override - public int getApiVersion() { - return API_VERSION; - } - - @Override - public Class getPluginType() { - return RecordingPlugin.class; - } - - @Override - public DurableExecutionPlugin createPlugin() { + public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { return plugin; } } diff --git a/sdk-testing/src/main/java/software/amazon/lambda/durable/testing/LocalDurableTestRunner.java b/sdk-testing/src/main/java/software/amazon/lambda/durable/testing/LocalDurableTestRunner.java index 06d59d5d3..012b743ae 100644 --- a/sdk-testing/src/main/java/software/amazon/lambda/durable/testing/LocalDurableTestRunner.java +++ b/sdk-testing/src/main/java/software/amazon/lambda/durable/testing/LocalDurableTestRunner.java @@ -21,7 +21,7 @@ import software.amazon.lambda.durable.execution.DurableExecutor; import software.amazon.lambda.durable.model.DurableExecutionInput; import software.amazon.lambda.durable.model.ExecutionStatus; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.serde.SerDes; import software.amazon.lambda.durable.testing.local.LocalMemoryExecutionClient; import software.amazon.lambda.durable.testing.local.OperationResult; @@ -70,7 +70,7 @@ private LocalDurableTestRunner( .withLoggerConfig(customerConfig.getLoggerConfig()) // Temporary: remove along with the checkpointEmptyMap flag in a future major version. .withCheckpointEmptyMap(customerConfig.shouldCheckpointEmptyMap()) - .withPlugins(customerConfig.getPluginRunner().getPlugins().toArray(new DurableExecutionPlugin[0])) + .withPlugins(customerConfig.getPluginFactories().toArray(new DurableExecutionPluginFactory[0])) .build(); } else { // Fallback to default config with in-memory client diff --git a/sdk-testing/src/test/java/software/amazon/lambda/durable/testing/LocalDurableTestRunnerTest.java b/sdk-testing/src/test/java/software/amazon/lambda/durable/testing/LocalDurableTestRunnerTest.java index 36f1bbced..23a30a141 100644 --- a/sdk-testing/src/test/java/software/amazon/lambda/durable/testing/LocalDurableTestRunnerTest.java +++ b/sdk-testing/src/test/java/software/amazon/lambda/durable/testing/LocalDurableTestRunnerTest.java @@ -95,7 +95,8 @@ public void onInvocationStart(InvocationInfo info) { executionStartTimes.add(info.executionStartTime()); } }; - var config = DurableConfig.builder().withPlugins(plugin).build(); + // One instance for both invocations, so the assertion below still compares what two invocations observed. + var config = DurableConfig.builder().withPlugins(info -> plugin).build(); var runner = LocalDurableTestRunner.create( String.class, (input, context) -> { diff --git a/sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java b/sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java index 5101b9fda..582fdbd06 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java @@ -23,8 +23,7 @@ import software.amazon.lambda.durable.client.DurableExecutionClient; import software.amazon.lambda.durable.client.LambdaDurableFunctionsClient; import software.amazon.lambda.durable.logging.LoggerConfig; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; -import software.amazon.lambda.durable.plugin.PluginRunner; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.retry.PollingStrategies; import software.amazon.lambda.durable.retry.PollingStrategy; import software.amazon.lambda.durable.serde.JacksonSerDes; @@ -100,10 +99,10 @@ public final class DurableConfig { private final Duration checkpointDelay; private final boolean deserializeAfterSerialization; private final boolean checkpointEmptyMap; - private final PluginRunner pluginRunner; + private final List pluginFactories; private DurableConfig(Builder builder) { - var plugins = DynamicPluginLoader.loadConfiguredPlugins(builder.plugins); + this.pluginFactories = DynamicPluginLoader.loadConfiguredPluginFactories(builder.pluginFactories); this.durableExecutionClient = Objects.requireNonNullElseGet( builder.durableExecutionClient, DurableConfig::createDefaultDurableExecutionClient); this.serDes = Objects.requireNonNullElseGet(builder.serDes, JacksonSerDes::new); @@ -114,7 +113,6 @@ private DurableConfig(Builder builder) { this.checkpointDelay = Objects.requireNonNullElseGet(builder.checkpointDelay, () -> Duration.ofSeconds(0)); this.deserializeAfterSerialization = builder.deserializeAfterSerialization; this.checkpointEmptyMap = builder.checkpointEmptyMap; - this.pluginRunner = plugins.isEmpty() ? PluginRunner.noOp() : new PluginRunner(plugins); validateConfiguration(); } @@ -215,14 +213,15 @@ public boolean shouldCheckpointEmptyMap() { } /** - * Gets the plugin runner that dispatches lifecycle events to registered plugins. + * Gets the plugin factories registered via the builder or loaded dynamically, in dispatch order. * - *

    Returns a no-op runner if no plugins were registered via the builder or loaded dynamically. + *

    Each factory is called once per Lambda invocation to create that invocation's plugin instance; the SDK never + * shares a plugin instance across invocations. * - * @return PluginRunner instance (never null) + * @return immutable list of plugin factories (never null, possibly empty) */ - public PluginRunner getPluginRunner() { - return pluginRunner; + public List getPluginFactories() { + return pluginFactories; } public void validateConfiguration() { @@ -321,7 +320,7 @@ public static final class Builder { private Duration checkpointDelay; private boolean deserializeAfterSerialization = true; private boolean checkpointEmptyMap = false; - private List plugins = new ArrayList<>(); + private List pluginFactories = new ArrayList<>(); public Builder() {} @@ -459,24 +458,29 @@ public Builder withCheckpointEmptyMap(boolean checkpointEmptyMap) { } /** - * Registers one or more plugins for lifecycle event instrumentation. + * Registers one or more plugin factories for lifecycle event instrumentation. * - *

    Plugins receive hooks at invocation, operation, and user function boundaries. Errors thrown by plugins are - * isolated and never disrupt SDK execution. + *

    Each factory is called once per Lambda invocation, with that invocation's {@code InvocationInfo}, and the + * instance it returns receives only that invocation's hooks. Plugin instances can therefore keep per-invocation + * state in plain fields even when the execution environment runs several executions concurrently. * - *

    Calling this method replaces any previously registered plugins. Plugins are called in registration order. + *

    Plugins receive hooks at invocation, operation, and user function boundaries. Errors thrown by a factory + * or a hook are isolated and never disrupt SDK execution. * - * @param plugins the plugins to register + *

    Calling this method replaces any previously registered factories. Plugins are called in registration + * order. + * + * @param pluginFactories the plugin factories to register * @return This builder - * @throws NullPointerException if any plugin is null + * @throws NullPointerException if any factory is null */ - public Builder withPlugins(DurableExecutionPlugin... plugins) { - Objects.requireNonNull(plugins, "Plugins array cannot be null"); - var newPlugins = new ArrayList(plugins.length); - for (var plugin : plugins) { - newPlugins.add(Objects.requireNonNull(plugin, "Plugin cannot be null")); + public Builder withPlugins(DurableExecutionPluginFactory... pluginFactories) { + Objects.requireNonNull(pluginFactories, "Plugins array cannot be null"); + var newFactories = new ArrayList(pluginFactories.length); + for (var pluginFactory : pluginFactories) { + newFactories.add(Objects.requireNonNull(pluginFactory, "Plugin cannot be null")); } - this.plugins = newPlugins; + this.pluginFactories = newFactories; return this; } diff --git a/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java b/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java index efd2e86f0..b38cd18b5 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java @@ -2,7 +2,6 @@ // SPDX-License-Identifier: Apache-2.0 package software.amazon.lambda.durable; -import java.lang.reflect.Modifier; import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.LinkedHashSet; @@ -10,7 +9,7 @@ import java.util.Map; import java.util.ServiceConfigurationError; import java.util.ServiceLoader; -import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; final class DynamicPluginLoader { @@ -18,38 +17,39 @@ final class DynamicPluginLoader { private DynamicPluginLoader() {} - static List loadConfiguredPlugins(List explicitPlugins) { + static List loadConfiguredPluginFactories( + List explicitFactories) { var configuredNames = System.getenv(PLUGINS_ENVIRONMENT_VARIABLE); if (configuredNames == null || configuredNames.isBlank()) { - return List.copyOf(explicitPlugins); + return List.copyOf(explicitFactories); } var classLoader = Thread.currentThread().getContextClassLoader(); if (classLoader == null) { classLoader = DurableExecutionPluginProvider.class.getClassLoader(); } - return loadConfiguredPlugins( + return loadConfiguredPluginFactories( configuredNames, ServiceLoader.load(DurableExecutionPluginProvider.class, classLoader), - explicitPlugins); + explicitFactories); } - static List loadConfiguredPlugins( + static List loadConfiguredPluginFactories( String configuredNames, Iterable providers, - List explicitPlugins) { + List explicitFactories) { if (configuredNames == null || configuredNames.isBlank()) { - return List.copyOf(explicitPlugins); + return List.copyOf(explicitFactories); } var requestedNames = parseProviderNames(configuredNames); var providersByName = indexProviders(providers); - var plugins = new ArrayList(); + var factories = new ArrayList(); for (var name : requestedNames) { - addPlugin(name, getProvider(name, providersByName), plugins); + factories.add(getProvider(name, providersByName)); } - plugins.addAll(explicitPlugins); - return List.copyOf(plugins); + factories.addAll(explicitFactories); + return List.copyOf(factories); } private static List parseProviderNames(String configuredNames) { @@ -121,58 +121,6 @@ private static DurableExecutionPluginProvider getProvider( return provider; } - private static void addPlugin( - String name, DurableExecutionPluginProvider provider, List plugins) { - var pluginType = validateProvider(name, provider); - var plugin = createPlugin(name, provider); - if (!pluginType.isInstance(plugin)) { - throw configurationError("Plugin provider '" + name + "' declared type '" + pluginType.getName() - + "' but created '" + plugin.getClass().getName() + "'"); - } - plugins.add(plugin); - } - - private static Class validateProvider( - String name, DurableExecutionPluginProvider provider) { - int apiVersion; - Class pluginType; - try { - apiVersion = provider.getApiVersion(); - pluginType = provider.getPluginType(); - } catch (RuntimeException | LinkageError e) { - throw configurationError( - "Plugin provider '" + name + "' is not compatible with this Durable Execution SDK version", e); - } - if (apiVersion != DurableExecutionPluginProvider.API_VERSION) { - throw configurationError("Plugin provider '" + name + "' uses provider API version " + apiVersion - + ", but this SDK requires version " + DurableExecutionPluginProvider.API_VERSION); - } - if (pluginType == null - || pluginType.isInterface() - || Modifier.isAbstract(pluginType.getModifiers()) - || !DurableExecutionPlugin.class.isAssignableFrom(pluginType)) { - throw configurationError( - "Plugin provider '" + name + "' must declare a concrete DurableExecutionPlugin type"); - } - return pluginType; - } - - private static DurableExecutionPlugin createPlugin(String name, DurableExecutionPluginProvider provider) { - DurableExecutionPlugin plugin; - try { - plugin = provider.createPlugin(); - } catch (RuntimeException | LinkageError e) { - throw configurationError( - "Plugin provider '" + name + "' failed to create its plugin. " - + "Verify its settings and compatibility with this Durable Execution SDK version", - e); - } - if (plugin == null) { - throw configurationError("Plugin provider '" + name + "' returned a null plugin"); - } - return plugin; - } - private static IllegalStateException configurationError(String message) { return new IllegalStateException("Dynamic plugin configuration failed: " + message); } diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java index d8db91326..1ef55278c 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java @@ -55,8 +55,10 @@ public static DurableExecutionOutput execute( TypeToken inputType, BiFunction handler, DurableConfig config) { - var pluginRunner = config.getPluginRunner(); try (var executionManager = new ExecutionManager(input, config, lambdaContext)) { + // Scoped to this invocation: the runner creates this invocation's plugin instances from the configured + // factories when onInvocationStart fires below, and releases them when the manager closes. + var pluginRunner = executionManager.getPluginRunner(); var isFirstInvocation = !executionManager.isReplaying(); var requestId = lambdaContext != null ? lambdaContext.getAwsRequestId() : null; var executionArn = input.durableExecutionArn(); diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java index 0e9d8426e..f967a30a5 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java @@ -30,6 +30,7 @@ import software.amazon.lambda.durable.model.SafeCloseable; import software.amazon.lambda.durable.operation.BaseDurableOperation; import software.amazon.lambda.durable.plugin.PluginInfoConverter; +import software.amazon.lambda.durable.plugin.PluginRunner; /** * Central manager for durable execution coordination. @@ -66,6 +67,11 @@ public class ExecutionManager implements SafeCloseable { private final Set updatedOperationIdsSinceLastInvocation; private final Set initialOperationIds; + // ===== Plugins ===== + // Created per invocation, alongside this manager: the runner materializes one plugin instance per configured + // factory when the invocation starts, and releases them in close(), so instances never outlive the invocation. + private final PluginRunner pluginRunner; + // ===== Thread Coordination ===== private final Map registeredOperations = new ConcurrentHashMap<>(); private final Set activeThreads = Collections.synchronizedSet(new HashSet<>()); @@ -79,6 +85,7 @@ public class ExecutionManager implements SafeCloseable { public ExecutionManager(DurableExecutionInput input, DurableConfig config, Context lambdaContext) { durableConfig = config; + this.pluginRunner = new PluginRunner(config.getPluginFactories()); this.durableExecutionArn = input.durableExecutionArn(); this.lambdaContext = lambdaContext; @@ -124,6 +131,15 @@ public ExecutionManager(DurableExecutionInput input, DurableConfig config, Conte // ===== State Management ===== + /** + * Returns this invocation's plugin dispatcher. Scoped to this manager, i.e. to this invocation. + * + * @return PluginRunner instance (never null) + */ + public PluginRunner getPluginRunner() { + return pluginRunner; + } + /** Returns the ARN of the durable execution being managed. */ public String getDurableExecutionArn() { return durableExecutionArn; @@ -217,14 +233,8 @@ void onCheckpointComplete(List newOperations) { // Fire onOperationChange when a checkpoint response changed one or more operations if (!updatedOperations.isEmpty()) { var requestId = lambdaContext != null ? lambdaContext.getAwsRequestId() : null; - durableConfig - .getPluginRunner() - .onOperationChange(PluginInfoConverter.toOperationChangeInfo( - requestId, - durableExecutionArn, - updatedOperations, - operationStorage.values(), - initialOperationIds)); + pluginRunner.onOperationChange(PluginInfoConverter.toOperationChangeInfo( + requestId, durableExecutionArn, updatedOperations, operationStorage.values(), initialOperationIds)); } } @@ -412,6 +422,9 @@ public void close() { validateRunningThreads(); checkpointManager.shutdown(); + + // The invocation is over: drop this invocation's plugin instances so they cannot be reached again. + pluginRunner.releasePlugins(); } private void validateRunningThreads() { diff --git a/sdk/src/main/java/software/amazon/lambda/durable/operation/BaseDurableOperation.java b/sdk/src/main/java/software/amazon/lambda/durable/operation/BaseDurableOperation.java index 5cd40820e..32ada44c5 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/operation/BaseDurableOperation.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/operation/BaseDurableOperation.java @@ -556,10 +556,13 @@ public CompletableFuture getRunningUserHandler() { // ─── Plugin hook helpers ───────────────────────────────────────────── - /** Returns the plugin runner from config, or no-op if config is unavailable. */ + /** + * Returns this invocation's plugin runner, scoped to the ExecutionManager of this invocation. Falls back to a no-op + * runner when the manager does not provide one (mocked managers in unit tests). + */ private PluginRunner getPluginRunner() { - var config = getContext().getDurableConfig(); - return config != null ? config.getPluginRunner() : PluginRunner.noOp(); + var pluginRunner = executionManager.getPluginRunner(); + return pluginRunner != null ? pluginRunner : PluginRunner.noOp(); } /** Fires onOperationStart plugin hook. */ diff --git a/sdk/src/main/java/software/amazon/lambda/durable/plugin/DurableExecutionPluginFactory.java b/sdk/src/main/java/software/amazon/lambda/durable/plugin/DurableExecutionPluginFactory.java new file mode 100644 index 000000000..264dc12ea --- /dev/null +++ b/sdk/src/main/java/software/amazon/lambda/durable/plugin/DurableExecutionPluginFactory.java @@ -0,0 +1,35 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.plugin; + +/** + * Creates one {@link DurableExecutionPlugin} instance per Lambda invocation. + * + *

    The SDK builds the {@link InvocationInfo} for an invocation, calls this factory with it, dispatches that + * invocation's hooks to the returned instance, and drops the instance when the invocation returns. A plugin instance + * therefore serves exactly one invocation and can hold per-invocation state in plain fields — no keying by execution + * ARN is needed, even when the execution environment runs several executions concurrently. + * + *

    The {@link InvocationInfo} handed to the factory is the same instance the plugin's + * {@link DurableExecutionPlugin#onInvocationStart(InvocationInfo)} hook then receives. + * + *

    Factory failures are contained exactly like hook failures: a factory that throws or returns {@code null} is logged + * and skipped for that invocation, and never disrupts the execution. + * + * 

    {@code
+ * DurableConfig.builder()
+ *     .withPlugins(info -> new MyPlugin(info.durableExecutionArn()))
+ *     .build();
+ * }
    + */ +@FunctionalInterface +public interface DurableExecutionPluginFactory { + + /** + * Creates the plugin instance that serves the described invocation. + * + * @param invocationInfo the invocation the plugin instance will observe + * @return the plugin instance for this invocation + */ + DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo); +} diff --git a/sdk/src/main/java/software/amazon/lambda/durable/plugin/DurableExecutionPluginProvider.java b/sdk/src/main/java/software/amazon/lambda/durable/plugin/DurableExecutionPluginProvider.java index c527c2b38..e1d3d51e0 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/plugin/DurableExecutionPluginProvider.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/plugin/DurableExecutionPluginProvider.java @@ -3,16 +3,17 @@ package software.amazon.lambda.durable.plugin; /** - * Service provider interface for dynamically loading {@link DurableExecutionPlugin} implementations. + * A {@link DurableExecutionPluginFactory} that can be discovered through {@link java.util.ServiceLoader} and selected + * by name. * *

    Provider JARs register implementations in - * {@code META-INF/services/software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider}. The SDK only creates - * plugins from providers explicitly selected through {@code DURABLE_EXECUTION_PLUGINS}. + * {@code META-INF/services/software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider}. The SDK only uses + * providers explicitly selected through {@code DURABLE_EXECUTION_PLUGINS}; selection is by {@link #getName()}. + * + *

    A provider is itself the per-invocation factory: {@link #createPlugin(InvocationInfo)} is called once per + * invocation, and the returned instance serves only that invocation. */ -public interface DurableExecutionPluginProvider { - - /** Current version of the dynamic plugin provider contract. */ - int API_VERSION = 1; +public interface DurableExecutionPluginProvider extends DurableExecutionPluginFactory { /** * Returns the stable name used to select this provider. @@ -20,25 +21,4 @@ public interface DurableExecutionPluginProvider { * @return non-empty provider name */ String getName(); - - /** - * Returns the provider API version this implementation supports. - * - * @return provider API version - */ - int getApiVersion(); - - /** - * Returns the concrete plugin type created by this provider. - * - * @return plugin implementation class - */ - Class getPluginType(); - - /** - * Creates the plugin instance. - * - * @return plugin instance - */ - DurableExecutionPlugin createPlugin(); } diff --git a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java index e3a5707c4..3edfb0679 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 package software.amazon.lambda.durable.plugin; +import java.util.ArrayList; import java.util.Collections; import java.util.List; import java.util.function.Consumer; @@ -9,9 +10,16 @@ import org.slf4j.LoggerFactory; /** - * Composes multiple {@link DurableExecutionPlugin} instances into a single dispatcher. + * Dispatches the lifecycle hooks of a single Lambda invocation to that invocation's plugin instances. * - *

    Event hooks are fire-and-forget: each plugin is called in order, errors are swallowed. + *

    A runner is created per invocation from the configured {@link DurableExecutionPluginFactory factories} and holds + * no plugin instances until {@link #onInvocationStart(InvocationInfo)} materializes them — once, before the first hook + * fires, from the very {@link InvocationInfo} the first hook then receives. {@link #releasePlugins()} drops them when + * the invocation returns, so a plugin instance is never shared between invocations and never needs to key its state by + * execution ARN. + * + *

    Event hooks are fire-and-forget: each plugin is called in order, errors are swallowed. A factory that throws or + * returns {@code null} is contained the same way — the plugin is skipped for the invocation. * *

    {@code onInvocationEnd} is awaited (the SDK blocks until it returns) to allow plugins to flush data before Lambda * freezes. @@ -19,32 +27,65 @@ public class PluginRunner { private static final Logger logger = LoggerFactory.getLogger(PluginRunner.class); - private static final PluginRunner NO_OP = new PluginRunner(Collections.emptyList()); - private final List plugins; + private final List pluginFactories; - public PluginRunner(List plugins) { - this.plugins = plugins != null ? List.copyOf(plugins) : Collections.emptyList(); + /** + * This invocation's plugin instances. Written once on the thread that fires {@code onInvocationStart}, read from + * the user, checkpoint, and operation threads that fire the later hooks — volatile for that publication. + */ + private volatile List plugins = List.of(); + + public PluginRunner(List pluginFactories) { + this.pluginFactories = pluginFactories != null ? List.copyOf(pluginFactories) : Collections.emptyList(); } - /** Returns a no-op runner that does nothing. */ + /** Returns a runner with no plugin factories, which does nothing. */ public static PluginRunner noOp() { - return NO_OP; + return new PluginRunner(Collections.emptyList()); } - /** Returns true if no plugins are registered. */ + /** Returns true if no plugin factories are registered. */ public boolean isEmpty() { - return plugins.isEmpty(); + return pluginFactories.isEmpty(); } - /** Returns the list of registered plugins. */ - public List getPlugins() { - return plugins; + // ─── Per-invocation lifetime ───────────────────────────────────────── + + /** + * Creates this invocation's plugin instances, one per registered factory. + * + *

    Called from {@link #onInvocationStart(InvocationInfo)} so the instances exist before any hook is dispatched. + * Factories that throw or return null are logged and skipped. + */ + private void createPlugins(InvocationInfo info) { + var created = new ArrayList(pluginFactories.size()); + for (var factory : pluginFactories) { + try { + var plugin = factory.createPlugin(info); + if (plugin == null) { + logger.warn("Plugin factory {} returned null; skipping it for this invocation", factory); + continue; + } + created.add(plugin); + } catch (Exception e) { + logger.warn("Plugin factory threw exception; skipping it for this invocation", e); + } + } + this.plugins = List.copyOf(created); + } + + /** + * Drops this invocation's plugin instances. Called when the invocation returns so the instances are unreachable + * from the SDK and cannot leak into the next invocation the environment hosts. + */ + public void releasePlugins() { + this.plugins = List.of(); } // ─── Event hooks ───────────────────────────────────────────────────── - /** Calls a void hook on all plugins, swallowing any errors. */ + /** Calls a void hook on all of this invocation's plugins, swallowing any errors. */ private void run(Consumer hook) { for (var plugin : plugins) { try { @@ -55,7 +96,12 @@ private void run(Consumer hook) { } } + /** + * Called at the start of each invocation. Materializes this invocation's plugin instances from the registered + * factories, then dispatches the hook to them with the same {@link InvocationInfo} the factories received. + */ public void onInvocationStart(InvocationInfo info) { + createPlugins(info); run(p -> p.onInvocationStart(info)); } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/DurableConfigTest.java b/sdk/src/test/java/software/amazon/lambda/durable/DurableConfigTest.java index 266e43a6e..a79b8c515 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/DurableConfigTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/DurableConfigTest.java @@ -13,8 +13,10 @@ import static org.mockito.Mockito.mock; import java.time.Duration; +import java.time.Instant; import java.util.ArrayList; import java.util.List; +import java.util.Map; import java.util.concurrent.ExecutorService; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; @@ -23,6 +25,9 @@ import software.amazon.lambda.durable.client.DurableExecutionClient; import software.amazon.lambda.durable.client.LambdaDurableFunctionsClient; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; +import software.amazon.lambda.durable.plugin.InvocationInfo; +import software.amazon.lambda.durable.plugin.PluginRunner; import software.amazon.lambda.durable.retry.JitterStrategy; import software.amazon.lambda.durable.retry.PollingStrategies; import software.amazon.lambda.durable.serde.JacksonSerDes; @@ -507,47 +512,41 @@ private static void setField(Object target, String fieldName, Object value) thro // --- Plugin registration tests --- @Test - void testDefaultConfig_PluginRunnerIsNoOp() { + void testDefaultConfig_NoPluginFactories() { var config = DurableConfig.defaultConfig(); - assertNotNull(config.getPluginRunner()); - assertTrue(config.getPluginRunner().isEmpty()); + assertNotNull(config.getPluginFactories()); + assertTrue(config.getPluginFactories().isEmpty()); } @Test - void testBuilder_NoPlugins_PluginRunnerIsNoOp() { + void testBuilder_NoPlugins_NoPluginFactories() { var config = DurableConfig.builder().withDurableExecutionClient(mockClient).build(); - assertNotNull(config.getPluginRunner()); - assertTrue(config.getPluginRunner().isEmpty()); + assertNotNull(config.getPluginFactories()); + assertTrue(config.getPluginFactories().isEmpty()); } @Test - void testBuilder_WithPlugin_CreatesActivePluginRunner() { - var plugin = new DurableExecutionPlugin() {}; + void testBuilder_WithPlugin_RegistersFactory() { var config = DurableConfig.builder() .withDurableExecutionClient(mockClient) - .withPlugins(plugin) + .withPlugins(info -> new DurableExecutionPlugin() {}) .build(); - assertNotNull(config.getPluginRunner()); - assertFalse(config.getPluginRunner().isEmpty()); + assertEquals(1, config.getPluginFactories().size()); } @Test - void testBuilder_WithMultiplePlugins_AllRegistered() { + void testBuilder_WithMultiplePlugins_AllRegisteredInOrder() { var calls = new ArrayList(); - var plugin1 = new TestPlugin("p1", calls); - var plugin2 = new TestPlugin("p2", calls); var config = DurableConfig.builder() .withDurableExecutionClient(mockClient) - .withPlugins(plugin1, plugin2) + .withPlugins(info -> new TestPlugin("p1", calls), info -> new TestPlugin("p2", calls)) .build(); - config.getPluginRunner() - .onInvocationStart(new software.amazon.lambda.durable.plugin.InvocationInfo( - "req-1", "arn:test", true, java.time.Instant.now(), java.util.Map.of(), java.util.Map.of())); + new PluginRunner(config.getPluginFactories()).onInvocationStart(invocationInfo()); assertEquals(List.of("p1:onInvocationStart", "p2:onInvocationStart"), calls); } @@ -555,19 +554,14 @@ void testBuilder_WithMultiplePlugins_AllRegistered() { @Test void testBuilder_WithPlugins_CalledMultipleTimes_Replaces() { var calls = new ArrayList(); - var plugin1 = new TestPlugin("p1", calls); - var plugin2 = new TestPlugin("p2", calls); - var plugin3 = new TestPlugin("p3", calls); var config = DurableConfig.builder() .withDurableExecutionClient(mockClient) - .withPlugins(plugin1) - .withPlugins(plugin2, plugin3) + .withPlugins(info -> new TestPlugin("p1", calls)) + .withPlugins(info -> new TestPlugin("p2", calls), info -> new TestPlugin("p3", calls)) .build(); - config.getPluginRunner() - .onInvocationStart(new software.amazon.lambda.durable.plugin.InvocationInfo( - "req-1", "arn:test", true, java.time.Instant.now(), java.util.Map.of(), java.util.Map.of())); + new PluginRunner(config.getPluginFactories()).onInvocationStart(invocationInfo()); assertEquals(List.of("p2:onInvocationStart", "p3:onInvocationStart"), calls); } @@ -576,7 +570,8 @@ void testBuilder_WithPlugins_CalledMultipleTimes_Replaces() { void testBuilder_WithPlugins_NullArrayThrows() { var builder = DurableConfig.builder(); - var ex = assertThrows(NullPointerException.class, () -> builder.withPlugins((DurableExecutionPlugin[]) null)); + var ex = assertThrows( + NullPointerException.class, () -> builder.withPlugins((DurableExecutionPluginFactory[]) null)); assertEquals("Plugins array cannot be null", ex.getMessage()); } @@ -585,16 +580,19 @@ void testBuilder_WithPlugins_NullElementThrows() { var builder = DurableConfig.builder(); var ex = assertThrows( - NullPointerException.class, () -> builder.withPlugins(new DurableExecutionPlugin[] {null})); + NullPointerException.class, () -> builder.withPlugins(new DurableExecutionPluginFactory[] {null})); assertEquals("Plugin cannot be null", ex.getMessage()); } @Test void testBuilder_WithPlugins_FluentAPI() { var builder = DurableConfig.builder(); - var plugin = new DurableExecutionPlugin() {}; - assertSame(builder, builder.withPlugins(plugin)); + assertSame(builder, builder.withPlugins(info -> new DurableExecutionPlugin() {})); + } + + private static InvocationInfo invocationInfo() { + return new InvocationInfo("req-1", "arn:test", true, Instant.now(), Map.of(), Map.of()); } /** Simple test plugin that records hook calls. */ @@ -608,7 +606,7 @@ private static class TestPlugin implements DurableExecutionPlugin { } @Override - public void onInvocationStart(software.amazon.lambda.durable.plugin.InvocationInfo info) { + public void onInvocationStart(InvocationInfo info) { calls.add(name + ":onInvocationStart"); } } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderTest.java b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderTest.java index 17fe9f5b3..2c8992c9f 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderTest.java @@ -8,92 +8,93 @@ import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; -import java.util.ArrayList; +import java.time.Instant; import java.util.Iterator; import java.util.List; import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Supplier; import org.junit.jupiter.api.Test; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; +import software.amazon.lambda.durable.plugin.InvocationInfo; class DynamicPluginLoaderTest { @Test - void unsetConfigurationPreservesExplicitPluginsWithoutDiscoveringProviders() { - var explicitPlugin = new FirstPlugin(); + void unsetConfigurationPreservesExplicitFactoriesWithoutDiscoveringProviders() { + DurableExecutionPluginFactory explicitFactory = info -> new FirstPlugin(); Iterable providers = () -> { throw new AssertionError("Providers should not be discovered"); }; - var plugins = DynamicPluginLoader.loadConfiguredPlugins(null, providers, List.of(explicitPlugin)); + var factories = DynamicPluginLoader.loadConfiguredPluginFactories(null, providers, List.of(explicitFactory)); - assertEquals(1, plugins.size()); - assertSame(explicitPlugin, plugins.get(0)); + assertEquals(1, factories.size()); + assertSame(explicitFactory, factories.get(0)); } @Test - void loadsRequestedProvidersBeforeExplicitPluginsInConfiguredOrder() { - var creationOrder = new ArrayList(); - var explicitPlugin = new ExplicitPlugin(); - var firstProvider = provider("first", FirstPlugin.class, () -> { - creationOrder.add("first"); + void loadsRequestedProvidersBeforeExplicitFactoriesInConfiguredOrder() { + DurableExecutionPluginFactory explicitFactory = info -> new ExplicitPlugin(); + var firstProvider = provider("first", FirstPlugin::new); + var secondProvider = provider("second", SecondPlugin::new); + + var factories = DynamicPluginLoader.loadConfiguredPluginFactories( + " second, first ", List.of(firstProvider, secondProvider), List.of(explicitFactory)); + + assertSame(secondProvider, factories.get(0)); + assertSame(firstProvider, factories.get(1)); + assertSame(explicitFactory, factories.get(2)); + } + + @Test + void doesNotCreatePluginsAtConfigurationTime() { + var creations = new AtomicInteger(); + var requestedProvider = provider("requested", () -> { + creations.incrementAndGet(); return new FirstPlugin(); }); - var secondProvider = provider("second", SecondPlugin.class, () -> { - creationOrder.add("second"); - return new SecondPlugin(); - }); - var plugins = DynamicPluginLoader.loadConfiguredPlugins( - " second, first ", List.of(firstProvider, secondProvider), List.of(explicitPlugin)); + var factories = + DynamicPluginLoader.loadConfiguredPluginFactories("requested", List.of(requestedProvider), List.of()); - assertInstanceOf(SecondPlugin.class, plugins.get(0)); - assertInstanceOf(FirstPlugin.class, plugins.get(1)); - assertSame(explicitPlugin, plugins.get(2)); - assertEquals(List.of("second", "first"), creationOrder); + // Plugins are created per invocation, not while configuration is resolved. + assertEquals(1, factories.size()); + assertEquals(0, creations.get()); + assertInstanceOf(FirstPlugin.class, factories.get(0).createPlugin(invocationInfo())); + assertEquals(1, creations.get()); } @Test - void doesNotCreateProvidersOutsideTheAllowList() { - var unrequestedCreations = new AtomicInteger(); - var requestedProvider = provider("requested", FirstPlugin.class, FirstPlugin::new); - var unrequestedProvider = provider("unrequested", SecondPlugin.class, () -> { - unrequestedCreations.incrementAndGet(); - return new SecondPlugin(); - }); + void doesNotSelectProvidersOutsideTheAllowList() { + var requestedProvider = provider("requested", FirstPlugin::new); + var unrequestedProvider = provider("unrequested", SecondPlugin::new); - var plugins = DynamicPluginLoader.loadConfiguredPlugins( + var factories = DynamicPluginLoader.loadConfiguredPluginFactories( "requested", List.of(requestedProvider, unrequestedProvider), List.of()); - assertEquals(1, plugins.size()); - assertEquals(0, unrequestedCreations.get()); + assertEquals(List.of(requestedProvider), factories); } @Test - void loadsExplicitAndDynamicPluginsOfTheSameType() { - var creations = new AtomicInteger(); - var explicitPlugin = new FirstPlugin(); - var dynamicPlugin = new FirstPlugin(); - var duplicateProvider = provider("first", FirstPlugin.class, () -> { - creations.incrementAndGet(); - return dynamicPlugin; - }); + void loadsExplicitAndDynamicFactoriesOfTheSameType() { + DurableExecutionPluginFactory explicitFactory = info -> new FirstPlugin(); + var duplicateProvider = provider("first", FirstPlugin::new); - var plugins = - DynamicPluginLoader.loadConfiguredPlugins("first", List.of(duplicateProvider), List.of(explicitPlugin)); + var factories = DynamicPluginLoader.loadConfiguredPluginFactories( + "first", List.of(duplicateProvider), List.of(explicitFactory)); - assertEquals(2, plugins.size()); - assertSame(dynamicPlugin, plugins.get(0)); - assertSame(explicitPlugin, plugins.get(1)); - assertEquals(1, creations.get()); + assertEquals(2, factories.size()); + assertSame(duplicateProvider, factories.get(0)); + assertSame(explicitFactory, factories.get(1)); } @Test void rejectsEmptyConfiguredProviderName() { var error = assertThrows( IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins("first,,second", List.of(), List.of())); + () -> DynamicPluginLoader.loadConfiguredPluginFactories("first,,second", List.of(), List.of())); assertTrue(error.getMessage().contains("must be non-empty")); assertTrue(error.getMessage().contains(DynamicPluginLoader.PLUGINS_ENVIRONMENT_VARIABLE)); @@ -103,18 +104,19 @@ void rejectsEmptyConfiguredProviderName() { void rejectsDuplicateConfiguredProviderName() { var error = assertThrows( IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins("first,first", List.of(), List.of())); + () -> DynamicPluginLoader.loadConfiguredPluginFactories("first,first", List.of(), List.of())); assertTrue(error.getMessage().contains("listed more than once")); } @Test void rejectsUnknownProviderAndListsAvailableNames() { - var availableProvider = provider("available", FirstPlugin.class, FirstPlugin::new); + var availableProvider = provider("available", FirstPlugin::new); var error = assertThrows( IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins("missing", List.of(availableProvider), List.of())); + () -> DynamicPluginLoader.loadConfiguredPluginFactories( + "missing", List.of(availableProvider), List.of())); assertTrue(error.getMessage().contains("No DurableExecutionPluginProvider named 'missing'")); assertTrue(error.getMessage().contains("available")); @@ -122,12 +124,12 @@ void rejectsUnknownProviderAndListsAvailableNames() { @Test void rejectsDuplicateDiscoveredProviderNames() { - var firstProvider = provider("duplicate", FirstPlugin.class, FirstPlugin::new); - var secondProvider = provider("duplicate", SecondPlugin.class, SecondPlugin::new); + var firstProvider = provider("duplicate", FirstPlugin::new); + var secondProvider = provider("duplicate", SecondPlugin::new); var error = assertThrows( IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins( + () -> DynamicPluginLoader.loadConfiguredPluginFactories( "duplicate", List.of(firstProvider, secondProvider), List.of())); assertTrue(error.getMessage().contains("Multiple DurableExecutionPluginProvider implementations")); @@ -135,38 +137,15 @@ void rejectsDuplicateDiscoveredProviderNames() { } @Test - void rejectsIncompatibleProviderApiVersion() { - var provider = new TestProvider("first", 2, FirstPlugin.class, FirstPlugin::new); - - var error = assertThrows( - IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins("first", List.of(provider), List.of())); - - assertTrue(error.getMessage().contains("uses provider API version 2")); - assertTrue(error.getMessage().contains("requires version " + DurableExecutionPluginProvider.API_VERSION)); - } - - @Test - void rejectsInvalidDeclaredPluginType() { - var provider = provider("invalid", DurableExecutionPlugin.class, FirstPlugin::new); - - var error = assertThrows( - IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins("invalid", List.of(provider), List.of())); - - assertTrue(error.getMessage().contains("must declare a concrete DurableExecutionPlugin type")); - } - - @Test - void rejectsPluginThatDoesNotMatchDeclaredType() { - var provider = provider("first", FirstPlugin.class, SecondPlugin::new); + void rejectsProviderWithInvalidName() { + var blankNameProvider = provider(" ", FirstPlugin::new); var error = assertThrows( IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins("first", List.of(provider), List.of())); + () -> DynamicPluginLoader.loadConfiguredPluginFactories( + "first", List.of(blankNameProvider), List.of())); - assertTrue(error.getMessage().contains("declared type")); - assertTrue(error.getMessage().contains(SecondPlugin.class.getName())); + assertTrue(error.getMessage().contains("returned an invalid name")); } @Test @@ -185,38 +164,21 @@ public DurableExecutionPluginProvider next() { var error = assertThrows( IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins("first", providers, List.of())); + () -> DynamicPluginLoader.loadConfiguredPluginFactories("first", providers, List.of())); assertTrue(error.getMessage().contains("Failed to discover")); assertInstanceOf(LinkageError.class, error.getCause()); } - @Test - void wrapsPluginCreationFailure() { - var provider = provider("first", FirstPlugin.class, () -> { - throw new IllegalArgumentException("bad settings"); - }); - - var error = assertThrows( - IllegalStateException.class, - () -> DynamicPluginLoader.loadConfiguredPlugins("first", List.of(provider), List.of())); - - assertTrue(error.getMessage().contains("failed to create its plugin")); - assertInstanceOf(IllegalArgumentException.class, error.getCause()); + private static InvocationInfo invocationInfo() { + return new InvocationInfo("req-123", "arn:test", true, Instant.now()); } - private static TestProvider provider( - String name, - Class pluginType, - Supplier pluginSupplier) { - return new TestProvider(name, DurableExecutionPluginProvider.API_VERSION, pluginType, pluginSupplier); + private static TestProvider provider(String name, Supplier pluginSupplier) { + return new TestProvider(name, pluginSupplier); } - private record TestProvider( - String name, - int apiVersion, - Class pluginType, - Supplier pluginSupplier) + private record TestProvider(String name, Supplier pluginSupplier) implements DurableExecutionPluginProvider { @Override @@ -225,17 +187,7 @@ public String getName() { } @Override - public int getApiVersion() { - return apiVersion; - } - - @Override - public Class getPluginType() { - return pluginType; - } - - @Override - public DurableExecutionPlugin createPlugin() { + public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { return pluginSupplier.get(); } } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/operation/BaseDurableOperationPluginTest.java b/sdk/src/test/java/software/amazon/lambda/durable/operation/BaseDurableOperationPluginTest.java index 463a824ea..acae60678 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/operation/BaseDurableOperationPluginTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/operation/BaseDurableOperationPluginTest.java @@ -7,6 +7,7 @@ import static org.mockito.Mockito.when; import java.time.Duration; +import java.time.Instant; import java.util.ArrayList; import java.util.Collections; import java.util.List; @@ -27,6 +28,7 @@ import software.amazon.lambda.durable.model.OperationIdentifier; import software.amazon.lambda.durable.model.OperationSubType; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.InvocationInfo; import software.amazon.lambda.durable.plugin.OperationInfo; /** @@ -55,7 +57,7 @@ void execute_firesOnOperationStart_withIsReplayTrue_forNonTerminalWait() { .build(); var executionManager = createExecutionManager(List.of(waitOp), plugin); - var durableContext = mockDurableContext(executionManager, plugin); + var durableContext = mockDurableContext(executionManager); var operation = new WaitOperation( OperationIdentifier.of(OPERATION_ID, OPERATION_NAME, OperationSubType.WAIT), @@ -87,7 +89,7 @@ void execute_doesNotFireOnOperationStart_forTerminalOperation(OperationStatus te .build(); var executionManager = createExecutionManager(List.of(waitOp), plugin); - var durableContext = mockDurableContext(executionManager, plugin); + var durableContext = mockDurableContext(executionManager); var operation = new WaitOperation( OperationIdentifier.of(OPERATION_ID, OPERATION_NAME, OperationSubType.WAIT), @@ -107,7 +109,7 @@ void execute_firesOnOperationStart_withIsReplayFalse_forFirstExecution() { var plugin = new RecordingPlugin(); // No existing operations — first execution var executionManager = createExecutionManager(List.of(), plugin); - var durableContext = mockDurableContext(executionManager, plugin); + var durableContext = mockDurableContext(executionManager); var operation = new WaitOperation( OperationIdentifier.of(OPERATION_ID, OPERATION_NAME, OperationSubType.WAIT), @@ -125,6 +127,10 @@ void execute_firesOnOperationStart_withIsReplayFalse_forFirstExecution() { // ─── Helpers ───────────────────────────────────────────────────────── + /** + * Builds the per-invocation ExecutionManager and starts its invocation, which is what materializes the plugin + * instance the operation hooks are then dispatched to. + */ private ExecutionManager createExecutionManager(List additionalOps, RecordingPlugin plugin) { var client = TestUtils.createMockClient(); var operations = new ArrayList(); @@ -138,19 +144,20 @@ private ExecutionManager createExecutionManager(List additionalOps, R CheckpointUpdatedExecutionState.builder().operations(operations).build(); var config = DurableConfig.builder() .withDurableExecutionClient(client) - .withPlugins(plugin) + .withPlugins(info -> plugin) .build(); var executionManager = new ExecutionManager( new DurableExecutionInput(EXECUTION_ARN, "test-token", initialState), config, null); executionManager.setCurrentThreadContext(new ThreadContext("Root", ThreadType.CONTEXT)); + executionManager + .getPluginRunner() + .onInvocationStart(new InvocationInfo("req-1", EXECUTION_ARN, true, Instant.now())); return executionManager; } - private DurableContextImpl mockDurableContext(ExecutionManager executionManager, RecordingPlugin plugin) { + private DurableContextImpl mockDurableContext(ExecutionManager executionManager) { var durableContext = mock(DurableContextImpl.class); when(durableContext.getExecutionManager()).thenReturn(executionManager); - when(durableContext.getDurableConfig()) - .thenReturn(DurableConfig.builder().withPlugins(plugin).build()); return durableContext; } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java index dc21c9e8b..cf6f9deb1 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java @@ -8,6 +8,7 @@ import java.util.ArrayList; import java.util.List; import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; import org.junit.jupiter.api.Test; class PluginRunnerTest { @@ -24,7 +25,7 @@ void noOpRunner_doesNothing() { } @Test - void emptyPluginList_behavesAsNoOp() { + void emptyFactoryList_behavesAsNoOp() { var runner = new PluginRunner(List.of()); assertTrue(runner.isEmpty()); @@ -32,21 +33,152 @@ void emptyPluginList_behavesAsNoOp() { } @Test - void nullPluginList_behavesAsNoOp() { + void nullFactoryList_behavesAsNoOp() { var runner = new PluginRunner(null); assertTrue(runner.isEmpty()); assertDoesNotThrow(() -> runner.onOperationStart(operationInfo())); } + // ─── Per-invocation lifetime ───────────────────────────────────────── + + @Test + void invocationStart_createsOnePluginPerFactory_andPassesTheHookInfo() { + var calls = new ArrayList(); + var receivedByFactory = new ArrayList(); + var receivedByHook = new ArrayList(); + var runner = new PluginRunner(List.of(info -> { + receivedByFactory.add(info); + return new TestPlugin("p1", calls) { + @Override + public void onInvocationStart(InvocationInfo hookInfo) { + receivedByHook.add(hookInfo); + super.onInvocationStart(hookInfo); + } + }; + })); + var info = invocationInfo(); + + runner.onInvocationStart(info); + + assertEquals(List.of("p1:onInvocationStart"), calls); + assertEquals(1, receivedByFactory.size()); + assertSame(info, receivedByFactory.get(0), "the factory must receive this invocation's info"); + assertSame(info, receivedByHook.get(0), "the first hook must receive the same info instance"); + } + + @Test + void factoriesAreCalledOncePerInvocation_notPerHook() { + var creations = new AtomicInteger(); + var calls = new ArrayList(); + var runner = new PluginRunner(List.of(info -> { + creations.incrementAndGet(); + return new TestPlugin("p", calls); + })); + + runner.onInvocationStart(invocationInfo()); + runner.onOperationStart(operationInfo()); + runner.onOperationEnd(operationEndInfo()); + runner.onInvocationEnd(invocationEndInfo()); + + assertEquals(1, creations.get()); + assertEquals( + List.of("p:onInvocationStart", "p:onOperationStart", "p:onOperationEnd", "p:onInvocationEnd"), calls); + } + + @Test + void eachInvocationGetsItsOwnPluginInstance() { + var instances = new ArrayList(); + DurableExecutionPluginFactory factory = info -> { + var plugin = new TestPlugin("p", new ArrayList<>()); + instances.add(plugin); + return plugin; + }; + + // One runner per invocation, as the SDK creates one per ExecutionManager. + new PluginRunner(List.of(factory)).onInvocationStart(invocationInfo()); + new PluginRunner(List.of(factory)).onInvocationStart(invocationInfo()); + + assertEquals(2, instances.size()); + assertNotSame(instances.get(0), instances.get(1), "invocations must not share a plugin instance"); + } + + @Test + void hooksBeforeInvocationStart_dispatchToNothing() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of(info -> new TestPlugin("p", calls))); + + // Plugins only exist between onInvocationStart and the end of the invocation. + runner.onOperationStart(operationInfo()); + + assertTrue(calls.isEmpty()); + } + + @Test + void releasePlugins_dropsThisInvocationsInstances() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of(info -> new TestPlugin("p", calls))); + runner.onInvocationStart(invocationInfo()); + calls.clear(); + + runner.releasePlugins(); + runner.onOperationStart(operationInfo()); + runner.onInvocationEnd(invocationEndInfo()); + + assertTrue(calls.isEmpty(), "released plugin instances must not receive further hooks"); + } + + @Test + void factoryList_isCopiedAtConstruction() { + var calls = new ArrayList(); + var mutableList = new ArrayList(); + mutableList.add(info -> new TestPlugin("p1", calls)); + var runner = new PluginRunner(mutableList); + + // Modifying the original list should not affect the runner + mutableList.add(info -> new TestPlugin("p2", calls)); + + runner.onInvocationStart(invocationInfo()); + + // Only p1 should be called — p2 was added after construction + assertEquals(List.of("p1:onInvocationStart"), calls); + } + + // ─── Factory error isolation ───────────────────────────────────────── + + @Test + void throwingFactory_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> { + throw new RuntimeException("boom"); + }, + info -> new TestPlugin("p2", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + runner.onInvocationEnd(invocationEndInfo()); + + assertEquals(List.of("p2:onInvocationStart", "p2:onInvocationEnd"), calls); + } + + @Test + void nullReturningFactory_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of(info -> null, info -> new TestPlugin("p2", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + runner.onOperationStart(operationInfo()); + + assertEquals(List.of("p2:onInvocationStart", "p2:onOperationStart"), calls); + } + // ─── Fire-and-forget event hooks ───────────────────────────────────── @Test void fireAndForget_callsAllPlugins() { var calls = new ArrayList(); - var plugin1 = new TestPlugin("p1", calls); - var plugin2 = new TestPlugin("p2", calls); - var runner = new PluginRunner(List.of(plugin1, plugin2)); + var runner = + new PluginRunner(List.of(info -> new TestPlugin("p1", calls), info -> new TestPlugin("p2", calls))); runner.onInvocationStart(invocationInfo()); @@ -56,9 +188,7 @@ void fireAndForget_callsAllPlugins() { @Test void fireAndForget_swallowsExceptions() { var calls = new ArrayList(); - var throwingPlugin = new ThrowingPlugin(); - var normalPlugin = new TestPlugin("p2", calls); - var runner = new PluginRunner(List.of(throwingPlugin, normalPlugin)); + var runner = new PluginRunner(List.of(info -> new ThrowingPlugin(), info -> new TestPlugin("p2", calls))); assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); assertEquals(List.of("p2:onInvocationStart"), calls); @@ -67,26 +197,25 @@ void fireAndForget_swallowsExceptions() { @Test void fireAndForget_callsAllHookTypes() { var calls = new ArrayList(); - var plugin = new TestPlugin("p", calls); - var runner = new PluginRunner(List.of(plugin)); + var runner = new PluginRunner(List.of(info -> new TestPlugin("p", calls))); runner.onInvocationStart(invocationInfo()); - runner.onInvocationEnd(invocationEndInfo()); runner.onOperationStart(operationInfo()); runner.onOperationEnd(operationEndInfo()); runner.onOperationChange(operationChangeInfo()); runner.onUserFunctionStart(attemptInfo()); runner.onUserFunctionEnd(attemptEndInfo()); + runner.onInvocationEnd(invocationEndInfo()); assertEquals( List.of( "p:onInvocationStart", - "p:onInvocationEnd", "p:onOperationStart", "p:onOperationEnd", "p:onOperationChange", "p:onUserFunctionStart", - "p:onUserFunctionEnd"), + "p:onUserFunctionEnd", + "p:onInvocationEnd"), calls); } @@ -95,9 +224,10 @@ void fireAndForget_callsAllHookTypes() { @Test void awaitedHooks_callAllPlugins() { var calls = new ArrayList(); - var plugin1 = new TestPlugin("p1", calls); - var plugin2 = new TestPlugin("p2", calls); - var runner = new PluginRunner(List.of(plugin1, plugin2)); + var runner = + new PluginRunner(List.of(info -> new TestPlugin("p1", calls), info -> new TestPlugin("p2", calls))); + runner.onInvocationStart(invocationInfo()); + calls.clear(); runner.onInvocationEnd(invocationEndInfo()); @@ -107,32 +237,14 @@ void awaitedHooks_callAllPlugins() { @Test void awaitedHooks_swallowExceptions_butCallRemainingPlugins() { var calls = new ArrayList(); - var throwingPlugin = new ThrowingPlugin(); - var normalPlugin = new TestPlugin("p2", calls); - var runner = new PluginRunner(List.of(throwingPlugin, normalPlugin)); + var runner = new PluginRunner(List.of(info -> new ThrowingPlugin(), info -> new TestPlugin("p2", calls))); + runner.onInvocationStart(invocationInfo()); + calls.clear(); assertDoesNotThrow(() -> runner.onInvocationEnd(invocationEndInfo())); assertEquals(List.of("p2:onInvocationEnd"), calls); } - // ─── Thread safety (basic) ─────────────────────────────────────────── - - @Test - void pluginRunner_isImmutable() { - var calls = new ArrayList(); - var mutableList = new ArrayList(); - mutableList.add(new TestPlugin("p1", calls)); - var runner = new PluginRunner(mutableList); - - // Modifying the original list should not affect the runner - mutableList.add(new TestPlugin("p2", calls)); - - runner.onInvocationStart(invocationInfo()); - - // Only p1 should be called — p2 was added after construction - assertEquals(List.of("p1:onInvocationStart"), calls); - } - // ─── Execution input / result components ───────────────────────────── @Test From 698a0012d69843f0990f1214c70fbec16a59a829 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Thu, 17 Sep 2026 12:16:01 -0700 Subject: [PATCH 03/19] fix(plugin): refuse waits from fan-out workers, contain LinkageError MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings, both blocking, plus one test-determinism fix. With two or more exporters configured, `forEachExporterSettled` submits one task per exporter and the pump blocks waiting for them, so an exporter callback runs on a thread the pump cannot outlive. The re-entry guard only recognized the pump thread itself. An exporter that called `flush()` or `drain()` from a fan-out worker therefore waited for the pump while the pump waited for it, and a two-exporter test deadlocked both calls past a five-second deadline. The one-exporter regression passed because with a single exporter the callback runs directly on the pump thread, which the guard did recognize. Fan-out workers are now marked through a per-scheduler `ThreadLocal` and refused the same way the pump thread is: reported as `IllegalStateException` through the failure handler, returning without touching state, so the record stays in the invocation's slot and the waiting pump exports it as soon as the fan-out settles. The mark is per scheduler rather than static so one scheduler's worker is not refused by another, and the previous mark is restored rather than cleared so the rejected-execution fallback, where the fan-out thread is the pump, cannot clear a mark an enclosing frame needs. Second, `PluginRunner` caught `Exception` but not `LinkageError`. This matters because of the migration this change forces: a provider compiled against the previous interface throws `AbstractMethodError`, and one with a missing optional dependency throws `NoClassDefFoundError`. Both are `Error`, so both escaped invocation startup even though the factory contract promises a factory failure cannot disrupt the execution — end to end, a factory throwing `NoClassDefFoundError` failed the whole execution. Both `createPlugins` and the hook dispatch path now catch `Exception | LinkageError`. The catch stays narrow deliberately: `OutOfMemoryError` and `StackOverflowError` still propagate, and tests assert that. `releasePlugins` needs no equivalent change, because it drops references without calling into plugin code. `DynamicPluginLoader.getProviderName` still lets a `LinkageError` escape without the wrapping message, which is a follow-up rather than a defect: that path is fatal either way and only the message quality differs. Third, `StateCleanupLifecycleTest` polled weak references after `System.gc()`, which the JVM may ignore, so a correct implementation could fail it. It now asserts scheduler ownership and queue state directly, plus a reflective sweep proving no scheduler field reaches a plugin instance, and keeps the weak-reference count as a diagnostic that cannot fail. The sweep replaces what the weak-reference check was genuinely catching: an empty queue alone would not notice a registry reintroduced under a new field name. Verified by mutation — adding an uncleaned plugin registry to the scheduler fails all three assertions. Two suspend tests now emit on change, because under the default ON_COMPLETE mode a PENDING invocation queues nothing and the retention assertions passed trivially. --- .../durable/insight/ExportScheduler.java | 142 +++++++++--- .../ExportSchedulerFanOutReentryTest.java | 199 +++++++++++++++++ .../insight/StateCleanupLifecycleTest.java | 205 ++++++++++++++---- .../lambda/durable/PluginIntegrationTest.java | 51 +++++ .../lambda/durable/plugin/PluginRunner.java | 34 ++- .../durable/plugin/PluginRunnerTest.java | 108 +++++++++ 6 files changed, 659 insertions(+), 80 deletions(-) create mode 100644 insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFanOutReentryTest.java diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java index 3cc3562a3..4ae67384a 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java @@ -100,6 +100,24 @@ final class ExportScheduler { */ private final AtomicReference pumpThread = new AtomicReference<>(); + /** + * Marks the thread currently running one exporter's share of a fan-out for this scheduler, so that a + * {@code flush()} or {@code drain()} re-entered from an exporter callback can tell that the pump is waiting for it. + * + *

    With a single exporter the fan-out runs on the pump thread and {@link #pumpThread} already recognizes it. With + * two or more, {@link #forEachExporterSettled} submits one task per exporter and the pump then joins them all, so + * the callback runs on a thread that is not the pump but that the pump cannot outlive: a wait for the pump issued + * from there is the same wait-for cycle, two threads wide instead of one. The pump parks in the join, so it never + * reaches the point in its loop that would complete the future the worker is parked on. + * + *

    An instance field rather than a static: a fan-out worker of one scheduler is not pump-dependent on any other + * scheduler, and refusing its waits there would be a false positive. Set and cleared around each callback by the + * thread that runs it, restoring whatever was there before rather than blindly removing, so a callback that the + * pump ran inline (the rejected-worker fallback, where the fan-out thread is the pump) cannot clear a mark + * an enclosing frame still needs. + */ + private final ThreadLocal exporterFanOutThread = new ThreadLocal<>(); + /** * The invocations with a record no pump has picked up yet, in the order they first queued work. Ordering only — the * record itself lives on the invocation's plugin instance. Guarded by {@code this}. @@ -276,13 +294,14 @@ private void dropRecord(InsightPlugin execution) { * describe a different one. It tells the pump that this record gates an invocation return, so the pump exports it * before spending a flush fan-out. See {@link #exportRecordsADrainIsWaitingFor}. * - *

    Called from the pump thread itself, the wait is refused and reported instead of made: see - * {@link #refuseWaitFromThePumpThread}. + *

    Called from the pump thread itself, or from an exporter fan-out worker that pump is waiting for, the wait is + * refused and reported instead of made: see {@link #refuseWaitThatWouldBlockThePump}. */ void drain(InsightPlugin execution) { - // Re-entered from the pump: this thread is the one that would settle the signal it is about to wait for. Refuse - // and return; the record stays queued and this same pump exports it when it resumes its loop. - if (refuseWaitFromThePumpThread("drain(execution)")) { + // Re-entered from a thread the pump's progress depends on: waiting here would park on a signal only that pump + // can settle. Refuse and return; the record stays queued and that same pump exports it once it resumes its + // loop. + if (refuseWaitThatWouldBlockThePump("drain(execution)")) { return; } synchronized (this) { @@ -375,7 +394,7 @@ private synchronized boolean nothingCanSettle(InsightPlugin execution, Completab void drainAll() { // Every pass below is a drain, and each one would be refused; without this the loop spends all of its passes // reporting the same refusal. - if (refuseWaitFromThePumpThread("drainAll()")) { + if (refuseWaitThatWouldBlockThePump("drainAll()")) { return; } for (int pass = 0; pass < MAX_DRAIN_ALL_PASSES; pass++) { @@ -411,6 +430,22 @@ void drainAll() { } } + /** + * Test seam: how many invocations the scheduler still holds a reference to. + * + *

    {@link #queue} is the only collection of per-invocation objects the scheduler has, so this is the whole of the + * per-invocation state the environment retains. Zero means the environment — which outlives every invocation — + * holds nothing belonging to any invocation it has served. + */ + synchronized int retainedInvocationCount() { + return queue.size(); + } + + /** Test seam: whether the scheduler still holds a reference to one particular invocation. */ + synchronized boolean retains(InsightPlugin execution) { + return queue.contains(execution); + } + /** Gives up one invocation's outstanding work: drops its queued record and releases every drain waiting on it. */ private void abandon(InsightPlugin execution) { CompletableFuture signal; @@ -648,13 +683,15 @@ private void signalSettled(InsightPlugin execution) { *

    A queue that never runs dry cannot starve a request either: the pump alternates one record and one batch of * requests, so a flush waits at most one export fan-out. * - *

    Called from the pump thread itself — which only something the pump invokes synchronously can do — the request - * is refused and reported instead of made: see {@link #refuseWaitFromThePumpThread}. + *

    Called from the pump thread itself — or from an exporter fan-out worker that pump is waiting for, which is + * what a callback re-entering the scheduler does when two or more exporters are configured — the request is refused + * and reported instead of made: see {@link #refuseWaitThatWouldBlockThePump}. */ void flush() { - // Re-entered from the pump: this thread is the only one that could serve the request it is about to make, so it - // must not make it. Refuse and return rather than enqueue a request nobody can serve. - if (refuseWaitFromThePumpThread("flush()")) { + // Re-entered from a thread the pump's progress depends on: the pump is the only thread that could serve the + // request, and it cannot while this caller has not returned. Refuse rather than enqueue a request nobody + // serves. + if (refuseWaitThatWouldBlockThePump("flush()")) { return; } CompletableFuture request = new CompletableFuture<>(); @@ -737,12 +774,15 @@ private void exportToAll(WorkflowInsightRecord record) { /** Runs the action for every exporter concurrently and returns once all have settled, reporting each failure. */ private void forEachExporterSettled(Consumer action) { if (exporters.size() == 1) { + // On the pump thread itself, which the pump-thread check already refuses waits from. runSafely(() -> action.accept(exporters.get(0))); return; } List> settledExporters = new ArrayList<>(exporters.size()); for (InsightExporter exporter : exporters) { - Runnable task = () -> runSafely(() -> action.accept(exporter)); + // Marked as a fan-out task: the pump joins every one of these below, so a wait for the pump issued from + // inside one must be refused exactly as one issued from the pump itself. + Runnable task = () -> runSafely(() -> runAsExporterFanOut(() -> action.accept(exporter))); try { settledExporters.add(CompletableFuture.runAsync(task, executor)); } catch (Throwable t) { @@ -755,6 +795,26 @@ private void forEachExporterSettled(Consumer action) { } } + /** + * Runs one exporter's share of a fan-out with this thread marked pump-dependent, restoring the previous mark on the + * way out. The mark is what makes {@link #refuseWaitThatWouldBlockThePump} recognize a fan-out worker. + */ + private void runAsExporterFanOut(Runnable action) { + Boolean previous = exporterFanOutThread.get(); + exporterFanOutThread.set(Boolean.TRUE); + try { + action.run(); + } finally { + if (previous == null) { + // Removed rather than set back to null: these run on a shared, process-wide pool, so a thread must not + // keep an entry for this scheduler after its task ends. + exporterFanOutThread.remove(); + } else { + exporterFanOutThread.set(previous); + } + } + } + private void runSafely(Runnable action) { try { action.run(); @@ -772,34 +832,48 @@ private void reportFailure(Throwable t) { } /** - * Reports and refuses a wait for the pump that was issued from the pump. Returns whether the caller is the - * pump thread; when it is, the failure has already been reported and the caller must return without waiting. + * Reports and refuses a wait for the pump that was issued from a thread the pump's own progress depends on. Returns + * whether the caller is such a thread; when it is, the failure has already been reported and the caller must return + * without waiting. * - *

    Invariant: the thread that waits for the pump is never the thread that serves it. {@link #flush()} waits for a - * request only a pump can complete, and a drain waits for a signal only a pump can complete or for the running + *

    Invariant: the thread that waits for the pump is never a thread the pump waits for. {@link #flush()} waits for + * a request only a pump can complete, and a drain waits for a signal only a pump can complete or for the running * pump's own handle. All three are satisfied by the pump between records. * - *

    Without this, a wait issued from the pump is a wait-for cycle one thread wide: the pump parks on the future it - * would itself have completed, so it never reaches the point in its loop that completes it, and no other thread may - * take over because {@code inFlight} is this pump's. The invocation never returns, and nothing reports it — a - * {@link CompletableFuture} park cycle is not a monitor deadlock, so the JVM's deadlock detection cannot see it. - * Reachable through anything the pump calls synchronously: with a single exporter the fan-out runs on the pump - * thread, so a customer exporter's {@code export()} that asks for a flush, or a non-conforming {@code exportOne}, - * is enough. A conforming production {@code exportOne} does not re-enter the scheduler, so this is hardening. + *

    Two threads qualify. The pump thread itself: a wait issued from there is a wait-for cycle one thread wide — + * the pump parks on the future it would itself have completed, so it never reaches the point in its loop that + * completes it, and no other thread may take over because {@code inFlight} is this pump's. And an exporter fan-out + * worker: with two or more exporters the pump submits one task per exporter and joins them all, so a wait issued + * from a callback running on one of those workers is the same cycle two threads wide — the worker parks on a future + * only the pump can complete, and the pump is parked in the join waiting for that worker. Neither is a monitor + * deadlock, so the JVM's deadlock detection cannot see either one, and the invocation simply never returns. + * + *

    Reachable through anything a fan-out calls synchronously: with a single exporter the fan-out runs on the pump + * thread, so a customer exporter's {@code export()} or {@code flush()} that asks the scheduler for a flush, or a + * non-conforming {@code exportOne}, is enough; with several it runs on a worker instead, and the same call is + * refused for the same reason. A conforming production {@code exportOne} does not re-enter the scheduler, so this + * is hardening. * *

    So the call fails fast instead: the plugin's failure handler is told — it logs — and the caller returns as it * would from any other flush or drain, with nothing propagating into the execution. The queued work itself is not - * dropped by refusing a drain: the record stays in the invocation's slot and the pump asking the question is the - * one that will export it. Callers that are not the pump — every SDK hook thread — never enter this branch and - * behave exactly as before, and the check is a single volatile read, so no lock is added to that path. + * dropped by refusing a drain: the record stays in the invocation's slot, and the pump that is waiting for this + * caller exports it as soon as this caller returns and the fan-out it belongs to settles. Callers that are neither + * — every SDK hook thread — never enter this branch and behave exactly as before; the check is a volatile read plus + * a thread-local read, so no lock is added to that path. */ - private boolean refuseWaitFromThePumpThread(String call) { - if (pumpThread.get() != Thread.currentThread()) { - return false; - } - reportFailure(new IllegalStateException(call - + " was called from the export pump thread, the only thread able to serve it; the call was refused" - + " rather than deadlocking the invocation")); - return true; + private boolean refuseWaitThatWouldBlockThePump(String call) { + if (pumpThread.get() == Thread.currentThread()) { + reportFailure(new IllegalStateException(call + + " was called from the export pump thread, the only thread able to serve it; the call was refused" + + " rather than deadlocking the invocation")); + return true; + } + if (Boolean.TRUE.equals(exporterFanOutThread.get())) { + reportFailure(new IllegalStateException(call + + " was called from an exporter fan-out worker the export pump is waiting for, so the pump cannot" + + " serve it; the call was refused rather than deadlocking the invocation")); + return true; + } + return false; } } diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFanOutReentryTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFanOutReentryTest.java new file mode 100644 index 000000000..973fe97d3 --- /dev/null +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/ExportSchedulerFanOutReentryTest.java @@ -0,0 +1,199 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.Executor; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import org.junit.jupiter.api.Test; + +/** + * A {@code flush()} or {@code drain()} issued from an exporter fan-out worker is refused and reported, not waited on. + * + *

    With two or more exporters configured, the pump does not run the exporter callbacks itself: it submits one task + * per exporter and then waits for all of them. A callback therefore runs on a worker the pump is blocked on, and a wait + * for the pump issued from that worker is a wait-for cycle two threads wide — the worker parks on a future only the + * pump can complete, and the pump cannot resume its loop until that worker returns. The single-exporter case runs the + * callback on the pump thread itself and is covered by {@link ExportSchedulerReentrantFlushTest}; this covers the + * fan-out, which the pump-thread identity check alone does not recognize. + */ +class ExportSchedulerFanOutReentryTest { + + /** Longest a call that must return promptly may take before the property under test is considered broken. */ + private static final long DEADLINE_MILLIS = 5_000; + + private static String arn(int index) { + return "arn:aws:lambda:us-west-2:111122223333:function:f:$LATEST/durable-execution/exec-" + index + "/inv-1"; + } + + private static WorkflowInsightRecord record(String executionArn, String status) { + var r = new WorkflowInsightRecord(); + r.executionArn = executionArn; + r.status = status; + return r; + } + + /** Unbounded thread-per-task executor, like the cached pool the plugin injects in production. */ + private static Executor sharedWorkers() { + return command -> { + var thread = new Thread(command, "fan-out-reentry-worker"); + thread.setDaemon(true); + thread.start(); + }; + } + + private static final class CountingExporter implements InsightExporter { + final AtomicInteger exports = new AtomicInteger(); + final AtomicInteger flushes = new AtomicInteger(); + + @Override + public void export(WorkflowInsightRecord record) { + exports.incrementAndGet(); + } + + @Override + public void flush() { + flushes.incrementAndGet(); + } + } + + @Test + void flushReenteredFromAFanOutWorkerIsRefusedReportedAndLosesNoWork() throws Exception { + var first = new CountingExporter(); + var second = new CountingExporter(); + var failures = new CopyOnWriteArrayList(); + var holder = new AtomicReference(); + var reentrantFlushReturned = new CountDownLatch(1); + var flushesSeenByTheRefusedCall = new AtomicInteger(-1); + var reentered = new AtomicInteger(); + + var scheduler = new ExportScheduler( + List.of(first, second), + (rec, exp) -> { + exp.export(rec); + // Only the first exporter re-enters, and only once, so exactly one refusal is expected. + if (exp == first && "SUCCEEDED".equals(rec.status()) && reentered.getAndIncrement() == 0) { + holder.get().flush(); + flushesSeenByTheRefusedCall.set(first.flushes.get() + second.flushes.get()); + reentrantFlushReturned.countDown(); + } + }, + failures::add, + sharedWorkers()); + holder.set(scheduler); + + var drainReturned = new CountDownLatch(1); + var firstExecution = Executions.plugin(scheduler, arn(0)); + var invocation = new Thread( + () -> { + scheduler.schedule(firstExecution, record(arn(0), "SUCCEEDED")); + scheduler.drain(firstExecution); + drainReturned.countDown(); + }, + "fan-out-reentry-invocation"); + invocation.setDaemon(true); + invocation.start(); + + assertTrue( + reentrantFlushReturned.await(DEADLINE_MILLIS, MILLISECONDS), + "flush() re-entered from an exporter fan-out worker never returned: the pump is waiting for that" + + " worker, so nothing can serve the request it made"); + assertTrue( + drainReturned.await(DEADLINE_MILLIS, MILLISECONDS), + "drain() never returned after the re-entrant flush"); + invocation.join(DEADLINE_MILLIS); + + assertEquals(1, failures.size(), "exactly one failure reported: " + failures); + assertTrue( + failures.get(0) instanceof IllegalStateException, + "the refusal is reported as an IllegalStateException: " + failures.get(0)); + assertTrue( + failures.get(0).getMessage().contains("flush()"), + "the report names the refused call: " + failures.get(0).getMessage()); + assertEquals(0, flushesSeenByTheRefusedCall.get(), "the refused request must not have reached an exporter"); + + // No work lost: the record that was in flight reached both exporters. + assertEquals(1, first.exports.get(), "the record reached the first exporter"); + assertEquals(1, second.exports.get(), "the record reached the second exporter"); + + // Still usable from a thread that is not pump-dependent. + var secondExecution = Executions.plugin(scheduler, arn(1)); + scheduler.schedule(secondExecution, record(arn(1), "SUCCEEDED")); + scheduler.drain(secondExecution); + scheduler.flush(); + + assertEquals(2, first.exports.get(), "both records reached the first exporter"); + assertEquals(2, second.exports.get(), "both records reached the second exporter"); + assertEquals(1, first.flushes.get(), "the later flush is served normally on the first exporter"); + assertEquals(1, second.flushes.get(), "the later flush is served normally on the second exporter"); + assertEquals(1, failures.size(), "no further failure after the refusal: " + failures); + } + + @Test + void drainReenteredFromAFanOutWorkerIsRefusedReportedAndLosesNoWork() throws Exception { + var first = new CountingExporter(); + var second = new CountingExporter(); + var failures = new CopyOnWriteArrayList(); + var holder = new AtomicReference(); + var pluginHolder = new AtomicReference(); + var reentrantDrainReturned = new CountDownLatch(1); + var reentered = new AtomicInteger(); + + var scheduler = new ExportScheduler( + List.of(first, second), + (rec, exp) -> { + exp.export(rec); + if (exp == first && "SUCCEEDED".equals(rec.status()) && reentered.getAndIncrement() == 0) { + holder.get().drain(pluginHolder.get()); + reentrantDrainReturned.countDown(); + } + }, + failures::add, + sharedWorkers()); + holder.set(scheduler); + + var execution = Executions.plugin(scheduler, arn(0)); + pluginHolder.set(execution); + + var drainReturned = new CountDownLatch(1); + var invocation = new Thread( + () -> { + scheduler.schedule(execution, record(arn(0), "SUCCEEDED")); + scheduler.drain(execution); + drainReturned.countDown(); + }, + "fan-out-reentry-drain-invocation"); + invocation.setDaemon(true); + invocation.start(); + + assertTrue( + reentrantDrainReturned.await(DEADLINE_MILLIS, MILLISECONDS), + "drain() re-entered from an exporter fan-out worker never returned: the pump is waiting for that" + + " worker, so nothing can settle the signal it waited for"); + assertTrue( + drainReturned.await(DEADLINE_MILLIS, MILLISECONDS), + "the invocation's own drain() never returned after the re-entrant drain"); + invocation.join(DEADLINE_MILLIS); + + assertEquals(1, failures.size(), "exactly one failure reported: " + failures); + assertTrue( + failures.get(0) instanceof IllegalStateException, + "the refusal is reported as an IllegalStateException: " + failures.get(0)); + assertTrue( + failures.get(0).getMessage().contains("drain"), + "the report names the refused call: " + failures.get(0).getMessage()); + + // No work lost by refusing the drain: the record still reached every exporter, and the invocation's own drain + // returned only once it had. + assertEquals(1, first.exports.get(), "the record reached the first exporter"); + assertEquals(1, second.exports.get(), "the record reached the second exporter"); + assertTrue(!Executions.outstanding(execution), "the scheduler owes the invocation nothing after its drain"); + } +} diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/StateCleanupLifecycleTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/StateCleanupLifecycleTest.java index 3775cbf4e..e52504867 100644 --- a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/StateCleanupLifecycleTest.java +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/StateCleanupLifecycleTest.java @@ -4,11 +4,14 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertTrue; import java.lang.ref.WeakReference; +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; import java.time.Instant; import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -27,17 +30,28 @@ * *

    The plugin used to keep that state in an ARN-keyed map and remove the entry at every invocation end, so the test * counted the entries left behind. There is no map now — an invocation's state is its plugin instance, which - * the SDK creates per invocation and drops when it returns — so the two things worth proving are that the environment - * (the factory's scheduler, which does outlive invocations) owes a finished invocation nothing, and that it holds no - * reference to the instance once the invocation is over. A retained entry of any kind would fail the second assertion, - * which the old count could not make: it could only count the entries the plugin knew it had. + * the SDK creates per invocation and drops when it returns — so what is left to prove is about the one object that does + * outlive invocations: the factory's {@link ExportScheduler}. Two things are asserted, both read directly out of that + * scheduler under the monitor its fields are guarded by. First, that it owes a finished invocation nothing: no queued + * record, nothing inside the exporters, no uncompleted drain signal, no drain waiting. Second, that it holds no + * reference to the instance: {@link ExportScheduler#queue} is the only collection of per-invocation objects it has, so + * an empty queue after every invocation has ended is "the environment retains nothing", and a retained entry + * of any kind would fail it — which the old count could not do, because it could only count the entries the plugin knew + * it had. + * + *

    Reachability from the scheduler is what determines whether state accumulates, and that is a fact about the + * scheduler's own fields, not about the collector. Whether the JVM has actually reclaimed a finished instance is + * reported below as a diagnostic and never asserted: {@link System#gc()} is a request the JVM is free to ignore, so an + * implementation that retains nothing can still leave every weak reference set, and asserting on it would fail the + * build on garbage-collector behaviour rather than on this plugin's. */ class StateCleanupLifecycleTest { private static final Instant START = Instant.parse("2026-08-05T00:00:00Z"); private static final class CapturingExporter implements InsightExporter { - final List records = new ArrayList<>(); + /** Written on pump threads, read on the test thread after a drain; synchronized so the reads are sound. */ + final List records = Collections.synchronizedList(new ArrayList<>()); @Override public void export(WorkflowInsightRecord record) { @@ -45,6 +59,18 @@ public void export(WorkflowInsightRecord record) { } } + /** + * An execution environment that emits on every change, so each invocation below really does put records through the + * scheduler. With the default {@code ON_COMPLETE} mode a suspending invocation emits nothing, and "the environment + * retains nothing" would hold trivially because nothing was ever queued. + */ + private static DurableExecutionPluginFactory emittingEnvironment(CapturingExporter exporter) { + return WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .addExporter(exporter) + .build()); + } + private static String arn(int i) { return "arn:aws:lambda:us-west-2:1:function:f:$LATEST/durable-execution/exec-" + i + "/invocation-1"; } @@ -77,68 +103,158 @@ private static InvocationEndInfo end(int i, InvocationStatus status) { return new InvocationEndInfo("req", arn(i), true, START, ops(), status, null, "in-" + i, null); } + /** What a finished invocation leaves behind: the environment that served it, and a way to observe reclamation. */ + private record Finished(ExportScheduler environment, WeakReference instance) {} + /** - * Runs one whole invocation in the given environment and returns a weak reference to the instance that served it, - * keeping no strong reference of its own — so whatever the reference still points at afterwards is retained by the - * environment, not by this test. + * Runs one whole invocation in the given environment and returns the environment's scheduler plus a weak reference + * to the instance that served it, keeping no strong reference of its own — so whatever that reference still points + * at afterwards is retained by the environment, not by this test. + * + *

    Both assertions are made here, while the instance is still in hand: the scheduler's per-invocation fields for + * this instance are all clear, and the scheduler's queue does not contain it. Those are the two halves of one + * documented invariant — an invocation is in the queue exactly while its record is non-null — so checking both + * catches a state that satisfies one and not the other. */ - private static WeakReference runInvocation( - DurableExecutionPluginFactory environment, int i, InvocationStatus status) { + private static Finished runInvocation(DurableExecutionPluginFactory environment, int i, InvocationStatus status) { InsightPlugin plugin = Executions.started(environment, start(i)); plugin.onInvocationEnd(end(i, status)); + ExportScheduler scheduler = plugin.scheduler; assertFalse(Executions.outstanding(plugin), "the scheduler still owes execution " + i + " work"); - return new WeakReference<>(plugin); + assertFalse(scheduler.retains(plugin), "the environment still holds a reference to execution " + i); + return new Finished(scheduler, new WeakReference<>(plugin)); } - /** True once every instance has been collected; polls, because a single GC request need not clear them. */ - private static boolean allCollected(List> instances) { - for (int attempt = 0; attempt < 50; attempt++) { - if (instances.stream().allMatch(reference -> reference.get() == null)) { - return true; - } - System.gc(); - try { - Thread.sleep(10); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - return false; + /** + * Diagnostic only, never an assertion: how many finished instances the JVM has reclaimed after being asked to. Kept + * because it is the observation that first exposed the retained-state finding, and printed so a regression is + * visible in the build log without a nondeterministic failure. + */ + private static void reportReclamation(String scenario, List finished) { + System.gc(); + try { + Thread.sleep(10); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + long reclaimed = + finished.stream().filter(f -> f.instance().get() == null).count(); + System.out.printf( + "DIAGNOSTIC %s: %d of %d finished plugin instances reclaimed after a System.gc() request%n", + scenario, reclaimed, finished.size()); + } + + /** + * Every plugin instance the scheduler still reaches through any of its fields, described as {@code field -> + * plugin}. + * + *

    The seams above answer the same question for the one collection the scheduler is known to keep. This finds the + * collection it is not known to keep: a registry reintroduced under any name, keyed by execution ARN or + * otherwise, shows up here as soon as it holds an instance. That is what makes reachability, rather than + * collection, the thing this test asserts — and it is deterministic, unlike asking the collector. + * + *

    Read under the scheduler's monitor, which is the monitor its per-invocation fields are guarded by. + */ + private static List pluginsReachableFrom(ExportScheduler scheduler) { + var reachable = new ArrayList(); + synchronized (scheduler) { + for (Field field : ExportScheduler.class.getDeclaredFields()) { + if (Modifier.isStatic(field.getModifiers())) { + continue; + } + field.setAccessible(true); + Object value; + try { + value = field.get(scheduler); + } catch (ReflectiveOperationException e) { + throw new AssertionError("could not read ExportScheduler." + field.getName(), e); + } + for (Object element : elementsOf(value)) { + if (element instanceof InsightPlugin plugin) { + reachable.add(field.getName() + " -> " + plugin); + } + } } } - return instances.stream().allMatch(reference -> reference.get() == null); + return reachable; + } + + /** The elements a field value exposes, so a collection or map of any shape can be inspected uniformly. */ + private static Collection elementsOf(Object value) { + if (value instanceof Collection collection) { + return new ArrayList(collection); + } + if (value instanceof Map map) { + var elements = new ArrayList(map.keySet()); + elements.addAll(map.values()); + return elements; + } + return List.of(); } @Test void nDistinctPendingExecutionsLeaveNoRetainedState() { - var environment = - WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder().build()); + var exporter = new CapturingExporter(); + var environment = emittingEnvironment(exporter); int n = 25; - var instances = new ArrayList>(); + var finished = new ArrayList(); for (int i = 0; i < n; i++) { // Each execution suspends (PENDING) and never terminates in this container. - instances.add(runInvocation(environment, i, InvocationStatus.PENDING)); + finished.add(runInvocation(environment, i, InvocationStatus.PENDING)); } - assertTrue( - allCollected(instances), - "the environment still holds the state of a suspended execution after its invocation ended"); + ExportScheduler scheduler = finished.get(0).environment(); + // Quiesce: returns once nothing is queued and no pump owns the scheduler, so the count below is read at a point + // where a still-running pump cannot be mistaken for retained state. + scheduler.drainAll(); + + // Every invocation really did put records through the scheduler, so the assertions below are about state that + // existed and was released, not state that was never created. Counted by distinct execution rather than by + // record: a RUNNING snapshot that the end record supersedes before any pump takes it is coalesced away by + // design, so the number of records is not fixed, but every invocation drains its own final record. + assertEquals( + n, + exporter.records.stream() + .map(WorkflowInsightRecord::executionArn) + .distinct() + .count(), + "every invocation delivered at least one record through the environment's scheduler"); + assertEquals( + 0, + scheduler.retainedInvocationCount(), + "the environment still holds per-invocation state after all " + n + " invocations ended"); + assertEquals( + List.of(), + pluginsReachableFrom(scheduler), + "the environment still reaches plugin instances after all " + n + " invocations ended"); + reportReclamation(n + " pending executions", finished); } @Test void retryingSuspendAlsoLeavesNoRetainedState() { - var environment = - WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder().build()); - var instance = runInvocation(environment, 0, InvocationStatus.RETRYING); - assertTrue(allCollected(List.of(instance)), "a RETRYING suspend leaves nothing retained either"); + var exporter = new CapturingExporter(); + var environment = emittingEnvironment(exporter); + + var finished = runInvocation(environment, 0, InvocationStatus.RETRYING); + + finished.environment().drainAll(); + assertFalse(exporter.records.isEmpty(), "the invocation put at least one record through the scheduler"); + assertEquals( + 0, + finished.environment().retainedInvocationCount(), + "a RETRYING suspend leaves the environment holding per-invocation state"); + assertEquals( + List.of(), + pluginsReachableFrom(finished.environment()), + "a RETRYING suspend leaves the environment reaching its plugin instance"); + reportReclamation("one retrying execution", List.of(finished)); } @Test void resumeReSeedsStableStartTimeAndInput() { var exporter = new CapturingExporter(); - var environment = WorkflowInsight.workflowInsight(WorkflowInsightConfig.builder() - .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) - .addExporter(exporter) - .build()); + var environment = emittingEnvironment(exporter); // First invocation with input "alpha", then suspend. Its instance is dropped with it. var first = Executions.started( @@ -157,5 +273,14 @@ void resumeReSeedsStableStartTimeAndInput() { assertEquals(START.toString(), terminal.startTime(), "stable start time recreated across the suspend boundary"); assertEquals("alpha", terminal.input, "input re-seeded from resume onInvocationStart"); assertFalse(Executions.outstanding(resumed), "the terminal end leaves the scheduler owing nothing"); + assertFalse(resumed.scheduler.retains(resumed), "the environment holds no reference to the resumed invocation"); + assertEquals( + 0, + resumed.scheduler.retainedInvocationCount(), + "neither the suspended invocation nor the resumed one is retained by the environment"); + assertEquals( + List.of(), + pluginsReachableFrom(resumed.scheduler), + "the environment reaches neither the suspended invocation nor the resumed one"); } } diff --git a/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java b/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java index 01b1540d9..aab208fdf 100644 --- a/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java +++ b/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java @@ -837,6 +837,57 @@ void throwingPlugin_doesNotDisruptExecution() { assertFalse(recordingPlugin.invocationEnds.isEmpty()); } + @Test + void factoryThrowingLinkageError_doesNotDisruptExecution() { + // A provider whose optional dependency is missing from the deployment package fails this way. A LinkageError is + // an Error, not an Exception, so containment that catches only Exception lets it escape onInvocationStart and + // fail the whole execution. + var recordingPlugin = new RecordingPlugin(); + var config = DurableConfig.builder() + .withPlugins( + info -> { + throw new NoClassDefFoundError("software/amazon/example/OptionalExporter"); + }, + info -> recordingPlugin) + .build(); + + var runner = LocalDurableTestRunner.create( + String.class, (input, context) -> context.step("step", String.class, stepCtx -> "safe"), config); + + var result = runner.runUntilComplete("input"); + + assertEquals(ExecutionStatus.SUCCEEDED, result.getStatus()); + assertEquals("safe", result.getResult(String.class)); + assertFalse(recordingPlugin.invocationStarts.isEmpty(), "the surviving plugin must still receive its hooks"); + assertFalse(recordingPlugin.invocationEnds.isEmpty()); + } + + @Test + void factoryThrowingAbstractMethodError_doesNotDisruptExecution() { + // What a provider compiled against an earlier version of the factory interface throws the first time the SDK + // invokes the method it does not implement — the exact failure this SDK's factory-only plugin contract creates + // for a provider that has not been recompiled. + var recordingPlugin = new RecordingPlugin(); + var config = DurableConfig.builder() + .withPlugins( + info -> { + throw new AbstractMethodError( + "software.amazon.example.LegacyProvider.createPlugin(InvocationInfo)"); + }, + info -> recordingPlugin) + .build(); + + var runner = LocalDurableTestRunner.create( + String.class, (input, context) -> context.step("step", String.class, stepCtx -> "safe"), config); + + var result = runner.runUntilComplete("input"); + + assertEquals(ExecutionStatus.SUCCEEDED, result.getStatus()); + assertEquals("safe", result.getResult(String.class)); + assertFalse(recordingPlugin.invocationStarts.isEmpty(), "the surviving plugin must still receive its hooks"); + assertFalse(recordingPlugin.invocationEnds.isEmpty()); + } + // ─── Child context hooks ───────────────────────────────────────────── @Test diff --git a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java index 3edfb0679..de281eea6 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java @@ -19,7 +19,10 @@ * execution ARN. * *

    Event hooks are fire-and-forget: each plugin is called in order, errors are swallowed. A factory that throws or - * returns {@code null} is contained the same way — the plugin is skipped for the invocation. + * returns {@code null} is contained the same way — the plugin is skipped for the invocation. Containment covers + * {@link LinkageError} as well as {@link Exception}, because a plugin built against a different SDK version or missing + * an optional dependency fails with an {@code Error}; it deliberately stops short of the {@code Error}s that report the + * JVM itself failing. * *

    {@code onInvocationEnd} is awaited (the SDK blocks until it returns) to allow plugins to flush data before Lambda * freezes. @@ -57,6 +60,15 @@ public boolean isEmpty() { * *

    Called from {@link #onInvocationStart(InvocationInfo)} so the instances exist before any hook is dispatched. * Factories that throw or return null are logged and skipped. + * + *

    {@link LinkageError} is contained alongside {@link Exception} because it is how the two most likely + * version-skew failures of this contract present themselves, and neither is an {@code Exception}: a provider JAR + * compiled against an earlier version of {@link DurableExecutionPluginFactory} throws {@link AbstractMethodError} + * when the SDK invokes the method it does not implement, and a provider whose optional dependency is missing from + * the deployment package throws {@link NoClassDefFoundError} while building its plugin. The contract says a factory + * failure is skipped and never disrupts the execution, so both are skipped. Deliberately narrow: an {@code Error} + * that reports the JVM itself failing — {@link OutOfMemoryError}, {@link StackOverflowError} — is not a plugin + * defect and must keep propagating rather than be logged as one. */ private void createPlugins(InvocationInfo info) { var created = new ArrayList(pluginFactories.size()); @@ -68,8 +80,8 @@ private void createPlugins(InvocationInfo info) { continue; } created.add(plugin); - } catch (Exception e) { - logger.warn("Plugin factory threw exception; skipping it for this invocation", e); + } catch (Exception | LinkageError e) { + logger.warn("Plugin factory failed; skipping it for this invocation", e); } } this.plugins = List.copyOf(created); @@ -78,6 +90,9 @@ private void createPlugins(InvocationInfo info) { /** * Drops this invocation's plugin instances. Called when the invocation returns so the instances are unreachable * from the SDK and cannot leak into the next invocation the environment hosts. + * + *

    No containment here: this only replaces the field, and calls nothing on the plugins it drops. There is no + * {@code close()} in the plugin contract, so releasing cannot run plugin code and cannot fail. */ public void releasePlugins() { this.plugins = List.of(); @@ -85,13 +100,20 @@ public void releasePlugins() { // ─── Event hooks ───────────────────────────────────────────────────── - /** Calls a void hook on all of this invocation's plugins, swallowing any errors. */ + /** + * Calls a void hook on all of this invocation's plugins, swallowing any errors. + * + *

    {@link LinkageError} is contained alongside {@link Exception} for the reason given on {@link #createPlugins}: + * a plugin compiled against a different SDK version, or one missing an optional dependency, fails a hook with an + * {@code Error} rather than an {@code Exception}, and the fire-and-forget contract makes no distinction. Narrow on + * purpose — {@link OutOfMemoryError} and {@link StackOverflowError} still propagate. + */ private void run(Consumer hook) { for (var plugin : plugins) { try { hook.accept(plugin); - } catch (Exception e) { - logger.warn("Plugin hook threw exception", e); + } catch (Exception | LinkageError e) { + logger.warn("Plugin hook failed", e); } } } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java index cf6f9deb1..e9e9e8e32 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java @@ -172,6 +172,85 @@ void nullReturningFactory_isContained_andRemainingPluginsStillRun() { assertEquals(List.of("p2:onInvocationStart", "p2:onOperationStart"), calls); } + // ─── Factory and hook linkage failures ─────────────────────────────── + // + // A LinkageError is an Error, not an Exception, so a catch of Exception does not contain it. Both of the shapes + // below are reachable through the plugin contract rather than hypothetical: a provider JAR compiled against an + // earlier version of DurableExecutionPluginProvider throws AbstractMethodError the first time the SDK invokes the + // method it does not implement, and a provider whose optional dependency is absent from the deployment package + // throws NoClassDefFoundError when it first touches that class. Both must be contained, because the contract says a + // factory or hook failure is logged and skipped and never disrupts the execution. + + @Test + void factoryThrowingAbstractMethodError_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> { + // What a provider compiled against the previous interface throws when the new factory method is + // invoked on it. + throw new AbstractMethodError( + "software.amazon.example.LegacyProvider.createPlugin(InvocationInfo)"); + }, + info -> new TestPlugin("p2", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + runner.onInvocationEnd(invocationEndInfo()); + + assertEquals(List.of("p2:onInvocationStart", "p2:onInvocationEnd"), calls); + } + + @Test + void factoryThrowingNoClassDefFoundError_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> { + // What a provider with a missing optional dependency throws while building its plugin. + throw new NoClassDefFoundError("software/amazon/example/OptionalExporter"); + }, + info -> new TestPlugin("p2", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + runner.onInvocationEnd(invocationEndInfo()); + + assertEquals(List.of("p2:onInvocationStart", "p2:onInvocationEnd"), calls); + } + + @Test + void hookThrowingLinkageError_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> new LinkageErrorPlugin(), + info -> new TestPlugin("p2", calls), + info -> new TestPlugin("p3", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + calls.clear(); + assertDoesNotThrow(() -> runner.onOperationStart(operationInfo())); + assertDoesNotThrow(() -> runner.onInvocationEnd(invocationEndInfo())); + + assertEquals( + List.of("p2:onOperationStart", "p3:onOperationStart", "p2:onInvocationEnd", "p3:onInvocationEnd"), + calls); + } + + @Test + void factoryThrowingAJvmError_stillPropagates() { + // The containment is deliberately narrow: an Error that says the JVM itself is failing must not be swallowed as + // if it were a plugin defect, because the process cannot be assumed able to continue. + var runner = new PluginRunner(List.of(info -> { + throw new OutOfMemoryError("Java heap space"); + })); + + assertThrows(OutOfMemoryError.class, () -> runner.onInvocationStart(invocationInfo())); + } + + @Test + void hookThrowingAJvmError_stillPropagates() { + var runner = new PluginRunner(List.of(info -> new StackOverflowPlugin())); + + assertThrows(StackOverflowError.class, () -> runner.onInvocationStart(invocationInfo())); + } + // ─── Fire-and-forget event hooks ───────────────────────────────────── @Test @@ -428,4 +507,33 @@ public void onInvocationEnd(InvocationEndInfo info) { throw new RuntimeException("boom"); } } + + /** + * Plugin whose hooks fail to link, as a plugin compiled against a different SDK version or missing an optional + * dependency does. + */ + private static class LinkageErrorPlugin implements DurableExecutionPlugin { + @Override + public void onInvocationStart(InvocationInfo info) { + throw new NoClassDefFoundError("software/amazon/example/OptionalExporter"); + } + + @Override + public void onOperationStart(OperationInfo info) { + throw new AbstractMethodError("software.amazon.example.LegacyPlugin.onOperationStart(OperationInfo)"); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + throw new IncompatibleClassChangeError("software.amazon.example.LegacyPlugin"); + } + } + + /** Plugin whose hook reports that the JVM itself is failing, which must not be contained. */ + private static class StackOverflowPlugin implements DurableExecutionPlugin { + @Override + public void onInvocationStart(InvocationInfo info) { + throw new StackOverflowError(); + } + } } From c4428fe261ab2f87e8a3af80f5223263097f66a4 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Thu, 17 Sep 2026 15:25:17 -0700 Subject: [PATCH 04/19] docs: add the 2.x to 3.x plugin migration guide Review accepted shipping the factory-only plugin contract as a new major rather than keeping deprecated bridges, on the condition that the required changes are documented. This is that document. It covers the three paths a consumer can be on. Direct registration: `withPlugins(new MyPlugin())` becomes `withPlugins(info -> new MyPlugin())`, with worked examples for environment-lifetime state held outside the lambda and for per-invocation state that used to sit in an ARN-keyed map. Service providers: the before and after of a provider, now that the interface keeps only `getName()` and inherits `createPlugin(InvocationInfo)`. And `PluginRunner`, which was never customer API and should not be reached for at all. Two things the guide states that are easy to get wrong. A provider JAR that is not rebuilt still loads and is still selected, because its class file references nothing 3.x removed. Its inherited `createPlugin(InvocationInfo)` then throws `AbstractMethodError`, which the SDK now contains and logs, so the execution succeeds with no instrumentation at all. A deployment can therefore lose its telemetry while every invocation reports success, and the only signal is a per-invocation warning. That is the failure mode an operator most needs to know about, so it gets its own section along with three ways to confirm a provider actually loaded. A plugin instance does not survive suspension. A resume is a new invocation with a new instance, so state that must be stable across an execution has to be derived from `InvocationInfo` on each invocation rather than accumulated. Every "after" snippet was compiled against this branch rather than written from memory, which caught two errors in the drafts. Compiling the "before" snippets confirmed something worth stating in the guide: `DurableExecutionPlugin` declares only default methods, so it is not a functional interface and an instance cannot be silently coerced into a factory. Every direct registration site fails to compile until it is updated, rather than compiling and failing later. Linked from README.md and AGENTS.md alongside the 1.x to 2.x guide. Packaging and ordering details are cross-referenced to docs/advanced/configuration.md rather than duplicated. --- AGENTS.md | 1 + README.md | 1 + docs/migration-2.x-to-3.x.md | 361 +++++++++++++++++++++++++++++++++++ 3 files changed, 363 insertions(+) create mode 100644 docs/migration-2.x-to-3.x.md diff --git a/AGENTS.md b/AGENTS.md index 7894488db..6cc0114c9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -335,6 +335,7 @@ Run `mvn spotless:apply` after Java changes. Then run the narrowest relevant tes - [Error Handling](docs/advanced/error-handling.md) - [Logging](docs/advanced/logging.md) - [Migration from 1.x to 2.x](docs/migration-1.x-to-2.x.md) +- [Migration from 2.x to 3.x](docs/migration-2.x-to-3.x.md) ### Official AWS SDKs diff --git a/README.md b/README.md index 766a71b02..3c48e776b 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,7 @@ See [Deploy Lambda durable functions with Infrastructure as Code](https://docs.a - [Error Handling](docs/advanced/error-handling.md) - SDK exceptions for handling failures - [Logging](docs/advanced/logging.md) - How to use DurableLogger - [Migrating from 1.x to 2.x](docs/migration-1.x-to-2.x.md) - Upgrade guide for breaking changes since `v1.2.1` +- [Migrating from 2.x to 3.x](docs/migration-2.x-to-3.x.md) - Upgrade guide for the factory-only, per-invocation plugin contract - [Release Process](RELEASE.md) - Prepare and publish Maven releases - [Testing](docs/advanced/testing.md) - Utilities for local development and cloud-based integration testing diff --git a/docs/migration-2.x-to-3.x.md b/docs/migration-2.x-to-3.x.md new file mode 100644 index 000000000..084c56b69 --- /dev/null +++ b/docs/migration-2.x-to-3.x.md @@ -0,0 +1,361 @@ +# Migrating from 2.x to 3.x + +This guide helps teams upgrade from the `2.x` line to `3.x`. + +The `3.x` line contains one breaking change: the plugin contract is now factory-only and per-invocation. Nothing outside the plugin surface changed. If your application registers no plugins, ships no plugin provider, and consumes no plugin JAR, you can upgrade the dependency version and stop reading here. + +A plugin instance used to live as long as the execution environment and serve every execution that landed on it. Under Lambda Managed Instances several executions run concurrently in one environment, so a plugin had to key its own state by execution ARN and remove those entries itself. In `3.x` the SDK creates one plugin instance per Lambda invocation and drops it when the invocation returns, so per-invocation state is a plain instance field. + +## There Is No Compatibility Bridge + +`3.x` removes the instance-based registration path rather than keeping it alongside the factory path. There is no deprecated overload, no adapter, and no shim. + +That is deliberate. While an instance path exists, a plugin registered through it still serves several executions at once, so it still needs its ARN-keyed per-execution state and still cannot delete it. Deleting that state is the entire point of the change. Both bundled plugins had a concurrency defect in exactly that ARN-keyed code under Managed Instances, and the measured effect was records lost without any error surfacing. A bridge would have preserved the defect class it was meant to retire. + +The practical consequence is that recompilation against `3.x` is mandatory. Bytecode compiled against `2.x` links against `DurableConfig$Builder.withPlugins(DurableExecutionPlugin[])`, which no longer exists, and fails at runtime with: + +```text +java.lang.NoSuchMethodError: 'software.amazon.lambda.durable.DurableConfig$Builder + software.amazon.lambda.durable.DurableConfig$Builder.withPlugins( + software.amazon.lambda.durable.plugin.DurableExecutionPlugin[])' +``` + +Recompiling against `3.x` turns that runtime failure into a compile error at every call site, which is the outcome you want. `DurableExecutionPlugin` declares only default methods, so it is not a functional interface and a plugin instance cannot be silently coerced into a factory. Every direct registration site therefore fails to compile until it is updated. + +## Upgrade Checklist + +- Replace every `withPlugins(pluginInstance)` argument with a `DurableExecutionPluginFactory`. +- Move plugin state that must outlive one invocation out of the plugin and into the factory's enclosing scope. +- Replace ARN-keyed per-execution maps inside plugins with plain instance fields. +- Replace `DurableConfig.getPluginRunner()` with `DurableConfig.getPluginFactories()`. +- Rebuild every plugin provider JAR against `3.x` and redeploy it, including provider JARs delivered as Lambda layers. +- Update bundled OTel registrations from `new InvocationOtelPlugin(...)` to `InvocationOtelPlugin.factory(...)`. +- Confirm after deployment that each configured provider is still producing telemetry. + +Useful searches before upgrading: + +```bash +rg -n "withPlugins\(" . +rg -n "getPluginRunner|getPlugins\(\)" . +rg -n "DurableExecutionPluginProvider|getApiVersion|getPluginType" . +rg -n "durableExecutionArn\(\)\s*\)|ConcurrentHashMap" --glob '*Plugin*.java' . +``` + +## 1. Register Plugin Factories Instead of Plugin Instances + +`DurableConfig.Builder.withPlugins(DurableExecutionPlugin...)` is replaced by `withPlugins(DurableExecutionPluginFactory...)`. + +`DurableExecutionPluginFactory` is a `@FunctionalInterface` with one method, `DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo)`. A lambda or a constructor reference satisfies it directly, so no adapter class is needed. + +### Stateless plugin + +For a plugin that holds no state, the change is mechanical: wrap the constructor call in a lambda. + +Before: + +```java +@Override +protected DurableConfig createConfiguration() { + return DurableConfig.builder() + .withPlugins(new LoggingPlugin()) + .build(); +} +``` + +After: + +```java +@Override +protected DurableConfig createConfiguration() { + return DurableConfig.builder() + .withPlugins(info -> new LoggingPlugin()) + .build(); +} +``` + +If the plugin's constructor takes exactly one `InvocationInfo` argument, a constructor reference works instead: + +```java +return DurableConfig.builder() + .withPlugins(LoggingPlugin::new) + .build(); +``` + +### State that must be shared across invocations + +Some plugin state belongs to the execution environment rather than to one invocation: an exporter, a connection pool, a background scheduler, a resolved configuration object. That state must not be recreated per invocation. Hold it outside the lambda, and the lambda captures it. + +The handler is constructed once per execution environment, and `createConfiguration()` runs during that construction, so a handler field or a local variable in `createConfiguration()` both have execution-environment lifetime. + +```java +public class AuditingHandler extends DurableHandler { + + // Created once per execution environment, because the handler is. + private final AuditSink sink = new AuditSink(); + + @Override + protected DurableConfig createConfiguration() { + return DurableConfig.builder() + .withPlugins(info -> new AuditPlugin(sink, info)) + .build(); + } + + @Override + public OrderResult handleRequest(Order order, DurableContext ctx) { + // Your handler logic + } +} +``` + +Caveat: sharing an object across invocations means it is reachable from concurrent invocations in the same environment, so it still has to be thread-safe. The change removes the need for ARN keying inside plugin instances; it does not remove the need for thread safety in the objects those instances share. + +### Per-invocation state that used to be keyed by execution ARN + +This is the substantive part of the migration. A `2.x` plugin instance was shared, so per-execution state had to live in a map keyed by execution ARN, and the plugin had to remove the entry itself. + +Before: + +```java +public final class AuditPlugin implements DurableExecutionPlugin { + + private final AuditSink sink = new AuditSink(); + private final Map statesByArn = new ConcurrentHashMap<>(); + + @Override + public void onInvocationStart(InvocationInfo info) { + statesByArn.put( + info.durableExecutionArn(), + new ExecutionState(info.executionStartTime())); + } + + @Override + public void onOperationEnd(OperationEndInfo info) { + // OperationEndInfo carries no execution ARN, so this hook cannot look up + // its own execution's entry at all. + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + var state = statesByArn.remove(info.durableExecutionArn()); + if (state != null) { + sink.write(info.durableExecutionArn(), state.completedOperationIds()); + } + } + + private static final class ExecutionState { + // start time, sampling decision, accumulated operation ids, ... + } +} +``` + +After: + +```java +public final class AuditPlugin implements DurableExecutionPlugin { + + // Environment lifetime: handed in by the factory, shared by every invocation. + private final AuditSink sink; + + // Per-invocation state: plain fields, because this instance serves one invocation. + private final String executionArn; + private final Instant executionStartTime; + private final List completedOperationIds = new CopyOnWriteArrayList<>(); + + AuditPlugin(AuditSink sink, InvocationInfo info) { + this.sink = sink; + this.executionArn = info.durableExecutionArn(); + this.executionStartTime = info.executionStartTime(); + } + + @Override + public void onOperationEnd(OperationEndInfo info) { + completedOperationIds.add(info.id()); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + sink.write(executionArn, completedOperationIds); + } +} +``` + +Migration rules for this shape: + +- Delete the ARN-keyed map. There is nothing left to key: the instance belongs to one invocation. +- Delete the entry-removal code in `onInvocationEnd`. The SDK drops the instance when the invocation returns. +- Move anything the map's value type held into instance fields, and assign them in the constructor from the `InvocationInfo` the factory received. +- Prefer constructor assignment over assignment in `onInvocationStart`. The SDK publishes the plugin instance to the operation, checkpoint, and user function threads with a volatile write before firing the first hook, so a field assigned in the constructor is visible to those threads without being `volatile`. A field assigned inside `onInvocationStart` has no such guarantee for a thread that already existed. +- Collections that hooks mutate still need to be concurrent. Hooks for one invocation fire on several threads. + +Caveat about resumes: a plugin instance does not survive suspension. When an execution suspends on a `wait()` or a callback and later resumes, the resume is a new invocation with a new plugin instance, and any state accumulated in the previous instance is gone. State that has to be stable across the whole execution must be derivable from the `InvocationInfo` of each invocation, not accumulated. `InvocationInfo.executionStartTime()` is stable across all invocations of an execution for exactly this reason, and `InvocationInfo.operations()` carries the checkpointed operations delivered at invocation start. A sampling decision should be computed deterministically from the execution ARN rather than stored. + +The `2.x` code above illustrates a second reason for the change. `OperationInfo`, `OperationEndInfo`, `UserFunctionStartInfo`, and `UserFunctionEndInfo` carry no execution ARN, so a shared plugin instance could not determine which execution an operation-level hook belonged to. With one instance per invocation, that question does not arise. + +### Bundled plugins + +The OTel plugin's public constructors are replaced by static factory methods: + +```java +// Before +DurableConfig.builder().withPlugins(new InvocationOtelPlugin()).build(); + +// After +DurableConfig.builder().withPlugins(InvocationOtelPlugin.factory()).build(); +``` + +The same applies to `ExecutionOtelPlugin` and to the overloads that take an `SdkTracerProviderBuilder` and an `OtelPluginConfig`. See the [OTel plugin README](../otel-plugin/README.md#configuration) for the full set. + +`WorkflowInsight.workflowInsight(config)` now returns a `DurableExecutionPluginFactory` instead of a `DurableExecutionPlugin`, so the registration source line is unchanged: + +```java +DurableConfig.builder() + .withPlugins(WorkflowInsight.workflowInsight(config)) + .build(); +``` + +Caveat: the source line is unchanged but the return type is not, so this call site still has to be recompiled. An un-recompiled caller fails at runtime with `NoSuchMethodError`. + +## 2. `PluginRunner` and `getPluginRunner()` + +`PluginRunner` was never intended as customer API, and the honest migration advice is to stop using it. + +It is public because the SDK dispatches hooks to it from packages other than the one it lives in. It has no documented compatibility guarantee, and this release changed it without a deprecation cycle. Treat it as an SDK internal. + +What actually changed: + +- `DurableConfig.getPluginRunner()` is removed. `DurableConfig.getPluginFactories()` replaces it and returns an immutable `List` in dispatch order. +- `PluginRunner.getPlugins()` is removed. A runner holds no plugin instances until `onInvocationStart(InvocationInfo)` materializes them, and there is no accessor for them. +- `PluginRunner`'s constructor takes `List` instead of `List`. +- `PluginRunner.releasePlugins()` is added. The SDK calls it when the invocation returns, which is what bounds a plugin instance's lifetime to one invocation. +- `ExecutionManager.getPluginRunner()` exists in `3.x` and returns the runner for the current invocation. It is new in this release, not a renamed `2.x` method, and `ExecutionManager` is an internal coordination class. It is reachable only through `BaseContextImpl.getExecutionManager()`, which is declared on the implementation class and not on the `DurableContext` or `BaseContext` interfaces that handlers are given. + +What to do instead, by what you were trying to achieve: + +- **Reading which plugins are configured.** Use `DurableConfig.getPluginFactories()`. It returns factories, not instances, because instances do not exist outside an invocation. + + ```java + List factories = config.getPluginFactories(); + ``` + +- **Copying plugin registration into a derived `DurableConfig`.** Read the factories and pass them back through `withPlugins(...)`. This is what the SDK's own `LocalDurableTestRunner` does: + + ```java + var derived = DurableConfig.builder() + .withPlugins(config.getPluginFactories().toArray(new DurableExecutionPluginFactory[0])) + .build(); + ``` + +- **Firing hooks yourself in a test.** Construct the plugin directly and call its hook methods. Do not construct a `PluginRunner`. Building an `InvocationInfo` and calling `new MyPlugin(sink, info).onOperationEnd(...)` tests the plugin without depending on SDK internals. For end-to-end coverage, register the factory on a `DurableConfig` and drive it through `LocalDurableTestRunner`, which exercises the real dispatch path. + +- **Reaching a plugin instance from handler code at runtime.** There is no supported way to do this, and there was none in `2.x` either. Give the plugin and the handler a shared collaborator — the same object the factory captures — and communicate through it. + +## 3. Rebuild Service Providers Against 3.x + +`DurableExecutionPluginProvider` now extends `DurableExecutionPluginFactory` and declares only `getName()`. A provider is therefore itself the per-invocation factory. + +Removed from the interface: + +- `API_VERSION` +- `getApiVersion()` +- `getPluginType()` +- the zero-argument `createPlugin()` + +Discovery is unchanged: providers are still registered in `META-INF/services/software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider` and still selected by name through the `DURABLE_EXECUTION_PLUGINS` environment variable. Packaging, layer layout, ordering relative to `withPlugins(...)`, and the configuration errors that stop startup are documented in [Configuration](advanced/configuration.md#dynamic-plugin-loading) and are not repeated here. + +Before: + +```java +public final class AuditPluginProvider implements DurableExecutionPluginProvider { + + @Override + public String getName() { + return "com.example.audit"; + } + + @Override + public int getApiVersion() { + return API_VERSION; + } + + @Override + public Class getPluginType() { + return AuditPlugin.class; + } + + @Override + public DurableExecutionPlugin createPlugin() { + return new AuditPlugin(); + } +} +``` + +After: + +```java +public final class AuditPluginProvider implements DurableExecutionPluginProvider { + + // Environment lifetime: built once, when ServiceLoader instantiates the provider. + private final AuditSink sink = new AuditSink(); + + @Override + public String getName() { + return "com.example.audit"; + } + + @Override + public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { + return new AuditPlugin(sink, invocationInfo); + } +} +``` + +The provider instance itself is created once per execution environment by `ServiceLoader`, so provider fields are the right place for environment-lifetime state. `createPlugin(InvocationInfo)` runs once per invocation. + +### What happens to a provider that is not rebuilt + +This is the failure mode to understand before you deploy, because the function keeps succeeding while its instrumentation stops. + +A provider JAR compiled against `2.x` still loads. Its class file references nothing that `3.x` removed, so `ServiceLoader` instantiates it, `getName()` returns its name, and selection through `DURABLE_EXECUTION_PLUGINS` succeeds. Configuration therefore does not fail. + +The provider does not implement `createPlugin(InvocationInfo)` — it implements the zero-argument `createPlugin()` that no longer exists on the interface. When the SDK calls the method the class does not implement, the JVM throws `AbstractMethodError`. As of this release that error is contained: the SDK logs it and skips that factory for the invocation, exactly as it does for a factory that throws an exception. The execution proceeds and completes normally. + +The result is a deployment that runs correctly and emits nothing from its configured instrumentation. No execution fails, no invocation errors, and the only signal is a warning in the function's own logs, repeated once per invocation, from the `software.amazon.lambda.durable.plugin.PluginRunner` logger: + +```text +WARN software.amazon.lambda.durable.plugin.PluginRunner - Plugin factory failed; skipping it for this invocation +java.lang.AbstractMethodError: com.example.AuditPluginProvider.createPlugin(...) +``` + +If your instrumentation is the thing that produces your traces or audit records, losing it silently is worse than a failed deployment. Rebuild every provider JAR against `3.x` and redeploy it before or with the SDK upgrade. That includes provider JARs shipped as Lambda layers, which are versioned and deployed separately from the function package and are easy to leave behind. + +Caveat: containment is what makes this quiet, and containment is not the same as no failure at all. A stale provider whose class body also references an SDK symbol that `3.x` removed can instead fail during discovery, which throws `IllegalStateException` from `DurableConfig` construction and fails loudly. Both outcomes are possible depending on what the provider's code touches; neither is a substitute for rebuilding it. + +### Confirming a provider loaded + +There is no log line confirming successful provider selection, so confirmation is indirect. Check all three: + +1. The function's logs contain no `Plugin factory failed` warning from `PluginRunner`. +2. The provider's own output is present for a recent execution — spans in your trace backend, records at your exporter's destination, or whatever the plugin emits. +3. The deployed provider artifact is the one built against `3.x`. Check the layer version or JAR checksum you deployed, not just the version you built. + +A useful pre-deployment check is to run one execution locally with the provider on the class path and `DURABLE_EXECUTION_PLUGINS` set, using `LocalDurableTestRunner`, and assert that the plugin's output appears. + +## Recommended Validation After Upgrading + +1. Build your application against the `3.x` dependency and fix every `withPlugins(...)` compile error. There should be one per direct registration site. +2. Rebuild every plugin provider JAR you own against `3.x`. +3. Run your test suite. Tests that constructed a plugin instance and registered it will fail to compile; tests that assert on plugin output should still pass once registration is updated. +4. Exercise one workflow that suspends and resumes, and verify the plugin output for the resumed invocation is correct. This is where accumulated per-instance state that should have been derived from `InvocationInfo` shows up as missing data. +5. Exercise one workflow with concurrent child contexts, using `parallel()` or `map()`, and verify the plugin's collections tolerate concurrent hooks. +6. If you rely on dynamic loading, deploy to a pre-production stage and confirm the provider loaded using the three checks above. +7. Grep one stage's logs for `Plugin factory failed` before promoting. +8. Check that no plugin retains state after an invocation ends. A plugin instance should have no static or shared mutable collection keyed by execution ARN left in it. + +## Summary + +- `withPlugins(...)` takes `DurableExecutionPluginFactory` instead of `DurableExecutionPlugin`; pass `info -> new MyPlugin()` where you passed `new MyPlugin()` +- Environment-lifetime state moves outside the factory lambda; per-invocation state becomes plain instance fields and ARN-keyed maps are deleted +- `DurableConfig.getPluginRunner()` is removed in favor of `getPluginFactories()`; `PluginRunner` is an SDK internal and should not be used +- `DurableExecutionPluginProvider` keeps only `getName()` and inherits `createPlugin(InvocationInfo)`; `API_VERSION`, `getApiVersion()`, `getPluginType()`, and the zero-argument `createPlugin()` are removed +- A provider that is not rebuilt still loads and is still selected, but produces no instrumentation and only logs a warning, so rebuild and redeploy every provider JAR +- There is no compatibility bridge, and recompilation against `3.x` is required From 3cdec0315ed8c357223eadc5e9061d5f476d862e Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Thu, 17 Sep 2026 16:18:44 -0700 Subject: [PATCH 05/19] fix(plugin): contain every non-fatal throwable from plugin code The plugin contract says a factory or hook failure is logged, skipped, and never disrupts the execution. The catch was a list of types, so that promise held only while the list was complete. It was not complete. Four shapes reach the boundary and none is an Exception. A provider JAR compiled against an earlier interface throws AbstractMethodError. A provider whose optional dependency is absent throws NoClassDefFoundError. A plugin running with assertions enabled throws AssertionError. A plugin that loads its own exporter back ends through ServiceLoader throws ServiceConfigurationError. Only the first two are LinkageError, so the previous catch let the other two escape and fail an execution the plugin was only observing. Measured before the change: AssertionError and ServiceConfigurationError both escaped onInvocationStart, from the factory path and the hook path. The boundary now catches Throwable and rethrows only the fatal cases, so the promise stops depending on a list. Fatal means VirtualMachineError and nothing else. It is the JVM reporting it can no longer run correctly, which is not a plugin defect, and the process cannot be assumed able to continue. Naming the supertype covers OutOfMemoryError, StackOverflowError, InternalError and UnknownError, and keeps the rule stable if the JVM adds another. ThreadDeath is deliberately absent, on measurement rather than taste. The JVM delivers it only through Thread.stop(), which throws UnsupportedOperationException on JDK 20 and later, so it cannot arrive from the JVM on a current runtime. It is also deprecated for removal since JDK 20, so naming it would add a removal warning to every compile of this class and need a suppression that would then mask genuine removal warnings here. A ThreadDeath that plugin code throws itself is a plugin defect and is contained like any other. One deliberate behaviour change beyond containment. Throwing an InterruptedException clears the throwing thread's interrupt status, and the previous catch of Exception already contained it, so containment discarded the cancellation request silently. The threads that run factories and hooks are SDK threads that carry SDK work after the plugin returns. So the status is now restored before returning, immediately rather than after the dispatch loop, which keeps the flag true for every later read on that thread. A remaining plugin whose blocking call then fails fast is contained by this same rule, so every plugin is still called. One correction to an earlier claim in this branch. A previous round reported that DynamicPluginLoader.getProviderName lets a LinkageError escape without the wrapping message. That is wrong. getProviderName runs inside indexProviders' try block, whose LinkageError catch already wraps it, and with the version-compatibility message rather than the name-specific one. Widening getProviderName would replace the better diagnostic with the worse one, so it stays as it is. The real residual gap is narrower: only an AssertionError from getName() escapes bare, on a fail-fast startup path where startup fails either way and only the message differs. --- .../lambda/durable/plugin/PluginRunner.java | 77 +++++-- .../durable/plugin/PluginRunnerTest.java | 196 ++++++++++++++++++ 2 files changed, 252 insertions(+), 21 deletions(-) diff --git a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java index de281eea6..973f60985 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java @@ -19,10 +19,10 @@ * execution ARN. * *

    Event hooks are fire-and-forget: each plugin is called in order, errors are swallowed. A factory that throws or - * returns {@code null} is contained the same way — the plugin is skipped for the invocation. Containment covers - * {@link LinkageError} as well as {@link Exception}, because a plugin built against a different SDK version or missing - * an optional dependency fails with an {@code Error}; it deliberately stops short of the {@code Error}s that report the - * JVM itself failing. + * returns {@code null} is contained the same way — the plugin is skipped for the invocation. Containment covers every + * non-fatal throwable, not only {@link Exception}, because a plugin built against a different SDK version, one missing + * an optional dependency, and one running with assertions enabled all fail with an {@code Error}. It stops short of the + * errors that report the JVM itself failing, which keep propagating. * *

    {@code onInvocationEnd} is awaited (the SDK blocks until it returns) to allow plugins to flush data before Lambda * freezes. @@ -61,14 +61,16 @@ public boolean isEmpty() { *

    Called from {@link #onInvocationStart(InvocationInfo)} so the instances exist before any hook is dispatched. * Factories that throw or return null are logged and skipped. * - *

    {@link LinkageError} is contained alongside {@link Exception} because it is how the two most likely - * version-skew failures of this contract present themselves, and neither is an {@code Exception}: a provider JAR - * compiled against an earlier version of {@link DurableExecutionPluginFactory} throws {@link AbstractMethodError} - * when the SDK invokes the method it does not implement, and a provider whose optional dependency is missing from - * the deployment package throws {@link NoClassDefFoundError} while building its plugin. The contract says a factory - * failure is skipped and never disrupts the execution, so both are skipped. Deliberately narrow: an {@code Error} - * that reports the JVM itself failing — {@link OutOfMemoryError}, {@link StackOverflowError} — is not a plugin - * defect and must keep propagating rather than be logged as one. + *

    Every non-fatal throwable is contained, not just {@link Exception}. The contract says a factory failure is + * skipped and never disrupts the execution, and a throwable that escapes here fails an execution the plugin was + * only observing. Narrowing the catch to a list of types would leave that promise conditional on the list being + * complete, and it was not: a provider JAR compiled against an earlier version of + * {@link DurableExecutionPluginFactory} throws {@link AbstractMethodError}, a provider whose optional dependency is + * missing from the deployment package throws {@link NoClassDefFoundError}, a provider running with assertions + * enabled throws {@link AssertionError}, and a provider that loads its own exporter back ends through + * {@link java.util.ServiceLoader} throws {@link java.util.ServiceConfigurationError}. Only the first two are + * {@link LinkageError} and none is an {@link Exception}. Catching {@code Throwable} and rethrowing only the fatal + * cases makes the promise unconditional. See {@link #contain} for which cases stay fatal. */ private void createPlugins(InvocationInfo info) { var created = new ArrayList(pluginFactories.size()); @@ -80,8 +82,8 @@ private void createPlugins(InvocationInfo info) { continue; } created.add(plugin); - } catch (Exception | LinkageError e) { - logger.warn("Plugin factory failed; skipping it for this invocation", e); + } catch (Throwable t) { + contain(t, "Plugin factory failed; skipping it for this invocation"); } } this.plugins = List.copyOf(created); @@ -101,23 +103,56 @@ public void releasePlugins() { // ─── Event hooks ───────────────────────────────────────────────────── /** - * Calls a void hook on all of this invocation's plugins, swallowing any errors. + * Calls a void hook on all of this invocation's plugins, swallowing any non-fatal throwable. * - *

    {@link LinkageError} is contained alongside {@link Exception} for the reason given on {@link #createPlugins}: - * a plugin compiled against a different SDK version, or one missing an optional dependency, fails a hook with an - * {@code Error} rather than an {@code Exception}, and the fire-and-forget contract makes no distinction. Narrow on - * purpose — {@link OutOfMemoryError} and {@link StackOverflowError} still propagate. + *

    Containment here follows the same rule as {@link #createPlugins}, because the fire-and-forget contract makes + * no distinction between the two boundaries. A plugin fails a hook with the same shapes a factory fails with, and + * one plugin's failure must not stop the remaining plugins from receiving the hook or fail the execution. See + * {@link #contain} for which cases stay fatal. */ private void run(Consumer hook) { for (var plugin : plugins) { try { hook.accept(plugin); - } catch (Exception | LinkageError e) { - logger.warn("Plugin hook failed", e); + } catch (Throwable t) { + contain(t, "Plugin hook failed"); } } } + /** + * Logs a throwable that plugin code produced, or rethrows it if it is fatal. + * + *

    A {@link VirtualMachineError} is the JVM reporting that it can no longer run correctly, which covers + * {@link OutOfMemoryError}, {@link StackOverflowError}, {@link InternalError} and {@link UnknownError}. That is not + * a plugin defect, and the process cannot be assumed able to continue past it. Logging it as a contained plugin + * failure would therefore hide a condition the caller has to see, so it is rethrown unchanged. The rule names the + * supertype rather than the four subclasses so that a subclass added later is fatal without an edit here. + * + *

    {@code ThreadDeath} is the other error conventionally called fatal, and it is deliberately absent. The JVM + * delivers it only through {@code Thread.stop()}, which throws {@link UnsupportedOperationException} on JDK 20 and + * later, so on a current runtime it cannot arrive from the JVM at all. It is also deprecated for removal since JDK + * 20, so naming it would add a removal warning to every compile of this class and require a suppression that would + * then also mask genuine removal warnings here. A {@code ThreadDeath} that plugin code constructs and throws itself + * is a plugin defect, and is contained like any other. + * + *

    Throwing an {@link InterruptedException} clears the throwing thread's interrupt status. The threads that run + * factories and hooks are SDK threads that carry SDK work after the plugin returns, so containing the interrupt + * without restoring the status would hide the cancellation request from that later work and from the SDK's own + * blocking calls. The status is therefore restored before returning. Restoring it immediately rather than after the + * dispatch loop keeps the flag true for every subsequent read on this thread; a remaining plugin whose blocking + * call then fails fast is contained by this same rule, so every plugin is still called. + */ + private static void contain(Throwable t, String message) { + if (t instanceof VirtualMachineError fatal) { + throw fatal; + } + if (t instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } + logger.warn(message, t); + } + /** * Called at the start of each invocation. Materializes this invocation's plugin instances from the registered * factories, then dispatches the hook to them with the same {@link InvocationInfo} the factories received. diff --git a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java index e9e9e8e32..da11e63cd 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java @@ -8,6 +8,7 @@ import java.util.ArrayList; import java.util.List; import java.util.Map; +import java.util.ServiceConfigurationError; import java.util.concurrent.atomic.AtomicInteger; import org.junit.jupiter.api.Test; @@ -233,6 +234,127 @@ void hookThrowingLinkageError_isContained_andRemainingPluginsStillRun() { calls); } + // ─── Factory and hook throwables that are neither Exception nor LinkageError ── + // + // AssertionError and ServiceConfigurationError extend Error and Error respectively, and neither is a LinkageError, + // so a catch of `Exception | LinkageError` lets both escape. Escaping the plugin boundary fails the invocation the + // plugin was only observing. The contract says a factory or hook failure is logged and skipped and never disrupts + // the execution, so both must be contained. Both shapes are reachable through the plugin contract: a plugin that + // ships with assertions enabled, or that calls a library which asserts internally, throws AssertionError, and a + // plugin that runs its own ServiceLoader over its exporter back ends throws ServiceConfigurationError when one of + // them is misdeclared. + + @Test + void factoryThrowingAssertionError_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> { + throw new AssertionError("plugin invariant violated"); + }, + info -> new TestPlugin("p2", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + runner.onInvocationEnd(invocationEndInfo()); + + assertEquals(List.of("p2:onInvocationStart", "p2:onInvocationEnd"), calls); + } + + @Test + void factoryThrowingServiceConfigurationError_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> { + throw new ServiceConfigurationError("software.amazon.example.Exporter: provider not found"); + }, + info -> new TestPlugin("p2", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + runner.onInvocationEnd(invocationEndInfo()); + + assertEquals(List.of("p2:onInvocationStart", "p2:onInvocationEnd"), calls); + } + + @Test + void hookThrowingAssertionError_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> new AssertionErrorPlugin(), + info -> new TestPlugin("p2", calls), + info -> new TestPlugin("p3", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + calls.clear(); + assertDoesNotThrow(() -> runner.onOperationStart(operationInfo())); + assertDoesNotThrow(() -> runner.onInvocationEnd(invocationEndInfo())); + + assertEquals( + List.of("p2:onOperationStart", "p3:onOperationStart", "p2:onInvocationEnd", "p3:onInvocationEnd"), + calls); + } + + @Test + void hookThrowingServiceConfigurationError_isContained_andRemainingPluginsStillRun() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> new ServiceConfigurationErrorPlugin(), + info -> new TestPlugin("p2", calls), + info -> new TestPlugin("p3", calls))); + + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + calls.clear(); + assertDoesNotThrow(() -> runner.onInvocationEnd(invocationEndInfo())); + + assertEquals(List.of("p2:onInvocationEnd", "p3:onInvocationEnd"), calls); + } + + // ─── Interrupts ────────────────────────────────────────────────────── + // + // Throwing InterruptedException clears the throwing thread's interrupt status. The thread that fires a hook is an + // SDK thread that carries SDK work after the hook returns, so a runner that contains the InterruptedException + // without restoring the status hides the cancellation request from that later SDK work. The runner therefore + // contains the throwable, as the contract requires, and restores the interrupt status before returning. + // + // No hook and no factory method declares a checked exception, so plugin code reaches the boundary with an + // InterruptedException only by rethrowing it undeclared. The tests below use that shape deliberately. + + @Test + void factoryThrowingInterruptedException_isContained_andRestoresTheInterruptStatus() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of( + info -> { + sneakyThrow(new InterruptedException("flush interrupted")); + return null; + }, + info -> new TestPlugin("p2", calls))); + + try { + assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); + + assertTrue(Thread.currentThread().isInterrupted(), "the interrupt status must survive containment"); + assertEquals(List.of("p2:onInvocationStart"), calls); + } finally { + // Clear the status so it does not leak into whatever else runs on this thread. + Thread.interrupted(); + } + } + + @Test + void hookThrowingInterruptedException_isContained_andRestoresTheInterruptStatus() { + var calls = new ArrayList(); + var runner = new PluginRunner(List.of(info -> new InterruptingPlugin(), info -> new TestPlugin("p2", calls))); + runner.onInvocationStart(invocationInfo()); + calls.clear(); + + try { + assertDoesNotThrow(() -> runner.onInvocationEnd(invocationEndInfo())); + + assertTrue(Thread.currentThread().isInterrupted(), "the interrupt status must survive containment"); + assertEquals(List.of("p2:onInvocationEnd"), calls, "remaining plugins must still be called"); + } finally { + Thread.interrupted(); + } + } + @Test void factoryThrowingAJvmError_stillPropagates() { // The containment is deliberately narrow: an Error that says the JVM itself is failing must not be swallowed as @@ -251,6 +373,24 @@ void hookThrowingAJvmError_stillPropagates() { assertThrows(StackOverflowError.class, () -> runner.onInvocationStart(invocationInfo())); } + @Test + void factoryThrowingAnyVirtualMachineError_stillPropagates() { + // The fatal set is named by the VirtualMachineError supertype rather than by listing its subclasses, so an + // InternalError propagates for the same reason OutOfMemoryError does. This pins the supertype, not the list. + var runner = new PluginRunner(List.of(info -> { + throw new InternalError("JVM internal invariant violated"); + })); + + assertThrows(InternalError.class, () -> runner.onInvocationStart(invocationInfo())); + } + + @Test + void hookThrowingAnyVirtualMachineError_stillPropagates() { + var runner = new PluginRunner(List.of(info -> new UnknownErrorPlugin())); + + assertThrows(UnknownError.class, () -> runner.onInvocationStart(invocationInfo())); + } + // ─── Fire-and-forget event hooks ───────────────────────────────────── @Test @@ -536,4 +676,60 @@ public void onInvocationStart(InvocationInfo info) { throw new StackOverflowError(); } } + + /** Plugin whose hook throws a VirtualMachineError other than the two the older tests pin. */ + private static class UnknownErrorPlugin implements DurableExecutionPlugin { + @Override + public void onInvocationStart(InvocationInfo info) { + throw new UnknownError("unknown JVM failure"); + } + } + + /** Plugin whose hooks fail an assertion, as a plugin running with assertions enabled does. */ + private static class AssertionErrorPlugin implements DurableExecutionPlugin { + @Override + public void onInvocationStart(InvocationInfo info) { + throw new AssertionError("plugin invariant violated"); + } + + @Override + public void onOperationStart(OperationInfo info) { + throw new AssertionError("plugin invariant violated"); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + throw new AssertionError("plugin invariant violated"); + } + } + + /** Plugin whose hook fails its own service lookup, as a plugin loading its exporter back ends does. */ + private static class ServiceConfigurationErrorPlugin implements DurableExecutionPlugin { + @Override + public void onInvocationStart(InvocationInfo info) { + throw new ServiceConfigurationError("software.amazon.example.Exporter: provider not found"); + } + + @Override + public void onInvocationEnd(InvocationEndInfo info) { + throw new ServiceConfigurationError("software.amazon.example.Exporter: provider not found"); + } + } + + /** Plugin whose awaited hook is interrupted while flushing and rethrows the InterruptedException undeclared. */ + private static class InterruptingPlugin implements DurableExecutionPlugin { + @Override + public void onInvocationEnd(InvocationEndInfo info) { + sneakyThrow(new InterruptedException("flush interrupted")); + } + } + + /** + * Throws {@code t} without declaring it, which is how plugin code can reach the runner with an + * {@link InterruptedException} even though no hook signature permits a checked exception. + */ + @SuppressWarnings("unchecked") + private static void sneakyThrow(Throwable t) throws E { + throw (E) t; + } } From 942505504da01e7cf3c786cd615e9f125bfd380c Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Thu, 17 Sep 2026 23:02:05 -0700 Subject: [PATCH 06/19] fix(insight): drop a record whose build was overtaken A reviewer found this defect in the Python plugin, and a probe has since confirmed it in the JS plugin. Both are fixed. This makes Java the third, and it is a latent defect here rather than a live one. Three properties produce it. Customer code runs while a record is being built, on the hook thread, before anything is scheduled: the input and output content transforms, an operation's result transform, and any Jackson serializer registered for a customer type. The scheduler's per-invocation slot takes whichever record is handed over last and compares no ages, so there is no sequence number, revision or timestamp anywhere in this module. And customer code can call back into a hook of the same instance, so a build can be overtaken by a newer build that starts and finishes inside it. The first and third properties hold here. The second does not, as shipped: the SDK serialises change hooks for one execution, because CheckpointManager.checkpointBatch wraps the plugin dispatch in synchronized (pollingFutures) and batches for one execution are chained sequentially through thenRunAsync. So the defect is not reachable today. Nothing pins that. Neither CheckpointManagerTest nor ApiRequestDelayedBatcherTest asserts that change hooks are serialised per execution, so the plugin is correct only because of an SDK property no test defends. A probe that forced the re-entry produced both Python outcomes: with the pump held, the newer two-operation snapshot was coalesced away and only the older one-operation snapshot was exported; with the pump running immediately, the exporter saw the newer snapshot and then the older one. Each invocation now counts the record builds it has started. A build takes the next value before it begins, and the scheduler queues the record only while that value is still the newest, so an overtaken build's record is dropped. Nothing is lost, because a record is a complete snapshot of one execution and the record that superseded it carries everything the older one carried. That is the same property that already makes the slot's coalescing sound. The counter is an AtomicLong rather than a volatile long. Incrementing a volatile long is a read-modify-write, so two concurrent builds could take the same value and each would conclude its own record is the newest. That is the case the check exists for, so a racy counter would guard nothing. The check itself is read inside the critical section that queues the record, which is the monitor `closed` is already read under, so the two checks and the hand-off are one section. The revision is taken before the build and never after. A value read after the build would already belong to the build that customer code started from inside the transform, so the older record would pass the check and overwrite the newer one. The final record takes no revision and is not checked. Customer code running inside its build can start a newer RUNNING build, which would make a revision taken there stale, and a checked hand-off would then drop the final record and leave a RUNNING snapshot as the execution's last exported state. Exempting it cannot let a stale record win, because closeAndSchedule sets `closed` in the same critical section that queues the final record and every RUNNING record handed over afterwards is rejected. So the two orderings are separate and neither subsumes the other. `closed` orders RUNNING records against the final record. The revision orders RUNNING records against each other. A boolean cannot say which of two RUNNING builds is newer, and the revision cannot reject a RUNNING record that follows the final one. Four tests. Two pin the outcomes, one per pump timing, and both fail when the revision term is removed. Two pin the failure mode a revision check can introduce, and both fail when the final record is checked as well. --- .../durable/insight/ExportScheduler.java | 53 +++- .../lambda/durable/insight/InsightPlugin.java | 65 +++- .../insight/RecordSupersessionTest.java | 282 ++++++++++++++++++ 3 files changed, 390 insertions(+), 10 deletions(-) create mode 100644 insight-plugin/src/test/java/software/amazon/lambda/durable/insight/RecordSupersessionTest.java diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java index 4ae67384a..2f84feabc 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/ExportScheduler.java @@ -43,6 +43,12 @@ * that invocation are coalesced into its slot — intermediate records are dropped because the latest one already * contains all of their information. A record from a different invocation never displaces another's record. * + *

    The slot takes whichever record is handed to it last and compares nothing, so "newer" has to be established before + * the hand-off. Customer code runs while a record is being built and can re-enter a hook of the same invocation, which + * builds and hands over a newer record first; the build it re-entered from then hands over an older snapshot last. + * {@code InsightPlugin}'s build revision identifies each build and {@link #scheduleIfNotSuperseded} drops a record + * whose build has been overtaken, so the slot only ever advances. + * *

    A single pump exports the queued records one at a time, in the order the invocations first queued work * ({@link #queue}, which is ordering only — membership in it is the same fact as "this invocation has a record", * written in one place), so exporters still never see two exports at once and each record keeps its per-exporter @@ -164,9 +170,14 @@ final class ExportScheduler { // --- Scheduling. --- /** - * Queues the latest record of one invocation for export. If an export is already running, the record is held in - * that invocation's own slot (replacing only an earlier record of the same invocation) and exported once - * the pump reaches it. + * Queues the latest record of one invocation for export, with no ordering check. If an export is already running, + * the record is held in that invocation's own slot (replacing only an earlier record of the same + * invocation) and exported once the pump reaches it. + * + *

    The slot takes whichever record is handed over last and does not compare record ages, so this is the right + * entry point only for a record that cannot be superseded. The plugin's RUNNING records go through + * {@link #scheduleIfNotSuperseded} and its final record through {@link #closeAndSchedule}; both add the ordering + * checks this one omits. */ void schedule(InsightPlugin execution, WorkflowInsightRecord record) { CompletableFuture handle; @@ -178,13 +189,31 @@ void schedule(InsightPlugin execution, WorkflowInsightRecord record) { } /** - * Schedules the record unless this invocation has already ended; the check and the hand-off are one critical - * section, on the same monitor that owns the {@code closed} flag. + * Schedules a non-terminal record unless it has been superseded, which is two separate facts. + * + *

    The invocation may already have ended. No RUNNING snapshot may follow the final record, so + * {@link InsightPlugin#closed} rejects it. + * + *

    A newer build of this same invocation may already have started. Customer code runs inside a build — the + * content transforms, an operation result transform, a serializer for a customer type — and can re-enter a hook, so + * the build that hands its record over last is not necessarily the build that started last. Without the revision + * check the slot would take that older snapshot and the newer one would be lost, or, if a pump had already taken + * the newer one, an exporter would see the older snapshot after the newer one. + * + *

    The superseded record is dropped rather than queued. Nothing is lost: a record is a complete snapshot of one + * execution, so the record that superseded it carries everything it carries. That is the same property that makes + * the slot's coalescing sound. + * + *

    Both checks and the hand-off are one critical section, on the monitor that owns both fields, so a record + * cannot pass the checks and then be queued after the record that supersedes it. + * + * @param buildRevision the revision the caller took before it started building this record + * @return whether the record was queued */ - boolean scheduleIfOpen(InsightPlugin execution, WorkflowInsightRecord record) { + boolean scheduleIfNotSuperseded(InsightPlugin execution, WorkflowInsightRecord record, long buildRevision) { CompletableFuture handle; synchronized (this) { - if (execution.closed) { + if (execution.closed || !execution.isNewestBuild(buildRevision)) { return false; } queueRecord(execution, record); @@ -194,7 +223,15 @@ boolean scheduleIfOpen(InsightPlugin execution, WorkflowInsightRecord record) { return true; } - /** Marks the invocation ended and, when given a record, schedules it as the last one for that invocation. */ + /** + * Marks the invocation ended and, when given a record, schedules it as the last one for that invocation. + * + *

    The final record is queued without the build-revision check {@link #scheduleIfNotSuperseded} makes. Customer + * code running inside the final record's build can start a newer RUNNING build, which would leave the final + * record's revision stale, and a checked hand-off would then drop it and leave a RUNNING snapshot as the + * execution's last exported state. Exempting it cannot let a stale record win, because {@code closed} is set in + * this same critical section and every RUNNING record handed over afterwards is rejected. + */ void closeAndSchedule(InsightPlugin execution, WorkflowInsightRecord finalRecord) { if (finalRecord == null && execution.closed) { // The idempotent second call from the hook's `finally`. A volatile read, so the common case of an diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightPlugin.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightPlugin.java index 82b67ba1a..3223522f8 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightPlugin.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightPlugin.java @@ -8,6 +8,7 @@ import java.util.List; import java.util.Map; import java.util.concurrent.CompletableFuture; +import java.util.concurrent.atomic.AtomicLong; import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.InvocationEndInfo; import software.amazon.lambda.durable.plugin.InvocationInfo; @@ -37,6 +38,9 @@ *

  • The input snapshot — {@link #cachedInput} — is written by the thread that fires * {@code onInvocationStart} and read by the operation-change and invocation-end threads of the same invocation, * which the SDK does not promise are the same thread; {@code volatile} for that publication. + *
  • The build revision — {@link #buildRevision} — counts the record builds this invocation has started, so + * that a build which was overtaken can be recognized at hand-off time and its record dropped. Atomic rather than + * {@code volatile}, because the case it exists for is two builds running at once. See the field. *
  • Scheduling state — {@link #record}, {@link #settled}, {@link #exporting}, {@link #drainWaiters} and * {@link #closed} — is shared with the export pump and guarded by the monitor of {@link #scheduler}. One monitor * for the whole environment, not one per invocation, so the {@code closed} check and the hand-off of a record are @@ -82,6 +86,27 @@ final class InsightPlugin implements DurableExecutionPlugin { */ private volatile Object cachedInput; + // --- Build ordering. --- + + /** + * Counts the record builds this invocation has started. The value a build takes identifies that build. + * + *

    Customer code runs inside a build, on the hook thread: the input and output content transforms, an operation's + * result transform, and any Jackson serializer registered for a customer type. That code can call back into a hook + * of this same instance, and it runs before anything is scheduled, so a build can be overtaken by a newer build + * that starts and finishes inside it. Two hook threads for one invocation would produce the same overlap. + * + *

    The scheduler's slot holds one record per invocation and takes whichever record is handed to it last, with no + * comparison of age. An overtaken build would therefore write its older snapshot over the newer one. Every build + * takes the next value here before it starts, and the scheduler queues the record only while that value is still + * the newest, so an overtaken build's record is dropped instead. + * + *

    An {@link AtomicLong} rather than a {@code volatile long}: {@code ++} on a {@code volatile long} is a + * read-modify-write, so two concurrent builds can take the same value and each conclude its own record is the + * newest. That is the very case the check exists for, so a racy counter would guard nothing. + */ + private final AtomicLong buildRevision = new AtomicLong(); + // --- Scheduling state: guarded by the scheduler's monitor. --- /** @@ -90,6 +115,10 @@ final class InsightPlugin implements DurableExecutionPlugin { *

    A newer record replaces an older one here — each record is a complete snapshot, so the older one carries * nothing the newer one lacks. That is the whole of coalescing: one slot, on the instance, which no other * invocation can reach. + * + *

    Which record is newer is decided by {@link #buildRevision}, not by the order the records reach this slot. The + * slot itself takes the last hand-off unconditionally, and the last hand-off is not the newest build when a build + * was overtaken by one that customer code started from inside it. */ WorkflowInsightRecord record; @@ -121,6 +150,11 @@ final class InsightPlugin implements DurableExecutionPlugin { * *

    A checkpoint that completes while the end record is being drained still delivers an operation-change hook to * this same instance, and that RUNNING snapshot must not supersede the final record. + * + *

    This orders RUNNING records against the final record; {@link #buildRevision} orders RUNNING records against + * each other. Neither covers the other's case. A boolean cannot say which of two RUNNING builds is newer, and the + * revision cannot reject a RUNNING record that follows the final one, because the final record is queued without a + * revision check. See {@link ExportScheduler#closeAndSchedule}. */ volatile boolean closed; @@ -163,7 +197,12 @@ public void onInvocationStart(InvocationInfo info) { cachedInput = null; } if (settings.emitMode == WorkflowInsightConfig.EmitMode.ON_CHANGE) { - scheduler.schedule(this, buildRecord("RUNNING", info.operations(), null, cachedInput, null, null)); + // The revision is taken before the build, never after. Customer code runs inside buildRecord and can + // re-enter a hook of this instance, which builds a newer record; a revision read afterwards would + // already be that newer build's, and this older record would pass the check and overwrite it. + long revision = beginBuild(); + scheduler.scheduleIfNotSuperseded( + this, buildRecord("RUNNING", info.operations(), null, cachedInput, null, null), revision); } } catch (Throwable t) { WorkflowInsight.logSafely("onInvocationStart failed", t); @@ -181,7 +220,9 @@ public void onOperationChange(OperationChangeInfo info) { if (closed) { return; } - scheduler.scheduleIfOpen(this, buildRecord("RUNNING", info.operations(), null, cachedInput, null, null)); + long revision = beginBuild(); + scheduler.scheduleIfNotSuperseded( + this, buildRecord("RUNNING", info.operations(), null, cachedInput, null, null), revision); } catch (Throwable t) { WorkflowInsight.logSafely("onOperationChange failed", t); } @@ -212,6 +253,10 @@ public void onInvocationEnd(InvocationEndInfo info) { WorkflowInsightRecord finalRecord = null; if (sampledIn && shouldEmit) { + // No build revision is taken here. Customer code running inside this build can start a newer RUNNING + // build, which would make a revision taken here stale, and a checked hand-off would then drop the final + // record and leave a RUNNING snapshot as this execution's last exported state. The final record is + // instead ordered by `closed`, which closeAndSchedule sets in the same critical section that queues it. finalRecord = buildRecord( status, info.operations(), @@ -271,6 +316,22 @@ private void drainAndFlush() { // --- Record building. --- + /** Starts a record build and returns the revision that identifies it. */ + private long beginBuild() { + return buildRevision.incrementAndGet(); + } + + /** + * Whether the identified build is still the newest one this invocation has started. + * + *

    Read by the scheduler inside the critical section that queues the record, so a record that passes cannot be + * queued after a record that supersedes it. A build that starts after the check passes still supersedes this one: + * its record is handed over later and replaces this one in the slot, which is the order the slot should have. + */ + boolean isNewestBuild(long revision) { + return buildRevision.get() == revision; + } + private WorkflowInsightRecord buildRecord( String status, Map operations, diff --git a/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/RecordSupersessionTest.java b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/RecordSupersessionTest.java new file mode 100644 index 000000000..faf7dbaf4 --- /dev/null +++ b/insight-plugin/src/test/java/software/amazon/lambda/durable/insight/RecordSupersessionTest.java @@ -0,0 +1,282 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.insight; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.time.Instant; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Queue; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.Executor; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import org.junit.jupiter.api.Test; +import software.amazon.awssdk.services.lambda.model.OperationStatus; +import software.amazon.lambda.durable.plugin.InvocationEndInfo; +import software.amazon.lambda.durable.plugin.InvocationInfo; +import software.amazon.lambda.durable.plugin.InvocationStatus; +import software.amazon.lambda.durable.plugin.OperationChangeInfo; +import software.amazon.lambda.durable.plugin.OperationChangeItemInfo; + +/** + * Build order, not hand-off order, decides which record an invocation exports. + * + *

    Customer code runs while a record is being built, on the hook thread, before anything is scheduled: the input and + * output content transforms, an operation's result transform, and any Jackson serializer registered for a customer + * type. That code can call back into a hook of the same plugin instance, which builds and hands over a newer record + * while the outer build is still running. The outer build then hands over an older snapshot last, and the scheduler's + * per-invocation slot takes the last hand-off with no comparison of record ages. + * + *

    Two outcomes follow if nothing orders the two records. With the pump held, the newer record is coalesced away and + * only the older snapshot is exported. With the pump running immediately, the exporter sees the newer record and then + * the older one, so the last state a destination records for the execution is stale. + * + *

    The plugin takes a build revision before each build and the scheduler queues the record only while that revision + * is still the newest, so an overtaken build's record is dropped. The final record is exempt from that check and is + * ordered by the invocation's {@code closed} flag instead, so a newer RUNNING build started from inside the final + * record's own transforms cannot drop it. + * + *

    The SDK serializes change hooks for one execution today, so the re-entrant hook here is forced rather than + * observed in production. The plugin must not depend on that: nothing in the SDK pins it, and the three language SDKs + * carry the same guard. + */ +class RecordSupersessionTest { + + private static final String ARN = + "arn:aws:lambda:us-west-2:111122223333:function:f:$LATEST/durable-execution/exec-1/invocation-1"; + private static final Instant START = Instant.parse("2026-08-05T00:00:00Z"); + private static final String INPUT = "payload"; + + /** Records every record handed to an exporter, in the order the exporter saw them. */ + private static final class RecordingExporter implements InsightExporter { + final List exported = Collections.synchronizedList(new ArrayList<>()); + + @Override + public void export(WorkflowInsightRecord record) { + exported.add(record); + } + } + + /** Holds every pump the scheduler starts until the test runs it, so the coalescing window is under test control. */ + private static final class HeldExecutor implements Executor { + private final Queue pending = new ConcurrentLinkedQueue<>(); + + @Override + public void execute(Runnable command) { + pending.add(command); + } + + void runPending() { + Runnable task; + while ((task = pending.poll()) != null) { + task.run(); + } + } + } + + /** + * One invocation's plugin, wired to a scheduler whose pump the test controls, with an input transform that can be + * armed to re-enter a hook of that same plugin. Re-entry through a content transform is the reachable path: + * {@code buildRecord} calls the transform before it returns the record to be scheduled. + */ + private static final class Fixture { + final RecordingExporter exporter = new RecordingExporter(); + final List failures = Collections.synchronizedList(new ArrayList<>()); + final AtomicReference armed = new AtomicReference<>(); + final ExportScheduler scheduler; + final InsightPlugin plugin; + + Fixture(Executor executor) { + var config = WorkflowInsightConfig.builder() + .emitMode(WorkflowInsightConfig.EmitMode.ON_CHANGE) + .content(ContentConfig.builder() + .inputTransform(value -> { + Runnable reentry = armed.getAndSet(null); + if (reentry != null) { + reentry.run(); + } + return value; + }) + .build()) + .addExporter(exporter) + .build(); + scheduler = new ExportScheduler( + List.of(exporter), (record, target) -> target.export(record), failures::add, executor); + plugin = new InsightPlugin(new InsightSettings(config), scheduler, Executions.info(ARN)); + } + + /** Arms the next build's transform to run this once, before the build that triggered it finishes. */ + void arm(Runnable reentry) { + armed.set(reentry); + } + + boolean isDraining() { + synchronized (scheduler) { + return plugin.drainWaiters > 0; + } + } + } + + // --- The two outcomes an unordered hand-off produces. --- + + @Test + void anOvertakenBuildDoesNotOverwriteTheNewerRecordInTheSlot() { + var executor = new HeldExecutor(); + var fixture = new Fixture(executor); + + // The start record claims the pump, which this executor holds, so every record below coalesces into the one + // slot this invocation has. + fixture.plugin.onInvocationStart(startInfo(operations(1))); + // The change hook this arms builds a two-operation record and hands it over while the outer build below is + // still inside its transform. The outer build then hands over its one-operation record last. + fixture.arm(() -> fixture.plugin.onOperationChange(changeInfo(operations(2)))); + fixture.plugin.onOperationChange(changeInfo(operations(1))); + + executor.runPending(); + + assertEquals( + List.of(2), + operationCounts(fixture), + "the slot must hold the newest build's record; the overtaken build's older snapshot is dropped"); + assertEquals(List.of(), fixture.failures, "no scheduler failure was reported"); + } + + @Test + void anOvertakenBuildIsNotExportedAfterTheRecordThatOvertookIt() { + // Runnable::run makes the pump drain inside schedule(), so each record reaches the exporter before the next + // hand-off. Nothing is coalesced, and a superseded record shows up as a stale export rather than a lost one. + var fixture = new Fixture(Runnable::run); + + fixture.plugin.onInvocationStart(startInfo(operations(1))); + fixture.arm(() -> fixture.plugin.onOperationChange(changeInfo(operations(2)))); + fixture.plugin.onOperationChange(changeInfo(operations(1))); + + assertEquals( + List.of(1, 2), + operationCounts(fixture), + "the overtaken build's record must not be exported after the record that overtook it"); + assertEquals(List.of(), fixture.failures, "no scheduler failure was reported"); + } + + // --- The failure mode a revision check can introduce. --- + + @Test + void theFinalRecordSurvivesANewerBuildStartedInsideIt() { + var fixture = new Fixture(Runnable::run); + + fixture.plugin.onInvocationStart(startInfo(operations(1))); + // Re-entered from the final record's own build, so the final record's revision is no longer the newest by the + // time it is handed over. Dropping it would leave a RUNNING snapshot as this execution's last exported state. + fixture.arm(() -> fixture.plugin.onOperationChange(changeInfo(operations(2)))); + fixture.plugin.onInvocationEnd(endInfo(operations(2))); + + var statuses = statuses(fixture); + assertTrue(statuses.contains("SUCCEEDED"), "the final record must be exported; exported: " + statuses); + assertEquals( + "SUCCEEDED", + statuses.get(statuses.size() - 1), + "no RUNNING record may be exported after the final one; exported: " + statuses); + assertEquals(List.of(), fixture.failures, "no scheduler failure was reported"); + } + + @Test + void theFinalRecordIsTheOnlyRecordExportedWhenThePumpRunsAfterTheInvocationEnds() throws Exception { + var executor = new HeldExecutor(); + var fixture = new Fixture(executor); + + // Runs the held pump only once the invocation end is inside its drain. Every record that end built is in the + // slot by then, so which record is exported is decided by the slot rather than by when this thread wakes up. + var invocationEnded = new AtomicBoolean(); + var pumper = new Thread(() -> { + while (!fixture.isDraining()) { + Thread.onSpinWait(); + } + // Kept pumping until the hook returns: the flush the end requests after its drain needs a pump too. + while (!invocationEnded.get()) { + executor.runPending(); + Thread.onSpinWait(); + } + executor.runPending(); + }); + pumper.setDaemon(true); + pumper.start(); + + fixture.plugin.onInvocationStart(startInfo(operations(1))); + fixture.arm(() -> fixture.plugin.onOperationChange(changeInfo(operations(2)))); + fixture.plugin.onInvocationEnd(endInfo(operations(2))); + invocationEnded.set(true); + pumper.join(30_000); + assertFalse(pumper.isAlive(), "the invocation end never completed its drain and flush"); + + assertEquals( + List.of("SUCCEEDED"), + statuses(fixture), + "the final record supersedes both RUNNING records in the slot and is the one exported"); + assertEquals(List.of(), fixture.failures, "no scheduler failure was reported"); + } + + // --- Fixture helpers. --- + + private static List operationCounts(Fixture fixture) { + List counts = new ArrayList<>(); + synchronized (fixture.exporter.exported) { + for (WorkflowInsightRecord record : fixture.exporter.exported) { + counts.add(record.operations().size()); + } + } + return counts; + } + + private static List statuses(Fixture fixture) { + List statuses = new ArrayList<>(); + synchronized (fixture.exporter.exported) { + for (WorkflowInsightRecord record : fixture.exporter.exported) { + statuses.add(record.status()); + } + } + return statuses; + } + + private static InvocationInfo startInfo(Map operations) { + return new InvocationInfo("req", ARN, true, START, INPUT, operations, Map.of()); + } + + private static OperationChangeInfo changeInfo(Map operations) { + return new OperationChangeInfo("req", ARN, operations, operations); + } + + private static InvocationEndInfo endInfo(Map operations) { + return new InvocationEndInfo( + "req", ARN, true, START, operations, InvocationStatus.SUCCEEDED, null, INPUT, "result"); + } + + /** A snapshot of {@code count} completed steps; the count is how a test tells two records apart. */ + private static Map operations(int count) { + Map snapshot = new LinkedHashMap<>(); + for (int i = 1; i <= count; i++) { + snapshot.put( + "op-" + i, + new OperationChangeItemInfo( + "op-" + i, + "step-" + i, + "STEP", + "Step", + null, + START.plusMillis(i), + START.plusMillis(i + 1), + OperationStatus.SUCCEEDED, + 1, + false, + null, + null)); + } + return snapshot; + } +} From 1846a629fcc35fd74a39ecc19ea256a3b9d42009 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 11:01:02 -0700 Subject: [PATCH 07/19] fix(plugin): rethrow ThreadDeath as fatal An earlier commit made the plugin boundary contain every non-fatal throwable, treating only VirtualMachineError as fatal. A reviewer filed twice that ThreadDeath belongs on the fatal side, and they are right. Thread.stop() delivers ThreadDeath by throwing it into the target thread, which unwinds that thread's stack from wherever it stood and releases the monitors it held over state it had only half updated. So it is not a report of a failure but a termination already under way. The threads that create plugins and fire hooks are SDK threads that carry SDK and user work after the plugin returns. Containing it would therefore return one of those threads to that work with its invariants broken and the termination silently dropped. The earlier decision rested on two facts that are true and one conclusion that does not follow. Thread.stop() throws UnsupportedOperationException on JDK 20 and later, so the JVM cannot deliver a ThreadDeath there. It is also deprecated for removal since JDK 20, so naming it emits a removal warning. But maven.compiler.source and maven.compiler.target are both 17, verified in the root pom at lines 53 and 54, and Thread.stop() still delivers on 17. So the delivery is possible on the runtime this SDK declares support for, which is what decides it. A ThreadDeath that plugin code constructs and throws itself is rethrown on every runtime. The boundary cannot distinguish it from a delivered one, and treating the ambiguous case as fatal is the safe direction. The removal warning is suppressed on the contain method rather than the class, so the suppression cannot mask a removal warning that appears elsewhere in PluginRunner. The full reactor emits no removal warning after the change. Deleting the suppression and recompiling the sdk module produces exactly one, and that compile still exits zero, so the warning was never fatal: no pom sets -Werror, failOnWarning, showDeprecation or any compilerArg. The javadoc that argued the other way is rewritten rather than left contradicting the code, and the class-level sentence that described the containment rule now names both propagating cases. Two tests pin it, one per boundary, and both fail before the change with "Expected java.lang.ThreadDeath to be thrown, but nothing was thrown". They throw it directly, because the build runs on a JDK where Thread.stop() would raise UnsupportedOperationException instead of delivering one. The existing tests that assert OutOfMemoryError, StackOverflowError, InternalError and UnknownError propagate, and that AssertionError, ServiceConfigurationError and both LinkageError shapes are contained, are unchanged and still pass: the change moves exactly one throwable from one side to the other. --- .../lambda/durable/plugin/PluginRunner.java | 32 +++++++++++---- .../durable/plugin/PluginRunnerTest.java | 39 +++++++++++++++++++ 2 files changed, 63 insertions(+), 8 deletions(-) diff --git a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java index 973f60985..5bf4d835e 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java @@ -21,8 +21,9 @@ *

    Event hooks are fire-and-forget: each plugin is called in order, errors are swallowed. A factory that throws or * returns {@code null} is contained the same way — the plugin is skipped for the invocation. Containment covers every * non-fatal throwable, not only {@link Exception}, because a plugin built against a different SDK version, one missing - * an optional dependency, and one running with assertions enabled all fail with an {@code Error}. It stops short of the - * errors that report the JVM itself failing, which keep propagating. + * an optional dependency, and one running with assertions enabled all fail with an {@code Error}. It stops short of two + * cases, which keep propagating: the errors that report the JVM itself failing, and the {@code ThreadDeath} that + * reports the thread running the plugin has already been terminated. * *

    {@code onInvocationEnd} is awaited (the SDK blocks until it returns) to allow plugins to flush data before Lambda * freezes. @@ -129,12 +130,23 @@ private void run(Consumer hook) { * failure would therefore hide a condition the caller has to see, so it is rethrown unchanged. The rule names the * supertype rather than the four subclasses so that a subclass added later is fatal without an edit here. * - *

    {@code ThreadDeath} is the other error conventionally called fatal, and it is deliberately absent. The JVM - * delivers it only through {@code Thread.stop()}, which throws {@link UnsupportedOperationException} on JDK 20 and - * later, so on a current runtime it cannot arrive from the JVM at all. It is also deprecated for removal since JDK - * 20, so naming it would add a removal warning to every compile of this class and require a suppression that would - * then also mask genuine removal warnings here. A {@code ThreadDeath} that plugin code constructs and throws itself - * is a plugin defect, and is contained like any other. + *

    {@code ThreadDeath} is fatal for a different reason. It is not a report of a failure but a thread termination + * that has already begun: {@code Thread.stop()} delivers it by throwing it into the target thread, which unwinds + * that thread's stack from wherever it stood and releases the monitors it held over state it had only half updated. + * The threads that create plugins and fire hooks are SDK threads that carry SDK and user work after the plugin + * returns. Containing the {@code ThreadDeath} would therefore return one of those threads to that work with its + * invariants already broken and the termination it was sent silently dropped. It is rethrown unchanged so the + * termination completes. + * + *

    {@code Thread.stop()} throws {@link UnsupportedOperationException} on JDK 20 and later, so the JVM cannot + * deliver a {@code ThreadDeath} on those runtimes. It can deliver one on JDK 17, and {@code maven.compiler.source} + * is 17, so the rethrow is reachable on a runtime this SDK supports. A {@code ThreadDeath} that plugin code + * constructs and throws itself is rethrown on every runtime; the boundary cannot distinguish it from a delivered + * one, and treating the ambiguous case as fatal is the safe direction. + * + *

    {@code ThreadDeath} is deprecated for removal since JDK 20, so naming it emits a removal warning when this + * class is compiled on a JDK 20 or later compiler. The {@code @SuppressWarnings("removal")} below is scoped to this + * method rather than the class so it cannot mask a removal warning that appears elsewhere in {@code PluginRunner}. * *

    Throwing an {@link InterruptedException} clears the throwing thread's interrupt status. The threads that run * factories and hooks are SDK threads that carry SDK work after the plugin returns, so containing the interrupt @@ -143,10 +155,14 @@ private void run(Consumer hook) { * dispatch loop keeps the flag true for every subsequent read on this thread; a remaining plugin whose blocking * call then fails fast is contained by this same rule, so every plugin is still called. */ + @SuppressWarnings("removal") // ThreadDeath is deprecated for removal since JDK 20; see the javadoc above. private static void contain(Throwable t, String message) { if (t instanceof VirtualMachineError fatal) { throw fatal; } + if (t instanceof ThreadDeath fatal) { + throw fatal; + } if (t instanceof InterruptedException) { Thread.currentThread().interrupt(); } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java index da11e63cd..ec1f7d5bb 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java @@ -391,6 +391,36 @@ void hookThrowingAnyVirtualMachineError_stillPropagates() { assertThrows(UnknownError.class, () -> runner.onInvocationStart(invocationInfo())); } + // ─── Thread termination ────────────────────────────────────────────── + // + // Thread.stop() terminates a thread by throwing ThreadDeath into it, which unwinds that thread's stack from + // wherever it stood and releases the monitors it held over state it had only half updated. maven.compiler.source is + // 17, and Thread.stop() still delivers ThreadDeath on a JDK 17 runtime, so the delivery is possible on a runtime + // this SDK supports. Containing the ThreadDeath would return the factory or hook thread to the SDK and user work it + // carries after the plugin returns, with that thread's invariants already broken and the termination dropped. The + // runner therefore rethrows it, at both the factory boundary and the hook boundary. + // + // These tests throw the ThreadDeath directly. Thread.stop() throws UnsupportedOperationException on the JDK 20 or + // later runtime the build uses, so a test cannot ask the JVM to deliver one. + + @Test + @SuppressWarnings("removal") // ThreadDeath is deprecated for removal since JDK 20. + void factoryThrowingThreadDeath_stillPropagates() { + var runner = new PluginRunner(List.of(info -> { + throw new ThreadDeath(); + })); + + assertThrows(ThreadDeath.class, () -> runner.onInvocationStart(invocationInfo())); + } + + @Test + @SuppressWarnings("removal") // ThreadDeath is deprecated for removal since JDK 20. + void hookThrowingThreadDeath_stillPropagates() { + var runner = new PluginRunner(List.of(info -> new ThreadDeathPlugin())); + + assertThrows(ThreadDeath.class, () -> runner.onInvocationStart(invocationInfo())); + } + // ─── Fire-and-forget event hooks ───────────────────────────────────── @Test @@ -685,6 +715,15 @@ public void onInvocationStart(InvocationInfo info) { } } + /** Plugin whose hook thread has been terminated by {@code Thread.stop()}, which must not be contained. */ + @SuppressWarnings("removal") // ThreadDeath is deprecated for removal since JDK 20. + private static class ThreadDeathPlugin implements DurableExecutionPlugin { + @Override + public void onInvocationStart(InvocationInfo info) { + throw new ThreadDeath(); + } + } + /** Plugin whose hooks fail an assertion, as a plugin running with assertions enabled does. */ private static class AssertionErrorPlugin implements DurableExecutionPlugin { @Override From 9fc44f8638888b3c03d150f4d5158b5d9ae3caf8 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 12:59:35 -0700 Subject: [PATCH 08/19] feat(plugin): reject a stale plugin provider at load A provider JAR compiled against the 2.x provider interface still loads under 3.x. Nothing it references was removed, so ServiceLoader instantiates it and selection by name succeeds. The first createPlugin(InvocationInfo) call then throws AbstractMethodError, which the SDK contains per invocation and logs as a warning. The function keeps succeeding while the provider emits nothing. DynamicPluginLoader.getProvider now reads the resolved createPlugin method and fails configuration when it is still abstract. Reading the modifier runs no provider code. The failure names the provider class, the artifact it came from, and the fact that a provider shipped as a Lambda layer is deployed separately from the function package. The check runs on the selected provider, not on every indexed one, so a stale JAR that sits unused on the class path does not fail a function that does not select it. --- docs/advanced/configuration.md | 2 +- docs/migration-2.x-to-3.x.md | 34 +- .../lambda/durable/DynamicPluginLoader.java | 90 +++++ .../DynamicPluginLoaderStaleProviderTest.java | 314 ++++++++++++++++++ .../durable/DynamicPluginLoaderTest.java | 97 ++++++ 5 files changed, 528 insertions(+), 9 deletions(-) create mode 100644 sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java diff --git a/docs/advanced/configuration.md b/docs/advanced/configuration.md index eaa936e8c..d4072c224 100644 --- a/docs/advanced/configuration.md +++ b/docs/advanced/configuration.md @@ -50,7 +50,7 @@ DURABLE_EXECUTION_PLUGINS=otel-invocation,com.example.audit When the variable is unset or blank, the SDK does not perform provider discovery. During `DurableConfig` construction, the SDK uses `ServiceLoader` and the thread context class loader to find `DurableExecutionPluginProvider` implementations. Only named providers create plugins. -Dynamically loaded plugins run first in the order listed in `DURABLE_EXECUTION_PLUGINS`. Plugins registered through `withPlugins(...)` follow in configuration order. Both sources are additive: if the same plugin is selected dynamically and registered explicitly, both factories are registered and each produces an instance that receives lifecycle hooks. Duplicate configured provider names, duplicate discovered provider names, missing providers, and provider discovery failures stop configuration with an `IllegalStateException`. +Dynamically loaded plugins run first in the order listed in `DURABLE_EXECUTION_PLUGINS`. Plugins registered through `withPlugins(...)` follow in configuration order. Both sources are additive: if the same plugin is selected dynamically and registered explicitly, both factories are registered and each produces an instance that receives lifecycle hooks. Duplicate configured provider names, duplicate discovered provider names, missing providers, provider discovery failures, and selected providers that were built against an older SDK and do not implement `createPlugin(InvocationInfo)` stop configuration with an `IllegalStateException`. To distribute a provider in a Lambda layer, package its JAR under `java/lib`: diff --git a/docs/migration-2.x-to-3.x.md b/docs/migration-2.x-to-3.x.md index 084c56b69..76cbdda48 100644 --- a/docs/migration-2.x-to-3.x.md +++ b/docs/migration-2.x-to-3.x.md @@ -313,28 +313,46 @@ The provider instance itself is created once per execution environment by `Servi ### What happens to a provider that is not rebuilt -This is the failure mode to understand before you deploy, because the function keeps succeeding while its instrumentation stops. +A provider JAR compiled against `2.x` still loads. Its class file references nothing that `3.x` removed, so `ServiceLoader` instantiates it and `getName()` returns its name. -A provider JAR compiled against `2.x` still loads. Its class file references nothing that `3.x` removed, so `ServiceLoader` instantiates it, `getName()` returns its name, and selection through `DURABLE_EXECUTION_PLUGINS` succeeds. Configuration therefore does not fail. +It does not implement `createPlugin(InvocationInfo)` — it implements the zero-argument `createPlugin()` that no longer exists on the interface. Calling the method the class does not implement throws `AbstractMethodError`. -The provider does not implement `createPlugin(InvocationInfo)` — it implements the zero-argument `createPlugin()` that no longer exists on the interface. When the SDK calls the method the class does not implement, the JVM throws `AbstractMethodError`. As of this release that error is contained: the SDK logs it and skips that factory for the invocation, exactly as it does for a factory that throws an exception. The execution proceeds and completes normally. +`3.x` detects that at configuration time for every provider selected through `DURABLE_EXECUTION_PLUGINS`. Selecting a stale provider throws `IllegalStateException` from `DurableConfig` construction, so the handler fails to initialize and the invocation fails: -The result is a deployment that runs correctly and emits nothing from its configured instrumentation. No execution fails, no invocation errors, and the only signal is a warning in the function's own logs, repeated once per invocation, from the `software.amazon.lambda.durable.plugin.PluginRunner` logger: +```text +java.lang.IllegalStateException: Dynamic plugin configuration failed: Plugin provider 'com.example.audit' + (com.example.AuditPluginProvider from file:/opt/java/lib/audit-plugin.jar) does not implement + createPlugin(InvocationInfo). It was compiled against an older Durable Execution SDK whose provider + interface declared a different createPlugin method. Rebuild the provider against this SDK version and + redeploy it. A provider shipped as a Lambda layer is versioned and deployed separately from the function + package, so upgrading the function's SDK dependency does not update the layer. +``` + +The check reads whether `createPlugin(InvocationInfo)` resolves to an abstract method on the provider's class, and calls no provider code. A provider written against `3.x` does not trip it, including one that declares `createPlugin` with a narrowed return type, inherits it from an abstract base class, or inherits it as a default method from a subinterface of `DurableExecutionPluginFactory`. + +Two cases are outside the check. Both produce a function that runs correctly and emits nothing from that provider. + +- **A stale provider on the class path that no name in `DURABLE_EXECUTION_PLUGINS` selects.** An unselected provider is never called, so failing startup for it would break a deployment that works. It is left alone. +- **A stale provider registered directly through `withPlugins(...)` rather than discovered.** `DurableExecutionPluginProvider` extends `DurableExecutionPluginFactory`, so a provider instance is a valid `withPlugins(...)` argument, and the instance passed can come from a stale JAR even though the call site itself was recompiled. That registration is not checked. + +In the second case the outcome is the one per-invocation containment produces. `AbstractMethodError` is thrown once per invocation and contained: the SDK logs it and skips that factory for the invocation, exactly as it does for a factory that throws an exception. The execution proceeds and completes normally. No execution fails, no invocation errors, and the only signal is a warning in the function's own logs, repeated once per invocation, from the `software.amazon.lambda.durable.plugin.PluginRunner` logger: ```text WARN software.amazon.lambda.durable.plugin.PluginRunner - Plugin factory failed; skipping it for this invocation java.lang.AbstractMethodError: com.example.AuditPluginProvider.createPlugin(...) ``` +Per-invocation containment is deliberate and did not change. Instrumentation never decides whether an execution runs, so a failure at the runtime boundary is logged and skipped rather than propagated. Configuration is the boundary where failing fast is already the policy, which is why the startup check is there and not in `PluginRunner`. + If your instrumentation is the thing that produces your traces or audit records, losing it silently is worse than a failed deployment. Rebuild every provider JAR against `3.x` and redeploy it before or with the SDK upgrade. That includes provider JARs shipped as Lambda layers, which are versioned and deployed separately from the function package and are easy to leave behind. -Caveat: containment is what makes this quiet, and containment is not the same as no failure at all. A stale provider whose class body also references an SDK symbol that `3.x` removed can instead fail during discovery, which throws `IllegalStateException` from `DurableConfig` construction and fails loudly. Both outcomes are possible depending on what the provider's code touches; neither is a substitute for rebuilding it. +Caveat: a stale provider whose class body also references an SDK symbol that `3.x` removed can fail earlier still, during discovery, which throws `IllegalStateException` from `DurableConfig` construction with a different message. Both outcomes fail startup, and neither is a substitute for rebuilding the provider. ### Confirming a provider loaded -There is no log line confirming successful provider selection, so confirmation is indirect. Check all three: +A selected provider that was not rebuilt now fails startup, so the case left to confirm is a provider that loads and is selected but produces nothing, and a provider registered directly through `withPlugins(...)`. There is no log line confirming successful provider selection, so confirmation is indirect. Check all three: -1. The function's logs contain no `Plugin factory failed` warning from `PluginRunner`. +1. The function's logs contain no `Plugin factory failed` warning from `PluginRunner` and no `Dynamic plugin configuration failed` initialization error. 2. The provider's own output is present for a recent execution — spans in your trace backend, records at your exporter's destination, or whatever the plugin emits. 3. The deployed provider artifact is the one built against `3.x`. Check the layer version or JAR checksum you deployed, not just the version you built. @@ -357,5 +375,5 @@ A useful pre-deployment check is to run one execution locally with the provider - Environment-lifetime state moves outside the factory lambda; per-invocation state becomes plain instance fields and ARN-keyed maps are deleted - `DurableConfig.getPluginRunner()` is removed in favor of `getPluginFactories()`; `PluginRunner` is an SDK internal and should not be used - `DurableExecutionPluginProvider` keeps only `getName()` and inherits `createPlugin(InvocationInfo)`; `API_VERSION`, `getApiVersion()`, `getPluginType()`, and the zero-argument `createPlugin()` are removed -- A provider that is not rebuilt still loads and is still selected, but produces no instrumentation and only logs a warning, so rebuild and redeploy every provider JAR +- A provider selected through `DURABLE_EXECUTION_PLUGINS` that was not rebuilt fails startup with an `IllegalStateException` naming the provider and its JAR; a stale provider registered directly through `withPlugins(...)` instead produces no instrumentation and only logs a warning, so rebuild and redeploy every provider JAR - There is no compatibility bridge, and recompilation against `3.x` is required diff --git a/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java b/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java index b38cd18b5..1df1278b7 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java @@ -2,6 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 package software.amazon.lambda.durable; +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.LinkedHashSet; @@ -11,10 +13,22 @@ import java.util.ServiceLoader; import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; +import software.amazon.lambda.durable.plugin.InvocationInfo; final class DynamicPluginLoader { static final String PLUGINS_ENVIRONMENT_VARIABLE = "DURABLE_EXECUTION_PLUGINS"; + /** + * Closing sentences shared by the failures a stale provider JAR produces. A provider distributed as a Lambda layer + * has its own version and its own deployment, so raising the function's SDK dependency leaves the deployed provider + * untouched. An operator who does not know that reads "rebuild the provider" as something the function build + * already did, so the remedy names the layer explicitly. + */ + private static final String REBUILD_PROVIDER_REMEDY = + "Rebuild the provider against this SDK version and redeploy it. A provider shipped as a Lambda layer is " + + "versioned and deployed separately from the function package, so upgrading the function's SDK " + + "dependency does not update the layer."; + private DynamicPluginLoader() {} static List loadConfiguredPluginFactories( @@ -118,9 +132,85 @@ private static DurableExecutionPluginProvider getProvider( throw configurationError("No DurableExecutionPluginProvider named '" + name + "' was found on the application class path. Available providers: " + available); } + requireCreatePluginImplementation(name, provider); return provider; } + /** + * Fails when a selected provider does not implement + * {@link DurableExecutionPluginFactory#createPlugin(InvocationInfo)}. + * + *

    A provider JAR compiled against an SDK version whose provider interface declared a different + * {@code createPlugin} method still loads. Its class file references nothing this version removed, so + * {@link ServiceLoader} instantiates it, {@link DurableExecutionPluginProvider#getName()} returns its name, and + * selection by name succeeds. The first call to {@code createPlugin(InvocationInfo)} then throws + * {@link AbstractMethodError}, which is contained per invocation and logged as a warning. Without this check the + * function keeps succeeding while the provider emits nothing, and the only signal is one warning per invocation. + * This check reports the condition as a startup failure instead, which is how every other provider configuration + * problem on this path is already reported. + * + *

    {@link Class#getMethod} resolves to the most specific declaration reachable from the runtime class. A provider + * that declares the method itself, inherits a concrete implementation from a superclass, or inherits a default + * implementation from a subinterface of {@link DurableExecutionPluginFactory} therefore resolves to a non-abstract + * method. A provider that has none of those resolves to the abstract declaration on + * {@link DurableExecutionPluginFactory} itself. The abstract modifier on the resolved method distinguishes the two + * cases, and reading it runs no provider code. + * + *

    A class that inherits a {@code createPlugin(InvocationInfo)} default from an interface unrelated to + * {@link DurableExecutionPluginFactory} does not compile, because an unrelated default does not override the + * factory interface's abstract declaration. That shape cannot reach this check from Java source. + * + *

    The provider reaching this method was already cast to this SDK's {@link DurableExecutionPluginProvider}, so it + * inherits this SDK's {@code createPlugin(InvocationInfo)} declaration and {@link Class#getMethod} finds at least + * that declaration. A failure to resolve the method therefore means the provider's class hierarchy resolves + * {@link InvocationInfo} to a different class than this SDK does, which is a class path problem with the same + * remedy. It is reported as a configuration failure rather than allowed to escape configuration as an unexplained + * {@link NoSuchMethodException}. + */ + private static void requireCreatePluginImplementation(String name, DurableExecutionPluginProvider provider) { + var providerClass = provider.getClass(); + Method createPlugin; + try { + createPlugin = providerClass.getMethod("createPlugin", InvocationInfo.class); + } catch (NoSuchMethodException | LinkageError e) { + throw configurationError( + "Plugin provider '" + name + "' (" + describe(providerClass) + + ") does not expose a createPlugin method that accepts this SDK's InvocationInfo type. " + + REBUILD_PROVIDER_REMEDY, + e); + } + if (Modifier.isAbstract(createPlugin.getModifiers())) { + throw configurationError("Plugin provider '" + name + "' (" + describe(providerClass) + + ") does not implement createPlugin(InvocationInfo). It was compiled against an older " + + "Durable Execution SDK whose provider interface declared a different createPlugin method. " + + REBUILD_PROVIDER_REMEDY); + } + } + + /** Returns the provider class name, with the artifact it was loaded from when the JVM reports one. */ + private static String describe(Class providerClass) { + var location = codeSourceLocation(providerClass); + return location == null ? providerClass.getName() : providerClass.getName() + " from " + location; + } + + /** + * Returns the location of the artifact a class was loaded from, or null when the JVM does not report one. + * + *

    A class defined by a loader that supplies no code source has no location, and a security manager can refuse + * the protection domain. Neither case says anything about whether the provider is usable, so neither may replace + * the configuration failure being reported. Both are therefore reported as an absent location. + */ + private static String codeSourceLocation(Class providerClass) { + try { + var protectionDomain = providerClass.getProtectionDomain(); + var codeSource = protectionDomain == null ? null : protectionDomain.getCodeSource(); + var location = codeSource == null ? null : codeSource.getLocation(); + return location == null ? null : location.toString(); + } catch (RuntimeException e) { + return null; + } + } + private static IllegalStateException configurationError(String message) { return new IllegalStateException("Dynamic plugin configuration failed: " + message); } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java new file mode 100644 index 000000000..270b6a662 --- /dev/null +++ b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java @@ -0,0 +1,314 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; +import static org.junit.jupiter.api.Assumptions.assumeTrue; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.lang.reflect.Modifier; +import java.net.URL; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.CodeSigner; +import java.security.CodeSource; +import java.security.ProtectionDomain; +import java.time.Instant; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; +import javax.tools.JavaCompiler; +import javax.tools.ToolProvider; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; +import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; +import software.amazon.lambda.durable.plugin.InvocationInfo; + +/** + * Covers the startup check that rejects a plugin provider compiled against the older provider interface. + * + *

    The condition being checked is a property of a class file, not of source: the provider's class file declares that + * it implements {@code DurableExecutionPluginProvider} but contains no {@code createPlugin(InvocationInfo)} method. + * That class file cannot be produced from this source tree, because this source tree contains only the current + * interface and a class that fails to implement one of its abstract methods does not compile. Each test here therefore + * compiles a provider against a stub of the older interface and then loads the result against the current interface, + * which is the deployment it stands in for: a provider JAR built against an earlier SDK and left in place while the + * function's SDK dependency was raised. + * + *

    The fixture reproduces the condition rather than approximating it, so these tests establish that the check fires + * on a class file with the shape a stale provider JAR has. A cheaper fixture would not: a + * {@link java.lang.reflect.Proxy} over the provider interface generates a concrete + * {@code createPlugin(InvocationInfo)}, so it does not reproduce the condition at all. + * + *

    What these tests do not establish is that a provider JAR built by some other toolchain against some other 2.x + * point release produces exactly this class file shape. They cover one stale shape, the one the migration guide + * describes. + */ +class DynamicPluginLoaderStaleProviderTest { + + private static final String PROVIDER_CLASS = "com.example.audit.StaleAuditProvider"; + private static final String PROVIDER_NAME = "com.example.audit"; + + /** The plugin interface, which is unchanged, so the stale provider's references to it still resolve. */ + private static final String PLUGIN_SOURCE = """ + package software.amazon.lambda.durable.plugin; + + public interface DurableExecutionPlugin {} + """; + + /** The provider interface as an earlier SDK declared it, against which the fixture provider is compiled. */ + private static final String OLD_PROVIDER_INTERFACE_SOURCE = """ + package software.amazon.lambda.durable.plugin; + + public interface DurableExecutionPluginProvider { + + int API_VERSION = 1; + + String getName(); + + int getApiVersion(); + + Class getPluginType(); + + DurableExecutionPlugin createPlugin(); + } + """; + + /** A provider written against the interface above, exactly as the migration guide's "before" example is. */ + private static final String PROVIDER_SOURCE = """ + package com.example.audit; + + import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; + import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; + + public final class StaleAuditProvider implements DurableExecutionPluginProvider { + + @Override + public String getName() { + return "com.example.audit"; + } + + @Override + public int getApiVersion() { + return API_VERSION; + } + + @Override + public Class getPluginType() { + return StaleAuditPlugin.class; + } + + @Override + public DurableExecutionPlugin createPlugin() { + return new StaleAuditPlugin(); + } + + public static final class StaleAuditPlugin implements DurableExecutionPlugin {} + } + """; + + @Test + void staleProviderIsIndistinguishableFromACurrentOneUntilCreatePluginIsResolved(@TempDir Path workDir) + throws Exception { + var provider = staleProvider(workDir); + + // Nothing the provider's class file references was removed, so it loads, instantiates, and reports its name. + assertEquals(PROVIDER_NAME, provider.getName()); + + // Its createPlugin(InvocationInfo) resolves to the abstract declaration on the interface it inherits. + var createPlugin = provider.getClass().getMethod("createPlugin", InvocationInfo.class); + assertTrue(Modifier.isAbstract(createPlugin.getModifiers())); + assertTrue(createPlugin.getDeclaringClass().isInterface()); + + // Calling it fails, which is the outcome the startup check exists to reach first. + assertThrows(AbstractMethodError.class, () -> provider.createPlugin(invocationInfo())); + } + + @Test + void rejectsSelectedStaleProviderAtConfigurationTime(@TempDir Path workDir) throws Exception { + var provider = staleProvider(workDir); + + var error = assertThrows( + IllegalStateException.class, + () -> DynamicPluginLoader.loadConfiguredPluginFactories(PROVIDER_NAME, List.of(provider), List.of())); + + var message = error.getMessage(); + assertTrue(message.contains("Dynamic plugin configuration failed"), message); + assertTrue(message.contains("Plugin provider '" + PROVIDER_NAME + "'"), message); + assertTrue(message.contains(PROVIDER_CLASS), message); + assertTrue(message.contains("does not implement createPlugin(InvocationInfo)"), message); + assertTrue(message.contains("compiled against an older Durable Execution SDK"), message); + assertTrue(message.contains("Rebuild the provider against this SDK version and redeploy it"), message); + assertTrue(message.contains("Lambda layer is versioned and deployed separately"), message); + + // The artifact to rebuild is named, because an operator with several provider layers deployed needs to know + // which one is stale. + var artifactLocation = + provider.getClass().getProtectionDomain().getCodeSource().getLocation(); + assertTrue(message.contains(artifactLocation.toString()), message); + } + + @Test + void doesNotRejectAStaleProviderThatWasNotSelected(@TempDir Path workDir) throws Exception { + var staleProvider = staleProvider(workDir); + var selectedProvider = new CurrentProvider(); + + // A stale provider JAR on the class path that no name in the environment variable selects is never called, so + // rejecting it would fail startup for a deployment that works. Only selected providers are checked. + var factories = DynamicPluginLoader.loadConfiguredPluginFactories( + "current", List.of(staleProvider, selectedProvider), List.of()); + + assertEquals(List.of(selectedProvider), factories); + } + + @Test + void namesTheProviderClassWhenNoArtifactLocationIsReported(@TempDir Path workDir) throws Exception { + // A class whose loader reports no code source has no artifact to name. That says nothing about whether the + // provider is usable, so the failure is still reported and only the location is left out of the message. + var provider = staleProvider(workDir, null); + + var error = assertThrows( + IllegalStateException.class, + () -> DynamicPluginLoader.loadConfiguredPluginFactories(PROVIDER_NAME, List.of(provider), List.of())); + + var message = error.getMessage(); + assertTrue(message.contains("(" + PROVIDER_CLASS + ")"), message); + assertTrue(message.contains("does not implement createPlugin(InvocationInfo)"), message); + } + + private static InvocationInfo invocationInfo() { + return new InvocationInfo("req-123", "arn:test", true, Instant.now()); + } + + /** + * Compiles the fixture provider against the older interface and returns an instance of it loaded against the + * current interface. + * + *

    The stub interfaces are compiled only so the provider source has something to compile against, and they are + * not handed to the loader. Class loading for every {@code software.amazon.lambda.durable} name therefore reaches + * the parent loader and resolves to this SDK's classes, which is the same resolution a deployed provider JAR gets + * from the function class path. + */ + private static DurableExecutionPluginProvider staleProvider(Path workDir) throws Exception { + return staleProvider(workDir, workDir.resolve("classes").toUri().toURL()); + } + + /** @param artifactLocation reported as the fixture classes' code source, or null to report none */ + private static DurableExecutionPluginProvider staleProvider(Path workDir, URL artifactLocation) throws Exception { + var compiler = ToolProvider.getSystemJavaCompiler(); + assumeTrue(compiler != null, "This test compiles a fixture and needs a JDK rather than a JRE"); + + var classDir = compileFixture(compiler, workDir); + var loader = new FixtureClassLoader( + DynamicPluginLoaderStaleProviderTest.class.getClassLoader(), + fixtureClasses(classDir), + artifactLocation); + var type = loader.loadClass(PROVIDER_CLASS); + return (DurableExecutionPluginProvider) type.getDeclaredConstructor().newInstance(); + } + + private static Path compileFixture(JavaCompiler compiler, Path workDir) throws Exception { + var sourceDir = Files.createDirectories(workDir.resolve("source")); + var classDir = Files.createDirectories(workDir.resolve("classes")); + var sources = new String[] { + write(sourceDir, "DurableExecutionPlugin.java", PLUGIN_SOURCE), + write(sourceDir, "DurableExecutionPluginProvider.java", OLD_PROVIDER_INTERFACE_SOURCE), + write(sourceDir, "StaleAuditProvider.java", PROVIDER_SOURCE), + }; + + // The class path holds only the output directory, which is empty when the compile starts. This SDK's current + // interfaces are therefore not visible to the compile, and the provider is compiled against the stub above + // rather than against the interface it is meant to predate. + var arguments = Stream.concat( + Stream.of("--release", "17", "-classpath", classDir.toString(), "-d", classDir.toString()), + Stream.of(sources)) + .toArray(String[]::new); + var diagnostics = new ByteArrayOutputStream(); + var exitCode = compiler.run(null, null, diagnostics, arguments); + if (exitCode != 0) { + fail("Failed to compile the stale provider fixture: " + diagnostics.toString(StandardCharsets.UTF_8)); + } + return classDir; + } + + /** + * Returns the compiled fixture classes outside the {@code software.amazon.lambda.durable} packages, keyed by binary + * name. + * + *

    Excluding those packages is what leaves the stub interfaces behind. A stub that reached the loader would + * shadow this SDK's interface of the same name, and the provider would then implement the stub rather than the + * current interface, which is not the condition under test. + */ + private static Map fixtureClasses(Path classDir) throws Exception { + var classes = new HashMap(); + try (var files = Files.walk(classDir)) { + for (var file : files.filter(f -> f.toString().endsWith(".class")).toList()) { + var relativePath = classDir.relativize(file).toString(); + var binaryName = relativePath + .substring(0, relativePath.length() - ".class".length()) + .replace(File.separatorChar, '.'); + if (!binaryName.startsWith("software.amazon.lambda.durable.")) { + classes.put(binaryName, Files.readAllBytes(file)); + } + } + } + return classes; + } + + private static String write(Path sourceDir, String fileName, String source) throws Exception { + var file = sourceDir.resolve(fileName); + Files.writeString(file, source); + return file.toString(); + } + + /** Defines the fixture classes and delegates every other name to the parent loader. */ + private static final class FixtureClassLoader extends ClassLoader { + + private final Map fixtureClasses; + private final ProtectionDomain protectionDomain; + + /** + * @param artifactLocation where the fixture classes were loaded from, reported as their code source so the + * failure message can name it as it names a deployed provider's JAR, or null to report no code source + */ + FixtureClassLoader(ClassLoader parent, Map fixtureClasses, URL artifactLocation) { + super(parent); + this.fixtureClasses = Map.copyOf(fixtureClasses); + this.protectionDomain = artifactLocation == null + ? null + : new ProtectionDomain(new CodeSource(artifactLocation, (CodeSigner[]) null), null); + } + + @Override + protected Class findClass(String name) throws ClassNotFoundException { + var bytes = fixtureClasses.get(name); + if (bytes == null) { + return super.findClass(name); + } + return defineClass(name, bytes, 0, bytes.length, protectionDomain); + } + } + + /** A provider written against the current interface, used to show that only selected providers are checked. */ + private static final class CurrentProvider implements DurableExecutionPluginProvider { + + @Override + public String getName() { + return "current"; + } + + @Override + public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { + return new CurrentPlugin(); + } + } + + private static final class CurrentPlugin implements DurableExecutionPlugin {} +} diff --git a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderTest.java b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderTest.java index 2c8992c9f..1aa7a572b 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderTest.java @@ -170,6 +170,55 @@ public DurableExecutionPluginProvider next() { assertInstanceOf(LinkageError.class, error.getCause()); } + // ─── Providers that do implement createPlugin(InvocationInfo) ──────── + // + // The startup check that rejects a provider compiled against the older provider interface reads whether + // createPlugin(InvocationInfo) resolves to an abstract method on the runtime class. These cases cover the shapes + // in which a provider written against this SDK supplies that method without declaring it on its own class, so the + // check must accept all of them. DynamicPluginLoaderStaleProviderTest covers the case the check rejects. + + @Test + void acceptsProviderThatDeclaresCreatePluginItself() { + var declaringProvider = provider("declaring", FirstPlugin::new); + + var factories = + DynamicPluginLoader.loadConfiguredPluginFactories("declaring", List.of(declaringProvider), List.of()); + + assertInstanceOf(FirstPlugin.class, factories.get(0).createPlugin(invocationInfo())); + } + + @Test + void acceptsProviderThatInheritsCreatePluginFromAbstractBaseClass() { + var inheritingProvider = new InheritsFromBaseProvider(); + + var factories = DynamicPluginLoader.loadConfiguredPluginFactories( + "inherits-from-base", List.of(inheritingProvider), List.of()); + + assertInstanceOf(FirstPlugin.class, factories.get(0).createPlugin(invocationInfo())); + } + + @Test + void acceptsProviderThatInheritsCreatePluginAsDefaultMethod() { + var inheritingProvider = new InheritsDefaultMethodProvider(); + + var factories = DynamicPluginLoader.loadConfiguredPluginFactories( + "inherits-default-method", List.of(inheritingProvider), List.of()); + + assertInstanceOf(FirstPlugin.class, factories.get(0).createPlugin(invocationInfo())); + } + + @Test + void acceptsProviderThatNarrowsTheCreatePluginReturnType() { + // A narrowed return type makes the compiler emit a bridge method, so createPlugin(InvocationInfo) resolves to + // one of two declarations on the provider class. Neither is abstract. + var covariantProvider = new NarrowedReturnTypeProvider(); + + var factories = DynamicPluginLoader.loadConfiguredPluginFactories( + "narrowed-return-type", List.of(covariantProvider), List.of()); + + assertInstanceOf(FirstPlugin.class, factories.get(0).createPlugin(invocationInfo())); + } + private static InvocationInfo invocationInfo() { return new InvocationInfo("req-123", "arn:test", true, Instant.now()); } @@ -197,4 +246,52 @@ private static final class ExplicitPlugin implements DurableExecutionPlugin {} private static final class FirstPlugin implements DurableExecutionPlugin {} private static final class SecondPlugin implements DurableExecutionPlugin {} + + /** A provider whose {@code createPlugin} implementation is inherited from a superclass. */ + private abstract static class BaseProvider implements DurableExecutionPluginProvider { + + @Override + public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { + return new FirstPlugin(); + } + } + + private static final class InheritsFromBaseProvider extends BaseProvider { + + @Override + public String getName() { + return "inherits-from-base"; + } + } + + /** A provider whose {@code createPlugin} implementation is inherited as a default method. */ + private interface DefaultMethodProvider extends DurableExecutionPluginProvider { + + @Override + default DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { + return new FirstPlugin(); + } + } + + private static final class InheritsDefaultMethodProvider implements DefaultMethodProvider { + + @Override + public String getName() { + return "inherits-default-method"; + } + } + + /** A provider that declares {@code createPlugin} with a narrowed return type. */ + private static final class NarrowedReturnTypeProvider implements DurableExecutionPluginProvider { + + @Override + public String getName() { + return "narrowed-return-type"; + } + + @Override + public FirstPlugin createPlugin(InvocationInfo invocationInfo) { + return new FirstPlugin(); + } + } } From 489490fe1e8ef0f800c89827a57ff86b47a4bfab Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 13:06:15 -0700 Subject: [PATCH 09/19] fix(plugin): stop interrupting the handler thread onInvocationStart and the factories run on the handler thread, by design: a plugin sets a ThreadLocal or an MDC key there and the handler's own logging reads it. Restoring the interrupt status after containing an InterruptedException therefore left the handler's next blocking call to fail with an interrupt no user code asked for, which is the containment contract broken by the boundary that enforces it. A thrown InterruptedException is also no proof of interruption. No hook and no factory method declares a checked exception, so plugin code reaches the boundary with one only by rethrowing it undeclared, and it can construct one with the status clear. No SDK code interrupts these threads or reads their interrupt status, so the restored flag served no reader. The boundary now neither sets nor clears the flag. An interrupt the thread already carried still survives, which a new test pins. --- .../lambda/durable/plugin/PluginRunner.java | 20 ++++---- .../durable/plugin/PluginRunnerTest.java | 48 ++++++++++++++----- 2 files changed, 47 insertions(+), 21 deletions(-) diff --git a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java index 5bf4d835e..7ec785a79 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java @@ -148,12 +148,17 @@ private void run(Consumer hook) { * class is compiled on a JDK 20 or later compiler. The {@code @SuppressWarnings("removal")} below is scoped to this * method rather than the class so it cannot mask a removal warning that appears elsewhere in {@code PluginRunner}. * - *

    Throwing an {@link InterruptedException} clears the throwing thread's interrupt status. The threads that run - * factories and hooks are SDK threads that carry SDK work after the plugin returns, so containing the interrupt - * without restoring the status would hide the cancellation request from that later work and from the SDK's own - * blocking calls. The status is therefore restored before returning. Restoring it immediately rather than after the - * dispatch loop keeps the flag true for every subsequent read on this thread; a remaining plugin whose blocking - * call then fails fast is contained by this same rule, so every plugin is still called. + *

    An {@link InterruptedException} is contained like any other non-fatal throwable, and the interrupt status is + * not restored. Three facts decide it. The thread that creates plugins and fires {@code onInvocationStart} is the + * handler thread — the hook runs there on purpose, so a plugin can set a {@code ThreadLocal} or an MDC key the + * handler's own logging then reads — so setting the flag there leaves the handler's next blocking call to fail with + * an {@code InterruptedException} that no user code asked for, which is the containment contract broken by the + * boundary meant to enforce it. A thrown {@code InterruptedException} is also no proof that the thread was + * interrupted: no hook and no factory method declares a checked exception, so the only way one arrives is plugin + * code rethrowing it undeclared, and plugin code can construct one and throw it with the interrupt status clear. + * And no SDK code interrupts these threads or reads their interrupt status, so restoring the flag serves no waiting + * reader. The interrupt is reported the way every other contained plugin failure is, as a logged warning naming the + * plugin boundary that produced it. */ @SuppressWarnings("removal") // ThreadDeath is deprecated for removal since JDK 20; see the javadoc above. private static void contain(Throwable t, String message) { @@ -163,9 +168,6 @@ private static void contain(Throwable t, String message) { if (t instanceof ThreadDeath fatal) { throw fatal; } - if (t instanceof InterruptedException) { - Thread.currentThread().interrupt(); - } logger.warn(message, t); } diff --git a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java index ec1f7d5bb..cefbabe2c 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java @@ -309,16 +309,15 @@ void hookThrowingServiceConfigurationError_isContained_andRemainingPluginsStillR // ─── Interrupts ────────────────────────────────────────────────────── // - // Throwing InterruptedException clears the throwing thread's interrupt status. The thread that fires a hook is an - // SDK thread that carries SDK work after the hook returns, so a runner that contains the InterruptedException - // without restoring the status hides the cancellation request from that later SDK work. The runner therefore - // contains the throwable, as the contract requires, and restores the interrupt status before returning. - // - // No hook and no factory method declares a checked exception, so plugin code reaches the boundary with an - // InterruptedException only by rethrowing it undeclared. The tests below use that shape deliberately. + // An InterruptedException from plugin code is contained like any other non-fatal throwable, and the interrupt + // status is left alone. onInvocationStart runs on the handler thread, so setting the flag there would leave the + // handler's next blocking call to fail with an interrupt no user code asked for. A thrown InterruptedException is + // also no proof of interruption: no hook and no factory method declares a checked exception, so plugin code reaches + // the boundary with one only by rethrowing it undeclared, and it can construct one with the status clear. The tests + // below use that shape deliberately. @Test - void factoryThrowingInterruptedException_isContained_andRestoresTheInterruptStatus() { + void factoryThrowingInterruptedException_isContained_andLeavesTheThreadUninterrupted() { var calls = new ArrayList(); var runner = new PluginRunner(List.of( info -> { @@ -330,16 +329,18 @@ void factoryThrowingInterruptedException_isContained_andRestoresTheInterruptStat try { assertDoesNotThrow(() -> runner.onInvocationStart(invocationInfo())); - assertTrue(Thread.currentThread().isInterrupted(), "the interrupt status must survive containment"); + assertFalse( + Thread.currentThread().isInterrupted(), + "containment must not interrupt the thread that runs the handler"); assertEquals(List.of("p2:onInvocationStart"), calls); } finally { - // Clear the status so it does not leak into whatever else runs on this thread. + // Clear the status so a failure here does not leak into whatever else runs on this thread. Thread.interrupted(); } } @Test - void hookThrowingInterruptedException_isContained_andRestoresTheInterruptStatus() { + void hookThrowingInterruptedException_isContained_andLeavesTheThreadUninterrupted() { var calls = new ArrayList(); var runner = new PluginRunner(List.of(info -> new InterruptingPlugin(), info -> new TestPlugin("p2", calls))); runner.onInvocationStart(invocationInfo()); @@ -348,13 +349,36 @@ void hookThrowingInterruptedException_isContained_andRestoresTheInterruptStatus( try { assertDoesNotThrow(() -> runner.onInvocationEnd(invocationEndInfo())); - assertTrue(Thread.currentThread().isInterrupted(), "the interrupt status must survive containment"); + assertFalse( + Thread.currentThread().isInterrupted(), + "containment must not interrupt the thread that runs the handler"); assertEquals(List.of("p2:onInvocationEnd"), calls, "remaining plugins must still be called"); } finally { Thread.interrupted(); } } + @Test + void containmentPreservesAnInterruptTheThreadAlreadyCarried() { + // The boundary neither sets nor clears the flag: a thread that was already interrupted before it entered plugin + // code still carries the interrupt when containment returns. + var calls = new ArrayList(); + var runner = new PluginRunner(List.of(info -> new ThrowingPlugin(), info -> new TestPlugin("p2", calls))); + runner.onInvocationStart(invocationInfo()); + calls.clear(); + + try { + Thread.currentThread().interrupt(); + + assertDoesNotThrow(() -> runner.onInvocationEnd(invocationEndInfo())); + + assertTrue(Thread.currentThread().isInterrupted(), "an interrupt the thread already carried must survive"); + assertEquals(List.of("p2:onInvocationEnd"), calls); + } finally { + Thread.interrupted(); + } + } + @Test void factoryThrowingAJvmError_stillPropagates() { // The containment is deliberately narrow: an Error that says the JVM itself is failing must not be swallowed as From 0ed998a0d91db285b5001ed5895506985e4928c2 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 13:32:10 -0700 Subject: [PATCH 10/19] fix(plugin): check the resolved createPlugin fully The load-time check read only the abstract modifier, which the resolved method being usable does not follow from. Class#getMethod searches the class before the interfaces it implements and returns static methods, so a stale class carrying a static createPlugin(InvocationInfo) helper resolved to that helper while the instance method the interface call dispatches to was still missing. A class file whose createPlugin(InvocationInfo) returns an unrelated type does not override the interface method either, and is also concrete. Both passed the check and then threw AbstractMethodError on the first call, which is contained per invocation, so the provider emitted nothing while the function kept succeeding. The check now rejects a resolved method that is static, abstract, or does not return a DurableExecutionPlugin, and names which of the three it was. A covariant override and the bridge javac generates for one both return a DurableExecutionPlugin subtype, so the return check refuses no valid provider. All three are class-file properties, so none runs provider code. Two fixtures compile the new shapes against the older interface and load them against this one, and both assert the AbstractMethodError the check now precedes. --- .../lambda/durable/DynamicPluginLoader.java | 42 ++++- .../DynamicPluginLoaderStaleProviderTest.java | 151 +++++++++++++++++- 2 files changed, 183 insertions(+), 10 deletions(-) diff --git a/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java b/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java index 1df1278b7..00ecedc43 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java @@ -11,6 +11,7 @@ import java.util.Map; import java.util.ServiceConfigurationError; import java.util.ServiceLoader; +import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; import software.amazon.lambda.durable.plugin.DurableExecutionPluginFactory; import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; import software.amazon.lambda.durable.plugin.InvocationInfo; @@ -153,8 +154,17 @@ private static DurableExecutionPluginProvider getProvider( * that declares the method itself, inherits a concrete implementation from a superclass, or inherits a default * implementation from a subinterface of {@link DurableExecutionPluginFactory} therefore resolves to a non-abstract * method. A provider that has none of those resolves to the abstract declaration on - * {@link DurableExecutionPluginFactory} itself. The abstract modifier on the resolved method distinguishes the two - * cases, and reading it runs no provider code. + * {@link DurableExecutionPluginFactory} itself. + * + *

    Three properties of the resolved method are read, because the resolved method is not necessarily the one the + * interface call dispatches to. It must not be abstract, which is the stale provider above. It must not be static: + * {@link Class#getMethod} searches the class before the interfaces it implements and returns static methods, so a + * stale class carrying a static {@code createPlugin(InvocationInfo)} helper resolves to that helper while the + * instance method the interface call needs is still missing. And its return type must be a + * {@link DurableExecutionPlugin}: a class file whose {@code createPlugin(InvocationInfo)} returns something else + * does not override the interface method at all, and a covariant override or the bridge javac generates for one + * both return a {@link DurableExecutionPlugin} subtype, so requiring it refuses no valid provider. Each of the + * three is a class-file property, so reading them runs no provider code. * *

    A class that inherits a {@code createPlugin(InvocationInfo)} default from an interface unrelated to * {@link DurableExecutionPluginFactory} does not compile, because an unrelated default does not override the @@ -179,12 +189,32 @@ private static void requireCreatePluginImplementation(String name, DurableExecut + REBUILD_PROVIDER_REMEDY, e); } - if (Modifier.isAbstract(createPlugin.getModifiers())) { + var reason = unimplementedReason(createPlugin); + if (reason != null) { throw configurationError("Plugin provider '" + name + "' (" + describe(providerClass) - + ") does not implement createPlugin(InvocationInfo). It was compiled against an older " - + "Durable Execution SDK whose provider interface declared a different createPlugin method. " - + REBUILD_PROVIDER_REMEDY); + + ") does not implement createPlugin(InvocationInfo): " + reason + + ". It was compiled against an older Durable Execution SDK whose provider interface declared a " + + "different createPlugin method. " + REBUILD_PROVIDER_REMEDY); + } + } + + /** Returns why the resolved method cannot serve the interface call, or null when it can. */ + private static String unimplementedReason(Method createPlugin) { + if (Modifier.isStatic(createPlugin.getModifiers())) { + return "the createPlugin(InvocationInfo) it declares is static, so it cannot implement the interface's " + + "instance method"; + } + if (Modifier.isAbstract(createPlugin.getModifiers())) { + return "the only declaration is the abstract one on " + + createPlugin.getDeclaringClass().getName(); + } + if (!DurableExecutionPlugin.class.isAssignableFrom(createPlugin.getReturnType())) { + return "its createPlugin(InvocationInfo) returns " + + createPlugin.getReturnType().getName() + " rather than a " + + DurableExecutionPlugin.class.getName() + + ", so it does not override the interface method"; } + return null; } /** Returns the provider class name, with the artifact it was loaded from when the JVM reports one. */ diff --git a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java index 270b6a662..d145d7af0 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java @@ -81,6 +81,94 @@ public interface DurableExecutionPluginProvider { } """; + /** + * The invocation info as a stub, so a fixture can name it in a signature. + * + *

    Not handed to the loader, so a compiled reference to it resolves to this SDK's class at load time and the + * fixture's method descriptor matches the one {@code getMethod} is asked for. + */ + private static final String INVOCATION_INFO_SOURCE = """ + package software.amazon.lambda.durable.plugin; + + public final class InvocationInfo {} + """; + + /** A provider whose only createPlugin(InvocationInfo) is static, which cannot implement an instance method. */ + private static final String STATIC_PROVIDER_SOURCE = """ + package com.example.audit; + + import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; + import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; + import software.amazon.lambda.durable.plugin.InvocationInfo; + + public final class StaleAuditProvider implements DurableExecutionPluginProvider { + + @Override + public String getName() { + return "com.example.audit"; + } + + @Override + public int getApiVersion() { + return API_VERSION; + } + + @Override + public Class getPluginType() { + return StaleAuditPlugin.class; + } + + @Override + public DurableExecutionPlugin createPlugin() { + return new StaleAuditPlugin(); + } + + public static DurableExecutionPlugin createPlugin(InvocationInfo info) { + return new StaleAuditPlugin(); + } + + public static final class StaleAuditPlugin implements DurableExecutionPlugin {} + } + """; + + /** A provider whose createPlugin(InvocationInfo) returns something that is not a plugin. */ + private static final String WRONG_RETURN_PROVIDER_SOURCE = """ + package com.example.audit; + + import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; + import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; + import software.amazon.lambda.durable.plugin.InvocationInfo; + + public final class StaleAuditProvider implements DurableExecutionPluginProvider { + + @Override + public String getName() { + return "com.example.audit"; + } + + @Override + public int getApiVersion() { + return API_VERSION; + } + + @Override + public Class getPluginType() { + return StaleAuditPlugin.class; + } + + @Override + public DurableExecutionPlugin createPlugin() { + return new StaleAuditPlugin(); + } + + public String createPlugin(InvocationInfo info) { + return "not a plugin"; + } + + public static final class StaleAuditPlugin implements DurableExecutionPlugin {} + } + """; + /** A provider written against the interface above, exactly as the migration guide's "before" example is. */ private static final String PROVIDER_SOURCE = """ package com.example.audit; @@ -155,6 +243,48 @@ void rejectsSelectedStaleProviderAtConfigurationTime(@TempDir Path workDir) thro assertTrue(message.contains(artifactLocation.toString()), message); } + @Test + void rejectsAProviderWhoseCreatePluginIsStatic(@TempDir Path workDir) throws Exception { + // getMethod searches the class before the interfaces it implements and returns static methods, so a stale class + // carrying a static createPlugin(InvocationInfo) helper resolves to that helper. The instance method the + // interface call dispatches to is still missing, so the absence of the abstract modifier proves nothing here. + var provider = staleProviderOfShape(workDir, STATIC_PROVIDER_SOURCE); + + var createPlugin = provider.getClass().getMethod("createPlugin", InvocationInfo.class); + assertTrue(Modifier.isStatic(createPlugin.getModifiers())); + assertTrue(!Modifier.isAbstract(createPlugin.getModifiers())); + assertThrows(AbstractMethodError.class, () -> provider.createPlugin(invocationInfo())); + + var error = assertThrows( + IllegalStateException.class, + () -> DynamicPluginLoader.loadConfiguredPluginFactories(PROVIDER_NAME, List.of(provider), List.of())); + + var message = error.getMessage(); + assertTrue(message.contains("does not implement createPlugin(InvocationInfo)"), message); + assertTrue(message.contains("is static"), message); + assertTrue(message.contains("Rebuild the provider against this SDK version and redeploy it"), message); + } + + @Test + void rejectsAProviderWhoseCreatePluginReturnsSomethingElse(@TempDir Path workDir) throws Exception { + // A createPlugin(InvocationInfo) whose return type is unrelated to DurableExecutionPlugin does not override the + // interface method, so it is concrete and still leaves the interface call unimplemented. + var provider = staleProviderOfShape(workDir, WRONG_RETURN_PROVIDER_SOURCE); + + var createPlugin = provider.getClass().getMethod("createPlugin", InvocationInfo.class); + assertEquals(String.class, createPlugin.getReturnType()); + assertTrue(!Modifier.isAbstract(createPlugin.getModifiers())); + assertThrows(AbstractMethodError.class, () -> provider.createPlugin(invocationInfo())); + + var error = assertThrows( + IllegalStateException.class, + () -> DynamicPluginLoader.loadConfiguredPluginFactories(PROVIDER_NAME, List.of(provider), List.of())); + + var message = error.getMessage(); + assertTrue(message.contains("does not implement createPlugin(InvocationInfo)"), message); + assertTrue(message.contains("returns java.lang.String"), message); + } + @Test void doesNotRejectAStaleProviderThatWasNotSelected(@TempDir Path workDir) throws Exception { var staleProvider = staleProvider(workDir); @@ -197,15 +327,27 @@ private static InvocationInfo invocationInfo() { * from the function class path. */ private static DurableExecutionPluginProvider staleProvider(Path workDir) throws Exception { - return staleProvider(workDir, workDir.resolve("classes").toUri().toURL()); + return staleProviderOfShape(workDir, PROVIDER_SOURCE); + } + + /** @param providerSource the stale shape to compile, one of the provider sources above */ + private static DurableExecutionPluginProvider staleProviderOfShape(Path workDir, String providerSource) + throws Exception { + return staleProvider(workDir, workDir.resolve("classes").toUri().toURL(), providerSource); } /** @param artifactLocation reported as the fixture classes' code source, or null to report none */ private static DurableExecutionPluginProvider staleProvider(Path workDir, URL artifactLocation) throws Exception { + return staleProvider(workDir, artifactLocation, PROVIDER_SOURCE); + } + + /** @param artifactLocation reported as the fixture classes' code source, or null to report none */ + private static DurableExecutionPluginProvider staleProvider( + Path workDir, URL artifactLocation, String providerSource) throws Exception { var compiler = ToolProvider.getSystemJavaCompiler(); assumeTrue(compiler != null, "This test compiles a fixture and needs a JDK rather than a JRE"); - var classDir = compileFixture(compiler, workDir); + var classDir = compileFixture(compiler, workDir, providerSource); var loader = new FixtureClassLoader( DynamicPluginLoaderStaleProviderTest.class.getClassLoader(), fixtureClasses(classDir), @@ -214,13 +356,14 @@ private static DurableExecutionPluginProvider staleProvider(Path workDir, URL ar return (DurableExecutionPluginProvider) type.getDeclaredConstructor().newInstance(); } - private static Path compileFixture(JavaCompiler compiler, Path workDir) throws Exception { + private static Path compileFixture(JavaCompiler compiler, Path workDir, String providerSource) throws Exception { var sourceDir = Files.createDirectories(workDir.resolve("source")); var classDir = Files.createDirectories(workDir.resolve("classes")); var sources = new String[] { write(sourceDir, "DurableExecutionPlugin.java", PLUGIN_SOURCE), write(sourceDir, "DurableExecutionPluginProvider.java", OLD_PROVIDER_INTERFACE_SOURCE), - write(sourceDir, "StaleAuditProvider.java", PROVIDER_SOURCE), + write(sourceDir, "InvocationInfo.java", INVOCATION_INFO_SOURCE), + write(sourceDir, "StaleAuditProvider.java", providerSource), }; // The class path holds only the output directory, which is empty when the compile starts. This SDK's current From b222d0135f65bde35ed648491115bc7eb75f9b06 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 14:20:34 -0700 Subject: [PATCH 11/19] fix(plugin): require the dispatchable createPlugin The check tested that the resolved method's return type was assignable to DurableExecutionPlugin, which is looser than what the call needs. A class compiled against an older interface can declare a concrete MyPlugin createPlugin(InvocationInfo) that overrides nothing, so javac emits no bridge carrying the interface's erased descriptor. Assignability passed it; invokeinterface looks for that descriptor, finds none, and throws AbstractMethodError, which is contained per invocation. The check now requires what dispatch requires: a public, non-static, non-abstract createPlugin taking this SDK's InvocationInfo and returning exactly DurableExecutionPlugin. It cannot reject a provider that would have worked, because a covariant override compiles to the specific method plus that bridge, and a class without the descriptor is one the JVM cannot dispatch to either. The whole public method set is examined rather than the one getMethod resolves. That resolution prefers the most specific return type, so it hides the bridge behind the covariant declaration, and it searches the class before its interfaces, so it returns a static same-signature helper in preference to the interface declaration. Two fixtures: the no-bridge shape, which asserts an assignability test would have accepted it and that the call throws, and a covariant provider compiled against the current interface, which must still be accepted. --- .../lambda/durable/DynamicPluginLoader.java | 105 +++++++++++------- .../DynamicPluginLoaderStaleProviderTest.java | 104 +++++++++++++++++ 2 files changed, 168 insertions(+), 41 deletions(-) diff --git a/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java b/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java index 00ecedc43..7d596f09e 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/DynamicPluginLoader.java @@ -150,46 +150,40 @@ private static DurableExecutionPluginProvider getProvider( * This check reports the condition as a startup failure instead, which is how every other provider configuration * problem on this path is already reported. * - *

    {@link Class#getMethod} resolves to the most specific declaration reachable from the runtime class. A provider - * that declares the method itself, inherits a concrete implementation from a superclass, or inherits a default - * implementation from a subinterface of {@link DurableExecutionPluginFactory} therefore resolves to a non-abstract - * method. A provider that has none of those resolves to the abstract declaration on - * {@link DurableExecutionPluginFactory} itself. + *

    What is checked is the condition {@code invokeinterface} itself needs: a public, non-static, non-abstract + * method named {@code createPlugin} taking this SDK's {@link InvocationInfo} and returning exactly + * {@link DurableExecutionPlugin}, which is the erased descriptor the interface declares. Checking anything looser + * accepts class files the call cannot dispatch to. A concrete {@code MyPlugin createPlugin(InvocationInfo)} that + * overrides nothing -- which is what a class compiled against an older interface declares -- is such a file: its + * return type is a {@link DurableExecutionPlugin} subtype, so an assignability test passes it, while the interface + * call still finds no matching descriptor and throws. * - *

    Three properties of the resolved method are read, because the resolved method is not necessarily the one the - * interface call dispatches to. It must not be abstract, which is the stale provider above. It must not be static: - * {@link Class#getMethod} searches the class before the interfaces it implements and returns static methods, so a - * stale class carrying a static {@code createPlugin(InvocationInfo)} helper resolves to that helper while the - * instance method the interface call needs is still missing. And its return type must be a - * {@link DurableExecutionPlugin}: a class file whose {@code createPlugin(InvocationInfo)} returns something else - * does not override the interface method at all, and a covariant override or the bridge javac generates for one - * both return a {@link DurableExecutionPlugin} subtype, so requiring it refuses no valid provider. Each of the - * three is a class-file property, so reading them runs no provider code. + *

    Requiring the exact descriptor cannot reject a provider that would have worked, because the descriptor is what + * dispatch resolves. A covariant override compiles to the specific method plus a bridge that returns + * {@link DurableExecutionPlugin}, and it is the bridge the interface call reaches; a compiler that emitted no + * bridge would produce a class the JVM cannot dispatch to either. The whole public method set is examined rather + * than the one {@link Class#getMethod} resolves, because that resolution prefers the most specific return type and + * so hides the bridge behind the covariant declaration, and because it searches the class before the interfaces and + * so returns a static same-signature helper in preference to the interface's declaration. * - *

    A class that inherits a {@code createPlugin(InvocationInfo)} default from an interface unrelated to - * {@link DurableExecutionPluginFactory} does not compile, because an unrelated default does not override the - * factory interface's abstract declaration. That shape cannot reach this check from Java source. + *

    Every property read is a class-file property, so this runs no provider code. * - *

    The provider reaching this method was already cast to this SDK's {@link DurableExecutionPluginProvider}, so it - * inherits this SDK's {@code createPlugin(InvocationInfo)} declaration and {@link Class#getMethod} finds at least - * that declaration. A failure to resolve the method therefore means the provider's class hierarchy resolves - * {@link InvocationInfo} to a different class than this SDK does, which is a class path problem with the same - * remedy. It is reported as a configuration failure rather than allowed to escape configuration as an unexplained - * {@link NoSuchMethodException}. + *

    {@link Class#getMethods} can raise a {@link LinkageError} while resolving a method's parameter or return type + * against a class path that cannot supply it. That is a class path problem with the same remedy, so it is reported + * as this configuration failure rather than escaping as an unexplained {@code NoClassDefFoundError}. */ private static void requireCreatePluginImplementation(String name, DurableExecutionPluginProvider provider) { var providerClass = provider.getClass(); - Method createPlugin; + String reason; try { - createPlugin = providerClass.getMethod("createPlugin", InvocationInfo.class); - } catch (NoSuchMethodException | LinkageError e) { + reason = undispatchableCreatePluginReason(providerClass); + } catch (LinkageError e) { throw configurationError( "Plugin provider '" + name + "' (" + describe(providerClass) - + ") does not expose a createPlugin method that accepts this SDK's InvocationInfo type. " + + ") declares a createPlugin method whose types this class path cannot resolve. " + REBUILD_PROVIDER_REMEDY, e); } - var reason = unimplementedReason(createPlugin); if (reason != null) { throw configurationError("Plugin provider '" + name + "' (" + describe(providerClass) + ") does not implement createPlugin(InvocationInfo): " + reason @@ -198,23 +192,52 @@ private static void requireCreatePluginImplementation(String name, DurableExecut } } - /** Returns why the resolved method cannot serve the interface call, or null when it can. */ - private static String unimplementedReason(Method createPlugin) { - if (Modifier.isStatic(createPlugin.getModifiers())) { + /** + * Returns why no public method can serve the interface call, or null when one can. + * + *

    The reason names what was found instead, because an operator reading the failure has to be able to tell a + * provider that predates the current interface from a class path that resolves {@link InvocationInfo} to two + * different classes. + */ + private static String undispatchableCreatePluginReason(Class providerClass) { + var abstractOn = (Class) null; + var staticFound = false; + var otherReturnType = (Class) null; + for (var method : providerClass.getMethods()) { + if (!isCreatePluginCandidate(method)) { + continue; + } + var modifiers = method.getModifiers(); + if (Modifier.isStatic(modifiers)) { + staticFound = true; + } else if (Modifier.isAbstract(modifiers)) { + abstractOn = method.getDeclaringClass(); + } else if (method.getReturnType() == DurableExecutionPlugin.class) { + return null; + } else { + otherReturnType = method.getReturnType(); + } + } + if (otherReturnType != null) { + return "its createPlugin(InvocationInfo) returns " + otherReturnType.getName() + + " and the class carries no method returning " + DurableExecutionPlugin.class.getName() + + ", so it overrides nothing the interface call can dispatch to"; + } + if (staticFound) { return "the createPlugin(InvocationInfo) it declares is static, so it cannot implement the interface's " + "instance method"; } - if (Modifier.isAbstract(createPlugin.getModifiers())) { - return "the only declaration is the abstract one on " - + createPlugin.getDeclaringClass().getName(); - } - if (!DurableExecutionPlugin.class.isAssignableFrom(createPlugin.getReturnType())) { - return "its createPlugin(InvocationInfo) returns " - + createPlugin.getReturnType().getName() + " rather than a " - + DurableExecutionPlugin.class.getName() - + ", so it does not override the interface method"; + if (abstractOn != null) { + return "the only declaration is the abstract one on " + abstractOn.getName(); } - return null; + return "it declares no createPlugin method taking this SDK's " + InvocationInfo.class.getName(); + } + + /** Whether a method is named and parameterized like the factory method, whatever it returns. */ + private static boolean isCreatePluginCandidate(Method method) { + return "createPlugin".equals(method.getName()) + && method.getParameterCount() == 1 + && method.getParameterTypes()[0] == InvocationInfo.class; } /** Returns the provider class name, with the artifact it was loaded from when the JVM reports one. */ diff --git a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java index d145d7af0..87f70a798 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/DynamicPluginLoaderStaleProviderTest.java @@ -131,6 +131,51 @@ public static final class StaleAuditPlugin implements DurableExecutionPlugin {} } """; + /** + * A provider whose createPlugin(InvocationInfo) returns a plugin subtype and overrides nothing. + * + *

    Compiled against the older interface, so the method overrides no abstract declaration and javac emits no + * bridge returning {@code DurableExecutionPlugin}. The return type is still a plugin, so a check that asked only + * whether the return type were assignable to {@code DurableExecutionPlugin} would accept it, while + * {@code invokeinterface} looks for the interface's erased descriptor and finds none. + */ + private static final String SUBTYPE_RETURN_PROVIDER_SOURCE = """ + package com.example.audit; + + import software.amazon.lambda.durable.plugin.DurableExecutionPlugin; + import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider; + import software.amazon.lambda.durable.plugin.InvocationInfo; + + public final class StaleAuditProvider implements DurableExecutionPluginProvider { + + @Override + public String getName() { + return "com.example.audit"; + } + + @Override + public int getApiVersion() { + return API_VERSION; + } + + @Override + public Class getPluginType() { + return StaleAuditPlugin.class; + } + + @Override + public DurableExecutionPlugin createPlugin() { + return new StaleAuditPlugin(); + } + + public StaleAuditPlugin createPlugin(InvocationInfo info) { + return new StaleAuditPlugin(); + } + + public static final class StaleAuditPlugin implements DurableExecutionPlugin {} + } + """; + /** A provider whose createPlugin(InvocationInfo) returns something that is not a plugin. */ private static final String WRONG_RETURN_PROVIDER_SOURCE = """ package com.example.audit; @@ -285,6 +330,51 @@ void rejectsAProviderWhoseCreatePluginReturnsSomethingElse(@TempDir Path workDir assertTrue(message.contains("returns java.lang.String"), message); } + @Test + void rejectsAProviderWhoseCreatePluginHasNoBridge(@TempDir Path workDir) throws Exception { + // The shape an assignability test accepts and the JVM does not. The method is concrete, takes this SDK's + // InvocationInfo, and returns a DurableExecutionPlugin subtype, but it overrides nothing, so there is no bridge + // carrying the interface's erased descriptor and invokeinterface finds nothing to dispatch to. + var provider = staleProviderOfShape(workDir, SUBTYPE_RETURN_PROVIDER_SOURCE); + + var resolved = provider.getClass().getMethod("createPlugin", InvocationInfo.class); + assertTrue(!Modifier.isAbstract(resolved.getModifiers())); + assertTrue(!Modifier.isStatic(resolved.getModifiers())); + assertTrue( + DurableExecutionPlugin.class.isAssignableFrom(resolved.getReturnType()), + "the fixture is only interesting while an assignability test would accept it"); + assertTrue( + Stream.of(provider.getClass().getMethods()) + .noneMatch(method -> "createPlugin".equals(method.getName()) + && method.getParameterCount() == 1 + && method.getParameterTypes()[0] == InvocationInfo.class + && method.getReturnType() == DurableExecutionPlugin.class + && !Modifier.isAbstract(method.getModifiers())), + "the fixture must carry no bridge method, which is what makes the call fail"); + assertThrows(AbstractMethodError.class, () -> provider.createPlugin(invocationInfo())); + + var error = assertThrows( + IllegalStateException.class, + () -> DynamicPluginLoader.loadConfiguredPluginFactories(PROVIDER_NAME, List.of(provider), List.of())); + + var message = error.getMessage(); + assertTrue(message.contains("does not implement createPlugin(InvocationInfo)"), message); + assertTrue(message.contains("carries no method returning"), message); + } + + @Test + void acceptsAProviderWhoseCreatePluginReturnsASubtype() { + // The same covariant return, compiled against the current interface: javac emits the bridge, the interface call + // dispatches, and the check must accept it. This is what keeps the exact-descriptor rule from rejecting a + // provider that works. + var provider = new CovariantProvider(); + + var factories = DynamicPluginLoader.loadConfiguredPluginFactories("covariant", List.of(provider), List.of()); + + assertEquals(List.of(provider), factories); + assertTrue(provider.createPlugin(invocationInfo()) instanceof CurrentPlugin); + } + @Test void doesNotRejectAStaleProviderThatWasNotSelected(@TempDir Path workDir) throws Exception { var staleProvider = staleProvider(workDir); @@ -453,5 +543,19 @@ public DurableExecutionPlugin createPlugin(InvocationInfo invocationInfo) { } } + /** A provider written against the current interface with a covariant return, so javac emits a bridge. */ + private static final class CovariantProvider implements DurableExecutionPluginProvider { + + @Override + public String getName() { + return "covariant"; + } + + @Override + public CurrentPlugin createPlugin(InvocationInfo invocationInfo) { + return new CurrentPlugin(); + } + } + private static final class CurrentPlugin implements DurableExecutionPlugin {} } From 43982fe904e91e5fa455a4e260886b18769d819b Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 15:15:58 -0700 Subject: [PATCH 12/19] test(plugin): pin the factory and start-hook order The order is part of the contract, so it belongs in a test rather than only in a reply on a review thread. Every factory runs, then every start hook, and a plugin therefore cannot observe what another plugin's start hook installed. That is deliberate. A plugin depending on it would be depending on the order entries appear in a customer's withPlugins call, and instrumentation that changes what other instrumentation records is not something the SDK can promise across three languages. Both halves run on the same thread, so the test's ThreadLocal is visible where it is set; only the interleaving is asserted. --- .../durable/plugin/PluginRunnerTest.java | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java index cefbabe2c..db257aff5 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java @@ -68,6 +68,51 @@ public void onInvocationStart(InvocationInfo hookInfo) { assertSame(info, receivedByHook.get(0), "the first hook must receive the same info instance"); } + @Test + void everyFactoryRunsBeforeAnyStartHook_andNoPluginSeesAnothersHookState() { + // The order is part of the contract, so it is pinned rather than left to the reply on a review thread. Every + // factory runs, then every start hook, and a plugin therefore cannot observe what another plugin's start hook + // installed. That is deliberate: a plugin that depended on it would be depending on the order entries appear in + // a customer's withPlugins call, and instrumentation that changes what other instrumentation records is not + // something the SDK can promise across three languages. + // + // Both run on the same thread, so the ThreadLocal below is visible where it is set; only the interleaving is + // being asserted, not visibility. + var order = new ArrayList(); + var seenByLaterConstructor = new ArrayList(); + var installed = new ThreadLocal(); + + DurableExecutionPluginFactory first = info -> { + order.add("construct:first"); + return new DurableExecutionPlugin() { + @Override + public void onInvocationStart(InvocationInfo hookInfo) { + order.add("start:first"); + installed.set("from-first-start-hook"); + } + }; + }; + DurableExecutionPluginFactory second = info -> { + order.add("construct:second"); + seenByLaterConstructor.add(String.valueOf(installed.get())); + return new DurableExecutionPlugin() { + @Override + public void onInvocationStart(InvocationInfo hookInfo) { + order.add("start:second"); + } + }; + }; + + try { + new PluginRunner(List.of(first, second)).onInvocationStart(invocationInfo()); + } finally { + installed.remove(); + } + + assertEquals(List.of("construct:first", "construct:second", "start:first", "start:second"), order); + assertEquals(List.of("null"), seenByLaterConstructor, "a constructor must not observe another plugin's hook"); + } + @Test void factoriesAreCalledOncePerInvocation_notPerHook() { var creations = new AtomicInteger(); From 32c13815ff7719a5f20860c0077d118599ce90d0 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 15:49:01 -0700 Subject: [PATCH 13/19] fix(otel): reject a null config at registration The global-provider factory overload only stores the config, so a null one was first dereferenced when an invocation's plugin instance was built. PluginRunner contains a factory failure, so the function ran without the telemetry it had asked for and reported one warning per invocation. OtelPluginEnvironment's constructor now requires it. Every factory overload on both plugins reaches that constructor, so one check covers all four, and registration is where a caller can still act on the failure. --- .../durable/otel/OtelPluginEnvironment.java | 10 ++++- .../otel/OtelPluginFactoryConfigTest.java | 45 +++++++++++++++++++ 2 files changed, 54 insertions(+), 1 deletion(-) create mode 100644 otel-plugin/src/test/java/software/amazon/lambda/durable/otel/OtelPluginFactoryConfigTest.java diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java index 91caba1b2..029462e70 100644 --- a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java @@ -3,6 +3,7 @@ package software.amazon.lambda.durable.otel; import io.opentelemetry.sdk.trace.SdkTracerProviderBuilder; +import java.util.Objects; /** * Everything the OTel plugins need that belongs to the execution environment rather than to one invocation. @@ -32,9 +33,16 @@ final class OtelPluginEnvironment { */ private volatile OtelPluginSupport.ProviderSetup resolvedGlobalSetup; + /** + * @throws NullPointerException if the config is null. The check belongs here because every factory overload on both + * plugins reaches this constructor, and because the alternative is silence: the global-provider path only + * stores the config, so a null one would first be dereferenced when an invocation's plugin instance is built, + * where {@code PluginRunner} contains the failure. The function would then run without the telemetry it asked + * for, reporting one warning per invocation. Registration is where a caller can still act on it. + */ private OtelPluginEnvironment( OtelPluginConfig config, DeterministicIdGenerator idGenerator, OtelPluginSupport.ProviderSetup ownedSetup) { - this.config = config; + this.config = Objects.requireNonNull(config, "config must not be null"); this.idGenerator = idGenerator; this.ownedSetup = ownedSetup; } diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/OtelPluginFactoryConfigTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/OtelPluginFactoryConfigTest.java new file mode 100644 index 000000000..abe3624de --- /dev/null +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/OtelPluginFactoryConfigTest.java @@ -0,0 +1,45 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +package software.amazon.lambda.durable.otel; + +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.opentelemetry.sdk.trace.SdkTracerProvider; +import org.junit.jupiter.api.Test; + +/** + * Covers registration-time rejection of a null config on both plugins' factory overloads. + * + *

    The global-provider overload only stores the config, so a null one used to be dereferenced when an invocation's + * plugin instance was built. {@code PluginRunner} contains a factory failure, so the function ran without the telemetry + * it had asked for and reported one warning per invocation. Registration is where a caller can still act on it. + */ +class OtelPluginFactoryConfigTest { + + @Test + void invocationPluginRejectsANullConfigOnTheGlobalProviderOverload() { + var error = + assertThrows(NullPointerException.class, () -> InvocationOtelPlugin.factory((OtelPluginConfig) null)); + + assertTrue(error.getMessage().contains("config"), error.getMessage()); + } + + @Test + void executionPluginRejectsANullConfigOnTheGlobalProviderOverload() { + var error = + assertThrows(NullPointerException.class, () -> ExecutionOtelPlugin.factory((OtelPluginConfig) null)); + + assertTrue(error.getMessage().contains("config"), error.getMessage()); + } + + @Test + void invocationPluginRejectsANullConfigOnTheProviderBuilderOverload() { + assertThrows(NullPointerException.class, () -> InvocationOtelPlugin.factory(SdkTracerProvider.builder(), null)); + } + + @Test + void executionPluginRejectsANullConfigOnTheProviderBuilderOverload() { + assertThrows(NullPointerException.class, () -> ExecutionOtelPlugin.factory(SdkTracerProvider.builder(), null)); + } +} From 6aaa35f25269e0d4fc59a0effb9434be5aa54e23 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 19:55:04 -0700 Subject: [PATCH 14/19] fix(plugin): pair the end hook on a result-delivery failure The success branch serialized the handler's result, and checkpointed an oversized one, before firing onInvocationEnd. Either step can fail -- SerDes is a public extension point and JacksonSerDes throws on a value it cannot write, and the large-payload path adds a checkpoint that can fail on its own -- and the invocation then left without its end hook. That was survivable while a plugin instance outlived the invocation. It is not now: releasePlugins() calls nothing on the instances it drops and the contract has no close(), so the end hook is the only point at which a plugin can finish. Measured on this path before the fix: Insight emitted no record at all in on-complete mode, and in on-change mode the exporter received three RUNNING snapshots and was never flushed, so a buffering exporter lost those too. Both OTel plugins never ended the invocation span, never materialized the Workflow span, and never forceFlush. It is also the only exit that returned without draining, so a record could stay queued for a pump that exports it after the invocation returned -- the out-of-order delivery drainUntilSettled exists to prevent. The two steps are now contained, the end hook fires, and the failure is rethrown unchanged. The status is RETRYING rather than FAILED because the result never reached the backend: the execution is not finished, and the backend decides whether a new invocation follows. --- .../lambda/durable/PluginIntegrationTest.java | 43 +++++++++++++++++++ .../durable/execution/DurableExecutor.java | 35 +++++++++++++-- 2 files changed, 75 insertions(+), 3 deletions(-) diff --git a/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java b/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java index aab208fdf..304a8f4ef 100644 --- a/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java +++ b/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java @@ -355,6 +355,49 @@ public T deserialize(String data, TypeToken typeToken) { } } + @Test + void plugin_hooksStayPaired_whenTheResultCannotBeSerialized() { + var plugin = new RecordingPlugin(); + var config = DurableConfig.builder() + .withPlugins(info -> plugin) + .withSerDes(new ResultRejectingSerDes()) + .build(); + + var runner = LocalDurableTestRunner.create(String.class, (input, context) -> "unserializable", config); + + // The invocation fails on the way out, after the handler has already returned. + assertThrows(Exception.class, () -> runner.run("input")); + + // The end hook is the only point at which a plugin can flush: releasePlugins() calls nothing on the + // instances it drops and the contract has no close(). An exit that skips it therefore discards the whole + // invocation's telemetry -- Insight's record and every exporter's flush, and both OTel plugins' spans -- + // and can leave a record queued for a pump that exports it after this invocation has returned. + assertEquals(1, plugin.invocationStarts.size()); + assertEquals(1, plugin.invocationEnds.size(), "a start hook must not be left without its end hook"); + // RETRYING, not SUCCEEDED: the result never reached the backend, so the execution is not finished. + assertEquals(InvocationStatus.RETRYING, plugin.invocationEnds.get(0).invocationStatus()); + assertNotNull( + plugin.invocationEnds.get(0).executionError(), "the plugin must be told why the invocation ended"); + } + + /** SerDes that refuses to serialize the handler's result, as JacksonSerDes does for an unwritable value. */ + static class ResultRejectingSerDes implements SerDes { + private final JacksonSerDes delegate = new JacksonSerDes(); + + @Override + public String serialize(Object value) { + if ("unserializable".equals(value)) { + throw new IllegalStateException("cannot serialize the result"); + } + return delegate.serialize(value); + } + + @Override + public T deserialize(String data, TypeToken typeToken) { + return delegate.deserialize(data, typeToken); + } + } + // ─── Operation-level hooks ─────────────────────────────────────────── @Test diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java index 1ef55278c..8c118d03b 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java @@ -177,9 +177,38 @@ public static DurableExecutionOutput execute( } // user handler complete successfully logger.debug("Execution completed"); - var outputPayload = config.getSerDes().serialize(result); - var output = - DurableExecutionOutput.success(handleLargePayload(executionManager, outputPayload)); + // Serializing the result and checkpointing an oversized one can both fail, and this + // invocation ends either way. The end hook is the only point at which a plugin can finish: + // releasePlugins() calls nothing on the instances it drops and the contract has no close(), + // so an exit that skips the hook discards everything the plugin holds -- Insight's record + // for the execution and every exporter's flush, and both OTel plugins' invocation and + // Workflow spans. It also leaves a record queued for a pump that will export it after this + // invocation has returned, which is the out-of-order delivery drainUntilSettled exists to + // prevent. The status is RETRYING rather than FAILED because the throw below leaves the + // invocation the way a retryable failure does: the execution is not finished, and the + // backend decides whether a new invocation follows. + DurableExecutionOutput output = null; + Throwable resultDeliveryFailure = null; + try { + var outputPayload = config.getSerDes().serialize(result); + output = DurableExecutionOutput.success( + handleLargePayload(executionManager, outputPayload)); + } catch (Throwable failure) { + resultDeliveryFailure = failure; + } + if (resultDeliveryFailure != null) { + fireOnInvocationEnd( + pluginRunner, + executionManager, + requestId, + executionArn, + isFirstInvocation, + InvocationStatus.RETRYING, + resultDeliveryFailure, + pluginExecutionInput.get(), + null); + ExceptionHelper.sneakyThrow(resultDeliveryFailure); + } fireOnInvocationEnd( pluginRunner, executionManager, From f212142b1464e16c796812fa3fd99e7b1b736400 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 20:22:51 -0700 Subject: [PATCH 15/19] docs(insight): scope the flush exclusivity to one factory The flush() contract stated "never called concurrently with export()" as if it were an environment-wide property. It is a property of one scheduler, and workflowInsight() creates a scheduler per call, so an exporter instance handed to two factories is served by two pumps that can call its export and flush at once. The contract now says which scope it holds in and what to do about it. --- .../amazon/lambda/durable/insight/InsightExporter.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightExporter.java b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightExporter.java index a6acb8c88..26c2d2b7a 100644 --- a/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightExporter.java +++ b/insight-plugin/src/main/java/software/amazon/lambda/durable/insight/InsightExporter.java @@ -22,7 +22,12 @@ public interface InsightExporter { * all of them, because it starts only after each of their records has been handed to every exporter. An execution * that is sampled out neither exports nor flushes. * - *

    Never called concurrently with {@link #export(WorkflowInsightRecord)} on the same plugin instance. + *

    Never called concurrently with {@link #export(WorkflowInsightRecord)} by the plugins one + * {@link WorkflowInsight#workflowInsight} factory creates. That factory owns the scheduler serializing them, so the + * guarantee is per factory rather than per environment: an exporter instance handed to two factories is served by + * two schedulers, which can call its {@code export} and {@code flush} at the same time. Build the factory once per + * handler — which is what a {@code DurableConfig} created once per handler does — and give each factory its own + * exporter instances if a single exporter cannot tolerate concurrent calls. * *

    May cover records belonging to other executions running in the same environment, so it is not a per-execution * barrier. From 39a84aa4cdf4c23132b47a6388f71b34f6a39d43 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 20:44:59 -0700 Subject: [PATCH 16/19] fix(plugin): unwrap a wrapped result-delivery failure handleLargePayload waits with join(), so a failed checkpoint of an oversized result reaches the new containment wrapped in a CompletionException. Passing the wrapper to the end hook would have Insight's record and the OTel span status name the wrapper rather than the failure, while the failure branches above already unwrap before they report. --- .../lambda/durable/PluginIntegrationTest.java | 45 +++++++++++++++++++ .../durable/execution/DurableExecutor.java | 9 +++- 2 files changed, 53 insertions(+), 1 deletion(-) diff --git a/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java b/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java index 304a8f4ef..21ba26b45 100644 --- a/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java +++ b/sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java @@ -398,6 +398,51 @@ public T deserialize(String data, TypeToken typeToken) { } } + @Test + void plugin_seesTheUnderlyingFailure_whenResultDeliveryFailsWrapped() { + // handleLargePayload waits with join(), so a failed checkpoint of an oversized result reaches the same catch + // wrapped in a CompletionException. Plugins are told what failed, not how it was delivered. + var plugin = new RecordingPlugin(); + var cause = new IllegalStateException("underlying delivery failure"); + var config = DurableConfig.builder() + .withPlugins(info -> plugin) + .withSerDes(new WrappedFailureSerDes(cause)) + .build(); + + var runner = LocalDurableTestRunner.create(String.class, (input, context) -> "unserializable", config); + + assertThrows(Exception.class, () -> runner.run("input")); + + assertEquals(1, plugin.invocationEnds.size()); + assertSame( + cause, + plugin.invocationEnds.get(0).executionError(), + "the plugin must be told the underlying failure, not the CompletionException wrapper"); + } + + /** SerDes whose result failure arrives wrapped, as a failed oversized-result checkpoint does. */ + static class WrappedFailureSerDes implements SerDes { + private final JacksonSerDes delegate = new JacksonSerDes(); + private final Throwable cause; + + WrappedFailureSerDes(Throwable cause) { + this.cause = cause; + } + + @Override + public String serialize(Object value) { + if ("unserializable".equals(value)) { + throw new CompletionException(cause); + } + return delegate.serialize(value); + } + + @Override + public T deserialize(String data, TypeToken typeToken) { + return delegate.deserialize(data, typeToken); + } + } + // ─── Operation-level hooks ─────────────────────────────────────────── @Test diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java index 8c118d03b..567c95ace 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/DurableExecutor.java @@ -194,7 +194,14 @@ public static DurableExecutionOutput execute( output = DurableExecutionOutput.success( handleLargePayload(executionManager, outputPayload)); } catch (Throwable failure) { - resultDeliveryFailure = failure; + // handleLargePayload waits with join(), so a failed checkpoint arrives wrapped in a + // CompletionException. The plugins are told what failed, not how it was delivered, and + // the failure branches above already unwrap before they report -- so unwrap here too, + // or Insight's record and the OTel span status would name the wrapper. + resultDeliveryFailure = ExceptionHelper.unwrapCompletableFuture(failure); + if (resultDeliveryFailure == null) { + resultDeliveryFailure = failure; + } } if (resultDeliveryFailure != null) { fireOnInvocationEnd( From 860ee07e022e91bfaa9fe01c5e38afe3a1ca2187 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 21:12:45 -0700 Subject: [PATCH 17/19] fix(otel): reject the config before consuming the builder The null-config check lived in OtelPluginEnvironment's constructor, which runs after forProviderBuilder has installed the ID generator and the sampler on the caller's builder and built a provider. A provider built and then thrown away is unreachable, so its span processors and their worker threads are never shut down. The check is now the first statement of that method. The constructor keeps its own check, so a future entry point cannot skip it. --- .../durable/otel/OtelPluginEnvironment.java | 4 ++++ .../otel/OtelPluginFactoryConfigTest.java | 18 ++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java index 029462e70..69b87515b 100644 --- a/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java +++ b/otel-plugin/src/main/java/software/amazon/lambda/durable/otel/OtelPluginEnvironment.java @@ -53,6 +53,10 @@ private OtelPluginEnvironment( */ static OtelPluginEnvironment forProviderBuilder( SdkTracerProviderBuilder tracerProviderBuilder, OtelPluginConfig config) { + // Checked before anything is consumed. The constructor below checks it too, but by then this method has + // installed the ID generator and the sampler on the caller's builder and built a provider -- and a provider + // that fails validation is unreachable, so its span processors and their worker threads are never shut down. + Objects.requireNonNull(config, "config must not be null"); var idGenerator = DeterministicIdGenerator.installOn(tracerProviderBuilder); // Wrap the configured sampler so durable spans use the execution's single precomputed decision. DurableSampler.installOn(tracerProviderBuilder); diff --git a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/OtelPluginFactoryConfigTest.java b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/OtelPluginFactoryConfigTest.java index abe3624de..23675c8d0 100644 --- a/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/OtelPluginFactoryConfigTest.java +++ b/otel-plugin/src/test/java/software/amazon/lambda/durable/otel/OtelPluginFactoryConfigTest.java @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 package software.amazon.lambda.durable.otel; +import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -42,4 +43,21 @@ void invocationPluginRejectsANullConfigOnTheProviderBuilderOverload() { void executionPluginRejectsANullConfigOnTheProviderBuilderOverload() { assertThrows(NullPointerException.class, () -> ExecutionOtelPlugin.factory(SdkTracerProvider.builder(), null)); } + + @Test + void aRejectedConfigLeavesTheBuilderUsable() { + // The check is the first statement of forProviderBuilder, so it precedes the ID-generator and sampler + // installation and the provider build. That ordering matters because a provider built and then thrown away is + // unreachable: its span processors and their worker threads are never shut down. The ordering itself is a + // property of the source rather than something this test can observe -- SdkTracerProviderBuilder exposes no + // getters -- so what is asserted here is the consequence a caller can see: the builder they passed still + // works. + var builder = SdkTracerProvider.builder(); + + assertThrows(NullPointerException.class, () -> InvocationOtelPlugin.factory(builder, null)); + + try (var provider = builder.build()) { + assertNotNull(provider.get("probe")); + } + } } From 51a235fe07f6cbc3421b3a7a361046f0c232415b Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 21:41:36 -0700 Subject: [PATCH 18/19] fix(plugin): publish instances built before a fatal failure createPlugins assigned the plugin list after the loop, so a factory failure that is fatal -- VirtualMachineError or ThreadDeath, the two contain() rethrows -- left the runner looking empty. A plugin constructor is where both OTel plugins bind their tracer and start the Invocation span, so an instance built before the fatal one already owns spans that only onInvocationEnd ends and flushes, and the end hook the failure path fires reached nothing. The list is published in a finally, so what was built is reachable whatever left the loop. --- .../lambda/durable/plugin/PluginRunner.java | 28 ++++++++++++------- .../durable/plugin/PluginRunnerTest.java | 20 +++++++++++++ 2 files changed, 38 insertions(+), 10 deletions(-) diff --git a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java index 7ec785a79..4aed4b9f5 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/plugin/PluginRunner.java @@ -75,19 +75,27 @@ public boolean isEmpty() { */ private void createPlugins(InvocationInfo info) { var created = new ArrayList(pluginFactories.size()); - for (var factory : pluginFactories) { - try { - var plugin = factory.createPlugin(info); - if (plugin == null) { - logger.warn("Plugin factory {} returned null; skipping it for this invocation", factory); - continue; + try { + for (var factory : pluginFactories) { + try { + var plugin = factory.createPlugin(info); + if (plugin == null) { + logger.warn("Plugin factory {} returned null; skipping it for this invocation", factory); + continue; + } + created.add(plugin); + } catch (Throwable t) { + contain(t, "Plugin factory failed; skipping it for this invocation"); } - created.add(plugin); - } catch (Throwable t) { - contain(t, "Plugin factory failed; skipping it for this invocation"); } + } finally { + // Published even when a factory failure is fatal and propagates. A plugin constructor is where both OTel + // plugins bind their tracer and start the Invocation span, so an instance built before the fatal one + // already owns spans that only onInvocationEnd ends and flushes. Assigning after the loop meant a + // VirtualMachineError or ThreadDeath from a later factory left the runner looking empty, so the end hook + // the failure path fires reached nothing and those spans were dropped un-ended. + this.plugins = List.copyOf(created); } - this.plugins = List.copyOf(created); } /** diff --git a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java index db257aff5..7780793cf 100644 --- a/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java +++ b/sdk/src/test/java/software/amazon/lambda/durable/plugin/PluginRunnerTest.java @@ -424,6 +424,26 @@ void containmentPreservesAnInterruptTheThreadAlreadyCarried() { } } + @Test + void aFatalFactoryFailure_stillPublishesTheInstancesAlreadyBuilt() { + // A plugin constructor is where both OTel plugins bind their tracer and start the Invocation span, so an + // instance built before a fatal failure already owns spans that only onInvocationEnd ends and flushes. + // Publishing after the loop meant a VirtualMachineError from a later factory left the runner looking empty, + // and the end hook the failure path fires reached nothing. + var calls = new ArrayList(); + var runner = new PluginRunner(List.of(info -> new TestPlugin("p1", calls), info -> { + throw new OutOfMemoryError("fatal factory"); + })); + + assertThrows(OutOfMemoryError.class, () -> runner.onInvocationStart(invocationInfo())); + + // The start hook never ran -- the fatal throw left createPlugins -- but the instance exists and its end hook + // must still reach it. + assertEquals(List.of(), calls); + runner.onInvocationEnd(invocationEndInfo()); + assertEquals(List.of("p1:onInvocationEnd"), calls, "an instance already built must still be finalized"); + } + @Test void factoryThrowingAJvmError_stillPropagates() { // The containment is deliberately narrow: an Error that says the JVM itself is failing must not be swallowed as From 3a99cdd57d370100a83a0f6586f5e10ef9997f78 Mon Sep 17 00:00:00 2001 From: Pooya Paridel Date: Fri, 18 Sep 2026 22:05:33 -0700 Subject: [PATCH 19/19] fix(plugin): release instances even if shutdown throws close() called validateRunningThreads() and the checkpoint shutdown before releasePlugins(), and validateRunningThreads throws on a stuck user handler. The instances then stayed reachable from the runner and were carried into the next invocation the environment hosts, which is the cross-execution sharing the per-invocation lifetime exists to prevent. The release moves into a finally. No dedicated test: reaching the throwing path needs a user handler that never completes, and the release is unconditional rather than conditional on what threw. --- .../durable/execution/ExecutionManager.java | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java b/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java index f967a30a5..3dc6feb82 100644 --- a/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java +++ b/sdk/src/main/java/software/amazon/lambda/durable/execution/ExecutionManager.java @@ -419,12 +419,16 @@ public CompletableFuture pollForOperationUpdates(String operationId, /** Shutdown the checkpoint batcher. */ @Override public void close() { - validateRunningThreads(); - - checkpointManager.shutdown(); - - // The invocation is over: drop this invocation's plugin instances so they cannot be reached again. - pluginRunner.releasePlugins(); + try { + validateRunningThreads(); + checkpointManager.shutdown(); + } finally { + // The invocation is over: drop this invocation's plugin instances so they cannot be reached again. + // In a finally, because validateRunningThreads throws on a stuck user handler: leaving the instances + // in place then carries them into the next invocation the environment hosts, which is the + // cross-execution sharing the per-invocation lifetime exists to prevent. + pluginRunner.releasePlugins(); + } } private void validateRunningThreads() {