Repository navigation
MLE-32970 Prevent retries of multipart streaming requests - #1979
rjdew-progress wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Multipart bodies wrapping InputStreamHandle may still be replayed because one-shot status is not propagated through ObjectRequestBody.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Prevents retries of multipart requests containing one-shot streaming bodies.
Changes:
- Marks
StreamingOutputImplas one-shot. - Honors OkHttp’s
isOneShot()contract in retry handling. - Adds multipart streaming regression coverage.
| File | Reviewed change |
|---|---|
marklogic-client-api/src/test/java/com/marklogic/client/impl/StreamingOutputImplTest.java |
Tests one-shot multipart behavior. |
marklogic-client-api/src/main/java/com/marklogic/client/impl/StreamingOutputImpl.java |
Declares streaming bodies as one-shot. |
marklogic-client-api/src/main/java/com/marklogic/client/impl/okhttp/RetryIOExceptionInterceptor.java |
Avoids retries for one-shot request bodies. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
af0a197 to
c634447
Compare
c634447 to
3c26e65
Compare
…ndability StreamingOutputImpl previously hardcoded isRetryable()=false / isOneShot()=true for every wrapped OutputStreamSender, even when the originating write handle was fully buffered and resendable (e.g. JacksonHandle, StringHandle). This meant any request body built from such a handle -- including Row Manager plan execution, document writes, and Data Services parameters -- was needlessly blocked from being retried by RetryIOExceptionInterceptor after a transient connection error, even though the content could safely be written again. Thread each call site's already-known isResendable()/handleBase.isResendable() value into StreamingOutputImpl instead of assuming non-resendable: - writeDocumentImpl-style document writes - putPostValueImpl - postResource / putResource (used by RowManagerImpl for Optic plan execution) - postIteratedResourceImpl (multipart-mixed eval/invoke style resources) - addParts multipart builder - makeRequestBodyForContent and BaseProxy's makeRequestBody (Data Services single-node parameters) Call sites without an existing resendable concept (structured query search, alert match) keep the prior conservative false/one-shot default via new 3-arg doPost/doPut overloads, so behavior is unchanged where resendability was never determined. Added StreamingOutputImplTest cases covering: a resendable body being retried after a simulated connection failure, and a non-resendable part inside a MultipartBody still propagating isOneShot()=true so it is never retried. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Java client always sends Accept-Encoding: gzip, and MarkLogic can close a connection after replying to such a request even though OkHttp keeps it pooled. A Data Services call that streams a one-shot body (e.g. a Reader), made right after another call such as RowManager.resultRows(), could be written to that closed connection. The body was consumed before the failure was noticed, so the retry failed with 'Stream closed'. The ping that guards against this only ran for the first request on a thread, and only for digest authentication, so it had already been used up by the earlier call. It now runs before every request that has a streaming part, so a closed connection is found by a ping that has no body to lose. This relies on StreamingOutputImpl reporting one-shot bodies as such: with the ping but without that, a failure after the ping is resent by OkHttp and reports 'Stream closed' again. StreamingPartRequestTest (mock server) checks that the ping happens before each streaming request and that a failed streaming request is not resent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| Function<Request.Builder, Response> sendRequestFunction = requestBldr -> { | ||
| if (isFirstRequest() && hasStreamingPart) { | ||
| // Ping the server before streaming; if unavailable, wait and retry the ping | ||
| if (hasStreamingPart) { |
There was a problem hiding this comment.
Revisit pinging on every streaming Data Services request (OkHttpServices.java:5412) vs. fixing the Gzip connection handling
Changing if (isFirstRequest() && hasStreamingPart) to if (hasStreamingPart) in CallRequestImpl.executeRequest means every streaming Data Services call now incurs an extra synchronous HEAD round-trip over the network, even when the connection pool is healthy. Furthermore, doPut (lines 3757, 3778) and doPost (lines 3802, 3830) still check isFirstRequest(), so a streaming ReaderHandle document write right after a gzipped RowManager.resultRows() call would still hit the closed connection.
- Cleaner alternatives to consider:
- Check why
RowManager.resultRows()(makeResults, line 4148) leaves the gzipped connection in a bad state:readMultipartBodyParts(response.body())reads the entire multipart stream into memory vianew ByteArrayDataSource(body.byteStream(), contentType), which stops readingGzipSourceat the 8-byte gzip footer without closingbody.byteStream()orresponse. Because allBodyParts are already buffered in memory inByteArrayDataSource, closingresponseinsidemakeResults(after reading trailers) ensuresGzipSource/ChunkedSourceis properly closed or evicted right whenresultRows()finishes, even if the caller doesn't closeRowSet(or whenRowSethas 0 rows inRowSetBase.init()).
- Check why
- Use Expect: 100-continue for one-shot bodies (or only set the ping flag when a gzipped response is received): Adding Expect: 100-continue on one-shot requests causes OkHttp's CallServerInterceptor to flush the request headers and verify the connection is alive before invoking RequestBody.writeTo(sink), avoiding an extra HEAD request on every call.
There was a problem hiding this comment.
Following up on my earlier comment, with two corrections to it.
- I no longer think
makeResultsleaves the connection in a bad state.ByteArrayDataSourcereads the body to the end,GzipSourcethen exhausts the chunked source, and the connection is released cleanly. An unread body would leak the connection, not pool it. Expect: 100-continueis not a clean alternative. OkHttp does not recover a one-shot request even if the failure happens before the body is written, and withreadTimeout(0)an intermediary that ignoresExpectwould block forever.
On the change itself:
- The cause stated in the comment ("MarkLogic can close a connection after replying to a request that accepts gzip") is not shown anywhere in the PR. The new test has no gzip. Is there a server ticket or a capture? Does it reproduce with
withGzippedResponsesDisabled()? What is the gap between the two calls, and what is the app server's keep-alive timeout? - A simpler explanation fits the symptoms. OkHttp only probes a pooled connection before a non-GET request once it has been idle for 10 seconds, and
new ConnectionPool()keeps idle connections for 5 minutes. If the server closes an idle connection inside that 10-second window, the client cannot see it. If that is the cause, the fix is one line inOkHttpUtil: give the pool a keep-alive shorter than the server's. That covers every request path at no per-request cost. - The ping is not atomic with the request. The pool is static and shared by all clients, so with more than one thread the ping can validate one connection and the POST can be sent on another.
- It only covers Data Services.
doPut,doPost,putPostValueImpl(1221),putPostDocumentImpl(1373) and 2473 still ping only on the first request, so a rows call followed bywrite(uri, new InputStreamHandle(...))has the same problem. - It reuses the digest first-request ping for a different purpose.
useDigestAuthPingand theisFirstRequestbookkeeping in this method no longer describe what the code does. - Every streaming Data Services call now costs two round trips, and the new test asserts that as the contract.
My suggestion: keep the isOneShot() work in this PR, since it turns "Stream closed" into the real error. Move the ping to its own PR once the cause is confirmed. If a ping is still needed then, put it in one helper used by every one-shot path, keyed on body.isOneShot().
| Function<Request.Builder, Response> doPutFunction = new Function<Request.Builder, Response>() { | ||
| public Response apply(Request.Builder funcBuilder) { | ||
| return doPut(reqlog, funcBuilder, inputBase.sendContent()); | ||
| return doPut(reqlog, funcBuilder, inputBase.sendContent(), isResendable); |
There was a problem hiding this comment.
Using OkHttp's native RequestBody.isOneShot() so MultipartBody.isOneShot() automatically aggregates one-shot status across its parts is the right direction. However, the follow-up changes in OkHttpServices (cdb2f2c and 4779f8f) touch a lot of base methods and introduce a few gaps that we can solve much more cleanly:
- Avoid threading boolean isResendable through doPut / doPost (and 4 missed call sites)
Adding 4-arg doPut(..., boolean isResendable) and doPost(..., boolean isResendable) overloads where the 3-arg versions default isResendable to false requires every caller that calls baseHandle.sendContent() to also pass baseHandle.isResendable().
Four handle-based call sites in OkHttpServices.java were missed and still call the 3-arg doPost(reqlog, requestBldr, baseHandle.sendContent()):
Line 1934 (RawQueryDefinition in search)
Line 1936 (RawCtsQueryDefinition in search)
Line 4667 (match with StructureWriteHandle)
Line 4765 (match with RawQueryDefinition) For those calls, even a resendable StringHandle or JacksonHandle will now be wrapped in StreamingOutputImpl(..., false) (isOneShot() = true), disabling retries on stale connections.
Cleaner approach: The only reason StreamingOutputImpl didn't know if the handle was resendable is because baseHandle.sendContent() returns an OutputStreamSender without the handle's isResendable() metadata. If OutputStreamSender (or an internal package-private sub-interface / HandleImplementation check) exposes isResendable(), StreamingOutputImpl(OutputStreamSender, RequestLogger, MediaType) can determine resendable directly from the sender without changing doPut, doPost, or any of their callers in OkHttpServices.
prgs-rsingh
left a comment
There was a problem hiding this comment.
Need to revisit the implementation
Minor Cleanup Nits
Duplicate method in OkHttpServices.java: CallRequestImpl.makeRequestBody(AbstractWriteHandle) (lines 5521–5534) is identical to OkHttpServices.makeRequestBodyForContent(AbstractWriteHandle) (lines 5294–5307) and can simply delegate to it.
RetryableRequestBody.isRetryable() vs RequestBody.isOneShot(): In both StreamingOutputImpl and ObjectRequestBody, isRetryable() is now just !isOneShot(). Since RetryableRequestBody is an internal interface in com.marklogic.client.impl.okhttp, we can either retire RetryableRequestBody in favor of OkHttp's RequestBody.isOneShot() or give it a default implementation.
| } | ||
|
|
||
| private Response doPost(RequestLogger reqlog, Request.Builder requestBldr, Object value) { | ||
| // Default to non-resendable for callers that have not determined whether the content can be sent again. |
There was a problem hiding this comment.
The 3-arg overloads that default to isResendable = false are what let call sites get missed. There are five, not four: 1934, 1936, 2193 (values() with a raw query), 4667 and 4765.
This matters more than it first appears. RetryIOExceptionInterceptor is only installed by the test clients (Common.java, ConnectedRESTQA). In production the effect of isOneShot() == true is on OkHttp itself: it stops recovering from a failed pooled connection and stops following up a 401. On develop these bodies were replayable, so a raw-query search with a StringHandle on a stale connection was retried silently. With this PR it fails, and a digest re-challenge surfaces as "Unauthorized". That is a regression for resendable handles at those five sites.
Rather than thread a boolean through every caller, I would read resendability in the one place the content is read:
private static RequestBody toRequestBody(HandleImplementation<?, ?> handle, MediaType type, RequestLogger reqlog) {
Object content = handle.sendContent();
if (content instanceof OutputStreamSender sender) {
return new StreamingOutputImpl(sender, reqlog, type, handle.isResendable());
}
return new ObjectRequestBody(reqlog != null ? reqlog.copyContent(content) : content, type);
}
Then doPut/doPost take a RequestBody instead of an Object plus a flag. The overloads and the defaults go away, and a new call site cannot forget the flag. The same wrap-the-content block is currently written inline five times (putPostValueImpl, putPostValueImpl for values, doPut, doPost, addParts) and twice more as methods (makeRequestBodyForContent and CallRequestImpl.makeRequestBody, which are identical). All seven can use the one helper.
This needs no change to the public OutputStreamSender interface, so I prefer it to my earlier suggestion of exposing isResendable() there.
| if (nextValue instanceof OutputStreamSender) { | ||
| sentValue = new StreamingOutputImpl( | ||
| (OutputStreamSender) nextValue, reqlog, mediaType); | ||
| (OutputStreamSender) nextValue, reqlog, mediaType, handle != null && handle.isResendable()); |
There was a problem hiding this comment.
This method now has two definitions of "resendable". Here it is handle != null && handle.isResendable(). Ten lines down, isResendable is computed as (handle == null) ? !isStreaming : handle.isResendable(). Please compute it once and use it for both the body and the retry decision.
Related, and already on develop: isStreaming(sentValue) on line 2469 is passed the RequestBody, not the content, so it is always true. It should be isStreaming(nextValue).
| // InputStreams cannot be retried as they are consumed on first read. | ||
| return !(obj instanceof InputStream); | ||
| } | ||
|
|
There was a problem hiding this comment.
This is a good change, and it fixes more than the multipart case. On develop, when OkHttp recovered a failed connection or followed up a 401, it called writeTo again on the same InputStream and sent only whatever was left unread: an empty or truncated body. Marking it one-shot stops that.
Two small things. isRetryable() is now exactly !isOneShot(), so one of them should go (see my note on RetryableRequestBody). And the class was widened from private only for the test. If the test builds the body through makeRequestBodyForContent(new InputStreamHandle(...)), it can stay private.
| * Assumes the wrapped handle cannot be resent, which is the safe default for callers that have not determined | ||
| * whether the underlying content can be sent again (e.g. because it is backed by a one-shot stream). | ||
| */ | ||
| StreamingOutputImpl(OutputStreamSender handle, RequestLogger logger, MediaType contentType) { |
There was a problem hiding this comment.
The 3-arg constructor is no longer used by any main code, only by one test. Please remove it: a silent false default is the same trap as the 3-arg doPost.
isRetryable() and isOneShot() must now always be exact opposites, here and in ObjectRequestBody. Keeping two methods for one fact invites them to drift. I would drop RetryableRequestBody from both classes and let the interceptor ask isOneShot() only.
Nits: resendable (and the other fields) can be final, and the comment in isRetryable() describes what the code used to do. That history belongs in the commit message.
| RequestBody requestBody = request.body(); | ||
|
|
||
| if (request.body() instanceof RetryableRequestBody body && !body.isRetryable()) { | ||
| if (requestBody != null && (requestBody.isOneShot() || |
There was a problem hiding this comment.
Honoring isOneShot() is the right contract, and the class Javadoc already describes it that way. Once the two body classes stop implementing RetryableRequestBody, the second half of this condition can go and the interface can be deprecated or removed (it is public, in an impl package, added in 8.0.0).
If it stays for now, please parenthesize the instanceof ... && ... part. It relies on && binding tighter than ||.
Out of scope, but the same class of bug: ProgressDataCloudAuthenticationConfigurer.TokenAuthenticationInterceptor calls chain.proceed a second time on a 401 without checking isOneShot(), so a streaming body is written twice after a token expiry. Worth a follow-up ticket.
| import static org.junit.jupiter.api.Assertions.assertThrows; | ||
| import static org.junit.jupiter.api.Assertions.assertTrue; | ||
|
|
||
| class StreamingOutputImplTest { |
There was a problem hiding this comment.
- These tests exercise
RetryIOExceptionInterceptor, which production clients do not install. The behaviour that matters in production is whether real handles produce the right flag. A parameterized test over handles would cover that:makeRequestBodyForContent(handle).isOneShot()is false forStringHandle,JacksonHandle,BytesHandleandFileHandle, and true forReaderHandleandInputStreamHandle. nonResendableHandleInMultipartIsNotRetriedrepeatsmultipartBodyWithStreamingPartIsNotRetriedand, despite its name, only assertsisOneShot().- The opposite case is missing: a multipart body whose parts are all resendable should not be one-shot and should be retried.
- The client and failing-interceptor setup is copied in three tests. A small helper would do.
- Nits: import
Response,ProtocolandResponseBodyrather than using full names, and declarethrows IOExceptionrather than wrapping it inRuntimeException.
| } | ||
|
|
||
| @Test | ||
| void serverIsPingedBeforeEveryStreamingRequest() throws Exception { |
There was a problem hiding this comment.
This test fixes the mechanism (HEAD, POST, HEAD, POST) rather than the behaviour, so any better fix would fail it. It also never reproduces the reported problem.
A more useful test: the server answers a first call and then closes the socket without sending Connection: close; a streaming call made straight after should succeed, and its body should arrive once and intact. That test passes for any correct fix.
| } | ||
|
|
||
| @Test | ||
| void failedStreamingRequestIsNotResent() { |
There was a problem hiding this comment.
The request-count assertion is the valuable one. The two message assertions depend on OkHttp's wording and on the platform (this can be "Connection reset" rather than "unexpected end of stream"), so I would drop them or loosen them.
Both tests use basic auth. Digest is the default and is the reason the ping exists, so one digest case would be worth having.
prgs-rsingh
left a comment
There was a problem hiding this comment.
The description covers only the first commit and is now inaccurate: StreamingOutputImpl is no longer always one-shot, and neither the resendability plumbing nor the ping on every streaming Data Services request is mentioned. The title, branch and commits also mix MLE-32970 and MLE-32953.
I would split this in two. The one-shot contract plus a single body factory is one reviewable change. The ping is a behaviour and performance change that needs its root cause confirmed first.

Why
A Data Services request with streaming multipart input can fail with
Stream closedafter a preceding gzipped Row Manager response. The IOException retry interceptor sees only the outerMultipartBody, so it can retry the request even though a nestedStreamingOutputImplhas already consumed and closed its source.Approach
StreamingOutputImplas one-shot using OkHttp's nativeRequestBody.isOneShot()contract.isOneShot()in addition to the Java Client'sRetryableRequestBodycontract.MultipartBodyand prevents a second request attempt.Validation
./gradlew marklogic-client-api:test --tests com.marklogic.client.impl.StreamingOutputImplTestlocalhost:8012.