Skip to content

MLE-32970 Prevent retries of multipart streaming requests - #1979

Open
rjdew-progress wants to merge 4 commits into
developfrom
MLE-32953
Open

rjdew-progress wants to merge 4 commits into
developfrom
MLE-32953

Conversation

@rjdew-progress

Copy link
Copy Markdown
Contributor

Why

A Data Services request with streaming multipart input can fail with Stream closed after a preceding gzipped Row Manager response. The IOException retry interceptor sees only the outer MultipartBody, so it can retry the request even though a nested StreamingOutputImpl has already consumed and closed its source.

Approach

  • Mark StreamingOutputImpl as one-shot using OkHttp's native RequestBody.isOneShot() contract.
  • Make the IOException retry interceptor honor isOneShot() in addition to the Java Client's RetryableRequestBody contract.
  • Add a regression test proving that one-shot status propagates through MultipartBody and prevents a second request attempt.

Validation

  • ./gradlew marklogic-client-api:test --tests com.marklogic.client.impl.StreamingOutputImplTest
  • The complete core test task was also attempted; its server-dependent tests could not run because no deployed MarkLogic test instance was available on localhost:8012.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Multipart bodies wrapping InputStreamHandle may still be replayed because one-shot status is not propagated through ObjectRequestBody.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity

Open (1)
What changed in this PR

Prevents retries of multipart requests containing one-shot streaming bodies.

Changes:

  • Marks StreamingOutputImpl as one-shot.
  • Honors OkHttp’s isOneShot() contract in retry handling.
  • Adds multipart streaming regression coverage.
File Reviewed change
marklogic-client-api/​src/​test/​java/​com/​marklogic/​client/​impl/​StreamingOutputImplTest.java Tests one-shot multipart behavior.
marklogic-client-api/​src/​main/​java/​com/​marklogic/​client/​impl/​StreamingOutputImpl.java Declares streaming bodies as one-shot.
marklogic-client-api/​src/​main/​java/​com/​marklogic/​client/​impl/​okhttp/​RetryIOExceptionInterceptor.java Avoids retries for one-shot request bodies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

All reviewed changes are covered and no unresolved blocking issues were identified.

Review effort: Lite
Findings: None

Resolved since last review (1)

@rjdew-progress rjdew-progress changed the title MLE-32953 Prevent retries of multipart streaming requests MLE-32970 Prevent retries of multipart streaming requests Sep 24, 2026
…ndability

StreamingOutputImpl previously hardcoded isRetryable()=false / isOneShot()=true
for every wrapped OutputStreamSender, even when the originating write handle
was fully buffered and resendable (e.g. JacksonHandle, StringHandle). This
meant any request body built from such a handle -- including Row Manager plan
execution, document writes, and Data Services parameters -- was needlessly
blocked from being retried by RetryIOExceptionInterceptor after a transient
connection error, even though the content could safely be written again.

Thread each call site's already-known isResendable()/handleBase.isResendable()
value into StreamingOutputImpl instead of assuming non-resendable:
- writeDocumentImpl-style document writes
- putPostValueImpl
- postResource / putResource (used by RowManagerImpl for Optic plan execution)
- postIteratedResourceImpl (multipart-mixed eval/invoke style resources)
- addParts multipart builder
- makeRequestBodyForContent and BaseProxy's makeRequestBody (Data Services
  single-node parameters)

Call sites without an existing resendable concept (structured query search,
alert match) keep the prior conservative false/one-shot default via new 3-arg
doPost/doPut overloads, so behavior is unchanged where resendability was
never determined.

Added StreamingOutputImplTest cases covering: a resendable body being retried
after a simulated connection failure, and a non-resendable part inside a
MultipartBody still propagating isOneShot()=true so it is never retried.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Java client always sends Accept-Encoding: gzip, and MarkLogic can close a connection after replying to such a request even though OkHttp keeps it pooled. A Data Services call that streams a one-shot body (e.g. a Reader), made right after another call such as RowManager.resultRows(), could be written to that closed connection. The body was consumed before the failure was noticed, so the retry failed with 'Stream closed'.

The ping that guards against this only ran for the first request on a thread, and only for digest authentication, so it had already been used up by the earlier call. It now runs before every request that has a streaming part, so a closed connection is found by a ping that has no body to lose.

This relies on StreamingOutputImpl reporting one-shot bodies as such: with the ping but without that, a failure after the ping is resent by OkHttp and reports 'Stream closed' again.

StreamingPartRequestTest (mock server) checks that the ping happens before each streaming request and that a failed streaming request is not resent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Function<Request.Builder, Response> sendRequestFunction = requestBldr -> {
if (isFirstRequest() && hasStreamingPart) {
// Ping the server before streaming; if unavailable, wait and retry the ping
if (hasStreamingPart) {

@prgs-rsingh prgs-rsingh Nov 10, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revisit pinging on every streaming Data Services request (OkHttpServices.java:5412) vs. fixing the Gzip connection handling
Changing if (isFirstRequest() && hasStreamingPart) to if (hasStreamingPart) in CallRequestImpl.executeRequest means every streaming Data Services call now incurs an extra synchronous HEAD round-trip over the network, even when the connection pool is healthy. Furthermore, doPut (lines 3757, 3778) and doPost (lines 3802, 3830) still check isFirstRequest(), so a streaming ReaderHandle document write right after a gzipped RowManager.resultRows() call would still hit the closed connection.

  • Cleaner alternatives to consider:
    1. Check why RowManager.resultRows() (makeResults, line 4148) leaves the gzipped connection in a bad state: readMultipartBodyParts(response.body()) reads the entire multipart stream into memory via new ByteArrayDataSource(body.byteStream(), contentType), which stops reading GzipSource at the 8-byte gzip footer without closing body.byteStream() or response. Because all BodyParts are already buffered in memory in ByteArrayDataSource, closing response inside makeResults (after reading trailers) ensures GzipSource / ChunkedSource is properly closed or evicted right when resultRows() finishes, even if the caller doesn't close RowSet (or when RowSet has 0 rows in RowSetBase.init()).
  1. Use Expect: 100-continue for one-shot bodies (or only set the ping flag when a gzipped response is received): Adding Expect: 100-continue on one-shot requests causes OkHttp's CallServerInterceptor to flush the request headers and verify the connection is alive before invoking RequestBody.writeTo(sink), avoiding an extra HEAD request on every call.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on my earlier comment, with two corrections to it.

  1. I no longer think makeResults leaves the connection in a bad state. ByteArrayDataSource reads the body to the end, GzipSource then exhausts the chunked source, and the connection is released cleanly. An unread body would leak the connection, not pool it.
  2. Expect: 100-continue is not a clean alternative. OkHttp does not recover a one-shot request even if the failure happens before the body is written, and with readTimeout(0) an intermediary that ignores Expect would block forever.

On the change itself:

  • The cause stated in the comment ("MarkLogic can close a connection after replying to a request that accepts gzip") is not shown anywhere in the PR. The new test has no gzip. Is there a server ticket or a capture? Does it reproduce with withGzippedResponsesDisabled()? What is the gap between the two calls, and what is the app server's keep-alive timeout?
  • A simpler explanation fits the symptoms. OkHttp only probes a pooled connection before a non-GET request once it has been idle for 10 seconds, and new ConnectionPool() keeps idle connections for 5 minutes. If the server closes an idle connection inside that 10-second window, the client cannot see it. If that is the cause, the fix is one line in OkHttpUtil: give the pool a keep-alive shorter than the server's. That covers every request path at no per-request cost.
  • The ping is not atomic with the request. The pool is static and shared by all clients, so with more than one thread the ping can validate one connection and the POST can be sent on another.
  • It only covers Data Services. doPut, doPost, putPostValueImpl (1221), putPostDocumentImpl (1373) and 2473 still ping only on the first request, so a rows call followed by write(uri, new InputStreamHandle(...)) has the same problem.
  • It reuses the digest first-request ping for a different purpose. useDigestAuthPing and the isFirstRequest bookkeeping in this method no longer describe what the code does.
  • Every streaming Data Services call now costs two round trips, and the new test asserts that as the contract.

My suggestion: keep the isOneShot() work in this PR, since it turns "Stream closed" into the real error. Move the ping to its own PR once the cause is confirmed. If a ping is still needed then, put it in one helper used by every one-shot path, keyed on body.isOneShot().

Function<Request.Builder, Response> doPutFunction = new Function<Request.Builder, Response>() {
public Response apply(Request.Builder funcBuilder) {
return doPut(reqlog, funcBuilder, inputBase.sendContent());
return doPut(reqlog, funcBuilder, inputBase.sendContent(), isResendable);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using OkHttp's native RequestBody.isOneShot() so MultipartBody.isOneShot() automatically aggregates one-shot status across its parts is the right direction. However, the follow-up changes in OkHttpServices (cdb2f2c and 4779f8f) touch a lot of base methods and introduce a few gaps that we can solve much more cleanly:

  1. Avoid threading boolean isResendable through doPut / doPost (and 4 missed call sites)

Adding 4-arg doPut(..., boolean isResendable) and doPost(..., boolean isResendable) overloads where the 3-arg versions default isResendable to false requires every caller that calls baseHandle.sendContent() to also pass baseHandle.isResendable().

Four handle-based call sites in OkHttpServices.java were missed and still call the 3-arg doPost(reqlog, requestBldr, baseHandle.sendContent()):
Line 1934 (RawQueryDefinition in search)
Line 1936 (RawCtsQueryDefinition in search)
Line 4667 (match with StructureWriteHandle)
Line 4765 (match with RawQueryDefinition) For those calls, even a resendable StringHandle or JacksonHandle will now be wrapped in StreamingOutputImpl(..., false) (isOneShot() = true), disabling retries on stale connections.
Cleaner approach: The only reason StreamingOutputImpl didn't know if the handle was resendable is because baseHandle.sendContent() returns an OutputStreamSender without the handle's isResendable() metadata. If OutputStreamSender (or an internal package-private sub-interface / HandleImplementation check) exposes isResendable(), StreamingOutputImpl(OutputStreamSender, RequestLogger, MediaType) can determine resendable directly from the sender without changing doPut, doPost, or any of their callers in OkHttpServices.

@prgs-rsingh prgs-rsingh left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to revisit the implementation

Minor Cleanup Nits
Duplicate method in OkHttpServices.java: CallRequestImpl.makeRequestBody(AbstractWriteHandle) (lines 5521–5534) is identical to OkHttpServices.makeRequestBodyForContent(AbstractWriteHandle) (lines 5294–5307) and can simply delegate to it.
RetryableRequestBody.isRetryable() vs RequestBody.isOneShot(): In both StreamingOutputImpl and ObjectRequestBody, isRetryable() is now just !isOneShot(). Since RetryableRequestBody is an internal interface in com.marklogic.client.impl.okhttp, we can either retire RetryableRequestBody in favor of OkHttp's RequestBody.isOneShot() or give it a default implementation.

}

private Response doPost(RequestLogger reqlog, Request.Builder requestBldr, Object value) {
// Default to non-resendable for callers that have not determined whether the content can be sent again.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 3-arg overloads that default to isResendable = false are what let call sites get missed. There are five, not four: 1934, 1936, 2193 (values() with a raw query), 4667 and 4765.

This matters more than it first appears. RetryIOExceptionInterceptor is only installed by the test clients (Common.java, ConnectedRESTQA). In production the effect of isOneShot() == true is on OkHttp itself: it stops recovering from a failed pooled connection and stops following up a 401. On develop these bodies were replayable, so a raw-query search with a StringHandle on a stale connection was retried silently. With this PR it fails, and a digest re-challenge surfaces as "Unauthorized". That is a regression for resendable handles at those five sites.

Rather than thread a boolean through every caller, I would read resendability in the one place the content is read:

private static RequestBody toRequestBody(HandleImplementation<?, ?> handle, MediaType type, RequestLogger reqlog) {
    Object content = handle.sendContent();
    if (content instanceof OutputStreamSender sender) {
        return new StreamingOutputImpl(sender, reqlog, type, handle.isResendable());
    }
    return new ObjectRequestBody(reqlog != null ? reqlog.copyContent(content) : content, type);
}

Then doPut/doPost take a RequestBody instead of an Object plus a flag. The overloads and the defaults go away, and a new call site cannot forget the flag. The same wrap-the-content block is currently written inline five times (putPostValueImpl, putPostValueImpl for values, doPut, doPost, addParts) and twice more as methods (makeRequestBodyForContent and CallRequestImpl.makeRequestBody, which are identical). All seven can use the one helper.

This needs no change to the public OutputStreamSender interface, so I prefer it to my earlier suggestion of exposing isResendable() there.

if (nextValue instanceof OutputStreamSender) {
sentValue = new StreamingOutputImpl(
(OutputStreamSender) nextValue, reqlog, mediaType);
(OutputStreamSender) nextValue, reqlog, mediaType, handle != null && handle.isResendable());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method now has two definitions of "resendable". Here it is handle != null && handle.isResendable(). Ten lines down, isResendable is computed as (handle == null) ? !isStreaming : handle.isResendable(). Please compute it once and use it for both the body and the retry decision.

Related, and already on develop: isStreaming(sentValue) on line 2469 is passed the RequestBody, not the content, so it is always true. It should be isStreaming(nextValue).

// InputStreams cannot be retried as they are consumed on first read.
return !(obj instanceof InputStream);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good change, and it fixes more than the multipart case. On develop, when OkHttp recovered a failed connection or followed up a 401, it called writeTo again on the same InputStream and sent only whatever was left unread: an empty or truncated body. Marking it one-shot stops that.

Two small things. isRetryable() is now exactly !isOneShot(), so one of them should go (see my note on RetryableRequestBody). And the class was widened from private only for the test. If the test builds the body through makeRequestBodyForContent(new InputStreamHandle(...)), it can stay private.

* Assumes the wrapped handle cannot be resent, which is the safe default for callers that have not determined
* whether the underlying content can be sent again (e.g. because it is backed by a one-shot stream).
*/
StreamingOutputImpl(OutputStreamSender handle, RequestLogger logger, MediaType contentType) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 3-arg constructor is no longer used by any main code, only by one test. Please remove it: a silent false default is the same trap as the 3-arg doPost.

isRetryable() and isOneShot() must now always be exact opposites, here and in ObjectRequestBody. Keeping two methods for one fact invites them to drift. I would drop RetryableRequestBody from both classes and let the interceptor ask isOneShot() only.

Nits: resendable (and the other fields) can be final, and the comment in isRetryable() describes what the code used to do. That history belongs in the commit message.

RequestBody requestBody = request.body();

if (request.body() instanceof RetryableRequestBody body && !body.isRetryable()) {
if (requestBody != null && (requestBody.isOneShot() ||

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honoring isOneShot() is the right contract, and the class Javadoc already describes it that way. Once the two body classes stop implementing RetryableRequestBody, the second half of this condition can go and the interface can be deprecated or removed (it is public, in an impl package, added in 8.0.0).

If it stays for now, please parenthesize the instanceof ... && ... part. It relies on && binding tighter than ||.

Out of scope, but the same class of bug: ProgressDataCloudAuthenticationConfigurer.TokenAuthenticationInterceptor calls chain.proceed a second time on a 401 without checking isOneShot(), so a streaming body is written twice after a token expiry. Worth a follow-up ticket.

import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

class StreamingOutputImplTest {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • These tests exercise RetryIOExceptionInterceptor, which production clients do not install. The behaviour that matters in production is whether real handles produce the right flag. A parameterized test over handles would cover that: makeRequestBodyForContent(handle).isOneShot() is false for StringHandle, JacksonHandle, BytesHandle and FileHandle, and true for ReaderHandle and InputStreamHandle.
  • nonResendableHandleInMultipartIsNotRetried repeats multipartBodyWithStreamingPartIsNotRetried and, despite its name, only asserts isOneShot().
  • The opposite case is missing: a multipart body whose parts are all resendable should not be one-shot and should be retried.
  • The client and failing-interceptor setup is copied in three tests. A small helper would do.
  • Nits: import Response, Protocol and ResponseBody rather than using full names, and declare throws IOException rather than wrapping it in RuntimeException.

}

@Test
void serverIsPingedBeforeEveryStreamingRequest() throws Exception {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test fixes the mechanism (HEAD, POST, HEAD, POST) rather than the behaviour, so any better fix would fail it. It also never reproduces the reported problem.

A more useful test: the server answers a first call and then closes the socket without sending Connection: close; a streaming call made straight after should succeed, and its body should arrive once and intact. That test passes for any correct fix.

}

@Test
void failedStreamingRequestIsNotResent() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The request-count assertion is the valuable one. The two message assertions depend on OkHttp's wording and on the platform (this can be "Connection reset" rather than "unexpected end of stream"), so I would drop them or loosen them.

Both tests use basic auth. Digest is the default and is the reason the ping exists, so one digest case would be worth having.

@prgs-rsingh prgs-rsingh left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description covers only the first commit and is now inaccurate: StreamingOutputImpl is no longer always one-shot, and neither the resendability plumbing nor the ping on every streaming Data Services request is mentioned. The title, branch and commits also mix MLE-32970 and MLE-32953.

I would split this in two. The one-shot contract plus a single body factory is one reviewable change. The ping is a behaviour and performance change that needs its root cause confirmed first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants