Skip to content

fix(ilp): fix a leaked socket and native memory when an HTTP sender fails to start - #78

Merged
bluestreak01 merged 10 commits into
mainfrom
puzpuzpuz_http_ctor_rollback
Sep 24, 2026
Merged

bluestreak01 merged 10 commits into
mainfrom
puzpuzpuz_http_ctor_rollback

Conversation

@puzpuzpuz

@puzpuzpuz puzpuzpuz commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Found while reviewing questdb/questdb#7434, which made the equivalent sender
construction path exception-safe in the core repository. The ILP HTTP sender
uses this repository's HttpClientFactory, so it needs the same ownership
guarantee here.

Problem

An HTTP sender can fail after acquiring a client but before the sender escapes
to its caller. In particular, newRequest() can throw while writing the request
preamble. The same risk exists when the OIDC token provider is installed and the
factory rebuilds the initial request. In both cases, the client, its socket and
its native buffers were left without an owner that could close them.

HttpHeaderParser also allocated its DirectUtf8Sink in a field initializer.
Because field initializers run before the constructor body, a later header-buffer
allocation failure could strand the sink outside the constructor's cleanup path.

Finally, if closing a staged socket during HttpClient construction failed, that
cleanup failure could replace the exception that explained why construction
failed.

Changes

  • Move the parser sink allocation into the constructor's guarded region so both
    native allocations are released if construction fails.
  • Preserve the primary HttpClient construction exception and attach any socket
    close failure as suppressed.
  • Close the owned client if the sender constructor fails while creating its
    initial request.
  • Apply the same rollback when rebuilding the initial request after wiring an
    OIDC token provider.

There is no happy-path behavior change. A client supplied to a sender is now
closed if sender construction fails, matching the existing ownership contract on
the successful path, where closing the sender also closes that client.

The ILP HTTP senders build their client through this fork's
HttpClientFactory, whose constructors took a socket and native buffers
with no rollback. A throw past the first acquisition left a half-built
client the caller never receives, so close() never ran and everything
taken so far leaked, the socket file descriptor included.

HttpClient now takes the socket, both buffers and the response parser
inside one try and frees what it took in reverse order, attaching a
close failure to the primary exception rather than replacing it.
ResponseHeaders does the same for the parser its super() call
completed. HttpHeaderParser moves the direct UTF-8 sink out of its
field initialiser, which runs before the constructor body, so the
constructor's own catch can reach it. The three platform subclasses
close the completed base client when their poller fails to open.

HttpClientWindows also reads the select facade before it takes the FD
set. Nothing between that acquisition and the end of the constructor
can then throw, which removes a partial-FDSet cleanup branch that only
a Windows host could ever have reached.

HttpClientConstructorTest covers each rollback point with a
close-counting socket, and HttpHeaderParserTest covers the sink. Both
inject the fault deterministically and without a host dependency: a
negative buffer size makes Unsafe.allocateMemory throw, and a throwing
configuration getter fails each platform subclass before it touches
platform natives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@puzpuzpuz puzpuzpuz added the bug Something isn't working label Jul 30, 2026
puzpuzpuz added a commit to questdb/questdb that referenced this pull request Jul 30, 2026
The client-side port of the HTTP constructor rollback lives in
questdb/java-questdb-client#78, which is not merged yet. Pinning the
submodule at that branch commit would make this PR depend on an unmerged
revision and, if merged first, would permanently fix the pointer at a
commit that is not on the client's main branch.

Point the submodule back at 2e84db07, the commit this PR already carried
and the one main was on when the branch was cut. Once #78 merges, the
pointer moves forward to a main commit that contains the fix.

Chosen over the current main head, 9ccfadbe: that is 1.3.7-SNAPSHOT and
would require bumping questdb.client.version in three parent poms, and it
would drag in a version bump and a PEM CA feature unrelated to this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
puzpuzpuz and others added 3 commits July 30, 2026 11:23
AbstractLineHttpSender's constructor assigned this.client - either the
one createLineSender() handed in or one it built through
HttpClientFactory - and then called newRequest(), with neither statement
inside a try. newRequest() writes the POST line, the path, the
User-Agent header and any auth header into the client's request buffer,
and growBuffer() throws once they exceed the maximum. A throw there left
a fully built client - a socket plus two native buffers plus the
response parser - that no caller ever receives a reference to, so
nothing closed it. A Java-heap OutOfMemoryError from the version lookup
or the User-Agent concatenation lands in the same window.

createLineSender() cannot clean this up. Its three Misc.free(cli) calls
all sit before the switch, and the switch builds the sender inside a
return expression that no try covers. Both entry shapes leak: an
explicit protocol version makes the constructor build the client itself,
and an auto-detected one hands in the client the version probe used.

The constructor now takes the client, the version and the request inside
one try and closes the client on the way out, attaching a close failure
to the primary exception rather than replacing it. It frees a handed-in
client too: close() already frees the client regardless of where it came
from, so the sender owns it from the assignment on either way.

LineHttpSenderConstructorTest covers both shapes. A configuration whose
maximum request buffer is smaller than the preamble makes newRequest()
throw deterministically, with no server and no new production seam. The
self-built case asserts against the enclosing leak check; the handed-in
case adds a close-counting socket, since the leak check cannot see a
file descriptor. Both fail without the fix: the first on 65552 bytes
leaked under NATIVE_DEFAULT, the second on the missing socket close.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… probe

facadeCloseIsBoundedUnderRepeatedInterruptsDuringSenderCreation failed on
windows-msvc-2022-x64 with "sender close left its creation wait before the
interrupt storm landed twice; interrupts landed: 6" -- a false negative, not
a product defect.

Both pools' hasCreationWaiterForTesting() reported the wait region as
lock.hasWaiters(creationFinished). An interrupt does not leave the waiter on
the condition queue: AQS transfers the node to the lock's sync queue to
reacquire before awaitNanos rethrows, so under the test's interrupt storm the
probe reads false for a measurable slice of a wait close() never left. A
harness replicating close()'s loop measures 5-8% of polls reading false with
the 1s deadline honoured exactly.

Count the threads inside the wait loop instead: the state no longer flickers,
and the rising edge no longer depends on the closer having actually parked.
awaitRepeatedInterrupts() also re-reads the interrupt count before failing,
since the count it samples precedes the predicate read that fails on it.

creationWaitStateDoesNotFlickerUnderInterruptStorm pins the contract: red on
the old predicate (flickered after 46 polls, 2 interrupts landed), green now.

Full suite green locally: 2766 tests, 0 failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ollback

# Conflicts:
#	core/src/main/java/io/questdb/client/cutlass/http/client/HttpClient.java
#	core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientLinux.java
#	core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientOsx.java
#	core/src/main/java/io/questdb/client/cutlass/http/client/HttpClientWindows.java
#	core/src/main/java/io/questdb/client/cutlass/line/http/AbstractLineHttpSender.java
#	core/src/main/java/io/questdb/client/impl/QueryClientPool.java
#	core/src/main/java/io/questdb/client/impl/SenderPool.java
#	core/src/test/java/io/questdb/client/test/impl/QuestDBImplCloseLifecycleTest.java
@jerrinot

Copy link
Copy Markdown
Contributor

Review (level 3): approve. Nothing blocking, and no Moderate or Minor findings.

What I checked

  • Supplied-client cleanup: the only caller that passes an existing client into the sender constructor is the version-detection path in createLineSender, and it has no cleanup of its own around that call. The new rollback in AbstractLineHttpSender fixes the leak without any double close.
  • Token provider: passing it into the constructor reaches the same end state as before (token pending, request stopped after the headers), and drops a redundant second client.newRequest() during setup.
  • Other files: HttpHeaderParser, HttpClient, QwpWebSocketSender and SenderPool keep the same cleanup order. Misc.freeSuppressing behaves exactly like the SenderPool.addSuppressed helper it replaces. Java 8 floor respected; no change on the steady-state send path.
  • Tests: LineHttpSenderConstructorTest, HttpClientConstructorTest, HttpHeaderParserTest, MiscTest and LineHttpSenderTokenProviderTest pass on head (5b450b7). testHandedInClientIsReleasedWhenRequestPreambleDoesNotFit asserts the socket is closed twice, so it would fail if the rollback were removed.

Scope

  • No submodule pointer moves.
  • No tandem questdb/questdb-enterprise PR needed: the change only affects cleanup when a sender fails to start, which is not server-observable and is unit-testable here.

@jerrinot jerrinot added READY and removed READY labels Sep 23, 2026
@mtopolnik

Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 38 / 39 (97.44%)

file detail

path covered line new line coverage
🔵 io/questdb/client/std/Misc.java 8 9 88.89%
🔵 io/questdb/client/cutlass/http/client/HttpClient.java 1 1 100.00%
🔵 io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java 1 1 100.00%
🔵 io/questdb/client/cutlass/http/HttpHeaderParser.java 14 14 100.00%
🔵 io/questdb/client/impl/SenderPool.java 1 1 100.00%
🔵 io/questdb/client/cutlass/line/http/AbstractLineHttpSender.java 13 13 100.00%

@jerrinot jerrinot added the READY label Sep 23, 2026

@bluestreak01 bluestreak01 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #78 review: HTTP sender and client constructor rollback (level 3)

Verdict: approve with comments. Nothing blocks the merge and the test gate passes. One Moderate item should be fixed before merge: the change removes some released public constructor signatures without keeping the old ones. The rollback fix itself is correct, and I found no issues with it.

Critical

None.

Moderate

M1. Released multi-host sender constructors were changed in place instead of overloaded

  • Problem: four constructors from the 1.3.9 release no longer exist. There is no overload with the old parameter list.
  • Net impact: any outside code that calls these constructors directly, or subclasses these senders, stops compiling or linking. I found no such callers.
  • Evidence:
    • Release 1.3.9 (git show 1.3.9:…) has these 17-argument constructors, ending (…, int currentAddressIndex, Rnd rnd):
      • LineHttpSenderV2 and LineHttpSenderV3: public.
      • AbstractLineHttpSender and LineHttpSenderV1: protected. The classes are public and not final, so subclasses can call them.
    • At head 5b450b7, javap -protected lists only the 18-argument forms, which add HttpTokenProvider at the end.
    • A small caller that uses the old LineHttpSenderV2 constructor compiles against base 916e51f (javac exit 0). Against head it fails with "no suitable constructor found".

Where it is:

  • core/src/main/java/io/questdb/client/cutlass/line/http/AbstractLineHttpSender.java:168-187
  • LineHttpSenderV1.java:82-99
  • LineHttpSenderV2.java:86-105
  • LineHttpSenderV3.java:46-65

module-info.java:52 exports io.questdb.client.cutlass.line.http. The repo's own rules apply here:

  • ExportedApiCompatibilityTest says: "Adding an overload is fine … retyping or removing one turns it red". It also explains that a break with no known callers was still restored in the past.
  • createLineSender keeps a provider-less overload "so callers compiled against the pre-httpTokenProvider signature keep linking".

This PR goes against that. The test doesn't catch it because it only checks createLineSender and connect, not constructors.

Why Moderate, not Critical: I searched the questdb and questdb-enterprise checkouts and found no callers. They only use the 13-argument single-host LineHttpSenderV2 constructor, which is unchanged. The removed constructors also take internal plumbing types (HttpClient, Rnd, currentAddressIndex). So I can't name anyone who is actually affected.

Suggested fix: add the four 17-argument constructors back as overloads that pass null for the provider. Also add the constructor signatures to ExportedApiCompatibilityTest, for example with a constructor variant of assertSignaturesPresent.

Minor

None.

Coverage map

Test gate: pass. No coverage gaps admitted.

Every behaviour change has a local test that fails without the fix. I ran the new tests on JDK 8 at head (all pass) and at base with the tests adapted to compile there (base 916e51f):

New test Result at base
HttpHeaderParserTest.testConstructorFailureFreesNativeAllocations fails: 32 B leaked under NATIVE_DIRECT_UTF8_SINK
LineHttpSenderConstructorTest.testHandedInClient… fails: socket closed once, expected twice
LineHttpSenderConstructorTest.testProviderClient… fails: leak under NATIVE_DEFAULT
LineHttpSenderConstructorTest.testSelfBuiltClient… fails: leak under NATIVE_DEFAULT
HttpClientConstructorTest.testBaseConstructorFailurePreservesOriginalFailure… errors: the socket close failure replaces the original exception

The wider run (*Http*, *LineSender*, *SenderPool*, *TokenProvider*, Misc*) passed at head: 649 tests, 0 failures. The code also compiles and test-compiles on JDK 25.

No tandem PR is needed. The change only affects what happens when a sender fails to start, which a server can't observe and unit tests here can cover. I searched for tandem PRs on the puzpuzpuz_http_ctor_rollback branch: there are none in questdb or questdb-enterprise.

Summary

  • Submodules: no submodule pointer moves.
  • Scope of the finding: M1 is an in-diff issue with no breakage outside the diff. I checked every caller of the changed symbols across this repo and the local questdb/questdb-enterprise checkouts.
  • Rollback fix:
    • The client a sender is handed is closed exactly once. createLineSender has no cleanup of its own around the constructor switch, so there is no double close.
    • Passing the token provider through the constructor leaves the request in the same state as the old rebuild. The only difference is one fewer client.newRequest() call.
    • Misc.freeSuppressing behaves the same as the old SenderPool helper. It is slightly safer in QwpWebSocketSender, where it can no longer throw on self-suppression.
    • Nothing changes on the per-row or flush path.
  • Findings: 0 Critical, 1 Moderate (M1), 0 Minor.

Before approving, I'd ask the author to add the constructor overloads from M1.

@bluestreak01 bluestreak01 added the QUEUED FOR MERGE Approved PR in the merge queue. Do not merge master into this PR. label Sep 24, 2026
@bluestreak01
bluestreak01 merged commit c9968f2 into main Sep 24, 2026
19 checks passed
@bluestreak01
bluestreak01 deleted the puzpuzpuz_http_ctor_rollback branch September 24, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working QUEUED FOR MERGE Approved PR in the merge queue. Do not merge master into this PR. READY

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants