branch-4.1: [fix](be) Fix SIGSEGV in bvar::take_sample caused by ~AgentCombiner walking freed TLS agents #67276 - #68045
Open
github-actions[bot] wants to merge 1 commit into
Open
github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
…alking freed TLS agents (#67276) ### What problem does this PR solve? Issue Number: close #66895 Related PR: apache/brpc#3291 Problem Summary: Follow-up to #64040 (backport of apache/brpc#2949). That fix made `Agent::~Agent()` safe by using a `weak_ptr` for `Agent::combiner`, but `~AgentCombiner()` still calls `clear_all_agents()`. When the last `shared_ptr` to the combiner is released while another thread is exiting, the TLS agents' `weak_ptr`s are already expired: `~Agent` skips `commit_and_erase()` and leaves its `LinkNode` in `_agents`, then the `ThreadBlock` is freed - and `clear_all_agents()` walks that freed TLS storage, causing a heap-use-after-free. In production this shows up as SIGSEGV in `bvar::SeriesSampler::take_sample` (#66895). Fix: port apache/brpc#3291 - `~AgentCombiner()` no longer traverses `_agents`. This is safe: `butil::LinkNode` has a trivial destructor and is never dereferenced when the list is torn down, and surviving agents observe `combiner.expired() == true` in `~Agent` and skip `commit_and_erase()`. There is no memory leak: agents are freed together with their `ThreadBlock` at thread exit. This closes the gap that was flagged during review of #64040. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. Manual test: 1. Applied the patch on top of the existing `brpc-1.4.0-fix-agent-combiner-thread-safety.patch` and rebuilt brpc 1.4.0 thirdparty + BE (4.1.3-based image). 2. Ran the same workload that previously triggered the crashes in #66895: sustained high-EPS stream load while rewriting a ~40B-row table from storage format V2 to V3 on the affected production cluster (compute-storage coupled mode, 6 BE nodes, Kubernetes). 3. Before the fix: BEs crashed with SIGSEGV in `bvar::SeriesSampler::take_sample` (two BEs independently, see issue). After the fix: the production cluster has been running stable for 2 days with the patch applied - no recurrence of the SIGSEGV. ### Check List (For Reviewer who merge this PR) - Confirm the release note - Confirm test cases - Confirm document - Add branch pick label Co-authored-by: stinger1206 <4540066+stinger1206@users.noreply.github.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #67276