Conversation
…hange job ### What problem does this PR solve? Issue Number: close apache#66177 Problem Summary: Reproduction: 1. Make a tablet of the table stay in the tablet scheduler for a long time. For example, an orphan CLONE replica whose pathHash is -1 makes the scheduler always fail with "dest replica has no slot" and remove the tablet ctx after too many failures, while the colocate checker keeps re-adding the tablet to the pending queue on every check round. 2. Submit an ALTER TABLE schema change on the table. The job's checkTableStable() finds the tablet in the tablet scheduler, sets the table state to WAITING_STABLE and the job keeps waiting in PENDING. 3. Execute CANCEL ALTER TABLE COLUMN. FE logs "set table's state to NORMAL when cancel", but the table state is actually still WAITING_STABLE. The job has already been marked CANCELLED, so cancelling again fails with "could not find related job", and any subsequent DDL on the table is rejected because the table state is not NORMAL. The table is stuck until someone manually runs ADMIN SET TABLE ... STATUS PROPERTIES("state" = "NORMAL"). Root cause: SchemaChangeJobV2.changeTableState() only resets the table state when the current state is SCHEMA_CHANGE, and misses the WAITING_STABLE case. When the job is cancelled (or finished/replayed) while the table is still WAITING_STABLE, the table state is never reset to NORMAL, while the job itself is already in a final state and will never touch the table again. Fix: also allow resetting the table state from WAITING_STABLE in SchemaChangeJobV2.changeTableState(). WAITING_STABLE is only set by this job's own checkTableStable(), so it is always safe to reset it here. ### Release note Fix the issue that cancelling a schema change job which is still waiting for the table to become stable leaves the table in WAITING_STABLE state forever, blocking all subsequent DDL on the table. ### Check List (For Author) - Test: Unit Test - Added SchemaChangeJobV2Test#testCancelSchemaChangeWhileTableWaitingStable. It fails without the fix (expected: <NORMAL> but was: <WAITING_STABLE>) and passes with the fix. The whole SchemaChangeJobV2Test (14 tests) passes. - Behavior changed: Yes - after cancelling a schema change job that is waiting in PENDING state, the table state is now correctly reset to NORMAL instead of staying WAITING_STABLE - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #66177
Problem Summary:
Reproduction:
Root cause: SchemaChangeJobV2.changeTableState() only resets the table state when the current state is SCHEMA_CHANGE, and misses the WAITING_STABLE case. When the job is cancelled (or finished/replayed) while the table is still WAITING_STABLE, the table state is never reset to NORMAL, while the job itself is already in a final state and will never touch the table again.
Fix: also allow resetting the table state from WAITING_STABLE in SchemaChangeJobV2.changeTableState(). WAITING_STABLE is only set by this job's own checkTableStable(), so it is always safe to reset it here.
Release note
Fix the issue that cancelling a schema change job which is still waiting for the table to become stable leaves the table in WAITING_STABLE state forever, blocking all subsequent DDL on the table.
Check List (For Author)