Skip to content

[fix](fe) Fix table stuck in WAITING_STABLE after cancelling schema change job - #68049

Closed
Ryan19929 wants to merge 1 commit into
apache:masterfrom
Ryan19929:fix/fe-sc-cancel-waiting-stable
Closed

Ryan19929 wants to merge 1 commit into
apache:masterfrom
Ryan19929:fix/fe-sc-cancel-waiting-stable

Conversation

@Ryan19929

@Ryan19929 Ryan19929 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #66177

Problem Summary:

Reproduction:

  1. Make a tablet of the table stay in the tablet scheduler for a long time. For example, an orphan CLONE replica whose pathHash is -1 makes the scheduler always fail with "dest replica has no slot" and remove the tablet ctx after too many failures, while the colocate checker keeps re-adding the tablet to the pending queue on every check round.
  2. Submit an ALTER TABLE schema change on the table. The job's checkTableStable() finds the tablet in the tablet scheduler, sets the table state to WAITING_STABLE and the job keeps waiting in PENDING.
  3. Execute CANCEL ALTER TABLE COLUMN. FE logs "set table's state to NORMAL when cancel", but the table state is actually still WAITING_STABLE. The job has already been marked CANCELLED, so cancelling again fails with "could not find related job", and any subsequent DDL on the table is rejected because the table state is not NORMAL. The table is stuck until someone manually runs ADMIN SET TABLE ... STATUS PROPERTIES("state" = "NORMAL").

Root cause: SchemaChangeJobV2.changeTableState() only resets the table state when the current state is SCHEMA_CHANGE, and misses the WAITING_STABLE case. When the job is cancelled (or finished/replayed) while the table is still WAITING_STABLE, the table state is never reset to NORMAL, while the job itself is already in a final state and will never touch the table again.

Fix: also allow resetting the table state from WAITING_STABLE in SchemaChangeJobV2.changeTableState(). WAITING_STABLE is only set by this job's own checkTableStable(), so it is always safe to reset it here.

Release note

Fix the issue that cancelling a schema change job which is still waiting for the table to become stable leaves the table in WAITING_STABLE state forever, blocking all subsequent DDL on the table.

Check List (For Author)

  • Test: Unit Test
    • Added SchemaChangeJobV2Test#testCancelSchemaChangeWhileTableWaitingStable. It fails without the fix (expected: but was: <WAITING_STABLE>) and passes with the fix. The whole SchemaChangeJobV2Test (14 tests) passes.
  • Behavior changed: Yes - after cancelling a schema change job that is waiting in PENDING state, the table state is now correctly reset to NORMAL instead of staying WAITING_STABLE
  • Does this need documentation: No

…hange job

### What problem does this PR solve?

Issue Number: close apache#66177

Problem Summary:

Reproduction:
1. Make a tablet of the table stay in the tablet scheduler for a long time. For example, an orphan CLONE replica whose pathHash is -1 makes the scheduler always fail with "dest replica has no slot" and remove the tablet ctx after too many failures, while the colocate checker keeps re-adding the tablet to the pending queue on every check round.
2. Submit an ALTER TABLE schema change on the table. The job's checkTableStable() finds the tablet in the tablet scheduler, sets the table state to WAITING_STABLE and the job keeps waiting in PENDING.
3. Execute CANCEL ALTER TABLE COLUMN. FE logs "set table's state to NORMAL when cancel", but the table state is actually still WAITING_STABLE. The job has already been marked CANCELLED, so cancelling again fails with "could not find related job", and any subsequent DDL on the table is rejected because the table state is not NORMAL. The table is stuck until someone manually runs ADMIN SET TABLE ... STATUS PROPERTIES("state" = "NORMAL").

Root cause: SchemaChangeJobV2.changeTableState() only resets the table state when the current state is SCHEMA_CHANGE, and misses the WAITING_STABLE case. When the job is cancelled (or finished/replayed) while the table is still WAITING_STABLE, the table state is never reset to NORMAL, while the job itself is already in a final state and will never touch the table again.

Fix: also allow resetting the table state from WAITING_STABLE in SchemaChangeJobV2.changeTableState(). WAITING_STABLE is only set by this job's own checkTableStable(), so it is always safe to reset it here.

### Release note

Fix the issue that cancelling a schema change job which is still waiting for the table to become stable leaves the table in WAITING_STABLE state forever, blocking all subsequent DDL on the table.

### Check List (For Author)

- Test: Unit Test
    - Added SchemaChangeJobV2Test#testCancelSchemaChangeWhileTableWaitingStable. It fails without the fix (expected: <NORMAL> but was: <WAITING_STABLE>) and passes with the fix. The whole SchemaChangeJobV2Test (14 tests) passes.
- Behavior changed: Yes - after cancelling a schema change job that is waiting in PENDING state, the table state is now correctly reset to NORMAL instead of staying WAITING_STABLE
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Ryan19929 Ryan19929 closed this Sep 16, 2026
@Ryan19929
Ryan19929 deleted the fix/fe-sc-cancel-waiting-stable branch September 16, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Table stuck in WAITING_STABLE after cancelling a schema-change job that never became stable

2 participants