[CELEBORN-2451] Make DRA safe under AUTO fallback by tying supportsReliableStorage to fallback policy - #3834
Open
venkata91 wants to merge 2 commits into
Open
Conversation
…bleStorage to fallback policy Report reliable storage only for fallback policy NEVER. Under AUTO/ALWAYS a shuffle can fall back to the local-disk SortShuffleManager, whose output is lost when the executor is reclaimed, so claiming reliable storage lets DRA release those executors and triggers FetchFailed. Reporting false makes Spark's ExecutorAllocationManager require shuffle tracking (or a shuffle service) under DRA, failing fast at startup otherwise. Make the SparkShuffleManager DRA guidance policy-aware and downgrade the registerShuffle fallback log from error to warn when shuffle tracking is on.
github-actions
Bot
added
correctness
venkata91
changed the title
venkata91
marked this pull request as ready for review
August 31, 2026 18:20
Author
|
cc @SteNicholas for review. |
Author
|
cc @sunchao |
Author
|
Gentle ping @SteNicholas @1fanwang !! Thanks !! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
supportsReliableStorage()to the fallback policy:trueonly forNEVER,falseforAUTO/ALWAYS(was keyed on deprecatedforceFallback, so it returnedtrueunderAUTO).SparkShuffleManagerDRA warnings policy-aware; log theregisterShufflefallback aterroronly when shuffle tracking is also off.Why are the changes needed?
supportsReliableStorage()is one global flag.Under
AUTOa shuffle can fall back to local-diskSortShuffleManager;that output dies with a DRA-reclaimed executor, causing
FetchFailedException.Nothing prevents the unsafe config today.
falseunderAUTO/ALWAYSlets Spark'sExecutorAllocationManagerfail fast unless a shuffle service, shuffle tracking, or decommissioning is on,
so
AUTOis safe withshuffleTracking.enabled=true;NEVERkeeps the fast path.AUTO,reliableStorage=falsemakes Spark recompute a lostexecutor's map stages even though Celeborn still holds the data;
NEVERavoidsthis. Removing it needs a Spark-side change (per-shuffle reliable-storage
granularity), tracked as a follow-up.
Does this PR resolve a correctness bug?
Does this PR introduce any user-facing change?
AUTO(default), DRA without shuffle service and without shuffletracking now fails fast at startup. Enable tracking or set
NEVER.How was this patch tested?
testReliableStorageFollowsFallbackPolicy;-Dtest=SparkShuffleManagerSuitepasses (5/5), spotless clean.