[CELEBORN-XXXX][WIP] Mark CelebornShuffleHandle as reliably stored for per-shuffle DRA safety - #3835
Draft
venkata91 wants to merge 1 commit into
Draft
[CELEBORN-XXXX][WIP] Mark CelebornShuffleHandle as reliably stored for per-shuffle DRA safety#3835venkata91 wants to merge 1 commit into
venkata91 wants to merge 1 commit into
Conversation
…r per-shuffle DRA safety Spark recomputes a lost executor's map stages on executor loss. With the Spark-side per-shuffle reliable-storage change (SPARK-59138), a shuffle can tell Spark its output survives executor loss via ShuffleHandle.isReliablyStored, avoiding a needless recompute. A CelebornShuffleHandle is only created for shuffles stored on Celeborn workers (fallback shuffles use Spark's SortShuffleManager), so it can report reliable storage. The override is left commented out here because it does not compile against currently released Spark; this PR is blocked on the OSS Spark PR landing and Celeborn's Spark dependency being bumped. Follow-up to CELEBORN-2451.
venkata91
force-pushed
the
dra-per-shuffle-reliable-storage
branch
from
August 31, 2026 23:23
7d8762d to
da25f4c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
DAGSchedulerunregisters the executor's map outputs and recomputes the map stage. ACelebornShuffleHandleis only created for shuffles stored on Celeborn workers (fallback shuffles use Spark'sSortShuffleManager), so their output survives executor loss and the recompute is wasteful.ShuffleHandle.isReliablyStoredto returntrueonCelebornShuffleHandle; fallback shuffles keep Spark'sBaseShuffleHandle(defaultfalse), so the mixed case is handled per shuffle.ShuffleHandle.isReliablyStoreddoes not exist in any released Spark, so enabling it would break compilation against the Spark versions Celeborn builds against.supportsReliableStorageto theNEVERfallback policy to stay DRA-safe. This PR lets AUTO fallback avoid the recompute penalty per shuffle.Why are the changes needed?
supportsReliableStorage()flag cannot express "this shuffle is on Celeborn, that one fell back to local disk," so it forcesNEVERto be safe. Per-shuffle reliability removes that constraint.Does this PR resolve a correctness bug?
Does this PR introduce any user-facing change?
How was this patch tested?