Skip to content

[CELEBORN-XXXX][WIP] Mark CelebornShuffleHandle as reliably stored for per-shuffle DRA safety - #3835

Draft
venkata91 wants to merge 1 commit into
apache:mainfrom
venkata91:dra-per-shuffle-reliable-storage
Draft

[CELEBORN-XXXX][WIP] Mark CelebornShuffleHandle as reliably stored for per-shuffle DRA safety#3835
venkata91 wants to merge 1 commit into
apache:mainfrom
venkata91:dra-per-shuffle-reliable-storage

Conversation

@venkata91

@venkata91 venkata91 commented Aug 31, 2026

Copy link
Copy Markdown

What changes were proposed in this pull request?

  • Adds a per-shuffle reliable-storage signal so Spark does not recompute a lost executor's map stages for shuffles whose data lives on Celeborn workers.
  • On executor loss, Spark's DAGScheduler unregisters the executor's map outputs and recomputes the map stage. A CelebornShuffleHandle is only created for shuffles stored on Celeborn workers (fallback shuffles use Spark's SortShuffleManager), so their output survives executor loss and the recompute is wasteful.
  • Overrides ShuffleHandle.isReliablyStored to return true on CelebornShuffleHandle; fallback shuffles keep Spark's BaseShuffleHandle (default false), so the mixed case is handled per shuffle.
  • The override is committed commented out: ShuffleHandle.isReliablyStored does not exist in any released Spark, so enabling it would break compilation against the Spark versions Celeborn builds against.
  • Blocked on the Spark-side PR ([SPARK-59138][CORE] Preserve reliably-stored shuffle map outputs on executor loss spark#58437, SPARK-59138). Do not merge until that lands, ships in a release, and Celeborn's Spark dependency is bumped; then uncomment the override and update the JIRA IDs.
  • Follow-up to CELEBORN-2451, which ties supportsReliableStorage to the NEVER fallback policy to stay DRA-safe. This PR lets AUTO fallback avoid the recompute penalty per shuffle.

Why are the changes needed?

  • Under DRA with AUTO fallback, losing an executor needlessly recomputes map stages whose output is safely stored on Celeborn workers.
  • The app-global supportsReliableStorage() flag cannot express "this shuffle is on Celeborn, that one fell back to local disk," so it forces NEVER to be safe. Per-shuffle reliability removes that constraint.

Does this PR resolve a correctness bug?

  • Yes

Does this PR introduce any user-facing change?

  • Yes

How was this patch tested?

  • Not yet exercised end-to-end: the override is commented out pending the Spark dependency.
  • Once unblocked, validated against a Spark build carrying SPARK-59138 with a DRA + AUTO-fallback job that loses an executor, asserting the Celeborn-backed shuffle's map stage is not recomputed while a fallback shuffle's is.

…r per-shuffle DRA safety

Spark recomputes a lost executor's map stages on executor loss. With the
Spark-side per-shuffle reliable-storage change (SPARK-59138), a shuffle can
tell Spark its output survives executor loss via
ShuffleHandle.isReliablyStored, avoiding a needless recompute.

A CelebornShuffleHandle is only created for shuffles stored on Celeborn
workers (fallback shuffles use Spark's SortShuffleManager), so it can report
reliable storage. The override is left commented out here because it does not
compile against currently released Spark; this PR is blocked on the OSS Spark
PR landing and Celeborn's Spark dependency being bumped.

Follow-up to CELEBORN-2451.
@venkata91
venkata91 force-pushed the dra-per-shuffle-reliable-storage branch from 7d8762d to da25f4c Compare August 31, 2026 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant