Skip to content

[FEATURE] Run Wan2.2 on Ascend 910B with a CANN 9.3.0-beta.1 image - #42

Open
yx0716 wants to merge 22 commits into
Tele-AI:mainfrom
yx0716:feat/npu-adapt
Open

yx0716 wants to merge 22 commits into
Tele-AI:mainfrom
yx0716:feat/npu-adapt

Conversation

@yx0716

@yx0716 yx0716 commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Run Wan2.2-TI2V-5B text-to-video on Atlas 910B2 with CANN 9.3.0-beta.1. Ascend uses TeleFuser’s existing platform and operator interfaces. Download the prebuilt arm64 Docker package or build from source. Docker combines the published TeleFuser application layer with the pinned official CANN image and Ubuntu FFmpeg. Public Docker package · Dockerfile · English guide · 中文手册

CUDA users who need Triton, torchao or bitsandbytes now install telefuser[cuda]. ROCm users need the Triton build paired with their PyTorch distribution.

Shared runtime changes

  • Device mesh and pipeline receive. Defaults previously selected CUDA. They now follow the active CPU, CUDA, ROCm or NPU platform.
  • Workers, tensor channels and ports. NPU tensor handles failed across process queues. Workers now transfer through host memory, bind each HCCL rank to its device and reserve separate port blocks. Shared rendezvous ports are released on shutdown and failed startup.
  • Wan VAE. NPU Conv3d uses contiguous NCDHW. Parallel decode now accepts already-batched tensors directly, fixing the torch.stack error for all callers, including CUDA/H100.
  • Attention. Wan and MiniMax DiT imports and LongCat and FlashVSR imports previously loaded Triton eagerly. FP8 and Sparse SageAttention now load their kernels on use. FP8 CPU smoothing matched main byte-for-byte.

Validation

  • CANN 9.3.0-beta.1, 910B2 (image from c27fabd). Wan took 157.07 s on one card and 92.05 s on two cards with CFG 2. The 121-frame 480p MP4s were byte-identical. Two-rank HCCL and tensor transfer also passed.
  • Public Docker install passed anonymous download, checksum verification, image assembly, CLI startup and an FFmpeg/TorchCodec round trip.
  • CI passed all seven checks on a9a3495. Four-card CANN 9.3, distributed VAE output parity and H100/ROCm hardware remain unverified. CI checks
  • Earlier CANN 8.2 runs took 180.9, 120.8 and 87.0 s on one, two and four cards at 480p. At 720p, one and four cards took 477.8 and 173.9 s.

@lzx1413

lzx1413 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Please add the tutorial document on how to run with Ascend NPU.

The VAE weight-side conversion to channels_last_3d is already guarded by cuDNN availability, but the activation-side casts in CausalConv3d.forward and the spatial-parallel halo conv were unconditional. NPU and CPU reject channels_last_3d activations, so Wan2.2 VAE forward failed on Ascend with ERR01007 ("NPU contiguous operator only supported contiguous memory format"). Route non-CUDA tensors through standard contiguous instead.

Verified: Wan2.2-TI2V-5B text-to-video smoke on Ascend 910B2 (CANN 8.2, torch_npu 2.9.0) completes end to end; CUDA path unchanged.
create_device_mesh_from_config defaulted device_type to "cuda" and 12 of 14 call sites rely on the default, so parallel denoising on NPU crashed inside torch DeviceMesh with "integer modulo by zero" (zero visible CUDA devices). Resolve the default from current_platform; explicit arguments keep working.

Verified: 4-card (cfg=2 x ulysses=2) Wan2.2-TI2V-5B run on Ascend 910B2 builds the [cfg, ulysses] mesh over hccl; on CUDA the default resolves to "cuda" as before.
Parallel workers exchange request and result tensors over torch.multiprocessing queues, which relies on device IPC. torch_npu cross-process sharing raised "devptr INTERNAL ASSERT FAILED ... entry in cache has missing shared_ptr" when rebuilding queued NPU tensors. Enable the existing queue_with_cpu marshalling by default on non-CUDA platforms, mirror it on the result path, and move results back to the stage device in the main process. Worker dispatch already moves inputs to the local device, so the contract is unchanged and CUDA keeps direct device-queue transport.

Verified: 4-card Wan2.2-TI2V-5B denoising on Ascend 910B2 completes with workers exiting cleanly.
ops/fp8_attention.py defined @triton.jit kernels at module scope, making "import triton" unconditional for every consumer of the wan_video and minimax DiT import chains even though the fused path is CUDA-only by contract. Move the two kernels and their launcher to telefuser/kernel/triton/fp8_attention.py and import them lazily inside quantize_fp8_qkv after its existing CUDA validation, matching the established ops -> kernel.triton dispatch pattern (see ops/rotary.py and kernel/__init__.py). The pure-torch quantize/dequantize helpers are unchanged.

Verified: wan22 pipeline import and Wan2.2-TI2V-5B smoke succeed on an NPU host without triton installed; tests/unit/models/test_wan_video_sol_attention.py passes with CUDA-only cases skipped.
recv, recv_latent, and recv_latent_async allocated receive buffers with device="cuda", which fails on NPU-only hosts before irecv can run. Allocate on current_platform.device_type (identical behavior on CUDA) and parameterize the test tensor device the same way so the suite runs on CUDA, NPU, and CPU hosts.

Verified: tests/unit/distributed/test_pp_comm.py 12 passed on Ascend 910B2 (previously 6 device-related failures).
The non-CUDA result path introduced for CPU-marshalled queues moved results back to self._stage.device inside _wait_result. That breaks worker unit tests on non-CUDA hosts (mocked stages resolve .device through the method proxy or to a MagicMock), and it is unnecessary: stage entry points already place their inputs, and the Wan VAE decode paths move latents themselves. Drop the move so _wait_result returns results untouched; workers still marshal results through CPU when device IPC is unavailable.

Verified: tests/unit/worker passes on an Ascend host (same non-CUDA branch as CPU CI) and the 4-card Wan2.2-TI2V-5B smoke still completes; ruff clean.
The example hardcoded device="cuda", so running it on an NPU host required editing the script. Default the pipeline device to current_platform.device_type so the same command runs unmodified on CUDA and NPU; on CUDA hosts this resolves to "cuda" as before.

Verified: python examples/wan_video/wan22_t2v_5b.py --gpu_num 1 and --gpu_num 4 run unmodified on Ascend 910B (50 steps, 121 frames, default prompt).
Wan22VideoVAE.decode's parallel branch called torch.stack on its input, assuming a list of per-video tensors, but VAEStage.decode_video passes a batched [B, C, T, H, W] tensor; the serial branch only worked because iterating a tensor yields its batch slices. Any enable_vae_parallel run of the Wan2.2 48-channel VAE therefore failed with TypeError regardless of platform. Normalize tensor inputs before stacking.

Found while enabling spatially parallel VAE decode on Ascend 910B2; the parallel path now proceeds to communicator setup (further multi-communicator progress on that host is limited by its CANN driver, see branch notes).
Concurrent worker groups sharing the same NPU devices (e.g. denoising plus VAE workers) collide on HCCL's default data-plane socket range and fail comm init with EJ0003 ("IP address and port have been bound already"). Allocate a distinct HCCL_IF_BASE_PORT per spawned group on NPU platforms, mirroring the existing per-group MASTER_PORT allocation; other platforms are untouched.

Verified: single-group 4-card Wan2.2-TI2V-5B regression on Ascend 910B2 passes with the env applied (33.4s generate, parity with baseline).
…o guides

Mirror the ROCm registration pattern for the NPU support validated by this
branch, updating the guides introduced on main:

- docs/en|zh/platforms.md: expand the "NPU and CPU" section with the
  native-fallback attention path (TORCH_SDPA, no triton/tf-kernel needed),
  forward_npu dispatch, HCCL with CPU-marshalled worker queues and per-group
  HCCL_IF_BASE_PORT isolation, and the validated wan22_t2v_5b.py entry point
  on Atlas 910B (CANN 8.2); note that Wan2.2 A14B is not yet exercised.
- docs/en|zh/installation.md: add an Ascend NPU requirements row and a
  verification note (torch_npu selection, ASCEND_RT_VISIBLE_DEVICES,
  torch.cuda.is_available() False is expected).
- examples/wan_video/README.md: add the Ascend requirements bullet and
  platform auto-detection / validated-hardware bullets to wan22_t2v_5b.py.

Verification: python scripts/docs/prepare_cookbook.py build passes with
rendered pages, assets, and anchors validated; the tests/docs unittest
failures on this host are macOS /var tempdir symlink artifacts unrelated to
these files. English additions stay within the 120-column style.
lzx1413 and others added 9 commits November 5, 2026 15:09
Restore the mainline smoothing, scale and correction paths for Wan and MiniMax when the optional Triton implementation is split out. The CPU paths import without Triton. All 18 FP8 function definitions match main apart from import placement. Four CPU smoothing modes matched bitwise. H100 execution remains a separate qualification.
Bind visible accelerator IDs before initializing the process group. Stage NPU tensors on the host for both multiprocessing transports, reserve and recycle aligned port blocks, and clean up a partially spawned HCCL group. Restore the CPU and GPU Conv3d activation layout while using NCDHW on Ascend. Exercise independent-port owners, shutdown, source metadata and uneven shards on CPU.
Pin the 910B CANN 9.3.0-beta.1 arm64 manifest, CPU PyTorch 2.11.0 and the Ascend 2.11.0.post4 wheel, including hashes for the runtime closure. Move GPU-specific packages into the cuda extra. Document source builds, release digests, device mapping and the 1/2/4-card offline T2V recipe in both languages. The documentation build and shell examples were checked. Hardware qualification is reported per card count.
CANN exports development tools through PYTHONPATH, including an optional profiler with absent dependencies and a separate NumPy constraint. Resolve and check the application venv with Python isolated mode while preserving the CANN entrypoint and its runtime library paths. Keep source labels after the locked dependency layer for cached application rebuilds.
Remove the stale Triton exclusions for the shared device mesh and pipeline communication suites. Exercise both list and batched Wan2.2 VAE inputs with tiled and spatial parallel dispatch.

Validated the focused CPU suite without Triton: 141 passed, 8 skipped. Ruff and formatting checks passed.
@yx0716 yx0716 changed the title [FEATURE] Run Wan2.2 pipelines on Ascend NPU [FEATURE] Run Wan2.2 on Ascend 910B with a CANN 9.3 image Nov 20, 2026
Import the sparse SageAttention kernels only when that backend executes. LongCat and FlashVSR can now load their normal service entry points without the optional CUDA extra. Keep the same Triton functions and arguments for sparse attention.

Skip the MiniMax Ulysses CUDA kernel tests when Triton itself is absent. Other missing dependencies still fail collection. Verified the CPU service parity tests and both model imports without Triton.
@yx0716 yx0716 changed the title [FEATURE] Run Wan2.2 on Ascend 910B with a CANN 9.3 image [FEATURE] Run Wan2.2 on Ascend 910B with a CANN 9.3.0-beta.1 image Nov 20, 2026
Export the locked Python environment and Wan example as an arm64 application image. Assemble it with the official CANN image and Ubuntu FFmpeg on the user host. Document the public GitHub Release download, checksum verification and Docker commands in both runbooks.

Verified the original source-build path, the application archive and no-context assembly, the TeleFuser and Wan CLI entry points, a TorchCodec and FFmpeg round trip, both docs builds, and Bash syntax.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants