Refactor/npu transformer cleanup - #272
Open
fxp19950422 wants to merge 19 commits into
Open
Conversation
Add platforms registry with Ascend capability probes, extend utils/platform for npu/hccl/compile kwargs, and register a MindIE attention backend for Qwen-Image on the v1 layout. Co-authored-by: Cursor <cursoragent@cursor.com>
Bind Ascend devices before HCCL init_process_group, align/bind config.device across engine workers, and reject MindIE with ring SP. Co-authored-by: Cursor <cursoragent@cursor.com>
Use addcmul for gated residual/modulate micro-opts, and gate MindIE RoPE / layernorm_scale_shift behind USE_MINDIESD_FUSE. Co-authored-by: Cursor <cursoragent@cursor.com>
under Ulysses SP, hiding all-to-all latency behind FA compute (compute-communication overlap). - Ulysses flow via all_to_all_4D_pre/single/after (SeqAllToAll4D). - FA_ALLTOALL_CUT/OVERLAP count semantics (0/1 off, >1 on, CUT wins): baseline single round trip; cut = head-chunked round trips; overlap = all-to-all on side stream2 overlapped with FA on main stream via npu Event/Stream sync. - current_stream refreshed at forward time. - Ulysses-only; falls back to USPAttention when not available.
…ction - Add Ascend long-context attention under Ulysses SP with a single shared all-to-all comm stream - Add fused RMSNorm and use it in the Qwen-Image transformer - Unify platform abstraction via current_platform - Centralize Ascend tuning knobs (op_fusion, fa_alltoall_overlap/cut) as platform attributes
…ps and attention factory
…stack Results: -0.27% speed (no gain), SSIM=0.849 (precision degradation) Conclusion: torch.compile not viable on CANN 9.1.0 + MindIE backend Code retained for future re-evaluation when CANN improves
GPU H20 (134): 13 configs tested (Ulysses/Ring/Hybrid/CFG) - Best: 8card_cfg_u4 = 5.04x speedup, 63% eff - 4card_ulysses = 2.92x, 73% eff (sweet spot) NPU 910B: 4-card and 8-card Ulysses - 4-card: 2.03x speedup, 50.7% eff - 8-card: 1.56x (ANTI-SCALING, comm overhead 81%) Key finding: CFG parallel is P0 optimization for NPU multi-card NPU single-card is 2.6x faster than GPU H20
… NPU) Results: - NPU 8card_cfg_u4: 175.87 ms/step (3.16x), vs pure Ulysses 354.59ms (+102%) - NPU 4card_cfg_u2: 249.04 ms/step (2.23x), vs pure Ulysses 272.18ms (+9.3%) - AllToAll overlap: NOT viable (overlap=2 is 26% slower, overlap=4/8 errors) - NPU still 1.62x faster than GPU H20 at best 8-card config Recommended config: parallelism=8, use_cfg_parallel=True, sp_ulysses_degree=4
Back-to-back same-script comparison results: Config Refactored PR#270 Diff 1card 661.29 ms 645.22 ms +2.5% 4card_pure_ulysses 273.28 ms 271.78 ms +0.6% 4card_cfg_u2 253.63 ms 250.66 ms +1.2% 8card_pure_ulysses 355.64 ms 357.93 ms -0.6% 8card_cfg_u4 173.31 ms 168.53 ms +2.8% Conclusion: Performance equivalent (within 3% noise). Refactoring preserved computation correctness with zero perf regression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.