Repository navigation
Conversation
Add native H3-Image six-head PDD generation and reference editing, plus PAN2 text-to-video and first-frame conditioning through existing pipeline, stage, module-loading and worker contracts. Preserve checkpoint sampling order, conditioning layouts, FP32 modulation and rotary buffers, and output conversion. Reuse H3 parallel attention and implement PAN2 Ulysses/TP with padding removal, FP8 linear quantization and Sol-Attn. Support repeated model CPU offload with sharded weights. Keep H3 FP8 prefix/correction inputs contiguous at the existing operator call sites. Add no shared framework configuration or service schema. Provide runnable examples, source and checkpoint provenance, focused numerical regressions, and reproducible distributed/worker validation tools. Twenty new unit tests and 41 existing H3/FP8 regressions pass. Tiny SP2, TP2 and SP2xTP2 checks pass with FP8 GEMM, FP8 Sol-Attn and repeated CPU offload. Released conditioner and image VAE outputs match independent references exactly. Released-transformer generation verification remains in progress while its weights download; the validation document records this boundary explicitly.
Preserve FP32 values only during dtype conversion. Device-only moves retain dtype directly and avoid duplicating the full modulation weights on GPU during CPU offload. Validated with four PAN2 model tests; rerun released-checkpoint generation after the observed offload OOM.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance and Quality
All measurements below are real checkpoint runs on 4× NVIDIA H100 80 GB. Timings are end-to-end generation wall time unless noted otherwise.
LightX2V-H3 vs TeleFuser-H3
TeleFuser BF16 is 1.75× faster than the matched official run and uses 30.2% less peak memory. TeleFuser Q-SPA(FP8 + Sol-Attn) is 2.64× faster and uses 40.3% less peak memory.
xychart-beta title "H3 generation time Comparison (lower is better)" x-axis [Official, TF-BF16, TF-QSPA] y-axis "seconds" 0 --> 150 bar [137.760, 78.506, 52.275]xychart-beta title "H3 peak memory Comparison (lower is better)" x-axis [Official, TF BF16, TF-QSPA] y-axis "MiB" 0 --> 80000 bar [72916, 50907, 43501]PAN2 with TeleFuser
Ablation:
xychart-beta title "PAN2 generation time by TeleFuser configuration" x-axis [BF16, SP2, TP2, FP8-Sol, SP2-FP8, TP2-FP8, I2V-TP2, 10s-TP2] y-axis "seconds" 0 --> 400 bar [360.61, 226.28, 215.88, 304.21, 218.66, 206.64, 191.86, 338.50]The PAN2 generates the 10-second, 241 frames, video. This has similar inputs and checkpoint to official PAN2 example.
TeleFuser PAN2 generated outputs
PAN2 TP2 FP8 Sol output
pan2-tp2-fp8-sol.mp4
TP2 + FP8 + Sol-Attn, 121 frames.
PAN2 10s output
pan2-10s-tp2-fp8-sol.mp4
10 s + TP2 + FP8 + Sol-Attn, 241 frames.
Tests
pytest -q tests/unit/models/test_pan2_dit.py tests/unit/models/test_h3_image_pan2_conditioning.py tests/unit/pipelines/test_h3_image_pan2_sampling.pyArchitecture support
The MiniMax-H3 LightX2V path is documented as already supported and is covered by the matched benchmark above.
PAN2 uses TeleFuser's BaseModel abstraction, explicit conditional/unconditional CFG forwards, FP32 latent/noise accumulation, SP/TP process groups, staged module loading, and optional FP8/Sol-Attention execution.