Skip to content

[PAN2] Add layerwise NVFP4 inference for Blackwell - #53

Closed
Uxito-Ada wants to merge 10 commits into
Tele-AI:mainfrom
Uxito-Ada:feat/minimax-h3-image-nvfp4
Closed

Uxito-Ada wants to merge 10 commits into
Tele-AI:mainfrom
Uxito-Ada:feat/minimax-h3-image-nvfp4

Conversation

@Uxito-Ada

@Uxito-Ada Uxito-Ada commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add PAN2 layerwise NVFP4 conversion and loading for Blackwell GPUs.
  • Stream one quantized transformer block at a time while keeping the VAE on CUDA; activation quantization uses the TorchAO NVFP4 Triton path (MSLK Python-only package is supported).
  • Add official PAN2 prompt/config options to the benchmark and document conversion/inference examples.

Validation

The official PAN2 T2V example was used: 704x1248, 241 frames, 24 FPS, seed 123, swimmer prompt from inference/prompts/examples.json, and the PAN2 scheduler. BF16 could not fit on the 24 GiB RTX 5090. A real one-step NVFP4 run completed with 241 frames, 229.54 s end-to-end latency, 14.36 GiB peak allocated / 20.71 GiB reserved, and 1.05 generated FPS (0.044x realtime). The one-step output is attached below; it is a capacity/path validation, not a quality claim. A 50-step run was started and measured at roughly 83-92 s per denoise step, so the single-card block-offload profile is not realtime.

Official PAN2 NVFP4 RTX 5090 metrics

Download the official-config one-step NVFP4 MP4

Model feasibility

MiniMax-H3-Image and LightX2V H3 were evaluated before adding a misleading integration: both use the same 33.1B H3-Image DiT. Official H3-Image lists 66.28 GB BF16 DiT weights, a 66.7 GB BF16 Qwen3-VL text encoder, and a 2.34 GB VAE. NVFP4 can reduce the DiT to roughly 17-19 GB, but the complete pipeline still cannot reside in a 24 GiB RTX 5090. LightX2V's official single-GPU profile therefore enables model-level CPU offload; its four-GPU profiles disable offload.

Generated reports and media are PR attachments only; no model output is committed to the repository.

@Uxito-Ada
Uxito-Ada marked this pull request as draft November 18, 2026 06:08
@Uxito-Ada Uxito-Ada closed this Nov 18, 2026
@Uxito-Ada
Uxito-Ada deleted the feat/minimax-h3-image-nvfp4 branch November 18, 2026 07:20
@Uxito-Ada

Copy link
Copy Markdown
Collaborator Author

Realtime Video Generation Exploration on 5090 NVFP4

Hardware: NVIDIA GeForce RTX 5090 Laptop GPU, 24 GiB

Measurement: PAN2

Tested on the first official swimmer prompt from the PAN2 example.

  • Resolution: 704x1248
  • Frames / Target FPS: 241 / 24
  • Video duration: 10.04 s
  • Denoise steps: 50 (1-step path for validation)
  • Seed: 123
  • Scheduler / Prompt: Official PAN2 example

NVFP4 single-step results:

  • End-to-end latency (1 step): 229.54 s
  • Denoise time: 82.35 s
  • Peak allocated: 14.36 GiB
  • Peak reserved: 20.71 GiB
  • Output frames: 241
  • Generation FPS: 1.05
  • Realtime ratio: 0.0437x

For the 50-step run, per-step latency stabilized at 82.4–91.8 s after step 5. Linear projection puts full denoise at ~70–77 minutes, excluding final VAE decode. Realtime generation is not feasible.

Why CPU Offload

Layerwise NVFP4 transformer blocks for PAN2 sum to ~15 GiB. The full pipeline additionally requires:

  • Unquantized transformer embeddings, norms, and output layers
  • Qwen text encoder
  • VAE
  • Denoise activations, attention buffers, and CUDA allocator reservation

Keeping all transformer blocks resident on GPU leaves no headroom for the text encoder, VAE, and runtime working set. Block-wise CPU offload caps allocation at 14.36 GiB / 20.71 GiB reserved, but introduces heavy PCIe transfer and reinitialization overhead on every denoise step.

MiniMax-H3-Image / LightX2V H3 Estimation

Model specs: 33.1B DiT, 66.28 GB BF16 transformer, 66.7 GB BF16 Qwen3-VL text encoder, 2.34 GB VAE.
Theoretical lower bound for DiT converted to 4-bit 6.28 GB * 4 / 16 = 16.57 GB

With NVFP4 block scales, padding, unquantized layers and runtime buffers, practical DiT footprint is estimated at 17–19 GB.

Component BF16/FP32 Estimate after DiT NVFP4
H3-Image DiT 66.28 GB 17–19 GB
Qwen3-VL text encoder 66.7 GB 66.7 GB
VAE 2.34 GB 2.34 GB
Activation / working set N/A Several GB minimum

The full pipeline exceeds 24 GiB. Even with 4-bit quantization on the text encoder, total weights land around 17–19 + 16.7 + 2.34 = 36–38 GB before activations and buffers. A single 5090 cannot hold the full pipeline.

LightX2V’s official single-card H3-Image preset enables cpu_offload=true, text_encoder_cpu_offload=true, and vae_cpu_offload=true. It improves runtime efficiency but still cannot hit realtime.

Conclusion

RTX 5090 supports NVFP4, but its 24 GiB VRAM is too small for these 33B video DiT pipelines. PAN2 can run only with block-level CPU offload, which kills latency. NVFP4 on DiT alone is not enough to fit the full stack. This GPU works for offline validation, not low-latency realtime video generation. @lzx1413

For the next step, I will support PAN2/LightX2V-H3 first on H100 platform, where there is 80G memory enough and multi GPUs for acceleation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant