Repository navigation
Conversation
Realtime Video Generation Exploration on 5090 NVFP4Hardware: NVIDIA GeForce RTX 5090 Laptop GPU, 24 GiB Measurement: PAN2Tested on the first official swimmer prompt from the PAN2 example.
NVFP4 single-step results:
For the 50-step run, per-step latency stabilized at 82.4–91.8 s after step 5. Linear projection puts full denoise at ~70–77 minutes, excluding final VAE decode. Realtime generation is not feasible. Why CPU OffloadLayerwise NVFP4 transformer blocks for PAN2 sum to ~15 GiB. The full pipeline additionally requires:
Keeping all transformer blocks resident on GPU leaves no headroom for the text encoder, VAE, and runtime working set. Block-wise CPU offload caps allocation at 14.36 GiB / 20.71 GiB reserved, but introduces heavy PCIe transfer and reinitialization overhead on every denoise step. MiniMax-H3-Image / LightX2V H3 EstimationModel specs: 33.1B DiT, 66.28 GB BF16 transformer, 66.7 GB BF16 Qwen3-VL text encoder, 2.34 GB VAE. With NVFP4 block scales, padding, unquantized layers and runtime buffers, practical DiT footprint is estimated at 17–19 GB.
The full pipeline exceeds 24 GiB. Even with 4-bit quantization on the text encoder, total weights land around LightX2V’s official single-card H3-Image preset enables ConclusionRTX 5090 supports NVFP4, but its 24 GiB VRAM is too small for these 33B video DiT pipelines. PAN2 can run only with block-level CPU offload, which kills latency. NVFP4 on DiT alone is not enough to fit the full stack. This GPU works for offline validation, not low-latency realtime video generation. @lzx1413 For the next step, I will support PAN2/LightX2V-H3 first on H100 platform, where there is 80G memory enough and multi GPUs for acceleation. |
Summary
Validation
The official PAN2 T2V example was used:
704x1248,241frames,24 FPS, seed123, swimmer prompt frominference/prompts/examples.json, and the PAN2 scheduler. BF16 could not fit on the 24 GiB RTX 5090. A real one-step NVFP4 run completed with 241 frames, 229.54 s end-to-end latency, 14.36 GiB peak allocated / 20.71 GiB reserved, and 1.05 generated FPS (0.044x realtime). The one-step output is attached below; it is a capacity/path validation, not a quality claim. A 50-step run was started and measured at roughly 83-92 s per denoise step, so the single-card block-offload profile is not realtime.Download the official-config one-step NVFP4 MP4
Model feasibility
MiniMax-H3-Image and LightX2V H3 were evaluated before adding a misleading integration: both use the same 33.1B H3-Image DiT. Official H3-Image lists 66.28 GB BF16 DiT weights, a 66.7 GB BF16 Qwen3-VL text encoder, and a 2.34 GB VAE. NVFP4 can reduce the DiT to roughly 17-19 GB, but the complete pipeline still cannot reside in a 24 GiB RTX 5090. LightX2V's official single-GPU profile therefore enables model-level CPU offload; its four-GPU profiles disable offload.
Generated reports and media are PR attachments only; no model output is committed to the repository.