Add SPEED-Bench AL collectors for Qwen3.8-2.4T-A95B and Qwen3.8-Flash-Next - #2780
Draft
qiching wants to merge 8 commits into
Draft
Add SPEED-Bench AL collectors for Qwen3.8-2.4T-A95B and Qwen3.8-Flash-Next#2780qiching wants to merge 8 commits into
qiching wants to merge 8 commits into
Conversation
Adds the acceptance-length collectors for deepseek-ai/DeepSeek-V4-Pro-DSpark, following the Kimi-K3 DSpark collector (#2352). DSpark ships the draft inside a separate checkpoint, so unlike K3 there is no external draft head to download and no "model" key in the speculative-config. dsv4dspark_fp4_b300_vllm.sh is a copy of the DSV4 MTP collector with only the speculative-config changed (method dspark + draft_sample_method); every serve flag is left identical so the DSpark and MTP curves stay comparable, and it already matches the published DSpark recipe. Benjamin asked for greedy and probabilistic drafting to be characterized on DSV4-Pro. speedbench-al.yml resolves the collector purely as ${model-prefix}_fp4_b300_vllm.sh, so the second arm needs its own dispatchable file; dsv4dsparkprob_fp4_b300_vllm.sh delegates to the base collector with DRAFT_SAMPLE_METHOD=probabilistic rather than duplicating it, so the two arms cannot drift. rejection_sample_method is left at the vLLM default and exposed as an opt-in env knob: the K3 variant bundled it with probabilistic drafting, but the forced-AL config has to stay on a sampling method TRT-LLM also supports. The golden YAML and its README row follow in a separate PR once the collection run is reviewed.
…the pass The first collection run failed the preflight in 50s: DeepSeek-V4-Pro-DSpark is not staged at /scratch/models on the B300 cluster, so listing it in the launcher's STAGED_MODELS resolved MODEL_PATH to a read-only mount that does not hold the weights. Drop it from the list so MODEL_PATH resolves to the writable models dir and the collector's existing download block pulls the ~960 GB checkpoint once, on the first run. Once the weights are staged, adding the basename back is a one-line change that switches to the faster read-only mount. Also raise CONCURRENCY from 1 to 32. AL is a per-draft accept/reject property and is independent of batch size, and nothing here sets speculative_disable_by_batch_size, so drafting stays on and the curve stays comparable to the MTP one while the pass takes far less wall-clock — which matters more now that the first run also pays for the download.
The collection run aborted in the shim: on v0.25.1 serve.py already declares --chat-template-kwargs natively (vllm-project/vllm#44244 has landed), so the marker check short-circuited there, but datasets.py no longer matches the pre-#44244 anchor and the assertion killed the run before the first server start. Replace the shim with the Kimi-K3 collector's version, which probes all three pieces of native support (CLI option, speed_bench forward, CustomDataset unpack) and no-ops when they are present, patching only what an older image is missing. Verified against the current vLLM benchmark source: all three probes match, so the shim reports native support and returns without touching anything. Note dsv4_fp4_b300_vllm.sh still carries the old unconditional shim and would hit the same assertion if the MTP curve were ever re-collected on a v0.25.x image.
vllm-project/vllm#44244 is in the images this collector runs on, so the site-packages monkey-patching the older collectors carry is dead code. Drop it. Keep a preflight assertion rather than assuming, because one direction fails silently: if the CLI option exists but the speed_bench path does not forward it, the flag is accepted and ignored, every thinking_on prompt renders without thinking, and the cell writes a non-thinking AL under the thinking_on key. A missing CLI option would instead fail loudly at argument parsing. The assertion turns the silent case into an abort before the first server start. Verified against the current vLLM benchmark source: all three pieces are present, so the preflight passes and the run proceeds.
…d cell The DSpark=4 cell OOMed during warmup: the rejection sampler copies logits to fp32 at max_num_seqs * (1 + num_speculative_tokens) * vocab, which the default max_num_seqs of 1024 makes 2.47 GiB at 4 speculative tokens and 4.4 GiB at 8. The memory profiler does not account for it, so the KV cache is already sized before the allocation fails. The collection only drives 32 concurrent requests, so pin max_num_seqs to 64 and expose gpu-memory-utilization to trade KV cache for headroom if a higher level still runs out. The run also aborted at that cell instead of recording N/A and continuing: wait_for_server_ready exits rather than returns, so call it in a subshell.
…mega_moe The collector inherited EP_SIZE / DP_ATTENTION knobs from the MTP collector, but speedbench-al.yml exports EP_SIZE=1 and DP_ATTENTION=false for the whole matrix, so both recipe flags were silently dropped and the run was plain TP on the auto MoE backend. TP-sharding the FP4 experts cost 141.53 GiB per GPU against an 831 GiB checkpoint (1132 GiB across 8 ranks), roughly 37 GiB per GPU of sharding overhead on weights expert parallel keeps whole, which is most of why there was nothing left for warmup after the KV cache. Hard-code TEP8 instead of honoring knobs the workflow overrides. TP stays 8 to match the MTP collector; AL does not depend on expert placement either way.
0813 is now staged on the B300 shared FS (AIAOPS-853), so add its basename to STAGED_MODELS. MODEL_PATH now resolves to /scratch/models instead of triggering the one-off download to the writable models dir on the first collection.
Contributor
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add two SPEED-Bench AL collector scripts, following the existing DSV4/Qwen3.5 pattern:
qwen3.8_fp4_b300_vllm.sh— Qwen3.8-2.4T-A95B (NVFP4, TP8, thinking-only)qwen3.8flashnext_fp4_b300_vllm.sh— Qwen3.8-Flash-Next (BF16, TP4, thinking on/off)Sampling params, reasoning-effort levels (xhigh/medium/low), reasoning/tool-call parsers, and chat-template kwargs match the official model cards. Models are downloaded via
hf downloadat runtime (not staged).Test plan
speedbench-al.ymlfor both model-prefixes on B300